Class 6/Lab 1: Seeing Support Vector Machines
Interactive Shiny simulations — runs in your browser
Introduction
In class we built support vector machines one idea at a time: a hyperplane that separates two classes, the maximal margin classifier that picks the widest such hyperplane, the soft margin that tolerates some violations, and kernels that bend the boundary. This lab lets you see each idea on simulated data. Move a hyperplane by hand, push support vectors around, turn the cost dial, switch kernels.
There is no code to run here. Each section is a small Shiny app that runs entirely in your browser (via Shinylive and webR) — no local R install needed. The first app takes 10–30 seconds to start while R and the packages download; the rest start faster after that.
The real data analysis — the tumor example from the slides and variable selection with the lasso — is in Lab 2.
A warning about C before we start
The slides (and ISLR) write the soft margin with a budget \(C\):
\[\sum_{i=1}^n \epsilon_i \le C .\]
A large budget means more tolerance for points on the wrong side of the margin — a wider margin.
Every R package we use (kernlab, e1071, caret) instead uses a cost C: the price paid per unit of violation. A large cost means less tolerance — a narrower margin. The two run in opposite directions:
| slides / ISLR budget \(C\) | software cost C |
|
|---|---|---|
| wider margin, more violations, more support vectors | large | small |
| narrower margin, fewer violations, fewer support vectors | small | large |
Both labs use the software meaning. When you read C = 100 below, think “violations are expensive”, not “a big budget”. The apps fit SVMs with kernlab::ksvm(), the same engine caret’s svm* methods call in Lab 2.
1. A hyperplane is just a sign
With \(p = 2\) features a hyperplane is a line, \(f(X) = \beta_0 + \beta_1 X_1 + \beta_2 X_2 = 0\). The slides normalize \(\beta_1^2 + \beta_2^2 = 1\); we build that in by writing \(\beta_1 = \cos\theta\), \(\beta_2 = \sin\theta\), so the two sliders are the direction of the normal vector (the angle \(\theta\)) and the offset \(\beta_0\).
The shading shows the sign of \(f(X)\). A point with label \(y_i = \pm 1\) is on the correct side when \(y_i f(x_i) > 0\). When every point is on the correct side, you have a separating hyperplane, and — because \(\beta_1^2 + \beta_2^2 = 1\) — the smallest \(y_i f(x_i)\) is the margin \(M\): the distance from the line to the closest point.
Your job: find a separating hyperplane, then make \(M\) as large as you can. The title tells you the best achievable value.
#| '!! shinylive warning !!': |
#| shinylive does not work in self-contained HTML documents.
#| Please set `embed-resources: false` in your metadata.
#| standalone: true
#| viewerHeight: 640
library(shiny)
library(bslib)
library(ggplot2)
# Two well-separated clouds; y = -1 (lower left) and y = +1 (upper right)
set.seed(6)
n <- 20
sep <- data.frame(X1 = c(rnorm(n, 1, 0.5), rnorm(n, 3, 0.5)),
X2 = c(rnorm(n, 1, 0.5), rnorm(n, 3, 0.5)),
y = rep(c(-1, 1), each = n))
grid <- expand.grid(X1 = seq(-0.5, 4.5, length.out = 120),
X2 = seq(-0.5, 4.5, length.out = 120))
ui <- page_sidebar(
sidebar = sidebar(
sliderInput("theta", "angle θ (degrees)", min = 0, max = 355,
value = 0, step = 5),
sliderInput("beta0", "offset β₀", min = -5, max = 5,
value = -2, step = 0.05)
),
plotOutput("plot", height = "560px")
)
server <- function(input, output, session) {
output$plot <- renderPlot({
# The hyperplane you chose (beta1^2 + beta2^2 = 1)
b1 <- cos(input$theta * pi / 180)
b2 <- sin(input$theta * pi / 180)
f <- input$beta0 + b1 * sep$X1 + b2 * sep$X2
n_wrong <- sum(sep$y * f <= 0)
M <- min(sep$y * f)
grid$f <- input$beta0 + b1 * grid$X1 + b2 * grid$X2
headline <- if (n_wrong == 0) {
sprintf("Separating hyperplane! Your margin M = %.2f", M)
} else {
sprintf("%d point(s) on the wrong side (circled)", n_wrong)
}
ggplot() +
geom_raster(data = grid, aes(X1, X2, fill = f > 0), alpha = 0.25) +
geom_contour(data = grid, aes(X1, X2, z = f), breaks = 0,
colour = "black", linewidth = 1) +
geom_point(data = sep, aes(X1, X2, colour = factor(y)), size = 3) +
geom_point(data = sep[sep$y * f <= 0, ], aes(X1, X2), shape = 21,
size = 7, stroke = 1.2) +
scale_fill_manual(values = c(`FALSE` = "#3B4992", `TRUE` = "#EE0000"),
labels = c("f(X) < 0", "f(X) > 0"), name = NULL,
drop = FALSE) +
scale_colour_manual(values = c(`-1` = "#3B4992", `1` = "#EE0000"),
labels = c("y = -1", "y = +1"), name = NULL) +
coord_equal(xlim = c(-0.5, 4.5), ylim = c(-0.5, 4.5), expand = FALSE) +
labs(title = headline,
subtitle = "best possible margin: M = 0.49") +
theme_bw(base_size = 15) +
theme(legend.position = "bottom")
}, res = 96)
}
shinyApp(ui, server)Notice how many different lines separate these two clouds — infinitely many. Separation alone does not pick a classifier; that is why we ask for the one with the largest margin. (Hint if you are stuck: the best angle is about 40° and the best offset about −2.75.)
2. The maximal margin classifier and its support vectors
ksvm() with a linear kernel and a huge cost (C = 10000, so no violations are affordable) finds the maximal margin hyperplane for us. The solid line is the hyperplane; the dashed lines are the margins; circled points are the support vectors.
The slides claim the support vectors are the only points that matter. Test it: pick a point and slide it away from (positive) or toward (negative) the boundary. The grey line is the original hyperplane for comparison.
#| '!! shinylive warning !!': |
#| shinylive does not work in self-contained HTML documents.
#| Please set `embed-resources: false` in your metadata.
#| standalone: true
#| viewerHeight: 640
library(shiny)
library(bslib)
library(ggplot2)
library(kernlab)
set.seed(6)
n <- 20
sep <- data.frame(X1 = c(rnorm(n, 1, 0.5), rnorm(n, 3, 0.5)),
X2 = c(rnorm(n, 1, 0.5), rnorm(n, 3, 0.5)),
y = factor(rep(c(-1, 1), each = n)))
fit_mmc <- function(d) {
ksvm(y ~ X1 + X2, data = d, kernel = "vanilladot", kpar = list(),
C = 10000, scaled = FALSE)
}
margin_M <- function(m) { # M = 1 / ||beta||
w <- colSums(coef(m)[[1]] * xmatrix(m)[[1]])
1 / sqrt(sum(w^2))
}
m0 <- fit_mmc(sep)
w0 <- colSums(coef(m0)[[1]] * xmatrix(m0)[[1]])
u <- w0 / sqrt(sum(w0^2)) # unit normal to the hyperplane
f0 <- predict(m0, sep, type = "decision")[, 1]
grid <- expand.grid(X1 = seq(-0.5, 4.5, length.out = 150),
X2 = seq(-0.5, 4.5, length.out = 150))
grid$f0 <- predict(m0, grid, type = "decision")[, 1]
ui <- page_sidebar(
sidebar = sidebar(
radioButtons("which_point", "Move",
c("a support vector", "a point far from the margin")),
sliderInput("shift", "distance moved", min = -0.4, max = 1.5,
value = 0, step = 0.05)
),
plotOutput("plot", height = "560px")
)
server <- function(input, output, session) {
output$plot <- renderPlot({
# Choose the point: the first support vector, or the point farthest from the line
i <- if (input$which_point == "a support vector") alphaindex(m0)[[1]][1] else
which.max(abs(f0))
# Move it along the normal: positive = away from the boundary
moved <- sep
moved[i, c("X1", "X2")] <- sep[i, c("X1", "X2")] + input$shift * sign(f0[i]) * u
m1 <- fit_mmc(moved)
grid$f1 <- predict(m1, grid, type = "decision")[, 1]
sv <- moved[alphaindex(m1)[[1]], ]
ggplot() +
geom_contour(data = grid, aes(X1, X2, z = f0), breaks = 0,
colour = "grey60", linewidth = 1) +
geom_contour(data = grid, aes(X1, X2, z = f1), breaks = 0,
colour = "black", linewidth = 1) +
geom_contour(data = grid, aes(X1, X2, z = f1), breaks = c(-1, 1),
colour = "black", linetype = "dashed") +
geom_point(data = moved, aes(X1, X2, colour = y), size = 3) +
geom_point(data = sv, aes(X1, X2), shape = 21, size = 7, stroke = 1.2) +
geom_point(data = moved[i, ], aes(X1, X2), shape = 4, size = 5, stroke = 2) +
scale_colour_manual(values = c(`-1` = "#3B4992", `1` = "#EE0000"),
labels = c("y = -1", "y = +1"), name = NULL) +
coord_equal(xlim = c(-0.5, 4.5), ylim = c(-0.5, 4.5), expand = FALSE) +
labs(title = sprintf("Margin M = %.2f (was %.2f) | %d support vectors",
margin_M(m1), margin_M(m0), length(alphaindex(m1)[[1]])),
subtitle = "x marks the point you are moving") +
theme_bw(base_size = 15) +
theme(legend.position = "bottom")
}, res = 96)
}
shinyApp(ui, server)What you should see:
- Far-away point: slide it anywhere in its range — the hyperplane does not move at all. It is not a support vector, so it has no say.
- Support vector, pulled toward the boundary (negative): the margin shrinks immediately. The whole classifier is at the mercy of this one point.
- Support vector, pushed away (positive): the margin widens until another point takes over as a support vector.
That sensitivity is the maximal margin classifier’s weakness, and the reason the soft margin exists.
3. The soft margin: turning the cost dial
Now the classes overlap, so no separating hyperplane exists. The soft margin lets some points sit inside the margin, or even on the wrong side of the hyperplane. Slide the cost from cheap (left) to expensive (right), or press ▶ under the slider to sweep it automatically.
#| '!! shinylive warning !!': |
#| shinylive does not work in self-contained HTML documents.
#| Please set `embed-resources: false` in your metadata.
#| standalone: true
#| viewerHeight: 640
library(shiny)
library(bslib)
library(ggplot2)
library(kernlab)
set.seed(3)
n <- 50
ovl <- data.frame(X1 = c(rnorm(n, 1.2, 0.8), rnorm(n, 2.8, 0.8)),
X2 = c(rnorm(n, 1.2, 0.8), rnorm(n, 2.8, 0.8)),
y = factor(rep(c(-1, 1), each = n)))
grid <- expand.grid(X1 = seq(-1.5, 5.5, length.out = 150),
X2 = seq(-1.5, 5.5, length.out = 150))
ui <- page_sidebar(
sidebar = sidebar(
sliderInput("log10C", "log₁₀(cost C)", min = -2, max = 2,
value = 0, step = 0.25,
animate = animationOptions(interval = 1200))
),
plotOutput("plot", height = "560px")
)
server <- function(input, output, session) {
output$plot <- renderPlot({
C <- 10^input$log10C
m <- ksvm(y ~ X1 + X2, data = ovl, kernel = "vanilladot", kpar = list(),
C = C, scaled = FALSE)
w <- colSums(coef(m)[[1]] * xmatrix(m)[[1]])
grid$f <- predict(m, grid, type = "decision")[, 1]
sv <- ovl[alphaindex(m)[[1]], ]
ggplot() +
geom_contour(data = grid, aes(X1, X2, z = f), breaks = 0,
colour = "black", linewidth = 1) +
geom_contour(data = grid, aes(X1, X2, z = f), breaks = c(-1, 1),
colour = "black", linetype = "dashed") +
geom_point(data = ovl, aes(X1, X2, colour = y), size = 2.5) +
geom_point(data = sv, aes(X1, X2), shape = 21, size = 5, stroke = 0.8) +
scale_colour_manual(values = c(`-1` = "#3B4992", `1` = "#EE0000"),
labels = c("y = -1", "y = +1"), name = NULL) +
coord_equal(xlim = c(-1.5, 5.5), ylim = c(-1.5, 5.5), expand = FALSE) +
labs(title = sprintf("cost C = %.2f | margin width = %.2f",
C, 2 / sqrt(sum(w^2))),
subtitle = sprintf("%d support vectors (circled) | training error = %.0f%%",
nSV(m), 100 * error(m))) +
theme_bw(base_size = 15) +
theme(legend.position = "bottom")
}, res = 96)
}
shinyApp(ui, server)Sweep the slider end to end and watch three numbers move together:
cost C |
margin width | support vectors | training error |
|---|---|---|---|
| 0.01 (cheap violations) | 3.32 | 70 | 4% |
| 1 | 0.91 | 16 | 4% |
| 100 (expensive violations) | 0.59 | 10 | 3% |
- Cheap violations → wide margin, many support vectors. 70 of the 100 points help set the boundary, so no single point can drag it around: low variance, higher bias.
- Expensive violations → narrow margin, few support vectors. The fit bends to a handful of points near the boundary: low bias, higher variance.
- Training error barely changes (4% → 3%). So training error cannot pick
Cfor us — it will always favor expensive violations. We chooseCthe way we choose every tuning parameter: cross-validation. That is exactly whatcaret::train()does in Lab 2.
4. Kernels: when a straight line won’t do
Here class +1 sits inside a ring and class -1 outside it — no straight line can separate them. We fit on 200 training points and report the error on 1,000 fresh test points from the same process. Try each kernel; for the radial kernel, also slide \(\gamma\) (kernlab calls it sigma).
#| '!! shinylive warning !!': |
#| shinylive does not work in self-contained HTML documents.
#| Please set `embed-resources: false` in your metadata.
#| standalone: true
#| viewerHeight: 680
library(shiny)
library(bslib)
library(ggplot2)
library(kernlab)
make_ring <- function(n, noise) {
X1 <- runif(n, -2, 2)
X2 <- runif(n, -2, 2)
inside <- sqrt(X1^2 + X2^2) + rnorm(n, 0, noise) < 1.2
data.frame(X1, X2, y = factor(ifelse(inside, 1, -1), levels = c(-1, 1)))
}
grid <- expand.grid(X1 = seq(-2, 2, length.out = 150),
X2 = seq(-2, 2, length.out = 150))
ring_colours <- scale_colour_manual(
values = c(`-1` = "#3B4992", `1` = "#EE0000"),
labels = c("y = -1 (outside)", "y = +1 (inside)"), name = NULL)
ui <- page_sidebar(
sidebar = sidebar(
selectInput("kern", "kernel",
c("linear", "polynomial, degree 2", "polynomial, degree 3",
"radial")),
conditionalPanel(
"input.kern == 'radial'",
sliderInput("log10gamma", "log₁₀(γ)", min = -1, max = 2,
value = 0, step = 0.25)
),
sliderInput("noise", "label noise near the ring (SD)", min = 0.05,
max = 0.5, value = 0.15, step = 0.05)
),
navset_card_tab(
nav_panel("Original space: X₁, X₂",
plotOutput("fit_plot", height = "520px")),
nav_panel("Enlarged space: X₁², X₂²",
plotOutput("enlarged_plot", height = "520px"))
)
)
server <- function(input, output, session) {
ring_train <- reactive({ set.seed(4); make_ring(200, input$noise) })
ring_test <- reactive({ set.seed(5); make_ring(1000, input$noise) })
output$fit_plot <- renderPlot({
d <- ring_train()
gamma <- 10^input$log10gamma
m <- switch(input$kern,
"linear" = ksvm(y ~ ., d, kernel = "vanilladot",
kpar = list(), C = 1),
"polynomial, degree 2" = ksvm(y ~ ., d, kernel = "polydot",
kpar = list(degree = 2), C = 1),
"polynomial, degree 3" = ksvm(y ~ ., d, kernel = "polydot",
kpar = list(degree = 3), C = 1),
"radial" = ksvm(y ~ ., d, kernel = "rbfdot",
kpar = list(sigma = gamma), C = 1))
train_err <- mean(predict(m, d) != d$y)
test_err <- mean(predict(m, ring_test()) != ring_test()$y)
grid$pred <- predict(m, grid)
ggplot() +
geom_raster(data = grid, aes(X1, X2, fill = pred), alpha = 0.25) +
geom_point(data = d, aes(X1, X2, colour = y), size = 2) +
scale_fill_manual(values = c(`-1` = "#3B4992", `1` = "#EE0000"),
guide = "none") +
ring_colours +
coord_equal(expand = FALSE) +
labs(title = sprintf("%s%s", input$kern,
if (input$kern == "radial") sprintf(" (γ = %.2f)", gamma) else ""),
subtitle = sprintf("training error %.1f%% | test error %.1f%%",
100 * train_err, 100 * test_err)) +
theme_bw(base_size = 15) +
theme(legend.position = "bottom")
}, res = 96)
output$enlarged_plot <- renderPlot({
d <- ring_train()
ggplot(d, aes(X1^2, X2^2, colour = y)) +
geom_point(size = 2) +
geom_abline(intercept = 1.44, slope = -1, linetype = "dashed") +
ring_colours +
labs(x = expression(X[1]^2), y = expression(X[2]^2),
title = "The ring becomes (almost) linearly separable",
subtitle = "dashed line: X1² + X2² = 1.44, the true boundary") +
theme_bw(base_size = 15) +
theme(legend.position = "bottom")
}, res = 96)
}
shinyApp(ui, server)Things to try (at the default noise, 0.15):
| kernel | training error | test error |
|---|---|---|
| linear | 30.5% | 28.5% |
| polynomial, degree 2 | 7.0% | 7.4% |
| radial, γ = 1 | 8.5% | 6.3% |
| radial, γ = 100 | 3.0% | 21.0% |
- linear gives up and calls everything “outside” — one straight cut can’t carve out a disc, so it misclassifies the whole inside class.
- polynomial, degree 2 gets close to the best possible: the true boundary \(X_1^2 + X_2^2 = 1.2^2\) is a degree-2 curve. (Nothing reaches 0% — the labels near the ring are noisy on purpose.)
- radial, then slide \(\gamma\) all the way right. The training error keeps falling while the test error climbs as the boundary shrink-wraps individual points. Large \(\gamma\) = very local kernel = overfitting. \(\gamma\) is a flexibility dial, just like \(k\) in k-NN.
Why the degree-2 kernel works. Open the Enlarged space tab. The slides’ trick was to add transformed features and fit a linear classifier in the bigger space. On \(X_1^2\) and \(X_2^2\) instead of \(X_1\) and \(X_2\), a straight line does the job. The polynomial kernel gets the same result without ever building those columns: it only needs inner products, which is what makes kernels cheap even when the enlarged space is huge (or, for the radial kernel, infinite-dimensional).
Break the ring. Raise the noise slider to 0.4. The test error of the degree-2 kernel climbs, and switching kernels does not bring it back down. Why can no kernel fix this?
5. More than two classes
SVMs are binary by construction. The slides gave two ways round that: one-versus-all (fit \(K\) classifiers, each class against the rest) and one-versus-one (fit all \(\binom{K}{2}\) pairwise classifiers and let them vote). kernlab uses one-versus-one. With three iris species that means \(\binom{3}{2} = 3\) binary SVMs inside one fitted model — the subtitle counts them from the fitted object.
The app shows the combined vote as three-colored decision regions. The two feature pairs give very different problems:
#| '!! shinylive warning !!': |
#| shinylive does not work in self-contained HTML documents.
#| Please set `embed-resources: false` in your metadata.
#| standalone: true
#| viewerHeight: 640
library(shiny)
library(bslib)
library(ggplot2)
library(kernlab)
ui <- page_sidebar(
sidebar = sidebar(
radioButtons("features", "features", c("petal", "sepal")),
radioButtons("kernel", "kernel", c("linear", "radial"))
),
plotOutput("plot", height = "560px")
)
server <- function(input, output, session) {
output$plot <- renderPlot({
vars <- if (input$features == "petal") c("Petal.Length", "Petal.Width") else
c("Sepal.Length", "Sepal.Width")
d <- data.frame(x1 = iris[[vars[1]]], x2 = iris[[vars[2]]],
Species = iris$Species)
set.seed(1)
m <- if (input$kernel == "linear") {
ksvm(Species ~ x1 + x2, data = d, kernel = "vanilladot", kpar = list(),
C = 1, cross = 10)
} else {
ksvm(Species ~ x1 + x2, data = d, kernel = "rbfdot", C = 1, cross = 10)
}
grid <- expand.grid(x1 = seq(min(d$x1) - 0.3, max(d$x1) + 0.3, length.out = 150),
x2 = seq(min(d$x2) - 0.3, max(d$x2) + 0.3, length.out = 150))
grid$pred <- predict(m, grid)
species_cols <- c(setosa = "#3B4992", versicolor = "#EE0000",
virginica = "#008B45")
ggplot() +
geom_raster(data = grid, aes(x1, x2, fill = pred), alpha = 0.25) +
geom_jitter(data = d, aes(x1, x2, colour = Species, shape = Species),
width = 0.03, height = 0.03, size = 2.5) +
scale_colour_manual(values = species_cols) +
scale_fill_manual(values = species_cols, guide = "none") +
coord_cartesian(expand = FALSE) +
labs(x = vars[1], y = vars[2],
title = sprintf("%s kernel on %s features | 10-fold CV error %.1f%%",
input$kernel, input$features, 100 * cross(m)),
subtitle = sprintf("one-versus-one: %d pairwise classifiers in this model",
length(coef(m)))) +
theme_bw(base_size = 15) +
theme(legend.position = "bottom")
}, res = 96)
}
shinyApp(ui, server)- Petal features: both kernels separate the three species almost perfectly (10-fold CV error 4.7% linear, 4.0% radial). Setosa sits far off on its own; the two classifiers that involve it are easy.
- Sepal features: the error jumps to about 21% (linear) and 25% (radial). Versicolor and virginica overlap heavily on sepal measurements, and no kernel can separate points that sit on top of each other — the radial kernel just draws a more contorted boundary through the overlap and does worse. Better features beat a fancier kernel.
Things worth remembering
- The margin is decided by a few points. Only the support vectors shape the SVM; everything else could move (within limits) without changing the fit.
- Cost
Cin software is the opposite of the budget \(C\) on the slides. Large softwareC= expensive violations = narrow margin = low bias, high variance. Tune it by cross-validation, never by training error. - Kernels buy flexibility, and flexibility has to be earned. On the ring a degree-2 kernel is exactly right; when the labels themselves are noisy, no kernel helps.
- Multi-class SVMs are a vote.
kernlabfits one binary SVM per pair of classes.
Next: Lab 2 fits these models to real tumor data and uses the lasso to choose which features to keep.