Class 6/Lab 2: SVMs and Variable Selection on Real Tumor Data

Live in-browser R — edit and re-run any cell

Published

October 2, 2026

Introduction

Lab 1 showed the SVM ideas on simulated data: hyperplanes, support vectors, the cost dial, kernels. This lab puts them to work on real tumor biopsies, in two parts:

  1. The tumor example from the slides. Real biopsy data, the same two features (cell size and cell shape uniformity), then all nine, across three kernels.
  2. Variable selection for classification. Thirty correlated tumor features: ordinary logistic regression breaks, and the lasso picks a handful.

Every code cell runs live in your browser via webR — no local R install needed. Packages and data load once when the page opens; give it a few seconds on first load. Cells run when you click Run Code. Run them in order from the top of each part, because later cells use objects made by earlier ones.

Remember the warning from Lab 1: the software cost C runs opposite to the slides’ budget \(C\). Large software C = violations are expensive = narrow margin.

We fit every SVM with caret::train(), following Lab 5: method = "svmLinear", "svmPoly", and "svmRadial" all call kernlab underneath.

Part 1 — The tumor example from the slides

The slides motivated SVMs with benign vs. malignant tumors plotted on cell size uniformity and cell shape uniformity. Those come from the Wisconsin breast-cancer biopsy data, which ships with the MASS package as biopsy: 699 biopsies, nine cytology scores each rated 1–10, and the diagnosis.

683 complete biopsies, 65% benign. The scores are whole numbers from 1 to 10, so many patients share exactly the same (size, shape) pair — we jitter the points slightly so they don’t stack on top of each other:

Mostly separable, with a band of overlap in the middle — a job for the soft margin.

Split, then tune the cost by cross-validation

Cross-validation picks C = 8, but look at the y-axis: accuracy only runs from 0.934 to 0.951 across a 1,000-fold range of C. On this data the cost barely matters. That is common — and it’s why we tune on a log-spaced grid rather than agonizing over one value.

The fitted boundary

predict() on a grid of (size, shape) values paints the regions the SVM assigns to each class. caret applies the same centering and scaling to the grid that it applied to the training data.

How well does it do on the test set?

As in Lab 5 we set positive = "Malignant", so sensitivity is the share of cancers we catch.

With just two features the linear SVM gets 95.3% of the 170 test biopsies right, catching 56 of 59 cancers (sensitivity 0.949) while clearing 106 of 111 benign tumors (specificity 0.955).

All nine features, three kernels

Now use every score, and compare the three kernels from class. All three go through the same train() call; only method changes. This is the heaviest cell in Part 1 (the polynomial grid fits 270 models) — give it a moment.

  • More features help: going from two scores to nine lifts accuracy from 0.953 to 0.971 and sensitivity from 0.949 to 0.983 (one missed cancer instead of three).
  • More flexible kernels don’t: linear, polynomial, and radial land within one test biopsy of each other. Cross-validation even picked the smallest cost on the grid for the linear kernel — the widest, most forgiving margin.
  • The lesson from the ring in Lab 1, in reverse: a kernel only pays off when the true boundary is curved. These cytology scores rise together as a tumor turns malignant, so a straight cut is already about as good as it gets. Try the linear kernel first; reach for a kernel only when CV says it helps.

Part 2 — Variable selection for classification

Everything so far used a few hand-picked features. Real problems often come with many correlated candidates, and we would like the data to tell us which ones matter. Back to the Wisconsin Diagnostic (WDBC) data from Lab 5, this time with all 30 features: ten nuclear measurements, each recorded as a mean, a standard error, and a “worst” value.

Why not just throw everything into logistic regression?

R warns that the algorithm did not converge and that fitted probabilities are numerically 0 or 1. The training tumors are perfectly separable in 30 dimensions, so the likelihood keeps improving as the coefficients grow — the median coefficient is about 500 and the largest about 44,000, on the log-odds scale. These estimates mean nothing, and (spoiler) the model does worst of everything we’ll try on the test set.

The lasso: logistic regression with a penalty

The fix is the same one we used for regression in Lab 3: add a penalty. The lasso maximizes

\[\underbrace{\sum_{i=1}^n \log L(\beta;\, y_i, x_i)}_{\text{logistic log-likelihood}} \; - \; \lambda \sum_{j=1}^{p} |\beta_j| .\]

The absolute-value penalty does two jobs at once: it keeps the coefficients finite, and it sets many of them exactly to zero — so fitting the model is selecting variables. \(\lambda\) controls how many survive. glmnet fits the whole path of \(\lambda\) values in one call (alpha = 1 is the lasso; alpha = 0 would be ridge).

The features are on very different scales (areas in the hundreds, smoothness around 0.1), and the penalty treats every \(\beta_j\) alike, so we standardize first, using the training means and SDs for both sets.

Read the plot right to left: at large \(\lambda\) (right) every coefficient is zero; as \(\lambda\) shrinks, features enter one at a time. The numbers along the top count how many are in the model.

Pick your own λ

Choose a \(\lambda\) and see which features survive and how the test accuracy responds. Change log10_lambda, then click Run Code again. This cell reuses lasso_path, X_test, and wtest from the cells above.

A few settings to try:

log₁₀(λ) features kept test accuracy
−3.5 21 97.9%
−2.0 8 97.9%
−1.0 4 95.1%
−0.5 2 69.7%

From 21 features down to 8 the test accuracy does not move: those extra 13 features were redundant. Around 6 features it starts to slip, and at 2 the model has thrown away real signal. Somewhere in between is the smallest model that is still as good as the big one — which is what cross-validation will find.

Letting cross-validation choose λ

cv.glmnet() runs 10-fold CV along the whole path. It reports two choices: lambda.min (the lowest CV error) and lambda.1se (the largest \(\lambda\) within one standard error of that minimum — the simplest model that is statistically as good). For selection we usually prefer lambda.1se.

The lambda.1se model keeps 10 of the 30 features. Notice what’s there: mostly “worst” measurements (the most abnormal cells in the image) and the concavity / concave points family — the irregular, indented cell outlines a pathologist looks for. Of radius, perimeter, and area — three near-copies of “how big is the nucleus” — the lasso keeps area_worst and radius_worst but not perimeter: with correlated features, the lasso tends to keep one or two representatives and drop the rest.

Does selecting features cost us anything?

Compare, on the same 142 test tumors: the unpenalized logistic model, the lasso, ridge (penalized but no selection), and a linear SVM trained on all 30 features vs. only the lasso’s picks.

  • The unpenalized model is worst (0.944) — the inflated coefficients from above do real damage on new data.
  • Lasso with 10 features = ridge with 30 = SVM with 30 (all 0.979). We got a model a third the size at no cost in accuracy, and one that names which measurements matter.
  • The SVM on the lasso’s 10 features is one tumor behind (0.972). Variable selection by one method can be handed to another.
  • Everything — the standardization, the λ choice, the selected list — came from the training tumors only. Selecting features on all the data and then “testing” on part of it leaks information and makes the test accuracy look better than it is.

A connection worth knowing

The SVM is secretly a penalized method too. ISLR §9.5 shows the support vector classifier solves

\[\min_{\beta}\; \sum_{i=1}^n \max\big[0,\, 1 - y_i f(x_i)\big] \;+\; \lambda \sum_{j=1}^p \beta_j^2 ,\]

a hinge loss plus a ridge penalty — and \(\lambda\) plays the role of \(1/C\) for the software cost. Swap the hinge loss for the logistic loss and you have ridge logistic regression; swap the ridge penalty for the lasso penalty and you have the lasso. “Regularization for classification” is one recipe — loss + penalty — with interchangeable parts.

Try it yourself

1. Elastic net. alpha between 0 and 1 mixes the lasso and ridge penalties. Correlated features (radius, perimeter, and area are nearly the same measurement) tend to enter together under the elastic net, where the lasso picks one and drops the rest. Fill in an alpha and compare how many features survive — and whether the test accuracy changes at all.

2. A different split. In Part 1, change set.seed(2155) before createDataPartition() to another number and re-run the part. Does the linear kernel still hold its own against the radial one?

Things worth remembering

  • Cost C in software is the opposite of the budget \(C\) on the slides. Tune it by cross-validation on a log-spaced grid, never by training error.
  • Try the linear kernel first. On the tumor data a straight line is already as good as anything fancier; reach for a kernel only when CV says it helps.
  • With many correlated features, penalize. Unpenalized logistic regression broke on 30 correlated features; the lasso kept the coefficients sane and selected a short list.