Class 2/Lab1: Training, Evaluating, and Resampling Models

Live in-browser R — edit and re-run any cell

Published

September 4, 2026

Introduction

In this lab, we will build the toolkit for training and evaluating statistical learning models. Every code cell below runs live in your browser via webR — no local R install needed. Edit any cell and re-run it to see what changes.

Packages load once when the page opens (see the webr: packages: list in the YAML front matter above) — give it a few seconds on first load.

Using simulated data, we will:

  1. Calculate common performance metrics (RMSE, Accuracy, Silhouette Score, Explained Variance) by hand, so the formulas behind caret/yardstick output stop being a black box.
  2. Fit polynomials of increasing degree to the same data and watch training error keep falling while test error eventually turns back up — the bias-variance tradeoff.
  3. Use the bootstrap to estimate the standard error and a confidence interval for a statistic without relying on a textbook formula.

Evaluation Metrics — RMSE (Regression)

RMSE measures the average magnitude of the errors between predicted and actual values.

\[RMSE = \sqrt{\frac{1}{n}\sum_{i=1}^{n}(predicted_i - actual_i)^2}\]

Try it live: change set.seed(42) to a different number and re-run all three cells above — watch RMSE change.

Accuracy (Classification)

Accuracy is the proportion of correct predictions among all cases evaluated — the most common classification metric, though it can be misleading on imbalanced data.

\[Accuracy = \frac{True\,Positives + True\,Negatives}{Total\,Samples}\]

Live-coding demo: the blank marks the one piece of the accuracy formula left to fill in — true negatives, added alongside true positives in the numerator.

Silhouette Score (Clustering)

The silhouette score measures how similar a point is to its own cluster (cohesion) versus the nearest other cluster (separation). It ranges from -1 to +1; higher is better.

\[s(i) = \frac{b(i) - a(i)}{\max(a(i), b(i))}\]

where \(a(i)\) is the average distance from point \(i\) to other points in its own cluster, and \(b(i)\) is the average distance from \(i\) to points in the nearest neighboring cluster.

Try it live: change centers = 2 to centers = 3 or 4 in the first cell and re-run both cells — watch the average silhouette score change as the clustering gets forced into the wrong number of groups.

Explained Variance (Dimensionality Reduction)

In PCA, explained variance is the proportion of total variance captured by each principal component — used to decide how many components to keep.

Try it live: change cov_matrix <- matrix(c(5, 4, 4, 5), nrow = 2) to matrix(c(5, 0, 0, 5), nrow = 2) (zero covariance) and re-run all three cells — watch the first component’s explained variance drop from dominant to about 50%, since the features are no longer correlated.

Bias-Variance Tradeoff

Here we generate data from a known nonlinear function, then fit polynomials of increasing degree and track training vs. test error. Low-degree models underfit (high bias); high-degree models overfit (high variance).

Live-coding demo: fill in the blank with a low degree (e.g. 1) — model underfits. Re-run with a high degree (e.g. 10) — model overfits. This #| exercise: blank is the code-blanks pattern: everything else is given, only the one concept-defining value is left for the class to fill in live.

Interpretation: Training error decreases (almost) monotonically as we add flexibility — the model can always fit the training points more closely. Test error decreases at first as bias drops, then rises again once the model starts fitting noise instead of signal. The degree at which test error is minimized is the sweet spot in the bias-variance tradeoff for this dataset.

The Bootstrap

The bootstrap estimates the sampling distribution of a statistic by resampling the observed data with replacement, without relying on a closed-form formula for its standard error.

Interpretation: The bootstrap SE and CI closely match what the Central Limit Theorem predicts for the mean of a normal sample (\(\sigma/\sqrt{n}\)), which is why the bootstrap is trustworthy here — and why it’s most useful for statistics (medians, correlations, model coefficients) where no such closed-form formula exists.

Try it live: change statistic_fn <- mean to statistic_fn <- sd or statistic_fn <- median and re-run both cells — the whole pipeline updates. The standard deviation’s bootstrap distribution is noticeably right-skewed in small samples, unlike the mean’s.