\hrule

## Instructions

* This file serves as a template for your assignment. \textcolor{red}{\textbf{Remember to set eval = TRUE below when compiling the report.}}
```{r setup}
knitr::opts_chunk$set(echo = TRUE, eval = FALSE,
                      fig.height = 4, fig.width = 7, fig.align = "center",
                      message=FALSE)
```
* Use comments to annotate and explain your code whenever possible.
* This assignment covers **Regression II** (non-linear regression, splines, kernel trick) and **Classification I** (logistic regression, generative models). It uses two data sets: `Boston Housing` (`assign1_regData.csv`, continued from Assignment 1) for Part A, and `Pima Diabetes` (`assign2_diabetes.csv`) for Part B.
\newline
\hrule

## Theory (12 points)

### T1. Continuity and smoothness of cubic splines (12 points)

ISLR Problem 7.9.1, page 321. A cubic regression spline with one knot at $\xi$ can be obtained using a basis of the form $x, x^2, x^3, (x-\xi)_+^3$, where $(x - \xi)_+^3 = (x-\xi)^3$ if $x > \xi$ and equals $0$ otherwise. We will now show that a function of the form
$$f(x) = \beta_0 + \beta_1 x + \beta_2 x^2 + \beta_3 x^3 + \beta_4 (x-\xi)_+^3$$
is indeed a cubic regression spline, regardless of the values of $\beta_0, \beta_1, \beta_2, \beta_3, \beta_4$.

(a) Find a cubic polynomial $f_1(x) = a_1 + b_1 x + c_1 x^2 + d_1 x^3$ such that $f(x) = f_1(x)$ for all $x \leq \xi$. Express $a_1, b_1, c_1, d_1$ in terms of $\beta_0, \beta_1, \beta_2, \beta_3, \beta_4$.

(b) Find a cubic polynomial $f_2(x) = a_2 + b_2 x + c_2 x^2 + d_2 x^3$ such that $f(x) = f_2(x)$ for all $x > \xi$. Express $a_2, b_2, c_2, d_2$ in terms of $\beta_0, \beta_1, \beta_2, \beta_3, \beta_4$. We have now established that $f(x)$ is a piecewise polynomial.

(c) Show that $f_1(\xi) = f_2(\xi)$, i.e. that $f(x)$ is continuous at $\xi$.

(d) Show that $f_1'(\xi) = f_2'(\xi)$, i.e. that $f'(x)$ is continuous at $\xi$.

(e) Show that $f_1''(\xi) = f_2''(\xi)$, i.e. that $f''(x)$ is continuous at $\xi$.

Together, (c), (d), and (e) establish that $f(x)$ is indeed a cubic spline.

## Application, Part A: Regression II — Non-linear modeling (38 points)
Dataset: `Boston Housing`, saved as `assign1_regData.csv` (same data as Assignment 1).

```{r, echo = TRUE, eval = TRUE, message=FALSE}
# Load required libraries and dataset
pacman::p_load(tidyverse, caret, splines, gam)
set.seed(42)
data <- read_csv("assign1_regData.csv") |> drop_na()
data$CHAS <- as.factor(data$CHAS)

train_index <- createDataPartition(data$MEDV, p = 0.8, list = FALSE)
train <- data[train_index, ]
test  <- data[-train_index, ]
```

### B1. Polynomial Regression (5 + 5 + 5 = 15 points)

1. Consider a polynomial regression with `LSTAT` as the only predictor. Use 10-fold CV on the training data to select the optimal polynomial degree (1 to 10). Plot how CV error varies with polynomial degree.
```{r}
# your code here
```

2. Report the optimal degree and the test set RMSE for that model.
```{r}
# your code here
```

3. Plot the fitted polynomial curve over a scatterplot of `MEDV` vs `LSTAT`. Does the relationship look like it needed a non-linear fit, or would a straight line have done nearly as well?
```{r}
# your code here
```

### B2. Splines (5 + 5 = 10 points)

1. Fit a natural spline (natural spline = cubic spline + linearity constraint in the tails) with `RM` as the predictor, using 4 degrees of freedom. Report the test set RMSE.
```{r}
# your code here
```

2. Compare this test RMSE to a simple linear fit of `MEDV` on `RM` alone. Did the spline improve prediction? Briefly explain, in terms of the shape of the `RM`-`MEDV` relationship, why or why not.
```{r}
# your code here
```

### B3. Generalized Additive Model (GAM) (8 + 5 = 13 points)

1. Fit a GAM for `MEDV` using natural splines for both `LSTAT` and `RM` (4 df each), plus `CHAS` as a linear term. Report the test set RMSE.
```{r}
# your code here
```

2. Compare the test RMSE of this GAM to the polynomial model from B1 and the spline model from B2. Which model would you recommend, and why?
```{r}
# your code here
```

## Application, Part B: Classification I — Logistic regression and LDA (40 points)
Dataset: `Pima Diabetes`, saved as `assign2_diabetes.csv`.

This is a classic dataset of `n = 768` women of Pima Indian heritage, used to predict the onset of diabetes within five years based on diagnostic measurements. Your goal is to build and compare two classifiers: logistic regression (a discriminative model) and linear discriminant analysis (a generative model). **Please set a random seed (`set.seed(42)`) before any step involving data splitting or randomization.**

- `Pregnancies`: Number of times pregnant
- `Glucose`: Plasma glucose concentration (2-hour oral glucose tolerance test)
- `BloodPressure`: Diastolic blood pressure (mm Hg)
- `SkinThickness`: Triceps skin fold thickness (mm)
- `Insulin`: 2-Hour serum insulin (mu U/mL)
- `BMI`: Body mass index
- `DiabetesPedigreeFunction`: A function that scores likelihood of diabetes based on family history
- `Age`: Age in years
- `Outcome`: 1 if the patient developed diabetes, 0 otherwise

```{r, echo = TRUE, eval = TRUE, message=FALSE}
pacman::p_load(tidyverse, caret, MASS, pROC)
set.seed(42)
diab <- read_csv("assign2_diabetes.csv") |> drop_na()
diab$Outcome <- factor(diab$Outcome, levels = c(0, 1), labels = c("No", "Yes"))
```

### C1. Data Exploration & Splitting (5 + 5 = 10 points)

1. Use `summary()` to explore the dataset. Comment on any variables that look like they may have implausible zero values (e.g. `Glucose` or `BMI` of zero) and briefly explain why that matters for the analysis (you do not need to fix it — just note the issue).
```{r}
# your code here
```

2. Randomly split the data into a training set (75%) and a test set (25%), stratified by `Outcome` (use `caret::createDataPartition`, `seed = 42`).
```{r}
# your code here
```

### C2. Logistic Regression (5 + 5 + 5 = 15 points)

1. Fit a logistic regression model predicting `Outcome` from all eight predictors on the training data. Which predictors are statistically significant at the 0.05 level?
```{r}
# your code here
```

2. Using a 0.5 probability threshold, generate class predictions on the test set and report the confusion matrix (`caret::confusionMatrix`).
```{r}
# your code here
```

3. Plot the ROC curve for this model on the test set and report the AUC.
```{r}
# your code here
```

### C3. Linear Discriminant Analysis & Comparison (5 + 5 + 5 = 15 points)

1. Fit an LDA model with the same eight predictors on the training data, and generate class predictions on the test set.
```{r}
# your code here
```

2. Report the LDA confusion matrix and AUC (using the posterior probability of `Yes`) on the test set.
```{r}
# your code here
```

3. Interpretation: *Compare logistic regression and LDA on accuracy, sensitivity, specificity, and AUC. LDA assumes predictors are approximately multivariate normal within each class and share a common covariance matrix — do you expect that assumption to hold reasonably well for this data? Which model would you recommend for this problem, and why?*