F5 โ€” Cross-Validation in Practice

1 Concept: Using CV for Model Selection

1.1 The one-standard-error rule

When selecting the best model via CV, the minimum CV error may not be the best choice. The 1-SE rule selects the simplest model whose CV error is within one standard error of the minimum.

\[d^* = \min \{d : \text{CV}(d) \leq \text{CV}(d_{\min}) + \text{SE}(d_{\min})\}\]

This guards against selecting an unnecessarily complex model when a simpler one performs nearly as well.

1.2 CV in glmnet: choosing ฮป

For ridge and LASSO regression, cv.glmnet() automates k-fold CV over a grid of \(\lambda\) values and returns:

  • lambda.min: the \(\lambda\) minimising CV error
  • lambda.1se: the largest \(\lambda\) within 1 SE of the minimum (more regularisation, simpler model)

In practice, lambda.1se is often preferred for parsimony.

1.3 Pitfalls

  1. Information leakage: never standardise predictors on all data before CV โ€” standardise inside each fold
  2. Nested CV: if you select both the model class and its hyperparameters, you need outer CV for honest evaluation
  3. Classification: use stratified folds to preserve class proportions across folds

2 Quiz

The 1-SE model achieves lower test error than lambda.min in almost every case
Models within 1 SE are statistically tied โ€” the 1-SE rule picks the simpler one by Occam's razor
Minimising CV error directly inflates optimism; the 1-SE adjustment corrects this
glmnet enforces the 1-SE rule and no alternative selection criterion is available
CV error estimates have uncertainty (measured by the SE across folds). If two models' CV errors are within 1 SE of each other, we cannot distinguish them statistically. The 1-SE rule then picks the simpler one โ€” Occam's razor applied to model selection. This is often more robust when deployed on new data.
Minimum CV error is at lambda = 0.08; the model at lambda = 0.01 is less penalised
lambda.1se = 0.08 gives a sparser model within 1 SE of the minimum CV error
A larger lambda = 0.08 always yields lower test error than the smaller lambda = 0.01
LASSO defines lambda.1se as always larger than lambda.min by mathematical necessity
lambda.min = 0.01 minimises the cross-validated MSE. lambda.1se = 0.08 is the largest lambda whose CV MSE is within 1 SE of the minimum โ€” meaning it produces a sparser (more zeros) but statistically comparable model. Higher lambda โ†’ more shrinkage โ†’ fewer non-zero coefficients in LASSO.

3 Worked Example: cv.glmnet on Hitters

library(ISLR2)
library(glmnet)

data(Hitters)
Hitters <- na.omit(Hitters)

X <- model.matrix(Salary ~ ., data=Hitters)[, -1]
y <- Hitters$Salary

set.seed(42)
cv_lasso <- cv.glmnet(X, y, alpha=1, nfolds=10)

cat("lambda.min:", round(cv_lasso$lambda.min, 4), "\n")
lambda.min: 2.2203 
cat("lambda.1se:", round(cv_lasso$lambda.1se, 4), "\n")
lambda.1se: 83.5934 
cat("\nNumber of non-zero coefficients:\n")

Number of non-zero coefficients:
cat("  at lambda.min:", sum(coef(cv_lasso, s="lambda.min") != 0) - 1, "\n")
  at lambda.min: 14 
cat("  at lambda.1se:", sum(coef(cv_lasso, s="lambda.1se") != 0) - 1, "\n")
  at lambda.1se: 5 
plot(cv_lasso)

cv_ridge <- cv.glmnet(X, y, alpha=0, nfolds=10)
cat("Ridge lambda.min:", round(cv_ridge$lambda.min, 4), "\n")
Ridge lambda.min: 25.5282 
cat("Ridge lambda.1se:", round(cv_ridge$lambda.1se, 4), "\n")
Ridge lambda.1se: 2935.124 
# Compare LASSO vs Ridge test MSE
cat("\nTest MSE (LASSO, lambda.min):",
    round(min(cv_lasso$cvm), 2), "\n")

Test MSE (LASSO, lambda.min): 115250.2 
cat("Test MSE (Ridge, lambda.min):",
    round(min(cv_ridge$cvm), 2), "\n")
Test MSE (Ridge, lambda.min): 115726.1

4 Guided Practice

R is loading โ€” exercises interactive once ready.

  • coef(cv_fit, s="lambda.1se") returns a sparse matrix โ€” use !=0 to find non-zero entries
  • The true model has non-zero coefficients at X1 (ฮฒ=50), X3 (ฮฒ=30), X9 (ฮฒ=โˆ’20)
  • Does LASSO recover them? Compare with lambda.min
NoteFurther reading

James et al. (2021). An Introduction to Statistical Learning. โ€” Chapter 6.2 (LASSO + CV) and Chapter 5.1 (CV theory)