Exam SRM formulas, and when to use each

Statistical learning

• Expected test MSE at a point = variance of the fit + squared bias + variance of the error.
• k-fold cross-validation: the average of the k fold MSEs. Leave-one-out for least squares: (1 ÷ n) Σ ((yᵢ − ŷᵢ) ÷ (1 − hᵢ))².

Least squares

• Simple regression: b₁ = Σ(x − x̄)(y − ȳ) ÷ Σ(x − x̄)², and b₀ = ȳ − b₁x̄.
• R² = 1 − RSS ÷ TSS. The residual standard error is √(RSS ÷ (n − p − 1)).
• t statistic: estimate ÷ standard error, on n − p − 1 degrees of freedom.
• Overall F: ((TSS − RSS) ÷ p) ÷ (RSS ÷ (n − p − 1)). Partial F for q dropped terms: ((RSS₀ − RSS) ÷ q) ÷ (RSS ÷ (n − p − 1)).
• A prediction interval adds the error variance to the variance of the fitted mean, so it is always wider than the confidence interval at the same x.
• Leverage averages (p + 1) ÷ n. VIF for a predictor = 1 ÷ (1 − R²) from regressing it on the others.

Generalized linear models

Using a GLM

• Log link: a coefficient b multiplies the mean by e^b for each unit of its predictor. Logit link: it multiplies the odds by e^b.
• A Poisson model of counts with exposure E uses ln(E) as an offset: the mean is E × e^(Xβ).
• Likelihood ratio test: the drop in deviance between nested models, compared with chi-square on the difference in parameters.

Time series

• Random walk: yₜ = yₜ₋₁ + cₜ. With drift μ, the k-step forecast is the last value + kμ, and its variance grows as kσ².
• AR(1): yₜ = β₀ + β₁yₜ₋₁ + εₜ, stationary when |β₁| < 1, with mean β₀ ÷ (1 − β₁) and lag-k autocorrelation β₁ᵏ.
• Single exponential smoothing: ŝₜ = (1 − w)yₜ + w ŝₜ₋₁; the forecast of every later value is the last smoothed value.
• ARCH(1): the conditional variance is ω + αε²ₜ₋₁; GARCH(1, 1) adds βσ²ₜ₋₁, with long-run variance ω ÷ (1 − α − β).

Trees and ensembles

• Regression tree: predict the mean of the training responses in the leaf; choose the split that most reduces RSS.
• Classification tree: Gini Σ p̂(1 − p̂), entropy −Σ p̂ ln p̂, or error 1 − max p̂, weighted by the share of observations in each child.
• Cost complexity pruning: minimize RSS + α × (number of leaves). A larger α gives a smaller tree.
• Bagging averages trees grown on bootstrap samples. A random forest considers m of the p predictors at each split, often m ≈ √p.
• Boosting fits small trees in sequence, each to the current residuals, shrunk by λ.

Principal components and clustering

• Proportion of variance explained by component m: its variance ÷ the sum of all components' variances. prcomp prints standard deviations, so square them first.
• A score is the sum of each loading times the matching centered (and scaled) variable.
• K-means alternates two steps, assign each point to its nearest centroid and recompute each centroid, until nothing changes. The within-cluster variation never rises.
• Linkage: complete uses the largest pairwise distance between clusters, single the smallest, average the mean, centroid the distance between centroids.