Basics of Statistical Learning

Judge a model by its test error, not its training error. More flexibility always lowers training error; test error falls, then rises as variance takes over.

A lesson for this topic

Question 1

A claim-triage classifier has sensitivity 80% and specificity 90% for detecting claims that will exceed the reinsurance retention. Of all claims, 5% exceed the retention.

Calculate the precision of the classifier: the proportion of claims it flags that actually exceed the retention.

Question 2

An actuary fits a logistic regression of whether a policy filed a claim on its annual mileage, using 12 policies. The fitted claim probabilities are below.

Policy	Annual mileage (000s)	Claim filed	Fitted probability
1	5	No	0.160
2	7	No	0.203
3	8	Yes	0.228
4	10	No	0.283
5	11	No	0.314
6	13	Yes	0.379
7	14	No	0.414
8	16	No	0.486
9	17	Yes	0.523
10	19	Yes	0.595
11	21	No	0.663
12	24	Yes	0.753

The actuary predicts a claim for every policy whose fitted probability exceeds 0.40.

Calculate the training sensitivity of this classifier.

Question 3

A K-nearest neighbors classifier with K = 3 predicts whether a policyholder lapses from the number of years they have been insured, with distance measured as the absolute difference in years. The training data are:

Years insured	1	2	4	5	7	8	10	11	13	15
Outcome	Lapsed	Lapsed	Stayed	Lapsed	Lapsed	Stayed	Stayed	Lapsed	Stayed	Stayed

The classifier is applied to eight test policyholders:

Years insured	1.6	3.5	4.4	6.8	9.4	10.6	12.3	14.2
Outcome	Lapsed	Lapsed	Stayed	Stayed	Stayed	Lapsed	Lapsed	Stayed

Calculate the test error rate of the classifier.

Question 4

A model predicts claim severity, in thousands. At a particular predictor value $x_0$ , the estimate $\hat f(x_0)$ has bias −0.6 and variance 0.45 over repeated training sets, and the irreducible error has variance 1.20.

Calculate the expected test MSE at $x_0$ .

Question 5

In a simulation study the true regression function at $x_0$ is $f(x_0)$ $= 5.0$ and the error variance is $\text{Var}(\varepsilon)$ $= 0.8$ . A model is fitted to five training sets, giving these estimates of $f(x_0)$ :

5.6, 4.8, 6.1, 5.3, 5.7

Treat the five estimates as the whole distribution of $\hat f(x_0)$ over training sets, each with probability 0.2.

Calculate the expected test MSE at $x_0$ .