Thursday, November 11, 2010

Cross-validation with one model

This is essentially a repost of Rob J Hyndman's blog post on the relevance of cross-validation for statisticians.

Within this very nice piece, Rob drops this bomb of mathematical knowledge:

It is not necessary to actually fit separate models when computing the CV statistic for linear models.

Say what?

Here is a broader excerpt and the method itself (after the jump).

While cross-validation can be computationally expensive in general, it is very easy and fast to compute LOOCV for linear models. A linear model can be written as Then and the fitted values can be calculated using where is known as the “hat-matrix” because it is used to compute (“Y-hat”).
If the diagonal values of are denoted by , then the cross-validation statistic can be computed using where is the residual obtained from fitting the model to all observations. See Christensen’s book Plane Answers to Complex Questions for a proof. Thus, it is not necessary to actually fit separate models when computing the CV statistic for linear models. This remarkable result allows cross-validation to be used while only fitting the model once to all available observations.

Very cool.