Cross-Validation: Splits, Nested Selection and Honest Generalisation Estimates

Ask what will be new at deployment

Predicting a new record from an existing customer differs from predicting a new customer. Forecasting next month differs from interpolating among shuffled historical rows. A random split can be appropriate for independent identically distributed observations and misleading when records share entities or time-dependent information.

Define the unit of independence and the information cutoff first. Patients, devices, households, documents and conversations can each produce many correlated rows. Keeping related rows together is often more important than selecting an elaborate score.

Training, validation and test have different jobs

Training data fits model parameters. Validation data guides hyperparameters, features, thresholds and model families. Test data estimates performance after those decisions are fixed. Repeatedly examining a test score and adjusting the model turns that test set into another validation set.

A small dataset can use cross-validation for selection while retaining a final test set, or nested cross-validation to estimate a selection procedure. Neither manufactures independence where the underlying observations lack it. The split must still reflect the intended deployment population.

Compute out-of-fold loss consistently

For K folds, each observation receives a prediction from a model that did not train on its fold. A sample-weighted loss is

If one validation fold contains two observations with mean loss 1 and another contains eight with mean loss 3, pooled loss is \((2\cdot1+8\cdot3)/10=2.6\). The unweighted mean of fold means is 2, which answers a different question. With equal fold sizes these coincide.

For nonlinear metrics such as AUC or F1, averaging fold metrics and computing one metric from pooled predictions can also differ. Scores from separately fitted models may have different scales. State the aggregation and interpret it in relation to deployment.

Choose a splitter deliberately

Stratification preserves class proportions approximately; it does not make correlated observations independent. A temporal split may need a gap if features or labels use overlapping windows. Group and time constraints can coexist and may require a custom design.

Put the entire pipeline inside the fold

Fit imputation, scaling, vocabulary, PCA and feature selection using only the training fold. Oversampling and synthetic examples must also be generated within that training portion. Otherwise held-out information can influence the fitted representation or even create close relatives of validation observations.

Use the preprocessing chapter to trace the boundary. A convenient pipeline abstraction helps enforce it, but code structure is not proof: any transformation performed before entering the pipeline must be audited separately.

Early stopping uses validation feedback to choose training duration. Its stopping data is therefore part of selection, not an untouched estimate of final generalisation. The same applies to choosing a classification threshold or probability calibrator.

Nested selection evaluates a procedure

In nested cross-validation, an inner loop selects hyperparameters within each outer training subset. The outer held-out fold evaluates the resulting selected model. Repeating this estimates the performance of the selection procedure, including its opportunity to overfit inner validation noise.

After evaluation, a final deployment model can be selected and fitted using the available development data under the chosen protocol. Outer scores do not correspond to one identical fitted model; they correspond to several instances of the procedure.

Nested evaluation is costly. Use it when model selection is substantial relative to the dataset and the estimate matters. A carefully protected test set can also provide an honest final assessment. Reusing one validation set for hundreds of decisions without accounting for selection is the problem to avoid.

Uncertainty and comparisons

Fold scores are dependent because their training sets overlap. Their standard deviation describes variation across those particular folds; dividing it by the square root of K does not automatically give a valid independent-sample standard error.

Compare models on the same held-out observations or splits. For a fixed test set, paired resampling at the independent unit can estimate variation in a score difference under suitable assumptions. Repeated training seeds capture another source of variation and should not be confused with resampling the population.

Report the score distribution, sample count, split rule, search space and selection budget. Small differences should be interpreted relative to variability and practical effect, not only the ordering of mean scores.

Keep a reproducible experimental record

Store fold assignments or the exact splitter and seed, feature definitions, model settings, fitted preprocessing and prediction outputs. Include a simple baseline and relevant failure slices. This allows a later reviewer to distinguish a model improvement from a changed dataset or evaluation rule.

The goal is a believable estimate for a specified deployment situation. More folds, more decimals and a more complex model cannot compensate for a split that answers the wrong question.

References

1. scikit-learn. Cross-validation: evaluating estimator performance.
2. scikit-learn. Nested versus non-nested cross-validation.
3. James, Witten, Hastie, Tibshirani and Taylor. An Introduction to Statistical Learning with Applications in Python. Springer, 2023.