MACHINE LEARNING · QUESTION 18
What is cross-validation, and when should you use it?
On this page
Cross-validation estimates how a model may perform on unseen data by repeatedly training on some folds and validating on a held-out fold. In k-fold cross-validation, each fold serves as validation once, and the results are summarized. It makes efficient use of limited data for model selection, but folds must respect dependencies such as time, users, or groups to avoid leakage.
LET'S MAKE IT SIMPLE
First, the idea in plain English
A single train-validation split can give a noisy result, especially when the dataset is small. Cross-validation repeats the comparison with different parts of the data held out, so each example can contribute to validation in one round. The scores show how sensitive a model’s performance is to the split. The method is only fair when the split matches how new examples will arrive.
A USEFUL COMPARISON
Picture it this way
Imagine asking several different groups of students to take turns as the practice exam group while everyone else studies. You get more than one estimate of how the lesson works for students not used in that round. If related students appear in both groups, though, the estimate can be too optimistic—just as related records can leak across folds.
FOLLOW THE JOURNEY
Each fold takes a turn as validation data
Show how k-fold cross-validation rotates training and validation subsets.
The number of folds and split strategy depend on data size and dependence structure. Time series usually needs chronological splits rather than random folds.
A dataset is split into five folds; across five rounds, one fold is validation and the remaining four folds are training.
How to read this visual
The number of folds and split strategy depend on data size and dependence structure. Time series usually needs chronological splits rather than random folds.
UNDER THE HOOD
How it works, step by step
Choose a split strategy
Select folds that preserve class balance or respect time, groups, and other dependencies.
Why it matters: Validation must imitate the independence and chronology of future predictions.
Train each round
Fit on all but one fold and evaluate on the held-out fold.
Why it matters: Rotating folds gives multiple estimates from the available data.
Compare results
Summarize the scores and variability across folds.
Why it matters: Large variation can signal sensitivity to sample composition.
Select and evaluate carefully
Use cross-validation for development, then reserve a final test set when an unbiased final estimate is needed.
Why it matters: Repeated model selection can overfit the cross-validation results too.
OUTSIDE THE TEXTBOOK
A practical example
A team has a limited set of labeled customer requests and wants to compare two classifiers. It uses stratified folds so each validation round includes a similar class mix. If multiple requests come from the same customer, it instead groups by customer so that one person’s examples cannot appear in both training and validation for the same round.
GO A LITTLE DEEPER
Technical details that matter
- Stratified k-fold helps preserve approximate class proportions for classification.
- Grouped splits keep related entities together; time-series splits train on the past and validate on later periods.
- Preprocessing such as feature selection and scaling should be fitted inside each training fold, not once using the full dataset.
- Nested cross-validation can separate hyperparameter selection from performance estimation when data is limited, at additional computational cost.
MAKE A GOOD CHOICE
When is this useful?
- Comparing models or hyperparameters when labeled data is limited.
- Estimating variability due to a particular random train-validation split.
- Testing workflows where data can be partitioned into reasonably independent examples.
A fixed holdout split is simpler and faster for large datasets. Use a time-based or group-based holdout when deployment has those dependencies; no split strategy can compensate for unrepresentative data.
WATCH OUT FOR
Common misunderstandings
- Randomly splitting time-dependent records and training on future information.
- Preprocessing the full dataset before cross-validation, causing leakage.
- Treating the mean score as an unbiased final result after extensive model tuning.
REAL-WORLD TRADE-OFFS
Limitations and failure cases
- Training k models costs more compute than one holdout split.
- Fold scores are not fully independent because training sets overlap.
- Poorly chosen folds can leak information or fail to represent deployment.
WHAT TO REMEMBER
Key takeaways
- 1Cross-validation rotates held-out folds to estimate generalization.
- 2Use group-aware or time-aware splits when examples are dependent.
- 3Fit preprocessing within each training fold.
- 4Keep final evaluation separate from repeated model selection when possible.
PRACTICE THE FOLLOW-UP
What might an interviewer ask next?
When would you use stratified cross-validation?
For classification, especially with imbalanced classes, it helps keep class proportions similar across folds. It does not replace checking for other dependencies.
Why is preprocessing inside each fold important?
If preprocessing learns statistics from held-out data, information leaks into training and makes validation performance look better than it should.
Should you still keep a test set?
Often yes, particularly after trying many model choices. A test set used only at the end gives a cleaner final estimate than the repeatedly consulted validation folds.
SAY IT OUT LOUD
How to explain this in an interview
“Cross-validation rotates which subset of data is held out, so model performance is measured across multiple splits rather than relying on one. It is useful when data is limited and I need to compare models. The split strategy matters: I would keep related users together or use chronological folds for time-dependent data, and fit preprocessing inside each training fold to prevent leakage.”
Connect this question to the roadmap topic it supports.