MACHINE LEARNING · QUESTION 50
What is data leakage, and how do you detect it?
On this page
Data leakage occurs when training or model selection uses information that would not legitimately be available at prediction time, or when evaluation data influences fitting. It can produce unrealistically strong validation scores and poor deployment performance. I inspect feature timestamps and joins, split by time or entity where needed, fit preprocessing within folds, and keep the final test set isolated.
LET'S MAKE IT SIMPLE
First, the idea in plain English
A model should be tested as if it were making a real future prediction. If the training data accidentally contains the answer, a future event, or information from the evaluation set, the model can appear brilliant without learning a usable relationship. Leakage is often a data-process bug rather than a sophisticated model failure.
A USEFUL COMPARISON
Picture it this way
A student’s practice exam is not a fair test if the answer key is printed on the same page. The score will look excellent, but it says little about solving a new problem. A model similarly needs an evaluation setup that withholds information it could not have in real use.
FOLLOW THE JOURNEY
Keep future and held-out information outside training
Illustrate how leakage crosses a time or evaluation boundary.
This simplified visual highlights the main relationships. In practice, the exact design depends on the requirements; A held-out estimate is useful only if it stayed unseen.
A prediction-time boundary separates allowed historical features from future labels; preprocessing is fitted only on training data before validation and test evaluation.
How to read this visual
This simplified visual highlights the main relationships. In practice, the exact design depends on the requirements; A held-out estimate is useful only if it stayed unseen.
UNDER THE HOOD
How it works, step by step
Set prediction time
Define exactly when the model would make its decision.
Why it matters: Feature availability must be judged relative to that moment.
Audit sources and joins
Inspect timestamps, target-derived fields, duplicated entities, and aggregation windows.
Why it matters: Information can cross boundaries through data preparation.
Choose valid splits
Use time-aware, group-aware, or stratified splits as the task requires.
Why it matters: The split should reflect how unseen cases arise.
Fit and evaluate cleanly
Train transformations only within training data and reserve final test data.
Why it matters: A held-out estimate is useful only if it stayed unseen.
OUTSIDE THE TEXTBOOK
A practical example
A churn model predicts next month’s cancellation. A feature table accidentally includes a “closed account” flag populated after cancellation. Random validation produces near-perfect results, but the field does not exist when the prediction is made. Reviewing feature timestamps reveals the leak; a time-based evaluation with valid features gives a more realistic estimate.
GO A LITTLE DEEPER
Technical details that matter
- Entity leakage happens when related records from the same user or device cross train and validation splits.
- Target leakage can arrive through post-outcome fields, aggregates that include labels, or preprocessing performed before splitting.
- Unexpectedly high scores are a useful audit signal but are not proof of leakage by themselves.
MAKE A GOOD CHOICE
When is this useful?
- Auditing unusually strong model results.
- Designing evaluation for time-dependent or repeated-entity data.
- Reviewing feature pipelines before deployment.
If data truly arrives as independent, identically distributed examples, a random split may be suitable. Confirm that assumption rather than applying a time split automatically.
WATCH OUT FOR
Common misunderstandings
- Looking only for a column named after the target.
- Splitting rows randomly when users or time periods are related.
- Fitting imputation, scaling, or feature selection before cross-validation.
REAL-WORLD TRADE-OFFS
Limitations and failure cases
- Leakage can be subtle in derived data and upstream pipelines.
- Strict splits can reduce the amount of data available for training.
- A leak-free evaluation may still be unrepresentative of future distribution shifts.
WHAT TO REMEMBER
Key takeaways
- 1Define prediction time and what data is available then.
- 2Choose splits that respect time and entity relationships.
- 3Keep preprocessing and final test data isolated.
PRACTICE THE FOLLOW-UP
What might an interviewer ask next?
Can random train-test splitting cause leakage?
It can when records share users, devices, sessions, or future information. The appropriate split depends on the deployment scenario.
How can a very high validation score be explained?
It may reflect a genuinely easy task, but also check target-derived fields, duplicates, timestamp errors, and preprocessing across the split.
SAY IT OUT LOUD
How to explain this in an interview
“Leakage means the model or evaluation has access to information unavailable at prediction time, including information that crossed the train-test boundary. I define the decision timestamp, audit feature lineage and joins, split by time or entity when needed, and fit preprocessing inside each training fold. Strong metrics deserve inspection, and the final test set must remain isolated.”
Connect this question to the roadmap topic it supports.