MLOPS & DEPLOYMENT · QUESTION 65
What should you track for a machine-learning experiment?
On this page
Track enough information to understand, compare, and reproduce each run: code revision, data version, configuration, environment, model artifact, metrics, and evaluation split. Record meaningful parameters and artifacts rather than every transient detail. Experiment tracking supports collaboration and auditability, but it only helps when runs are named, data is accessible, and comparisons use consistent evaluation.
LET'S MAKE IT SIMPLE
First, the idea in plain English
A score without context is hard to trust. To reproduce an experiment, a teammate needs to know which code, data, settings, and environment produced the model. A tracker stores these details alongside results so teams can compare runs and identify what actually changed.
A USEFUL COMPARISON
Picture it this way
A laboratory notebook records the ingredients, steps, and observations for an experiment. If a result is promising, another scientist can investigate how it was produced. A digital tracker does this for model runs, though it still depends on accurate and complete records.
FOLLOW THE JOURNEY
Connect each training run to its inputs and results
Show run metadata, artifact lineage, and comparable evaluation results.
This simplified visual highlights the main relationships. In practice, the exact design depends on the requirements; Inconsistent splits or metrics make rankings misleading.
A training run records code, data, parameters, and environment alongside metrics and artifacts in a searchable experiment history.
How to read this visual
This simplified visual highlights the main relationships. In practice, the exact design depends on the requirements; Inconsistent splits or metrics make rankings misleading.
UNDER THE HOOD
How it works, step by step
Identify the run
Record a stable run ID and code revision.
Why it matters: Results need an unambiguous owner and source.
Capture inputs
Link data version, configuration, dependencies, and evaluation split.
Why it matters: These inputs determine what the experiment means.
Store outputs
Save metrics, plots, and model artifacts with checksums or references.
Why it matters: Teams need evidence and a retrievable candidate.
Compare deliberately
Group runs by experiment and compare like-for-like metrics.
Why it matters: Inconsistent splits or metrics make rankings misleading.
OUTSIDE THE TEXTBOOK
A practical example
A team compares three forecasting models. Each run records the same time-based split, feature revision, dependency lock, parameters, and error metrics by region. Reviewers can see that one apparent improvement came from a different data window rather than a better model.
GO A LITTLE DEEPER
Technical details that matter
- Track immutable identifiers for large datasets and artifacts rather than relying on mutable filenames.
- Record environment and random seeds where reproducibility matters, while recognizing hardware nondeterminism may remain.
- Avoid storing sensitive raw data or credentials in experiment metadata.
MAKE A GOOD CHOICE
When is this useful?
- Collaborative research with many model and parameter trials.
- Auditing how a deployed artifact was produced.
- Comparing candidates under consistent datasets and evaluation.
A small one-off experiment can use version-controlled scripts and structured result files. A tracking platform helps when run volume or collaboration makes manual records unreliable.
WATCH OUT FOR
Common misunderstandings
- Logging only the final metric and losing data or code lineage.
- Comparing scores from different splits as if they were equivalent.
- Saving sensitive inputs or secrets in a broadly accessible tracker.
REAL-WORLD TRADE-OFFS
Limitations and failure cases
- Trackers do not make experiments reproducible if source data is deleted.
- Metadata can be incomplete or inconsistent without automation.
- Artifact storage and access control need ongoing management.
WHAT TO REMEMBER
Key takeaways
- 1Track code, data, config, environment, artifacts, and evaluation.
- 2Use consistent splits and metrics for meaningful comparisons.
- 3Protect sensitive data and make records retrievable.
PRACTICE THE FOLLOW-UP
What might an interviewer ask next?
Why record the data version?
The same code and settings can produce different results on changed data. A stable data reference is essential for interpreting and reproducing a run.
Do fixed random seeds guarantee exact reproducibility?
Not always. Hardware, libraries, parallel operations, and nondeterminism can still change results, though seeds improve repeatability.
SAY IT OUT LOUD
How to explain this in an interview
“I track the code revision, data snapshot, configuration, environment, evaluation split, model artifact, and metrics for each run. That makes comparisons and reproduction possible. I keep evaluation consistent and store sensitive data carefully. A tracker is only useful if records are complete and the referenced code and data remain accessible.”
Connect this question to the roadmap topic it supports.