A demo shows possibility, not coverage
A demo often uses a small number of hand-picked examples. It can show that a model is capable of producing a good result, but not how often it succeeds or who it fails for.
Evaluation starts by defining the intended task and what different errors cost. A spam filter, for example, has different consequences for missed spam and a legitimate message sent to junk.
Use data that represents the decision
A held-out evaluation set should resemble the situations where the model will be used. If time changes the data, a time-aware split may be more realistic than a random split.
Keep validation data for development choices and a final test set for a last estimate. Repeatedly tuning against the test set turns it into part of the training process.
Inspect failures as well as averages
Aggregate metrics can hide weak performance in important segments. Review slices, examples, label quality, and failure modes alongside overall scores.
Offline evaluation is not the whole story. Production monitoring should check input quality, service reliability, and outcomes when trustworthy labels become available.
Make the result actionable
A useful evaluation leads to a decision: ship, gather more data, adjust a threshold, or change the product workflow. A score without a defined decision threshold rarely answers whether a model is ready.
The best metric is not the most familiar one; it is the one connected to the product’s actual objective and risks.