Evaluation is often delegated to the machine-learning team and summarized as a benchmark score. That is necessary but insufficient. A product can improve on a technical metric while becoming less useful, less fair, or more dangerous in the workflow where it operates.
Begin with the decision
Write down what changes because the model produced an output. Does a person receive information, reorder a queue, approve a claim, contact a patient, or move money? The consequence determines which errors matter and how quickly they must be detected.
False positives and false negatives rarely cost the same. Average accuracy hides this asymmetry. Product leaders must define the acceptable tradeoff with domain experts and affected users.
Build an evaluation portfolio
- Capability: Can the system perform the intended task on representative examples?
- Safety: Does it avoid prohibited or harmful behavior under pressure?
- Robustness: What happens with incomplete, unusual, or adversarial inputs?
- Equity: Are outcomes materially different across relevant groups?
- Experience: Can users understand, correct, and act on the output?
- Operations: Are latency, cost, and availability acceptable at real volume?
A score is only meaningful when the organization has agreed what failure means.
Use examples the business recognizes
Generic benchmarks help compare models, but release decisions require domain-specific cases. Build a living evaluation set from historical incidents, expert judgment, user research, edge cases, and production feedback. Record why each example matters and which expected behavior it tests.
Keep a protected holdout set so teams do not unconsciously optimize for the visible test. Refresh the set as users, policies, data, and model behavior change.
Connect offline and online evidence
Pre-release testing estimates behavior. Production reveals behavior under real incentives and context. Monitor outcome quality, override rates, escalation patterns, user corrections, subgroup performance, and downstream business effects. A model that tests well but is routinely ignored has failed as a product.
Make thresholds govern action
Define what happens when a metric moves: alert, investigate, restrict a feature, route to human review, or roll back. Without action thresholds, monitoring becomes a dashboard that documents deterioration instead of controlling it.
Evaluation belongs in product strategy because it defines the promise being made. Engineering can measure that promise, but the whole organization must decide what is worth measuring.