Evals
5 current practices.
StandardEvalsEV-001Read your traces before you build any metricHand-read production traces, categorise the failures, and build a vocabulary of what's actually broken before writing a single eval.
Reviewed Aug 2026EvalsEV-002Make judges binary and anchor them to expert critiquesUse pass/fail rather than a 1–5 scale, anchored to one domain expert's written critiques.Reviewed Aug 2026EvalsEV-003Validate the judge against human labelsMeasure agreement between judge and human, and treat a 100% pass rate as a warning rather than a win.Reviewed Aug 2026EvalsEV-004Only build an automated eval when the failure justifies itCatch what you can with assertions and regex. Reserve judge-based evals for failure modes that warrant a hundred labels and weekly maintenance.Reviewed Aug 2026EvalsEV-005Gate model, prompt and tool changes through an eval suiteWire the agent into a test suite so changes are checked before they ship, and watch for drift after.
Reviewed Aug 2026
Reviewed Aug 2026EvalsEV-002Make judges binary and anchor them to expert critiquesUse pass/fail rather than a 1–5 scale, anchored to one domain expert's written critiques.Reviewed Aug 2026EvalsEV-003Validate the judge against human labelsMeasure agreement between judge and human, and treat a 100% pass rate as a warning rather than a win.Reviewed Aug 2026EvalsEV-004Only build an automated eval when the failure justifies itCatch what you can with assertions and regex. Reserve judge-based evals for failure modes that warrant a hundred labels and weekly maintenance.Reviewed Aug 2026EvalsEV-005Gate model, prompt and tool changes through an eval suiteWire the agent into a test suite so changes are checked before they ship, and watch for drift after.Reviewed Aug 2026