There is a specific failure that shows up in most stalled AI pilots. The system performs well in testing, ships, and then slowly loses the confidence of the people using it, without any single incident anyone can point to. Nobody can say it broke. Everybody can tell it is not trusted any more.
That failure is almost always an evaluation problem rather than a model problem. The team measured the thing that was easy to measure offline, and the operation was judging something else entirely.
What offline benchmarks are actually good for
Benchmarks are useful for one decision: choosing which model to start from. They are a capability filter. They tell you whether a model can, in principle, follow a multi-step instruction, use a tool correctly, or hold a coherent conversation.
What they cannot tell you is whether your system, with your prompts, your tools, your data, and your callers, produces the outcome your operation needs. Capability is a precondition. It is not evidence.
Score the outcome the operation already cares about
The most useful discipline we have adopted is refusing to invent a new metric. Every operation already has a number it lives by, and the agent should be scored on that same number, computed the same way, next to the humans doing the same work.
- Task completion: of the interactions that arrived, what proportion ended in the outcome the operation wanted, using the operation's own definition of the outcome.
- Escalation rate and escalation quality: how often the agent hands off, and whether the handoff arrived with enough context that the person did not have to start over.
- Silent failure: cases where the agent believed it succeeded and did not. This is the most expensive category and the one benchmarks never surface.
- Latency distribution, not average: the tail is what callers experience and what makes a system feel broken.
If a metric would not have existed before the agent was introduced, be suspicious of it. It was probably invented to make the agent look good.
Continuous, on real traffic
A one-off evaluation before launch answers a question nobody will ask again after week two. Traffic shifts, the business changes what it sells, a prompt gets edited, an upstream model is updated. Any of those can move behaviour without anyone touching the part of the system that appears responsible.
So evaluation has to be a running process rather than a gate. Every interaction is logged with its inputs, its actions, and its outcome, a sample is scored continuously, and regressions raise an alert before a customer finds them. This is the layer we treat as infrastructure rather than as a per-product concern, which is why it belongs to the group rather than to any one venture.
Keep a human in the scoring loop
Automated scoring, including using a model to grade another model, scales well and drifts quietly. It is worth exactly as much as its agreement with human judgement, and that agreement has to be re-measured periodically on a fresh sample. When the two diverge, the automated scorer is the thing that is wrong.
None of this is exotic. It is the same discipline any operations team applies to any process it depends on. The only thing that changes with an agent is that the process can now degrade without anybody noticing, which raises the cost of not measuring it.