Evaluating AI agents in production: getting past vibes
Most teams ship agent changes on vibes: it felt better in the demo. But an agent silently regresses, and "it seems good" isn't a measurement. Evals are a golden set of real tasks with gradeable outcomes, run on every change, with the judge itself checked.
tsukumo
The short answer
Most teams change an agent and judge it on vibes: it felt better in the demo. That is not a measurement, and an agent silently regresses on the cases you didn't look at. Evaluating an agent in production means a golden set of real tasks with gradeable outcomes, run on every prompt, model, or tool change, so a regression shows up as a number that dropped instead of a complaint two weeks later. And if you use a model to grade, you have to check the grader, or you've just automated the vibes.
Short version: most teams change an agent and decide it's better because it felt better in the demo. That is not a measurement, and it is exactly how agents silently regress. An agent's behavior shifts across thousands of cases you never look at, so "it seems good" tells you about the three you did. Evaluating an agent means a golden set of real tasks with gradeable outcomes, run on every prompt, model, or tool change, so a regression shows up as a number that dropped instead of a support ticket two weeks later. We run agents in production to ship our own software, and the evals are what let us change them without holding our breath.
Because a demo is a sample you chose, and an agent fails on the distribution you didn't.
When you tweak a prompt or upgrade a model, you naturally test the handful of cases you have in your head. They improve, so you ship. The problem is that an agent operates across a wide, messy distribution of inputs, and a change that helps your three favorites can quietly degrade a whole category you never sample. There is no error, no crash, just a slow drift in quality that nobody is measuring. By the time it surfaces as a complaint, it has been affecting real work for a while, and you have no idea which change caused it because you never measured any of them.
A golden set of real tasks, a way to grade each one, and a score you compare every time.
There are three parts, and skipping any of them breaks it.
A golden set. Real, representative tasks (not three demo prompts), including the hard and weird ones, frozen so you test the same thing every run.
A grader. A way to score each outcome: an exact check or assertion where the answer is well-defined, and a model-judge where it isn't, validated against human labels.
A run-on-every-change discipline. The eval runs on every prompt, model, or tool change, like a test suite, and you read the score before you ship.
It is not exotic. It is the same instinct that made you write tests for code, applied to a component whose behavior is statistical instead of deterministic.
Where does your team actually stand on this? A short agent-ops assessment is the low-risk way to find out.
Use the strongest grader the task allows, and don't reach for a model when a check will do.
Some outcomes are checkable: did the agent call the right tool, did the output parse, did the number match, did the test pass. Grade those with code, because a deterministic check never has an off day. Other outcomes are judgement calls: is this summary faithful, is this fix reasonable, is this answer helpful. For those, an LLM-as-judge earns its place, because a human can't grade thousands of these on every change. The mistake is using a model-judge for things a simple assertion would catch more reliably, and cheaper.
If you grade with a model and never check the model, you have automated your vibes, not replaced them.
LLM-as-judge is genuinely useful and genuinely dangerous, because a grader you don't validate can be biased, inconsistent, or wrong in the same direction as the agent it is grading. A judge that quietly prefers longer answers, or rubber-stamps confident ones, gives you a green dashboard over a degrading agent, which is worse than no eval at all. You validate the judge the boring way: have humans label a sample, check that the judge agrees with them at a rate you'd accept, and re-check when you change the judge's own prompt or model. An unverified judge is just a confident vibe with a number attached.
If you are running agents on anything that matters, build the eval before you build the tenth feature. A frozen golden set of real tasks, deterministic grading where you can and a validated judge where you can't, run on every change like a test suite. It is the difference between an agent you can improve with confidence and one you are afraid to touch because you can't tell whether you made it worse. The model gives you capability; the eval is what lets you keep it.
We set up this measurement discipline when we install agents on a client's environment, because an agent you can't evaluate is an agent you can't safely change. If your team is shipping agent changes on vibes and feeling the wobble, that's the work we do.
We map where your agents run unmeasured and build the golden set first.
An eval is a repeatable test of an agent against a fixed set of representative tasks with a way to grade each outcome as pass or fail (or scored). You run it on every change to the agent's prompt, model, or tools, and compare the score to the last run. It turns "does this feel better" into "did the number go up or down."
Why isn't testing in the demo enough for an AI agent?
Because the demo is a handful of cases you chose, and an agent's behavior shifts across thousands you didn't. A change that improves your three demo prompts can quietly break a category you never look at. Without a fixed, representative set scored every time, you only learn about regressions when a user hits one.
Can you use an LLM to grade an AI agent's output?
Yes, LLM-as-judge is useful for outputs that are hard to grade with a script, but only if you validate the judge against human labels first. An unchecked judge can be biased, inconsistent, or wrong in the same direction as the agent. If you never verify the grader, you've automated your vibes, not replaced them.
How often should you run agent evals?
On every change that can move behavior: a prompt edit, a model upgrade, a new or changed tool. Treat the eval like a test suite in CI. The whole value is catching a regression at the moment you introduce it, not discovering it in production after it has affected real work.
