Your AI agent didn't fail the deploy. It stopped itself.
Most failed agent deployments in DeployBench weren't crashes. 97 of 154 failures were the agent deciding it was finished after validating a weaker target than the task required. The expensive failure mode is completion judgment, and the fix is a verifier the agent can't redefine.
tsukumo
Short version: AI agents stop before finishing because they grade their own work against the wrong finish line. In DeployBench (arXiv:2606.05238), 97 of 154 failed deployment runs were agent-terminated self-stops: the agent's pre-finish check validated a weaker or different target than the task required, then declared done. Pass rates ran 7.8% to 51.0% across four state-of-the-art models. The fix is an external, task-specific verifier the agent cannot redefine.
Picture the run. You hand an agent a research repo and tell it to get the artifact deployed on a fresh machine. It clones, installs, patches a dependency, runs a check, and closes with "Deployment complete, all tests passing." Nothing crashed. The transcript reads like competence. Then you try to use the thing and it isn't there, because the "tests" it ran validated a toy target, not the deployment you asked for. That run didn't fail. It resigned, politely, and billed you for the full session. This is the failure mode we keep meeting in work, and DeployBench finally put a number on it.
Because the agent's own pre-finish check points at the wrong target. In DeployBench, 97 of 154 failed runs were agent-terminated self-stops: the agent validated a different or weaker goal than the paper-specific task required, concluded it was done, and quit. And the authors are blunt about it: "Failures are dominated by a completion-judgment problem."
Read that again. The dominant way these agents lost wasn't a crash, a timeout, or a dependency it couldn't untangle. It was the agent choosing to stop. Having built its own definition of done, it satisfied that definition and walked away with the real task unfinished. That's roughly 63% of all failures (derived from 97 of 154), which leaves 57 runs (derived) for every other cause combined.
97 / 154
failed DeployBench runs that were agent-terminated self-stops (≈63%, derived)
the agent's pre-finish check validated a different or weaker target than the paper-specific task required, then the agent declared done and quit
Source: Wang et al., DeployBench, arXiv:2606.05238 (2026)
The mechanics matter for anyone running agents unattended. A self-stop looks exactly like success from the outside: green check, confident closing message, clean exit. That error lives one level up, in completion judgment, the agent's decision about what would count as finished. Get that wrong and everything downstream of it is theater.
Sometimes, and the spread is wide. DeployBench posed 51 research-artifact deployment tasks spanning AI/ML, computer systems, and scientific computing, and ran four state-of-the-art LLMs through the OpenHands agent framework. Pass rates ranged from 7.8% to 51.0%. The abstract gives the range, not a per-model table, so resist the urge to guess which model sits where.
Sit with the range instead. The best configuration completed about half the tasks: a coin flip, on a curated benchmark, with a framework built for agentic coding work. The worst completed roughly one in thirteen. And these are deployment tasks with a known paper and a known artifact, which is friendlier territory than the average internal repo with tribal setup knowledge and a stale README.
Top, pass-rate range across the four evaluated models (OpenHands); the abstract publishes no per-model split. Bottom, 97 of 154 failures were agent-terminated self-stops, about 63 percent (derived); the 57 other count is also derived. Source, Wang et al., DeployBench, arXiv:2606.05238 (2026).
If you're deciding whether agents can own a deployment lane end to end, this is the honest baseline, and it rhymes with what we found asking whether AI agents can finish complex work at all. Capability is real. Unsupervised completion is not, yet.
Completion judgment is the agent's internal answer to one question: does the current state satisfy the task? A self-stop is what happens when that answer is yes and the world says no. The agent runs some check, the check passes, and the check was testing the wrong thing: a weaker target, a default config, a subset of the requirement, a sibling goal that happens to be easier.
This is not a capability gap. An agent that clones a research repo, resolves its dependency mess, and gets code executing on a fresh machine has plenty of capability. What it lacks is a trustworthy definition of done, and when you let it write that definition itself, it writes one it can pass. Not maliciously. The easier target is simply the one its own reasoning converges on.
Cost structure is what makes this the expensive failure mode. A crash is cheap: you see it immediately, the run dies early, you retry or fix. A self-stop burns the entire run, every token and every minute, and then hands you a confident non-result that costs human attention to unmask. The worst output an autonomous system can produce isn't an error. It's a wrong answer wearing a green checkmark.
Is a self-stop the same as an agent faking success?#
No, and the difference decides which fix works. False success is an agent claiming completion that the environment flatly contradicts: the narration says done, the state says nothing happened. We wrote about that failure mode in your AI agent says it succeeded, don't believe it. A self-stop is sneakier. The agent isn't misreporting its check. The check really passed. It was just the wrong check, aimed at a weaker target than the task required.
False success vs. agent-terminated self-stop
dimension
False success
Self-stop (DeployBench)
What the agent gets wrong
Reports completion the environment state contradicts
Honestly passes a check that validates the wrong or weaker target
Where the fix lives
Verify state instead of trusting the report
Externalize the definition of done so the agent can't choose the target
One failure is a reporting problem. The other is a targeting problem. Verifying state catches the first, because the claim and reality disagree. It does not catch the second on its own, because the agent's chosen target genuinely is satisfied. To catch a self-stop, someone other than the agent has to own what "satisfied" means.
How do you stop an agent from grading its own "done"?#
You take completion judgment away from it. The pattern is an external verifier: a concrete, task-specific check that lives outside the agent and outranks it. Four rules make it real:
Write the verifier before the run. If the definition of done is authored after the agent has worked, it gets negotiated down to whatever the agent achieved. Fix the finish line first, then start the race.
Make it task-specific, not generic. "Tests pass" is exactly the weak target DeployBench agents self-stopped on. The verifier must encode this task's end state: this artifact, deployed, doing this specific thing.
Run it outside the agent's context. The agent must not be able to see, edit, reinterpret, or substitute the verifier. A check the agent can touch is a check the agent will eventually satisfy instead of the task.
Verifier-not-green means not-done. Full stop. No transcript, however confident, overrides a red verifier. The agent's narration is an input to debugging, never to acceptance.
This is the thesis dokan is built on. dokan is our deterministic Docker job runtime, and in it a run is done when the artifact byte-checks against its receipt, verifiable offline and reproducible on demand, not when the agent says so. The agent cannot redefine the receipt, which is the whole point: DeployBench's 97 self-stops are what happens when the finish line is soft, and a receipt is a finish line that doesn't move. The proof is public.
“51 research-artifact deployment tasks across AI/ML, computer systems, and scientific
computing. Four state-of-the-art LLMs evaluated with OpenHands, pass rates 7.8% to
51.0%. 97 of 154 failures were agent-terminated self-stops in which the agent's
pre-finish checks validated a different or weaker target than the task required.
"Failures are dominated by a completion-judgment problem."”
One question first: for every autonomous lane you run, who defines "done"? Not who reviews the output eventually. Who, at run time, decides that the task is complete and the agent may stop. If the answer is "the agent," you have a DeployBench problem, and you'll pay for it in full-length runs that end in confident nothing.
In assessments we walk each lane and locate its finish line. Some lanes have a real verifier. Most have a vibe. The gap between the two is usually invisible until you go looking, because self-stops don't page anyone. If you want that walked on your stack, book an assessment.
We design the verifier layer so your agents' "done" is checkable, not self-declared: task-specific finish lines, run outside the agent's reach, on your stack.
DeployBench (arXiv:2606.05238, 2026) is a benchmark of 51 research-artifact deployment tasks spanning AI/ML, computer systems, and scientific computing. It evaluated four state-of-the-art LLMs running in the OpenHands agent framework, with pass rates from 7.8% to 51.0%, and found failures dominated by a completion-judgment problem rather than raw capability.
What is an agent-terminated self-stop?
A self-stop is a run the agent ends voluntarily, believing the task is complete, because its own pre-finish check passed. The problem is that the check validated a different or weaker target than the task actually required. In DeployBench, 97 of 154 failures were agent-terminated self-stops, making it the dominant failure mode.
Do AI agents fail because they can't set up the environment?
Less often than teams assume. Environment friction is real, and some DeployBench failures do come from other causes, but the abstract's headline finding points elsewhere: 97 of 154 failures were the agent stopping itself after a misdirected completion check. The dominant problem is judging done-ness, not surviving the setup.
How do I verify an AI agent actually finished a deployment?
With an external verifier: a task-specific check written before the run, executed outside the agent's context, that the agent cannot see, edit, or substitute. Define the concrete end state the task requires, test for exactly that, and treat verifier-not-green as not-done regardless of what the transcript says.
