We benchmarked the token cost of rereading docs, and shipped the command
Resolving a question to one canonical doc instead of rereading the top candidate files cut about 60% of the tokens per markdown lookup, measured at equal task-success on our own repo. Here is the number, the disclosures, and the one command to reproduce it on your repo.
tsukumo
Short version: we got tired of quoting a representative number, so we benchmarked it and shipped the benchmark as a command. The finding: when an agent resolves a question to one canonical doc instead of rereading the top candidate files to work out which is current, it spends about 60% fewer tokens per markdown lookup. That floor sits under a measured run on our own repo, counted only where both approaches answered equally well: a median of 69% across 26 queries. The proof is that you can run the same benchmark on your repo in about a minute.
That post explains why the waste exists. This one measures it, then hands you the command. Resolving to one canonical doc instead of rereading is context engineering made measurable.
~60%
fewer tokens per markdown lookup
Conservative floor, measured at equal task-success on the trovex repo
Source: trovex bench, --eval
About 60% is the floor we put our name on, not the average. We lead with it because the measured median is higher, and a higher number on a single repo with an LLM judge is exactly the kind of figure that deserves a footnote rather than a headline.
The honest version of "measured" is narrow on purpose. A token cut that quietly degrades the answer is not a saving, so we only counted a query when both approaches reached an equally correct answer.
Two things in there matter more than the median. First, the judge is an LLM with no gold answers, on one repo, so 69% is evidence that 60% is not overstated, not a universal benchmark. Second, we counted three losses and a whole category that did not clear the bar, because a benchmark that only reports its wins is marketing wearing a lab coat.
This is the deliverable, more than the number. trovex bench ships in the CLI. The method is the claim, so run it.
bash
uvx trovex bench . # any folder of markdown, local embedder, no API key
uvx trovex bench . --eval # full answer + judge at equal task-success (needs a key)
uvx trovex bench . --json # machine-readable: median, spread, per-query, per-category
Instant token-model by default, --eval for the answer-and-judge run, --json for machine output
The default run is the instant token-accounting model: it counts the tokens to read the top candidates versus the one routed doc, with a local embedder, no API key, deterministic, no state left behind. The --eval flag runs the full answer-and-judge version above. Point it at your own markdown and you get your real reduction on your real docs.
To show the command does what it says, we pointed it at FastAPI, a public, doc-heavy project with over 1,500 markdown files. This is the instant token-model, not the equal-task-success measure, so read it as modeled, not judged.
uvx trovex bench on the FastAPI repo (modeled token-model at k=3)
Query
Would read
With trovex
Reduction
how do I install this
17,628
6,193
64.9%
how do I configure it
1,982
835
57.9%
what is the architecture
1,444
212
85.3%
how do I contribute
2,599
121
95.3%
Across the built-in question set the modeled median came out at about 82%, with the lowest query still near 58%. A different repo gives a different number. That is the point: the command reports the shape of your docs, not a slide we made.
The number to trust over any benchmark is your own traffic. trovex logs each markdown read a routed agent makes, and trovex measure compares a window before you routed through trovex against a window after, reporting the real change in tokens per read.
bash
trovex measure # last 7 days vs the prior 7
trovex measure --baseline-days 14 --current-days 14
Before and after on your agent's own logged reads, not a synthetic benchmark
This assumes nothing about your baseline behavior, the way the modeled run assumes the agent would have read three candidates. It reads what your agent actually did, before and after, and reports the change. If you run one number on your own setup, run this one.
Because the expensive part was never the answer. It was the guessing.
Without a canonical layer, the agent reads three files to find the one that is current and discards two. With one, it reads a pointer and the single doc that answers. The discarded reads vanish, and so does the quieter cost of a polluted context window nudging the model toward a stale answer. This is the canonical-doc layer the agent-memory taxonomies keep missing, and the benchmark is the arithmetic underneath it.
“The waste was never the answer. It was the agent rereading two files to work out which third file to trust.”
No, and any benchmark that promised otherwise would be lying to you.
We measured about 60% on the trovex repo. A repo with little overlapping markdown saves less, because there was less rereading to remove. A repo thick with duplicates and stale runbooks saves more. The benchmark is not a guarantee you will hit our number. It is a tool to find yours.
We did not want to be one more vendor quoting a tidy percentage with no way to check it. So we shipped the command before we leaned on the claim. The number we put our name on, about 60%, is the conservative floor of what we measured on our own repo at equal task-success, with the losses reported and a command you can run on your own docs tonight. That discipline, measure on first-party data, publish the weak result next to the strong one, and hand you the tool to falsify it, is how we work.
If you are rolling agents out across a fleet and the token bill is becoming a line item, that is the layer we fit and the number we help you keep honest.
We map where your agents reread instead of resolve, run the benchmark on your repo, and leave you with a figure you can reproduce without us.
About 60% fewer tokens per markdown lookup, the conservative floor of what we measured on our own repo. The measured run, counted only where both approaches reached an equally correct answer, had a median of 69% across 26 queries with a range of 41 to 81%. We publish about 60% because 69% on one repo with an LLM judge is not a number to lead with. Your repo will land somewhere of its own.
Is the benchmark reproducible?
Yes, that is the whole point. It ships in the trovex CLI as `trovex bench`. Point it at any folder of markdown. The default is an instant token-accounting model with a local embedder and no API key. Add `--eval` for the full answer-and-judge run that counts savings only at equal task-success. The method is the claim, so run it yourself.
Was answer quality held equal?
In the measured run, yes. A token cut at degraded quality is not a win, so a saving counted only when both the baseline approach and trovex produced an equally correct answer, scored by an LLM judge (gpt-5.4-mini, no gold answers). Of 26 queries, 17 reached equal success and 3 were trovex losses, which we count and report rather than hide.
Where does trovex do worst?
Needle-in-a-large-corpus questions underperformed in the measured run: that category reached no equal-success pairs, so we do not claim it as a strength here. The honest shape is that single-doc and canonical-versus-stale questions save the most, and ambiguous or needle questions save the least or lose. We report the per-category spread, including the losses.
Does this hold on my repo?
We do not know, and we will not pretend to. The number is measured on the trovex repo with an LLM judge, not a universal guarantee. A repo with little overlapping markdown saves less. A repo thick with half-stale duplicates saves more. The benchmark exists so you get your own number in about a minute instead of taking ours on faith.
