Benchmarks
Compare dctx with grep plus file reads, then measure the whole agent session. Startup instructions, tool definitions and extra turns can outweigh smaller answers.
Grep plus file reads
Measured whole-session results, 11 November 2026. Full report and method
Measured on 11.11.2026. NOT STABLE: dctx remains an advanced tool. On this corpus and model dctx did not save 25 % tokens. The pre-registered success rule was not met: CLI median total tokens were -5.01 % versus grep. The claim supported here is accuracy and bounded context, not token savings.
Method#
- Corpus: a private 13-crate Rust workspace; it is not public.
- Questions: 20, covering code location, call sites and multi-hop traces.
- Model:
claude-sonnet-5-5via Claude CLI. - dctx: 1.3.0 (
156e1bb), not current HEAD. - Three repeats per arm. Only tool exposure differs; prompt, model and question order are identical. Blind grading uses opaque ids without arm labels.
| Arm | Tools and cache conditions |
|---|---|
| A | Read + Grep/Glob: grep plus file reads baseline |
| B | Read + Bash restricted to dctx *; index warm before measurement |
These are whole-session measurements, including conversation replay and provider cache usage, rather than estimates of tool stdout. The warm index in B excludes its preparation cost; these results do not measure cold indexing.
Cost and turns#
Totals per arm cover all 20 questions; each cell is the median [min, max] of three runs.
Total tokens = uncached input (including cache creation) + cached input + output.
Each metric is summarized independently, so component medians need not sum to
the median total. Turns are CLI num_turns, an assistant-turn proxy, not exact
tool calls.
| Metric | A: grep plus reads | B: dctx CLI |
|---|---|---|
| Total tokens | 14,904,014 [14,604,149, 15,622,381] | 14,157,974 [11,221,908, 15,613,375] |
| Uncached input tokens | 286,289 [284,340, 290,043] | 250,479 [220,537, 258,313] |
| Cached input tokens | 14,543,028 [14,241,089, 15,246,829] | 13,833,705 [10,939,641, 15,278,757] |
| Output tokens | 78,720 [74,697, 85,509] | 73,790 [61,730, 76,305] |
| Wall time (s) | 700.87 [697.78, 811.59] | 719.84 [628.75, 768.06] |
Turns (num_turns) |
172 [159, 175] | 130 [129, 137] |
B versus A median total tokens: -5.01 %.
Accuracy: both metrics#
The official metric is strict citation recall: exact repo-relative path and line, with a tolerance of +/-3 lines, against 574 required line references per run. The diagnostic metric is file-level recall: the required file named in full anywhere in the answer, against 106 file/question pairs per run.
| Metric | A: grep plus reads | B: dctx CLI |
|---|---|---|
| Official strict citation recall, median [min, max] | 10.10 % [10.10, 20.73] | 14.46 % [8.36, 22.65] |
| Diagnostic file-level recall, median | 53.77 % | 71.70 % |
A grader audit found no grader defect. Low strict recall comes from answers
citing shorthand paths such as store.rs:313, and from line-level granularity:
10 trace questions hold 82 % of required lines. File-level recall is a
diagnostic, not a replacement metric. No file-level spread is supplied.
Where savings would come from#
dctx on its own does not save tokens by default. Token savings come from systems that use dctx: a harness or agent system built around its cache-hit philosophy, reusing byte-identical, prefix-stable context across long sessions over a large body of work. In that setting it behaves like any sufficiently advanced caching mechanism, with one difference: you have complete control over every byte of the context. The system must arrange that reuse; merely exposing dctx tools does not do it. There is no measured, citable savings figure for such a system yet. See the philosophy.
Verdict and limitations#
The pre-registered rule required B to save at least 25 % median total tokens and match or exceed A's recall. NOT MET on tokens (-5.01 %), even though B's median official recall exceeded A's. On this corpus and model dctx did not save 25 % tokens.
B named more of the required files than grep at about equal token cost. This supports an accuracy and bounded context claim, not a whole-session token-savings claim. The bounded context behavior is described in Choosing how to use dctx; these results do not establish a general accuracy advantage.
Limitations: one corpus, one model, three repeats, and no significance claim. The corpus and raw answers are private and cannot be published; these aggregate results cannot be independently reproduced from the public repository alone. dctx 1.3.0 is not current HEAD. The guidance Under ~200 files, use grep stays.
The earlier measurement evidence preserves the historical tool-output protocols separately from this whole-session study.
Secondary historical evidence
All other-tool tables below preserve the October research record. Their tokens are tool stdout estimates, not whole-session cost; scores measure gold facts in raw output without an agent model. The dctx team designed the 70-question protocol. The independent run selected repositories by seed and used language-server gold.
Historical run 7: task-level protocol
| Tool | Points /140 | Gold facts /354 | Tokens | Facts per 1k tokens |
|---|---|---|---|---|
dctx 53f8b6b | 140 (run 6: 136) | 353 (342) | 29,340 | 12.0 |
| Trail-Code 0.28.0 | 122 | 312 | 88,923 | 3.51 |
| graphify 0.9.97, code-only | 96 | 238 | 197,203 | 1.21 |
By question type
Points out of 14 per type, from run 5 and task re-run 3 (dctx a906884), the last run with a per-type table; run 7 then scored every point on all ten types. run 7 note
| Type | dctx | Trail-Code | graphify |
|---|---|---|---|
| definition and callers | 14 | 10 | 10 |
| impact | 14 | 13 | 12 |
| outline | 14 | 14 | 12 |
| config key | 14 | 14 | 6 |
| exports | 14 | 14 | 14 |
| source | 14 | 12 | 0 |
| callees | 10 | 8 | 8 |
| overview | 14 | 13 | 11 |
| collision | 14 | 12 | 13 |
| concept | 14 | 12 | 10 |
Independent benchmark, 31 October 2026
Seeded selection of 3 repos the tools never saw (zoxide, requests, gin), gold from language servers and repo history, published regardless of outcome. Tasks: dctx 22/28 points, 31/36 facts, 6.85 facts per 1k tokens (Trail-Code 19/28, graphify 14/28). Callers against LSP gold, first page: dctx P 0.730 / R 0.341, Trail-Code P 0.731 / R 0.313, graphify P 0.920 / R 0.264. Honest read: dctx leads on task answers and token efficiency; caller recall on unseen repos is far below the in-house corpus (0.851), so the caller floors above hold only for the curated corpora. Full table and raw JSON: docs/research/bench-independent/results/20261031T191225Z/.
Cold index
Run 7 medians of three runs: dctx 0.13–1.20 s on 6 corpora, Trail-Code 1.04–52.83 s, graphify 4.59–59.56 s (5 corpora; no index on docs).
| corpus | dctx | Trail-Code | graphify |
|---|---|---|---|
| flask | 0.13 s | 3.82 s | 8.02 s |
| ripgrep | 0.88 s | 3.21 s | 10.33 s |
| pocketbase | 0.93 s | 50.54 s | 59.56 s |
| trailcode | 1.20 s | 52.83 s | 45.55 s |
| mixed | 0.39 s | 3.97 s | 4.59 s |
| docs | 0.19 s | 1.04 s | no index |
Run 1 (30 October), with peak memory and index size: median wall s, peak RSS of the largest process, index size.
| corpus | dctx | Trail-Code | graphify |
|---|---|---|---|
| flask (Python) | 0.16 s, 19 MB, 1.1 MB | 4.17 s, 225 MB, 4.4 MB | 8.13 s, 102 MB, 4.5 MB |
| ripgrep (Rust) | 0.54 s, 25 MB, 2.2 MB | 3.01 s, 303 MB, 8.4 MB | 10.45 s, 93 MB, 8.6 MB |
| pocketbase (Go+JS) | 0.91 s, 63 MB, 9.9 MB | 50.63 s, 987 MB, 41.8 MB | 58.96 s, 243 MB, 26.3 MB |
| trailcode (TS) | 1.16 s, 90 MB, 14.8 MB | 58.34 s, 1,020 MB, 41.1 MB | 44.32 s, 394 MB, 11.9 MB |
| mixed (10 langs) | 0.24 s, 74 MB, 2.8 MB | 3.84 s, 470 MB, 12.8 MB | 4.46 s, 74 MB, 2.2 MB |
| docs | 0.18 s, 74 MB, 1.4 MB | 1.17 s, 121 MB, 0.0 MB | failed: no index |
Warm queries and answer size
| corpus | query | dctx | Trail-Code | graphify |
|---|---|---|---|---|
| flask | search | 81 ms, 218 | 1,066 ms, 262 | 564 ms, 1,845 |
| flask | callers | 47 ms, 479 | 988 ms, 136 | 381 ms, 32 |
| flask | skeleton | 44 ms, 866 | 991 ms, 852 | 514 ms, 474 |
| flask | overview | 45 ms, 491 | 1,009 ms, 1,122 | 7 ms, 5,930 |
| pocketbase | search | 213 ms, 259 | 1,573 ms, 358 | 1,393 ms, 97 (exit 3) |
| pocketbase | callers | 214 ms, 464 | 1,195 ms, 175 | 777 ms, 24 |
| pocketbase | overview | 226 ms, 652 | 1,232 ms, 1,514 | 8 ms, 11,591 |
| trailcode | callers | 200 ms, 472 | 1,180 ms, 1,064 | 715 ms, 196 |
| trailcode | overview | 195 ms, 500 | 1,159 ms, 499 | 7 ms, 10,076 |
Byte stability
| Measurement | Text | JSON |
|---|---|---|
| Answers identical after unrelated edit | 33/33 | 33/33 |
| Answers identical over five separate invocations | 33/33 | 33/33 |
| Baseline post-edit common-prefix ratio, minimum / mean | 1.0 / 1.0 | 1.0 / 1.0 |
| Consecutive same-request transitions, five per case | 165 | 165 |
| Same-request M7, minimum / mean | 1.0 / 1.0 | 1.0 / 1.0 |
| Consecutive changing-question transitions | 195 | 195 |
| Changing-question M7, minimum / median / mean | 0.007795889 / 0.107142857 / 0.214830257 | 0.008776458 / 0.067594433 / 0.182446794 |
| Changing-question M7, total prefix bytes / total current bytes | 0.061846978 | 0.079892615 |
Not claimed
These exclusions apply to the historical tool-output protocols below the whole-session summary above.
- Whole-session cost or task-level answer quality from these historical protocols. Historical fact scores describe raw tool outputs only; the 11 November study measures whole sessions with a frozen model.
- Anything about graphify's LLM semantic graph, or Trail-Code's learnings and LSP edges.
- SWE-bench scores or agent-run numbers from these historical tool-output protocols.
- Equal language depth. dctx's 25 new formats are outline tier: symbols, not call edges.
Historical raw data
The whole-session corpus and raw answers are private and cannot be published. The links below cover only the historical studies.
- docs/research/bench-3way-20261030/: questions, gold facts, scorer, raw results and scripts
- The full write-up, with every run and the method
- The earlier two-way study (October 2026, before call edges)