dctx1.3.1
Evidence

Benchmarks

Compare dctx with grep plus file reads, then measure the whole agent session. Startup instructions, tool definitions and extra turns can outweigh smaller answers.

Grep plus file reads

Measured whole-session results, 11 November 2026. Full report and method

Measured on 11.11.2026. NOT STABLE: dctx remains an advanced tool. On this corpus and model dctx did not save 25 % tokens. The pre-registered success rule was not met: CLI median total tokens were -5.01 % versus grep. The claim supported here is accuracy and bounded context, not token savings.

Method#

  • Corpus: a private 13-crate Rust workspace; it is not public.
  • Questions: 20, covering code location, call sites and multi-hop traces.
  • Model: claude-sonnet-5-5 via Claude CLI.
  • dctx: 1.3.0 (156e1bb), not current HEAD.
  • Three repeats per arm. Only tool exposure differs; prompt, model and question order are identical. Blind grading uses opaque ids without arm labels.
Arm Tools and cache conditions
A Read + Grep/Glob: grep plus file reads baseline
B Read + Bash restricted to dctx *; index warm before measurement

These are whole-session measurements, including conversation replay and provider cache usage, rather than estimates of tool stdout. The warm index in B excludes its preparation cost; these results do not measure cold indexing.

Cost and turns#

Totals per arm cover all 20 questions; each cell is the median [min, max] of three runs. Total tokens = uncached input (including cache creation) + cached input + output. Each metric is summarized independently, so component medians need not sum to the median total. Turns are CLI num_turns, an assistant-turn proxy, not exact tool calls.

Metric A: grep plus reads B: dctx CLI
Total tokens 14,904,014 [14,604,149, 15,622,381] 14,157,974 [11,221,908, 15,613,375]
Uncached input tokens 286,289 [284,340, 290,043] 250,479 [220,537, 258,313]
Cached input tokens 14,543,028 [14,241,089, 15,246,829] 13,833,705 [10,939,641, 15,278,757]
Output tokens 78,720 [74,697, 85,509] 73,790 [61,730, 76,305]
Wall time (s) 700.87 [697.78, 811.59] 719.84 [628.75, 768.06]
Turns (num_turns) 172 [159, 175] 130 [129, 137]

B versus A median total tokens: -5.01 %.

Accuracy: both metrics#

The official metric is strict citation recall: exact repo-relative path and line, with a tolerance of +/-3 lines, against 574 required line references per run. The diagnostic metric is file-level recall: the required file named in full anywhere in the answer, against 106 file/question pairs per run.

Metric A: grep plus reads B: dctx CLI
Official strict citation recall, median [min, max] 10.10 % [10.10, 20.73] 14.46 % [8.36, 22.65]
Diagnostic file-level recall, median 53.77 % 71.70 %

A grader audit found no grader defect. Low strict recall comes from answers citing shorthand paths such as store.rs:313, and from line-level granularity: 10 trace questions hold 82 % of required lines. File-level recall is a diagnostic, not a replacement metric. No file-level spread is supplied.

Where savings would come from#

dctx on its own does not save tokens by default. Token savings come from systems that use dctx: a harness or agent system built around its cache-hit philosophy, reusing byte-identical, prefix-stable context across long sessions over a large body of work. In that setting it behaves like any sufficiently advanced caching mechanism, with one difference: you have complete control over every byte of the context. The system must arrange that reuse; merely exposing dctx tools does not do it. There is no measured, citable savings figure for such a system yet. See the philosophy.

Verdict and limitations#

The pre-registered rule required B to save at least 25 % median total tokens and match or exceed A's recall. NOT MET on tokens (-5.01 %), even though B's median official recall exceeded A's. On this corpus and model dctx did not save 25 % tokens.

B named more of the required files than grep at about equal token cost. This supports an accuracy and bounded context claim, not a whole-session token-savings claim. The bounded context behavior is described in Choosing how to use dctx; these results do not establish a general accuracy advantage.

Limitations: one corpus, one model, three repeats, and no significance claim. The corpus and raw answers are private and cannot be published; these aggregate results cannot be independently reproduced from the public repository alone. dctx 1.3.0 is not current HEAD. The guidance Under ~200 files, use grep stays.

The earlier measurement evidence preserves the historical tool-output protocols separately from this whole-session study.

Secondary historical evidence

All other-tool tables below preserve the October research record. Their tokens are tool stdout estimates, not whole-session cost; scores measure gold facts in raw output without an agent model. The dctx team designed the 70-question protocol. The independent run selected repositories by seed and used language-server gold.

Historical run 7: task-level protocol

70 questions, 354 gold facts, 31 October 2026. BENCHMARK-20261030.md
ToolPoints /140Gold facts /354TokensFacts per 1k tokens
dctx 53f8b6b140 (run 6: 136)353 (342)29,34012.0
Trail-Code 0.28.012231288,9233.51
graphify 0.9.97, code-only96238197,2031.21

By question type

Points out of 14 per type, from run 5 and task re-run 3 (dctx a906884), the last run with a per-type table; run 7 then scored every point on all ten types. run 7 note

BENCHMARK-20261030.md
TypedctxTrail-Codegraphify
definition and callers141010
impact141312
outline141412
config key14146
exports141414
source14120
callees1088
overview141311
collision141213
concept141210

Independent benchmark, 31 October 2026

Seeded selection of 3 repos the tools never saw (zoxide, requests, gin), gold from language servers and repo history, published regardless of outcome. Tasks: dctx 22/28 points, 31/36 facts, 6.85 facts per 1k tokens (Trail-Code 19/28, graphify 14/28). Callers against LSP gold, first page: dctx P 0.730 / R 0.341, Trail-Code P 0.731 / R 0.313, graphify P 0.920 / R 0.264. Honest read: dctx leads on task answers and token efficiency; caller recall on unseen repos is far below the in-house corpus (0.851), so the caller floors above hold only for the curated corpora. Full table and raw JSON: docs/research/bench-independent/results/20261031T191225Z/.

BENCHMARK-20261030.md

Cold index

Run 7 medians of three runs: dctx 0.13–1.20 s on 6 corpora, Trail-Code 1.04–52.83 s, graphify 4.59–59.56 s (5 corpora; no index on docs).

run7 aggregate.json
corpusdctxTrail-Codegraphify
flask0.13 s3.82 s8.02 s
ripgrep0.88 s3.21 s10.33 s
pocketbase0.93 s50.54 s59.56 s
trailcode1.20 s52.83 s45.55 s
mixed0.39 s3.97 s4.59 s
docs0.19 s1.04 sno index

Run 1 (30 October), with peak memory and index size: median wall s, peak RSS of the largest process, index size.

BENCHMARK-20261030.md
corpusdctxTrail-Codegraphify
flask (Python)0.16 s, 19 MB, 1.1 MB4.17 s, 225 MB, 4.4 MB8.13 s, 102 MB, 4.5 MB
ripgrep (Rust)0.54 s, 25 MB, 2.2 MB3.01 s, 303 MB, 8.4 MB10.45 s, 93 MB, 8.6 MB
pocketbase (Go+JS)0.91 s, 63 MB, 9.9 MB50.63 s, 987 MB, 41.8 MB58.96 s, 243 MB, 26.3 MB
trailcode (TS)1.16 s, 90 MB, 14.8 MB58.34 s, 1,020 MB, 41.1 MB44.32 s, 394 MB, 11.9 MB
mixed (10 langs)0.24 s, 74 MB, 2.8 MB3.84 s, 470 MB, 12.8 MB4.46 s, 74 MB, 2.2 MB
docs0.18 s, 74 MB, 1.4 MB1.17 s, 121 MB, 0.0 MBfailed: no index

Warm queries and answer size

Run 1, median ms of 5 after one warm-up, including process start, and estimated tokens. BENCHMARK-20261030.md
corpusquerydctxTrail-Codegraphify
flasksearch81 ms, 2181,066 ms, 262564 ms, 1,845
flaskcallers47 ms, 479988 ms, 136381 ms, 32
flaskskeleton44 ms, 866991 ms, 852514 ms, 474
flaskoverview45 ms, 4911,009 ms, 1,1227 ms, 5,930
pocketbasesearch213 ms, 2591,573 ms, 3581,393 ms, 97 (exit 3)
pocketbasecallers214 ms, 4641,195 ms, 175777 ms, 24
pocketbaseoverview226 ms, 6521,232 ms, 1,5148 ms, 11,591
trailcodecallers200 ms, 4721,180 ms, 1,064715 ms, 196
trailcodeoverview195 ms, 5001,159 ms, 4997 ms, 10,076

Byte stability

Build 0495f5f, 6 November 2026, three pinned corpora, 11 query shapes each. REPORT.md · full report
MeasurementTextJSON
Answers identical after unrelated edit33/3333/33
Answers identical over five separate invocations33/3333/33
Baseline post-edit common-prefix ratio, minimum / mean1.0 / 1.01.0 / 1.0
Consecutive same-request transitions, five per case165165
Same-request M7, minimum / mean1.0 / 1.01.0 / 1.0
Consecutive changing-question transitions195195
Changing-question M7, minimum / median / mean0.007795889 / 0.107142857 / 0.2148302570.008776458 / 0.067594433 / 0.182446794
Changing-question M7, total prefix bytes / total current bytes0.0618469780.079892615

Not claimed

These exclusions apply to the historical tool-output protocols below the whole-session summary above.

  • Whole-session cost or task-level answer quality from these historical protocols. Historical fact scores describe raw tool outputs only; the 11 November study measures whole sessions with a frozen model.
  • Anything about graphify's LLM semantic graph, or Trail-Code's learnings and LSP edges.
  • SWE-bench scores or agent-run numbers from these historical tool-output protocols.
  • Equal language depth. dctx's 25 new formats are outline tier: symbols, not call edges.

Historical raw data

The whole-session corpus and raw answers are private and cannot be published. The links below cover only the historical studies.