Agent test

Date: 2026-11-08. Model: claude-sonnet-5-5 (the id the CLI reports for the sonnet alias). Pre-registration: PREREGISTRATION.md, committed before the first paid session, with one amendment (written into it) made after the 3-trial checkpoint and before the extension. Machine-readable numbers: summary.json. Every table for every repo is in tables.md; raw per-session JSON is in results/; the cost ledger is ledger.jsonl.

Headline

Stacklit did not make these sessions cheaper, and on this task set it made the answers slightly less often right. It did cut tool calls and turns by about a quarter to a third, and it nearly removed Grep and Glob use. Sessions here were already cheap (about 0.026 USD, 3.6 turns), because the questions can be answered with one or two greps on code the model has not seen.

Pooled over the three repos, 150 paired sessions per arm (10 tasks x 5 trials x 3 repos), stacklit arm vs control, 95 percent bootstrap intervals:

Strict arm (crawl guard in redirect mode plus steering variant d, 150 sessions, binary from origin/feat/source-of-truth at fdc0f40) vs control:

The strict arm did worse than the plain stacklit arm on cost, tokens and correctness. The crawl guard itself fired in only 8 of 150 sessions (17 redirects, all on grep -r style Bash calls), so most of the difference comes from the stronger rules text and the agent routing its search through stacklit commands instead of Grep.

Where Stacklit helped and where it did not

Pooled stacklit arm vs control by task kind (per-repo tables with intervals are in tables.md):

• Helped: add-feature placement and locate questions. Fewer tool calls (about 0.4x) and 12 to 22 percent lower cost.
• No effect on cost: trace questions.
• Hurt: impact questions. A single exact-string Grep answers “who imports X” in one call (1.2 calls on average). The stacklit arm runs a stacklit query first and then usually greps anyway, so it takes about twice the calls and 1.6x the cost. In the strict arm, which pushes the agent away from grep, impact questions cost 2.5x and were right only 77 percent of the time.
• Correctness losses concentrate in trace tasks (97.5% to 82.5%, difference -15 points [-27.5, -5]). The clearest case is lr-10 (three files for waking a sleeping app): in both treatment arms the agent ran one stacklit query, answered at once with internal/api/app_sleep.go instead of internal/api/app_sleep_wake.go, and was wrong in 5 of 5 trials in each arm; the control arm grepped and was right in 5 of 5.
• By repo, stacklit vs control cost ratio: levelrail 0.78 [0.70, 0.86] (the largest repo, where it helped), theauth-go 1.10 [0.98, 1.25], featuredrop 1.23 [1.14, 1.33] (the smallest repo, where the injected map is mostly overhead).

Memorisation check

Six tasks per repo, all tools disabled (--tools ""), one trial each, 18 sessions. Correct: 0 of 18 (0 of 6 on every repo). Several answers were plausible layout guesses and none matched, so no repo was excluded. This check asks for exact paths, so it shows the model does not know the layout, not that it cannot guess a file name.

Method

• Repos, pinned by full SHA: glincker/levelrail (Go and TypeScript, about 16,000 files), glincker/theauth-go (Go, about 1,300 files), glincker/featuredrop (TypeScript, about 400 files). All public, pushed Oct to Nov 2026.
• Tasks: 10 per repo in tasks/, written by reading the code, answers verified by grep and file existence. Kinds: locate, trace, impact, add. Grading: every all_of path and at least one any_of path named in the answer, boundary-aware match.
• Arms: control (Read, Grep, Glob only); stacklit (stacklit generate --graph, then stacklit setup claude --session-hook --no-mcp --no-git-hook, plus a Bash tool restricted to Bash(stacklit *)); strict (as stacklit, with --crawl-guard=redirect and STACKLIT_STEERING=d).
• Each session: claude -p with --output-format stream-json --verbose --no-session-persistence --setting-sources project,local --strict-mcp-config --disable-slash-commands --tools ... --max-turns 25 --max-budget-usd 1.5 --model sonnet, in a fresh shallow clone in a temp dir, sequential, seed-shuffled. --strict-mcp-config --disable-slash-commands cut the fixed context of an empty session from about 64,000 to about 5,400 tokens (account connectors and skills were otherwise loaded). I added them after a first trivial test, before any measured session.
• Trials: 5 per task and arm. The pre-registration asked for 3; trials 4 and 5 were added in Amendment 1 because the observed cost (0.03 USD per session) left most of the cap unused.
• Statistics: paired by repo, task and trial; seeded bootstrap (seed 20260508, 10,000 resamples of pairs) for the mean difference and ratio of means; Wilson intervals for rates.
• Sessions: 150 control, 150 stacklit, 150 strict, 18 memory check. The 10 pilot sessions are the first part of the full run (same task, trial and arm), not duplicated. Total notional spend from the ledger: 12.54 USD of the 60 USD cap. No rate limit, usage limit or authentication error occurred, and no session failed for harness reasons.
• Spot check: 45 of 450 sessions (10 percent, seeded random) were compared by hand, expected paths against the answer. Nothing new beyond the three defects below; the known fd-07 defect appeared twice. The false answers I saw (lr-10, fd-05, tg-05) were genuinely wrong or incomplete.

Grading defects found

The correct column is the pre-registered grader. correct_audited adds three fixes, applied to both arms identically and decided from the spot check and the failure list: (a) the matcher rejected answers written as ./path or as an absolute path inside the checkout (this hit the treatment arms more, since they often answer from shell grep output), (b) fd-07 asks where to add a new adapter, and the model correctly names a new file in src/adapters/, which the pinned any-of list (existing files) did not allow, (c) fd-09: src/analytics.ts (PostHogAdapter) also forwards events to PostHog and was missing from the pinned answer. After the fixes the gap between control and stacklit shrinks from -6.0 to -3.0 points, and its interval includes zero. For the strict arm the gap stays clear (-9 points).

Pooled tables

pooled: control vs stacklit (150 pairs)

Correctness after audit fixes: control 147/150 = 98.0% [94.3, 99.3]; stacklit 143/150 = 95.3% [90.7, 97.7].

Correctness (pre-registered grader): control 141/150 = 94.0% [89.0, 96.8]; stacklit 132/150 = 88.0% [81.8, 92.3]. Stacklit commands used in 150/150 = 100.0% [97.5, 100.0] of treatment sessions; session hook fired in 150/150 = 100.0% [97.5, 100.0].

By task kind (pooled):

pooled: control vs strict (150 pairs)

Correctness after audit fixes: control 147/150 = 98.0% [94.3, 99.3]; strict 134/150 = 89.3% [83.4, 93.3].

Correctness (pre-registered grader): control 141/150 = 94.0% [89.0, 96.8]; strict 124/150 = 82.7% [75.8, 87.9]. Stacklit commands used in 150/150 = 100.0% [97.5, 100.0] of treatment sessions; session hook fired in 150/150 = 100.0% [97.5, 100.0].

By task kind (pooled):

Limitations

• One model (Sonnet), one day of runs, one machine. Service load can differ across the day; the strict arm ran after control and stacklit and is not interleaved with control, so time-of-day or cache effects are not balanced for that comparison.
• Notional cost is the CLI’s list-price figure, not membership usage. Cache reads dominate tokens, so cost and the uncached-equivalent view are both given.
• Tasks are written by us, with a known file answer, from code we know. Ten per repo, three repos, so n is small and the by-kind rows rest on 30 to 50 pairs. Trials of one task are not independent draws of tasks; the intervals understate uncertainty about other tasks.
• These tasks can be answered with one or two exact-string greps, so a map has little room to save exploration. Opening files was almost absent in both arms (0.1 Read calls per control session), so the claim about files read cannot be tested here. Tasks that need reading and reasoning across files, or editing, are not covered.
• Correctness is a path match, not an understanding check. The audited column is a post-hoc view.
• Control has no Bash tool; treatment arms have one limited to stacklit *. That is the existing harness design. A control with a shell would grep more flexibly.
• seconds includes the clone and, for treatment arms, stacklit generate and setup (about 30 seconds on levelrail). It is not model time and must not be quoted as latency.
• The repos ship their own agent guide files, present in both arms, and the user-level CLAUDE.md was loaded in all arms (a few thousand tokens).
• The harness looked for the crawl guard state file in the wrong directory, so redirect counts for the strict arm were taken from the transcripts afterwards (8 sessions, 17 redirects). The harness was fixed after the run.
• Transcripts are not committed. A call log (tool and key argument) per session is in each result JSON.

What we claim, and what we do not

Source: RESULTS.md and summary.json. Model claude-sonnet-5-5, 3 repos, 30 tasks, 5 trials, 150 pairs per arm, bootstrap 95 percent intervals. Each claim names the cell it comes from. Wording is meant to be copied as is.

Claims the data supports

1. “In a measured A/B with real Claude Code sessions on three repos, sessions with Stacklit used about a third fewer tool calls than sessions without it (1.77 vs 2.58 per session, ratio 0.68, 95% CI 0.61 to 0.77).” Cell: pooled, stacklit, tool_calls.
2. “They also took about a quarter fewer model turns (2.77 vs 3.58, ratio 0.77, CI 0.71 to 0.84).” Cell: pooled, stacklit, turns.
3. “Grep and Glob calls fell by more than four fifths (2.48 to 0.45 per session, ratio 0.18, CI 0.14 to 0.23).” Cell: pooled, stacklit, grep_glob_calls.
4. “On the largest repo (about 16,000 files) notional cost per session fell 22 percent (ratio 0.78, CI 0.70 to 0.86) and uncached-equivalent tokens fell 18 percent (0.82, CI 0.73 to 0.92).” Cell: levelrail, stacklit, cost_usd and uncached_eq_tokens. One repo, 50 pairs.
5. “For locate and add-feature questions, tool calls fell to about 0.4x (0.43 [0.38, 0.48] and 0.41 [0.35, 0.48]) and cost fell 12 to 22 percent (0.88 [0.79, 0.98] and 0.78 [0.68, 0.90]).” Cell: pooled by kind, stacklit.
6. “Stacklit commands were used in every treatment session (150 of 150), and the session hook fired in every one.” Cell: stacklit_used_in_treatment.
7. “The models had not memorised these repos: with all tools off they got 0 of 18 tasks right.” Cell: memorisation check.

Claims the data does NOT support (do not make them)

1. That Stacklit lowers cost or tokens in general. Pooled cost ratio is 1.00 [0.93, 1.07] and uncached-equivalent tokens 1.03 [0.97, 1.11]: no saving. On the small repo featuredrop it cost more (1.23 [1.14, 1.33]), and on theauth-go the interval is 0.98 to 1.25.
2. That Stacklit improves answer quality. Correctness was 94.0% control vs 88.0% stacklit (pre-registered grader, difference -6.0 points [-10.7, -1.3]) and 98.0% vs 95.3% after audited grading (difference -3.0 points, interval includes zero). Trace tasks were worse (97.5% to 82.5%).
3. That Stacklit reduces files read. Control sessions barely opened files (0.1 per session), so there was nothing to reduce; the measured drop to 0 is not a meaningful saving.
4. That Stacklit helps “who imports X” (impact) questions. They cost 1.59x [1.47, 1.72] with the plain arm because one exact grep is enough without it.
5. Anything about the strict arm (crawl guard redirect plus steering d) being better. It cost more (1.12 [1.02, 1.24] in dollars, 1.37 [1.18, 1.59] in total tokens) and was less often right (-11.3 points [-17.3, -6.0]; audited -9 points [-14, -3]). The guard fired in only 8 of 150 sessions, so this result is about the stronger rules text, not mainly about redirects.
6. Any latency claim. The seconds metric includes clone and stacklit generate.
7. Any claim on editing tasks, other models, other repos, harder multi-file reasoning tasks, or membership usage (the dollars are notional list-price figures).
8. That the numbers are final for the strict arm: it was not interleaved with control.

Generated from bench/agent-ab/RESULTS.md in the repository, the single source of truth.