did-it — Checking an AI Agent's Claims Against Its Transcript

Coding agents routinely claim work they didn’t do — “the tests pass”, “I created the file”, “the suite is green”. Verifying that today is manual; METR reports it as “the majority of the work” in an eval run. did-it makes it a command: point it at a Claude Code session transcript and it emits a per-claim receipt saying whether the transcript’s own evidence backs the claim, contradicts it, or supports neither.

Context

The core problem is claim-versus-evidence: an agent’s prose says one thing, and the tool calls it actually made — exit codes, test-framework summaries, file operations — may say another. The nearest prior art names the problem without solving it for this surface. NabaOS (“Tool Receipts, Not Zero-Knowledge Proofs”, arXiv:2603.10060) frames it for generic tool use; METR’s evaluators reconcile agent claims against reality by hand. did-it is the coding-specialized instrument: one agent’s claims, checked against its own session’s execution record.

The design constraint that shapes everything is that a false accusation is the credibility-killing error. A tool that falsely accuses an honest agent is worse than no tool, so did-it is built precision-first: it accuses only when a claimed pass meets the test framework’s own failure marker on a temporally-valid run, and every ambiguity routes to a safe abstention instead.

What it does

For each procedural claim the agent made, did-it returns one of six verdicts:

• BACKED-transcript — in-transcript evidence supports the claim at the moment it was made.
• BACKED-verified — an optional --verify <repo> re-ran the claim’s own test command and it passed.
• UNSUPPORTED — no supporting evidence found. This is the safe abstention; all ambiguity lands here, never in an accusation.
• CONTRADICTED — the framework’s own failure marker contradicts a pass-claim, temporally valid, verbatim span in hand. The only accusation, held to a high bar.
• NOT-CHECKABLE — a semantic claim (“fixed the bug”) the transcript can’t adjudicate.
• NOT-EVALUABLE — an unparseable transcript or unknown schema version. Unknown fails closed here, never to a verdict.

The logic is temporal: a claim is judged against evidence as of the moment it was uttered. A green run must precede the claim, and a code edit between the run and the claim voids the evidence. A green run followed by a source edit backs nothing; a red run followed by a fix accuses nobody. There is no LLM in the hot path — every pattern, from the claim grammar to the runner matchers, is published in the source, greppable and auditable, with no judge model to introduce circularity.

Show it working

No Claude Code session handy? The repo ships a fabricated fixture whose “tests pass” claim is contradicted by a failing run:

$ pip install -e '.[dev]'
$ did-it fixtures/corpus/test-green-0-flip_exit_code.jsonl
CONTRADICTED  [toolu_fx0002]  The test suite is green: 158 passed.
                  · last test run: 'Exit code 1; test result: FAILED'

did-it: 1 claim(s) — CONTRADICTED: 1

The exit code is non-zero only on CONTRADICTED, so it drops into CI or a Claude Code Stop hook, where it prints the receipt and never blocks the stop.

Honest scope

The narrow scope is deliberate. did-it reads Claude Code transcripts (schema versions 2.1.156–2.1.205) and adjudicates procedural claims only; a runner whose failure summary it can’t read is excluded rather than mislabeled. Its two ground truths are measured separately. The synthetic corpus — truthful sessions mutated into labeled lies by published operators — carries the reproducible headline: precision 1.0 on the held-out split, where any unexpected CONTRADICTED on any fixture counts against it. Alongside that, a private execution-labeled anchor of my own real coding sessions cross-checks external validity: across 400 real sessions the tool issued zero CONTRADICTED verdicts, so zero false accusations — after calibration surfaced and eliminated three real false-accusation classes. The real anchor fired no accusations at all, so it bounds the false-positive rate rather than measuring precision directly; the 1.0 is a synthetic-corpus number, and I keep those two claims apart.

did-it is the detect half of a pair — reconciling what an agent claimed against what it did. The prevent half is excubitor, the guards that stop an unattended loop from taking an irreversible action on its own say-so.