SX1 results: the science-evaluation comparison, run once
2026-11-10. Pre-registration: docs/dd/sx1-prereg.md (sealed in commit 9521dd3, re-audit cleared in 4f0cd16, before any test read). Score: data/dd/sx1/score.json, sha256 755ce132b4b26a14f5644ec5ac0caf83997edee214e6010c9a7499494880529b. 24 packs, 121 hard planted errors, 423 unaltered claims. No void read, no dropped pack. Every agent ran in-session on Sonnet, medium effort. Scoring is by code against the key.
The two primaries
- Fan-out against one pass: no win. Witnessed rate on hard errors: fan-out (S) 89%, one pass (B1R) 91%. Difference -1.7 points (95% interval -10 to +5), p 0.77, Holm p 0.77. The ceiling rule had not fired. Rule R1: one pass is the default reader. Fan-out is used only when a pack exceeds one read.
- Retrieval against none: a win. Youden's J (flag rate on hard planted claims minus flag rate on unaltered claims): one pass with retrieval 0.92, one pass without 0.50. Difference +0.42 (95% interval +0.33 to +0.51), p under 0.001 (no bootstrap draw at or below zero in 10,000), Holm p under 0.001. Rule R2 is met: retrieval adds detection, not only the quote.
All arms
| Arm | Hard: found with the right paper | Hard: flagged at all | Hard: restated as true | Unaltered: flagged |
|---|---|---|---|---|
| One pass with retrieval (B1R) | 110 of 121 (91%, 85 to 96) | 112 of 121 (93%, 87 to 98) | 8 of 121 (7%, 2 to 12) | 2 of 423 (0%, 0 to 1) |
| Fan-out (S) | 108 of 121 (89%, 82 to 96) | 110 of 121 (91%, 84 to 96) | 13 of 121 (11%, 4 to 18) | 7 of 423 (2%, 0 to 3) |
| One pass, no retrieval (B0) | 0 of 121 (0%, 0 to 0) | 63 of 121 (52%, 43 to 61) | 0 of 121 (0%, 0 to 0) | 7 of 423 (2%, 0 to 3) |
| Lens plus string rule (L0) | 3 of 121 (2%, 0 to 5) | 3 of 121 (2%, 0 to 5) | 0 of 121 (0%, 0 to 0) | 14 of 423 (3%, 2 to 5) |
By type (found with the right paper; for B0, flagged)
| Type | One pass (B1R) | Fan-out (S) | No retrieval (B0), flagged only | String rule (L0) |
|---|---|---|---|---|
| X1 direction flipped | 58 of 66 (88%) | 55 of 66 (83%) | 32 of 66 (48%) | 1 of 66 (2%) |
| X2 animal and human swapped | 17 of 18 (94%) | 17 of 18 (94%) | 17 of 18 (94%) | 0 of 18 (0%) |
| X4 number from the same abstract | 35 of 37 (95%) | 36 of 37 (97%) | 14 of 37 (38%) | 2 of 37 (5%) |
| X5 number from nowhere (easy) | 32 of 32 (100%) | 32 of 32 (100%) | 2 of 32 (6%) | 32 of 32 (100%) |
There is no X3 (negation) error in the test: the survey found none in eligible packs.
Wording licensed for the brief (default arm, one pass)
- Hard errors: "catches". 91% (85 to 96) on 121, with a false-alarm rate of 0.5% (2 of 423).
- X1 and X4: "catches". X2: counts only (17 of 18; under 30 errors).
Beside the primaries (descriptive)
- On the 58 hard errors the no-retrieval read did not flag, one pass found 51 (88%) and fan-out 48 (83%).
- The checker changed almost nothing. After it, one pass stands at 109 of 121 found and 2 false alarms; fan-out at 108 found and 6 false alarms. It removed one true catch and no false alarm from one pass. On this test it does not pay for itself.
- Fan-out raised more false alarms than one pass (7 against 2 of 423): chunk readers that hold no paper for a claim fall back on
doubt. It also restated more errors as true (13 against 8). - Records. One pass: 544 passed and 32 refused in the first round, 26 recovered by the correction round. Fan-out: 2205 passed, 51 refused, 51 recovered. No retrieval: 570 passed, 6 refused.
- Cost, fresh input tokens over 24 packs. One pass reader 1.52M in 38 reads (corrections included); fan-out reader 2.64M in 124; no retrieval 0.34M in 29; each checker about 1.1M.
Caveats (read before using any number above)
- Planted errors are an upper bound. One word or number was changed, and the paper that shows the truth was in the pack. A real misstatement is messier, and the contradicting paper may not be in the corpus at all.
- The claims were built from the same abstracts. Each claim is a model's extraction of the paper it is checked against. The test shows a reader can find the source among 24 to 60 abstracts and notice a changed detail. It does not show it can judge a company's own wording, or find a different paper that disagrees.
- The no-retrieval read is not weak. It flagged 52% of hard errors from plausibility alone, 94% of animal-and-human swaps. Retrieval's edge is largest on numbers (95% against 38%) and smallest on model-system swaps (no edge on flagging).
- Fan-out is not shown to be worse. The interval runs from 10 points worse to 5 better. One pass is the default because it is cheaper and not measurably worse, at packs of 24 to 60 abstracts. Larger packs were not tested.
- "Supported" verdicts, independent support, the ranking and the framed read are not measured here.
- About 1 in 50 unaltered claims is itself a wrong extraction, so some of the false alarms may be real. They were not read.
- All readers are Claude models, as is the extractor. Scoring is code.
- Types are uneven. X1 is 55% of hard errors; X2 has 18.
Independent audit (docs/audits/sx1-results-audit.md): pass with notes
A fresh backtest-auditor recomputed every count and both primaries with its own script and found no mismatch. It read all 191 reader transcripts (1,692 tool calls): none left its listed files or touched a key. Nothing was re-run. Its notes, which change no number:
- "Found with the right paper" is strict. Of the 11 hard errors one pass did not find, 8 are real misses: the reader marked the altered claim supported, twice after noticing the tension. The other 3 are real flags the rule does not credit: one quoted a second copy of the source abstract under another paper id, one contradicted the claim from a different paper, and one was a correct record refused for a quote that code could not find. The flagged column (112 of 121) is the looser measure.
- Both of one pass's false alarms look like wrong extractions, not reader errors. The auditor read them: in each, the unaltered claim disagrees with its own abstract. So 0.5% is an upper bound here, and the reader found two real errors in the Lens extractions without being asked.
- Eight random catches were read and all are genuine: the quoted sentence shows the altered word or number.
- Not audited: checker-stage numbers, token counts and the string-rule row.