A specialist science agent lost to one careful read. Here is what I built instead, and what it measurably does.

2026-11-10. Plan: docs/plans/2026-11-10-science-eval-plan.md. Everything ran on this machine, in-session, at no API cost. Nothing was fetched and nothing was pushed.

1. The short version

• The setup. dd-science, a specialist agent, had lost its one test: it proofread trial abstracts no better than a single general read. Separately, the Science Lens could find labs and papers as of a date. The two had never met.
• What I built. A pipeline: the Lens finds, one sealed reader evaluates, code ranks. For a topic or a target as of a date, it checks each extracted claim against the retrieved abstracts and returns a verdict with two exact quotes, one from the claim and one from the paper. Code checks both quotes, computes independence and tier, and sorts.
• What it measurably does. On 121 planted misstatements across 24 topics, the reader found 110 and named the paper that shows each (91%, 95% interval 85 to 96). It raised 2 false alarms on 423 unaltered claims. A plain read with no literature flagged 52% and could name no paper. The Lens with string rules found 2%.
• What it does not do. Splitting the reading across several agents did not help: 89% against 91% for one pass. So the "agent" is one read. And on five real subjects it found no independent support and no contradiction between labs. Today the real brief is a verified, quoted evidence listing with a tier sort. The cross-lab layer works on planted errors and has found nothing real yet.
• The main weakness. Every error was planted by code, and the true paper was always in the pack. No person has checked a real case.

2. What I decided and why

Routes I cut: reshaping the proofreading test again (spent); giving the agent search tools (nothing sealed, nothing reproducible); AI-judged "usefulness" (AI grading AI prose); an agent that grades each paper's design in prose (unmeasured opinion, when the Lens field is already measured at 95%).

Success and the change-course rules were written before any build step. None of the stop rules fired.

3. What I built

1. Three fixes the handoff named.
   • A correction round can only rewrite a refused finding. An added finding is refused with a reason.
   • dd-verifier2: a claim that a source lacks something is judged against the whole block, not a quote.
   • An unseeded pack is no longer described to a judge as changed.
2. The pack builder (src/diaspore/dd/scieval.py). Topic mode: the top labs' claims, at most 24, and up to 60 abstracts. Target mode: a lab's own claims, or a company's possible founding labs' claims, plus other labs' papers. The claims sheet never says which paper a claim came from. The key stays outside the pack.
3. The reader contract (prompts/dd/scieval-v1.md). Per claim and paper: supported, contradicted, doubt or not found, with two quotes. No number of its own.
4. Code that does the rest (scibrief.py): independence by lab and author overlap, tier from the Lens model-system field, counts, and a published sort rule (scirank-v1). A second prompt checks each record; a third writes a framed read (the claim that carries the most weight, what must hold, data-room questions) from validated records only. Its numbers are checked by code.
5. The test (mutate_claims.py, sx1.py): code plants one false detail in a claim; code scores against a key. No AI judge anywhere.
6. The command. /sci-brief topic "<words>" <date>, or lab <id>, or company <id>; underneath, scripts/dd_sci_run.py. An MCP tool, science_brief, serves built briefs.

Try it:

PYTHONPATH=src ~/.venvs/diaspore-sk122/bin/python scripts/dd_sci_run.py brief topic "KRAS inhibitor" --as-of 2025-12-31

Then follow .claude/commands/sci-brief.md. The agents run in a Claude Code session, so a brief is not one shell command: it is a build step, one to four agent reads, and a render step.

4. Results against baselines

Pre-registered in docs/dd/sx1-prereg.md, audited, sealed and committed before any test read. Run once. 24 topics no earlier step had used, 121 hard planted errors, 423 unaltered claims. Detail: docs/dd/sx1-results.md.

• Fan-out against one pass: −1.7 points (interval −10 to +5), p 0.77. No win. One pass ships.
• Retrieval against none: the gap between flagging altered and unaltered claims is 0.42 larger with retrieval (interval 0.33 to 0.51), p under 0.001.
• By type, one pass: a flipped direction 58 of 66; a number swapped for another in the same abstract 35 of 37; animal and human swapped 17 of 18. The no-retrieval read was best where plausibility helps (17 of 18 swaps) and worst on numbers (14 of 37).
• The checker added nothing measurable. It removed one true catch and no false alarm.
• Both false alarms were real. The auditor read them: in each, the Lens extraction disagreed with its own abstract. The reader found two wrong extractions without being asked.

The other direction, graded in session. The test above measures catching a misstatement. Most of a real brief is the opposite: records that say a claim is supported. So 20 of those were drawn at random from the five real briefs and re-read by two blind agents, with a negative control built by code from each one's own paper (a direction word flipped, a number changed, or the claim paired with another paper). Both graders had to agree.

All 80 grader runs were unanimous. The controls are what make the first row mean anything: a grader that accepts everything scores 20 of 20 on real records and 0 of 20 here. What it does not show: the controls are plain contradictions, so a subtly wrong record (the same result in a different population) is not represented; 20 items give a lower bound of 83%, not a guarantee; and the graders are the same model family as the reader, so this is agreement, not independence. Method and limits: docs/methodology/scibrief-grading.md, rerunnable with scripts/dd_sci_grade.py.

On real data (five briefs, 99 claims, as of 2025-12-31): 96 claims verified against their own paper with quotes, 0 contradictions, 0 independent supports, 1 same-group support. Subjects: KRAS inhibitor, in vivo CAR-T, antibody-drug conjugate, one lab target, one company target. Outputs are in data/processed/scibrief/ and are not committed: they pair named PIs with name-matched companies.

5. How a skeptical investor should attack this

1. "You planted the errors and hid the answer in the pack." Correct. 91% is an upper bound. A real misstatement is messier, and the paper that contradicts it may not be in the corpus.
2. "The claim and the paper are the same text." Each claim is a model's summary of the paper it is checked against. The test shows the reader can find a source among 24 to 60 abstracts and notice one changed detail. It does not show it can judge a company's own wording.
3. "So your agent is one prompt." Yes. The test says a second architecture adds nothing at this size. The value is the retrieval, the contract and the code around the read.
4. "It found nothing on real topics." Correct, on five. The corpus is three recent abstracts per lab; two labs rarely report the same result in it. The rule for "supported" is strict: same intervention, system, direction and numbers.
5. "The ranking is the Lens field sorted." Nearly. With no independent support and no contradictions, the order is evidence tier, then ties. It is a sort rule and the brief says it predicts nothing.
6. "Company links are name matches." Correct. The company brief for one target drew claims from six labs joined by officer-name matches. It prints "the company link is weak" at the top.
7. "AI graded AI." The reader and the extractor are Claude models, and so are the two graders of the supported records. The planted-error scoring is code against a key, with no model in the loop. The grading check runs negative controls so that at least a grader which accepts everything is detectable. No person has checked any of it, and the briefs say so.

6. What failed

• Fan-out. Not better than one pass, and it raised more false alarms (7 against 2), because a chunk reader with no paper for a claim falls back on doubt.
• The checker stage. No measurable gain on this test. It stays in the pipeline for supported records, which the test does not measure, and the brief says so.
• The cross-lab layer on real data. Zero independent supports and zero contradictions in five briefs.
• My first mutator. An independent reviewer read 18 planted claims and found 6 could be spotted from the claim line alone, because the line still stated the original direction elsewhere. Fixed before sealing; it cost a third of the direction errors and all negation errors.
• My first pre-registration. The auditor returned FAIL: the numbers were stale against the survey file, primary 1 had no false-alarm guard, and an asymmetry that favours fan-out was undisclosed. All fixed, re-audited and cleared before the run.
• My first note rule refused any note that named PD-1 or MRTX1133, because names carry digits. Fixed on development packs.
• New agent files do not register mid-session. The plan named three new agents. They run instead as versioned prompt files through the existing generic agent, with the prompt hash on every record.
• The first test launcher stopped early on the 20-agent limit. No read was lost.
• Three of the reader's 11 "misses" were real flags the strict rule does not credit: a duplicate abstract under another id, a contradiction from a different paper, and one correct record refused on its quote.
• The lab target had only 3 claims, because the corpus holds 3 recent papers per lab at each year-end.
• Six tests fail in this environment, none in files this work touched: the five the handoff names, plus test_idea001_m8_probe, which tries to write to a read-only path. Final run: 1,993 passed, 6 failed, 5 skipped.

7. Checks by independent agents

• Milestone reviewers: M1 to M3 (pass with notes; defects fixed), M4 (pass with notes; mutator fixed), M5 (pass with notes), M6 (pass).
• **backtest-auditor:** the pre-registration before sealing (FAIL, then cleared), and the results (pass with notes). It recomputed every count and both primaries with its own script, found no mismatch, and read all 191 reader transcripts.
• Acceptance: see section 9.

8. What is left, and what needs you

• No human check is planned. You said the point is that this is automated, so the grading runs in session and every number is labelled model-graded. The limit stands and is stated wherever a number appears: no person has read any of it, and two instances of one model agreeing is not independence. If you ever want one outside read, the cheapest version is scripts/dd_sci_grade.py build and handing someone the item files.
• The real gap is the corpus, not the reader. To find a paper that contradicts a target, the pack needs more than three abstracts per lab. Two routes, both need your approval: a PubMed fetch on the target's terms (a fetch), or full-text extraction for shortlisted labs (API spend).
• A real target's own words. The acquisition case is a company's claims against the literature. That needs a claims sheet built from a 10-K or a deck. The reader contract already takes any sheet.
• Drop or keep the checker. It costs one read per brief and showed no gain. I kept it; your call.
• Larger packs. One pass was tested at 24 to 60 abstracts. Above 32,000 tokens the pipeline falls back to fan-out, which is measured as not worse, not as good.
• Decision-log lines (I did not commit docs/decisions.md, which holds another session's edits):
  • 2026-11-10: science evaluation uses one reader contract (prompts/dd/scieval-v1.md); dd-science is retired from this pipeline, its file kept because R2 seals its hash.
  • 2026-11-10: SX1 run once; one pass with retrieval is the default reader (R1); fan-out only above 32,000 tokens.
  • 2026-11-10: a correction round may only rewrite a refused finding; dd-verifier2 adds an absence rule; R2 and M13 drivers stay on the sealed dd-verifier.
  • 2026-11-10: a science brief's rank is scirank-v1, an unfitted sort; it is never called a score.
  • 2026-11-10: supported records are graded in session by two blind agents against code-built negative controls (scripts/dd_sci_grade.py); no owner grading, and every such number is labelled model-graded.
• Not pushed. All commits are local.

9. Acceptance

A fresh agent, given only .claude/commands/sci-brief.md, ran the acceptance test fixed in the plan and returned ACCEPTANCE: PASS.

• It built and evaluated two briefs itself: topic "targeted protein degradation" as of 2024-12-31 (45 papers, 24 claims) and lab 1489 as of 2025-12-31 with topic "single-cell sequencing" (60 papers, 4 claims). No void.
• It checked 14 records: every claim quote was on its claim's line and every paper quote inside the named paper.
• No paper was dated after its as-of. The same topic built at 2022-12-31 held 32 papers, the latest dated 2022-12-28.
• A malformed date and a missing date were refused and built nothing.
• Both briefs print the measured rates with intervals, the extraction accuracy line and "unvalidated by any person".
• Its findings: the sixth test failure above; the command's paths are relative while it says never cd; the doc did not say to wait for agents before collect. The last two are fixed in the command file.