Science Lens: extraction accuracy on a fresh sample
Run 2026-11-09. Plan: docs/plans/2026-11-09-extraction-use-plan.md, milestone M3. Machine-readable copy: lens-accuracy.json (the Lens prints its summary). Verdicts: evals/lens/verdicts/.
Result
All three thresholds are met. The stop rule did not fire. No field is marked low confidence.
| Check | Items | Share correct, weighted | 95% interval | Reader A | Reader B | Readers agree | Threshold |
|---|---|---|---|---|---|---|---|
| output_type right | 200 | 96.9% | 92.2% to 98.8% | 97.0% | 96.5% | 98.5% | ≥ 85% met |
| model_system right | 200 | 94.9% | 89.6% to 97.6% | 97.0% | 96.5% | 97.5% | ≥ 85% met |
| scale right | 200 | 97.4% | 92.9% to 99.1% | 98.0% | 98.0% | 99.0% | none set |
| claim supported | 200 | 97.9% | 93.6% to 99.3% | 98.5% | 98.5% | 99.0% | ≥ 95% met |
| effect numbers supported | 131 | 98.6% | 92.1% to 99.8% | 100.0% | 100.0% | 99.2% | none set |
| limitations supported | 140 | 93.1% | 85.4% to 96.8% | 95.7% | 95.0% | 99.3% | none set |
- "Correct" is strict: both readers must mark the item correct.
- "Weighted" means weighted to the mix of paper types in the 202,638 extractions.
- Items is fewer than 200 where the field was null for some papers.
Correct by paper type (both readers, of items checked)
| output_type | output_type right | model_system right | scale right | claim supported | effect numbers supported | limitations supported |
|---|---|---|---|---|---|---|
| therapeutic_candidate | 24/25 | 25/25 | 25/25 | 25/25 | 19/19 | 20/20 |
| diagnostic | 24/25 | 24/25 | 24/25 | 24/25 | 21/21 | 21/22 |
| device | 24/25 | 23/25 | 25/25 | 25/25 | 20/20 | 18/19 |
| platform_method | 23/25 | 22/25 | 23/25 | 24/25 | 16/16 | 14/16 |
| basic_mechanism | 25/25 | 23/25 | 25/25 | 25/25 | 16/17 | 14/15 |
| observational | 24/25 | 24/25 | 24/25 | 24/25 | 22/22 | 16/18 |
| review | 25/25 | 25/25 | 25/25 | 25/25 | 4/4 | 14/14 |
| other | 23/25 | 25/25 | 24/25 | 23/25 | 12/12 | 16/16 |
How it was done
- Sample: 200 extractions, 25 per
output_type, drawn withrandom.Random(11)from the sorted ids. The 200 pilot papers were excluded, so no reader had seen these items. - Readers: two agent readers (Claude Sonnet) per item, in separate sessions. Each saw only the title, the abstract, the extraction and the category definitions from the extraction prompt.
- Instructions:
evals/lens/m3_reader_instructions.md, committed before any reader ran. - Scoring:
evals/lens/score.py, committed before any verdict file was opened.
Full-corpus number check (a description, not a test)
A script compared every number in effect_size_text with the paper's title and abstract, over all 202,638 extractions. 108,727 extractions quote at least one number. Of 637,539 numbers quoted, 99.3% appear verbatim in the source text. 3.0% of those extractions hold at least one number that does not. In the cases I read, most were counts derived from a stated percentage, or a reformatted decimal. I looked at this number while scoping the plan, before writing thresholds.
Weaknesses
- No human read these. Both readers and the extractor are Claude models. They can share blind spots. The two readers' notes are often near-identical, so their agreement is weak evidence of independence. A human spot check is the missing step;
data/interim/lens/m3/items.jsonholds the 200 items. - The standard is "a reasonable reading". Readers were told to accept a defensible category. A stricter standard would score lower.
- Abstract only. A correct extraction of an abstract can still misdescribe the paper.
- **
limitationsis the weakest field** (93.1%). The common failure is invented or placeholder text: about 6.5% of non-nulllimitationsvalues across the corpus contain "not stated" wording. The brief view hides the pure placeholders. - Small per-type counts. 25 items per type cannot separate, say, 92% from 100% for one type.
- **
humanandclinicalwere treated as the same answer,** because the prompt does not define the difference. - **A reader's
nacounts against the item.** In three cases one reader wrotenawhere the other gave a verdict. The strict rule scores those as not correct. This makes the numbers slightly conservative. - Independent review. A reviewing agent recomputed every number from the verdict files with its own script (no mismatch), redrew the sample from the seed (identical), and agreed with 8 of 8 spot-checked items.