Science Lens: search against keyword baselines
Run 2026-11-09, once. Plan: docs/plans/2026-11-09-extraction-use-plan.md, milestone M4. Machine-readable copy: lens-search-eval.json. Verdicts and arm results: evals/lens/verdicts/.
Result
The extraction filter beat plain keyword search. On the 19 queries that returned labs, precision at 10 rose from 0.58 to 0.86.
| Arm | Queries with results | Mean precision at 10 | Mean relevant labs in top 10 (all 20 queries) | Mean labs returned |
|---|---|---|---|---|
| Title baseline (titles only) | 17 | 0.60 | 3.05 | 6.10 |
| Abstract baseline (title and abstract) | 19 | 0.58 | 5.50 | 9.45 |
| Lens (abstract match, product-directed papers only) | 19 | 0.86 | 7.95 | 9.25 |
- Primary, Lens minus abstract baseline, precision at 10: +0.28 (95% bootstrap interval +0.19 to +0.36; 19 queries; 15 wins, 4 ties, 0 losses).
- Relevant labs found in the top 10, Lens minus abstract baseline: +2.58 (+1.74 to +3.42). The filter did not win by returning fewer labs.
- Abstracts against titles: precision is the same, but abstracts find 2.24 more relevant labs per query (+1.41 to +3.06). Before this work the database held no abstracts.
- One query found nothing in any arm: "wearable glucose biosensor". The corpus has no such lab papers matching all three words. The query was kept as written.
Per query (relevant / returned)
| Query | Title | Abstract | Lens |
|---|---|---|---|
| CAR-T solid tumors | 7/10 | 8/10 | 8/10 |
| lipid nanoparticle mRNA delivery | 10/10 | 10/10 | 10/10 |
| CRISPR base editing | 6/10 | 5/10 | 7/10 |
| antibody-drug conjugate | 4/10 | 6/10 | 10/10 |
| gut microbiome therapeutic | 0/8 | 2/10 | 8/10 |
| circulating tumor DNA liquid biopsy | 3/3 | 6/10 | 10/10 |
| Alzheimer tau | 1/10 | 2/10 | 7/10 |
| AAV gene therapy retina | 0/0 | 7/9 | 7/9 |
| KRAS inhibitor | 4/10 | 6/10 | 10/10 |
| organoid drug screening | 3/4 | 9/10 | 10/10 |
| antibiotic resistance new antibiotic | 0/0 | 4/10 | 8/10 |
| targeted protein degradation | 6/10 | 8/10 | 10/10 |
| single-cell sequencing method | 5/6 | 6/10 | 10/10 |
| neoantigen cancer vaccine | 4/6 | 6/10 | 10/10 |
| antifibrotic fibrosis | 2/3 | 5/10 | 9/10 |
| GLP-1 obesity | 0/10 | 2/10 | 3/10 |
| sickle cell gene editing | 1/1 | 5/10 | 5/6 |
| bispecific antibody | 4/10 | 6/10 | 10/10 |
| wearable glucose biosensor | 0/0 | 0/0 | 0/0 |
| machine learning protein design | 1/1 | 7/10 | 7/10 |
The weak spot is "GLP-1 obesity" (3 of 10). The judge's notes on the seven misses: four case reports or observational studies of approved drugs, one mechanism paper, two off topic.
How it was done
- Queries: 20, written in the plan before any search was run (
evals/lens/queries.txt). - Data: the live panel, as of 2025-12-31, through the gate.
- Match rule, all arms: every query word must appear (stemmed). Labs are ranked by number of matching papers, then by best text score. Top 10.
- Judging: 347 (query, lab) pairs pooled from all arms, shuffled, judged once by an agent (Claude Sonnet) that did not know the arm. The judge read up to 3 of the lab's matching papers (title and abstract, never the extraction). Relevant = on topic and product-directed. 207 of 347 were relevant. Instructions:
evals/lens/m4_judge_instructions.md. - Scoring:
evals/lens/score.py, committed before any verdict file was opened. Bootstrap over queries, 2,000 draws, seed 0.
How a skeptic should attack this
- The test is partly circular. "Relevant" requires product-directed work, and the Lens filter selects papers the extractor called product-directed. The result shows the extractor's label agrees with a second model's reading of the abstracts. It does not show the labs are good investments.
- Both are Claude models. The extractor is Haiku and the judge is Sonnet. Shared bias would inflate the gain. One judge read each item; there is no second judge and no human.
- Twenty queries, written by the builder. They are common biotech themes. Rare or oddly worded topics will do worse, as the empty query shows. The match rule needs every word, so it misses synonyms.
- The corpus is thin. Each lab has at most its three most recent last-author papers at four year-ends. A lab whose relevant paper is older or middle-authored is missing.
- This measures retrieval, not outcomes. Nothing here says a top-ranked lab forms a company or raises.
- The "relevant labs found" gain mixes two effects. The Lens arm is not a subset of the abstract arm's top 10. It ranks within the filtered set, so it reaches labs the abstract arm ranked lower. The gain is filtering plus depth.
- The title arm is handicapped in the secondary comparison. 23 judgments showed papers with no abstract, so the judge had only a title. All 23 came from the title arm and 5 were judged relevant. This does not touch the primary Lens-against-abstract result.
- The judge is lenient. The reviewing agent disagreed with 1 of 8 spot-checked judgments (a yeast mechanism paper counted as relevant). Leniency raises relevance in every arm.