What is checked by code, what is graded by an AI, what a person verified
One row per measured claim. The third column is what the result was compared with. The fourth says who did the grading.
Read this first. No person has checked the AI-graded rows. Two copies of one model agreeing with each other is not independent verification: they can share the same blind spots. Rows tagged code were computed by a program against a key or an equality test. Rows tagged AI-graded rest on a model's judgement. Rows tagged person were verified by a human.
| Claim | Result | Compared with | Who graded | Method |
|---|---|---|---|---|
| Extraction: output_type right | 96.9% of 200 items; interval 92.2% to 98.8% | a bar of 85.0% set before grading: met | AI-graded | lens-accuracy |
| Extraction: model_system right | 94.9% of 200 items; interval 89.6% to 97.6% | a bar of 85.0% set before grading: met | AI-graded | lens-accuracy |
| Extraction: scale right | 97.4% of 200 items; interval 92.9% to 99.1% | no bar was set | AI-graded | lens-accuracy |
| Extraction: claim supported | 97.9% of 200 items; interval 93.6% to 99.3% | a bar of 95.0% set before grading: met | AI-graded | lens-accuracy |
| Extraction: effect numbers supported | 98.6% of 131 items; interval 92.1% to 99.8% | no bar was set | AI-graded | lens-accuracy |
| Extraction: limitations supported | 93.1% of 140 items; interval 85.4% to 96.8% | no bar was set | AI-graded | lens-accuracy |
| Lens search: share of the top 10 labs that are relevant (precision at 10) | 85.8% of the top 10 relevant; difference +27.5 points, interval +19.1 points to +35.8 points; 19 queries: 15 better, 4 tied, 0 worse | keyword search over title and abstract: 58.3%; over title only: 59.5% | AI-graded | lens-search-eval |
| Lens search finds more relevant labs, not fewer results | 7.95 relevant labs per query; difference +2.58, interval +1.74 to +3.42; 14 better, 5 tied, 0 worse | keyword search over title and abstract: 5.50 per query | AI-graded | lens-search-eval |
| The search in the browser returns exactly what the command-line search returns | identical on 360 of 360 searches (30 queries at 6 dates), 0 mismatches; word-stemming differences: 0 in 329,906 words. Checked 2026-11-10. | the command-line search (lens.search_labs); an equality test, so no interval | code | docs/methodology/lens-browser-parity.json |
| Science reader finds a planted misstatement (a changed direction or number, or animal for human) and names the paper that shows it. Scored by code against a key; the reader itself is an AI. | 90.9% (110 of 121) found and the paper that shows it named; interval 85.0% to 95.9% | a read with no literature: 52.1% flagged, 0.0% with a paper named; a string-matching rule: 2.5% | code | sx1-results |
| Science reader's false alarms on correct claims | 0.5% (2 of 423) unaltered claims wrongly flagged; interval 0.0% to 1.2% | no literature: 1.7%; string-matching rule: 3.3% | code | sx1-results |
| Reading in many passes instead of one | fan-out minus one pass: -1.7 points, interval -10.0 points to +5.4 points: no gain from splitting the reading across many passes | one pass over the retrieved abstracts: 90.9% | code | sx1-results |
| Records the reader called supported hold up against their own paper | 100.0% (20 of 20); interval 83.2% to 100.0% | the wrong records beside it: a grader that accepts everything passes here, so read both rows | AI-graded | scibrief-grading |
| Deliberately wrong records (altered direction or number, or the wrong paper) are rejected | 100.0% (20 of 20); interval 83.2% to 100.0% | a grader that accepts everything would pass every real record and reject none of these | AI-graded | scibrief-grading |
| Audit of top-ranked labs: founders the SEC-based labels miss | 70% of founders among top-ranked labs are not in SEC filings: 6 found in filings, 14 more by web search, over 70 labs; no interval, small count | labels built from SEC filings alone | AI-graded | docs/audits/founder-audit-2021-result.md |
| Audit of a stratified sample: how complete the SEC-based founder labels are | SEC labels capture an estimated 41% of founders (interval 26% to 68%), from a sample of 400 labs; the estimate is biased upward because search depth was limited | complete labels would capture every founder | AI-graded | docs/audits/founder-audit-2021-stratified-result.md |
| Opportunity memos where a person verified every claim | 0 memos pass human verification: none published | the rule: no memo publishes unverified | person | v_publishable_memos |
Notes
- Extraction accuracy: 200 extractions, 25 per output type, graded by 2 AI readers; an item counts as correct only if both agree. The readers and the extractor are all Claude models.
- Search: 20 queries, 347 results judged relevant or not by an AI judge, as of 2025-12-31. No person checked a judgement. Queries that returned nothing in an arm count as zero found.
- Science reader: planted errors are an upper bound on real ones, because one word or number was changed and the true paper was in the pack. No person has checked a real case. Supported records, the framed read and the ranking are measured separately, below.
- On real data the science reader's cross-lab layer found no contradictions and no independent supports. A brief's claims are checked against their own paper, so a verified claim means the quoted text is in that paper; it does not mean another lab confirmed it. Detail: science evaluation report.
- A percentage is shown with the counts behind it where the denominator is small. An interval is a 95% interval: the range the true rate plausibly lies in.