Data sources (all public)

Never use: employer-internal data, paid databases accessed through an employer, non-public information.

UKRLP and OfS Register: terms and robots.txt (quoted 2026-10-26, before the first lookup)

• UKRLP. https://www.ukrlp.co.uk/robots.txt 301-redirects to the home page of https://ukrlp.education.gov.uk/; https://ukrlp.education.gov.uk/robots.txt returns HTTP 404. No robots.txt exists, so no path is disallowed. The public search needs no key or login ("Search provider name or UKPRN"; sign-in is for providers' own records). Terms (https://ukrlp.education.gov.uk/Help/TermsAndConditions), quoted: "By using the UK Register of Learning Providers service, you agree to these terms and conditions." Users must "agree not to use the service for any illegal purpose"; personal data must be "used only for the purposes for which it was obtained". Footer: "All content is available under the Open Government Licence v3.0, except where otherwise stated". We read provider-level records (name, UKPRN, status, dates) only, no personal data, at one request at a time with a pause.
• OfS Register. https://www.officeforstudents.org.uk/robots.txt reads "User-agent: *" / "Disallow: /umbraco/" (the register page /for-providers/registering-with-the-ofs/the-ofs-register/ is not disallowed). https://register.officeforstudents.org.uk/robots.txt and https://register-api.officeforstudents.org.uk/robots.txt return HTTP 404 (no robots.txt). Copyright page (https://www.officeforstudents.org.uk/copyright/), quoted: "All content owned by the OfS is available for re-use under the terms of the Open Government Licence (OGL)." and "If you re-use any of the content owned by the OfS you must acknowledge the source." The register page loads its data from https://register-api.officeforstudents.org.uk/api/Provider (no key).
• If either needs a key or login later, or a page becomes disallowed, the lookup stops and is reported; no workaround.

Internet Archive snapshots and HEFCE publications (row added 2026-10-26, addendum-4 conformance m5)

Companies House: terms and robots.txt (quoted 2026-10-26, before the first CH data request; IDEA-009 M3)

Fetched by enrich/registry.py (Fetcher, User-Agent "diaspore-research/0.3 (public-data research project, IDEA-009; no contact address)", <= 1 request/s), each with a sources row in data/diaspore_registry.db (rows 1-17; status per request in data/raw/ch_robots/fetch_log.jsonl, bodies saved there). Only these robots.txt and terms pages were requested before this section was written.

• **Bulk-download host download.companieshouse.gov.uk.** robots.txt (HTTP 200, sha256 8b1e4278…, row 1) reads "User-agent: *" / "Disallow:" (empty: nothing disallowed). The product page en_output.html (row 3), quoted: "The Free Company Data Product is a downloadable data snapshot containing basic company data of live companies on the register." "This snapshot is provided free of charge and will not be supported." "The contents of the snapshot have been compiled up to the end of the previous month." It lists BasicCompanyDataAsOneFile-2026-10-01.zip (471Mb) and 7 part files. It states no record count for the file.
• **Public-page host find-and-update.company-information.service.gov.uk.** robots.txt returns HTTP 404 (row 2, null hash): no robots.txt, nothing disallowed. Footer: "© Crown copyright"; its "Policies" link goes to the GOV.UK guidance below.
• **Terms (GOV.UK guidance "Searching the Companies House register", www.gov.uk/guidance/searching-the-companies-house-register, row 9; www.gov.uk/robots.txt row 5 disallows only /*/print$ and /search/all* for "").* Quoted: "Our Find and update company information service makes all of the public data we hold on companies available free of charge. You do not need to register to view company information through this service." Included: "images of documents", "current and resigned officers", "full transaction filing". Acceptable use: "they are not intended to be a source for bulk downloads." and "Individuals, companies, IP addresses or blocks of IP addresses who deny or degrade service to other users by generating unusually high numbers of daily database accesses, whether generated manually or in an automated fashion, may be denied access to these services without notice." Item 7: the service "may also be used to search and take copies of material on the register", including "the filing history of registered material and associated document images - images are free from 2003 onwards". GOV.UK footer: "All content is available under the Open Government Licence v3.0, except where otherwise stated".
• Reading for this project. Scripted access is not prohibited by either robots.txt or the terms; the binding limit is volume ("unusually high numbers of daily database accesses"). M3 makes a few dozen page requests at <= 1/s; M8 (30-60 CRNs, a few pages each) stays in the hundreds. M11's per-CRN load of every in-scope CRN is the bulk-style use the terms warn against, so M11 uses the CH API (OWNER-QUEUE key row) or is re-scoped; it does not page-scrape at scale. The bulk file is the product meant for bulk use. Personal data (officers, PSCs) is used under the G14/H5 limits (no birth month/year; committed files carry design ranks only).
• **resources.companieshouse.gov.uk** (the bulk page's field-list PDF and FAQ): robots.txt returned HTTP 403 (row 7, null hash), so the host is refused by the fetcher and nothing else was requested from it.
• Register-size statistic (for the M3 row-count check; GOV.UK "Incorporated companies in the UK April to June 2026", rows 15 and 17, OGL v3.0): "there were 5,516,377 companies on the register as of end June 2026"; companies only (LLPs are in the tables alongside, not in that figure); "Next release: 29 October 2026."
• Document host(s). A filing-history document link redirects off the find-and-update host. No redirect target is fetched until its robots.txt and terms are quoted below and the host is added to registry.QUOTED_HOSTS.
• Document host, checked 2026-10-26 (M3 smoke). A "View PDF" link (/company/<CRN>/filing-history/<id>/document?format=pdf) returns HTTP 302 to a 60-second signed URL on s3.eu-west-2.amazonaws.com (bucket document-api-images-live.ch.gov.uk). https://s3.eu-west-2.amazonaws.com/robots.txt returned HTTP 403 (row 35, null hash). The fetcher treats any robots.txt status other than 200 or 404 as "host refused", so the host is not added to QUOTED_HOSTS and no PDF was fetched (the 302 itself has a row, null hash). Ruling queued in OWNER-QUEUE (RFC 9309 reads 4xx as "no robots.txt"; Python's robotparser reads 401/403 as disallow-all).
• Bulk data products: terms (G20; quoted 2026-10-26, before M6 uses the index; M3 review ruling note 4). Source: GOV.UK guidance "Companies House data products", www.gov.uk/guidance/companies-house-data-products (row 37, sha256 3d1abf01…; www.gov.uk/robots.txt row 36, unchanged 459835d2…; page "Last updated: 25 October 2022"). It names the bulk file: "The company data product is a free monthly snapshot of information for live companies on the public register." Its "Using Companies House data" section, quoted: "Companies House is required to make data about companies and their officials available for public inspection under the Companies Act 2006. We impose no rules or requirements on how the information on the public register is used, and we're not responsible for your use of the company data. You are responsible for complying with any applicable data protection, copyright and other legislation and regulations. You should check that you've met all the necessary legal requirements before you publish any information." It points to "Companies House public task and Crown copyright" (not fetched). No named licence (OGL or other) is stated for the bulk file itself; the page text above is the terms. Reading: reuse of the bulk file is unrestricted by CH; data-protection law still applies to officer data, which the bulk file does not carry. No licence gap, so no OWNER-QUEUE row. Only derived counts are published, as before. URLs tried for this quote: the one above (200). resources.companieshouse.gov.uk/infoAndGuide/faq/publicDataProduct.shtml (the bulk page's FAQ link) was not requested: that host stays refused (robots.txt 403, row 7).
• Disclosures (M3 review rulings, decisions.md 2026-10-26). (a) The M3 row-count substitute (Companies Act categories -0.55% vs CH's end-June figure) was chosen after both numbers were seen. It is post hoc and does not count as a pass. The replacement criterion: (i) completeness, registry.count_bulk_records (a separate counting function) = 5,704,712 CSV data records = 5,704,711 index rows + 1 skipped blank line, exactly (met; run 2026-10-26, 49 s); (ii) same-date plausibility, the fixed 9-category count registry.COMPANIES_ACT_CATEGORIES (n_ca = 5,486,025 in index c1ea9788…; all categories 5,704,711 reported beside it) vs CH's end-September 2026 register figure, not yet published (due 2026-10-29). The list's own origin is post hoc, too (the builder's set, kept so that no fresh choice is made). Edge categories outside it: "Other company type" 15,518 + "Other Company Type" 3, "Royal Charter Company" 909, and the non-company forms (LLPs 50,171, LPs 61,086, CIOs 40,659, overseas entities 30,207, others). (b) H5 fixtures carry "PLACEHOLDER-MONTH 1900" in DOB fields. They are synthetic and name no real person. M8's parser test will assert that no dob value reaches a committed output. (c) The H6 registry guard misses transitive imports and paths built at runtime. That is deferred to M11 (import-graph check plus a runtime open-hook fixture). (d) "Guard green before the first interim write" at M3 rests on the builder's word. From now on, pytest command, commit, time and summary line are logged before the act. (e) Non-data requests made before the quotes commit 608c9d4, which carried no company data and were made to locate the terms and the comparison figure: row 4 (find-and-update home page), rows 11 and 13 (GOV.UK search API), rows 15 and 17 (the GOV.UK statistics release). From now on, only robots.txt and terms pages are requested before a quote. (f) M3 smoke page count: 11 company pages (rows 22-32, each 200 with a hash matching its saved file: 4 for one smoke CRN, 7 for the other, 5 of them filing-history pages), not the 13 the M3 Log line said. (g) Stale saved paths of robots rows 1, 5, 8, 10, 12, 14, 16 in data/raw/ch_robots/fetch_log.jsonl are corrected by appended {correction_for_source_id, saved, at} records; no line was rewritten.
• Document host: the director's scoped exception (quoted 2026-10-26, before the first PDF fetch). RFC 9309 (Robots Exclusion Protocol), www.rfc-editor.org/rfc/rfc9309.txt (sources row 39, sha256 f633915d…; www.rfc-editor.org/robots.txt row 38), section 2.3.1.3 "Unavailable" Status, quoted in full: ""Unavailable" means the crawler tries to fetch the robots.txt file and the server responds with status codes indicating that the resource in question is unavailable. For example, in the context of HTTP, such status codes are in the 400-499 range. If a server status code indicates that the robots.txt file is unavailable to the crawler, then the crawler MAY access any resources on the server." The exception (director, decisions.md 2026-10-26 "IDEA-009 M3 review rulings", PDF host / M8, answer (a), narrowly): a 302 Location target of a find-and-update /company/<CRN>/filing-history/<id>/document?format=pdf link, on s3.eu-west-2.amazonaws.com, with its path in bucket document-api-images-live.ch.gov.uk, may be fetched at <= 1 request/s, at M3-smoke and M8 volumes only (smoke CRNs; design ranks <= 60). Grounds as ruled: section 2.3.1.3 reads a 4xx robots.txt as unavailable; the 403 comes from the AWS S3 service root, not from CH; CH's terms (item 7 above) allow taking copies of "associated document images". In code (registry.doc_exception_applies, Fetcher.get(doc_exception_from=...)): only that origin (302 status), host, https scheme and bucket path; the host's robots.txt is still requested first (own sources row) and only a 4xx is read as unavailable; >= 1 s between exception requests whatever the session pause; at most 300 exception fetches per session; the host is not added to QUOTED_HOSTS, so nothing else on it is fetched; the signed query (temporary credentials, 60-second expiry) is stripped from sources rows and fetch logs. resources.companieshouse.gov.uk stays refused (registry.REFUSED_HOSTS, any purpose). The owner may reverse this; M8 then runs on the refused default (every PDF unreadable, OPS a lower bound, band on OP).
• PDF smoke, measured 2026-10-26 (first fetches under the exception). 2 M3 smoke CRNs (SIC 47110, never in scope): entry 1 incorporated 2021, entry 2 incorporated 2009. Documents: each entry's IN01 (CH filing type "NEWINC"), first return (G6: AR01 or CS01, whichever first) and, for entry 2, the first CS01 too (entry 1's first return is its first CS01). 5 PDFs, all HTTP 200 via the 302 -> bucket path (sources rows 41-51; robots.txt rows 40 find-and-update 404, 42 S3 403). pdftotext -layout (installed, no OCR): entry 1 IN01 and CS01 have text layers (CH electronic filings); entry 2's three PDFs are image-only scans (tiff2pdf), text output is form feeds only. Readable share, ruled denominator (IN01 + first return, n = 4): 2/4 = 50%. Unreadable 2/4 = 50%: not retrieved 0, retrieved with no text 2. Absent (readable, no list) 0; present 2 (IN01 initial shareholdings; CS01 full shareholder list). The literal set (IN01 + first CS01, n = 4) gives the same split. n = 2 companies of one SIC code: this says only that pre-electronic-filing documents are image-only. M8's design companies (incorporated 2012-2020 at the earliest) will show the real share. Text fixtures (names, addresses, DOBs, nationalities, company names, CRNs and barcodes synthesised; rule 8, H5) are in tests/fixtures/idea009_ch/smoke_*.txt; the per-document record with CRNs is data/interim/idea009/m3_pdf_smoke.json (sha256 902018b8…, gitignored). One older line in a gitignored ch_docs fetch log (the M3 302 of 10:56, before public_url) still holds that redirect's signed query; it expired 60 s after issue and is left as written (append-only).