scrapers for icrc mayo spor

[mkieffer1107]

Summary

This PR adds and hardens permission-gated scrapers for ICRC, Mayo Clinic, and the SPOR Evidence Alliance.

The scrapers support official inventory discovery, clinical relevance filtering, HTML/PDF parsing, source-format metadata, timing instrumentation, stale-link handling, and quality-preserving OCR fallbacks.

What changed

ICRC

• Discovers publications through all 17 official child sitemaps.
• Enumerates 728 unique English publication pages.
• Applies a conservative clinical relevance gate before downloading PDFs or starting OCR.
• Excludes annual reports, legal and institutional publications, recruitment material, posters, economic material, and other nonclinical pages.
• Supports landing-page HTML, direct PDFs, shop-backed PDF resolution, native extraction, Docling/OCR fallback, and landing-page fallback when a PDF is unavailable.
• Reuses the shop browser during manifest runs and recovers cleanly when a reused Playwright context fails.
• Adds coverage for newly published clinical material, including 2025 guidance on managing severe bleeding.

Mayo Clinic

• Discovers condition articles from the official sitemap.
• Enumerates 2,401 supported pages:
  • 1,202 symptoms/causes pages;
  • 1,199 diagnosis/treatment pages.
• Excludes doctors, departments, organizations, procedures, translations, and other non-condition routes.
• Validates canonical redirects so a moved URL cannot silently produce a record for a different article.
• Removes recurring appointment, newsletter, advertising, product, and Mayo Clinic Press chrome.
• Supports both current condition-page layouts and retains publication metadata when available.

SPOR Evidence Alliance

• Discovers all eight current English WordPress child sitemaps for inventory auditing.
• Keeps document ingestion tied to the official clinical-practice-guideline asset-map report rather than crawling arbitrary publisher HTML.
• Reconstructs 460 unique PDF candidates across 132 report pages and 71 publisher hosts.
• Handles dynamic PDF download URLs and deduplicates equivalent HTTP/HTTPS links.
• Records stale candidates and continues to later documents instead of silently dropping failures or ending early.
• Supports native PDF extraction and Docling/OCR fallback.
• Preserves a warning that SPOR’s currently published database covers guidelines through April 2018, was not quality-assessed, and does not establish that a listed guideline remains current.

Format and provenance metadata

Every successful record includes:

• metadata.source_format_types, such as html and pdf;
• metadata.source_media_types, such as text/html and application/pdf;
• provenance.content_type = text/markdown for normalized output;
• retrieval receipts with the original server content type;
• phase-level timing for retrieval, inventory parsing, HTML normalization, shop resolution, PDF conversion, and request pacing;
• support for mean, median, p90, minimum, maximum, and total run timing.

Validation

The expanded live validation produced 204 successful documents:

Source	Successful documents	Coverage
ICRC	99	Direct PDFs, numbered and slug routes, shop-backed downloads, unavailable PDFs, native extraction, Docling, and OCR
Mayo Clinic	69/70	Both article route families across a wide range of clinical specialties; one stale redirect was safely rejected
SPOR	36	12 reachable publisher hosts plus 94 explicitly recorded stale candidates

Across those 204 records:

• 204 unique external IDs;
• 204 unique normalized-content hashes;
• no empty records or incomplete JSONL rows;
• no malformed content hashes;
• no missing source-format or media-type metadata;
• 88 HTML records;
• 38 PDF records;
• 78 combined HTML/PDF records;
• content sizes from 426 to 2,925,647 characters;
• section counts from 1 to 286.

A separate distributed validation sampled beginning, middle, and final corpus positions instead of only taking the first N items:

• 21 ICRC positions at five-percent sitemap intervals, including the first and final entries;
• 21 Mayo positions at five-percent sitemap intervals, including the first and final entries;
• 21 SPOR positions distributed throughout the official asset map, including its final page;
• 44 successful documents and 14 explicitly recorded stale SPOR links;
• no likely-corrupt text, invalid control characters, or Unicode replacement characters.

The full findings are documented in:

datasets/SCRAPER_VALIDATION_ICRC_MAYO_SPOR_2026-08-26.md

Throughput improvements

The request policy, retry behavior, conversion quality gates, and OCR behavior were not relaxed.

Processing time now counts toward the existing minimum request interval instead of always adding a complete delay afterward. ICRC manifest runs also reuse one shop browser.

Source	Before	After	Reduction
ICRC, 4 documents	118.292 s	87.658 s	25.90%
Mayo Clinic, 4 documents	33.632 s	31.968 s	4.95%
SPOR, 1 success after 3 stale links	184.145 s	171.857 s	6.67%

Before/after outputs matched exactly on IDs, titles, URLs, content hashes, byte counts, section counts, formats, media types, conversion backends, and source-PDF hashes. No throughput improvement came from skipping or shortening documents.

Main remaining bottlenecks:

• ICRC: scan-heavy Docling/OCR conversion and initial shop-browser startup;
• Mayo Clinic: browser retrieval and the required courtesy interval, not HTML parsing;
• SPOR: dead publisher-host timeouts and scan-heavy Docling conversion.

Safety and operational behavior

• All three sources remain explicitly permission-gated.
• Request pacing and bounded retries remain enabled.
• Redirect, DNS/public-address, content-size, PDF-size, and conversion-time protections remain intact.
• Signed query values and credentials are not written to the validation report.
• Publisher content and local inspection outputs under outputs/ are ignored by Git and are not included in this PR.
• The changes contain no machine-specific paths or macOS-only behavior.
• Linux deployments must install Playwright Chromium for Mayo pages and ICRC shop resolution.

Tests

• Focused ICRC/Mayo/SPOR/CLI suite: 120 passed
• Full repository suite with PDF dependencies: 216 passed
• uv run ruff check .: passed
• git diff --check: passed
• uv build --offline --package amfv-datasets: passed