add statpearls scraper

[nahatav]

Licensing — read this first

StatPearls chapters are not public domain, unlike the MedlinePlus source in #23. Confirmed straight from NCBI's own copyright dialog on the book page (https://www.ncbi.nlm.nih.gov/books/NBK430685/):

Copyright © 2026, StatPearls Publishing LLC. This book is distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).

NonCommercial and NoDerivatives. The ND term is the one that matters here: claim decomposition and training-data construction are both derivative uses of the source text, which is what this project intends to do with scraped sources. I'm not the right person to make that call, so I haven't tried to. Every document carries metadata["license"] and metadata["license_url"] so it's not silent in the corpus, and the module docstring spells the conflict out. Whoever owns the decomposition/training pipeline should decide whether StatPearls-sourced documents get included, held out, or something in between, before this feeds anything downstream. Happy to gate it behind a flag or exclude it from default runs if that's the safer default — just say so.

Given IDSA is already flagged in the project doc as not redistributable, this project clearly already tracks this distinction; StatPearls belongs in that same bucket.

Summary

Adds a StatPearls source scraper. StatPearls has no public listing of chapter accessions, so chapters are discovered through NCBI's E-utilities search API rather than a crawlable index page: searching the books database for statpearls[book] returns one hit per chapter section (Introduction, Treatment, Review Questions, etc.), so discovery deduplicates by each section's parent chapter accession to build the chapter list. Chapter content itself is scraped from its NCBI Bookshelf HTML page. StatPearls is listed as an encyclopedic-fallback source in the AMFV project doc, alongside MedlinePlus.

Compliance

• /books/NBK* is explicitly Allowed in NCBI's robots.txt with a 5 second crawl delay, which is applied between chapter fetches.
• E-utilities usage policy (documented at NBK25497) asks for at most 3 requests/second without an API key and for callers to identify themselves via the tool parameter — sent as EUTILS_TOOL on every request. Discovery issues two sequential requests per listing page, well inside that limit.
• eutils.ncbi.nlm.nih.gov/robots.txt itself sets a blanket Disallow: / for all agents, which reads as a generic crawler-exclusion default rather than a statement about the documented, versioned API this scraper actually calls (the same host that publishes the usage policy above). I want to flag that reasoning explicitly rather than let it pass silently — a maintainer might read that robots.txt differently than I did.

Handling a long crawl

A full run is on the order of 10,000+ chapters at a mandatory 5s delay, roughly 15 hours. Two things followed from that scale once I thought it through:

• scrape_chapter now returns None (logged) on a fetch failure or missing content instead of raising, so one bad chapter costs a document instead of the rest of the crawl. Verified against a real nonexistent NCBI accession, both that this path logs and skips, and that scrape_chapter_by_url (the --url single-document path) still raises clearly, since there's no other document to fall back on there.
• search_section_uids and summarize_chapters (the E-utilities discovery calls) raise StatpearlsFetchError rather than a raw httpx error, so callers catching this module's own error type don't miss the most likely failure.

Components affected

• datasets/amfv_datasets/scraping/statpearls.py (new)
• datasets/amfv_datasets/scraping/cli.py — one import, one SCRAPERS entry
• datasets/test/test_scraping_statpearls.py (new)
• datasets/test/test_scraping_cli.py — the unknown-source test asserts the registered source list

Testing

• uv run ruff check / uv run ruff format --check clean
• uv run pytest — 47 passing, run on Python 3.13 to match CI
• Live scrapes verified: normal chapter scraping with correct section counts (cross-checked against independently measured values), --url mode, and both error-handling paths against a real nonexistent chapter (one skips and logs, the other raises)
• Not run: the full ~10,000+ chapter corpus end to end