add medlineplus scraper - #23
Conversation
Discovers topics from the sitemap MedlinePlus advertises in its robots.txt, then scrapes each topic page for its summary and Dublin Core metadata (MeSH headings, alternate titles, dates). Deliberately avoids the daily bulk XML export, which would be a single request but is served from /xml/, a path MedlinePlus robots.txt disallows. Pages that share the topic URL shape but carry no summary container are skipped, so indexes and tools do not enter the corpus. Health topic summaries are public domain, and each document carries the acknowledgement NLM asks redistributors for. Content NLM licenses from third parties is excluded by construction: A.D.A.M. articles and ASHP drug monographs live off the flat topic path this scraper accepts. MedlinePlus is listed as an encyclopedic-fallback source in the AMFV project doc.
|
|
| their visible text (default: LinkMode.KEEP). | ||
| """ | ||
| response = client.get(url) | ||
| response.raise_for_status() |
There was a problem hiding this comment.
raise_for_status() here raises httpx.HTTPStatusError, which nothing catches, so a single 503 or timeout ends a full 1,080-topic run wherever it lands. It is also not MedlineplusFetchError, whose docstring says it is raised “when a MedlinePlus topic cannot be fetched or parsed”, so a caller catching the module’s own error type misses the most likely failure there is.
Measured on a three-topic sitemap with a 503 on the second: run ends with httpx.HTTPStatusError, 1 of 3 documents written.
Two asks. Wrap both this fetch and the sitemap fetch at line 95 so MedlineplusFetchError is what escapes. Then consider logging and returning None here instead of raising: scrape_listing_documents already treats None as “skip this item”, so one unreachable page would cost one document rather than the rest of the crawl. The sitemap fetch should stay fatal.
| list_topic_urls, | ||
| scrape_topic, | ||
| topic_slug_from_url, | ||
| ) |
There was a problem hiding this comment.
The 11 tests here cover list_topic_urls, scrape_topic and topic_slug_from_url well, but none of them imports scrape_medlineplus, which is the only function SCRAPERS can reach. So the arrangement that actually terminates the scrape, first_page_items=topic_urls with list_page returning [], and the www. normalization in the --url branch, are both unexercised.
One test calling scrape_medlineplus against a mock transport serving a two-topic sitemap plus two topic pages would cover both, and asserting the sitemap is requested exactly once would pin the “already-fetched first page” behaviour.
Worth adding a CLI dispatch test too. test_scraping_aafp.py on #22 does this with monkeypatch.setitem(SCRAPERS, ...), which is the form that works now that sources are dictionary entries.
| with default_client() as client: | ||
| topic_urls = list_topic_urls(client) | ||
|
|
||
| return ScrapeRun( |
There was a problem hiding this comment.
Optional. total=documents is None on a full run, so the progress bar has no denominator for about eighteen minutes, even though list_topic_urls returned the complete list one line above.
total=documents if documents is not None else len(topic_urls),
It would be an upper bound, since pages with no summary are skipped, but it beats no total. Fine to leave as is if you would rather not show a count that overshoots.
| (`/druginfo/`) cannot be redistributed without licensing from those vendors, | ||
| and neither matches the flat topic path this scraper accepts. See | ||
| https://medlineplus.gov/about/using/usingcontent/. | ||
| """ |
There was a problem hiding this comment.
Optional, and not a defect. Eight topics scraped live come out at 751 to 6,135 characters, median 1,634. The NICE scraper already on main has a median of 23,817 and its clinical guidelines reach 97,167, so this source is roughly one fifteenth the size and written for patients rather than clinicians.
Worth one sentence in the module docstring saying what a full run yields and how large a document is, next to the licensing paragraph. It saves the next person measuring it, and it is the figure whoever tunes retrieval will want. My own WikiDoc branch has the same gap in the other direction.
Per @zndr27's review on MedARC-AI#23: - list_topic_urls and scrape_topic now raise/return MedlinePlusFetchError- wrapped failures instead of a raw httpx error escaping. The sitemap fetch stays fatal (discovery can't proceed without it); a single topic page fetch failure is logged and skipped, since scrape_listing_documents already treats None as "skip this item". A 503 partway through a run now costs one document instead of the rest of the crawl. - total is now an upper bound from the discovered topic count instead of always None, so the progress bar has a denominator on a full run. - Documented typical document size next to the licensing paragraph. - Added the missing coverage: scrape_medlineplus end to end (including that the sitemap is fetched exactly once), the --url branch's www. host normalization, and a CLI dispatch test for the medlineplus source.
|
@zndr27 Pushed a fix for all four. Wrapped fetch failures into MedlineplusFetchError with the sitemap staying fatal, ran your exact 503 mid-crawl scenario and now get 2 of 3 instead of 1 of 3 and a crash, added the scrape_medlineplus/www normalization/CLI dispatch tests, and the total + docstring notes. Let me know |
3 participants
Summary
Adds a MedlinePlus source scraper to
datasets, following the shared scraping base from #21. MedlinePlus is listed as an encyclopedic-fallback source in the AMFV project doc.Topics are discovered from the sitemap MedlinePlus advertises in its own
robots.txt, then each topic page is scraped for its summary and Dublin Core metadata (MeSH headings, alternate titles, created/modified dates, publisher).On the bulk XML export
MedlinePlus also publishes a daily bulk XML export of every topic, which would be a single request instead of one per topic. I started there, then backed it out: it is served from
/xml/, whichmedlineplus.gov/robots.txtdisallows. The sitemap and topic pages are both allowed.Content is not lost by taking the compliant path — I diffed page-scraped output against the bulk export for the same topics and the markdown is byte-identical (
a1c,abdominalpain,abortion,asthma; delta +0 on all four).Happy to switch back if maintainers read that
Disallowdifferently, since NLM separately documents those files as available for download.Licensing / redistributability
Relevant since the project doc separates redistributable from non-redistributable sources — MedlinePlus sits on the redistributable side, with one condition:
metadata["attribution"] = "Courtesy of MedlinePlus from the National Library of Medicine". That way the credit survives into any corpus built from this./ency/) and ASHP drug monographs (/druginfo/) cannot be redistributed without licensing from those vendors, and neither matches the flat topic path this scraper accepts. There is a test pinning that.Terms: https://medlineplus.gov/about/using/usingcontent/
Components affected
datasets/amfv_datasets/scraping/medlineplus.py(new)datasets/amfv_datasets/scraping/cli.py— one import, oneSCRAPERSentrydatasets/test/test_scraping_medlineplus.py(new)datasets/test/test_scraping_cli.py— the unknown-source test asserts the registered source listNotes on the implementation
topic-summarycontainer NLM markssyndicate. Pages that share the topic URL shape but have no such container (indexes, tools, directories) returnNoneand are skipped, so they never enter the corpus./a1c.html) are scraped;/spanish/,/ency/,/druginfo/, and/lab-tests/are filtered out at discovery, and--urlrejects them with a message naming the expected shape.DOCUMENT_DELAY_SECONDS = 1.0—robots.txtsets noCrawl-delay, so this is a politeness floor rather than a required interval.section_countstays at the default1, which is accurate here: a topic summary is a single section.Testing
uv run ruff check/uv run ruff format --checkcleanuv run pytest— 41 passing, run on Python 3.13 to match CI--format markdown, non-topic skip behavior (healthchecktools.html→ skipped), and sitemap enumeration (1080 candidate URLs)