Skip to content

Add a scheduled check for dead external links in the docs and explainers #1127

Open

Description

@yakew7

Where: a new scripts/check_external_links.py, a scheduled workflow under .github/workflows/, Makefile.

Current gap: scripts/check_broken_links.py checks only internal links. The 108 explainers carry about 200 external references (papers, court cases, regulators, news), and four already return 404 (see the open issue). Nothing notices when a source moves. A check on every PR would be slow and flaky (many publishers return 403 to scripts), so it should not block merges.

Suggested approach: a script that collects external URLs from the markdown files, requests each with a browser-like user agent and a timeout, treats 2xx and 3xx as fine, 403 and 429 as unknown (reported separately, never failing), and 404 or a DNS failure as broken. Run it weekly from a scheduled workflow that opens or updates one issue listing the broken links instead of failing a build. Add a --limit option and a cache so a run stays under a few minutes.

Done when: the script has tests with a local test server for each status class, the weekly workflow runs with a timeout and a concurrency group, and it reports (never fails the repository's checks).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesthelp wantedExtra attention is needed

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions