Add versioned HTML index sources and systemd documentation

[TheRealBecks]

Systemd's reference manuals live in DocBook XML, so indexing its Markdown guides omits systemctl, journalctl, unit configuration and most APIs. This implements the rendered-HTML approach proposed in #116: a versioned HTML-index source and two systemd registry packages.

• html-index accepts explicit numeric releases and an HTTPS URL with a {version} directory. It fetches the index's HTML links within that origin and directory, with bounded concurrency, timeouts, retries, response/page limits, and redirects checked before following them. Failed downloads fail the build.
• Raw downloads are cached by URL; identical manual aliases are indexed once. The existing nightly publisher skips releases that are already published. The importer uses LinkeDOM, already present in the workspace. The schema, CLI listing and version discovery support explicit HTML releases.
• systemd/systemd@258 supplies the reference manuals; systemd/systemd-guides@latest supplies the guides from Git tag v258. Registry documentation explains the source schema, limits and cache refresh procedure.

Live validation also exposed that Turndown flattens DocBook's bare <pre> elements. A small HTML-parser fix preserves these as fenced code blocks, with regression tests and a patch changeset.

Validation:

• pnpm install --frozen-lockfile, pnpm lint, pnpm build, pnpm test — passed; 317 tests.
• Systemd 258: 1,229 manual URLs deduplicated to 432 documents, 3,234 sections and 1,108,416 tokens. Confirmed key manuals, service examples and SQLite integrity.
• Guides at v258: 531 sections and 220,331 tokens; passed the CI registry-validation command.
• The manuals also passed automatic version discovery and a cached build with network requests disabled.

Versions are advanced deliberately in the definitions. Cached release content is reused until its cache is removed; no native XML processing is added.

Closes #116.

[moshest]

Does this close #116?

Yes. The issue title says DocBook, but the maintainer's comment on #116 explicitly rejected native XML parsing and asked for exactly this — a pinned HTML-index source over freedesktop.org's rendered man pages. This is the requested mechanism, and it delivers the content the issue was actually about. Closes #116 is legitimate.

The one part of #116 not implemented is point 5 (multiple docs_path roots). Solved here by shipping two packages instead, which the maintainer also offered. Fine.

Verified numbers

Built from a clean worktree at a2cc15d:

• systemd (html-index, 258): 3,234 sections, 1,108,416 tokens, 432 documents from 1,229 index links after dedup. Matches your claim exactly.
• systemd-guides (git, v258): 531 sections, 220,331 tokens. Matches.
• pnpm lint clean, pnpm build clean, pnpm test 317 passed (224 context + 93 registry).
• Spot-checked the crawl: systemctl, journalctl, systemd.service, systemd.unit, systemd.network, systemd.exec, loginctl all present.

Nightly risk

Traced it. publish-all wraps both discoverVersions and the per-version build in try/catch, so a throwing html-index records one failure and keeps going — the other ~140 packages still publish, and the job exits 1 at the summary. The only path that aborts everything is listDefinitions throwing on a schema error, which is pre-existing for git and zip too. Acceptable.

Crawl cost is also one-time: checkPackageExists runs before the build, so after 258 publishes once, nightly makes zero fetches to freedesktop.org.

Notes, none blocking

1. Both systemd packages are frozen. systemd@258 needs a manual PR per release. Worse, systemd-guides is unversioned but pinned to ref: v258 — getHeadCommit on a tag never changes, so after the first publish the skip-if-unchanged check fires forever and latest is permanently v258. An unversioned definition normally tracks tip; this one silently won't. Consider dropping ref: there, or at least a comment saying it's deliberate.
2. isZipVersionEntry is now dead outside the index.ts barrel and one test assertion. Its meaning also changed ("versions" in entry → source.type === "zip"), so anything still importing it gets different behavior. Worth deleting.
3. registry/README.md "Where the file goes" still says non-package-manager projects get "a directory named after the project containing a single self-named file". registry/systemd/ now has two. The new html-index section explains it, but the earlier rule now contradicts the tree.
4. The barePre rule filters on node.firstChild?.nodeName !== "CODE". A <pre> with whitespace before <code class="language-x"> has a text node first, so it takes the bare path and loses the language fence. Edge case, but the third test only covers the no-whitespace form.
5. No automated robots.txt check — the README makes it a manual reviewer step. Reasonable for now, just flagging that a 1,229-page crawl of a third-party site rests on a prose instruction.

Docs and changeset are both right: registry/README.md has a real section for the new source type, and the changeset targets @neuledge/context (published) for the html.ts fix, not @neuledge/registry (private).

---

Generated by Claude Code