Cut an MkDocs query into words the way titles and bodies are

[kevin9327]

oc astral uv pyproject.toml answers that nothing matches, though uv's docs have two sections titled "The pyproject.toml". The MkDocs ranker split the query on spaces only, and titles and bodies at every character outside [a-z0-9_], so a query word holding a dot or a dash could never equal a word of the index.

Before, on the live uv index:

$ oc astral uv pyproject.toml
# docs.astral.sh search: pyproject.toml
nothing in the docs' own index matches; try fewer or different words

$ oc astral uv "how do I add a dependency?"
342 sections match in the docs' own index, ranked locally, top 20 shown; no section matches every word, so these match some:
[1] add-bounds

After:

$ oc astral uv pyproject.toml
342 sections match in the docs' own index, ranked locally, top 20 shown:
[1] The pyproject.toml (Project structure and files)
[2] The pyproject.toml (Migrating from pip to a uv project)
[3] Using pyproject.toml

$ oc astral uv "how do I add a dependency?"
108 sections match in the docs' own index, ranked locally, top 20 shown:
[1] Adding dependencies

The same happened to uv.lock, --group, and any question ending in ?, whose last word never matched. An accented word was cut apart on both sides and matched no title.

The fix is one wordsOf used for the query, titles, and bodies. It cuts at any character that is not a letter, digit, or underscore, in any script, and drops a contraction's tail first, so "what's" reads as "what" (a stop word) instead of "what" plus a stray "s" every section would have to match.

Test. tests/mkdocs.test.js gains one test over a small index: pyproject.toml, what is uv.lock?, --group, and démarrage each lead with the section that holds them, and "what's a dependency group?" keeps exactly dependency and group with no partial flag. On main it fails at the first assertion (undefined where 'The pyproject.toml' is expected). Full suite: 311 tests, 308 pass, 3 skipped, 0 fail.

Left alone. Stop words, stemming, and scores are unchanged. The shared ranker in nodedocs.js (Node and RDoc) is not touched: it falls back to a substring match, so fs.readFile already works there. No CHANGELOG line, since the MkDocs backend itself is still under Unreleased.

🤖 Generated with Claude Code