Speed up tokenization ~25%, add perf bench, clean up lint and the stale gold file

[vthorsteinsson]

Three independent commits; review them one at a time.

1. Clean up ruff findings from its 0.16 default rule set

Ruff 0.16 turned on pyupgrade, RUF, SIM and other rule families by default. Because the lint group pins ruff>=0.15, unchanged code suddenly had 213 findings. CI runs only mypy and pytest, so it stayed green.

• Safe autofixes: list[str], X | None, collections.abc imports (fine with the 3.10 floor).
• Not applied: the autofix that rewrote Tok.from_txt/from_token to use typing_extensions.Self. Tokenizer has no runtime dependencies, so that import would fail on a clean install. The tests didn't catch it because typing_extensions comes in through dev dependencies. The original bound TypeVar stays, with a noqa and the reason.
• Manual: ClassVar on the Abbreviations tables, tuple startswith/endswith, yield from, datetime.date for date validation.
• FURB166 is a false positive: its base=0 suggestion would break &#x...; HTML escapes, so that line gets a noqa.
• SIM102 is ignored in config rather than collapsing 9 nested ifs in the core matching logic. Some of them sit in elif chains, where collapsing would change behavior. If you'd rather collapse the safe ones, say so.

2. Speed up tokenization ~25% and add a performance bench

Found with the new CPython 3.15 sampling profiler (python -m profiling.sampling):

• The zero-width and composite-glyph replacement passes ran a regex scan on every token. Now one check per text decides whether they are needed at all. (~10%)
• parse_digits makes up to ~25 re.match("<pattern>", w) calls per number, each paying ~220 ns of re cache lookup. All string-pattern regexes in tokenizer.py are now precompiled. (~12%)

tokenize, large test set	before	after
CPython 3.13	1371 ms	1097 ms	+25%
CPython 3.14	1472 ms	1150 ms	+28%
CPython 3.15	1381 ms	1128 ms	+22%

Output is byte-identical: digests of (kind, txt, val, original, origin_spans) for every token match across all five bench workloads.

bench/bench.py records timings plus output digests (--save / --compare). bench/profile.sh writes a pstats summary, self time per function, a flamegraph and a line heatmap. It uses --blocking on purpose: without it, the 3.15 profiler fails to unwind ~70% of stacks in this generator pipeline (its "Error rate" header) and charges inner stages' work to the outer stages' next() lines.

On match statements (looked at while doing this): on 3.15, value patterns like case TOK.WORD: still compile to the same COMPARE_OP + jump sequence as an if/elif chain, with no jump table. In a microbenchmark they were 8–14% slower on 3.13–3.15, so this PR doesn't use them.

3. Regenerate the large test set gold file and enforce it with a test

toktest_large_gold_acceptable.txt was byte-identical to Tokenizer 2.0.0's output (Jan 2020). Running each release tag's own CLI shows the drift came from 2.2.0, 3.0.0 and 3.4.0, plus one line from #63, and that nothing changed from 3.4.0 through 3.6.4.

• The gold file is regenerated, and test/test_large_gold.py now fails on any CLI output drift.
• The [input|gold] ranges in test/Overview.txt are recomputed. The rule used reproduces 127/137 of the original ranges on the old file; the remainder were boundary inconsistencies plus four typos.
• ⚠️ Please check the Icelandic wording of the dated note added to the Overview.txt legend.
• ⚠️ The drift is not uniformly an improvement. Compared with 2.0.0 and measured against gold_perfect, current output gains 244 correct lines and loses 215. Examples of losses: sentence-final abbreviations keep their period (ehf. instead of ehf .), and some sentence splits after abbreviations are missed (…am.k. Bara að það væru 7 .). This PR records current behavior as the baseline; it doesn't fix those.

Verification

• pytest: 152 passed (CPython 3.13 env and 3.10); mypy clean; ruff check + format clean
• Clean non-editable installs on CPython 3.10 and PyPy 3.11 pull in no other packages, and their CLI output on the large test set is identical
• Each commit was tested on its own before committing

🤖 Generated with Claude Code