feat(trainer): longest-match symbol lookup (RFC 0020 clause 4)

[willwade]

Makes multi-codepoint symbols (digraphs, ZWJ families, VS16 skin tones) trainable: SymbolStream::next probes multi-codepoint map keys longest-first before the single-character path. First PR of RFC 0020.

• CAlphabetMap::Add collects multi-codepoint keys (copies, sorted longest-first) + LongestMatch() probe + MaxKeyLen()
• next() probes before single-char dispatch (❤️ beats its prefix ❤); ensureLookahead(want) extracted from findNext keeps MaxKeyLen bytes buffered across 1024-byte window refills
• 6 unit tests + CAPI end-to-end (flat alphabet, corpus of only multi-cp tokens → top child 1.94× uniform; failed pre-fix)
• Emoji corpus test relaxed to whole-node tokens per RFC
• Full suite 47/47 locally

Confidence Score: 4/5

The PR is not yet safe to merge because escaped adaptive-training contexts containing multi-codepoint symbols are reconstructed incorrectly.

Findings

1.  Raw parsing corrupts context ▶
2.  Longest match hides delimiter ▶

Summary

This PR adds longest-match training support for multi-codepoint alphabet symbols and protects structural annotation delimiters with raw, single-codepoint stream operations.

• Registers multi-codepoint alphabet keys and probes them longest-first across stream refills.
• Tracks the complete previously consumed token for peekBack().
• Adds raw stream operations for conversion annotations and context-switch delimiters.
• Adds internal, C API, refill-boundary, prefix, and delimiter regression coverage.
• The raw mode is applied too broadly to context-switch body text, which prevents multi-codepoint symbols there from seeding the language-model context correctly.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    A[Training stream] --> B{Context escape?}
    B -- No --> C[Longest-match next]
    C --> D[Learn alphabet symbol]
    B -- Yes --> E[Read delimiter as one raw codepoint]
    E --> F[Read saved document context]
    F --> G[Enter resolved symbols into model context]
    G --> H[Read closing delimiter structurally]
    H --> C

Loading

_{Reviews (4) · Last reviewed commit: "fix(trainer): greptile P1 r3 — raw mode ..."}