Fix pinned Qwen literal tokenization before native initialization #2

Gap and evidence

Merged main cannot initialize the pinned Qwen tokenizer in the literal-special-v2 path. With a SHA-verified tokenizer.json from Qwen/Qwen2.5-0.5B revision 060db6499f32faf8b98477b0a26969ef7d8b9987 and tokenizers 0.22.1, the native literal test fails while constructing QwenTokenizer: its self-check encodes <|fim_prefix|>, then encoding.py raises ValueError: control token detected in user input. The broad suite skips this test when the pinned snapshot is absent, so its green CPU result does not cover this path. The pinned tokenizer also normalizes decomposed Unicode, which the current exact round-trip contract rejects.

This blocks native initialization and every pretrained/native parity claim. It is a confirmed implementation defect, not just missing test evidence.

Acceptance

• A hash-verified pinned Qwen tokenizer initializes and passes the real native test, with the revision and tokenizer hash recorded.
• Literal delimiter spellings, escaped/decomposed Unicode, nested JSON and structured criteria preserve exact input content without producing control-token IDs, truncation or silent replacement. If the chosen tokenizer cannot support the claimed exact contract, define and test an explicit honest limitation instead.
• Native initialization, checkpoint reload and serving use one versioned preprocessing identity; legacy checkpoints retain explicit migration/refusal behavior.
• Add a real pinned-tokenizer regression to CI or an explicit reproducible native gate; do not count the byte-tokenizer fixture as proof.

Reproduction: provide the verified snapshot in KEV_LAYA_QWEN_DIR and run python -m pytest tests/test_stage3_edges.py::test_real_pinned_qwen_tokenizer_literals -q.

Related acceptance plan: NEXT_STAGE_PROMPT.md.

Continuation evidence — 2026-09-27: unpublished source candidate

A new six-file patch, layev-tokenizer-v3.patch, is implemented in the continuation delivery bundle against main 080aa7183bb5d8f6576bde9d41e3c5115c7ab132. SHA-256: 67f5354f2873a5c4ca284b9c068325a5004703f107b32ff4628519cd6e41454a. It is not committed, pushed, native-validated or merged. All acceptance boxes remain open.

The candidate defaults to literal-bpe-v3. It reads and fingerprints the original tokenizer bytes once, derives an in-memory literal backend without added-token extraction or Unicode normalization, retains the BPE model/pre-tokenizer/decoder, and forbids every original added-token ID, including IDs marked special: false. Exact decoder round trips remain mandatory. The original full vocabulary/control/output backend and source file are retained. The state wrapper remains serialization v2; the new literal-version tag gives a distinct preprocessing identity. Explicit legacy v1/v2 paths retain their historical behavior/refusals instead of silently switching existing checkpoints to v3. The existing stage3 recording double is updated for the construction API without deleting or weakening its assertions.

Measured source-only evidence: 30 failures / 8 passes against the unchanged encoding source; 38 passes after the patch, using an explicitly labeled UTF-8/added-token/NFC double. An additional 17 verifier-preflight tests pass, for 55 total, zero skipped. These are not Rust/Qwen/native/model/quality results. The complete original encoding.py, schema.py and test_stage3_defects.py preimages were checked against their live Git blob IDs before editing. git apply --check, application, reverse-check and all six postimage SHA-256 comparisons passed in a disposable uncommitted tree. Compilation passed for the fetched/changed source, tests and script.

The patch adds an explicit offline gate, scripts/verify_pinned_tokenizer.py, plus docs/NATIVE_TOKENIZER_GATE.md. It requires the existing pinned snapshot, source.json revision/hash agreement, a separately reviewed pinned-source tokenizer SHA-256 and tokenizers 0.22.1. It checks literal/Unicode/nested content, request accounting, exact budget boundaries, question-ID exclusion, tokenizer-metadata reload and unchanged source bytes. It makes no network calls or source writes. Missing prerequisites exit 2, not a passing skip. The actual missing-source invocation here exited 2 and reported zero literal cases executed. Even a future passing receipt explicitly leaves model initialization/checkpoint/serving, CUDA, trained context, quality and Jev parity untested.

Remaining prerequisites: execute the gate and native tests on the existing authorized model workstation, validate actual checkpoint/serving identity and the complete repository suite, then use the configured verified SSH signer and hosted checks for delivery. This runtime has neither the pinned snapshot/tokenizers installation nor the Windows checkout/signing agent. Full-suite collection in the partial source copy stopped at missing kev_laya.telemetry; this is not a full-checkout test result. No WSL or existing run was modified. #3 remains blocked by this issue.

Exact archived-source reconciliation — 2026-09-27

The actual bytes of the continuation candidate above (A) and the separately quarantined review candidate (B) have now been retrieved from their original delivery archives and compared. This replaces the earlier unknown source comparison with a measured finding; it does not validate or select either implementation.

• A: patch SHA-256 67f5354f2873a5c4ca284b9c068325a5004703f107b32ff4628519cd6e41454a; encoding.py SHA-256 e91379ff8f6b88725e150ebf96b109e2e23568ae7e38f29010fbbf7c542d9662.
• B: patch SHA-256 16462a846089a38ad59e59b98c0044ec9792335a23418053e2871bae2b48902c; encoding.py SHA-256 613c57222580be5001f0f4040af326b146de09baa74bc9ead0ae20ef65f21913.

Confirmed source-contract mismatch: both candidates report literal-bpe-v3 and identical preprocessing_identity() values for the same source bytes, but A's encode_output() retains the original added-token/NFC backend while B routes output through the lossless literal backend. With the same explicitly injected UTF-8/added-vocabulary/NFC double, metadata matches and all 12 tested literal inputs round-trip exactly without reserved IDs, yet output token IDs/counts differ in 10 cases. Examples: <|fim_prefix|> counts 1 versus 14; decomposed e plus U+0301 counts 2 versus 3; decomposed Hangul counts 3 versus 6. These are double-derived counts, not real Qwen counts or billed usage. This finding alone does not demonstrate different native logits or failed checkpoint compatibility; it demonstrates that the recorded identity does not distinguish these output-accounting implementations.

New offline diagnostic diagnostics/compare_candidates.py verifies exact archived-source hashes before loading, performs no network/service/repository writes, and exits 1 when this collision is reproduced. Its SHA-256 is 3e1589e20f0369dba22e00da1209508ddf541039cd9f027ef3300f0c6a176dde. Actual comparison receipt SHA-256: 71fe7d7a14267c9dfe9160091e6eae0aa585e84842f82b9c6dc6425cf4f4ccd1. The accompanying new diagnostic suite passed 9/9, including changed-source rejection. Independent reruns of the archived focused suites on Python 3.13.5 produced A: 55 passed, zero skipped; B: 76 passed, one skipped because the Rust runtime is unavailable. These counts belong to separate suites/candidates, not one combined accepted implementation. No full repository, locked, hosted, real pinned-tokenizer, checkpoint/serving, weights, CUDA, quality or Jev gate ran in this reconciliation.

Before native training or delivery, reconcile the actual pending workstation changes into one reviewed implementation and explicitly freeze its input preprocessing and output-accounting contracts. Do not stack these patches, silently relabel existing artifacts or regard a shared version string as equivalence. The distinct literal-special-v3 alternative remains a separate unselected candidate. No fourth tokenizer implementation, source commit, push, PR or merge was created here. All original acceptance criteria, historical failures and native dependencies remain unchanged and open.

Subsequent v4 reconciliation candidate — 2026-09-27

A subsequent execution session retrieved all three prior delivery archives, verified their checksum manifests (40/40, 33/33 and 53/53 entries), compared their actual encoder bytes, and implemented one replacement source candidate, layev-tokenizer-v4-reconciled.patch, against the inspected main b4cb9df97be61729ebd68b8e0f5ee9bb2d193c73. Patch SHA-256: d1e0baf152f4645c60dda52b6c013532e37a6d7a7dd98899fb1bc13056d507a7. This candidate is not a selection already applied to the workstation: no source commit, push, PR, hosted candidate checks, merge or native acceptance is claimed. The earlier sections above remain historical evidence for their respective candidates.

The candidate uses literal-bpe-v4, deriving an ordinary deterministic BPE literal backend from the single fingerprinted source read, without normalization or any added-token extraction. Exact input round trips and reserved-ID refusal remain mandatory. Output counting is explicitly source-json-output-v1, retaining the original source backend without padding, truncation or automatically inserted special tokens. The derived-backend hash and output policy are bound into a recomputable fingerprint in the literal_encoding identity string. This preserves the exact existing three-string-field telemetry v2 identity shape and original source-tokenizer identity; explicit components are also stored in checkpoint metadata and gate receipts. No telemetry schema is changed.

Both unpublished v3 tags are refused instead of silently accepted or relabeled. Explicit v1/v2 paths retain historical behavior and limitations. V4 checkpoint loading requires the full matching identity and components; omitted, null, conflicting or altered metadata is rejected. Checkpoint export verifies the source bytes again before copying them, rejecting a source changed since initialization before writing a replacement tokenizer. These are source-boundary repairs, not demonstrated native model/checkpoint behavior.

Measured in the isolated Python 3.13.5 runtime: 132 focused tests passed, zero failed/skipped, covering literal contract doubles, identity/output policy, the telemetry identity projection, checkpoint functions with explicit fake collaborators, and offline gate prerequisites. An exact extraction of the original tokenizer-only stage3 test functions passed 12/12, not the full stage3 suite. Against the unchanged main encoding preimage, the inherited 38 literal-double tests produced 30 failures / 8 passes. Compilation passed. All four recovered preimages were checked against the live main Git blobs. The nine-file patch passed isolated application, reverse application and exact postimage hash checks.

The required full-suite attempt in this partial source export stops at collection with missing kev_laya.telemetry, which exists in the live main tree; this is an export limitation, not evidence of a main-branch defect. Ruff/locked/hosted checks for this new patch have not run. The actual offline native-gate invocation exits 2, reason source_directory_missing, zero literal cases executed; pinned Rust/Qwen, actual weights/checkpoint/serving, CUDA, trained context, quality and Jev parity remain untested/unknown.

Delivery artifact: jev-tokenizer-reconciliation-2026-09-27.zip in the execution-session response, containing the single active v4 patch, postimages, preimages, source-test receipts, hash manifest and workstation gate notes. Do not apply it sequentially with A/B/the alternate v3 patch; reconcile any actual pending workstation work first. The Windows primary checkout, configured SSH signing agent, pinned snapshot and running Overwatch service were not accessible in this session. No WSL instance, existing run, private data or remote transport was modified. Keep this issue and its native dependents open until the original acceptance criteria are demonstrated.

Windows primary-checkout v4 delivery — 2026-09-27

The reconciled v4 source is merged in PR #13 as verified SSH-signed source commit 078eaf7195b118cdd5c907bc12b0c556d8dae32f; merge commit e1db691cc7710ff808fa1220b16628c0d675d16c. All five hosted PR checks passed. The primary checkout was fast-forwarded to main, clean, with divergence 0/0. Earlier unpublished v3 candidates remain historical only; neither was stacked or relabeled.

The real Qwen/Qwen2.5-0.5B tokenizer at revision 060db6499f32faf8b98477b0a26969ef7d8b9987 was independently matched byte-for-byte to a fresh upstream download: SHA-256 c0382117ea329cdf097041132f6d735924b697924d6f6fc3945713e96ce87539. Under Python 3.13.12 and tokenizers==0.22.1, the offline v4 gate passed 36 literal cases, source-output policy comparison, metadata reload, exact budget checks and v3 refusal. The complete local suite, including the real pinned-tokenizer regression, passed 413 tests with 4 explicit skips. Compileall, CI-pinned critical Ruff and source audit passed. The four skips are official SDK, CUDA FP32, CUDA BF16 and Windows symlink privilege.

Keep this issue open. The local PyTorch build is CPU-only and pinned model weights are absent. Actual pretrained model initialization, trained checkpoint reload, serving, CUDA, context, quality and Jev parity remain unverified. The post-merge main workflow 36312254077 completed successfully with all five jobs passing.

Final native readback — 2026-09-27/28

The tokenizer-v4 source and native short-CUDA/oracle corrections are delivered in protected PR #24, merged at 937fee343fdc758cd241e3e2bb48090a60ddd55f. All five PR checks and all five post-merge main jobs passed. The primary main checkout is clean and synchronized, with divergence 0/0.

The pinned Qwen/Qwen2.5-0.5B revision 060db6499f32faf8b98477b0a26969ef7d8b9987 tokenizer SHA-256 is c0382117ea329cdf097041132f6d735924b697924d6f6fc3945713e96ce87539. The real tokenizers 0.22.1 v4 gate passed 36 literal/Unicode/structured and boundary cases with exact source-output accounting, metadata reload and legacy v3 refusal. The explicit offline native gate and CI regression are retained in source; the fixture suite is not the basis of this claim.

An actual pinned-weight initialization, two-step FP32 LoRA training with same-run resume, and authenticated loopback serving used the same recorded literal-bpe-v4:002e76a7328e372b4ca78a015cc203a522994eb150be050f834a136572d4965e identity. The trained checkpoint SHA-256 is b6e0db32a65bbf240a5cdec8c938040054a45284e5d4cf35cd310160262b6efb. A fresh probe run from merged main reloaded that checkpoint and passed independent Transformers 4.57.1 hidden-state parity at the unchanged combined tolerance, with 62 input tokens and maximum absolute difference 0.00011444091796875. The private raw receipt SHA-256 is c3ff436ac722f9e5d49a76d404047f7f273888c601a81d41a012c1b2fa84bc12; portable source-bound receipts are in docs/evidence/native/ from PR #24. The service and checkpoints are local, outside Git.

This closes the tokenizer/preprocessing integration acceptance in #2. Broader full-weight/BF16 training, long-context exposure, quality, and Jev comparison remain open under #3–#7; this short synthetic probe does not establish them.