Qwen3.8-Flash-Next: transformers 5.16 drops \p{M} from the pre-tokenizer, so combining-mark text gets different ids #648

load_tokenizer (utils/hf.py) uses AutoTokenizer. With the pinned transformers 5.16.x, this checkpoint loads as Qwen2Tokenizer, which replaces the tokenizer.json pre-tokenizer with a split regex that lacks \p{M}. Text with combining marks then gets different ids than the model was trained on: नमस्ते दुनिया gives 10 ids instead of 6, and สวัสดีครับ gives 7 instead of 3. ASCII and precomposed Latin text are unaffected. It is the same on 5.15.1 and 5.17.0, and correct on 5.18.0+, which loads Qwen3_5Tokenizer.

from freetoken.utils.hf import load_tokenizer
from tokenizers import Tokenizer
p = "<local nvidia/Qwen3.8-Flash-Next-NVFP4 dir>"; t = "नमस्ते दुनिया"
print(len(load_tokenizer(p).encode(t, add_special_tokens=False)),
      len(Tokenizer.from_file(p + "/tokenizer.json").encode(t, add_special_tokens=False).ids))  # 10 6

Expected: ids match tokenizer.json. For example, raise the pin to >=5.18, or keep the file's pre-tokenizer in load_tokenizer.

Environment: main 07813248 (CPU-only check), transformers 5.16.1, Python 3.14.

Scope (transformers 5.16.1 Qwen2Tokenizer against tokenizer.json):

• Identical:
  • emoji with variation selectors (⚠️, ✅, ❤️, ✔️, 1️⃣);
  • Latin with a combining accent (café), precomposed Latin, Japanese, Korean, Greek, Cyrillic;
  • all 770 text files of this repository (.py, .md, .json, .sh, .cu, ...);
  • the last 3,000 commit messages.
• Different: scripts whose marks combine inside a word, e.g. Devanagari (नमस्ते) and Thai (สวัสดี).

ASCII and code-heavy traffic is unaffected. Indic and Southeast Asian text gets out-of-distribution ids.