WorldAlphabets alphabets: tree barely expands despite full training corpus consumption (flat/unweighted canvas)

[willwade]

What

All 342 autoconverted alphabet.wa.*.xml alphabets render a nearly flat tree: at rest only 2–3 top-level boxes show, and steering into the tree yields ~2 letter boxes with a 1.1× size spread — letters look unweighted/uniform. Maintained alphabets on the same build show 25+ boxes with 2.4×+ spread.

What it is NOT

• Not the training corpus: instrumenting CTrainer::Train shows the full corpus is consumed — English (WorldAlphabets) trains on 1,490,517 symbols (72,326 unmapped) from training_wa_en_Latn.txt (1.5 MB). The LM is learning.
• Not the lazy-alphabet work (feat: lazy alphabet loading + generated alphabet metadata index #68): reproduced identically on pre-feat: lazy alphabet loading + generated alphabet metadata index #68 main (eager full parse).
• Not the space symbol: WA files do define space (<node label="␣" text=" ">); the node-level text attribute is honored by the loader.

Reproducer (host)

// adaptive probe: steer into the largest box for 5 rounds, dump opcode-4 box heights
dasher_set_alphabet_id(ctx, "English (WorldAlphabets)");
// → letter-boxes=2  h_min=10 h_med=11 h_max=11  spread=1.1x
dasher_set_alphabet_id(ctx, "English with limited punctuation");
// → letter-boxes=25 h_min=10 h_med=16 h_max=24  spread=2.4x

Probe source available (adaptive steering into the largest frame box — rules out the pointer just missing the tree).

Suspects

• Group/child accounting in the WA file structure (paragraphSpace/Editing groups, iNumChildNodes) producing malformed expansion
• Node-probability query path returning uniform values for these alphabets despite the trained LM (symbol numbering mismatch between the alphabet map and tree children?)
• The WA palette (colorsName="Default") making what does render look uniformly coloured — cosmetic, but compounds the 'no weights' impression

Why it surfaced now

Dasher-Android's locale-follow (PR #31, first version) briefly defaulted en-locale devices onto these alphabets (scoring bug, since fixed) — the first code path to auto-select a WA alphabet. Before that, nothing selected them unless a user did, by hand, from a 474-entry menu.

Impact

Every language that ONLY has a WA alphabet (a large share of the 456 lang codes in the catalogue) effectively gets an unweighted Dasher. That's the next-biggest usability item after the startup work.

[willwade]

Diagnosis narrowed (Aug 28, follow-up session)

New evidence from instrumented probes (all reproducible via small CAPI programs — probes attached to this comment's thread on request):

1. The LM learns correctly. Dumping the language model through the trainer's own empty context right after CTrainer::Train shows a real learned distribution — for English (WorldAlphabets) the maximum sits at symbol 64 (the space character), value 2605/65536, exactly where ~17% of a 1.49M-symbol English corpus should put it.

2. The rendered tree never sees it. The root node's probability vector (CAlphNode::GetProbInfo → GetProbs) shows symbol 64 at the uniform floor (50) and every symbol in a near-flat band (min 50, max 1775, no space spike anywhere) — i.e. the tree queries symbol indices whose training was recorded under a different index mapping.

3. So this is a symbol-numbering / context mismatch between the training map and the node tree for WA alphabets, not a training or corpus problem. Candidate wiring to inspect: CAlphabetManager::InitMap vs node-child iteration for the WA group layout (which includes action-only nodes — the Editing group's ⌫/⌦ deleteTextAction nodes contribute characters whose Text is empty, and are counted in iEnd), and root-context creation.

4. Maintained alphabets don't show the flat symptom, so whatever diverges is triggered by the WA file structure (action nodes in groups, group ordering, or the paragraphSpace/Editing layout).

Related but separate: relative data dirs never loaded any alphabet (fixed)

While probing this we found GTK's relative "Data" data dir broke alphabet loading entirely (index paths were relative; LoadAlphabetFile treated them as regex patterns; every selection silently fell back to the builtin 26-letter Default → the same "flat unweighted letters" symptom, different cause). Fixed in #68 (0365124): the indexer now stores absolute paths, verified with relative and absolute data dirs. If you saw flat letters on GTK dev builds, that was this; the WA tree bug is what remains for WA alphabets.

Impact restated

321 of 323 languages in the catalogue are WA-only — until this is fixed, effectively every non-English language renders unweighted. This is the highest-priority engine bug in the v6 data stack.

[willwade]

Root cause found — and fixed data-side (no engine changes). da2e558

My earlier symbol-numbering hypothesis was wrong. Bisecting with identical nodes/corpus/palettes and mutating one attribute at a time pinned it exactly:

colorInfoName="lower case" (with a space) → 1 box, 1.0× spread.
colorInfoName="lowercase" → 28 boxes, 5.3× spread.

The autoconverter emitted free-form colorInfoName values (lower case, korean trailing consonants, hundreds of pinyin/syllabary group names…), but the shipped palettes define exactly eleven tokens: lowercase, uppercase, numbers, punctuation, accents, space, paragraph, paragraphSpace, limitedPunctuation, punctuationLong, lowercaseBackground. A group whose token no palette resolves gets no colour info, and the renderer draws nothing for its nodes — the probabilities exist but are invisible. (The training, LM and symbol numbering were all fine; that's why the trainer-side dumps looked healthy while the canvas looked dead.)

Fix shipped (converter-side, as preferred)

Scripts/fix-wa-colorinfo.py rewrote all 474 autoConverted files: recognised tokens preserved, everything else keyword-mapped (upper→uppercase, digits/numerals→numbers, punctuation→punctuation, accents/diacritics/combining→accents, editing→punctuation, unknown letter groups→lowercase, the engine's own default).

Verified (tree-spread probe, before → after):

• English (WA): 2 boxes 1.1× → 28 boxes 5.3×
• Arabic (WA, 36 duplicate symbols): → 31 boxes 2.8×
• Korean (2325 syllables): → 25 boxes 2.4× (matches maintained English)
• Suite 39/39

Rule for the WorldAlphabets Dasher exporter

Emit only the eleven palette tokens above, or omit colorInfoName entirely (the engine then defaults the group to lowercase). Never free-form group names.

Remaining (separate)

• Conversion-mode alphabets (Mandarin pinyin routings) still render sparsely at the top level — different mechanism from colour tokens, worth its own investigation.
• The upstream converter should adopt the token rule so future exports don't regress; the fix script stays as the safety net.

[willwade]

Primary cause fixed data-side in #68 (da2e558): colour-token normalization across all 474 autoConverted alphabets, verified across English/Arabic/Korean. Follow-ups tracked separately: conversion-mode rendering (Mandarin pinyin), and adopting the token rule in the upstream WorldAlphabets exporter.