Skip to content

feat: lazy alphabet loading + generated alphabet metadata index #68

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
willwade merged 8 commits into from
Aug 28, 2026
Merged

feat: lazy alphabet loading + generated alphabet metadata index #68

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
d32a79e
feat: lazy alphabet loading + generated alphabet metadata index
willwade Aug 28, 2026
3d2ec86
fix: tolerate duplicate symbols in alphabet files (crash on select)
willwade Aug 28, 2026
9d8d158
fix: tolerate digraph symbol outputs in AlphabetMap (second crash)
willwade Aug 28, 2026
adf099c
fix: index alphabets with real XML parsing semantics (PR review)
willwade Aug 28, 2026
c2d3089
fix: deterministic duplicate-ID resolution in the alphabet index (rev…
willwade Aug 28, 2026
702bb40
fix: alphabet tier by immediate parent directory, not path substring …
willwade Aug 28, 2026
0365124
fix: absolutize index paths — relative data dirs never loaded alphabets
willwade Aug 28, 2026
da2e558
fix(data): normalize colorInfoName to palette tokens — WA alphabets r…
willwade Aug 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
16,760 changes: 16,760 additions & 0 deletions Data/alphabets/alphabet_index.json
View file
Open in desktop

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions Data/alphabets/autoConverted/alphabet..klingon.tlhingan.hol.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,7 @@
<textCharAction />
</node>
</group>
<group name="Numerals" colorInfoName="numerals">
<group name="Numerals" colorInfoName="numbers">
<!--Old Group Color: 113-->
<node label="">
<!--Note: 0-->
Expand Down Expand Up @@ -146,7 +146,7 @@
<textCharAction />
</node>
</group>
<group name="Punctuation, etc" colorInfoName="punctuation, etc">
<group name="Punctuation, etc" colorInfoName="punctuation">
<!--Old Group Color: 112-->
<node label="'">
<!--Note: latin glottal stop ()-->
Expand Down
2 changes: 1 addition & 1 deletion Data/alphabets/autoConverted/alphabet..runic.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -303,7 +303,7 @@
<textCharAction />
</node>
</group>
<group name="Runic punctuation" colorInfoName="runic punctuation">
<group name="Runic punctuation" colorInfoName="punctuation">
<!--Old Group Color: 0-->
<node label="᛫">
<!--Note: RUNIC SINGLE PUNCTUATION-->
Expand Down
6 changes: 3 additions & 3 deletions Data/alphabets/autoConverted/alphabet.abc.musical.language.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
English order. The only changes here are notes.
See also http://staffweb.cms.gre.ac.uk/~c.walshaw/abc/
-->
<group name="Second octave notes" colorInfoName="second octave notes">
<group name="Second octave notes" colorInfoName="lowercase">
<node label="a">
<!--Old Char Color: 10-->
<textCharAction />
Expand Down Expand Up @@ -117,7 +117,7 @@
<textCharAction />
</node>
</group>
<group name="First octave notes" colorInfoName="first octave notes">
<group name="First octave notes" colorInfoName="lowercase">
<!--Old Group Color: 111-->
<node label="A">
<!--Old Char Color: 10-->
Expand Down Expand Up @@ -278,7 +278,7 @@
<textCharAction />
</node>
</group>
<group name="Annotation" colorInfoName="annotation">
<group name="Annotation" colorInfoName="lowercase">
<!--Old Group Color: 112-->
<node label="%">
<!--Old Char Color: 90-->
Expand Down
4 changes: 2 additions & 2 deletions Data/alphabets/autoConverted/alphabet.adangbe.with.lots.of.punctuation.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<!-- same as Adangbe and Ga and Akan and Ewe?? -->
<!-- Author: David J C MacKay 2005 -->
<!--- http://www.ethnologue.com/show_language.asp?code=TWS -->
<group name="Lower case Latin letters" colorInfoName="lower case latin letters">
<group name="Lower case Latin letters" colorInfoName="lowercase">
<node label="a">
<!--Old Char Color: 10-->
<textCharAction />
Expand Down Expand Up @@ -125,7 +125,7 @@
<textCharAction />
</node>
</group>
<group name="Upper case Latin letters" colorInfoName="upper case latin letters">
<group name="Upper case Latin letters" colorInfoName="uppercase">
<!--Old Group Color: 111-->
<node label="A">
<!--Old Char Color: 10-->
Expand Down
2 changes: 1 addition & 1 deletion Data/alphabets/autoConverted/alphabet.afaan.oromo.with.numerals.and.punctuation.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ Afaan Oromo is a Cushitic language spoken by about 17 million people in Ethiopia
<textCharAction />
</node>
</group>
<group name="Upper case Latin letters" colorInfoName="upper case latin letters">
<group name="Upper case Latin letters" colorInfoName="uppercase">
<!--Old Group Color: 111-->
<node label="A">
<textCharAction />
Expand Down
2 changes: 1 addition & 1 deletion Data/alphabets/autoConverted/alphabet.afrikaans.with.punctuation.and.numerals.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@
<textCharAction />
</node>
</group>
<group name="upper case letters" colorInfoName="upper case letters">
<group name="upper case letters" colorInfoName="uppercase">
<!--Old Group Color: 111-->
<node label="A">
<textCharAction />
Expand Down
4 changes: 2 additions & 2 deletions Data/alphabets/autoConverted/alphabet.akan.twi.with.lots.of.punctuation.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
<alphabet name="Akan (Twi) with lots of punctuation" orientation="LR" trainingFilename="training_akan_GH.txt" colorsName="European/Asian">
<!-- same as Adangbe and Ga and Akan and Ewe -->
<!--- http://www.ethnologue.com/show_language.asp?code=TWS -->
<group name="Lower case Latin letters" colorInfoName="lower case latin letters">
<group name="Lower case Latin letters" colorInfoName="lowercase">
<node label="a">
<!--Old Char Color: 10-->
<textCharAction />
Expand Down Expand Up @@ -124,7 +124,7 @@
<textCharAction />
</node>
</group>
<group name="Upper case Latin letters" colorInfoName="upper case latin letters">
<group name="Upper case Latin letters" colorInfoName="uppercase">
<!--Old Group Color: 111-->
<node label="A">
<!--Old Char Color: 10-->
Expand Down
2 changes: 1 addition & 1 deletion Data/alphabets/autoConverted/alphabet.amharic.with.punctuation.and.numerals.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
<!DOCTYPE alphabet SYSTEM "../alphabet.dtd">
<alphabet name="Amharic (አማርኛ) with punctuation and numerals" orientation="LR" trainingFilename="training_amharic_ET.txt" colorsName="European/Asian">
<!-- identical alphabet used for Tigrinya, Amharic and Ethiopic -->
<group name="ethiopic letters" colorInfoName="ethiopic letters">
<group name="ethiopic letters" colorInfoName="lowercase">
<node label="ሀ">
<!--Old Char Color: 70-->
<!--Note: ETHIOPIC SYLLABLE HA -->
Expand Down
2 changes: 1 addition & 1 deletion Data/alphabets/autoConverted/alphabet.ar.brezhoneg.breton.with.punctuation.and.numerals.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,7 +143,7 @@
<textCharAction />
</node>
</group>
<group name="upper case letters" colorInfoName="upper case letters">
<group name="upper case letters" colorInfoName="uppercase">
<!--Old Group Color: 111-->
<node label="A">
<!--Note: A -->
Expand Down
10 changes: 5 additions & 5 deletions Data/alphabets/autoConverted/alphabet.assamese..with.punctuation.and.numerals.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
The same script is used by Assamese.
-->
<group name="bengali letters">
<group name="bengali vowels" colorInfoName="bengali vowels">
<group name="bengali vowels" colorInfoName="lowercase">
<!--Old Group Color: 0-->
<node label="অ">
<!--Note: LETTER A -->
Expand Down Expand Up @@ -65,7 +65,7 @@
<textCharAction />
</node>
</group>
<group name="bengali consonants" colorInfoName="bengali consonants">
<group name="bengali consonants" colorInfoName="lowercase">
<!--Old Group Color: 0-->
<node label="ক">
<!--Note: LETTER KA -->
Expand Down Expand Up @@ -221,7 +221,7 @@
</node>
</group>
</group>
<group name="Bengali combining vowels" colorInfoName="bengali combining vowels">
<group name="Bengali combining vowels" colorInfoName="accents">
<!--Old Group Color: 52-->
<node label="◌া">
<!--Note: VOWEL SIGN AA -->
Expand Down Expand Up @@ -284,7 +284,7 @@
<textCharAction unicode="2519" />
</node>
</group>
<group name="Bengali numbers" colorInfoName="bengali numbers">
<group name="Bengali numbers" colorInfoName="numbers">
<!--Old Group Color: 113-->
<node label="১">
<!--Note: BENGALI DIGIT ONE-->
Expand Down Expand Up @@ -362,7 +362,7 @@
</group>
<group name="Punctuation" colorInfoName="punctuation">
<!--Old Group Color: 112-->
<group name="Bengali Punctuation" colorInfoName="bengali punctuation">
<group name="Bengali Punctuation" colorInfoName="punctuation">
<!--Old Group Color: 113-->
<node label="৲">
<!--Note: RUPEE MARK-->
Expand Down
2 changes: 1 addition & 1 deletion ...ts/autoConverted/alphabet.azrbaycanca.azeri.azerbaijani.with.punctuation.and.numerals.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@
<textCharAction />
</node>
</group>
<group name="upper case letters" colorInfoName="upper case letters">
<group name="upper case letters" colorInfoName="uppercase">
<!--Old Group Color: 111-->
<node label="A">
<textCharAction />
Expand Down
2 changes: 1 addition & 1 deletion ...bets/autoConverted/alphabet.bahasa.indonesia.indonesian.with.numerals.and.punctuation.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@
<textCharAction />
</node>
</group>
<group name="Upper case Latin letters" colorInfoName="upper case latin letters">
<group name="Upper case Latin letters" colorInfoName="uppercase">
<!--Old Group Color: 111-->
<node label="A">
<textCharAction />
Expand Down
2 changes: 1 addition & 1 deletion ...ets/autoConverted/alphabet.bahasa.melayu.malay.alphabet.with.numerals.and.punctuation.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@
<textCharAction />
</node>
</group>
<group name="Upper case Latin letters" colorInfoName="upper case latin letters">
<group name="Upper case Latin letters" colorInfoName="uppercase">
<!--Old Group Color: 111-->
<node label="A">
<textCharAction />
Expand Down
2 changes: 1 addition & 1 deletion Data/alphabets/autoConverted/alphabet.belarusian.with.punctuation.and.numerals.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -351,7 +351,7 @@
<textCharAction />
</node>
</group>
<group name="upper case letters" colorInfoName="upper case letters">
<group name="upper case letters" colorInfoName="uppercase">
<!--Old Group Color: 111-->
<node label="А">
<textCharAction />
Expand Down
2 changes: 1 addition & 1 deletion Data/alphabets/autoConverted/alphabet.bosanski.bosnian.with.punctuation.and.numerals.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,7 @@
<textCharAction />
</node>
</group>
<group name="upper case letters" colorInfoName="upper case letters">
<group name="upper case letters" colorInfoName="uppercase">
<!--Old Group Color: 111-->
<node label="A">
<!--Note: A -->
Expand Down
2 changes: 1 addition & 1 deletion Data/alphabets/autoConverted/alphabet.bulgarian..with.punctuation.and.numerals.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@
<textCharAction />
</node>
</group>
<group name="upper case letters" colorInfoName="upper case letters">
<group name="upper case letters" colorInfoName="uppercase">
<!--Old Group Color: 111-->
<node label="А">
<textCharAction />
Expand Down
2 changes: 1 addition & 1 deletion Data/alphabets/autoConverted/alphabet.burmese.myanmar.with.punctuation.and.numerals.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -191,7 +191,7 @@
<textCharAction />
</node>
</group>
<group name="Vowels Myanmar" colorInfoName="vowels myanmar">
<group name="Vowels Myanmar" colorInfoName="lowercase">
<!--Old Group Color: 111-->
<node label="◌ာ">
<!--Note: VOWEL SIGN AA -->
Expand Down
6 changes: 3 additions & 3 deletions .../autoConverted/alphabet.catal.catalan.combining.accents.with.punctuation.and.numerals.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE alphabet SYSTEM "../alphabet.dtd">
<alphabet name="Català / Catalan (combining accents) with punctuation and numerals" orientation="LR" trainingFilename="training_catalanC_ES.txt" colorsName="European/Asian">
<group name="Combining accents" colorInfoName="combining accents">
<group name="Combining accents" colorInfoName="accents">
<node label="◌̀">
<!--Old Char Color: 72-->
<!--Note: COMBINING GRAVE ACCENT ` -->
Expand Down Expand Up @@ -38,7 +38,7 @@
<textCharAction unicode="776" />
</node>
</group>
<group name="Lower case Latin letters" colorInfoName="lower case latin letters">
<group name="Lower case Latin letters" colorInfoName="lowercase">
<node label="a">
<!--Old Char Color: 10-->
<textCharAction />
Expand Down Expand Up @@ -144,7 +144,7 @@
<textCharAction />
</node>
</group>
<group name="Upper case Latin letters" colorInfoName="upper case latin letters">
<group name="Upper case Latin letters" colorInfoName="uppercase">
<!--Old Group Color: 111-->
<node label="A">
<!--Old Char Color: 10-->
Expand Down
2 changes: 1 addition & 1 deletion Data/alphabets/autoConverted/alphabet.catal.catalan.with.punctuation.and.numerals.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -147,7 +147,7 @@
<textCharAction />
</node>
</group>
<group name="upper case letters" colorInfoName="upper case letters">
<group name="upper case letters" colorInfoName="uppercase">
<!--Old Group Color: 111-->
<node label="A">
<!--Note: A -->
Expand Down
24 changes: 12 additions & 12 deletions Data/alphabets/autoConverted/alphabet.cherokee.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
<textCharAction />
</node>
</group>
<group name="GA" colorInfoName="ga">
<group name="GA" colorInfoName="lowercase">
<!--Old Group Color: 191-->
<node label="Ꭶ">
<!--Note: CHEROKEE LETTER GA -->
Expand Down Expand Up @@ -58,7 +58,7 @@
<textCharAction />
</node>
</group>
<group name="HA" colorInfoName="ha">
<group name="HA" colorInfoName="lowercase">
<!--Old Group Color: 192-->
<node label="Ꭽ">
<!--Note: CHEROKEE LETTER HA -->
Expand All @@ -85,7 +85,7 @@
<textCharAction />
</node>
</group>
<group name="LA" colorInfoName="la">
<group name="LA" colorInfoName="lowercase">
<!--Old Group Color: 193-->
<node label="Ꮃ">
<!--Note: CHEROKEE LETTER LA -->
Expand All @@ -112,7 +112,7 @@
<textCharAction />
</node>
</group>
<group name="MA" colorInfoName="ma">
<group name="MA" colorInfoName="lowercase">
<!--Old Group Color: 194-->
<node label="Ꮉ">
<!--Note: CHEROKEE LETTER MA -->
Expand All @@ -135,7 +135,7 @@
<textCharAction />
</node>
</group>
<group name="NA" colorInfoName="na">
<group name="NA" colorInfoName="lowercase">
<!--Old Group Color: 195-->
<node label="Ꮎ">
<!--Note: CHEROKEE LETTER NA -->
Expand Down Expand Up @@ -170,7 +170,7 @@
<textCharAction />
</node>
</group>
<group name="QUA" colorInfoName="qua">
<group name="QUA" colorInfoName="lowercase">
<!--Old Group Color: 196-->
<node label="Ꮖ">
<!--Note: CHEROKEE LETTER QUA -->
Expand All @@ -197,7 +197,7 @@
<textCharAction />
</node>
</group>
<group name="SA" colorInfoName="sa">
<group name="SA" colorInfoName="lowercase">
<!--Old Group Color: 197-->
<node label="Ꮜ">
<!--Note: CHEROKEE LETTER SA -->
Expand Down Expand Up @@ -228,7 +228,7 @@
<textCharAction />
</node>
</group>
<group name="DA" colorInfoName="da">
<group name="DA" colorInfoName="lowercase">
<!--Old Group Color: 198-->
<node label="Ꮣ">
<!--Note: CHEROKEE LETTER DA -->
Expand Down Expand Up @@ -267,7 +267,7 @@
<textCharAction />
</node>
</group>
<group name="DL" colorInfoName="dl">
<group name="DL" colorInfoName="lowercase">
<!--Old Group Color: 199-->
<node label="Ꮬ">
<!--Note: CHEROKEE LETTER DLA -->
Expand Down Expand Up @@ -298,7 +298,7 @@
<textCharAction />
</node>
</group>
<group name="TS" colorInfoName="ts">
<group name="TS" colorInfoName="lowercase">
<!--Old Group Color: 190-->
<node label="Ꮳ">
<!--Note: CHEROKEE LETTER TSA -->
Expand All @@ -325,7 +325,7 @@
<textCharAction />
</node>
</group>
<group name="WA" colorInfoName="wa">
<group name="WA" colorInfoName="lowercase">
<!--Old Group Color: 191-->
<node label="Ꮹ">
<!--Note: CHEROKEE LETTER WA -->
Expand All @@ -352,7 +352,7 @@
<textCharAction />
</node>
</group>
<group name="YA" colorInfoName="ya">
<group name="YA" colorInfoName="lowercase">
<!--Old Group Color: 192-->
<node label="Ꮿ">
<!--Note: CHEROKEE LETTER YA -->
Expand Down
4 changes: 2 additions & 2 deletions ...ets/autoConverted/alphabet.chinese.pin.yin.with.combining.tones.no.chinese.characters.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<!-- <s d="&#x2CA;" t="&#x2CA;" /> 2 -->
<!-- <s d="&#x2C7;" t="&#x2C7;" /> 3 -->
<!-- <s d="&#x2CB;" t="&#x2CB;" /> 4 -->
<group name="pin yin roman letters" colorInfoName="pin yin roman letters">
<group name="pin yin roman letters" colorInfoName="lowercase">
<node label="a">
<!--Old Char Color: 10-->
<textCharAction />
Expand Down Expand Up @@ -121,7 +121,7 @@
<textCharAction />
</node>
</group>
<group name="combining diacritics" colorInfoName="combining diacritics">
<group name="combining diacritics" colorInfoName="accents">
<!--Old Group Color: 110-->
<node label="◌̄">
<!--Old Char Color: 90-->
Expand Down
Loading
Loading