[goldyfruit]

The bug

word_tokenize drops Unicode combining marks (category Mn/Mc) and splits the word at each one. Thai, Hindi, Bengali, Tamil and every other abugida write their vowel signs, viramas and tone marks this way, so their words come back in pieces.

The cause is the word alternative of _WORD_REGEX in quebra_frases/tokenization.py (line 5 on dev):

_WORD_REGEX = re.compile(r"(?:-?\d+(?:[,.:\/]\d*)+)|\b\p{L}*(?:\.\p{L}+)+\.|[\p{L}\p{N}'-]+|[.,;:_!?<>|()=\[\]{}»«*~^`%\/\\\+#]", re.I)

[\p{L}\p{N}'-]+ has no \p{M}, so U+0E34 THAI CHARACTER SARA I, U+0E48 MAI EK, U+093F DEVANAGARI VOWEL SIGN I, U+094D VIRAMA and the like match none of the alternatives. finditer skips them and the word ends there.

It shows up in ovos-core because the default ovos-utterance-normalizer transformer tokenizes with quebra_frases.word_tokenize and joins the tokens back with spaces:

input	word_tokenize before	normalised utterance before	after
เพิ่มความสว่างไฟ	['เพ', 'มความสว', 'างไฟ']	เพ มความสว างไฟ	เพิ่มความสว่างไฟ
नमस्ते	['नमस', 'त']	नमस त	नमस्ते
बत्ती की चमक बढ़ाओ	['बत', 'त', 'क', 'चमक', 'बढ', 'ओ']	बत त क चमक बढ ओ	बत्ती की चमक बढ़ाओ
আলো বাড়াও	['আল', 'ব', 'ড', 'ও']	আল ব ড ও	আলো বাড়াও
விளக்கை அணை	['வ', 'ளக', 'க', 'அண']	வ ளக க அண	விளக்கை அணை

Intent matching on these languages fails because of this.

The fix

Add \p{M} next to \p{L} in the word class, and in the abbreviation alternative so the two stay consistent. Punctuation is still split off: เพิ่มความสว่างไฟ! becomes ['เพิ่มความสว่างไฟ', '!']. Latin text written in NFD (cafe followed by U+0301) also keeps its accent on the letter now.

Tests

New test_word_combining_marks in test/test_chunking.py covers Thai (with and without !, ,, ?), Hindi, Bengali, Tamil, an NFD Latin accent, and span_indexed_word_tokenize offsets on Devanagari. It fails on dev without the fix (['เพ', 'มความสว', 'างไฟ'] != ['เพิ่มความสว่างไฟ']). pytest test/ passes with it: 23 passed.

I also ran it end to end with ovos-utterance-normalizer 0.2.5a2 against this branch. The five sentences above come out unchanged, and turn on the lights, please! still normalises to turn on the lights , please.

🤖 Generated with Claude Code

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

• Configuration used: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: fcd20127-8e7b-4d74-8571-c682d51bb4db

---

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

---

_{Comment @coderabbitai help to get the list of available commands.}

[goldyfruit]

Reopening: this was closed by mistake. A PR in one of our own repositories referenced it with a closing keyword, and merging that PR closed this one. The fix here (keeping combining marks inside words in _WORD_REGEX) still stands and is ready for review.