Skip to content

test: word_tokenize keeps a fraction and a signed number whole; CI runs pytest - #21

Merged
JarbasAl merged 1 commit into
devfrom
fix/word-tokenize-fraction-contract-and-pytest-ci
Sep 19, 2026
Merged

JarbasAl merged 1 commit into
devfrom
fix/word-tokenize-fraction-contract-and-pytest-ci

Conversation

@openvoiceos-bot

Copy link
Copy Markdown

Problem

test/test_chunking.py::TestChunking::test_word fails on dev (c57316f, and at c8e1e10 before #18, on 3.11 and 3.14): ['2', 'and', '3/4'] != ['2', 'and', '3', '/', '4']. Nobody saw it because build_tests.yml builds and installs the wheel and runs no pytest, and license_tests.yml runs the licence file alone. Evidence: knowledge/wiki/audits/gate-ledger/merged-run-quebra_frases-20.md.

Which side is right

The test. Two assertions written in 2021 (3988f62, e461391) predate two merged tokenizer changes:

  • extend tokenizer regex #1 (783a0b6, 2023) added / to the in-number separators of _WORD_REGEX, (?:\d+(?:[,.:\/]\d*)+), with date: 11/11/2023 11/11 as its stated test string. A digit run joined by / is one token, the same as 1.5 and 13:00.
  • fix negative float #3 (720185e, "fix negative float", 2023) added -? in front of it, so -2 and -2.5 are one token.

Both were merged and neither touched the test. The consumer agrees with the code: ovos-number-parser reads a fraction as one 3/4 token and splits it on / itself (look_for_fractions(split_list), "list created by splitting on '/'"). Reverting the code instead would split 11/11/2023 into five tokens and -2.5 into two.

Change

  • test_word states the contract: 2 and 3/4 is ['2', 'and', '3/4'], on 11/11/2023 at 13:00 keeps the date and the time whole, -2 and -2.5 are one token. Counter-cases beside them: a / between spaces or between letters, and a - set apart by a space, are their own tokens.
  • build_tests.yml calls OpenVoiceOS/gh-automations/.github/workflows/build-tests.yml@dev with test_path: 'test/test_chunking.py test/test_tokenization.py' on 3.10 to 3.14, on push and pull request to dev. license_tests.py stays with its own workflow, which installs lichecker.

Checks

  • Fail before: at dev the same file gives 1 failed, 21 passed (this run and the ledger's).
  • After: python -m pytest test/test_chunking.py test/test_tokenization.py, 22 passed, Python 3.11.
  • The caller YAML parses; the two inputs it names exist on the shared workflow.

This PR was written by an AI agent (Claude) and is not reviewed by a human yet.

🤖 Generated with Claude Code

…ns pytest

test_chunking::test_word has failed since 2023 and no workflow ran it.
Two assertions from 2021 predate two merged tokenizer changes: #1 made
a digit run joined by "/" one token, the same as "1.5" and "13:00", so
"3/4" and "11/11/2023" reach a number parser whole; #3 made a leading
"-" the sign of a negative number, so "-2" and "-2.5" are one token.
ovos-number-parser reads a fraction as one "3/4" token and splits it on
"/" itself. The test now states that contract, with a date, a spaced
"/" and a spaced "-" as the counter-cases.

build_tests.yml calls the gh-automations shared workflow with test_path
set to the two unit files, on 3.10 to 3.14, so the suite runs on every
push and pull request to dev. license_tests.py stays with its own
workflow, which installs lichecker.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Sep 19, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: eb027a13-bfaa-47fe-af24-abc057823c99

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@JarbasAl
JarbasAl marked this pull request as ready for review September 19, 2026 01:10
@github-actions

Copy link
Copy Markdown

Beep! Automated checks have reached 100% completion. 🔋

I've aggregated the results of the automated checks for this PR below.

🔨 Build Tests

The build pipeline has finished its work. 🏁

✅ All versions pass

Python Build Install Tests pytest
3.10 ✅ ✅ ✅ 22 passed in 0.07s
3.11 ✅ ✅ ✅ 22 passed in 0.09s
3.12 ✅ ✅ ✅ 22 passed in 0.07s
3.13 ✅ ✅ ✅ 22 passed in 0.07s
3.14 ✅ ✅ ✅ 22 passed in 0.11s

Catch you at the next merge! 🌊

@JarbasAl
JarbasAl merged commit e9218fb into dev Sep 19, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants