Repository navigation
test: word_tokenize keeps a fraction and a signed number whole; CI runs pytest - #21
Conversation
…ns pytest test_chunking::test_word has failed since 2023 and no workflow ran it. Two assertions from 2021 predate two merged tokenizer changes: #1 made a digit run joined by "/" one token, the same as "1.5" and "13:00", so "3/4" and "11/11/2023" reach a number parser whole; #3 made a leading "-" the sign of a negative number, so "-2" and "-2.5" are one token. ovos-number-parser reads a fraction as one "3/4" token and splits it on "/" itself. The test now states that contract, with a date, a spaced "/" and a spaced "-" as the counter-cases. build_tests.yml calls the gh-automations shared workflow with test_path set to the two unit files, on 3.10 to 3.14, so the suite runs on every push and pull request to dev. license_tests.py stays with its own workflow, which installs lichecker. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Beep! Automated checks have reached 100% completion. 🔋I've aggregated the results of the automated checks for this PR below. 🔨 Build TestsThe build pipeline has finished its work. 🏁 ✅ All versions pass
Catch you at the next merge! 🌊 |
Problem
test/test_chunking.py::TestChunking::test_wordfails on dev (c57316f, and at c8e1e10 before #18, on 3.11 and 3.14):['2', 'and', '3/4'] != ['2', 'and', '3', '/', '4']. Nobody saw it becausebuild_tests.ymlbuilds and installs the wheel and runs no pytest, andlicense_tests.ymlruns the licence file alone. Evidence:knowledge/wiki/audits/gate-ledger/merged-run-quebra_frases-20.md.Which side is right
The test. Two assertions written in 2021 (
3988f62,e461391) predate two merged tokenizer changes:783a0b6, 2023) added/to the in-number separators of_WORD_REGEX,(?:\d+(?:[,.:\/]\d*)+), withdate: 11/11/2023 11/11as its stated test string. A digit run joined by/is one token, the same as1.5and13:00.720185e, "fix negative float", 2023) added-?in front of it, so-2and-2.5are one token.Both were merged and neither touched the test. The consumer agrees with the code:
ovos-number-parserreads a fraction as one3/4token and splits it on/itself (look_for_fractions(split_list), "list created by splitting on '/'"). Reverting the code instead would split11/11/2023into five tokens and-2.5into two.Change
test_wordstates the contract:2 and 3/4is['2', 'and', '3/4'],on 11/11/2023 at 13:00keeps the date and the time whole,-2and-2.5are one token. Counter-cases beside them: a/between spaces or between letters, and a-set apart by a space, are their own tokens.build_tests.ymlcallsOpenVoiceOS/gh-automations/.github/workflows/build-tests.yml@devwithtest_path: 'test/test_chunking.py test/test_tokenization.py'on 3.10 to 3.14, on push and pull request to dev.license_tests.pystays with its own workflow, which installs lichecker.Checks
1 failed, 21 passed(this run and the ledger's).python -m pytest test/test_chunking.py test/test_tokenization.py,22 passed, Python 3.11.This PR was written by an AI agent (Claude) and is not reviewed by a human yet.
🤖 Generated with Claude Code