Skip to content

Make RapidFuzzMatcher the default fuzzy matcher - #62

Merged
mcarans merged 2 commits into
mainfrom
rapidfuzz-matcher
Sep 29, 2026
Merged

mcarans merged 2 commits into
mainfrom
rapidfuzz-matcher

Conversation

@mcarans

@mcarans mcarans commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

Summary

  • RapidFuzzMatcher (new, matching.py): fuzzy matches names with rapidfuzz, using the same match() interface as Phonetics.

    • default_scorer: token_set_ratio, or ratio on the names with spaces and punctuation removed when that is at least 90 (for "Dawlatabad" vs "Dawlat Abad").
    • place_name_scorer: as default_scorer, but scores 0 when names share a word and differ in another that isn't a spelling variant (eg. "central kalimantan" vs "kalimantan utara").
    • Ties are broken by ratio, so "north west" picks "North West" over "West".
  • get_code_from_name defaults to RapidFuzzMatcher. Pass matcher=Phonetics() for the previous behaviour.

  • resolve_name_parts (new): a name split by , ; / : & newline "and" "et", or a hyphen, is resolved from its parts:

    • one part is a given qualifier, eg. the parent admin name in "Falcón, Acosta";
    • one part's code is the other's plus "-", eg. "Protection - GBV" → PRO-GBV.

    A list of names (3+ parts, or 2 parts with different codes) is not fuzzy matched.

  • rapidfuzz is now a required dependency. pyphonetics stays, since Phonetics is still available.

  • Docs: breaking change note and matching examples in documentation/index.md. The note says "From 4.2.0", which needs confirming.

Why

pyphonetics has had no release or activity in 12+ months. A comparison over admin and lookup names collected from the repos that use this code (2,762 admin names with an answer independent of either matcher, from test asserts and provider p-codes; pipeline mode):

Phonetics RapidFuzzMatcher
Right 2,696 2,703
Wrong p-code 36 35
No match 30 24
  • Most names resolve exactly, so the matchers differ on only ~18 names. Excluding Mali, whose expected p-codes look shifted, Phonetics alone is right on 1 (CMR "Sud-Ouest") and RapidFuzzMatcher alone on 9 (eg. VEN "Monagas, Santa Bárbara" → Santa Babara, HTI "Nord Ouest" → North-West where Phonetics gives North-East).
  • Phonetics also makes unrelated matches in names without independent answers, some in published output, eg. UKR "South" → Kyiv, THA "North" → Nan, KEN "Coast" → Kisii.
  • Part resolution resolves ~300 VEN "State, Municipality" names exactly for either matcher.

Impact

Matching results change for downstream users of get_code_from_name and of hdx-python-country's AdminLevel, which will get the same default in a follow-up PR once this is released. Known RapidFuzzMatcher limits:

  • A name with one extra word still matches its shorter version ("North Maluku" → Maluku, "Kenge city" → Kenge).
  • Composite names like COL "Sabanalarga (Atlántico)" go to the bare municipality.

Testing

  • tests/hdx/utilities/test_matching.py: 19 passed.
  • Full suite: 2 failures and 1 collection error also present on main (test_downloader::test_download size assert, test_email and test_html without the optional extras).
  • hdx-python-country's 36 tests pass against this branch with its matching changes.

Add RapidFuzzMatcher, which scores names with rapidfuzz, and make it the
default for get_code_from_name. Phonetics remains available via the
matcher parameter. RapidFuzzMatcher.place_name_scorer additionally rejects
names that share a word but differ in another (eg. "central kalimantan" vs
"kalimantan utara").

Add resolve_name_parts: a two-part name resolves when one part qualifies
the other (a parent name, or a sub-code such as "Protection - GBV"), and a
list of names is not fuzzy matched.

rapidfuzz is now a required dependency.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

Test Results

137 tests  +10   136 ✅ +9   11s ⏱️ +3s
  1 suites ± 0     0 💤 ±0 
  1 files   ± 0     1 ❌ +1 

For more details on these failures, see this check.

Results for commit 1b87283. ± Comparison against base commit 12b8d11.

@github-actions

github-actions Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

Test Results

137 tests  +10   137 ✅ +10   11s ⏱️ +3s
  1 suites ± 0     0 💤 ± 0 
  1 files   ± 0     0 ❌ ± 0 

Results for commit 8c35b11. ± Comparison against base commit 12b8d11.

♻️ This comment has been updated with latest results.

@coveralls

Copy link
Copy Markdown

Coverage Status

coverage: 95.796% (+0.2%) from 95.643% — rapidfuzz-matcher into main

@mcarans
mcarans merged commit 020f49a into main Sep 29, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants