Make RapidFuzzMatcher the default AdminLevel matcher - #92
Merged
Merged
Conversation
AdminLevel fuzzy matches with HDX Python Utilities' RapidFuzzMatcher and its place_name_scorer by default, and takes a matcher parameter so Phonetics can still be used. Before fuzzy matching, names made of parts are resolved with resolve_name_parts: a part that is the parent's name is dropped (eg. "Falcón, Acosta" with parent Falcón gives Acosta), and lists of names (eg. "Kampot & Kep") are not fuzzy matched. Parent names are read from the parent-level rows passed to setup_from_iterable. Fuzzy matching is now tried when parent is "" as well as None, as get_pcode already treated "" as no parent. test_adminlevel_parent's "city" names are replaced with names garbled with "qx", so the tests check replacement scoping rather than relying on Phonetics failing to match "Kenge city" to "Kenge". Requires HDX Python Utilities >= 4.2.0.
|
coverage: 95.769% (-0.04%) from 95.807% — rapidfuzz-matcher into main |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follows hdx-python-utilities#62, released in HDX Python Utilities 4.2.0.
AdminLevelfuzzy matches withRapidFuzzMatcher(RapidFuzzMatcher.place_name_scorer)instead ofPhonetics. A newmatcherparameter takes any matcher, eg.AdminLevel(config, matcher=Phonetics())for the previous behaviour.fuzzy_pcodecallsresolve_name_parts.setup_from_iterable(parent_pcode_to_name).fuzzy_pcodenow treatsparent=""as no parent, asget_pcodealready did. Before, it returned None, so names passed withparent=""were never fuzzy matched. hdx-python-pipelineutils'complete_adminspasses""for admin2 when no admin1 code is known, so IPC and HNO admin2 names are now fuzzy matched.test_adminlevel_parentused "Kenge city"-style names to test replacement scoping. They relied on Phonetics failing to match "Kenge city" to "Kenge", so they are now garbled with "qx", which only the scoped"qx": ""replacements can resolve.test_adminlevel_custom_matcherandtest_adminlevel_list_of_names.documentation/index.md. The example's removedsetup_from_admin_infois replaced withsetup_from_iterable. Requireshdx-python-utilities>=4.2.0.Impact
Results change for
AdminLevel.get_pcodecallers that fuzzy match. In a comparison over admin names collected from these repos:fuzzy_match=Falsefor admin2, oradmin_fuzzy_dontentries.admin_name_mappingsfor its 21 English-named Indonesian provinces.The "From 4.2.0" in the breaking change note assumes this is released as 4.2.0.
Testing
uv run pytest: 36 passed, against HDX Python Utilities 4.2.0 from PyPI.