Source-side header abbreviation expansion (Issue #87)

[Li-ChangC]

Summary

• Expands closed-set clinical/oncology abbreviations in source column headers (e.g. AGE_AT_DX → age at diagnosis, OS_MONTHS → overall survival months) before the exact/fuzzy/embedding matching cascade, so surface-form abbreviations resolve to the correct standard field.
• Implemented as double-matching: each name-based matcher (stage1_matchers, stage3_matchers) tries both the raw normalized header and its abbreviation-expanded form (schema_mapper_utils.header_variants), keeping the best score per field with the raw form winning ties. This guarantees an exact raw match (e.g. gene symbols like HER2) is never displaced by a bad expansion — expansion can only help, never regress an already-correct match.
• The alias generator does not currently use a true closed abbreviation set. Its prompt contains example mappings, allows the LLM to generate additional abbreviations, and includes ambiguous terms such as pt, ca, and rt. Therefore, this change temporarily uses a separately curated abbrev,expansion dictionary with conservative mappings. Once the alias vocabulary and provenance are confirmed, both sides should be updated to use one shared, model-neutral abbreviation file.

Design notes

• Singular/plural normalization was intentionally not added: many header tokens legitimately end in s (stripping is error-prone), and a true plural that misses exact/fuzzy is still caught by the stage-3 embedding matcher, which is largely number-insensitive.
• Word-order canonicalization was intentionally not added: the fuzzy stage already uses token_sort_ratio, which is word-order-insensitive; extending that to exact/embedding matching would require sorting both the query and the candidate index, which is a larger change for low marginal value given real header word-order mismatches are rare.
• Candidate-side (schema/alias) expansion was intentionally not added — this PR only expands the source-header (query) side. The double-match/keep-best design already prevents query-side expansion from regressing matches.

Test plan

• pytest tests/ — full suite passes (389 passed), including new tests/test_header_normalizer.py covering header_variants/expand_abbreviations and the double-match invariants (raw exact match never dropped, expansion can recover a miss, raw wins ties).
• Verified _load_abbrev_map degrades to {} with a WARNING (not a crash) when CLINICAL_ABBREV_PATH is None.
• A/B validated

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Adds query-side (source header) abbreviation expansion to improve schema mapping accuracy for clinical/oncology datasets by generating normalized header variants (raw + expanded) and double-matching across exact, fuzzy, and embedding matchers, while bundling a closed abbreviation dictionary into the package.

Changes:

• Introduces expand_abbreviations() and header_variants() with a lazily loaded, cached abbreviation dictionary.
• Updates stage-1 (exact/fuzzy) and stage-3 (embedding) matchers to evaluate both raw and expanded header variants and keep best-per-field scores (raw-first tie behavior).
• Adds unit tests for expansion/variants + matcher invariants, and bundles the abbreviation CSV via package data.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/test_header_normalizer.py	New tests covering abbreviation expansion, header variants behavior, loader degradation, and double-match invariants.
src/metaharmonizer/utils/schema_mapper_utils.py	Adds abbreviation loader + expansion + header variant generation utilities.
src/metaharmonizer/models/schema_mapper/matchers/stage1_matchers.py	Double-matching for exact/fuzzy name-based matchers using header_variants().
src/metaharmonizer/models/schema_mapper/matchers/stage3_matchers.py	Double-matching for embedding matchers using header_variants() and best-per-field merging.
src/metaharmonizer/models/schema_mapper/config.py	Resolves optional bundled abbreviation CSV path with safe fallback to None.
src/metaharmonizer/_bundled_data/schema/clinical_abbreviations_gpt-5-codex.csv	Adds bundled closed-set clinical abbreviation dictionary.
scripts/clinical_abbreviations_prompt.py	Adds the prompt used to generate the abbreviation CSV.
pyproject.toml	Includes the new abbreviation CSV in wheel package data.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.