Repository navigation
Remove tretspef grouped - #292
tomjemmett wants to merge 3 commits into
Conversation
There was a problem hiding this comment.
🔵 Needs a closer look
It changes the data pipeline output schema (removing a column) and alters tretspef semantics with cross-repository downstream dependencies (nhp_inputs #618), which warrants human verification despite the code itself being clean and consistent.
No open issues 🎉
What changed in this PR
This PR removes the tretspef_grouped treatment-function grouping from the NHP data pipeline so that specialty-level parameters can be set more granularly downstream (closes #296, see also nhp_inputs #618). The old add_tretspef_grouped_column helper (which collapsed specialty codes into ~18 buckets plus "Other (Surgical)"/"Other (Medical)"/"Other") is deleted, and a new replace_invalid_tretspef_values helper normalizes any tretspef code not present in the reference_tretspef_type table to "Other". All producers and consumers of tretspef_grouped are updated to use the raw tretspef column instead.
Changes:
- Replace
add_tretspef_grouped_columnwithreplace_invalid_tretspef_values, applied to inpatients and outpatients raw data. - Drop
tretspef_groupedfrom all raw/aggregated selects, inputs renames, A&E baseline, maternity remapping, and the synthetic-data filter. - A&E baseline now groups by
tretspef(value"Other") rather than emitting a nulltretspef, consistent with A&E expat/repat.
| File | Description |
|---|---|
| src/nhp/data/raw_data/helpers.py | Removes add_tretspef_grouped_column; adds replace_invalid_tretspef_values validating against reference_tretspef_type. |
| src/nhp/data/raw_data/inpatients.py | Swaps grouped-column call for invalid-value replacement; drops tretspef_grouped from final select. |
| src/nhp/data/raw_data/outpatients.py | Same swap and select cleanup as inpatients. |
| src/nhp/data/raw_data/ecds.py | Drops the hardcoded tretspef_grouped = "Other" column. |
| src/nhp/data/raw_data/aae.py | Drops the hardcoded tretspef_grouped = "Other" column. |
| src/nhp/data/aggregated_data/outpatients.py | Removes tretspef_grouped from the aggregated select. |
| src/nhp/data/aggregated_data/ecds.py | Removes tretspef_grouped from the aggregated select. |
| src/nhp/data/inputs_data/ip/__init__.py | Stops dropping tretspef / renaming tretspef_grouped. |
| src/nhp/data/inputs_data/op/__init__.py | Stops dropping tretspef / renaming tretspef_grouped. |
| src/nhp/data/inputs_data/ip/baseline.py | Removes maternity→"Other (Medical)" remapping. |
| src/nhp/data/inputs_data/ip/expat_repat.py | Removes maternity→"Other (Medical)" remapping in two functions. |
| src/nhp/data/inputs_data/ae/baseline.py | Groups by tretspef instead of nulling it. |
| src/nhp/data/model_data/generate_synthetic_data.py | Removes the obsolete tretspef_grouped startswith filter. |
Review effort: Balanced
closes #296