Repository navigation
move demographic/birth factors into inputs data pipeline - #293
Open
tomjemmett wants to merge 2 commits into
Open
tomjemmett wants to merge 2 commits into
tomjemmett wants to merge 2 commits into
Conversation
* creates the population data as tables in the reference layer * extracts these tables in the inputs-data layer * removes these extracts from the model data layer
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The mlcsu config uses the non-existent unhp catalog for the two new reference tables, which would break table resolution in that environment.
2 open issues
What changed in this PR
This PR relocates the demographic-factors and birth-factors population data out of the model_data layer and into the inputs-data pipeline, backed by new reference-layer tables. New reference tables (reference_population_provider_demographics / _births) are created once in the reference layer, extracted into the inputs-data layer, and the old per-run model_data extracts (and their shared helpers) are removed. It also bundles some stacked refactors: a parameterized get_spark(PartitionOverwriteMode), dropping the unused spark arg from a couple of aggregated-data generators, and renaming the IP efficiencies TPMA extract (plus fixing its type filter to "efficiency").
Changes:
- Add
reference/population_provider.pyto build provider demographic/birth factor tables, andinputs_data/demographics.pyto extract them; removemodel_data/birth_factors.py,demographic_factors.py, the related helper functions, and synthetic-data generators. - Point
generate_provider_gamsat the new inputs path and inline_create_icb_population_projections; wire new table names and workflow tasks/entry points. - Parameterize
get_sparkwithPartitionOverwriteMode(default DYNAMIC) and use STATIC in raw/aggregated extracts; renameip_efficiencyTPMAs toip_efficienciesand fix the mitigatortypefilter.
| File | Description |
|---|---|
| src/nhp/data/reference/population_provider.py | New reference-layer builder for provider demographic/birth factor tables; main() docstring is copied from an unrelated function |
| src/nhp/data/inputs_data/demographics.py | New inputs-data extract for demographic/birth factors and provider catchments |
| src/nhp/data/table_names/table_names.py | Adds the two new reference table-name fields |
| src/nhp/data/table_names/udal.py | UDAL mappings for the new reference tables |
| src/nhp/data/table_names/mlcsu.py | MLCSU mappings; catalog typo unhp instead of nhp |
| src/nhp/data/get_spark.py | Adds PartitionOverwriteMode enum and parameterizes the partition overwrite mode |
| src/nhp/data/raw_data/{aae,inpatients,outpatients,outpatients_mitigators}.py | Switch main() to get_spark(PartitionOverwriteMode.STATIC) |
| src/nhp/data/aggregated_data/{outpatients,ecds}.py | Use STATIC mode and drop unused spark arg from generators |
| src/nhp/data/model_data/ip_tpmas.py | Rename to ip_efficiencies and fix TPMA type filter to "efficiency" |
| src/nhp/data/model_data/helpers.py | Remove the now-relocated population-projection helpers |
| src/nhp/data/model_data/health_status_adjustment/generate_provider_gams.py | Read demographics from the inputs path; adjust the base-year filter |
| src/nhp/data/model_data/health_status_adjustment/generate_icb_gams.py | Inline the ICB population-projection helper |
| src/nhp/data/model_data/generate_synthetic_data.py | Remove synthetic birth/demographic factor generation |
| src/nhp/data/model_data/{birth_factors,demographic_factors}.py | Deleted (moved to reference/inputs layers) |
| pyproject.toml | Add/remove the corresponding entry points |
| databricks_workflows/*.yaml | Wire new reference/inputs tasks and remove old model-data tasks |
Review effort: Balanced
tomjemmett
force-pushed
the
move_pop_extract_to_inputs
branch
from
October 21, 2026 14:24
41349ba to
52ecc2d
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack created with GitHub Stacks CLI • Give Feedback 💬