Template existing and planned (ECAA) generators - #143
Open
EllieKallmier wants to merge 3 commits into
Open
Conversation
…move some functions to helpers, pull in required tables
…tor shared helpers Functions (and mappings) that were shared/very similar across new_entrants templater module are pulled out and gently refactored to become generic helpers.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
generators_existing_plannedto the new-format templater, and lays thegenerator/storage split that
storage_existing_plannedwill build on. A bit chunkierthan ideal prob but should be pretty similar to the new entrants so (hope) not too
much review work...
Everything comes from one IASR table —
existing_committed_anticipated_additional_generator_summary— which already lists onerow per real generating unit (DUID-level), generators and storage together. So the module
splits that table in two, then merges per-unit properties onto the generator side from six
further IASR tables.
storage_existing_plannedis left at just the split and column renamehere; property merges for that one coming up.
Things worth knowing before reading the diff
PHES routing can't rely on
Technology Typealone. Wivenhoe and Shoalhaven arelabelled plain "Hydro" in the summary, but have real entries in the PHES properties table
(and the numbers cross-check — Wivenhoe's 3000 MWh / 285 MW ≈ the properties table's 10h).
Routing by technology string alone would send them to generators and silently never read
their PHES properties. So the split is a technology match union name-presence in the PHES
properties table -> noting it could just be one or the other
_validate_phes_routingthen fails loud if any PHES station is missing from the summaryaltogether — the only way to catch a future Wivenhoe-shaped surprise, since routing's own
presence check can't distinguish "absent" from "missed".
Lower Tumutis the onedocumented tolerated exception: it's Tumut 3's reversible pump-turbine subset (units 1-3),
see #131 comment thread.
Two things to note:
minimum_loadfor coal now comes from Typical Lowest Band. See #142KiataWF1vsKIATAWF1is handled as a named-constant correction, not by fuzzymatching —
fuzz.ratioscores it 50, well below any threshold that would be safe to applyacross 642 IDs. It's the only ID mismatch in the whole set. More detail in comments.
Known gap - not fixed here 25 REZ-located rows use sub-zone ids (
Q8a/Q8b/Q8c) thatrenewable_energy_zonesdoesn't carry — only the parentQ8. Same class asthe existing
#133non-REZ placeholder gap, different cause, so it follows the sameprecedent: named as an exception in the CLI-level geo_id coverage test rather than solved
here. Fix to come shortly - also flagging a future need to reassess when we move to v7.8
workbook which introduces a (differently-purposed/different coverage and impact) split up of
N9->N9a/N9b.Refactors
Mostly consolidation, so both new-format modules draw on one set of helpers instead of
reaching sideways into each other:
helpers.py:_set_geo_id(was duplicated in both modules),geography.py,_assert_table_validand the property-map machinery fromnew_entrants.py, and_build_geo_region_lookupfromtransmission.py_map_geo_id_to_granularitynow handles all three granularities and leaves REZ geo_idsuntouched in every case, so that rule lives in one place rather than being reimplemented
per module.
technology_col→key_col, so one merge helperserves both modules. The maps gain a
numericflag (default true) controlling whether amerged value is coerced with
pd.to_numeric(errors="raise")— an input-side typo guard;only
commissioning_dateopts out.dtype enforcement (whether to implement here or let validator apply) - my thought is
towards implementing here to keep validator's scope tight but it's a small fix thing
so just left simple as-is for the moment.
commissioning_dategets an explicit reformat: the parser emits ISO strings, the schemadeclares
%d/%m/%Y.