Skip to content

Template existing and planned (ECAA) generators - #143

Open
EllieKallmier wants to merge 3 commits into
mainfrom
template-existing-planned-skeleton
Open

Template existing and planned (ECAA) generators#143
EllieKallmier wants to merge 3 commits into
mainfrom
template-existing-planned-skeleton

Conversation

@EllieKallmier

Copy link
Copy Markdown
Member

Adds generators_existing_planned to the new-format templater, and lays the
generator/storage split that storage_existing_planned will build on. A bit chunkier
than ideal prob but should be pretty similar to the new entrants so (hope) not too
much review work...

Everything comes from one IASR table —
existing_committed_anticipated_additional_generator_summary — which already lists one
row per real generating unit (DUID-level), generators and storage together. So the module
splits that table in two, then merges per-unit properties onto the generator side from six
further IASR tables. storage_existing_planned is left at just the split and column rename
here; property merges for that one coming up.

src/ispypsa/templater/
├── existing_planned.py      → new module: gen/storage split, PHES routing, property merges
├── mappings.py              → _GENERATORS_EXISTING_PLANNED_PROPERTY_MAP; key_col rename
├── helpers.py               → shared helpers consolidated here (see Refactors)
├── create_template.py       → wires the new table into the new-format branch
└── geography.py, transmission.py, new_entrants.py
                             → helpers moved out to helpers.py
src/ispypsa/iasr_table_caching/local_cache.py
                             → registers the seven newly-consumed IASR tables
tests/test_workbook_table_cache/7.5/
                             → frozen cache rebuilt with those tables

Things worth knowing before reading the diff

PHES routing can't rely on Technology Type alone. Wivenhoe and Shoalhaven are
labelled plain "Hydro" in the summary, but have real entries in the PHES properties table
(and the numbers cross-check — Wivenhoe's 3000 MWh / 285 MW ≈ the properties table's 10h).
Routing by technology string alone would send them to generators and silently never read
their PHES properties. So the split is a technology match union name-presence in the PHES
properties table -> noting it could just be one or the other

_validate_phes_routing then fails loud if any PHES station is missing from the summary
altogether — the only way to catch a future Wivenhoe-shaped surprise, since routing's own
presence check can't distinguish "absent" from "missed". Lower Tumut is the one
documented tolerated exception: it's Tumut 3's reversible pump-turbine subset (units 1-3),
see #131 comment thread.

Two things to note:

  • minimum_load for coal now comes from Typical Lowest Band. See #142
  • KiataWF1 vs KIATAWF1 is handled as a named-constant correction, not by fuzzy
    matching — fuzz.ratio scores it 50, well below any threshold that would be safe to apply
    across 642 IDs. It's the only ID mismatch in the whole set. More detail in comments.

Known gap - not fixed here 25 REZ-located rows use sub-zone ids (Q8a/Q8b/
Q8c) that renewable_energy_zones doesn't carry — only the parent Q8. Same class as
the existing #133 non-REZ placeholder gap, different cause, so it follows the same
precedent: named as an exception in the CLI-level geo_id coverage test rather than solved
here. Fix to come shortly - also flagging a future need to reassess when we move to v7.8
workbook which introduces a (differently-purposed/different coverage and impact) split up of
N9 -> N9a / N9b.

Refactors

Mostly consolidation, so both new-format modules draw on one set of helpers instead of
reaching sideways into each other:

  • Moved into helpers.py:
    • _set_geo_id (was duplicated in both modules),
    • the geo_id/granularity trio from geography.py,
    • _assert_table_valid and the property-map machinery from new_entrants.py, and
    • _build_geo_region_lookup from transmission.py
  • _map_geo_id_to_granularity now handles all three granularities and leaves REZ geo_ids
    untouched in every case, so that rule lives in one place rather than being reimplemented
    per module.
  • New entrant's property maps rename technology_colkey_col, so one merge helper
    serves both modules. The maps gain a numeric flag (default true) controlling whether a
    merged value is coerced with pd.to_numeric(errors="raise") — an input-side typo guard;
    only commissioning_date opts out.
    • There's a note-to-self in there that I'm happy to take opinions on re: float vs int
      dtype enforcement (whether to implement here or let validator apply) - my thought is
      towards implementing here to keep validator's scope tight but it's a small fix thing
      so just left simple as-is for the moment.
  • commissioning_date gets an explicit reformat: the parser emits ISO strings, the schema
    declares %d/%m/%Y.

…move some functions to helpers, pull in required tables
…tor shared helpers

Functions (and mappings) that were shared/very similar across new_entrants templater module are pulled out and gently refactored to become generic helpers.
@EllieKallmier EllieKallmier added type: feature New feature or request module: templater Covers contents of `templater` module labels Aug 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

module: templater Covers contents of `templater` module type: feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant