Repository navigation
Conversation
…pd_non_anticipativity) Two always-on derived frames in _derived_branch.py, consumed by nothing (recourse plan §6b Slice A; design specs/sliceA_lineage_frames_design.md r3): - dd_same_scenario (d, d_other): symmetric/reflexive pairs sharing a scenario leaf-path; domain = period_in_use only; all-pairs degenerate content for deterministic solves; leaf rule with the F6 authority convention (branch-local member displaces the anchor). - pd_non_anticipativity (d, b): period-level invest-NA pairs (strictly-pre-reveal anchor ties), parameterized on an optional branch_start frame (Slice E hook); provably empty for every currently-constructible solve under the min-anchor fallback. Wiring: unconditional assignment in apply_branch_cluster (step 2b), guarded-ctx input acquisition per the dtttdt_from_source null-guard precedent, ValueError failure semantics on inconsistent per-solve artefacts, Utf8-construct -> non-strict cast_dim -> live null assertion -> deterministic Utf8-key sort. FlexData registration (two typed fields, None only pre-cascade), d_other axis synonym. No consumers, no schema change, no migration, no autoscale entries (nothing enters the LP), no dump-side change (_copy_meta already glob-copies the rebuild inputs). Tests: T1-T12 per design §4/§5 (17 tests) — five exact shape tables, both mid-horizon shapes incl. the branch_start-driven [(p1, p1_low)] pair, symmetry property, metadata exclusion, failure semantics, _drop_master_rows pass-through, integration + enum hygiene on the stoch_two_period fixture, and the first stochastic dump/reload round-trip (pins the _copy_meta glob). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Slice B of the recourse-invest plan (specs/sliceB_walker_lineage_design.md). Add a keyword-only lineage=pl.DataFrame|None=None parameter to the shared period_walk_iterator, applied PRE-aggregation under the Slice A semi-join contract (remove (e,d,d_all) iff both ends in period_in_use and (d,d_all) not in lineage; history-anchored rows pass unconditionally). - _derived_walks: _apply_lineage_filter (anchor_col/dall_col params, fixed-direction lineage-side non-strict cast, mandatory .unique(), marker left-join) + _assert_lineage_castable null guard; zero-cost None path behind a single if. - Thread lineage=None through inventory #1-17 (cohort/variant/from_source chain in _derived_existing, _derived_npv, _derived_params) and apply the shared helper in the one non-walker pair-former #18 edd_divest_active_from_source (anchor_col="d_divest", dall_col="d"). - _derived_branch: check_recourse_npv_preconditions + assert_recourse_npv_preconditions — the three named NPV proof obligations (leaf-weight constancy, cohort sum-to-1, byte-equal fan-member year rows), shipped as a pure checker with the F4 soundness-scope docstring; added to __all__. Inert: every production call passes lineage=None; no flag, no consumer activation, no schema change, no autoscale entries, no LP change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Slice B test suite (specs/sliceB_walker_lineage_design.md §9). Capability harness (solver-free, hand-built frames + stub source): - W1 set-shape (history passes, cross-leaf both-ends-PIU pairs removed) - W2 factor-side PRE-aggregation proof (sums 7.0/3.0/4.0 vs unfiltered 7.0 — a post-aggregation filter cannot produce 3.0/4.0) - W3 lineage=None 12-row unfiltered pin - W4/W4b empty-frame contract (set: 4 history rows; factor: single (h2020,7.0) row, anchors absent not zeroed) - W5a duplicate-pair .unique() guard; W5b uncastable-token ValueError; W5c Enum x Enum missing-token raises / all-present identical Stochastic fixture pins (solver-required fixture, solver-free asserts): - W6 edd_invest_set today-bug witness ((wind,p2040,p2040_low) absent = erratum E1 first-stage-visibility break) - W7 edd/divest non-vacuity + repeat-determinism; NPV variant vacuity pins + invest-method overlay leg - P1/P2 obligation checker clean on stoch/deterministic workdirs - P3a-d hand-seeded obligation violations (i)/(ii)/(iii) - P4 walker year-source blindness diagnostic (erratum E1 witness pair) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
The Slice B NPV obligation checker's year-row reader (`_read_solve_data_csv`) fell back to a bare `pl.read_csv(...)` on the workdir CSV when the Provider didn't carry the frame. That disk arm tripped the post-Step-2 cascade invariant (test_meta_provider_invariants::test_no_disk_csv_reads_in_cascade, Rule 1) — the seed-funnel bug class the invariant guards. Make the reader Provider-only (`_provider_get` -> `provider.get`), matching every other reader in `_derived_branch.py`. The checker now does no disk I/O; the caller owns placing the true emitted year CSVs on the Provider (tests seed via seed_provider_from_dir / _mk_provider; the Slice D activation site inherits the same solve_data frames the rest of the cascade already consumes). Obligation (iii) semantics preserved. SOUNDNESS SCOPE docstring + design (local) sec 6.2 updated with the deviation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Add the opt-in solve.stochastic_invest_method enum (value list
stochastic_invest_methods: none/recourse, default none) via migration
v70. 'none' is byte-identical to prior behaviour; 'recourse' is
reserved for Slice D and hard-rejected by the Slice C guards.
- FLEXTOOL_DB_VERSION 69 -> 70; _migrate_v70_stochastic_invest_method
mirrors the v64 solve.scaling exemplar (value list + definition +
guarded solve_advanced grouping).
- PARAMETER_TYPES row ('stochastic_invest_method','solve') DIMENSIONLESS.
- _solve_config read/store + stochastic_invest_method_for resolver
(absent -> 'none').
- Regenerated spinedb_schema.json master template + the two managed
canonical DBs (templates_examples, howto_stochastics).
- export_settings.yaml split_params: surface the new solve knob.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Three flag-independent / flag-keyed validation guards that keep the recourse opt-in inert until Slice D lands. - Guard 1 (flag hard-reject): stochastic_invest_method != none raises FlexToolConfigError at the top of _orchestration.run() (authoritative, covers rolling/contained solves via the roll-suffix strip) plus a fail-fast pass in _validate_model_solve over the top-level solves. - Guard 2 (Benders x genuine stochastics): _is_genuinely_stochastic raises in _run_benders_solve (primary) and at the top of solve_benders (defensive). Detector = period_branch_full d!=b b-tokens intersected with period_in_use_set, so the rolling realized-only bookkeeping shape does NOT false-positive. - Guard 3 (handoff synthetic-name assertion): build_handoff_from_solution rejects any realized_invest / committed period that is a synthetic branch fan member; None-degrades when flex_data is unavailable. New shared cycle-free module _stochastic_detect.py holds the single token-extraction idiom (synthetic_branch_tokens) used by Guards 2 and 3 and the PIU-intersected is_genuinely_stochastic used by Guard 2. Both cast d/b to Utf8 before comparing — at runtime period_branch_full is Enum-typed with disjoint d/b category sets, and comparing two enums raises in polars (would otherwise crash Guard 3 on every stochastic handoff); null b (deterministic shape) is dropped. Empirically verified on stoch_two_period: genuine stochastic -> True (the _rlz metadata members are excluded by the PIU intersection, the _low branches survive); deterministic + rolling -> False. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
- test_v70_stochastic_invest_method.py (M1-M3): migration round-trip
(definition present, default none, bound to stochastic_invest_methods,
grouped solve_advanced; value list {none,recourse}); pre-v70 -> v70
reach + idempotent re-run; master-template --verify up to date.
- test_recourse_guards.py (G1, G2a-c, G3): resolver normalisation +
_validate_model_solve reject/pass + rolling-suffix-strip composition
(G1); PIU-intersected detector on genuine/deterministic/self-row/
rolling-realized-only shapes incl. the Enum disjoint-category
regression pin and solve_benders defensive raise (G2); solver-based
handoff clean-path / config-level synthetic raise / None-degrade (G3).
Detector shapes mirror the empirically verified stoch_two_period
runtime frames (Enum-typed d/b, PIU excludes _rlz metadata members).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Emit solve_data/stochastic_invest_method.csv (resolved per-solve mode) next to the realized-invest emit; _recourse_invest_active reader (provider-first, workdir fallback, absent -> False); FlexData.recourse_invest bool stamped in apply_derived_c; _has_branch_invest_axis capability conjunct reusing is_genuinely_stochastic. Dump-registry + accumulator-manifest lines so a dump/reload round-trip preserves the flag. Flag defaults none -> no behavior change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Define LineageFilterError(ValueError) in _solve_state; the two Slice B lineage guards (_assert_lineage_castable, assert_recourse_npv_preconditions) raise it. Narrow the four blanket except-Exception swallows in apply_derived_c / apply_synthetic_invest_sets (edd lookback / edd_invest / edd_divest_active / synthetic edd) to re-raise LineageFilterError so a lineage failure is loud instead of silently reverting to unfiltered first-stage sets. Raise path is unreachable while lineage is None (flag-off byte-parity). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
(A) _expand_invest_branch_periods — invest-side mirror of the dispatch
_expand_branch_periods; wraps the 5 axis-defining _solve_periods(invest_periods)
sites (ed_invest_set, ed_divest_set, ed_invest_forbidden, eager
ed_entity_annual_family, apply_npv) so period_invest gains {p_k, p_k_b}
under recourse. ctx/provider threaded through those readers + their
apply_derived_c / apply_synthetic callers. period_first + PIU-fallback
sites left unwrapped (design §3.1).
(B) _anchor_expand_explicit + module-scoped anchor-pairs holder set at each
Layer-4 boundary (apply_derived_c / apply_npv / apply_synthetic): explicit
per-period rows (invest_cost/discount/lifetime/fixed_cost Maps + per-period
caps) inherit their anchor's value onto branch periods, wired into the three
producers (_resolve_per_period_lf, _per_entity_period_cost,
_ed_explicit_period_param). Holder None flag-off/deterministic -> no-op ->
byte-parity.
Flag-off spot-suites (fanout, npv_cluster, lineage) green; new axis unit
tests cover expansion + anchor inheritance.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Thread provider through period_walk_iterator into _p_years_d_lf (year side) and _inflation_factors_lf -> _years_for_period_lf -> _years_for_period_from_source (factor side). Add a provider-gated canonical arm to _years_for_period_from_source reading the emitted p_years_represented.csv (which byte-copies fan-member year rows via derive_years_represented's branch loop). Under recourse the caller (D4 apply boundaries) forwards provider so branch periods resolve their anchor's year-from-start -> correct annuity windows; provider=None (flag-off) keeps the canonical arms dead -> today's fill_null behaviour (the W6 today-bug pin) -> byte-parity. Plumbing only this phase (no caller forwards provider yet); activation lands with lineage at D4. Flag-off npv_cluster/walk/fanout suites green; new canonical-arm unit tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Compute the scenario-lineage frame (flex_data.dd_same_scenario) at the three Layer-4 apply boundaries, gated on recourse AND a genuine branch invest axis (capability conjunct). Forward lineage=... into the edd builders threaded by Slice B: edd_invest_lookback_set, edd_invest_set_lf, edd_divest_active (#18), the synthetic edd, and apply_npv's four ed_*_from_source walks. Hoist assert_recourse_npv_preconditions above the edd try/except blocks so a violation raises LineageFilterError (loud, D1). alpha-1 walker Provider: a boundary-scoped module-global (_RECOURSE_WALK_PROVIDER) armed alongside the anchor-pairs scope; period_walk_iterator falls back to it so the canonical year/factor arms revive under the flag without threading provider through ~12 NPV/edd caller signatures. None flag-off -> arms dead -> W6 today-bug preserved. Activation is inert flag-off (lineage None) -> byte-parity; 103-test spot-suite (npv_cluster/fanout/lineage/axis/year) green. Flag-on edd cross-leaf integration lands at D5 (Guard 1 lift + fixture). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Objective: multiply ALL EIGHT invest/divest cost Params (process annu/lf/lfd/annd + node annu_n/lf_n/lfd_n/annd_n) by pd_branch_weight[d] via the _wt closure, gated on d.recourse_invest AND pd_branch_weight; replace the .mod "NOT weighted" comment with the recourse rationale. Flag-off skips the multiply (byte-parity). Guard 1 LIFTED for 'recourse': _stochastic_invest_raw_guard validates the RAW authored value (resolver collapses typos to 'none' so the guard must read the raw dict); both sites (run() + _validate_model_solve) switched; guard tests flipped per design §15.4 + NEW raw-typo rejection test. Activation fixes found by the gate (recorded in design implementation notes): (a) precondition checker needs the REAL workdir (Provider-only readers key paths off it — design §8 snippet's None arg made obligations fail spuriously); (b) lineage built ON-DEMAND per boundary via dd_same_scenario_df (flex_data.dd_same_scenario populates only at derived_g); (c) alpha-1 walker holders gain the workdir twin (_RECOURSE_WALK_WORKDIR) so the canonical p_years_d arm goes live; (d) _recourse_usable_ctx sanitizes the snapshot-reload SolveContext (all-null d_anchor artifact) with provider-arm fall-through. Gate fixture stoch_two_period_invest.json (+ builder, + conftest wiring): supply-side branch variation (deterministic demand 100/130 + wind upper_limit profile rlz .4/.4 low .2/.1 via group__unit -> peaker residual 60/90 / 80/120) because node-inflow branch resolution gates on group__node and a stoch-group node would fire NA net-charge pinning. D5 gate: objective 196875 EXACT; coefficients 525/262.5/1575/787.5; per-leaf invest 60/80/30/40; committed output realized-only; edd cross-leaf ABSENT (W6 re-pointed); pd_non_anticipativity empty; STRICT autoscale clean. Flag-off parity: 128 tests green (lineage/walk/fanout/ guards/npv_cluster + recourse unit suites). ruff clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Per-scenario-path total-capacity caps so a total cap applies once per
scenario leaf under recourse, never cross-scenario double-counting
(design sliceD §7 (G) / §14 (L)).
- d_leaf (d, leaf) frame in _derived_branch.py: partitions period_in_use
into scenario leaves (real anchors -> "__realized", synthetic members
-> time-branch id); registered on FlexData, built in apply_branch_cluster.
- model.py _emit_entity_total_cap: the four maxInvest/maxDivest entity
total builders emit the legacy over=("d",) shape + name for a single
leaf (deterministic / flag-off, byte-identical) and …_path indexed by
(entity, leaf) for >1 leaf, masking the inner sum to each leaf.
- _cumulative_invest.py _emit_group_total_divest_cap: the divest-side
group total gains the same per-path treatment (max/<= cap only).
- autoscale registry: maxInvest/maxDivest_entity_total_path(_n) (POWER/
ENERGY) + maxDivestGroup_entity_total_path (group_capacity resolver);
test_registry_coverage dynamic-name pins.
- Fixture invest_max_total tightened 250 -> 150 (§15.1): per-leaf totals
90/120 non-binding, legacy all-d sum 210 would have bound — the
asymmetry proves per-path. Gate test asserts two leaf rows.
Gate: objective 196,875 exact; STRICT autoscale clean; flag-off parity
(derived_branch/period_walk lineage, continuation fanout, recourse
guards, deterministic invest cumulative/total goldens) green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Make the recourse handoff's drop of branch-named (non-realized-scenario) v_invest rows EXPLICIT (design sliceD §10.1 (H)). - build_handoff_from_solution: partition invest_by_ed on synthetic branch-token membership BEFORE the commit loop and debug-log the dropped mass; the commit gate previously dropped these by accident. The Slice C Guard 3 assertion stays as a belt-and-braces invariant check (no synthetic token may survive into committed rows), not the mechanism. Flag-off / deterministic: synthetic set empty → no-op (byte-parity). - Gate test: the pre-filter drops the low leaf's 80+40 = 120 MW, debug- logs it (periods p2035_low/p2040_low), and commits only the realized 60/30; Guard 3 stays inert. Guard 1 raw-value lift was already landed at D5. Flag-off parity (handoff cumulative carriers, continuation fanout, recourse guards) green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Outputs, cost reconciliation, and the cross-scenario-coupling fixture (design sliceD §11 (I) / §15.5 / §15.6). Outputs (§11.2): - read_highs_solution: under recourse AND output_horizon, union the horizon-expanded d_realize_dispatch_or_invest set (branch periods included) into the invest-axis canonical order, so non-realized branch v_invest rows surface. Gated on both flags → byte-parity for deterministic models and for recourse without horizon output. Committed path stays realized-only (§11.1). Costs (§11.3, choice I-a): - calc_costs probability-weights the invest/divest + lifetime-fixed cost rows by pd_branch_weight under recourse, so the cost breakdown reconciles to the LP objective's (expected) invest contribution rather than over-reporting the realized spend by 1/w_realized. read_parameters threads par.pd_branch_weight + par.recourse_invest. Gated → factor 1.0 off-flag (byte-parity; deterministic cost-aggregation goldens green). - test_npv_cluster skips the eager-vs-lazy NPV byte-equality under recourse (the eager reference is not lineage-aware, §11.4). Fixtures/tests: - variant A gains a recourse_horizon scenario (model.output_horizon=yes); gate test asserts branch rows 80@p2035_low / 40@p2040_low surface. - variant B (stoch_two_period_invest_na): extends the proven stochastics.json storage shape with a recourse_na scenario making the stochastic-group storage node investable. Witnesses: non_anticipativity_storage_use fires (>0 rows), branch v_state == the NA-tied realized value, and the node-side annu_n objective coefficient carries pd_branch_weight (the F3 node-side pin). - cost reconciliation e2e: reported realized invest cost 0.039375 M = 0.25·(60·2100 + 30·1050); solver objective 0.196875 M. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
- how_to.md: replace the "only one period investment stochastic models are supported" line with the per-scenario (wait-and-see) recourse recipe — what the objective means (expected value of per-scenario plans, EVPI-style; not a hedged plan), the coupling change vs the default, the α-1 cross-comparability caveat, and the Benders exclusion (design sliceD §16). - reference.md: document solve.stochastic_invest_method (none/recourse), the output_horizon interaction, and the Benders exclusion. - CHANGELOG: Unreleased draft entry (public wording, no internal model names; schema change note). No version bump — release is the owner's decision. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
The end-to-end cost-reconciliation test invokes a real solve; add the ``solver`` marker so it deselects under ``-m "not solver"`` like the other recourse solver tests (it already carried ``slow``). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
…nvariant) D0 introduced _recourse_invest_active with a provider-first / workdir-disk fallback read of solve_data/stochastic_invest_method.csv. The pl.read_csv disk arm tripped test_meta_provider_invariants::test_no_disk_csv_reads_in_cascade (Rule 1) — the same seed-funnel bug class fixed for the Slice B obligation checker in 91db38b. Make the reader Provider-only (_provider_has_key -> _provider_read), matching every other reader in this module (_build_recourse_anchor_pairs et al.). The flag CSV reaches the cascade exclusively through the Provider on all production paths: cascade entry points thread an explicit Provider, and the snapshot-reload path (load_flextool off a dumped workdir) seeds the Provider from solve_data/ via seed_provider_from_dir before the cascade runs, so it carries the solve_data/stochastic_invest_method key by construction. The recourse gate is unchanged (returns True under flag='recourse' on both the normal and reload paths). Update test_recourse_plumbing to seed a Provider from the workdir (mirroring the cascade) instead of relying on the removed disk arm; add a no-Provider case asserting the byte-parity-safe default. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
The v70 migration (_migrate_v70_stochastic_invest_method) attached
solve.stochastic_invest_method to the solve_advanced parameter_group only
"when that group exists", but probed existence with the raising
db.item(db.mapped_table("parameter_group"), name="solve_advanced"). In the
current spinedb-api, db.item RAISES SpineDBAPIError when the item is absent
(it does not return None), so the `is not None` guard never helped — it
crashed before returning.
solve_advanced is created by the v44 migration (recoloured in v45), so real
DBs migrated from < v44 and the shipped template always carry it. But a DB
built from scratch at a version above v44 never ran the v44 step and has no
parameter_groups at all — e.g. the minimal v64 fixtures that
tests/test_v65_migration.py and tests/test_v66_migration.py seed and then
migrate forward through v70. Those two tests crashed on the v70 step
(regression vs main, where they pass).
Fix: use the non-raising db.get_item guard and get-or-create the group —
create solve_advanced (mirroring v44's priority 87 and the post-v45 colour
b56f6f) only when the DB lacks it, then always attach the param. This keeps
the param in its canonical schema home and satisfies the v69 invariant that
every parameter belongs to a group; it is a clean forward add, not
crash-tolerance. The v70 fixture (lh2_three_region) already has the group,
so the migrate-from-v69 path is unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
The three recourse process-globals (_RECOURSE_ANCHOR_PAIRS, _RECOURSE_WALK_PROVIDER, _RECOURSE_WALK_WORKDIR) were reset only inside the three flag-aware Layer-4 boundaries. Add _reset_recourse_scope() called once at the top of _apply_db_overrides (the single per-solve cascade entry, before pass 1a) so a future per-period producer wired into an earlier cascade step cannot inherit a stale global from a prior flag-on solve in a chained flag-on->flag-off multi-solve. The boundaries still overwrite unconditionally; this is belt-and-suspenders. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
…icit The final cast(d_dtype, strict=False) could silently null an out-of-vocab token, contradicting the docstring's claim that the round-trip "cannot silently null a token." Mirror the sibling _derived_branch.d_leaf_lf: on the non-Utf8 (Enum) path, collect and hard-raise on any nulled token instead of dropping it. Cannot fire in valid runs -- anchor tokens arrive already typed as d_dtype (lossless Utf8->Enum round-trip) and branch (br) tokens are covered by the 4.0.4 vocabulary splice. Utf8-d callers (both production call sites) skip the collect entirely, so no hot-path laziness/perf regression. Docstring corrected to match. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Enable create_stochastic_periods to trigger the branch fan at a period k>1 (mid-horizon reveal) while keeping pre-reveal periods as a single real-named trunk (Option B, non-anticipativity by construction). - _stochastic.py: replace the 'realized row at the solve first step' validation with a well-formedness check (period-boundary reveal + exactly-one-realized per branching period), admitting a boundary mid-horizon reveal byte-identically for fan-at-first-step. - _timeline.py make_step_jump + _derived_params.py dtttdt_from_source: a branch copy with no earlier same-time-branch period now links its first step to the nearest anchor STRICTLY BEFORE the copy's own anchor (shared pre-reveal trunk end-state) instead of self-cycling. First- period fan self-cycles unchanged (empty anchor range) -> byte-parity. Byte-parity gate green (recourse 196875, storage *_na goldens unmoved). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
…dedup
Option B correctness for the shared pre-reveal trunk:
- _lineage_leaves now returns (label, members) pairs (F4) so callers
carry the authoritative leaf label instead of a brittle zip-alignment.
- d_leaf: partition -> RELATION. A shared pre-reveal anchor belongs to
EVERY branch leaf's path, so the per-path total cap counts it into
each leaf's total (byte-parity for fan-at-first/deterministic: no
pre-reveal anchors).
- New dd_same_scenario_annuity lineage: the NPV annuity / fixed-cost
window walks dedup branch copies of the same CALENDAR slot by anchor,
so a shared trunk's window counts each future period once (window
{p2035,p2040}=2), not once per branch (=3). Reconciles design
§4.1 (capacity edd keeps the full frame with (p2035,p2040_low)) with
§8.1 (annuity=2). Fixes the latent NPV annuity over-count that the
shared trunk exposes. No-op for every non-shared-trunk solve.
- check_recourse_npv_preconditions obligation (i): re-scope leaf-weight
constancy to the POST-REVEAL segment (exclude shared pre-reveal
anchors, which carry the certain weight 1.0) per the function's own
Slice-E note; the two-stage weighting is correct, not a violation.
Fixture stoch_two_period_hedge (RP mid-horizon 173250 / WS fan-at-first
157500) + conftest wiring.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
- test_recourse_invest_hedge.py: T-RP (173250, hedge x=60/r=60/0), T-WS (157500, EVPI=15750>0), T-EEV (192937.5 hand-calc, VSS>0), T-F/T-V (mid-horizon fan + validation, solver-free unit harness), T-L/T-E2 (dd_same_scenario shape A, d_leaf relation, pd_non_anticipativity empty, no invest-NA family), T-C (per-path cap two-leaf), T-S (storage NA fires on (p2040,p2040_low), branch copy links to shared trunk, make_step_jump <-> dtttdt parity). - hedge fixture gains a hedge_storage scenario (stochastic-group storage node resv) for the storage-NA / continuity assertions. - docs/how_to.md + CHANGELOG: mid-horizon hedged two-stage reveal. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Completes the E1-E3 annuity-lineage addition (functions defined and consumed by _derived_npv; finish the public export). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
…ng fix Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
The v70 stochastic_invest_method migration (Slice C) regenerated only the 2 generate_canonical recipes; the 7 tests/fixtures and 8 canonical_databases JSONs were left at v69, failing the CI test_fixtures/canonical_databases verify steps. Pure schema-additive regen (new value list + param definition + version bump); no test behavior or golden change (tests migrate JSON on load). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
sync_master_template runs migrate_database, whose sqlite engine handle is still open when TemporaryDirectory cleanup runs -> WinError 32 on Windows. Same check runs on the ubuntu-only template-check CI job. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Document the two per-entity knobs (construction_lead_time + _method), the four seam-snap methods, the order-vs-commissioning / cost-vs-availability split, the out-of-horizon (voluntary warn / forced error) behaviour, and the orthogonal composition with the invest non-anticipativity window for automatic stochastic staging. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
…wallow) The Slice H silent-failure guards were defeated by broad `except Exception` swallows in the real solve path: * MAJOR-1 — the §2.5 [F3] join-desync guard raised a bare `ValueError` that the edd builders' `except Exception` in apply_derived_c / apply_synthetic_invest_sets caught, silently reverting to the unlagged path. Give it a dedicated `CommissioningLagError` (ValueError subclass, mirroring LineageFilterError) and re-raise it ahead of the broad except at all edd sites. * MINOR-3 — a commissioning_year_lf bug on a genuinely lag-configured model was swallowed to None → silently-unlagged wrong model. Route both build sites through a shared `_build_commission_lf` that re-raises when `construction_lead_time > 0` is configured, while still swallowing the no-lag case (helper returns None anyway → byte-parity). * MINOR-2 — mirror the F5 forced-out-of-horizon assert into the synthetic invest path, which also threads commission_lf into the walk. * NIT-4 — collect commissioning_year_lf's result once instead of twice. No-lag / immediate models never reach any new raise, so existing models stay byte-identical (recourse + invest-NA gates green, zero golden changes). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
…lobals
The Slice H construction-lead-time STAGING tests failed in the full sweep
(green in isolation) because the commissioning-year helper silently
resolved its seam calendar from the WRONG solve.
Root cause: commissioning_year_lf's [F2] fallback reads the boundary-scoped
recourse walker process-globals (_RECOURSE_WALK_PROVIDER / _WORKDIR via
_recourse_walk_provider()/_recourse_walk_workdir() in _derived_params).
A real recourse solve arms those globals and leaves them ARMED when it
finishes; in production the next solve's cascade clears them
(_apply_db_overrides -> _reset_recourse_scope, the F1 guarantee). But a
solver-free unit test that calls the helper (and period_walk_iterator)
directly never runs that cascade, so on the same pytest worker it inherited
a PRIOR recourse solve test's Provider + workdir and read that solve's
p_years_d.csv. Its periods don't intersect the stub model's period_in_use
-> the seam set collapses to empty -> every snap is null -> commission_lf
comes back None -> the availability walk keeps yr_d -> capacity is available
from the ORDER period (all periods serving) instead of the lagged
commissioning period. The staging assertion serving == {"p1"} then failed
with {"p1","p2","p3","p4"} — the lag silently vanished.
ea3795a (which added tests to the guard file) merely re-bucketed the
xdist loadfile distribution so the staging file co-located with a leaker
(test_recourse_invest_{hedge,na,gate}); the latent leak was always present.
Fix (two parts):
* Test isolation — add an autouse conftest fixture that resets the recourse
walker process-globals before/after every engine_polars test, mirroring
the existing axis-enums ContextVar-leak reset and giving every test the
same clean slate the F1 cascade already guarantees a real solve. This is
the direct fix: helper AND walk both read the (now-clean) globals.
* Loud-on-empty guard (belt-and-suspenders) — commissioning_year_lf now
raises CommissioningLagError when a lag-configured model
(_construction_lag_configured true) hits a calendar DESYNC: _p_years_d_lf
returns None, or the seam set over period_in_use is EMPTY. This converts
the dangerous silent-no-lag into a loud failure and closes the
stale-but-wrong-provider hole the F3 join guard misses (empty overlap
never fires F3). Keyed on the EMPTY-seam / no-calendar signal, NOT on
emptiness alone, so legitimate immediate / round-to-zero models (whose
seam set is non-empty and simply snap back onto the order period) keep
returning None — byte-parity preserved (no-lag/immediate unchanged, zero
golden changes).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0199XGsB22JSNNHH1cxcWbcQ
Add reserve_duration scalar to reserve__upDown__group (schema parameter_definitions + parameter_types), plumb the loader (_direct_params from_source + wiring, FlexData field, _warm deferred + shape), and cover the helper with in-memory loader unit tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> (cherry picked from commit 9031ce2dcb82451b0317852dd368019be3088f98)
…on (#322) Add reserve_storage_up_floor / reserve_storage_down_headroom in _reserve._add_storage_reserve_constraints: a storage-backed reserve provider must hold enough stored energy (up/discharge) or free headroom (down/charge) to sustain committed reserve MW for reserve_duration hours. Efficiency-corrected via p_slope (coalesced to 1.0 over the provider set), reliability excluded, ud pre-filtered before the topology join. Down headroom replicates maxState's invest/divest cross-Enum capacity block scaled by p_state_unitsize. No slack: with v_reserve=0 both reduce to existing bounds, so a short battery just commits less reserve and pays the vq_reserve penalty. Absent reserve_duration => byte-identical LP. Dispatch wiring threads v_state/v_invest_n/v_divest_n into _reserve (gated on has_storage/has_invest_n/has_divest_n). Registers the two ENERGY constraint families + the DURATION param in the autoscale registries. Synthetic engine tests (both directions, efficiency, connection provider, no-op, feasibility) run under FLEXTOOL_AUTOSCALE_STRICT=1; real-SpineDB loader e2e; docs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> (cherry picked from commit 054b7a47bac8490a7f56d9a0e471a3388f0c9ee1)
) Adds invest/divest-eligible storage coverage to the reserve_duration synthetic suite so the DOWN-headroom capacity-replication block in _reserve.py (the __at_reserve_down_{invest,divest} virtual-var views and the cross-Enum n->e / n->p casts) is exercised through canonicalise() and Layer-2 autoscale (STRICT). Previously no synthetic fixture built an invest/divest storage node, so v_invest_n / v_divest_n were never present when the DOWN-headroom constraint was emitted. - fixture: add invest/divest flags (edd_invest_set / edd_divest_active, p_entity_max_units, annuity cost) on the storage node bat. - new tests: closed-form proof that the +v_invest.unitsize ceiling term is real (invest reaches 5, vq 0.2 -> 0, v_reserve 0.4 -> 0.5), plus an invest+divest build+solve+Layer-2 STRICT roundtrip that canonicalises both capacity views without raising. - fix stale e2e comment: tests.json now ships at v71 with the reserve_duration definition (commit 30286318); the manual import_data is a harmless idempotent upsert. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> (cherry picked from commit 3e3fa37a6ebd294bf36a80a33d0ed34e9d0bb98a)
…_duration
Consolidate the reserve line onto the stochastics-finish line (construction-
lead-time, v73) and resolve the duplicate-v71 collision in a single schema
bump v73 -> v74:
- Rename the two unreleased stochastic non-anticipativity window knobs to
positive names (user-facing DB params only; internal non-anticipativity
mechanism names kept, since they genuinely implement non-anticipativity):
non_anticipativity_periods -> shared_operation_periods
non_anticipativity_invest_periods -> shared_invest_periods
- Add reserve_duration (reserve__upDown__group, issue #322) to existing DBs
via the v74 migration (reader/constraints cherry-picked from the reserve
line; the v71 reserve migration was dropped to avoid the collision and
re-authored at v74).
- _migrate_v74 does both (collision-safe rename helper + reserve_duration add,
idempotent); FLEXTOOL_DB_VERSION 73 -> 74; fixtures/canonical DBs regen via
the standard chain; new names registered in PARAMETER_TYPES (autoscale).
Verified (FLEXTOOL_AUTOSCALE_STRICT=1): migration v71-v74 23 passed; reserve
198 passed; stochastics/NA 31 passed (invest-NA tie binds under
shared_invest_periods, RP 173250); autoscale coverage 39 passed; ruff clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZpCPYcqig42z8JQW3Qj5c
…axes broadcast_to_period_time cast axis tokens with strict=False BEFORE the null-pattern split, so an unknown/inactive period or timestep token became null and was then misread as a "missing axis" default — broadcasting that row across the whole axis (reproduced: a MAP_TIME row (b,t01) got both 50 and 60). Fix (spec fix_map_reading.md §3.4): capture period/time null masks BEFORE the cast, drop rows whose token was non-null before but null after (unknown/inactive; DEBUG-logged), and drive the null-pattern split off the PRE-cast masks. Applied to all three branches (MAP_PERIOD_TIME four-way, MAP_PERIOD, MAP_TIME). No-op for valid data by construction (pre-cast masks == post-cast nulls when nothing fails), so byte-identical for the normal case. Tests: new test_broadcast_null_cast_fix.py (6, verified load-bearing against the pre-fix logic). Cascade/byte-parity gate 822 passed byte-identical; param-shape/instant-flow/silent-default suite 66 passed; ruff clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CZpCPYcqig42z8JQW3Qj5c
…arams
One parameter authored with different Map shapes across entities (e.g.
flowGroup.min_instant_flow: one entity Map(period), another Map(time)) was
silently corrupted: the reader chose ONE positional index-column list for the
whole parameter, so tokens landed in the wrong axis column and the production
scenario-trim then dropped the misplaced rows — outcome flipped with DB row
order, no warning. (spec fix_map_reading.md §3.1-§3.3, rev 2.)
Registry-scoped fix (entered ONLY for the 48 PERIOD_TIME_PARAMS keys — those
whose allow-list contains MAP_PERIOD/MAP_TIME/MAP_PERIOD_TIME):
- Reader (_spinedb_reader): _unroll_period_time_rows classifies each Map level
per-row (index_name -> silent-label value-domain vs global axis vocab ->
raise on ambiguity), canonical period-before-t column order, each row writes
only its own axes. Shared classifier reused by parameter_shape_info.
- New parameter_shape_variants on SpineDbReader + InMemoryReader + optional on
the InputSource Protocol.
- Resolver (_param_shapes): per-variant allow-list check — accepts a combined
MAP_PERIOD_TIME for commodity.price (every row shape individually allowed),
rejects a genuine 2-D row there, and rejects a time/2-D map on a period-only
param. Raises FlexToolConfigError naming the offending entities; one INFO
line per mixed (class,param).
Behavior change (CHANGELOG, spec §5 decided): an unaccepted shape on a
period-only parameter now RAISES (naming the entity) instead of being silently
dropped; keys matching both vocabularies raise FlexDataIntegrityError.
Byte-identical by construction for every non-PERIOD_TIME_PARAMS key and every
single-shape registry param (new branch not entered / same frame). Verified:
golden snapshot 5 passed (reader-level byte-identical across corpus); dual-axis
matrix 42 passed (13 cases x row-order x reader{enums,no-enums}); registry pin
4 passed; reader/shape regression 106 passed; cascade byte-parity 822 passed
byte-identical. ruff clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZpCPYcqig42z8JQW3Qj5c
unit.min_load allows period/time/period-time Maps, but the minFlow_minload
constraint read it via _entity_scalar_explicit, which drops the Map index
column — a Map yielded multiple (unit,value) rows that polar_high then SUMMED
in the constraint, inflating the floor. Confirmed end-to-end: a period Map
{d1:0.4,d2:0.3} gave floor 0.7 in every period (an hourly Map would sum
thousands of values → floor >>1, unit can't run). (spec fix_map_reading.md §5.)
Fix: produce p_min_load as a (p,d,t) Param via the same _eff_param_to_pdt
template the efficiency curve already uses (resolve_param_shape +
broadcast_to_period_time), instead of the scalar read.
- _derived_params: p_min_load_pdt_from_source wired into apply_derived_b beside
p_slope; added to DERIVED_B_FIELDS.
- _direct_params: dropped the scalar (p,) assignment + unused helper.
- input.py: field now (p,d,t); synthetic/rolling re-wire beside SECTION-1/
RESERVE-1 so the ONLINE-gate field isn't None on early-return sub-solves.
- _dump_csvs: min_load moves to pdtProcess.csv (period+time), mirroring
availability.
- model.py unchanged (polar_high broadcasts (p,d,t) identically to (p,)).
Scalar min_load byte-identical (0.4→0.4); only Map changes: period Map now
0.4/0.3 per period, time Map per timestep. Tests: new test_min_load_pdt.py
(6, producer + end-to-end), delta4 retargeted to the derived path; b04_uc +
flex_coal_min_load scalar parity green; min_load_rolling (synthetic) green;
csv-dump round-trip green. FLEXTOOL_AUTOSCALE_STRICT=1; ruff clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZpCPYcqig42z8JQW3Qj5c
Adds a solving e2e for the rolling stochastic fixture `1_week_rolling_wind`, landed BEFORE the W1/W3 fix (spec stochastics_reassessment.md §E: coverage for rolling must come first so the fix is observable and no future change silently re-breaks it). The rolling solve currently crashes at the E0 branch-reveal validation (_stochastic.py:466-476, FlexToolConfigError). The test documents that via a module-level strict xfail(raises=FlexToolConfigError) keyed on a one-line switch (W1_W3_LANDED), so the suite stays green now and a single edit flips all three tests to real pass-assertions once W1/W3 merge. strict=True guarantees any accidental future pass xpass-fails the suite, forcing the flip. Assertions (post-flip): - solve completes: 7 rolls, each strictly optimal; - PRIMARY structural signal: per roll, flex_data.period_branch_full (non-empty d!=b sub-cohort) + dt_non_anticipativity are populated — the exact precondition the operational-NA gate (model.py:4519-4522) requires; - storage NA ties v_state across branches at tied (d,t) cells (two reconciliation advisories recorded at the flip switch for the W1/W3 implementer). Builds the DB from JSON via stochastic_db_url (invariant #3); runs under FLEXTOOL_AUTOSCALE_STRICT=1 (invariant #1). No engine code touched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CZpCPYcqig42z8JQW3Qj5c
The v74 fold (6c2a24f) renamed the user-facing solve params non_anticipativity_periods -> shared_operation_periods and non_anticipativity_invest_periods -> shared_invest_periods in the schema, but export_settings.yaml still listed the old names. This failed the TestSplitParamsSchemaSync guards (schema param missing from yaml + yaml param stale vs schema) — the CLAUDE.md invariant #4 post-rename follow-up. Rename both occurrences (solve_period split group + nested_periodic type map) to the user-facing shared_* names. Internal mechanism names stay non_anticipativity_* by design (_solve_config.py, db_migration.py). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CZpCPYcqig42z8JQW3Qj5c
Replaces the E0 validation in create_stochastic_periods that crashed the rolling stochastic UC. The old code validated EVERY stochastic_branches row against the solve's first step and counted realized rows grouped by period NAME — in rolling, each roll sees the full row set (all 7 roll-boundary reveals under one period), so both checks fired (non-first-step rows + 7 realized across rolls) and every roll after the first crashed with FlexToolConfigError. Uncaught by tests (spec stochastics_reassessment.md §A.1). New rule: validate only the reveal that actually TRIGGERS this solve — the first declared start_step reachable while walking this solve's sliced active_time_list — matching the fan trigger loop so validation and fan agree on the cohort. For that cohort: (i) it must sit at its period's first step in this window (period-boundary reveal); (ii) exactly one realized:yes. This admits both fan-at-first-per-roll (Model 2) and mid-horizon period-boundary reveals (Model 1) without reintroducing Option-B's "first period can't branch" guard, and preserves single-solve fan-at-first byte-for-byte. Fix 1: a pre-reveal rolling roll (no reveal reachable in its window) stays a deterministic trunk instead of false-raising; the "declared branches never fire" safety-net raise is scoped to single solves, detected structurally via solve_modes[complete_solves[solve]] == "rolling_window" (getattr defaults to single, the safe direction). Fix 2: rolling vs single strictness split. A rolling roll sees only its slice, so a declared step that is another roll's legitimate trigger is indistinguishable from a misaligned one — each reveal is validated in the roll where it triggers (documented limitation). A SINGLE solve sees the whole reveal picture, so every declared reveal is validated strictly (each at its period's first step; one realized:yes per period) — restoring pre-W1 per-row strictness and the single-solve midperiod-reveal rejection. Adds 6 focused unit cases (pre-reveal-no-raise, fan-at-first, not-at-boundary, two/zero realized, single-no-reveal-raises). Empirically: all 7 rolls now solve optimally, but per-roll NA frames (period_branch_full / dt_non_anticipativity) are still None — NA coupling in rolling is W3. The coverage test's two NA assertions are therefore xfail(strict, raises=AssertionError, reason="W3 pending") and test_rolling_stochastic_solve_completes now asserts for real. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CZpCPYcqig42z8JQW3Qj5c
After W1 the rolling stochastic solve ran but NA never coupled branches: per roll, flex_data.period_branch_full and dt_non_anticipativity came back None, so the operational-NA gate (model.py:4519-4522) never fired. Root cause: rolling sub-solves (1week_rolling_roll_N) are not declared in Spine, so the synthetic-solve early-return in _apply_db_overrides fires and skips passes 3-10 — including apply_derived_g -> apply_branch_cluster, the sole producer of those two frames. The per-roll emit was already correct (period__branch.csv with the d!=b fan rows; dt_non_anticipativity_set.csv = the 24-step jump). Fix: an NA-1 re-wire in the synthetic-solve branch (alongside the existing RESERVE-1/SECTION-1/MINLOAD-1/INFLOW-1 re-wires) that populates ONLY period_branch_full and dt_non_anticipativity, replicating apply_branch_cluster's two seed-preserving assignments verbatim. It deliberately does NOT call the full apply_branch_cluster, which would overwrite the correctly-seeded period_in_use_set via an unsorted .unique() and reorder nested-invest set outputs (byte-parity regression caught in review). Non-stochastic synthetic solves stay unaffected: with no d!=b fan rows, period_branch_full is self-rows only -> db_pairs.height==0 -> the gate's inner guard skips the constraint. tie = jump (dt_non_anticipativity = the committed jump, naturally). The storage NA family ties NET CHARGE (per-(n,t) delta), not absolute v_state, so the coverage test's storage assertion is rewritten to witness the net-charge delta equality across the branch cohort (incl. the realized trunk) at tied timesteps; both NA-coupling tests' xfail decorators are removed and now assert for real. Deferred (NOT W3): absolute storage continuity (spec §B/§D.6) — forecast branches cyclic-self-link their storage start while the realized trunk is start-anchored, giving a constant realized<->forecast offset; closing it is a separate anchor-repointing slice. Also flagged: period_branch_full now also feeds the invest handoff guards and benders is_genuinely_stochastic per roll — intended single-solve parity, untested for rolling-stochastic-invest (follow-up). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CZpCPYcqig42z8JQW3Qj5c
The storage non-anticipativity family ties NET CHARGE (per-(n,t) flow delta, model.py:219-234), not absolute v_state. In a rolling solve the untied post-reveal tail breaks the telescoping that makes single-solve tie absolutely, so the forecast branch copies floated by a constant v_state offset vs the realized trunk (~0.06 in 1_week_rolling_wind) — i.e. §B/§D.6 branch-point continuity was not delivered (W3 deferred this). Adds a new constraint family non_anticipativity_storage_state_first (block "1b" in _add_non_anticipativity_constraints, mirroring non_anticipativity_online_integer) that pins absolute v_state[n,d,t_first] - v_state[n,b,t_first] == 0 across the branch cohort at the FIRST tied step of each anchor period. Telescoping the existing per-step net-charge ties over the contiguous jump off that absolute anchor closes the whole tied range absolutely; the offset collapses to ~0. It is optimum-conditional on zero balance slack + branch-invariant inflow (documented). v_online/v_reserve need no analog — they already pin the absolute variable at every tied step. Proactive horizon-guard: emit per (n,d) ONLY when the tied set is a strict subset of that node's per-period v_state timeline (an untied tail exists). On a fully-tied solve (single-solve 2day_dispatch; rolling roll whose horizon==jump) the node is already doubly-anchored (hard start + reference end), so the row is redundant and SUPPRESSED — single solves receive zero new rows (byte-parity). Verified: 2day emits 0 rows, 1_week_rolling_wind emits 18 (3 pairs x 6 rolls; the horizon==jump final roll suppressed). t_first is per-d (not a global min), robust to multi-reveal single solves. The v_state-domain double inner-join (on (n,d,t) and (n,b,t)) drops state-less cells. Registered in autoscale _layer2_types.py (static literal). Test: test_rolling_na_ties_storage_across_branches keeps the net-charge witness (diagnostic localization) AND adds an absolute-v_state cross-branch-equality assertion (branches equal EACH OTHER at tied steps, not the nominal 0.5 — the shared level is the soft roll_continue value in continuation rolls). Both keep checked>0. Load-bearing: 0.06 fail without the family, ~0 with it. Deferred (NOT this slice): the HARD roll-to-roll carry-in (all branches inheriting the previous roll's actual jump end-state vs the current soft roll_continue seed) — spec W8b. Touches neither make_step_jump nor dtttdt. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CZpCPYcqig42z8JQW3Qj5c
Investment is period-granular (v_invest is indexed (p,d)/(n,d), no timestep axis) and a rolling solve advances past period beginnings, so a flat rolling_window solve builds no v_invest at all — authored invest params are silently ignored (root: _resolve_synthetic_solve reads the `_roll_N` suffix as an invest-period anchor). The intended design is to run a separate INVESTMENT solve BEFORE the rolling (operational) solves; rolling is for operations. This is not a bug to fix and not a case to validate — but it should not be silent. Adds a once-per-parent logger.warning in RecursiveSolveBuilder._process_rolling_solve when a rolling_window solve ITSELF carries investment intent (invest_periods / realized_invest_periods non-empty, or stochastic_invest_method != none) — keyed on solve-authored params, NOT on invest-eligible entities existing, so the correct invest-solve-then-rolling pattern never warns. Routed through a module-level logger because the orchestrator deliberately raises the run logger to ERROR during the cascade (_drive_cascade), which would swallow self.logger; the module logger is the established engine-advisory convention and reaches stdout via the CLI basicConfig. Log-only: no LP/frame/output change; an idempotency guard keys the one-shot on the parent solve name. Adds tests/engine_polars/test_rolling_stochastic_invest_noop.py pinning the intended + safe behavior of rolling + stochastic + investment: all 7 rolls optimal; the W3-activated period_branch_full consumers (is_genuinely_stochastic, synthetic_branch_tokens, the handoff guards) run inert and never raise; the known no-build (pd_invest_set is None per roll) is asserted as a tripwire so a future change that starts building v_invest in rolling trips loudly; and the new warning fires exactly once naming the solve. Fixture built by in-test injection on a tmp DB (invariant #3); committed stochastics.json untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CZpCPYcqig42z8JQW3Qj5c
Establishes the W8 premise that unblocks W4: in a stochastic model, realized OPERATIONS can come from a separate deterministic realized-dispatch solve whose investments are FIXED (via the existing invest→dispatch handoff) from the stochastic solve. Proven by a new fixture + e2e with NO engine change — the machinery (handoff capture + synthetic-invest drop + realized last-wins + output union) already exists; W8.1 wires a stochastic→deterministic pair and locks the behaviour in. Fixture `w8_realized_dispatch` (new, JSON-built): model.solves = [stoch_rd, stoch_rd_realized_dispatch]. stoch_rd is single_solve fan-at-first, stochastic_invest_method=none (shared first-stage invest), branches rlz 0.25 / high 0.375 / low 0.375 (sum 1.0), genuinely stochastic; it commits v_invest[peaker]=60. stoch_rd_realized_dispatch is deterministic, listed AFTER (required for last-wins), consumes the 60 MW as fixed existing capacity (mirrors the proven lh2_invest→lh2_dispatch precedent). e2e asserts: (a) both solves optimal; (b) dispatch receives the stochastic solve's v_invest as fixed existing capacity (dispatch builds NO v_invest, p_entity_invested[peaker]=60, peaker existing=0 in base so the 60 arrives only via the handoff — not pre-baked); (c) last-wins empties the stochastic solve's realized window (stoch realized_dispatch None; dispatch owns p2035/t0001-t0003); (d) hand-calc dispatch objective 5,256,000 (60 MW · 8760 h · 10, annualized) at rel=1e-9. Runs under FLEXTOOL_AUTOSCALE_STRICT=1. Documented W4 boundary (not patched): the deterministic dispatch reads the realized series only while that series is DETERMINISTIC; once forecasts diverge from realized (W4), feeding the realized data to the dispatch solve needs a real mechanism. W8.1 keeps realized operational data deterministic (readable by both) with genuine forecast branches, fully exercising the handoff + last-wins + output routing. Author-declared vs engine-automatic realized-dispatch is a deferred W4-era decision. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CZpCPYcqig42z8JQW3Qj5c
…1 (W4a)
First slice of W4 (de-scenario the realized weight), now unblocked by W8.1. Adds
a new solve-level method `dispatch_realization_method` (schema v74→v75), value
list {within_solve (default), separate_dispatch}, as the foundation for the
engine-automatic realized-dispatch solve. Structural clone of the v70
stochastic_invest_method line.
- within_solve (default) = today's behaviour: realization happens inside the
solve's committed/jump region, no second solve → BYTE-PARITY (no engine branch
keys off the new value yet).
- separate_dispatch = the future auto-spawned realized-dispatch solve; HARD-REJECTED
for now with a clear "not yet implemented (later slice)" FlexToolConfigError.
The method is orthogonal to solve_mode and documented as the 2-stage seed of a
future multi-stage market structure (W11); dispatch_realization (not just
realization) to disambiguate from the investment-realization params
(realized_invest_periods, stochastic_invest_method).
Mechanics (mirror v70): migration _migrate_v75_dispatch_realization_method writes
ONLY the value-list + parameter_definition (default within_solve) + solve_advanced
group attach — NO parameter_value rows, so existing DBs stay byte-identical;
reader + dispatch_realization_method_for accessor (absent/blank/unknown →
within_solve); autoscale ('dispatch_realization_method','solve')=DIMENSIONLESS.
Hard-reject guard _dispatch_realization_raw_guard reads the RAW authored value
(so a typo can't silently normalize) with allowed-set explicitly {"", within_solve}
— wired at BOTH v70 sites: the _validate_model_solve fail-fast pass AND the
authoritative run() site (after the _roll_N suffix strip), so contained/chained/
rolling sub-solves authoring separate_dispatch raise rather than silently running
the default path.
Schema artifacts regenerated (definition/value-list only, verified no value rows):
spinedb_schema.json, 10 canonical_databases/*.json, 7 tests/fixtures/*.json.
Tests: test_v75_dispatch_realization_method.py (migration/default/group/value-list/
idempotent/master-template-verify) + test_w4a_dispatch_realization_guard.py (both
guard sites, allowed-set, message). test_recourse_guards.py stub gains the new
attribute (the shared _validate_model_solve guard reads it).
Follow-ups: W4b (engine-automatic spawn of the realized-dispatch solve), W4c
(realized-branch projection so the dispatch solve reads divergent realized data),
W4d (realized operational weight→0, forecasts normalized), W4e (fixture/golden
migration).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZpCPYcqig42z8JQW3Qj5c
…atch (W4b) W4 slice 2. Makes solve.dispatch_realization_method=separate_dispatch (schema added in W4a, previously hard-rejected) actually work: when a single TOP-LEVEL solve authors it, the engine automatically registers a deterministic realized-dispatch sibling <solve>__realized_dispatch that takes the FULL override path — so it is indistinguishable from the author-declared dispatch solve W8.1 proved. Generalizes W8.1 from author-declared to engine-synthesized. Mechanism (dual in-memory registration, no DB mutation; prototyped end-to-end): - SpineDbReader.register_derived_solve: mints a FRESH entity id and injects the sibling into _entities_by_class/_entity_by_id, cloning only the 4 dispatch- defining params (period_timeset/realized_periods/years_represented/solve_mode) into _param_rows under the fresh eid — so _solve_in_spine(sibling) is True and it routes the full apply_derived_a..g path (feature passes run → no synthetic- path silent drop). Reusing the parent eid would map rows back to the parent and break _solve_periods(sibling) → avoided. - SolveConfig.spawn_realized_dispatch_sibling: inserts the sibling right after the parent in model_solve (positional → last-wins + handoff), copies the timeline scaffolding, SETs realized_periods[sibling]=parent's, and DROPs invest/ stochastic/benders/decomposition/non_anticipativity/fix_storage/contains_solves/ dispatch_realization_method. scaling/solver/decomposition are deliberately NOT copied: run() resolves those from the separate cascade SolveConfig, so the sibling uses engine defaults — correct (monolithic) for a deterministic dispatch. - _spawn_realized_dispatch_siblings: called once in _drive_cascade after the reader is built, before native_run_model, on runner.state.solve. Guard (_dispatch_realization_raw_guard, both wired sites) narrowed: accept separate_dispatch ONLY for a single top-level single_solve (in model_solve, not rolling, no contains_solves, not nested); reject the rest with a reason; keep within_solve/absent byte-parity; sibling no-ops at the run()-site. Collision on a pre-existing <solve>__realized_dispatch raises. Storage: the sibling drops fix_storage_periods, so separate_dispatch is rejected (positive guard, loud) for any parent authoring a non-empty fix_storage_periods — storage pinning in the realized-dispatch solve is a later slice. Multi-period storage WITHOUT fix_storage rides the monolithic single_solve balance + node-level storage params (preserved on the full path). Reuses the W8.1 invest→dispatch handoff + realized last-wins + output-union unchanged. Byte-parity: spawn/guard are no-ops unless a solve authors separate_dispatch, so existing within_solve/absent fixtures are untouched. Tests: test_w4b_auto_dispatch.py reproduces the 4 W8.1 oracles against the auto-spawned sibling (objective 5,256,000 at rel=1e-9; no v_invest, 60 MW handed as fixed existing; last-wins empties the parent realized window) + a reserve full-path proof (v_reserve present) + register/spawn/collision/storage-guard unit tests. W4a guard tests rewritten to the narrowed semantics. W8.1 author-declared test kept as the baseline. Deferred: W4b-rolling (per-roll spawn), W4c (realized-branch projection for divergent forecasts), W4d (realized operational weight→0), W4e (fixture/golden). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CZpCPYcqig42z8JQW3Qj5c
… (W4c) W4 slice 3 — the hard piece, and the unblocker for W4d. The W4b auto-spawned realized-dispatch sibling is deterministic and drops stochastic_branches, so its solve_branch__time_branch scaffolding was empty; the per-solve fold that collapses a stochastic 3d_map (branch,time_start,time)->(d,t) — _derived_profile _profile_stochastic_lf / _pdt_lookup PdtLookup.get — then found no branch rows, dropped the 3d_map, and the zero-fallback filled 0 (wind->0->garbage). So the sibling only worked when realized data was deterministic (2d). This makes the sibling read each stochastic parameter PROJECTED onto the realized branch, so the stochastic solve's forecasts can genuinely diverge from the realized series. Fix (minimal, additive — no fold/_param_shapes/reader change): - SolveConfig.realized_dispatch_parent: dict[str,str], recorded by the W4b spawn_realized_dispatch_sibling (explicit map, not name-parsing). - A module-level helper project_realized_branch_time_branch_rows + a gated post-pass in _native_run_model (after create_stochastic_periods, before the per-solve emit loop): solve_branch__time_branch_lists[sibling] = the parent's rows filtered by (p,p) in period__branch_lists[parent] — reusing the parent's already-materialized realized-branch rows; the (p,p) predicate (same one derive_solve_branch_weight uses) keeps realized-named rows and excludes synthetic forecast copies. The existing folds then select the realized branch unchanged. Covers a multi-period realized window (one realized row per period via the continuation loop). The sibling stays non-genuinely-stochastic: is_genuinely_stochastic keys on period_branch_full/period_in_use_set, NOT solve_branch__time_branch, so zero branch LP variables are built (verified: sibling v_flow has 3 rows, no fan). Reviewed consumers confirm the injected rows reach only the pbt/profile fold (and derive_solve_branch_weight emits 1.0 = semantically "no branch"). Byte-parity: no-op without a sibling; inert on deterministic 2d-profile data (W4b/W8.1 fixtures byte-identical). No new var/cstr/param. Tests: test_w4c_divergent_dispatch.py — branch-distinct 3d_map wind (rlz 0.50 / high 0.80 / low 0.10); the auto-spawned sibling dispatches the REALIZED branch (wind 50, peaker 50, objective 4,380,000 at rel=1e-9) with negative controls (NOT high 20, NOT low 90, NOT collapsed 100-VOLL); parent-divergence proves the data is branch-distinct; last-wins; multi-period + empty helper unit tests; spawn records the parent map. Runs under FLEXTOOL_AUTOSCALE_STRICT=1. Next: W4d (realized operational weight->0, forecasts normalized) is now safe — realized ops come entirely from this sibling reading the projected realized series. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CZpCPYcqig42z8JQW3Qj5c
…e pass) Anti-fracture pass over the accumulated separate_dispatch feature (W4a-W4c) before W4d — no behavior change (doc/comment + one guard test only). The structure was already coherent (single orchestrator entry point _spawn_realized_dispatch_siblings; single authoritative sibling->parent map SolveConfig.realized_dispatch_parent that W4c reads; clean reader/SolveConfig responsibility split). The one genuine, drift-prone fracture: "what a realized-dispatch sibling inherits from its parent" is defined TWICE in two vocabularies with nothing tying them — the reader's RAW Spine-param allow-list (_REALIZED_DISPATCH_SIBLING_PARAMS, for the full override path) and the DERIVED SolveConfig-dict copy/set in spawn_realized_dispatch_sibling (for native_run_model). The two layers genuinely need raw vs derived forms (so no clean single-list de-dup without risky surgery on working code), but the four DEFINING facts must agree: period_timeset<->timesets_used_by_solves, years_represented<->solve_period_years_represented, solve_mode<->solve_modes, realized_periods<->realized_periods. This pass: - adds a single FEATURE MAP anchor at _spawn_realized_dispatch_siblings (the entry point) enumerating every touch-point (schema/guard/registration/ projection/weighting) so the feature is navigable from one place, and names the dual-definition invariant + the W4d directive (reuse the authoritative realized_dispatch_parent / per-solve channel, do not add a parallel sibling-detection); - cross-references the two config halves (reader allow-list <-> SolveConfig copy/set) so each points at the other and at the feature map; - adds tests/engine_polars/test_realized_dispatch_coherence.py pinning the dual-definition invariant: the reader allow-list == the four defining facts, and spawn populates every defining fact's derived dict consistently with the parent. Either half drifting now fails loudly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CZpCPYcqig42z8JQW3Qj5c
…e_dispatch (W4d-A)
The payoff of the W8→W4a→W4b→W4c arc. In a separate_dispatch stochastic hedge
solve the OPERATIONAL branch weights now de-scenario realized: forecasts
normalize among THEMSELVES (realized excluded from the denominator) and realized's
own operational weight → 0 — safe because W4c's realized-dispatch sibling produces
realized operations, so the hedge solve no longer needs realized weighted. The
parent hedge objective becomes the expected FORECAST cost (+ invest annuity)
instead of the realized-in-cohort artifact.
within_solve (default) is UNCHANGED in this slice (realized→1.0 re-derivation is
W4d-B, deferred). INVEST weight (pd_branch_weight) is UNTOUCHED (W7 owns invest).
Decisive safety constraint: pd_branch_weight_lf (invest) and pdt_branch_weight_lf
(operational) both read the shared solve_branch_weight.csv (realized forced 1.0),
so the realized self-weight change lives ONLY in the OPERATIONAL producers —
derive_solve_branch_weight, pd_branch_weight_lf, and the recourse NPV precondition
(reads pd only) are literally untouched → the precondition passes unchanged
(dissolving the old "realized=1.0→cohort 2.0→crash").
Both pdt producers patched in lock-step (semantically byte-equivalent, verified):
- lazy _derived_branch.pdt_branch_weight_lf (the authoritative LP Param);
- eager _emit_period_calc.emit_branch_weights pdt loop (pdt_branch_weight.csv).
Rule: realized_set={d:(d,d)∈pb}; separate_dispatch → forecast d denom excludes
realized, realized d → 0.0; within_solve/absent → legacy arm verbatim; deterministic
pb-empty early-return untouched.
Method channel (coherence directive — the parent's own flag on the standard
per-solve solve_data channel, not a parallel sibling-detection): emit a one-row
solve_data/dispatch_realization_method.csv at _native_run_model ~:854 from
state.solve.dispatch_realization_method_for(complete_solve[solve]); both producers
read it (absent → within_solve → byte-parity). The auto-spawned sibling drops the
method → resolves within_solve → its realized ops are NOT zeroed (keeps obj 4,380,000).
Realized branch at weight 0 is free/degenerate in the parent but harmless: not
surfaced (sibling owns the realized window via last-wins) and does not corrupt
invest (annuity 283,500 unchanged; forecasts drive capacity).
Tests (FLEXTOOL_AUTOSCALE_STRICT=1): test_w4c_divergent_dispatch adds
test_parent_objective_is_forecast_only_expectation (parent v_obj == 5,101,500 at
rel=1e-9; negative control != 4,851,214.2857, the old realized-in-denominator
value) and updates test_parent_forecasts_genuinely_diverge to assert only the
forecast divergence {20,90} (the realized branch is now weight-0/degenerate, owned
by the sibling). Sibling assertions kept. within_solve goldens byte-identical
(gate is the per-solve marker, not forecast-existence).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZpCPYcqig42z8JQW3Qj5c
…y (R1)
The stochastic fan is forecasts-only; "realized" is an optional 0-or-1
designation (realized_yn=="yes"), not a mandatory weighted member. Invert
the W1 triggering-cohort validation accordingly:
- reject 2+ realized (was: reject != exactly one realized)
- reject cohorts with zero forecast (realized:no, weight>0) members
- accept n_real in {0,1} with >=1 forecast
- Block B population guard: found>1 raises; found==0 appends nothing
- keep the period/roll-boundary reveal admission check unchanged
Validation-only: no weight-math, fan-out exclusion, or precondition code
touched (those re-ground in R2/R4/W7).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZpCPYcqig42z8JQW3Qj5c
Re-ground non-anticipativity to the corrected model: the stochastic fan
is forecasts only, so NA ties forecasts TO EACH OTHER via a deterministic
per-anchor-group reference-forecast star, never to the realized trunk.
Works at n_real=0.
model.py: in the db_pairs build (behind the existing height>0 gate),
group raw db_pairs by anchor trunk d; per group pick d_ref = lexically
smallest forecast token, build the star {(d_ref, b!=d_ref)}, and re-key
dt_non_anticipativity's anchor d -> d_ref (same t). Both frames must be
re-keyed: the five NA families inner-join db_pairs AND dtna on d, so a
db_pairs-only re-point emits zero constraints. Pass the re-keyed dtna via
a new dtna_override kwarg to _add_non_anticipativity_constraints. The
deterministic (no-fan) path stays byte-identical (empty raw db_pairs ->
no groups -> no call). No new var/cstr/param/autoscale name.
R-CONT: non_anticipativity_storage_state_first is load-bearing (proven)
and re-points for free via the shared re-grounded frames; kept, not
deleted (deletion deferred to W8b behind a float==0 guard). Docstrings
updated: ties are forecast-to-forecast, not to the realised trunk.
Tests: re-point every trunk-anchored NA assertion to forecast-to-forecast
(trunk excluded from over.d/over.b; forecasts tied equal) across
test_rolling_stochastic_na, test_recourse_invest_na,
test_configurable_nonanticipativity, test_non_anticipativity,
test_flex_2day_stochastic_dispatch_full_storage, and the two emission
tests; add a new n_real=0 fixture (build_stoch_two_period_nreal0) + test
witnessing the forecast star with no realized member. Re-ground the
2_day_stochastic_dispatch scenario golden + pinned objective
(269.406895209 -> 268.081475165; a relaxation, new <= old, verified).
Also completes R1 validation-test fallout in test_recourse_invest_hedge
(corrected-model messages; zero-realized cohorts with >=1 forecast now
accepted).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZpCPYcqig42z8JQW3Qj5c
…=0 (R-NAME) Under the forecasts-only model a mid-horizon fan with no realized designation had nothing owning the real (un-suffixed) period names, so pre-reveal branch-keyed data fell through to the zero fallback and the solve silently corrupted (e.g. the hedge over-invested because its pre-reveal wind profile read as 0). Fix: in create_stochastic_periods, when the realized-branch search finds none (found==0) for a PRE-REVEAL period, bind that period's solve_branch__time_branch to the reference forecast (d_ref = the lexically-smallest forecast token, the same hub R-NA's star uses), so its data binds correctly. Pre-reveal-scoped on purpose: binding the post-reveal weight-0 phantom would join the recourse lineage chain and trip the leaf-weight-constancy precondition. Keeps "realized" optional (not mandated for mid-horizon). Byte-parity on all existing tests: the guard fires only for the n_real=0 mid-horizon topology, which had no prior fixture. New witness fixture build_stoch_two_period_midhorizon_nreal0 + test asserting the pre-reveal profile binds nonzero, SBTB owns (pre_reveal_period, d_ref), and the here-and-now invest is correct (fails for the right reasons if the guard is removed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CZpCPYcqig42z8JQW3Qj5c
…W7+R4+R3)
Complete the forecasts-only re-grounding of branch weighting and the
stochastic-value fixtures. The stochastic fan is forecasts only: the
realized/trunk self-row never enters the forecast weight denominator.
R2/W7 (weight rule): in the three branch-weight producers
(_derived_branch.pd_branch_weight_lf, pdt_branch_weight_lf, and the eager
_emit_period_calc.emit_branch_weights) compute the structural phantom set
(a trunk self-row (d,d) that also has forecast fan members), emit weight 0
for phantom anchors and exclude them from the denominator, so forecasts
renormalize to 1; a lone pre-reveal trunk with no fan member stays 1.0.
This applies to invest (pd) too (W7) and is unconditional — the W4d-A
separate-dispatch weight surgery, dispatch_realization_method_read, and
the solve_data/dispatch_realization_method.csv marker channel are removed
(spawn reads SolveConfig.dispatch_realization_method_for, not the CSV;
schema v75 + migration + resolver + spawn are kept). Also guard the
in-process re-solve path: trust ctx.period_branch only when its anchor
columns are null-free, else fall back to the raw period__branch (mirrors
_collect_lineage_inputs) so the phantom detection isn't defeated under
enum activation.
R4: the three recourse-NPV preconditions pass unchanged (phantom anchor
-> 0, forecasts sum to 1); docstrings updated and test_p3b re-pointed to a
genuinely malformed cohort (obligation (ii) is otherwise a renormalization
tautology).
R3 (fixtures): build_stoch_two_period{,_invest,_hedge} become genuine
{high, low} forecast fans (rlz retagged to a forecast). Hand-calcs
re-derived and driven-confirmed (hedge RP 173250 / EVPI 15750 / VSS
19687.5; recourse-gate 196875; continuation 70080000). For fan-at-first
recourse there is no here-and-now committed stage, so committed invest is
0 (faithful). Retired test_stoch_realized_gating_and_flows (Model 2
realized-dispatch is covered e2e by W8/W4b/W4c). Re-ground the Model 2
fixtures to declare realized as a weight-0 designation (behavior-
preserving: all W8/W4b/W4c objectives byte-identical).
Post-process (A): calc_costs/calc_slacks now apply pdt_branch_weight to
per-(d,t) operational and slack-penalty cost terms (new _branch_weight
helper + a pdt_branch_weight reader), so the reported "Total cost
(calculated)" is the probability-weighted expected cost matching the LP
objective; invest (already weighted), physical diagnostics, and the
capacity-margin penalty are left unweighted to match the LP exactly;
deterministic/single-branch output is byte-identical (weight 1).
2day scenario: retag its stray realized branch to a forecast (genuine
4-forecast fan; the realized designation was vestigial in a single solve
with no realization region) and regenerate the golden + expected_objective
(268.081475165 -> 66.9291187913 = the LP objective, zero slack). Add
test_single_invest_stochastic_ops pinning the default structure (single
per-period invest + branch-fanned operations).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZpCPYcqig42z8JQW3Qj5c
The recourse lineage/annuity leaf labelled "__realized" is a misnomer under the corrected model: it is the real-named calendar-anchor chain used to dedupe a shared pre-reveal capacity's annuity to one charge per calendar period (not per branch-copy) — it is NOT a realized scenario (realized is an optional designation, never a weighted fan member). Pure, behavior-preserving relabel: the leaf is an opaque LP-row key that is produced in exactly one place (_lineage_leaves) and compared nowhere (the annuity dedup tie-break keys off member==anchor, not the label), so renaming "__realized" -> "__anchor" changes no numeric result. The annuity over-count fix it guards is untouched (hedge RP=173250, annuity windows K1/K2=2100/3150). Docstrings/comments across _derived_branch.py, input.py, model.py re-grounded to describe the corrected model; the three test assertion literals updated. Nothing deleted: dd_same_scenario, dd_same_scenario_annuity, the dedup, and the per-path invest caps all stay (they remain load-bearing for mid-horizon / multi-stage recourse-invest, a kept capability). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CZpCPYcqig42z8JQW3Qj5c
… (H1) The rolling-stochastic roll-to-roll storage carry (p_roll_continue_state) read the wrong value: it preferred the in-memory flex_data.realized_dispatch, which is loaded from the forecast-EXPANDED emit set (all branch periods × the full look-ahead window), so its lexically-last (d,t) was the discarded HORIZON-END of a FORECAST branch (e.g. period1_upper@t0096), not the committed JUMP-END of the realized TRUNK where the next roll continues. Fix (value-only, soft — no model.py / no variable pin): prefer the Provider anchor solve_data/realized_dispatch (trunk-only, committed jump-end, emitted by emit_realized_dispatch before the handoff is built) and fall back to the in-memory frame only when the Provider key is absent (unit-test synthesis). This restores the GMPL value source. Driven on 1_week_rolling_wind: all continuation rolls now carry the trunk jump-end (0.0) instead of 2500. Pre-existing (orphan-root) defect, latent on current fixtures because the soft carry is operationally inert under every wired storage-binding method (so solutions are byte-identical); it would distort a model whose storage economics bind across the roll boundary. A solution-level witness is therefore not constructible, so the regression guard is producer-level: test_rolling_handoff_carries_trunk_committed_jump_end_storage asserts the carried value is the trunk committed jump-end (fails on the old 2500, checking both the wrong-step and wrong-branch defects). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CZpCPYcqig42z8JQW3Qj5c
Integrate main's indirect-unit capacity / input shares / per-unit min load / reserve fixes (#346) and the 4.0.5 release into feat/stochastics-reserve-v74. Schema version collision resolved: base was v69; main added one migration (v70); this branch added six (to v75). Both defined a different v70. Kept main's v70 (capacity-coefficient / input-share migration) at slot 70 and RENUMBERED the branch's six migrations to 71-76 (FLEXTOOL_DB_VERSION=76); the migrations are disjoint and order-insensitive. Renamed the six migration test files test_v70..v75 -> test_v71..v76. Canonical DBs + fixtures regenerated to a consistent v76 (sync_master_json_template / test_fixtures / canonical_databases / generate_canonical); the stochastics 2day retag survived migrate-all. Engine files model.py (+1543 from main), _derived_params, read_parameters, drop_levels auto-merged. Three genuine code conflicts resolved deliberately: - p_min_load: composed structurally — main's per-unit min-load semantics live in the minFlow_minload constraint (sum over a unit's outputs), our (d,t) map-reading capability is the producer shape; a (p,d,t) p_min_load flows through the per-unit constraint unchanged (both sides' tests pass). - calc_slacks reserve-slack penalty: combined main's shortfall_scale/reindex with our Option-A pdt_branch_weight; shortfall_scale subsumes the old reservation factor (single-apply, no double-count), matching the LP. - input.py synthetic-solve wiring: additive union (MINLOAD-1 + DELAY-1 + CAPCOEF-1). CHANGELOG: both release-note sets. Full suite green (4145 passed). Merge performed on scratch branch integrate/main-4.0.5-stochastics; feat/stochastics-reserve-v74 untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CZpCPYcqig42z8JQW3Qj5c
…y cast The construction-lead-time commission_lf desync guard relied on polars <2.0 nulling an i64->Enum key cast under strict=False, then detecting the zero surviving lag rows and raising CommissioningLagError. polars >=2.0 treats i64->enum as an unsupported-type cast and raises ComputeError outright (regardless of strict), so CI (polars 2.0) surfaced the raw polars error instead of the dedicated guard error (and risked the downstream `except Exception` hiding it). Catch the polars cast error in the survivor-check and raise the same CommissioningLagError — the non-empty pre-cast overlap with an unmaterializable typed join IS the dtype desync the guard exists to catch. Byte-parity on polars 1.40 (the null-survivors path is unchanged); fixes the sole polars-2.0 failure in the 4.0.5 merge CI (test_construction_lead_time_guard). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CZpCPYcqig42z8JQW3Qj5c
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Re-grounds the rolling/stochastic model to the forecasts-only semantics and integrates
main(4.0.5 / #346).Stochastics re-grounding
recourse(per-branch wait-and-see) stays an explicit opt-in. Validation, fixtures, and EVPI/VSS/RP hand-calcs re-derived to genuine forecast fans.Integration
origin/main(Indirect-unit capacity, input shares, per-unit min load and reserve fixes #346 indirect-unit capacity / input shares / per-unit min load / reserve fixes, on top of the 4.0.5 release).main's v70 kept; the stochastics migrations renumbered 71–76 (FLEXTOOL_DB_VERSION=76).p_min_loadcomposed (per-unit semantics + (d,t) capability); reserve-slack penalty combinesshortfall_scalewith branch weighting.Full local suite green (4145 passed). Opened as a draft to exercise CI.
🤖 Generated with Claude Code
https://claude.ai/code/session_01CZpCPYcqig42z8JQW3Qj5c