Write gpec.h5 incrementally per module so a crash retains partial results #374

Motivation

A long GPEC run that crashes in the ODE integration currently loses everything, including the equilibrium that finished minutes earlier. Writing gpec.h5 incrementally per module — Equilibrium, then ForceFreeStates, then the downstream stages — means a crash leaves behind whatever completed. This is much more natural in HDF5 than it was in the Fortran netCDF interface, which is worth taking advantage of.

Current state — already partly done

Staged writing already exists at module granularity:

The Tearing stage is already the model for what we want: _run_slayer_stage wraps the whole thing in a try/catch whose comment states that a failure "must not discard the equilibrium / stability / PE results already computed."

The actual gap

Everything in row 1 is computed before anything is written. The equilibrium solve, the ODE integration, free_run!, the Δ′ BVP and the Galerkin solve all complete, and only then does write_outputs_to_HDF5 open the file with "w". So the longest, most crash-prone part of the run is also the part with nothing on disk behind it.

Proposed change

Split the stage-1 writer so Info/, Input/ and Equilibrium/ are written immediately after the equilibrium solve, and ForceFreeStates appends with "r+". The isfile(h5_path) ? "r+" : "w" idiom at GeneralizedPerturbedEquilibrium.jl:543 is already the pattern to follow.

What makes this more than a mechanical split

1. Partial files become readable-but-incomplete, and Rerun.jl will start from them.

Today a gpec.h5 either exists and is complete or doesn't exist. Once writes are incremental, a crashed run leaves a file containing a valid Input/gpec_toml_raw — which is exactly what the rerun path reads. A rerun would silently start from a truncated file.

This needs a completion marker: a root attribute (run_complete, or better a stages_written list so a reader knows which modules are trustworthy), plus readers taught to check it. #364 introduces Utilities.HDF5Annotations.write_root_attrs!, which is the natural home for this — so this work should be based on #364, not on #363.

2. There are no flush calls anywhere in src/.

Incremental writing without an explicit HDF5.flush at each stage boundary risks leaving a corrupt file on crash, which is strictly worse than no file — a corrupt HDF5 can't be opened at all, whereas today's all-or-nothing write at least fails cleanly.

3. The bit-for-bit rerun round-trip test needs re-verifying.

HDF5 orders groups by name rather than creation order by default, so changing write order should be invisible to test/runtests_rerun_from_h5.jl — but that should be confirmed, not assumed.

4. Idempotency. Any stage that can run twice needs the delete_object-then-recreate guard write_slayer_hdf5! already uses.

Sequencing

Should land after the #226 stack (#363 → #364 → #368), based on #364 for the root-attributes machinery. Deliberately not folded into #363: that PR is a pure rename whose safety evidence is "zero regression-harness movement," and a write-ordering change is a behaviour change needing different evidence (crash injection, partial-file reads). Bolting it on would void the three approvals already on it.

Origin

Raised by @jhalpern30 in review of #363 ("I'm wondering if it would simplify some code elsewhere to write other parts of the outputs incrementally too?"), refined by @d-burg into the per-module eq/FFS/tearing form described above.