Repository navigation
Write the result of a calibration that ends before the search finishes #359
Description
Description
When the search ends early, the runner consolidates evaluations.csv and re-raises (runner.py#L567-L582): on Ctrl+C, on a worker death (BrokenProcessPool) and on any other failure. result.json, stations.csv and <config>-calibrated.json are not written, and the documentation says so ("An interrupted run": what needs the finished search is not written). None of the three needs the finished search or a model run, though: the best record already carries the parameters, the objective, the NSE and the station_metrics of every station (_worker.py#L336-L384); only best_<variable>.csv runs the model again. A user who interrupts a calibration after hours, or whose worker was killed for memory, reconstructs the best candidate from the CSV by hand.
With the MODFLOW coupling of #357 a dead worker stops being exceptional: with dis.nstp > 1, a candidate whose solver fails ends its worker process, and with it the whole calibration.
Proposed solution
- On any early end after the first recorded evaluation, after consolidating the table, write
result.json,stations.csvand<config>-calibrated.jsonfrom the best record. No model run on the way out, since the pool may be broken:best_<variable>.csvis not written andartifactslists only what was. result.jsongains aterminationfield:finishedfor a search that ran to SciPy's end,interrupted(KeyboardInterrupt),worker_died(BrokenProcessPool) anderror(any other exception, described inmessage).successkeeps the optimizer's meaning and isfalseon an early end;nitis the last generation the callback saw andnfevthe number of records. This field is the vocabulary the stopping rules extend.- A finished search writes
result.jsonbefore the re-run of the best candidate and rewrites it withbest_seriesonce the re-run succeeded, so that a failed re-run leaves a complete summary minus that file instead of no summary (today "result.json not written at all", "Artifacts of a run"). - The exit codes and the messages of the command stay (2 on an interruption, 1 on a dead worker); the message names
result.jsonnext to the table.
Tests that prove it: the existing interrupted and crashed search tests (_record_then_raise(KeyboardInterrupt()), BrokenProcessPool) assert the three files and the termination value; a search that recorded nothing leaves no summary; a failed re-run of the best candidate leaves result.json without best_series.
Documentation: "An interrupted run", the result.json field list, the note on the failed re-run; changelog.
Alternative solutions
- Run the best candidate once more on the way out to write
best_<variable>.csv. Rejected: after aBrokenProcessPoolthe pool cannot be trusted, and a user who pressed Ctrl+C does not want one more model run.
Additional context
Part of the calibrator robustness work assessed on 2026-09-24. The termination field is born here; #360 (stopping rules) and #361 (resume) build on it. Related: #358, #362. Whoever lands after #357 rebases runner.py.
Activity