Skip to content

Write the result of a calibration that ends before the search finishes #359

Open

Description

@soaressgabriel

Description

When the search ends early, the runner consolidates evaluations.csv and re-raises (runner.py#L567-L582): on Ctrl+C, on a worker death (BrokenProcessPool) and on any other failure. result.json, stations.csv and <config>-calibrated.json are not written, and the documentation says so ("An interrupted run": what needs the finished search is not written). None of the three needs the finished search or a model run, though: the best record already carries the parameters, the objective, the NSE and the station_metrics of every station (_worker.py#L336-L384); only best_<variable>.csv runs the model again. A user who interrupts a calibration after hours, or whose worker was killed for memory, reconstructs the best candidate from the CSV by hand.

With the MODFLOW coupling of #357 a dead worker stops being exceptional: with dis.nstp > 1, a candidate whose solver fails ends its worker process, and with it the whole calibration.

Proposed solution

  • On any early end after the first recorded evaluation, after consolidating the table, write result.json, stations.csv and <config>-calibrated.json from the best record. No model run on the way out, since the pool may be broken: best_<variable>.csv is not written and artifacts lists only what was.
  • result.json gains a termination field: finished for a search that ran to SciPy's end, interrupted (KeyboardInterrupt), worker_died (BrokenProcessPool) and error (any other exception, described in message). success keeps the optimizer's meaning and is false on an early end; nit is the last generation the callback saw and nfev the number of records. This field is the vocabulary the stopping rules extend.
  • A finished search writes result.json before the re-run of the best candidate and rewrites it with best_series once the re-run succeeded, so that a failed re-run leaves a complete summary minus that file instead of no summary (today "result.json not written at all", "Artifacts of a run").
  • The exit codes and the messages of the command stay (2 on an interruption, 1 on a dead worker); the message names result.json next to the table.

Tests that prove it: the existing interrupted and crashed search tests (_record_then_raise(KeyboardInterrupt()), BrokenProcessPool) assert the three files and the termination value; a search that recorded nothing leaves no summary; a failed re-run of the best candidate leaves result.json without best_series.

Documentation: "An interrupted run", the result.json field list, the note on the failed re-run; changelog.

Alternative solutions

  • Run the best candidate once more on the way out to write best_<variable>.csv. Rejected: after a BrokenProcessPool the pool cannot be trusted, and a user who pressed Ctrl+C does not want one more model run.

Additional context

Part of the calibrator robustness work assessed on 2026-09-24. The termination field is born here; #360 (stopping rules) and #361 (resume) build on it. Related: #358, #362. Whoever lands after #357 rebases runner.py.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions