Repository navigation
Add a sensitivity check that runs one simulation per free parameter before a calibration #362
Description
Description
A parameter reaches the simulation and may still have no effect on the compared variable, and the search then spends a dimension and its share of the population on it:
x, the flow recession coefficient, enters only the routed runoffarn(_dynamic_model.py#L535-L538); a calibration with--variable rnfsearches it for nothing.- In a coupled run (Add the optional MODFLOW groundwater coupling #357) the baseflow comes from MODFLOW and
alpha_gwhas no effect; the documentation of the coupling tells the user to--fixit by hand. - A candidate can fail for a reason every candidate shares that the pre-checks cannot see (under the
zonesaggregation the station ids are not known before the run; a MODFLOW setup that never converges), and with the defaults the search spends 128 runs before the first generation line says so.
That the nine parameters are applied is guaranteed structurally, not by a check: the derived document writes them all (_worker.py#L286-L328), the format 1.0 model is strict with every field required, and the model reads each of them. What nothing says before the search is whether each one moves the compared series.
Proposed solution
rubem calibrate --sensitivity-check(sensitivity_check()in the runner, next tocalibrate()) runs, in the pool of the calibration, the reference configuration (the starting point of the search) and one run per free parameter with that parameter moved by a fraction of its bound (a quarter of the range, toward the side with room), clipped to the bound and checked withis_admissible(movingw_1up can leavew_3negative; the weights are moved withw_3derived). N + 1 runs, in parallel; the MODFLOW parameters named in--bound(Add the optional MODFLOW groundwater coupling #357) join like the others.- Each moved run is compared with the reference on the series the objective compares (aligned steps, the stations of the objective): maximum absolute difference and its value relative to the reference, plus the NSE of both. A parameter whose series is identical is reported as having no effect; a run that fails is reported with its error and does not end the check.
sensitivity.jsonin the run directory: the reference (parameters, NSE, station metrics), one entry per parameter (reference value, moved value, maximum absolute difference, relative difference, NSE, error), and the listswithout_effectandfailed. One log line per parameter and a closing warning naming the parameters without effect, with the advice to--fixthem.- The command stops after the check (exit 0) and writes nothing else; the search is a separate invocation with the
--fixthe check suggested.
Tests that prove it: on the synthetic dataset x has no effect on rnf and an effect on arn; in a coupled configuration alpha_gw has no effect; the moved values are admissible; a moved run that fails is recorded and the check ends normally; sensitivity.json is written and the command exits 0.
Documentation: a "Sensitivity check" section in the calibration page, before "The search"; changelog.
Alternative solutions
- Run the check and continue into the search in one invocation. Left out to keep the option orthogonal to the stopping rules (Let a calibration stop on a tolerance, on a stalled efficiency and on a generation that failed entirely #360) and the resume (Resume a calibration from the population of its last generation #361); can be added later as a flag.
- A Morris (elementary effects) screening with several trajectories. More runs and more information; not needed for the question "does this parameter move the series at all".
- A runtime assertion in the worker that the loaded configuration carries the candidate's parameters. Left out: the strict schema already refuses a wrong or missing key, so the assertion would only guard against a future format change.
Additional context
Part of the calibrator robustness work assessed on 2026-09-24. Independent of #358, #359, #360 and #361.
Activity