[qarnot-2] Qarnot for every solver: a Run option each adapter opts into (code_saturne, code_aster, openTELEMAC) #36

Vision

From the dashboard, a user picks Qarnot in the Run dialog and their cases run on their own Qarnot account, monitored live exactly like a local run, whatever the solver: code_saturne, code_aster or openTELEMAC.

Whether a solver offers that button is a decision written in its adapter. One declaration turns Qarnot on for a solver; removing it turns it off. A solver that has not opted in never shows the option. A solver that has opted in is guaranteed by a test to provide everything Qarnot needs, so the button never leads to a launch that fails after the user confirmed a billed run.

Qarnot is the first execution backend, not the last: the mechanism has to work for any backend implementing ExecutionBackend.

Where we start from

PR #34 (feat/qarnot-cloud) builds the foundation:

• the ExecutionBackend contract and its Qarnot implementation;
• the backend choice in the Run dialog;
• live monitoring via observability_globs.

It has been verified end to end with code_saturne on a real account. What it does not do yet:

• No declaration in the adapter. Whether a solver can run remotely is inferred from side effects: run_argv is non-empty and the adapter has a default_docker_image.
• The check lives in Qarnot code, twice. QarnotBackend.submit raises when run_argv is empty (csauto/backends/qarnot.py:200), and maintenance.py:99-110 repeats the checks for doctor --backend qarnot, inside a generic module.
• Every backend is offered to every solver. /api/app_config returns available_backends() whatever the solver, so a code_aster campaign shows Qarnot and fails only at submit.
• Actions ignore the backend. Buttons are derived from the solver alone (feat: derive dashboard panels and actions from solver capabilities #22). A case running on Qarnot still shows Restart and Stop, which [qarnot-1] Run a campaign on the Qarnot cloud #34 documents as unsupported remotely.
• Only code_saturne works. code_aster builds no remote command, and openTELEMAC has no adapter yet ([solvers-2] openTELEMAC adapter #33).

Plan

Step 1: Land #34 cleanly

• Extract first the fixes that also change local runs, into their own PR: registry lock re-entrancy (c0682e2), residual abscissa (cd15e1b), LAST ITER (fb8cc20), diverged run read as a success (400fbc6). A trial cherry-pick onto main only conflicts on CHANGELOG.md and test files. Since Qarnot comes last, do this early: otherwise these fixes wait for the last milestone.
• Review [qarnot-1] Run a campaign on the Qarnot cloud #34 in sections:
  • the contract and runner;
  • registry locking;
  • the Qarnot implementation;
  • secret handling (QARNOT_TOKEN must never reach a payload, a log line or doctor output);
  • API and frontend.
• Rebase on main once the first two milestones have landed. They rewrite solvers/base.py, the adapters, logs.py, residuals.py and web_support.py, which [qarnot-1] Run a campaign on the Qarnot cloud #34 also touches. Rebuild frontend/dist once at the tip instead of resolving hashed bundles by hand. Merge with a merge commit, to keep the atomic history.
• Before merging, run one local code_saturne campaign, to confirm local runs are unchanged, and one test_qarnot_live.py.
• Record two boundary choices in docs/architecture.md:
  • the TIME STEP NUMBER regexes the branch adds to logs.py and residuals.py move into the code_saturne adapter during the rebase, since [core-4] probes.py and residuals.py: let the adapter declare the results layout #29 and [core-5] Split logs.py into a generic log engine and solver-declared patterns #30 will have made those modules generic;
  • the [qarnot] settings in config.py are either accepted residue, or something the backend should declare itself.
• [qarnot-1] Run a campaign on the Qarnot cloud #34 is merged last, as the start of the Qarnot milestone, after the generic core and the solver adapters.

Step 2: The adapter declares its backends

class SolverAdapterBase(ABC):
    # Execution backends this solver has been made to work on, beside the
    # local runtimes it always supports. Empty by default: a solver is never
    # offered a backend nobody has tested it on.
    execution_backends: ClassVar[frozenset[str]] = frozenset()

class CodeSaturneAdapter(SolverAdapterBase):
    execution_backends = frozenset({"qarnot"})

Each backend states what it requires of a solver, in its own module:

class ExecutionBackend(Protocol):
    def solver_problems(self, adapter: SolverAdapter) -> list[str]: ...

For Qarnot, the requirements are:

• a docker image;
• a non-empty remote command;
• observability_globs covering the files that feed every tab the adapter declares ([core-1] Dashboard tabs are declared by the solver adapter, and by nothing else #38), without which those tabs stay empty during a remote run.

This replaces the check in submit and the Qarnot block in maintenance.py.

A test ties the two together: for every adapter and every backend it declares, solver_problems returns nothing. A declaration the adapter cannot honour fails CI instead of a user's launch. This follows 06535bf ("assert analytics parsers imply their declarations").

Step 3: The button follows the declaration

• /api/app_config returns only the backends the campaign's solver declares, intersected with those available on the machine. The Run dialog shows the Qarnot option only then.
• The launch route refuses an undeclared backend with a clear 400, as feat: derive dashboard panels and actions from solver capabilities #22 does for unsupported actions.
• csauto doctor --backend qarnot tells "this solver has not opted into qarnot" apart from "it opted in but a check fails".
• Actions become a function of (solver, backend): Restart and Stop are hidden for a case running on Qarnot until they work there. Dashboard tabs do not: they stay declared by the adapter alone ([core-1] Dashboard tabs are declared by the solver adapter, and by nothing else #38).

Behaviour for code_saturne users is unchanged by steps 2 and 3.

Step 4: code_aster on Qarnot

Requires #32 first:

• The F mess bug. observability_globs points at RESU/LOGS/run_solver.log, which the shipped example never writes, because its .export declares F mess output.mess. Without the fix, live monitoring pulls back nothing and every run ends FAILED.
• RESU in shared_dir_names. The Qarnot upload plan ships shared directories as one campaign-wide bucket, so every case would read and write the same RESU.

Then:

• A remote command running run_aster on the case's .export inside the container. Today run_argv returns [] because the local launch is composed in build_run_command. Either build a real argv there, or add a remote_argv() defaulting to run_argv. Decide here.
• A docker image with code_aster, usable by Qarnot's docker-batch profile.
• prepare_remote_case, if .export paths no longer resolve once the shared directories sit inside the case.
• Opt in, and pass test_qarnot_live.py on a code_aster case.

Step 5: openTELEMAC on Qarnot

Requires the adapter from #33. Write it with remote execution in mind from the start:

• a real run_argv, to avoid the code_aster shape;
• observability_globs for the listing and the result files worth watching live;
• a docker image;
• prepare_remote_case if the steering file references paths outside the case.

Then opt in, and pass the live test.

Order and dependencies

#34 merged ──► step 2 + 3 (code_saturne) ──► step 4 (code_aster, after #32)
                                         └─► step 5 (openTELEMAC, after #33)

Steps 4 and 5 are independent of each other. This issue is the Qarnot milestone, the last of three: it starts once Generic core (#26, #38) and Solver adapters (#32, #33) are done, so steps 4 and 5 find both adapters ready.

Out of scope, to track separately

• Restart and Stop on a Qarnot case. Both hit the same read-only results tree, as described in [qarnot-1] Run a campaign on the Qarnot cloud #34.
• Porting local and Slurm to ExecutionBackend.
• A case relaunched locally keeps its backend field, so it carries both a pid and a backend name.
• More launch options (memory, CPU model), per-campaign defaults, instancecount > 1.
• Deleting a campaign's buckets when it is finished.
• Restricting backends per campaign in csauto.toml, for example a site that forbids the cloud.

Definition of done

• The Qarnot option appears in the Run dialog if and only if the campaign's solver declares it in its adapter, and the API refuses it otherwise.
• A test proves every adapter's declaration is honoured by the backend's requirements.
• No Qarnot-specific solver check remains in maintenance.py or in QarnotBackend.submit.
• code_saturne, code_aster and openTELEMAC each pass test_qarnot_live.py on a real account and are reported DONE with their results on local disk. If one of them cannot, this issue records why.
• Local runs behave as before for every solver.