refactor[next]: add stage observability and Toolchain.translate

[egparedes]

Description

Fourth PR of the otf-toolchain-split stack (on top of #2741). This one gives the toolchain a sanctioned way to observe and partially run its pipelines, and then removes the one consumer that previously had to reach into them. It implements the "Stage observability" decision recorded in ADR 0027 (added in #2741).

What's new

• otf.workflow.stage_hook(name, artifact) — an event hook on the existing instrumentation.hook_machinery, emitted after each step by the two named pipelines (backend.Transforms and recipes.OTFCompileWorkflow, through the generic MultiWorkflow / NamedStepSequence loops) and by Toolchain.translate. Artifacts are passed opaquely; nothing is formatted or stringified at the emit site. With no subscriber the hook body is empty, so callbacks is () and an emit costs ~160 ns (about 1 µs per seven-step program compile).
• GT4PY_DUMP_STAGES=<dir> (config.DUMP_STAGES) plus the instrumentation.stage_dump subscriber, writing each stage artifact to <dir>/<program>/<NNN>_<step>.<ext>. It registers at import time when the variable is set, so a plain gtx.program run dumps its stages with no user code; enable() / disable() cover programmatic and test use. Files are created exclusively with an index bump on collision, so concurrent compilation workers and same-named programs never overwrite each other. Serialization reuses what the repo already has (the iterator pretty printer, foast_pretty_printer, inspect.getsource, the real generated source_code) and falls back to repr.
• Toolchain.translate(definition, compile_time_args) — the sanctioned partial run: frontend plus the translation step, stopping before bindings and compilation. It narrows to the standard OTFCompileWorkflow shape and raises NotImplementedError naming the toolchain and the offending backend type otherwise. Per-call step options are deliberately not offered — reconfiguration stays composition-time via dataclasses.replace, per ADR 0011's own rule.

The dace __sdfg__ reach-in is gone — and it was hiding a bug

The old code (runners/dace/program.py) duck-typed through backend.backend.translation, unwrapped the CachedStep by hand, called the mixin .replace(...), and used object.__setattr__ to write into the stage returned by Transforms.past_to_itir and its frozen CompileTimeArgs. Since past_to_itir is an in-memory cached step, that stage was the cache entry — and the __sdfg__-shaped input and the compile()-shaped input produce the same cache key (verified: both a54d493f8ee25792).

So calling __sdfg__ and then compile(offset_provider=<mesh>) on the same program hands the compile path a cached stage whose args.offset_provider has been overwritten with offset-provider types, stripped of the runtime connectivity tables. That fails hard at iterator/ir_utils/domain_utils.py:231 (assert common.is_neighbor_table(connectivity)) — and under python -O, which gt4py actively recommends, the assert disappears and you get silently wrong domain inference instead.

It's replaced by a dace-owned translate-only toolchain variant built with dataclasses.replace plus Toolchain.translate, which leaves the cache pristine. This is the one intentional behavior change in the PR, it is a bug fix, and test_sdfg_conversion_does_not_mutate_gtir_cache guards it (verified to fail against the old code). The pre-existing test_halo_exchange_helper_attrs and test_orchestration.py pass unmodified — that's the equivalence check for everything else.

Removing the reach-in also drops the last non-doc consumer of the workflow mixins' .replace, which is what unblocks the pipeline simplification in the next PR.

Notes for review

• ProgramSource.source_code is not always a str. dace stores SDFG JSON as a dict, despite the annotation. The dump subscriber therefore serializes non-str source as pretty JSON. Worth knowing independently of this PR: under the default BUILD_JOBS_MODE=PROCESS an exception in the compile pipeline surfaces only at wait_for_compilation(), so this kind of mismatch fails silently and late. Tightening that annotation belongs in its own change.
• stage_dump warns rather than raises on any failure. An observational debug subscriber must never turn a working compile into a failing one — especially when the failure would surface out of context from a worker process. Push back if you disagree.
• Stage names are now de-facto observable API (func_to_past, past_lint, field_view_prog_args_transform, past_to_itir, translation, bindings, compilation). The next PR freezes them in explicit __call__ methods.
• Dumping from compilation workers. Under the default BUILD_JOBS_MODE=PROCESS the compile pipeline runs in a spawned worker. The GT4PY_DUMP_STAGES env-var route works there (the worker inherits the environment and registers at import). Setting config.DUMP_STAGES programmatically and calling stage_dump.enable() does not reach workers, so it dumps frontend stages only — enable() now warns about this. Making workers register too is a follow-up, not something I wanted to widen this PR with.
• Monolithic backends (roundtrip) have no OTFCompileWorkflow, so they dump frontend stages only and translate() raises. That's per ADR 0027.
• ADR 0027 describes the fix as "a dace-owned translate-only step"; the implementation builds a translate-only toolchain variant and calls Toolchain.translate, which keeps the dace side from needing to know the pipeline's internals at all.
• No fingerprinted dataclass gains a field, so no cache keys rotate.

Requirements

• All fixes and/or new features come with corresponding tests. (New unit + integration tests for the hook, the dump subscriber, translate, and the dace cache-mutation regression.)
• Important design decisions have been documented in the appropriate ADR inside the docs/development/ADRs/ folder. (ADR 0027, added in refactor[next]: rename Backend to Toolchain and align field names #2741, records this decision.)

https://claude.ai/code/session_01R8zRtFMhdJ8c96XJYCXkRk