Failed job discards the step cache of its successful upstream steps

[dkackman]

tool/endpoint: run_workflow / validate_workflow — plan.cached_steps / job step cache (engine, surfaced via MCP) repro: run a multi-step workflow where the last step fails (e.g. an invalid final argument). Upstream steps succeed and write their files. Then validate the same workflow again with the same seed. - observe: validate_workflow returns plan.cached_steps: 0, and a re-run re-renders the upstream steps that had already produced files. expected: the step cache entries for the successful upstream steps survive the failed final step; the re-run reports cached_steps > 0 and reuses the prior files rather than re-rendering them. A partial failure should not invalidate cache entries that were already written successfully. actual: cached_steps: 0 on the re-run; four takes re-rendered for nothing. The cache is keyed or invalidated in a way that discards successful-step entries when the overall run fails. impact: every failed job that had expensive successful steps upstream costs a full re-render on retry — the exact waste the step cache exists to prevent (see regression case C-F012 in regression-suite-complete.md).

[dkackman]

triage: work — real and not a duplicate (nearest priors #150/#98 are about pipeline release, not the step cache). Not visible from a code read, so the fix session should reproduce on lem before editing: step_cache.put is per step (dw/workflow.py:1279, inside the step loop, before any later step can fail) and nothing on the failure path touches the cache (dw/worker.py error branch only evicts pipelines + _cleanup_between_runs). The only thing that drops entries wholesale is _cleanup_all → step_cache.clear() (dw/worker.py:536), reached on a workflow-identity switch (_load_workflow: ("path", workflow_path) / ("inline", id)) or memory clear. So the candidates to check first: (1) another job with a different identity ran on lem's shared FIFO queue between the failed run and the retry — that clears the cache by design and is not a failure bug, though it would still be worth the probe saying so; (2) _saved_files_exist missed — did the failed run's directory get pruned/deleted before the retry; (3) probe_cache (Workflow.cache_hits) being handed a different output_dir than the job — entries are keyed on the output root, so a validate from another workspace answers 0 while a re-run would still hit. Ask the tester for the job ids of the failed run and the retry if the lem history doesn't make the sequence obvious. Triaged by model opus via provider anthropic.

[dkackman]

Could not reproduce on lem, in either form the issue describes. (model sonnet, provider anthropic)

Tried the literal repro: a 2-step seeded workflow where step 1 (main, a real SD1.5 pipeline call) succeeds and writes its file, then step 2 fails with a genuine in-process runtime error (diffusers' own height/width divisible-by-8 check — not an upfront path/schema failure; step 1 has already executed and saved). Confirmed via the manifest that step 1's file was actually written (c184-repro4/.../test_image-0.0.jpg).

• Re-validate_workflowing the same inline definition with the same seed immediately after: plan.cached_steps: 1 — the cache survived.
• Actually exercising the described "re-run" path via rerun_job on the failed job (not just a fresh submission with the same seed): the new job's manifest shows "step": "main", "reused": true — the cache was hit, nothing re-rendered.

So on current develop (deployed on lem, includes PR #183's merge), a graceful/caught final-step failure does not discard the upstream step cache, for either "validate again" or "hit rerun."

Per the triage comment's own candidates, this leaves the more likely explanation as something not reproducible from a clean, isolated test:

• an actual worker-process crash (OOM kill, CUDA/native segfault) between the failed run and the retry — the step cache is an in-memory singleton in the worker subprocess (dw/step_cache.py, dw/repl_worker.py's WorkerManager), and a crash-and-respawn would silently wipe it with no code defect involved; not safely testable by deliberately crashing lem's shared worker
• something specific to the original failed run/retry's actual output_dir or run identity that a synthetic repro doesn't reconstruct

Requesting the job ids (or timestamps) of the original failed run and the retry that showed cached_steps: 0, per the triage comment's own suggestion — that would let me check the worker's logs/history around that window for a crash-and-respawn, or confirm output_dir/workspace identity matched between the two.

[dkackman]

cannot repro