idea: H3 audio-hold in the denoise loop + refine pass after latent upscale (follows #471/#500)

[dkackman]

Idea, not approved work. Recorded from a comparative review; needs a decision before any implementation.

Gap

#471 shipped the H3 latent upscale (dw/tasks/h3_latent_upscale.py), but the refine pass was deferred (docs/proposals/complete/h3-latent-upscale-complete.md §Deferred), and the #500 gate judged upscale-only output "close but soft".

Idea

Two linked pieces:

1. Audio-hold in the H3 denoise loop ("audio drive"). VAE-encode a fixed audio track (resample to the audio VAE rate), fit it to the template audio latent (crop/pad), and keep it un-denoised: per-stream mask of 1 for video, 0 for audio, so only video is sampled to the fixed audio. Mux the original waveform at the end, not the VAE round-trip.
2. Refine pass on the upscaled latent. Upscale (pixel or latent), concat the stage-1 audio latent held as in (1), re-noise to a low sigma and run a few steps. Their shipped settings: denoise ~0.2, 4-5 steps; stage 1 at 0.4-0.5 MP with the turbo LoRA. A 3-pass variant repeats stage 2 at 2 MP.

Possible value

• Closes the "soft" gap that kept the #471 stage 2: minimax/upscale-preview template + skill line (gated on Don's stage-1 comparison) #500 path from shipping.
• Audio-hold on its own makes music-video and dialogue shots generate to the real track. Today chain-matched-to-audio only passes the slice as a reference and H3 still generates its own audio latent. That should tighten lip sync.

Costs / open questions

• Needs a dw-owned denoise block: diffusers' modular_pipelines/minimax_h3/denoise.py steps audio_latents[num_condition_audio_rows:], so it must either skip the audio scheduler step or re-impose the encoded audio after each step.
• Cost estimate for the refine stage (the validate_workflow: templates/minimax/music estimate drops to basis unknown when audio_duration (a ceiling) is lowered #440 cost-driver story).

What would unpark it

A lem A/B on the #500 workflows: upscale-only vs upscale + 5-step 0.2 refine.

Their files: VRGDG_MiniMaxH3AudioDrive.py, Workflows/UsedForUIDoNotTouch/minimax_*2pass*/3pass*_api.json, scripts/build_minimax_h3_ref2va_2pass_audio_api.py.

Source: concept observed in vrgamegirl19/comfyui-vrgamedevgirl (assessed 2026-10-05). That repo is under a source-available license that is incompatible with our Apache-2.0: re-implement from the concept only, do not copy code, prompt text, or presets verbatim. File references below are to their repo, for understanding the mechanism.

[dkackman]

Approved by Don (2026-10-05) for a full build: plan both pieces, audio-hold in the H3 denoise loop and the refine pass after the latent upscale, as one feature. No A/B gate before planning. The lem A/B against upscale-only (and #612's LMS upscaler, if it lands first) belongs in the plan's acceptance cases. Handing to the feature lead for a plan.

[dkackman]

Plan v4: H3 audio-hold + refine pass after latent upscale (#598)

Written by model claude-opus-5-5 via provider anthropic. v1 was a design session; v2 a decompose session (folds in Don's approval of v1); v3 a design session (2026-10-06), which folded in Don's choice of refine schedule (B) from #620, recorded what stages 1 and 4 shipped, and corrected stage 2 against installed diffusers. v4 is a decompose session (2026-10-06): it folds in Don's approval of v3, turning Q1 and Q2 into decisions D7 and D8. Scope is unchanged: no new surface, and the remaining stages are the same two.

Status of the stages

• #598 stage 1: H3 audio-hold blocks in the denoise pipeline (hold_audio) #618, stage 1 (hold blocks, hold_audio): shipped and verified.
• #598 stage 4: hold_audio in music-video and chain-matched templates + lip-sync A/B #619, stage 4 (templates + lip-sync A/B): shipped and verified, on the fallback. The A/B (one seed, sung track) went against hold: the mouth was open at ~3/6 onsets with hold, against 6/6 with the reference, at 19.8 vs 18.3 min. So the plan's named fallback applies: the three templates keep the per-shot AudioReference, hold_audio is opt-in engine surface, and the guide, TASKS.md and SKILL say so with the numbers. D4 is superseded, and D7 makes the fallback final (see Decisions).
• #598 stage 2: H3 refine schedule after latent upscale (refine_strength) #620, stage 2 (refine schedule): not built. Plan v2's schedule was impossible (below). v3 replaced it.
• #598 stage 3: templates/minimax/upscale-refine + refine A/B gate #621, stage 3 (upscale-refine + A/B gate): not built.

Verdict: build the remaining two stages, with the refine template still gated on its A/B

• Value. The hold half has shipped, and its lip-sync value did not appear on the one measured seed. That doesn't touch refine. Refine needs hold only to keep the base pass's own audio, which was generated jointly with that video and already matches it. Refine's value is unchanged from v2: closing #471 stage 2: minimax/upscale-preview template + skill line (gated on Don's stage-1 comparison) #500's "close but soft". Its evidence is still the weaker half. Don's gate timed the upscale preview at 7.8 min and native 768p at 12.7 min, and the eval: H3 community LoRAs - Realism People stacked wins; 4-step v1.2 viable draft; Acc/HyperFlow/FastH3 need loaders #585 Realism LoRA on the native turbo path raised the bar. So the template (stage 3) stays gated on the A/B. The engine half (stage 2) is cheap now that the hold seam exists.
• Build cost. Stages 1 and 4 are spent. About $11-16 remains: stage 2 $6-9, stage 3 $5-7.
• Inertia. As v2 said, minus what has already landed:
  • one more block in h3_blocks.py, the one place dw modifies a modular pipeline (D1), covered by the anchor-pin test;
  • one call argument, refine_strength. There is no MCP tool, so SURFACE_BUDGET is untouched;
  • one template if the gate passes. COMPACT_BUDGET (tests/test_catalog_structure.py:431, 9_550) has about 90 tokens of headroom and needs a raise of about 100;
  • a second owner of an H3 sigma schedule. Option (B) computes the shift-spaced tail in dw, because set_timesteps(sigmas=...) applies no shift. It reads the shift from the loaded scheduler and holds no constants.
• Reversibility. refine_strength is opt-in. A run that doesn't name it is unchanged, and the no-op insertion keeps the cache key. If the gate fails, the template goes and the engine stays as guide surface, as with #471 stage 1: upscale_h3_latents + decode_h3_latents tasks, no refine (the measurement) #499.
• Cheaper alternatives.
  • (a) Stop here. Hold has shipped, and there is no refine. #471 stage 2: minimax/upscale-preview template + skill line (gated on Don's stage-1 comparison) #500's softness stays.
  • (b) A template-only refine: still impossible. PrepareLatentsStep treats a passed latents as pure noise, and SetTimesteps takes no strength.
  • (c) Schedules (A) and (C) from #598 stage 2: H3 refine schedule after latent upscale (refine_strength) #620: rejected by Don in favour of (B). (A) is a single evaluation at 0.2. (C) is ComfyUI's grid-fraction denoise, which starts near σ 0.6 at shift 6 and re-noises far more than detail.
• Overlap.
  • idea (parked): H3 latent-space chunk continuation - blocked on multi-frame keyframes in diffusers #611: its stage 1, #611 stage 1: H3 multi-frame video guides (guides argument, layout step) #648, is now blocked by #598 stage 2: H3 refine schedule after latent upscale (refine_strength) #620, because both edit h3_blocks.py and the anchor-pin test (Don's ordering). If #598 stage 2: H3 refine schedule after latent upscale (refine_strength) #620 is declined, #611 stage 1: H3 multi-frame video guides (guides argument, layout step) #648 goes ahead on #598 stage 1: H3 audio-hold blocks in the denoise pipeline (hold_audio) #618's seam alone.
  • idea: trial Alissonerdx H3 LoRAs + LMS sharpness latent upscaler (aligned-guide LoRAs blocked on #611) #612 (LMS upscaler) adds an A/B arm to stage 3 only if it has landed. It is not a dependency.

Proposed follow-up (Don decides): none.

Problem and non-goals

Problem. H3 can't re-denoise an upscaled latent from a low sigma. (Generating to a fixed soundtrack is solved: #618.)

Non-goals:

• no 3-pass (2 MP) variant;
• no LTX-2 changes;
• no persisted-latents result type;
• no change to downloads_required;
• no re-measure of hold's lip sync (D7);
• nothing taken from vrgamegirl19's repo.

Design (stage 2, corrected)

Correction (v3): plan v2's schedule could not run. v2 said the refine runs "the grid points below σ₀", about 4-5 evaluations at 9 steps, shift 6, strength 0.2. Against the installed MiniMaxH3Scheduler, the 9-point shift-6 grid is 1, .977, .947, .909, .857, .783, .667, .462, 0. No point lies in (0, 0.2], so v2's schedule gave zero evaluations (#620, build session).

The schedule (D6, Don's option B). Start at exactly σ₀ = refine_strength. Take num_inference_steps points from σ₀ down to 0, spaced by the scheduler's shift: linear in the unshifted coordinate u, from u₀ = σ₀ / (shift − (shift−1)·σ₀) down to 0, mapped through σ = shift·u / (1 + (shift−1)·u). That gives num_inference_steps − 1 evaluations. Some values:

shift	num_inference_steps	σ list	evaluations
6	5	0.2, .157, .109, .057, 0	4
6	6	0.2, .166, .129, .089, .046, 0	5

What the installed diffusers requires (from the v3 sweep):

• set_timesteps(sigmas=...) applies no shift and appends nothing. It raises unless the σ list is strictly decreasing and ends at exactly 0.0 (scheduling_minimax_h3.py:160-167; timesteps = 1 − σ[:-1]). So the block applies the shift itself, appends the 0, and refuses a list that collapses (num_inference_steps < 2).
• The audio scheduler needs a matching list. denoise.py:222-241 calls audio_scheduler.step even when every audio row is held. It needs audio_timesteps the same length as the video's, and an exact index_for_timestep match. The block therefore resets the audio scheduler with the same σ list. The values don't matter, because no audio row is stepped, and the block asserts that every audio row is a held condition row before the loop runs.
• What it rebuilds: SetTimesteps writes timesteps, audio_timesteps and row_timestep_plan (before_denoise.py:1245-1266). The block rebuilds all three, the plan through build_row_timesteps (:1197-1236), so condition rows keep max(t, keyframe_noise_aug).
• Re-noise only the generated rows. FL2VA and Ref2VA put condition rows at the front of latents (before_denoise.py:1030, 1109-1121). The block re-noises only latents[num_condition_video_rows:]: x = (1−σ₀)·x₀ + σ₀·ε, with ε from the step's generator so a run is seeded.
• Passed latents: a (1, 24, F, H, W) tensor skips the noise draw (before_denoise.py:865-879). That matches upscale_h3_latents' output, provided the step's width and height equal the upscale target.

Placement. The block runs after set_timesteps in each of the three core-denoise sequences, so it goes in just before denoise. insert_audio_hold (h3_blocks.py:325-354) inserts before an anchor, so the refine block anchors on the denoise block. Three things change with it:

• blocks() (:283-288) returns three blocks and caches them, and :333 changes from unpacking two;
• the "already inserted" check (:339) covers the refine block too;
• introspection.py:361-380 and components.py:634-636 already call insert_audio_hold, so get_pipeline_signature shows refine_strength with no further change.

It does nothing unless refine_strength is set (D2).

Validation. The static checks sit beside hold's, in dw/hold_audio.py:68-91 and dw/validation.py:315-317; the runtime check sits beside _with_held_audio in pipeline.py:615-639. A step is refused when:

• refine_strength is not a number in (0, 1);
• it has no latents;
• it has no hold_audio;
• it is not an H3 step;
• num_inference_steps is below 2.

Cost (corrected from v2). Refine time scales with num_inference_steps, not independently of it. dw/plan.py:398-409 resets a catalog cost to unknown only for a declared cost driver. So the stage 3 template declares refine_strength in its cost_drivers, along with num_frames, num_inference_steps, width and height. Otherwise a changed strength would be quoted at the template's measured cost. Progress needs no change: the total comes from len(timesteps) (progress.py:87-144), and N points report N−1 steps, which is what the docs already say.

Surfaces decided here:

• Engine: one more no-op block.
• Call argument: refine_strength (D3).
• REST, MCP and the workflow schema: none.
• Trust gate: not widened.
• docs/ARCHITECTURE.md:95 changes "two blocks go in" to three.

Stages remaining

Build order: 2 → 3. Each stage is blocked by the one before it. No cross-feature blockers. #620 blocks #611's #648.

Stage 2 (#620): refine schedule (engine), est. $6-9, deploy: server

• Builds:
  • DwH3RefineScheduleStep as designed above, inserted before denoise in all three core-denoise sequences;
  • the matched audio σ list;
  • the held-audio assertion;
  • re-noising of the generated rows only;
  • validation;
  • the ARCHITECTURE.md row.
• Unit tests:
  • the σ list for (shift 6, 5 points, 0.2) equals 0.2, .157, .109, .057, 0 to 3 dp, and is strictly decreasing and ends at 0;
  • the schedule survives set_timesteps (the block runs after it);
  • the audio timesteps match the video's length;
  • the assertion fires when an audio row isn't held;
  • condition rows are untouched by the re-noise (FL2VA and Ref2VA shapes);
  • the re-noise formula, and a seeded draw that reproduces;
  • the anchor-pin test covers the third block in all three sequences;
  • the refusals;
  • no refine_strength leaves outputs identical (CPU tiny-tensor).
• Docs: rewrite WORKFLOW_GUIDE.md Promoting an H3 take to 768p in latent space:
  • drop the "no refine pass" lines (:1646, :1675);
  • add the refine step: base → upscale_h3_latents → H3 at 1344x768 with latents: previous_result:up, refine_strength: 0.2, num_inference_steps: 5, hold_audio: previous_result:base.audio;
  • put the new JSON after the existing example, because test_modular_output_properties.py:96-102 reads the first JSON block after the heading;
  • keep upscale_h3_latents, pair_audio, previous_result:base.videos and "fps": 24 in the section, for test_server_guides.py:263-278;
  • add a sentence where hold is called opt-in (:1632-1639): refine uses hold to keep the base pass's own audio, which is a different case from lip sync to supplied audio;
  • add the cost note: refine evaluations = num_inference_steps − 1.
• Acceptance intent (MCP, lem). v3 changes the evaluation counts in M-F084:
  1. Runs the guide's workflow. get_guide serves it, and it validates and completes at 1344x768 with 124 frames. The output audio is the base pass's track. get_job_events show 4 denoise steps in the refine step at num_inference_steps: 5, and 5 at 6.
  2. Strength changes how far it starts, not how many steps it runs. At 0.4 and 5 points it still runs 4 evaluations, and moves the picture further from the upscale-only decode than 0.2 does. (v2's "0.4 runs more evaluations than 0.2" no longer holds under (B).)
  3. Changes only detail. At 0.2 it keeps the subject, layout and motion of the upscale-only decode of the same seed, judged side by side with get_output_frames.
  4. Refusals at validate:
     • refine_strength of 0, 1.0, −0.2, 1.5 or "0.2";
     • no latents;
     • no hold_audio;
     • on SD 1.5;
     • num_inference_steps: 1 with refine_strength.

Stage 3 (#621): templates/minimax/upscale-refine + the A/B gate, est. $5-7, deploy: server (catalog) + plugin

• Builds:

  • the template: 544p base with the turbo LoRA, then upscale_h3_latents, then refine at 1344x768 with the 768p LoRA at video_shift 6, num_inference_steps 5 and refine_strength 0.2 (D8), holding the base audio;
  • compliance with test_h3_schedule.py: scheduler.shift = variable:video_shift (12 or 6), audio_scheduler.shift = variable:audio_shift (3), LoRA alpha = variable:lora_alpha defaulting to null, and no 20 steps;
  • cost_drivers including refine_strength;
  • measured cost and vram_estimate, from arm (b);
  • COMPACT_BUDGET raised by about 100;
  • a short minimax-h3 SKILL row. The SKILL is at 10,872 of 12,288 bytes.

• Acceptance intent: the A/B, unchanged from v2. The #471 stage 2: minimax/upscale-preview template + skill line (gated on Don's stage-1 comparison) #500 crowd-faces prompt, seed 42, 124 frames, on lem, with these arms:

  • (a) upscale-only;
  • (b) this template;
  • (c) native video-with-audio-768p;
  • (d) idea: trial Alissonerdx H3 LoRAs + LMS sharpness latent upscaler (aligned-guide LoRAs blocked on #611) #612's LMS upscaler, if it has landed.

  For each arm the tester records minutes and face crops. (b)'s minutes must be within 50% of the template's cost.

  The gate is Don's call. If he says no, the template and SKILL row are reverted before close-out.

Risks

• Held clean audio against a re-noised video. In #598 stage 4: hold_audio in music-video and chain-matched templates + lip-sync A/B #619, held audio drove the picture worse than a reference. Here the audio is the base take's own, and the video starts at σ 0.2 from a picture that already matches it, so the risk is low. Stage 2's acceptance item 3 catches a drift.
• Audio and video lists of equal length despite different shifts. The block uses one σ list for both, and the assertion guards the case where an audio row would be stepped.
• Anchor drift with diffusers: covered by the pin test, now with three blocks.
• Refine shows no gain over native 768p: stage 3's gate catches it.
• Audio VAE VRAM at 768p: measured in stage 3.

Decisions

• D1 (2026-10-05): engine H3 blocks, with an ARCHITECTURE.md row naming h3_blocks.py.
• D2: the blocks are inserted always, as no-ops.
• D3: the arguments are hold_audio and refine_strength.
• D4: superseded (2026-10-06). The decision was that hold replaces the per-shot reference. #598 stage 4: hold_audio in music-video and chain-matched templates + lip-sync A/B #619's A/B went against hold, so the plan's named fallback shipped: the templates keep the reference and hold is opt-in.
• D5: build order 1 → 4 → 2 → 3. Stages 1 and 4 are done.
• D6 (Don, 2026-10-06, on #598 stage 2: H3 refine schedule after latent upscale (refine_strength) #620): refine schedule (B). σ₀ = refine_strength, then num_inference_steps shift-spaced points to 0, which is num_inference_steps − 1 evaluations. Refine cost scales with num_inference_steps.
• D7 (Don, 2026-10-06, v3 Q1): the hold fallback from #598 stage 4: hold_audio in music-video and chain-matched templates + lip-sync A/B #619 is final. The templates keep the reference path and hold_audio is opt-in. A re-measure of hold is out of this feature; it would come back as its own idea if a field report shows reference lip sync failing where hold might help.
• D8 (Don, 2026-10-06, v3 Q2): the stage 3 template uses num_inference_steps 5 (4 evaluations) and refine_strength 0.2. The guide example matches.

Questions

None open. #621's template stays gated on its A/B, and that gate is Don's call.

[dkackman]

Plan v1 approved by Don (2026-10-05).

• Q1-Q4: the defaults are accepted:
  • engine H3 blocks, with an ARCHITECTURE.md row;
  • insert them always, as no-ops;
  • the arguments are named hold_audio and refine_strength;
  • hold replaces the per-shot AudioReference.
• Q5: the order is 1 → 4 → 2 → 3. Hold in the audio-driven templates comes before the refine engine. The stage 4 lip-sync A/B is the bigger value and tests the "lip-syncs poorly when following supplied audio" risk before any money goes into refine.

Two corrections from checking installed diffusers (modular_pipelines/minimax_h3):

1. Hold insertion point. prepare_latents → set_timesteps exist only inside the three core-denoise sequences: MiniMaxH3CoreDenoiseStep, MiniMaxH3FL2VACoreDenoiseStep and MiniMaxH3Ref2VACoreDenoiseStep, the branches of MiniMaxH3AutoDenoiseStep. The hold block has to go into each of them, between prepare_latents and set_timesteps, not onto MiniMaxH3Blocks. The anchor-pin test should cover all three. Ref2VA's prepare-latents also runs a reference-row count check (before_denoise.py:1113). Extend the count after that check, never before it.
2. Refine block placement. MiniMaxH3SetTimestepsStep.__call__ calls scheduler.set_timesteps(num_inference_steps) itself (before_denoise.py:~1245), so a tail schedule set before it is overwritten. The refine block must run after set_timesteps. It has to:
   • reset the video scheduler with sigmas=;
   • rebuild timesteps and row_timestep_plan;
   • deal with the zip(timesteps, audio_timesteps) pairing. The shortened video schedule pairs with the head of the full audio schedule. That is harmless only while every audio row is a held condition row, so assert it rather than relying on it.

Confirmed while checking: target audio rows at t=1.0 are never stepped (denoise.py:235), and the checkpoint is guidance-distilled with no unconditional pass to drop them, so the "no replacement loop" correction stands.