idea: fit_to_model / restore_to_source - exact size and frame-count round-trip for v2v templates

[dkackman]

Idea, not approved work. Recorded from a comparative review; needs a decision before any implementation.

Gap

Our video-to-video templates refuse frame counts off the model grid, and handle aspect ratio inconsistently: upscale-clip centre-crops while refine-clip stretches. Users must pre-conform their footage.

Idea

A fit_to_model / restore_to_source task pair:

• Fit:
  • Resize to the model bucket by stretching, cropping to fill, or letterboxing, and record what was done.
  • Pad the frame count to the grid.
• Restore:
  • Crop out the letterbox and resize back to the exact source width and height.
  • Reconcile the frame count: tolerate a small difference, and keep the original tail frames the model dropped.
  • Optionally blend with the original (orig*(1-s) + out*s).

Optional extension: guided enhancement anchors

• Pick every Nth frame (16 recommended), always including the first and last.
• Enhance each anchor with an image model (Z-Image img2img).
• Feed the anchors to LTX v2v as keyframe conditions; their strengths are cond 0.5 and guiding 0.2.
• LTX2ConditionPipeline already takes conditions with latent indices. Their i % 8 == 1 index dodge is a ComfyUI quirk, not needed here.

Their files: VRGDG_VideoEnhanceNodes.py, VRGDG_StandaloneVideoEnhancerNodes.py.

Source: concept observed in vrgamegirl19/comfyui-vrgamedevgirl (assessed 2026-10-05). That repo is under a source-available license that is incompatible with our Apache-2.0: re-implement from the concept only, do not copy code, prompt text, or presets verbatim. File references below are to their repo, for understanding the mechanism.

[dkackman]

Approved by Don (2026-10-05) at priority:2, for the fit_to_model / restore_to_source pair only. The guided-enhancement-anchors extension is split out to #613 to be assessed on its own. The plan should also make upscale-clip and refine-clip handle aspect the same way. Handing to the feature lead for a plan.

[dkackman]

Plan v4: fit_to_model / restore_to_source (#602)

Plan by the lead agent, model claude-opus-5-5 via anthropic. Scope is the task pair only, plus making upscale-clip and refine-clip handle aspect ratio the same way, as Don's 2026-10-05 comment set it. The anchors extension is #613.

Approved by Don (2026-10-05) as v1/v2; that approval was withdrawn when building stage B showed two parts of v2 don't hold against the code (#632's comment). v3 approved by Don (2026-10-06), with Q2′ as recommended. v4 records that decision (see Decisions); scope is unchanged from v3. Stage A (#631) is verified and stands; v3 adds two small changes to its tasks, built in stage B.

Verdict: build smaller, in two stages

What it gives. Today the caller has to set width/height to match the source clip. Both templates spend most of their descriptions on this, and the two get it wrong in opposite ways:

• upscale-clip silently centre-crops a different aspect ratio. That's diffusers' preprocess_video(resize_mode="crop"), pipeline_ltx2_ic_lora.py:1424.
• refine-clip silently stretches it (pipeline_ltx2_latent_upsample.py:363).
• A short source is lapped from frame 0 in refine-clip (loop_frames). In upscale-clip it gets no reference past its end.

With the pair, the size variables become the model's working size. Any source fits into that size and comes back at exactly its own size (×2 for these templates) and its own frame count.

Value.

• Who gains: Don's driving sessions that run v2v on footage dw didn't make.
• Evidence is thin. In the last 200 jobs, every run of either template was in regression-model-specific. Neither is in any of Don's workspaces, there are no field reports, and the templates are two days old.
• What it rests on: Don's priority:2 approval, and the fact that this is exactly the setting both descriptions warn about.
• Prior rejection: idea: generalized video upscale/resize task #512 (a general resize_video task) was declined. This plan stays narrower than that. It is a model-bound round trip with a record, not a general resize task.

Build cost. About $9–14: stage A $4–6 (spent, verified), stage B $5–8 (v3).

Inertia.

• Surface: two tasks in list_tasks/get_task, one mode domain in dw/task_domains.py, and a record shape that templates come to depend on.
• No MCP tools or REST routes, and the SURFACE_BUDGET in tests/test_mcp_server.py doesn't count tasks.
• Template variables: refine-clip's names are unchanged; their meaning narrows from "the source's size" to "the model's working size". upscale-clip's width/height keep their names and their meaning, the output size (Q2′), so no variable is removed.
• No security boundary moves. Both tasks take in-memory frames, or a video reference through the existing loaders.

Reversibility. Good while the only consumers are templates. The record shape is internal to a fit→restore pair inside one workflow, so it can change without a breaking-change until a user's saved workflow names its fields.

Cheaper alternatives (considered; Don chose the full pair).

1. Only make the two templates agree. Add a crop step to refine-clip so both crop. About $2; gets consistency, but not the exact size, the uncropped content, or the frame count back.
2. Do nothing. The descriptions already tell the caller to match the source.

Overlap.

• idea: guided enhancement anchors - image-model-enhanced keyframes as LTX v2v conditions (split from #602) #613 (anchors) needs its keyframes in the fitted frame, so it builds on this.
• idea: overlap windowing for long video sources (split, process, cosine-blend join) #601 (overlap windowing) is complementary: fit once, window, join, restore once.
• idea: strength-selectable same-size LTX refine (sigma ladder) + per-segment strength decay in chains #606 (strength-selectable refine via a sigma ladder) is the model-native version of the issue's orig*(1-s)+out*s blend. That is why the blend is cut here.

Cut, and what brings each part back:

• The blend option. idea: strength-selectable same-size LTX refine (sigma ladder) + per-segment strength decay in chains #606 covers strength natively. It comes back if idea: strength-selectable same-size LTX refine (sigma ladder) + per-segment strength decay in chains #606 is declined, or if a field report asks for a weaker refine that idea: strength-selectable same-size LTX refine (sigma ladder) + per-segment strength decay in chains #606 can't give.
• "Keep the original tail frames the model dropped". Splicing unprocessed frames after processed ones shows a visible seam, and with the frame handling below nothing is dropped below the source length. It comes back on a field report that wants a long source refined only in its head.
• Adopting the pair in restore-deblur/restore-decompression. They have the same silent crop. A separate follow-up (Q3).

Problem and non-goals

Problem. v2v templates make the caller conform the footage, and the two 2x templates disagree on how they do it.

Non-goals:

• general-purpose resizing (idea: generalized video upscale/resize task #512 declined that);
• deriving width/height/num_frames from the source at run time;
• MCP or REST changes;
• the anchors (idea: guided enhancement anchors - image-model-enhanced keyframes as LTX v2v conditions (split from #602) #613).

The design

The proposal is corrected below where the code disagreed with it.

Task fit_to_model(video, width, height, num_frames, mode="letterbox", downscale=1)

What it does:

• Takes a video argument: an in-memory frame result, asset: or output:.
• Resizes every frame into width×height using bicubic resampling with antialiasing. mode is one of:
  • stretch: resize to fill the frame exactly.
  • crop: scale to fill and centre-crop.
  • letterbox: scale to fit, then centre it on black.
• Sets the frame count to exactly num_frames:
  • a longer source is cut to its first num_frames frames;
  • a shorter one is padded by holding its last frame, not by lapping to frame 0 as loop_frames does.
• downscale (v3, built in stage B): an optional positive integer, default 1. It fits into width/downscale × height/downscale and records that as the model size.
  • It refuses, naming both values, when width or height isn't divisible by it.
  • Why: LTX2InContextPipeline takes width/height at the output size and derives its reference as width // reference_downscale_factor (pipeline_ltx2_ic_lora.py:1403-1404). Workflow JSON can't divide a variable, so the task does it. upscale-clip passes downscale: 2, matching its reference_downscale_factor: 2.

What it returns: a dict, so later steps read it as previous_result:<step>.video and previous_result:<step>.fit (the precedent is outpaint.json's mask_image.bordered_image).

• video: the frames, float [0,1], with the source's fps.
  • v3, built in stage B: it becomes a float32 numpy.ndarray subclass carrying .fps (in dw/media_types.py, alongside AudioVideo), instead of an AudioVideo.
  • Why: nothing in the engine unwraps an AudioVideo handed to a pipeline argument or a condition's frames, and previous_result:fit.video.frames doesn't resolve (get_artifact_properties, dw/result.py:217, reads one level). An ndarray passes diffusers' own isinstance(np.ndarray) checks as-is, and stays float, which LTX2ReferenceCondition needs (ltx2/reference-sheet: succeeds with no warnings but every frame is blocky mush; default asset:reference_sheet.png missing #444).
  • The contract stage A's cases test (video, float [0,1], source fps, saved like any video) doesn't change.
• fit: the record:
  • mode;
  • source_width, source_height, source_frames;
  • model_width, model_height, model_frames;
  • content_box (x, y, w, h): where the source sits in the model frame. In crop mode, source_box is the part of the source that was kept.

Correction to the proposal: it does not pick a bucket or a grid.

• width, height and num_frames stay template variables that the task reads.
• The reason: variable_constraints (dw/variable_constraints.py:267,362), cost_drivers and the VRAM estimate (dw/plan.py:318, dw/vram_estimate.py:57) only see variables. A size or frame count computed at run time would lose the 32n / 8n+1 refusals and the cost quote.
• So "pad the frame count to the grid" means padding a short source up to the caller's 8n+1 num_frames. Restore then trims the padding back.
• No snap: the existing 8n+1 refusal stays as it is.

Refusals, before any work:

• width/height/num_frames not positive integers;
• unknown mode;
• an empty video.

Task restore_to_source(video, fit)

What it does:

• Scale: inferred as video.width / model_width. It must equal video.height / model_height, otherwise it refuses with both sizes named. So the 2x templates need no scale argument. This is a correction: the proposal assumed restore always goes back to 1×.
• Size:
  • Letterbox: crops content_box × scale, then resizes to source_width × scale × source_height × scale.
  • Stretch: resizes straight to that size.
  • Crop: the cut edges don't exist in the output, so it returns the kept source_box at the source's pixel density (× scale). It is exact in density, smaller on the cropped axis, and the description says so.
• Frames: trims to min(source_frames, frame count of video). Padding is removed. A pipeline that floored the count returns what it made, and the record says how many frames there should have been.
• fps: passes through.
• Refuses: a record missing fields or with the wrong types, and a mismatched scale.

Audio

• Neither task touches the soundtrack.
• The templates already take it from the source (normalize_audio) and pair_audio(fit="video") cuts it to the video.
• After restore, the video is the source's length, so the soundtrack now covers the whole output instead of being cut.

Templates (stage B)

Both get the same step sequence:

1. fit (mode from a new fit variable, default letterbox, per Q1);
2. the pipeline, fed previous_result:fit.video;
3. restore (video: previous_result:<pipeline>, fit: previous_result:fit.fit);
4. the existing pair_audio.

Per template:

• upscale-clip: fit to the IC reference size (pipeline_ltx2_ic_lora.py:1403), half the output. Diffusers' crop is then a no-op.
  • Q2′: width/height stay the output size (default 960×544, 32n), passed to the pipeline as today. fit gets width: variable:width, height: variable:height, downscale: 2, so the reference is 480×272. The reference is fed as previous_result:fit.video in place of variable:source_video.
  • cost_drivers and the VRAM estimate keep seeing the output size, as today.
  • v2's ref_width/ref_height (Q2) can't be wired: nothing could pass the pipeline 2 × ref_width. A cross-variable constraint would be an engine change, and reference_downscale_factor: 1 would give a 1× output from a 2× LoRA.
  • Cost: upscale-clip's size variables name the output, and refine-clip's name the working size. The descriptions say which.
• refine-clip: fit to width × height. loop_frames is replaced, and the upsampler's stretch becomes a no-op.

Changed behaviour, stated in both descriptions:

• a different aspect ratio is now letterboxed and restored, not cropped or stretched;
• a short source is held and trimmed back, not lapped or left without a reference;
• the final output is exactly 2× the source and the source's length.

intermediate/ keeps the raw model output in the model frame.

Surfaces:

• Engine: none changed.
• Syntax: none.
• MCP/REST: none.
• Tasks: two new ones (stage A), plus fit_to_model's downscale argument and video type (stage B, v3).
• Templates: a new fit variable on both. No variable is removed or renamed, so no breaking-change (Q2′).
• Plugin: skills/ltx-2.5/SKILL.md text, so the plugin version is bumped.

Stages

Stage A: the two tasks (server)

What it builds:

• fit_to_model and restore_to_source in dw/tasks/ (video_utils.py or a new fit.py);
• register_command entries with summaries and parameter descriptions;
• the mode domain in dw/task_domains.py;
• a docs/TASKS.md Video Processing row each, plus a short "round trip" paragraph.

Unit tests:

• a round trip per mode on synthetic frames: size, frame count, content box, and pixels within tolerance for letterbox and stretch at 1× and 2×;
• each refusal;
• discovery (tests/test_task_discovery.py, presence tests).

Deploy: server restart. Estimate: $4–6.

Acceptance intent (over MCP):

• Discovery: list_tasks lists both, and get_task shows their arguments and mode's three values.

• Letterbox round trip: with an uploaded 640×480, 50-frame, 24 fps clip, a utility workflow runs fit_to_model(width=512, height=288, num_frames=57, mode=letterbox).

  • The fitted video (saved) is 512×288 with 57 frames, with black bars at the sides.
  • Restoring it gives 640×480, 50 frames, 24 fps, with the bars gone (the frames match the source by eye in get_output_frames).

• Scale inferred: the same round trip with a 2× resize in between (an existing per-frame resize_rescale step) restores to 1280×960.

• Stretch: restores to 640×480.

• Crop: restores to the kept region at source density. A 640×480 source into 512×288 gives 640×360.

• Long source: a source longer than num_frames comes back at num_frames frames.

• Refusals:

  • an unknown mode;
  • zero or negative sizes or frame counts;
  • a restore whose input size isn't a uniform multiple of the model size;
  • a malformed fit record.

  Each names the argument, and fails at its step without loading a pipeline.

Stage B: rewire upscale-clip and refine-clip (server + plugin)

Needs stage A.

What it builds:

• in dw/tasks/fit.py (v3):
  • fit_to_model's downscale argument, with its refusal and its get_task description;
  • its video as the fps-carrying float ndarray, so previous_result:fit.video feeds a pipeline argument or a condition's frames directly;
  • unit tests for both, including one that realizes previous_result:fit.video into a pipeline's arguments and a condition's frames through the real argument path, not a mock;
• the step sequence above, in both templates;
• the fit variable (default letterbox);
• for upscale-clip, downscale: 2 on fit, with width/height unchanged;
• rewritten descriptions and summaries, within the COMPACT_BUDGET in tests/test_catalog_structure.py:411;
• workflows/templates/ltx2/README.md, plugins/dw/skills/ltx-2.5/SKILL.md (and a plugin version bump), docs/RECIPES_24GB.md:186.

Tests to update: these pin today's shape.

• tests/test_ltx2_ic_loras.py:
  • :212 unpacks two steps;
  • :174-185 pins the reference to variable:source_video. Only upscale-clip changes; the restore templates keep that pin;
  • :216 pins pair_audio's input.
• tests/test_catalog_structure.py:566-645 (TestLtxRefineClip pins loop_frames).
• The tests/test_step_cache.py:706-726 hashes, if the configuration changes.

Deploy: server, plus a plugin release. Estimate: $5–8 (v3 adds the task changes).

Acceptance intent (over MCP): two GPU runs, about 3 minutes each.

• Both templates validate clean on a 640×480, 50-frame, 24 fps asset with a soundtrack, using defaults except source_video.
• The final/ output is 1280×960, 50 frames, 24 fps, and its audio is the source's length.
  • The whole field of view survives: get_output_frames compared to the source shows no edge content lost.
  • The last frames are not frame 0 again.
  • There are no black bars.
• intermediate/ shows the model-frame output with bars.
• Source matching the working size: a source already at the working size gives the same output size as before this change.
• downscale (v3): get_task fit_to_model shows it. A utility run with width=512, height=288, downscale=2 gives a 256×144 fitted video, and the record's model size is 256×144. downscale=3 with width=512 is refused, naming both.
• Unchanged refusals:
  • size variables off 32n and num_frames off 8n+1 are still refused at validate;
  • a source with no soundtrack is still refused at source_audio, before any pipeline loads (refine-clip).
• Edge halo (Q1): if the runs show a halo at the letterbox edge, the templates' default flips to stretch within this stage.

Order and dependencies

• A, then B. No other issue blocks either.
• idea: guided enhancement anchors - image-model-enhanced keyframes as LTX v2v conditions (split from #602) #613 would build on A.

Risks

• The IC reference may not take in-memory frames. v2's fallback (save fit.video and reference the file) doesn't work: output: names only earlier runs, and an h264 round trip would lose detail before a 2× re-render. v3 makes fit.video an ndarray, the form refine-clip already feeds the upsampler, and stage B's unit test runs it through the real argument path first. If a condition's frames still won't take it, stage B stops and re-plans; it doesn't bend the engine.
• The ndarray subclass may lose .fps through a numpy op that returns a plain array. Restore reads its fps from the pipeline output or the template's result.fps, as today. .fps only matters for saving fit.video itself, which stage B's tests pin.
• Letterbox bars inside a generative re-render. The IC-LoRA may invent content in the bars, or soften the content edge. Restore crops the bars, but an edge halo is possible. If B's runs show one, the default flips to stretch (Q1).
• Double resampling (fit, then restore) costs a little sharpness on a 1× round trip. For these 2× templates it is outweighed by the 2× resize.

Follow-ups (filed to Don after stage B)

• After stage B's GPU runs confirm letterbox holds up under a generative model, file adopting the pair in restore-deblur and restore-decompression (Q3).
• If stage B leaves loop_frames with no template consumer, file its removal as a proposal.

Decisions (Don)

• Q1. Default mode (2026-10-05): letterbox for both templates. Flip to stretch if stage B's runs show an edge halo.
• Q2. upscale-clip's size variables (2026-10-05): move to ref_width/ref_height with the output at 2×, removing width/height. Labelled breaking-change. Superseded by Q2′: it can't be wired (see Templates).
• Q2′. upscale-clip's size variables (2026-10-06): as recommended. upscale-clip keeps width/height as the output size, and fit_to_model gains downscale (upscale-clip passes 2). No engine change, no breaking-change. The two templates' size variables mean different things (output vs working size), and the descriptions say which. fit.video as the fps-carrying float32 ndarray is approved.
• Q3. restore-deblur / restore-decompression (2026-10-05): a separate follow-up, not in this plan.

[dkackman]

Plan v1 approved by Don (2026-10-05).

Q1-Q3 defaults accepted:

• Q1: letterbox is the default mode. Flip it to stretch if stage B's runs show an edge halo.
• Q2: upscale-clip moves to ref_width/ref_height, with the output at 2x. This is labelled breaking-change.
• Q3: restore-deblur/restore-decompression become a separate follow-up.

Follow-ups: after stage B's GPU runs confirm letterbox holds up under a generative model, file the Q3 issue. If stage B leaves loop_frames with no template consumer, file its removal as a proposal too. Both go to Don.