TASKS.md

Task Commands

Tasks are utility operations that run outside of pipeline inference. Use them for image preprocessing, data gathering, and other non-model operations.

{
    "name": "step_name",
    "task": {
        "command": "command_name",
        "arguments": { ... }
    },
    "result": { "content_type": "image/jpeg" }
}

Any task that runs a model accepts a "device" argument to pin where it runs - useful for keeping a helper model (a captioner, an upscaler) off the accelerator a loaded pipeline is using, or on a second one. "device" is listed on every task's schema, since it is always safe to pass: a task that runs no model - slice_audio, compose_text, and the like - just ignores it.

Task argument schemas are discoverable: GET /api/tasks/{command} on the server returns each command's arguments read from its registered implementation's real signature, the web editor builds task forms from them, and workflow validation flags task-argument typos the same way it flags pipeline ones.

A signature carries no domain, though, so the numbers whose domain is not a judgement call are declared separately (dw/task_domains.py) and validation reports one outside it as an error at its JSON path: a count of frames or seconds to cut, and a sample rate or frame rate, have to be above zero, and an offset to start at zero or above. Those are refused rather than interpreted - num_frames: -10 used to answer with the track minus its last ten frames and target_sample_rate: 0 with the original samples under a 44100 Hz header, both reported as clean successes. The commands refuse the same values at run time, which is what catches one that arrived from a variable: or an earlier step rather than being written in the file.

Image Processing

ControlNet Preprocessors

Generate control images for ControlNet pipelines:

Command	Description
canny	Canny edge detection
canny_cv	OpenCV Canny (alternative)
depth	Depth estimation (DPT)
midas	Monocular depth (MiDaS)
zoe	Zoe depth estimation
zoe_depth	Zoe depth with colorization
leres	Relative depth (LeReS)
normal_bae	Surface normal estimation
openpose	Pose estimation
dw_pose	DW pose estimation
mlsd	Line segment detection
lineart	Line art extraction
lineart_standard	Standard line art
hed	HED edge detection
scribble	Scribble-style edges
pidi	Boundary detection
shuffle	Content-preserving shuffle
teed	TEED edge detection
anyline	Anyline edge detection
sam	Segment Anything
segmentation	Semantic segmentation
depth_estimator	Depth hint generation
depth_estimator_tensor	Depth hint as tensor

All accept an image argument with processing parameters:

{
    "task": {
        "command": "canny",
        "arguments": {
            "image": {
                "location": "https://example.com/photo.jpg",
                "low_threshold": 50,
                "high_threshold": 200,
                "detect_resolution": 1024,
                "image_resolution": 1024
            }
        }
    }
}

Image Manipulation

Command	Description	Extra Arguments
remove_background	Remove image background
resize_center_crop	Crop to a centered square, then stretch to width x height - distorts a non-square target	width, height
resize_resample	Resample to nearest 64px multiple
resize_rescale	Resize to exact dimensions	width, height
resize_bucket	Snap to closest model-native aspect ratio	resolution, ratios, alignment
crop_square	Center crop to square
recenter_crop	Re-frame around a chosen point at a chosen scale, so a series of images registers on one feature; the window may run off the source	center_x, center_y, crop, width, height, fill
add_border_and_mask	Add border with alpha mask
add_border_and_mask_with_size	Border with specific dimensions	width, height
strip_exif	Remove all EXIF/metadata from image
add_watermark	Add visible text watermark	text, position, opacity, font_size, color, margin
get_image_size	Return {width, height} dict

EXIF Stripping

Remove all EXIF metadata, GPS coordinates, camera info, and timestamps from images for privacy-safe preprocessing:

{
    "task": {
        "command": "strip_exif",
        "arguments": {
            "image": "previous_result:input_image"
        }
    },
    "result": { "content_type": "image/png" }
}

Returns a clean copy with pixel data only — no embedded metadata. Useful as a first step when processing user-uploaded images.

Watermark Embedding

Add a visible text watermark to images for responsible AI compliance:

{
    "task": {
        "command": "add_watermark",
        "arguments": {
            "image": "previous_result:generate",
            "text": "AI Generated",
            "position": "bottom-right",
            "opacity": 128
        }
    },
    "result": { "content_type": "image/png" }
}

Argument	Required	Description
text	No	Watermark text (default: "AI Generated")
position	No	"bottom-right", "bottom-left", "top-right", "top-left", or "center" (default: "bottom-right")
opacity	No	Text opacity 0-255 (default: 128)
font_size	No	Font size in pixels, 0 = auto-scale ~3% of image height (default: 0)
color	No	RGB array for text color (default: white)
margin	No	Pixel margin from edges (default: 10)

Aspect Ratio Bucketing

The resize_bucket command snaps an image to the closest model-native aspect ratio, then resizes with 64-pixel alignment. This avoids distortion and ensures the model generates at a resolution it was trained on.

{
    "task": {
        "command": "resize_bucket",
        "arguments": {
            "image": "previous_result:input_image",
            "resolution": 1024
        }
    },
    "result": { "content_type": "image/png" }
}

Argument	Required	Description
resolution	No	Target short-side size in pixels (default: 1024)
ratios	No	Custom list of [w, h] ratio pairs (default: standard SDXL/Flux ratios)
alignment	No	Round dimensions to this multiple (default: 64)

Default ratios: 1:1, 4:3, 3:4, 3:2, 2:3, 16:9, 9:16, 21:9, 9:21

For example, a 1600x900 photo (16:9) at resolution 1024 becomes 1792x1024. A 800x600 photo (4:3) becomes 1344x1024.

Video Processing

Command	Description	Extra Arguments
get_first_frame	Extract first video frame
get_last_frame	Extract last video frame
get_frame	Extract frame at index	frame_index

The frame commands accept videos in any shape a result carries them: PIL frame lists, numpy or torch frame arrays, and audio+video pairs (LTX-2, MiniMax H3). The extracted frame is always a PIL image.

concat_videos

Concatenate videos - and the audio generated with them - into one video. The standalone counterpart of a chained pipeline step's stitching (see "Chained video generation" in the workflow guide):

{
    "task": {
        "command": "concat_videos",
        "arguments": {
            "videos": ["previous_result:shot_1", "previous_result:shot_2"],
            "trim_frames": 1,
            "crossfade_ms": 75,
            "fps": 24
        }
    },
    "result": { "content_type": "video/mp4", "fps": 24 }
}

Argument	Required	Description
videos	Yes	The videos to join, in order - previous_result references, or the path or URL of a video file an earlier run wrote, which is read with the audio muxed into it; an entry may also be a {"location": ...} dict wrapping either. Every video must be the same frame size - unlike a sample-rate mismatch, this task does not resize one for you, so a statically-resolvable (asset:/output:/literal path, wrapped in a {"location": ...} dict or not) size disagreement is refused at validate; a previous_result: or other reference not yet resolved still fails only at run time (#504, #518). To fit the odd video: video_frames to get its frames, resize_rescale to the target size (resize_center_crop squares the frame first and then stretches it, distorting a non-square target), then pair_audio(fit="video") to put its soundtrack back before passing it here (#551)
trim_frames	No	Frames dropped from the head of every video after the first (default: 0)
crossfade_ms	No	Equal-power crossfade at each audio seam, drawn from the trimmed material - no effect when trim_frames is 0, and validation warns when one is written there (default: 75)
audio_bleed_ms	No	How long the outgoing video's tail rings on over the head of the next one, at seams with nothing trimmed to crossfade (default: 0, off)
audio_bleed_gain_db	No	Gain applied to the bled tail before it is added, in dB - negative ducks a tail that would otherwise push the seam over 0 dBFS (default: 0, unchanged)
seam_fade_ms	No	Fade on each side of a seam that gets neither a crossfade nor a bleed - for tonal material, not for a continuous bed (default: 3, just enough not to click)
fps	No	Frame rate of the videos - required to join audio when trimming, and the rate the joined file is written at unless result.fps overrides it
match_levels	No	Even the shots' loudness out before joining - "rms" for perceived level (the measurement get_gallery_metadata reports as mean_dbfs), "peak" for the loudest sample. Off by default
match_levels_dbfs	No	The level match_levels moves every shot to (default: -1 dBFS for peak, -20 dBFS for rms). A shot that would clip at the target is held at -0.5 dBFS peak instead, reported as a match_levels_held warning with a per-shot log event

A video may also be named by path or URL, which is how shots an earlier run already wrote are joined without regenerating them - the file is read with the audio muxed into it, and its track is fitted to the frames' own duration so the codec's block padding does not walk the sound off the picture over a dozen seams. A shot generated in memory through a previous_result: chain gets the same fit, applied where the file is written rather than where it is decoded, so per-shot drift does not accumulate across a cut the way it once did:

{
    "task": {
        "command": "concat_videos",
        "arguments": {
            "videos": [
                "/path/to/outputs/shot_01.mp4",
                "/path/to/outputs/shot_02.mp4",
                "previous_result:shot_03_rerendered"
            ],
            "trim_frames": 0,
            "fps": 24
        }
    },
    "result": { "content_type": "video/mp4", "fps": 24 }
}

Give each video its own entry. One previous_result reference naming a step that produced several videos does not hand them all over at once - it fans the step out over them, one concatenation per video, which is what makes the list form above the way to join a run's shots.

trim_frames and audio_bleed_ms address opposite situations. A chain carries its keyframe forward, so the trimmed head is material that covers the same stretch of time as the outgoing tail and the two can be crossfaded. A cut generates each shot independently, so there is nothing to fade with - and generated shots tend to open on near-silence and end mid-sound, leaving a butt-join that drops a running laugh track or a ringing room into a hole. audio_bleed_ms fills it the way an audience carries across a picture cut: a decaying copy of the outgoing tail is laid over the incoming head, added to whatever is already there, shortening neither side. Reach for more than the gap looks like it needs: a shot's head is silent for longer than the picture suggests, and the bleed has to outlast it. Measured on a five-shot H3 sitcom cut, 700 ms still left a 44 dB hole at the worst seam; 1800 ms brought it to 32 dB and 2500 ms gained almost nothing more, so the dialogue-short template defaults to 1800 and exposes it as audio_bleed_ms:

{
    "task": {
        "command": "concat_videos",
        "arguments": {
            "videos": ["previous_result:shot_1", "previous_result:shot_2"],
            "trim_frames": 0,
            "audio_bleed_ms": 1800,
            "fps": 24
        }
    },
    "result": { "content_type": "video/mp4", "fps": 24 }
}

A bleed works because it copies ambience, which has no pitch and no attacks to give the copy away. It is the wrong tool for anything tonal - a copied musical phrase or half-spoken word reads as a stutter whichever direction it runs. bleed_join checks the outgoing tail's spectral flatness and warns when it looks tonal or speech-like rather than noise-like, so this failure mode surfaces in the job's warnings list instead of only in the mix. When a shot ends on something tonal, either give the cut a continuous bed with slice_audio + loop_audio + mix_audio + pair_audio, which leaves no seam to treat at all, or fade the seam gracefully with seam_fade_ms (a hundred or so milliseconds) and accept the cut. That advice inverts on a continuous bed - a laugh track, room tone - where a longer fade only digs the hole deeper (the same sitcom cut measured 54-59 dB holes with a 250-500 ms fade and no bleed). audio_bleed_ms wins where both are set and there is material to bleed. A bleed covers the gap but cannot fill it: the silence is inside the incoming shot's own head, and the only complete fix is a continuous bed under the whole cut: slice_audio a few seconds of tone out of a shot, loop_audio it to the length of the cut, mix_audio it under the episode and pair_audio it back onto the picture.

Levels are the other thing a cut has to reconcile, and no fade control can touch it. Shots generated independently land wherever the model put them - two shots of one scene, same template, same cast, measured peak_dbfs -2.65 and -12.56 - and each reads as fine on its own, because a shot is only wrong relative to what it is cut against. Butt-joined, that is a 10 dB drop at the cut, and it is not an artifact at the seam that a fade could smooth: it is either side of it. match_levels scales each track before the join - "rms" matches perceived level, which is usually what "make these sound the same" means, and "peak" matches the loudest sample, which is the safer choice on material with big transients. A shot whose gain would clip at the target is held just below full scale and the log says so. Left off - the default, so nothing existing changes - a spread of 6 dB or more across the tracks being joined is reported as a warning rather than passing in silence: on the job's warnings and as a warning event in its stream, not only in the server's log, since the caller who can act on it is the one who asked for the run. The other end of the range warns too: a shot at or below -40 dBFS, or one that needs 20 dB or more of gain to reach the target, is noise floor rather than a quieter performance, and matching it up is reported as match_levels_near_silent. dissolve_videos takes the same pair.

The joined soundtrack is fitted to the joined frames. A track that comes out short of the frame grid - rounding in an input's own track, which otherwise compounds join after join - is padded with silence to it. A pad of a frame or more is a warning (joined_audio_padded_to_frames); less than a frame is rounding, and only logged. The file's AAC encode can then trim the track by a further handful of samples (typically 16-32, under a millisecond), which is logged the same way, or warned as joined_audio_short_after_mux if it reaches a frame. Either way the recorded shots are re-measured against the file as written, so media.shots stays accurate. Both apply to dissolve_videos the same way, and neither task warns about resampling inputs that already agree to a sample_rate the caller pinned.

dissolve_videos

Join videos with a cross-dissolve at every seam, and fade the whole piece in from and out to a colour. Where concat_videos cuts - right for shots that each carry their own sound - this melts one shot into the next, which is what a montage cut to a score wants:

{
    "task": {
        "command": "dissolve_videos",
        "arguments": {
            "videos": ["previous_result:shot_1", "previous_result:shot_2"],
            "dissolve_frames": 12,
            "fade_in_frames": 12,
            "fade_out_frames": 24,
            "fps": 24
        }
    },
    "result": { "content_type": "video/mp4", "fps": 24 }
}

Argument	Required	Description
videos	Yes	The videos to join, in order - previous_result references, or the path or URL of a video file an earlier run wrote, one entry per video as with concat_videos; an entry may also be a {"location": ...} dict wrapping either. Every video must be the same frame size - unlike a sample-rate mismatch, this task does not resize one for you, so a statically-resolvable (asset:/output:/literal path, wrapped in a {"location": ...} dict or not) size disagreement is refused at validate; a previous_result: or other reference not yet resolved still fails only at run time (#504, #518). To fit the odd video: video_frames to get its frames, resize_rescale to the target size (resize_center_crop squares the frame first and then stretches it, distorting a non-square target), then pair_audio(fit="video") to put its soundtrack back before passing it here (#551)
dissolve_frames	No	Frames of overlap at each seam, blended linearly (default: 12). 0 is a hard cut
fade_in_frames	No	Frames over which the first video rises out of fade_color (default: 0)
fade_out_frames	No	Frames over which the last video sinks into it (default: 0)
fade_color	No	The RGB colour the fades come from and go to (default: black)
fps	No	Frame rate of the videos - required to crossfade audio at a dissolve, and the rate the dissolved file is written at unless result.fps overrides it
match_levels	No	Even the shots' loudness out before joining - "rms" or "peak", as with concat_videos. Off by default
match_levels_dbfs	No	The level match_levels moves every shot to (default: -1 dBFS for peak, -20 dBFS for rms). A shot that would clip at the target is held at -0.5 dBFS peak instead, reported as a match_levels_held warning with a per-shot log event

Every seam shortens the result by one overlap, so eight 124-frame shots joined with 12-frame dissolves run 908 frames, not 992 - size a soundtrack slice to the joined length, not the sum. When every input carries audio, the tracks are crossfaded over exactly the seam's span so they stay in step with the picture; when any input is silent the result is, and pair_audio puts a score under it.

Example: dissolve-between-shots.json

join_into_song

Join a spoken scene into a musical number: dialogue shots keep their own audio, and the shots sung after them play over the unbroken song rather than the separate slices each was generated against. The song's entry point is dialogue length - cue_seconds, and a workflow cannot do that arithmetic itself - a hand-computed literal goes stale the moment one dialogue shot is regenerated at another length. This task measures the joined dialogue at run time and places the song from it, so the offset never goes stale:

{
    "task": {
        "command": "join_into_song",
        "arguments": {
            "dialogue": ["previous_result:line_a", "previous_result:line_b"],
            "song_shots": "gather:sung",
            "song": "asset:song.mp3",
            "cue_seconds": 1.5
        }
    },
    "result": { "content_type": "video/mp4" }
}

Argument	Required	Description
dialogue	Yes	The spoken shots, in order, each keeping its own audio - a non-empty list, the same entries concat_videos' videos takes: previous_result references, or the path or URL of a video file an earlier run wrote, each optionally wrapped in a {"location": ...} dict. A shot with no track is filled with silence for its length
song_shots	Yes	The sung shots, in order - a non-empty list in the same shapes as dialogue. Their own audio is discarded; they play over song
song	Yes	The unbroken track the song shots were sliced from - an audio result, a video or audio file's path, or anything carrying .audio and .sample_rate. Its rate is the output's
cue_seconds	No	The song time that lands on the first song shot's frame 0 - the start of the slice that shot was generated against. 0 (the default) starts the song exactly at the cut; above 0 the song enters that long before it, under the last spoken line. Longer than the dialogue is refused - the song would have to start before the film. Must be ≥ 0
dialogue_target_lufs	No	Integrated loudness (BS.1770) each dialogue shot is gained to, with one static gain per shot. Omitted (the default), the shots keep their own levels. A shot too short (under 400 ms) or too quiet to measure is left at its own level, with a dialogue_unmatched warning
duck_delay_ms	No	How long after the song enters the dialogue starts to duck (default: 0). At or past the dialogue's end, nothing ducks. Must be ≥ 0
duck_db	No	How far the dialogue ducks, in dB (default: -12). Must be ≤ 0
duck_ramp_ms	No	The length of the linear ramp into the duck (default: 250). Must be ≥ 0
fps	No	The rate the videos play at, needed only when none of them carries one of its own - a pipeline's frames carry none, a file brings its own. Must be > 0

The timeline, in samples at the song's rate: each dialogue shot's track is fitted to its own frames (trimmed or padded with silence, warning dialogue_fitted_to_frames past a frame's worth of difference), so the joined dialogue ends exactly where the first song shot's frame 0 is - call that sample D. The song is placed at D - cue_seconds * sample_rate, so song time cue_seconds lands on that frame, and it runs to the end of the picture; a song shorter than that warns song_short and is padded with silence rather than looped. The dialogue ducks by duck_db from duck_delay_ms after the song enters, over a linear duck_ramp_ms ramp. The song shots' own audio is discarded, and a dialogue shot with no track of its own is filled with silence for its length rather than skipped, so later shots do not land early.

Frames are joined one for one, so every video must share one frame size and one frame rate: a mismatch is refused rather than resampled, as is a join where no video carries a rate and fps is not given, or an fps that contradicts the rate the videos carry. A step that wrote its video with result.fps hands that rate on, so a shot written at 12 fps from 24 fps frames is a 12 fps shot to the join, as it is when read back with output:.

There is no final normalization here - what level a deliverable sits at is the workflow's to decide, with normalize_audio after this step; the templates that mux to video normalize to -3 dBFS peak, as with any pair_audio mux.

The result is one video/mp4: the dialogue's frames then the song shots', over the mix, with a shot record per input (shot@<key>, named from the step's dialogue then song_shots references, as for_each names a member), each shot's samples measured off the built waveform rather than derived from its frames.

stabilize_video

Remove a generated clip's accumulated framing drift - the slow wander a video model adds over a shot that was meant to hold still:

{
    "task": {
        "command": "stabilize_video",
        "arguments": {
            "clip": "variable:shot_1",
            "smooth": 0
        }
    }
}

Argument	Required	Description
clip	Yes	The video - a frame list, a frame array or tensor, an audio+video pair, or the path or URL of a video file, read with its audio, so a shot an earlier run wrote can be steadied without regenerating it
smooth	No	0 (the default) locks the framing to the first frame, which is what a shot generated from a pinned keyframe wants. A window in frames instead removes only the wander faster than that window, so a slow deliberate camera move survives and the drift around it does not

The argument is clip, not video, on purpose: the engine loads an argument named video itself, as bare frames, which would strip the soundtrack off before the task ever saw it. Frames are shifted back and the result is cropped to the region every frame covers, then resized to the original size; a soundtrack passes through untouched.

It is a stabilization pass, not a format pass. smooth: 0 on a shot with a deliberate camera move fights the move - every frame is shifted back toward the first, cropped and rescaled - and nothing downstream will notice, since the duration, size and sample rate all survive. Run it on a shot that drifts, as its own step; do not run it on every shot before a cut, which is what the assembly templates once did and what made their output visibly wider than the source. The join tasks refuse shots of different sizes, so no normalization step is needed before them.

video_frames

The frames of a generated video, as one (frames, height, width, channels) uint8 array. That is the shape an argument taking frames rather than a video wants - LTX-2's keyframe conditions, which are mapped from 0-255 - and it is one artifact where a list of frames would become one artifact per frame and multiply the step that consumed it:

{
    "name": "opening_frames",
    "task": {
        "command": "video_frames",
        "arguments": { "video": "previous_result:opening" }
    },
    "result": { "content_type": "video/mp4", "save": false, "fps": 24 }
}

Argument	Required	Description
video	Yes	The video - a frame list, a frame array or tensor, or an audio+video pair

An argument that goes through diffusers' video processor instead - LTX-2's IC-LoRA references - wants the [0, 1] frames the pipeline returned rather than this array; hand those over with previous_result:step.frames.

Example: extend-clip.json

pair_audio

Pair a video with an audio track, so the two are saved as one muxed file. A pipeline that generates its own soundtrack returns the pair together; anything working on the frames alone - a latent upsampler, an interpolator, an upscaler - returns frames without it, and this puts it back:

{
    "task": {
        "command": "pair_audio",
        "arguments": {
            "video": "previous_result:upscale",
            "audio": "previous_result:base"
        }
    },
    "result": { "content_type": "video/mp4", "fps": 24 }
}

Argument	Required	Description
video	Yes	The frames - a frame list, a frame array or tensor, or an audio+video pair whose own soundtrack is replaced; their own rate is carried through to the output, so result.fps is only needed to override it (frames that carry none are written at 8 fps)
audio	Yes	The soundtrack - a waveform, the earlier step whose video carried one, or the path or URL of an audio or video file; the last two bring their sample rate along. A mono track is fine: an mp4 audio stream takes stereo and nothing else, so saving duplicates the one channel into two and warns that it did
sample_rate	No	Sample rate of the waveform. Required unless audio carries one; given here it wins
fps	No	The rate the frames play at, only needed when they carry none of their own. Used solely to work out how long the video is - what fit and the length-mismatch check measure the track against - and is never written to the file; that's result.fps, which sets the rate the output plays at and defaults to 8 fps when the frames carry none
fit	No	"video" cuts or pads the track with silence to the length of the frames, warning either way (audio_padded_to_video / audio_trimmed_to_video). Left unset (the default) the track is used as it is, and a length that disagrees with the frames' is warned about rather than corrected (audio_video_length_mismatch). Any other value is refused at run time, not by validate_workflow

fit's guarantee is exact for the waveform handed to the encoder, not for the file the encoder writes: muxing is a lossy AAC encode, and it can still trim or pad the written track by a further handful of samples (#428 measured up to ~30, under a millisecond). That residual is logged, not warned; on a video with recorded shots it becomes a joined_audio_short_after_mux warning only if it reaches a frame. get_gallery_metadata's media.shots and assess_output's sync_length are measured against the written file, not the pre-encode prediction, so they are the number to trust for the track's actual length.

When the video carries recorded shots (from an earlier concat_videos, dissolve_videos or chain step), pair_audio remeasures each one's sample fields against the track it was handed. Every shot but the last is round(start_frame / fps * sample_rate); the last one runs to the track's actual end, and once the file is written it is measured again against what the file decodes to. So its num_samples can sit a few dozen samples off round(num_frames * sample_rate / fps): the encoder's trim, which the job's event log records. A real mismatch between the track and the video's length is a separate, thresholded warning (audio_video_length_mismatch, or audio_padded_to_video / audio_trimmed_to_video when fit: "video" corrected it), so a last shot short by less than a millisecond is expected, not a bug. get_gallery_metadata's media.shots reports the remeasured fields.

Example: assemble-and-score.json

slice_audio

Cut a slice out of an audio track, addressed in seconds or in video frames. Slices reaching past the end of the track are zero-padded — asking for more than the source holds returns a track of the length you asked for whose tail is digital silence, not a shorter track and not an error. Anything past a few milliseconds of that padding is reported as a slice_past_end warning on the job, because a score laid under a longer cut goes silent for the rest of the film without anything else saying so; to fill a cut longer than the recording, build a bed with loop_audio first and slice that. Either half of a pair may be left out - an omitted start begins at the head of the track, an omitted duration runs to the end of it - so a workflow that trims only when it is given a length still passes the whole track along:

{
    "task": {
        "command": "slice_audio",
        "arguments": {
            "audio": "./soundtrack.wav",
            "start_frame": 124,
            "num_frames": 124,
            "fps": 24
        }
    },
    "result": { "content_type": "audio/wav", "sample_rate": 44100 }
}

Argument	Required	Description
audio	Yes	Path or URL of an audio file (or of a video file, whose soundtrack is taken), a waveform from a previous step, or an earlier step's video generated with a soundtrack (which brings its sample rate along)
start_seconds / duration_seconds	One pair	The slice in seconds; either may be omitted
start_frame / num_frames / fps	One pair	The slice in video frames; fps is required, start and count may be omitted
sample_rate	With a waveform	Sample rate of a directly passed waveform (files carry their own)

gain_audio

Apply a gain, in decibels, to a region of an audio track - the rest of the track passes through unchanged. The region is addressed the same way slice_audio's is, in seconds or in video frames, so ducking a scene under another (lowering a dialogue track between two timestamps) is one step instead of the slice_audio → gain (a whole-track normalize_audio on the slice) → mix_audio → rejoin → pair_audio chain that used to be the only way to gain part of a track rather than all of it. Unlike slice_audio, a region reaching past the end of the track is clipped to it rather than zero-padded - there is no silence there to gain, only the end of the real material:

{
    "task": {
        "command": "gain_audio",
        "arguments": {
            "audio": "./dialogue.wav",
            "gain_db": -12,
            "start_frame": 124,
            "num_frames": 48,
            "fps": 24
        }
    },
    "result": { "content_type": "audio/wav", "sample_rate": 44100 }
}

Argument	Required	Description
audio	Yes	Path or URL of an audio file (or of a video file, whose soundtrack is taken), a waveform from a previous step, or an earlier step's video generated with a soundtrack (which brings its sample rate along)
gain_db	Yes	Gain to apply within the region, in decibels - negative ducks it, positive boosts it
start_seconds / duration_seconds	No	The region in seconds; either may be omitted
start_frame / num_frames / fps	No	The region in video frames; fps is required if either is given, start and count may be omitted
sample_rate	With a waveform	Sample rate of a directly passed waveform (files carry their own)

No region argument is required: with every one of them omitted, the gain applies to the whole track (#395) - the same "no region means everything" reading mix_audio's gains use. To gain everything from some point on instead, give just start_seconds: 0 and leave duration_seconds unset (or start_frame: 0 + fps and leave num_frames unset), which runs to the end of the track without needing to already know how long that is.

crossfade_audio

Join audio tracks with an equal-power crossfade. Each seam overlaps the two tracks by the fade window:

{
    "task": {
        "command": "crossfade_audio",
        "arguments": {
            "audios": "previous_result:slices",
            "crossfade_ms": 75,
            "sample_rate": 44100
        }
    },
    "result": { "content_type": "audio/wav", "sample_rate": 44100 }
}

fade_audio

Fade a track in from silence and out to it. A slice cut out of the middle of a piece ends on whatever was sounding at the cut; a fade turns that into an ending. The curve is the equal-power cosine the seam joins use:

{
    "task": {
        "command": "fade_audio",
        "arguments": {
            "audio": "previous_result:soundtrack",
            "fade_in_ms": 500,
            "fade_out_ms": 2500,
            "sample_rate": 44100
        }
    }
}

Argument	Required	Description
audio	Yes	Path or URL of an audio file (or of a video file, whose soundtrack is taken), a waveform from a previous step, or an earlier step's video generated with a soundtrack (which brings its sample rate along)
fade_in_ms	No	Length of the fade in, from the head of the track (default: 0)
fade_out_ms	No	Length of the fade out, to the tail of the track (default: 0)
sample_rate	With a waveform	Sample rate of a directly passed waveform (files carry their own)

Example: audio-trim-fade.json — slice a generated track to length, then fade the cut into an ending.

normalize_audio

Scale a track so its loudest sample sits at a level. Generated music comes out wherever the model happened to land - a quiet take needs lifting before it sits under a picture, a hot one needs headroom before the encoder. Only the gain changes, so the dynamics survive:

{
    "task": {
        "command": "normalize_audio",
        "arguments": {
            "audio": "previous_result:faded",
            "peak_dbfs": -1.0,
            "sample_rate": 44100
        }
    }
}

Argument	Required	Description
audio	Yes	Path or URL of an audio file (or of a video file, whose soundtrack is taken), a waveform from a previous step, or an earlier step's video generated with a soundtrack (which brings its sample rate along)
peak_dbfs	No	The level the loudest sample is moved to, in dB below full scale (default: -1.0). 0 is full scale. With limit, a true-peak (4x oversampled, BS.1770) ceiling the output never crosses
target_lufs	No	An integrated loudness (BS.1770) to scale the track to instead, in LUFS (0 or below). Without limit the gain is capped so the peak stays under peak_dbfs, and target_lufs_capped warns when that cap wins. A track shorter than 400 ms (or silent throughout) cannot be measured, warns target_lufs_unmeasurable, and falls back to peak_dbfs
limit	No	Reach target_lufs past a transient instead of letting it cap the gain: a true-peak look-ahead limiter (5 ms look-ahead, 20 ms hold, 150 ms release, linked across channels) holds peak_dbfs, and the gain is searched for until the limited track lands within 0.1 LU of target_lufs - limiting takes some loudness back, more on dense material. The limiter itself never adds gain, and it stops at 12 dB of reduction - past that the gain stops too, and target_lufs_capped warns with limited: true and shortfall_lu (as it does for any track left more than 0.1 LU short). More than 6 dB warns limiter_heavy (pumping can be audible). The step's log carries constraint: "limiter", gain_db, max_gain_reduction_db, limited_fraction (0 when the limiter was not needed), output_true_peak_dbfs and output_lufs (default: false, which behaves exactly as without it)
sample_rate	With a waveform	Sample rate of a directly passed waveform (files carry their own)

A silent track is returned unchanged.

To level dialogue with a laugh or a shout on it, limit is the switch: a peak-only gain lets that one transient set the level of every line around it.

"arguments": {"audio": "previous_result:dialogue", "target_lufs": -16, "peak_dbfs": -1.0, "limit": true}

Example: dissolve-between-shots.json

Headroom and clipping warnings. Saving audio or a video with a muxed soundtrack checks the written level against two thresholds, reported in get_job's warnings and readable back afterward as media.peak_dbfs from get_gallery_metadata:

• audio_no_headroom fires when a plain audio file's waveform, before encoding, peaks at or above -0.5 dBFS - encoding can push a level that already has no headroom over full scale.
• audio_clipped fires when the file is decoded back after writing and measures at or above 0.0 dBFS - the ground truth of what a consumer's decoder will actually see, since an encoder's own overshoot varies by codec and is not reliably predictable from the pre-encode level.

A video mux only ever reports the second one: its pre-encode prediction is held rather than emitted, because a mux's overshoot is not reliably positive the way a plain audio encode's is. That leaves a real gap between the two thresholds - a video whose soundtrack decodes back between -0.5 and 0.0 dBFS produces no warning at all, because it predicted risk but measured clean. That is the file's own measured level, not a threshold bug: read media.peak_dbfs against -0.5 and 0.0 to judge a specific file rather than relying on the warning alone.

mix_audio

Layer tracks on top of one another. crossfade_audio puts tracks one after another; this puts them on top of each other - a score laid under a film's own sound, where the music runs unbroken while the world underneath it is replaced at every cut:

{
    "task": {
        "command": "mix_audio",
        "arguments": {
            "audios": ["previous_result:soundtrack", "previous_result:world"],
            "gains": [0.5, 1.0],
            "sample_rate": 44100
        }
    }
}

Argument	Required	Description
audios	Yes	The tracks to layer - waveforms, audio or video file paths, or videos generated with a soundtrack
gains	No	One plain multiplier per track, in the same order - not decibels. Defaults to unity on every track
sample_rate	With a raw waveform	Sample rate of the waveforms. Required unless every track brings its own; given here it wins

Tracks of different lengths are padded with silence to the longest, so a score shorter than the picture leaves the tail dry rather than cutting the picture down to fit. Summing can push peaks past full scale and the sum is not rescaled - follow it with normalize_audio to bring the peak back down.

Example: dissolve-between-shots.json — a generated score mixed under the shots' own audio.

loop_audio

Make a bed of a given length out of a short recording — the room tone laid under a whole cut, which is the only complete fix for the hole at a seam. Each shot in a cut carries its own room and nothing runs underneath the join; a continuous bed does, the way a location's room tone is laid under a dialogue scene so the edits stop being audible:

{
    "task": {
        "command": "loop_audio",
        "arguments": {
            "audio": "previous_result:room_tone",
            "target_frames": 620,
            "fps": 24,
            "crossfade_ms": 250
        }
    }
}

Argument	Required	Description
audio	Yes	Path or URL of an audio or video file, a video generated with a soundtrack (which brings its sample rate along), or a waveform
duration_seconds	One of	How long the bed should be, in seconds
target_frames / fps	One of	How long the bed should be, in video frames — how a bed is matched to a cut exactly
crossfade_ms	No	Crossfade at each loop point, clamped to the material available (default 250)
sample_rate	With a waveform	Sample rate of a waveform passed directly; given for a file or a video it overrides the rate they carry

Laps are joined with an equal-power crossfade rather than butted together, so the loop point itself is not a click. That only smooths the seam: a transient in the source (a hit, a swell) still recurs once per lap at full strength, so the loop still reads as a level pulse at the lap rate — measured at 9.3 dB on a source with one such transient. Pick a source with even internal level to avoid the pulse; the crossfade does not remove it. The source is used whole every lap and only the last one is trimmed, so the bed lands exactly on the requested length; a source longer than the request is trimmed to it.

The bed is laid under the cut with mix_audio and attached to the picture with pair_audio:

{ "name": "bed",   "task": { "command": "loop_audio",
                             "arguments": { "audio": "previous_result:room_tone",
                                            "target_frames": 620, "fps": 24 } } },
{ "name": "mixed", "task": { "command": "mix_audio",
                             "arguments": { "audios": ["previous_result:episode",
                                                       "previous_result:bed"],
                                            "gains": [1.0, 0.25] } } },
{ "name": "cut",   "task": { "command": "pair_audio",
                             "arguments": { "video": "previous_result:episode",
                                            "audio": "previous_result:mixed" } },
  "result": { "content_type": "video/mp4", "fps": 24 } }

The bed's source: find_loop_bed picks the stretch. Run it against the cut (or a stem of it) over the range that should hold room tone, copy the top candidate's start_seconds/duration_seconds into slice_audio, loop_audio the slice to the cut's length, and mix_audio it in at the candidate's gain — the room the shots were generated in, picked by measurement rather than by ear.

find_loop_bed

Pick the window of a recording worth looping into the room-tone bed above, measured as it will sound looped rather than as it sits in the source. A level check alone misses three things, each found on a real episode:

• near-programme material — faint speech attenuated ~30 dB reads as quiet, and is audible once it repeats every lap.
• lap-rate modulation — the loop's repeat rate beats against the source's own level movement, invisible in one pass through the source.
• ticks — a 1 ms transient is invisible to 50 ms RMS and recurs once per lap.

{
    "task": {
        "command": "find_loop_bed",
        "arguments": {
            "audio": "output:episode/latest/final/cut.mp4",
            "start_seconds": 30.0,
            "end_seconds": 90.0
        }
    },
    "result": { "content_type": "application/json" }
}

Argument	Required	Description
audio	Yes	Path, asset:/output: reference of an audio or video file (a video is read audio-only - frames are never decoded), or a track or video from previous_result:
start_seconds / end_seconds	No	The range to search; the whole file if omitted
min_seconds / max_seconds	No	Shortest/longest window tried (default 0.5 / 2.0)
max_bin_dbfs	No	Every 50 ms bin of a window must be at or below this (default -55)
max_mean_dbfs	No	A window's mean (RMS) level must be at or below this (default -60)
max_spike_db	No	Most a window's largest 1 ms peak may sit above its median 1 ms peak (default 12)
crossfade_ms	No	The crossfade candidates are looped with - loop_audio's own default, so what is measured is what loop_audio will make (default 250)
loop_seconds	No	Length of the looped result that is measured (default 10.0)
target_bed_dbfs	No	The level each candidate's gain is computed to reach (default -60)
max_candidates	No	How many ranked candidates to return (default 5)
shots	No	Shot boundaries, the assessment probes' shape: [{name, start_frame, num_frames}], optionally with start_sample/num_samples; overrides the shots a video carries or its run records
fps	No	Frame rate the shots' frames count at, for a source with none of its own (an audio file); a video's own rate otherwise

Every window on a 50 ms grid, from min_seconds to max_seconds long, is judged against four rules in order and counted in rejected under the first one it fails, so rejected is a tally of the whole grid:

• too_loud — a 50 ms bin above max_bin_dbfs, or a mean above max_mean_dbfs.
• silent — digital silence: a mean of zero, or a median 1 ms peak of zero (more than half the window is exact zeros, whatever sits in the rest).
• spike — its largest 1 ms peak more than max_spike_db above its median 1 ms peak. The largest peak is also looked for in the 5 ms just outside each end, unless that neighbouring bin is itself too loud, so a window ending on a click is thrown out rather than putting the click's onset at the loop's seam.
• tonal — the flatness/harmonicity test bleed_join uses, taken over every 0.1 s and every 0.2 s block inside the window (one per 50 ms step; no longer than min_seconds). A window is tonal when any block in it is. Faint speech comes and goes, and over a whole window the pauses dilute a syllable below the threshold; in the block it sits in, it is not diluted. Two lengths, because 0.1 s sits inside one syllable and 0.2 s holds enough periods of a low hum. Flatness is measured over the band the source actually occupies: a source resampled up (a 16 kHz bed mixed at 24 kHz) has an empty band above its own Nyquist that reads as tonal whatever the material is, as bleed_join's tail does (#198). A candidate's flatness and harmonicity are the readings of its blocks closest to failing (lowest flatness, highest harmonicity).

On a cut, a bed must come from inside one shot: a window across a cut loops the seam's change of room as a once-per-lap step. Shots resolve in the probes' order - the shots argument (shots_source: "argument"), else the shots a video from an earlier step carries ("artifact"), else the ones the run manifest beside the file records ("manifest", which is why output: the joined file finds them unasked), else none (null, the search above unchanged). With shots, a window that crosses a boundary, or lies where no shot covers, is counted under rejected.shot_boundary before the four rules (so theirs count only in-shot windows), each candidate names its shot (shot@<name> from a manifest), and source.shots lists each shot's {name, start_seconds, end_seconds} as placed in the soundtrack. A shot with both frames and recorded samples is placed at their overlap - the picture's cut and the audio's can sit a few samples apart. A shot placed by frames on a source with no frame rate is refused; pass fps.

The survivors are thinned so no two overlap, steadiest source first, to a pool of up to 200 (LOOPED_POOL). The pool does not depend on max_candidates, which only cuts the final ranking, so asking for one candidate returns the default run's first. Each one in the pool is then looped with loop_audio's own crossfade to loop_seconds and measured: ripple_db (the 5-95% spread of the looped 50 ms bins), envelope_peak_db/ envelope_peak_hz (the strongest level wobble) and lap_component_db (the wobble at the lap rate). Candidates are ranked by looped ripple_db, lowest first; one whose envelope_peak_db is above -15 dB carries the lap_modulation warning.

{
    "source": { "duration_seconds": 620.4, "sample_rate": 44100, "searched": [30.0, 90.0], "shots_source": null },
    "criteria": { "min_seconds": 0.5, "max_seconds": 2.0, "target_bed_dbfs": -60.0 },
    "candidates": [
        {
            "rank": 1,
            "start_seconds": 41.28,
            "duration_seconds": 1.35,
            "end_seconds": 42.63,
            "shot": null,
            "mean_dbfs": -63.1,
            "max_bin_dbfs": -57.4,
            "spike_db": 4.2,
            "flatness": 0.61,
            "harmonicity": 0.08,
            "looped": {
                "ripple_db": 1.1,
                "envelope_peak_db": -24.0,
                "envelope_peak_hz": 0.4,
                "lap_hz": 0.096,
                "lap_component_db": -26.0
            },
            "gain_db": 3.1,
            "gain": 1.43,
            "warnings": []
        }
    ],
    "rejected": { "too_loud": 812, "silent": 0, "spike": 14, "tonal": 203 },
    "findings": []
}

start_seconds/duration_seconds are slice_audio's arguments and gain is mix_audio's multiplier for reaching target_bed_dbfs — the answer picks a window and builds nothing, so the remedy is a copy. Finding nothing is an answer, not an error: candidates is [], rejected counts the windows each rule threw out, and one no_loop_bed finding says which rule to relax.

Like the assessment probes it answers JSON and decides nothing, but it searches rather than checking a finished cut, so it is not one of them — it stays out of list_tasks' assessment list. The result must be saved as application/json.

{
    "variables": { "audio": "output:episode/latest/final/cut.mp4" },
    "steps": [
        { "name": "bed", "task": { "command": "find_loop_bed",
                                    "arguments": { "audio": "variable:audio" } },
          "result": { "content_type": "application/json" } }
    ]
}

run against the finished cut once it exists, then read back over MCP with get_output_text (JSON results are text) rather than get_output_audio.

Out-of-range literals (min_seconds/max_seconds/loop_seconds at or below zero, max_candidates at or below zero, a negative crossfade_ms, and the like) are refused at validation. end_seconds at or before start_seconds, min_seconds above max_seconds, a range past the end of the file, and a source shorter than min_seconds fail at run time — they depend on the file.

resample_audio

Convert a track to a different sample rate. A pipeline that conditions on audio wants it at its own rate (MiniMax H3 at its audio VAE's), and resampling a supplied recording once, up front, feeds it what it already wants:

{
    "task": {
        "command": "resample_audio",
        "arguments": {
            "audio": "previous_result:edit",
            "target_sample_rate": 44100
        }
    }
}

Argument	Required	Description
audio	Yes	Path or URL of an audio or video file, a video generated with a soundtrack (which brings its sample rate along), or a waveform
target_sample_rate	Yes	The rate to convert to
sample_rate	With a waveform	Sample rate of a waveform passed directly; given for a file or a video it overrides the rate they carry

A track already at the target rate is returned untouched. The conversion is PyAV's, which dw already needs for video - no torchaudio dependency.

Every audio task returns the waveform and the rate it is at, so one chains into the next without the rate being restated: a resample_audio fed previous_result: from a slice_audio takes the source rate from the slice. A sample_rate given on the step still wins, and one declared on the step's result still decides what is written to disk.

Example: assemble-and-score.json

compress_audio

Shape a track's dynamics with an envelope-follower - a compressor, a limiter and a gate are the same algorithm with different knob settings, so one task covers all three through mode:

{
    "task": {
        "command": "compress_audio",
        "arguments": {
            "audio": "previous_result:mixed",
            "threshold_dbfs": -18.0,
            "ratio": 4.0,
            "attack_ms": 10,
            "release_ms": 100,
            "sample_rate": 44100
        }
    }
}

Argument	Required	Description
audio	Yes	Path or URL of an audio file (or of a video file, whose soundtrack is taken), a waveform from a previous step, or an earlier step's video generated with a soundtrack (which brings its sample rate along)
threshold_dbfs	Yes	The level the envelope is measured against, in dB below full scale. Cannot be above 0
ratio	No	How hard the reduction is above the threshold, in compress/gate mode (default: 4.0). Ignored in limit mode, which always holds the signal at the threshold
attack_ms	No	How fast the envelope rises to a louder signal (default: 10.0). 0 means instantly
release_ms	No	How fast the envelope falls back after a louder signal ends (default: 100.0). 0 means instantly
mode	No	compress (turn down what's above the threshold), limit (hold the signal at the threshold), or gate (turn down what's below the threshold) (default: compress)
sample_rate	With a waveform	Sample rate of a directly passed waveform (files carry their own)

A silent track is returned unchanged.

mode: "limit" is a sample-peak limiter with no look-ahead: the envelope reacts to a transient as it arrives, so the transient's leading edge and any inter-sample peak get through. To hold a true-peak ceiling while reaching a loudness target, use normalize_audio with limit: true.

filter_audio

Run a track through a single biquad filter stage - trimming the frequencies a mix doesn't need, or carving out room for another element:

{
    "task": {
        "command": "filter_audio",
        "arguments": {
            "audio": "previous_result:world",
            "cutoff_hz": 120,
            "kind": "highpass",
            "sample_rate": 44100
        }
    }
}

Argument	Required	Description
audio	Yes	Path or URL of an audio file (or of a video file, whose soundtrack is taken), a waveform from a previous step, or an earlier step's video generated with a soundtrack (which brings its sample rate along)
cutoff_hz	Yes	The filter's corner frequency. Must be below the Nyquist frequency (half the sample rate)
kind	No	lowpass, highpass, bandpass, or notch (default: lowpass)
q	No	The filter's resonance/bandwidth (default: 0.707, a Butterworth response)
sample_rate	With a waveform	Sample rate of a directly passed waveform (files carry their own)

A silent track is returned unchanged.

analyze_audio

Measure a track without changing it - peak and RMS level, crest factor, and a rough low/mid/high spectral balance, the numbers a compress_audio or filter_audio step downstream is tuned against rather than guessed at:

{
    "task": {
        "command": "analyze_audio",
        "arguments": {
            "audio": "previous_result:mixed",
            "sample_rate": 44100
        }
    }
}

Argument	Required	Description
audio	Yes	Path or URL of an audio file (or of a video file, whose soundtrack is taken), a waveform from a previous step, or an earlier step's video generated with a soundtrack (which brings its sample rate along)
sample_rate	With a waveform	Sample rate of a directly passed waveform (files carry their own)

Returns a dict, not a track: peak_dbfs, rms_dbfs, crest_factor_db, low_dbfs (20-250 Hz), mid_dbfs (250-4000 Hz), high_dbfs (4000-20000 Hz). The three bands are each a share of the track's total power on the same scale as rms_dbfs (their powers sum to it), so the loudest band sits near rms_dbfs rather than tens of dB under it - comparable to compress_audio's threshold_dbfs. A silent track, or a band with no content at the track's sample rate, reads as null rather than -inf.

Assessment Probes

Three read-only commands measure a finished cut and say where to look - analyze_shots, analyze_seams, analyze_sync_drift. Each is registered with assessment=True (register_command, dw/tasks/task.py), which is what makes a command a probe - not merely answering JSON, since attribute_voices answers JSON too and is not one. Each takes a video (a stored file - asset:, output: or a path, read straight from disk rather than decoded first - or the video an earlier step returned; not a URL, whose download is a bare frame list with no soundtrack) and answers one JSON document: every measurement it took, plus findings (the measurements that crossed a rule in the table below), rules_applied (the rule names the probe checked) and shots_source (where the shot list came from). A probe reads the file streaming - a 64x36 grey thumbnail per frame and the soundtrack, never a full frame list - so it runs on a cut of any length.

Findings are places to look, not verdicts: nothing in the engine acts on one, no run fails for one, and a finding someone has looked at and accepted is simply left alone.

A probe's result must save as JSON:

{
    "task": {
        "command": "analyze_seams",
        "arguments": {
            "video": "output:<identity>/latest/final/cut.mp4"
        }
    },
    "result": { "content_type": "application/json" }
}

Any other content_type (or none) fails validation - a JSON document can only be saved whole under application/json; every other content type would explode it key by key or die trying to write a number.

Shot boundaries come, in order: the step's own shots argument, the shots carried by the video an earlier step returned, the run manifest beside the file, and otherwise the whole file is treated as one shot. shots_source reports which - argument, artifact, manifest, or none.

analyze_shots

Each shot's level and spectral balance, and how far apart the shots sit:

Field	Meaning
shots[].name	The shot's name
shots[].start_frame / num_frames	The shot's frame range, as the shot record gave it
shots[].peak_dbfs	Peak level within the shot
shots[].rms_dbfs	RMS level within the shot
shots[].crest_db	peak_dbfs minus rms_dbfs
shots[].low_dbfs / mid_dbfs / high_dbfs	Spectral balance (20-250 Hz / 250-4000 Hz / 4000-20000 Hz), on the same scale as rms_dbfs
shots[].samples	Whether the shot's sample span was recorded (carried by the shot record) or derived (scaled from its frames)
shots[].dead_air_seconds	The longest run of 50ms windows inside the shot at or below the DEAD_AIR_FLOOR_DBFS threshold (-65 dBFS)
shots[].dead_air_at	Where that run starts, in seconds into the file
shots[].dead_air_floor_dbfs	The quietest 50ms window measured inside that run - not the threshold. Null when the run is pure digital silence, and null when there is no run at all
rms_range_db	The spread between the loudest and quietest voiced shot
has_audio	Whether the file carries a soundtrack at all

analyze_seams

Every seam between shots, audio and picture:

Field	Meaning
seams[].seam	The seam's index (1-based)
seams[].between	[previous shot name, next shot name]
seams[].seconds	Where the seam sits in the file
seams[].kind	cut or dissolve (a dissolve has overlap_frames)
seams[].hard_cut	Whether the incoming shot is marked hard_cut: true
seams[].before_shot_rms_dbfs / after_shot_rms_dbfs	RMS level of the whole shot either side of the seam
seams[].level_step_db	The absolute difference between those two shot levels. Shot against shot, not the audio at the seam's edges: a take's own tail and head can sit 20 dB apart, which is not a step the cut made
seams[].before_rms_dbfs / after_rms_dbfs	RMS level of the 0.25 s either side of the seam - what seam_hole's both-sides-voiced guard reads
seams[].floor_dbfs	RMS level of the join itself (the fade, or a short window centred on a cut)
seams[].click_db	How far a spike at the join peaks above its immediate neighbours
seams[].spectral_shift	How much the low/mid/high balance shifts across the seam (0-1)
seams[].frame_delta	The largest single-frame picture change across the seam
seams[].typical_delta	The larger shot's own typical frame-to-frame change, floored
seams[].jump_ratio	frame_delta divided by typical_delta

analyze_sync_drift

How far the soundtrack sits from the picture, shot by shot and over the whole file:

Field	Meaning
shots[].name	The shot's name
shots[].start_offset_ms	How far the audio sits from the picture at the shot's start
shots[].end_offset_ms	How far the audio sits from the picture at the shot's end
max_offset_ms	The largest end_offset_ms across all shots, by magnitude
video_seconds / audio_seconds	Each stream's own duration
length_delta_ms	audio_seconds minus video_seconds

Rules

Each rule names the probe and field it reads, how the value is compared to its threshold, and the severity of a crossing:

Rule	Probe	Field	Threshold	Severity
shot_level_spread	analyze_shots	rms_range_db	>= 6.0 dB	warn
seam_level_step	analyze_seams	level_step_db	> 3.0 dB	warn
seam_click	analyze_seams	click_db	> 12.0 dB	warn
seam_hole	analyze_seams	floor_dbfs	< -50.0 dBFS	warn
seam_frame_jump	analyze_seams	jump_ratio	> 25.0	info
shot_dead_air	analyze_shots	dead_air_seconds	> 0.4 s	warn
sync_drift	analyze_sync_drift	end_offset_ms	> 40.0 ms (magnitude)	warn
sync_length	analyze_sync_drift	length_delta_ms	> 40.0 ms (magnitude)	warn

Three rules carry a guard beyond the threshold: seam_hole only fires while both sides of the seam are voiced above -30 dBFS (a quiet join between two quiet shots is not a hole, it's a pause the shots themselves hold); seam_frame_jump is skipped at a seam whose incoming shot is marked hard_cut: true - a cut meant as a cut; and shot_dead_air is skipped inside a shot whose own rms is at or below -30 dBFS - a shot that is quiet throughout, on purpose, rather than one holding a gap. A gap the guard lets through is what slice_audio -> loop_audio -> mix_audio is for: cut a room-tone bed from the take, loop it to the gap's length, and mix it under the line rather than leaving the drop silent.

A shots record reaching past the file's own length is a separate finding, shot_span_overrun, on all three probes - not a threshold crossing, since the engine clips the record to the file before any of the rules above run. validate_workflow reports the same mistake ahead of the run when the video's length is already knowable (a shots argument against an asset:/literal video); a previous_result:/output: video not yet written is left to the finding.

list_tasks names the probes in their own assessment list, alongside commands, so a caller looking for a way to check a cut can find them without reading every command's schema. The list is exactly the commands declared assessment=True; attribute_voices stays in commands only, since it analyzes a song rather than checking a cut.

Voice attribution

Which reference voice sings each line of a song, by timbre - staging lip-sync shots for a generated song needs its section-to-singer map, and a generation model (MiniMax Music3 included) does not hand one back. Pitch cannot stand in for it: a tenor and a mezzo share a range, and a pitch heuristic has called a tenor female.

attribute_voices separates the vocal stem (htdemucs), reduces every line and every voice's reference to its voiced frames, embeds each with speechbrain's ECAPA speaker encoder (spkrec-ecapa-voxceleb), scores every line against every voice by cosine, and rolls lines up into named windows (shots, say) by the voiced seconds they overlap. It decides nothing: the argmax is always reported, alongside the margin and how much of the line was voiced, so a weak answer is visible as weak rather than silently accepted.

{
    "task": {
        "command": "attribute_voices",
        "arguments": {
            "audio": "asset:song.mp3",
            "voices": {
                "lena": [{"start_seconds": 4.0, "duration_seconds": 6.0}],
                "marcus": [{"start_seconds": 32.0, "duration_seconds": 6.0}]
            },
            "windows": [
                {"name": "shot-1", "start": 0.0, "end": 12.0},
                {"name": "shot-2", "start": 12.0, "end": 24.0},
                {"name": "shot-3", "start": 24.0, "end": 40.0}
            ]
        }
    },
    "result": { "content_type": "application/json" }
}

Argument	Required	Description
audio	Yes	The song - a path, asset:/output: reference, or an earlier step's audio or video
voices	Yes	Each voice's name mapped to its reference: a list of {start_seconds, duration_seconds} (or {start, end}) spans into audio, or a path/asset: of a separate clip. At least 2 voices; names follow the variable-name pattern; each voice's reference must total at least min_reference_seconds
lines	No	Spans to attribute, {start, end, text?} (a transcribe_audio transcript, or its chunks list, drops in directly) or {start_seconds, duration_seconds, text?}. Omitted: the song is cut into fixed windows of window_seconds
windows	No	Named spans to roll lines up into, {name, start, end}. Omitted: mirrors the fixed windows when lines is also omitted, otherwise none
window_seconds	No	Length of the fixed windows used without lines (default 2.0)
min_reference_seconds	No	Least total reference length per voice; a shorter one is refused by name (default 3.0)
separate	No	Isolate the vocal stem with htdemucs before embedding (default true); false for audio that is already a dry vocal
device	No	Where the models run

A literal voices is checked at validation as well as on the step: fewer than two voices, a bad name, a malformed span, or a span-list reference under min_reference_seconds is refused at steps[i].task.arguments.voices, and a bare path as a voice meets the same location policy as audio. What needs the song itself - a span past its end, a clip's voiced length - is the step's to refuse.

The result: voices (the names), separated, duration_seconds, lines[] (start, end, text, scores, voice, margin, voiced_seconds, uncertain, reason), windows[] (name, start, end, voiced_seconds, share, voice, uncertain, reason), reference_similarity (pairwise cosine between the voices' references), voiced_floor_dbfs (the floor this song's stem was read against), warnings and the thresholds compared against.

The voiced floor is relative to the stem rather than a fixed level: a quiet sung verse under a loud chorus sits 20 dB or more beneath it, and a fixed -40 dBFS floor dropped such a verse as unvoiced and refused it as a reference. Separation leaves near-silence between phrases, so the floor sits well above that and well below the singing.

Threshold	Value	What crossing it does
voiced_level_percentile	95	The stem's level is this percentile of its 20 ms frames' rms
voiced_floor_below_level_db	35.0 dB	A frame at or above the stem's level less this is voiced
voiced_floor_min_dbfs	-60.0 dBFS	The floor never drops below this, however quiet the stem
min_voiced_seconds	0.5 s	A line or window under this much voiced time has no voice - voice: null, uncertain: true
uncertain_margin	0.05	A line whose best score beats the runner-up by less than this is uncertain - the argmax is still reported
uncertain_share_margin	0.2	A window whose leading voice's share beats the runner-up's by less than this is uncertain - a duet line, or a window straddling a hand-over
voices_too_similar	0.8	Two references scoring above this against each other make every line between them a weak answer whatever its scores say

A voices_too_similar pair is reported in reference_similarity, added to warnings, and emitted onto the job's warnings (voices_too_similar) - picking better-separated reference spans is the fix, not reading past it.

Both models are fixed - there is no model-name argument - and both run in fp32 (their STFT front ends are fp32-only in practice, and both are small enough that it costs nothing). Weights download on first use - htdemucs from Hugging Face (adefossez/HTDemucs via demucs 4.1), ECAPA from speechbrain - and are cached between calls like any other model. On MPS a separate: true run that fails in htdemucs falls back to the CPU and warns (separation_cpu_fallback) rather than failing the step.

Data Gathering

gather_images

Load images from URLs and/or file glob patterns:

{
    "task": {
        "command": "gather_images",
        "arguments": {
            "urls": ["https://example.com/a.jpg", "https://example.com/b.jpg"],
            "glob": "./images/*.jpg"
        }
    }
}

Returns a list of images that can be referenced by later steps with previous_result:.

gather_videos

Same as gather_images but for video files. Each video comes back as one artifact holding its frames and whatever audio was muxed alongside them, so a step referencing this one iterates over videos rather than over frames.

To join videos that are already on disk, give their paths to concat_videos directly rather than gathering them first: a previous_result reference to a gather step fans the consuming step out over the gathered videos instead of handing it all of them at once.

gather_inputs

Pass through arguments directly. Useful for organizing data flow.

Videos in image tasks

Every image command - the upscalers, face restoration, segmentation and the image processors - takes a video where it takes an image: an AudioVideo from a generation, concat_videos or dissolve_videos step, or a frame array from video_frames. The command runs over the frames one at a time and returns one video artifact, its soundtrack carried through untouched, so a generated clip can be upscaled without losing what was generated alongside it:

{
    "task": {
        "command": "upscale",
        "arguments": {
            "image": "previous_result:generate_video",
            "model_name": "Kim2091/UltraSharp"
        }
    },
    "result": { "content_type": "video/mp4", "fps": 24 }
}

Captioning (image_to_text) is the exception - describe a frame, taken with get_first_frame, rather than a video.

A file path or asset:/output: reference to a video is not accepted here, even though the route above takes a video from a step - route it through video_frames first: an image command's image argument must be an AudioVideo, a frame array, or a still image.

Image Upscaling

Upscale images using spandrel-compatible super-resolution models (ESRGAN, SwinIR, HAT, DAT, and 40+ other architectures). Models are auto-detected from weight files.

{
    "task": {
        "command": "upscale",
        "arguments": {
            "image": "previous_result:generate",
            "model_name": "Kim2091/UltraSharp",
            "filename": "4x-UltraSharp.pth"
        }
    }
}

Argument	Required	Description
image	Yes	PIL Image or previous_result: reference - a video runs frame by frame, see Videos in image tasks
model_name	Yes	HuggingFace repo ID or local file path
filename	No	Specific weight file in a HF repo (auto-detected if only one)
tile_size	No	Tile size for large images (default: 512)
tile_overlap	No	Overlap between tiles in pixels (default: 32)

Large images are automatically tiled to avoid GPU memory issues. Models can be loaded from HuggingFace Hub repos or local .pth/.safetensors files.

Examples:

• upscale-spandrel.json — Upscale any existing image 4x.
• upscale-spandrel.json — Upscale an image you already have; there is no generation step, so the input is a path or URL.

Diffusion Upscaling

Upscale images using Stable Diffusion upscale pipelines. Text-guided upscaling with better detail recovery than traditional super-resolution, especially for faces and textures.

Two modes are available:

• x4 (default): StableDiffusionUpscalePipeline — 4x upscale via stabilityai/stable-diffusion-x4-upscaler
• x2: StableDiffusionLatentUpscalePipeline — 2x upscale via stabilityai/sd-x2-latent-upscaler

{
    "task": {
        "command": "diffusion_upscale",
        "arguments": {
            "image": "previous_result:generate",
            "prompt": "high quality, detailed",
            "negative_prompt": "blurry, low quality, artifacts",
            "mode": "x4"
        }
    }
}

Argument	Required	Description
image	Yes	PIL Image or previous_result: reference - a video runs frame by frame, see Videos in image tasks
prompt	No	Text guidance for upscaling (default: "")
negative_prompt	No	Negative text guidance (default: none)
mode	No	"x4" or "x2" (default: "x4")
model_name	No	Override the default model for the selected mode
num_inference_steps	No	Denoising steps (default: 25)
guidance_scale	No	Classifier-free guidance scale (default: 9.0)
noise_level	No	Noise level for x4 mode (default: 20, ignored for x2)

Examples:

• upscale-diffusion.json — Upscale any existing image. mode selects which: x4 (the default) reaches 2048px, x2 reaches 1024px through the latent upscaler.
• upscale-diffusion.json — Prompt-guided upscale of an image you already have, with no generation step.

Face Restoration

Restore and enhance faces in images using spandrel-compatible face restoration models (GFPGAN, CodeFormer, RestoreFormer). Uses facexlib for face detection and alignment, then runs each detected face through the restoration model.

{
    "task": {
        "command": "restore_faces",
        "arguments": {
            "image": "previous_result:generate",
            "model_name": "leonelhs/gfpgan",
            "filename": "GFPGANv1.4.pth"
        }
    }
}

Argument	Required	Description
image	Yes	PIL Image or previous_result: reference - a video runs frame by frame, see Videos in image tasks
model_name	Yes	HuggingFace repo ID or local file path
filename	No	Specific weight file in a HF repo (auto-detected if only one)
upscale_factor	No	Background upscale factor (default: 1, no upscaling)
face_size	No	Cropped face size in pixels (default: 512)
use_parse	No	Use face parsing for better blending (default: true)
only_center_face	No	Only restore the largest/center face (default: false)
detection_resize	No	Resize shorter side for detection speed (default: 640)
eye_dist_threshold	No	Skip faces with eye distance below this (default: 5)
upsample_img	No	Pre-upscaled background image (e.g., from a prior upscale step)

Models are loaded via spandrel, so any .pth/.safetensors face restoration weights work. CodeFormer requires pip install spandrel-extra-arches (non-commercial license).

Example: restore-faces.json — Generate a portrait, then restore faces with GFPGAN v1.4.

Combining with Upscaling

You can chain upscaling and face restoration. Generate first, upscale the background, then paste restored faces onto the upscaled image:

{
    "steps": [
        {
            "name": "generate",
            "pipeline": { "..." : "..." },
            "result": { "content_type": "image/jpeg" }
        },
        {
            "name": "upscale",
            "task": {
                "command": "upscale",
                "arguments": {
                    "image": "previous_result:generate",
                    "model_name": "Kim2091/UltraSharp",
                    "filename": "4x-UltraSharp.pth"
                }
            },
            "result": { "content_type": "image/jpeg" }
        },
        {
            "name": "restore",
            "task": {
                "command": "restore_faces",
                "arguments": {
                    "image": "previous_result:generate",
                    "model_name": "leonelhs/gfpgan",
                    "filename": "GFPGANv1.4.pth",
                    "upscale_factor": 4,
                    "upsample_img": "previous_result:upscale"
                }
            },
            "result": { "content_type": "image/jpeg" }
        }
    ]
}

This gives the best results: the super-resolution model handles background detail while the face model handles facial features, composited together at the upscaled resolution.

Object Segmentation

Detect and segment objects using text prompts via GroundingDINO + SAM2. Returns a binary mask image suitable for inpainting workflows.

{
    "task": {
        "command": "segment",
        "arguments": {
            "image": "previous_result:input_image",
            "prompt": "dog"
        }
    },
    "result": { "content_type": "image/png" }
}

Argument	Required	Description
image	Yes	PIL Image or previous_result: reference - a video runs frame by frame, see Videos in image tasks
prompt	Yes	Text description of object(s) to detect (e.g., "dog", "red car")
model_name	No	GroundingDINO model ID (default: IDEA-Research/grounding-dino-base)
sam_model_name	No	SAM2 model ID (default: facebook/sam2-hiera-large)
threshold	No	Detection confidence threshold (default: 0.3)
invert	No	Invert the output mask (default: false)

Returns a grayscale PIL Image (mode "L") — white (255) for detected objects, black (0) for background. Use with inpainting pipelines like FluxFillPipeline.

Examples:

• segment.json — Segment an object from an image
• segment-and-inpaint.json — Segment, then inpaint the masked region

Image Captioning

Generate text captions from images using a vision-language model.

Transformers 5 removed the dedicated image-to-text pipeline this task used to build, along with the BLIP/ViT-GPT2/GIT captioning models that ran on it. Captioning now goes through the same image-text-to-text pipeline as any other VLM, so model_name needs a vision-language model (SmolVLM, Qwen2.5-VL, LLaVA, etc.) and prompt is a question put to the model rather than a text fragment to continue.

{
    "task": {
        "command": "image_to_text",
        "arguments": {
            "image": "previous_result:input_image"
        }
    },
    "result": { "content_type": "text/plain" }
}

Argument	Required	Description
image	Yes	PIL Image, URL/path, or previous_result: reference
model_name	No	HuggingFace vision-language model ID (default: HuggingFaceTB/SmolVLM-256M-Instruct)
prompt	No	What to ask about the image (default: Describe this image.) — ask a narrower question for a narrower caption
system_prompt	No	System instruction for the model
max_new_tokens	No	Maximum tokens to generate (default: 50)

The default model is deliberately tiny, matching the footprint of the old captioning default; it produces short, plain captions. Point model_name at something larger for detail.

Returns a caption string. Save as text/plain for .txt output, or pass to a downstream step via previous_result: as a prompt for image generation.

For a detailed caption, hand the image to text_generation with a question and a larger vision-language model; that is what describe-and-regenerate.json does ahead of its prompt expansion.

Examples:

• image-to-text.json — Basic captioning with the default model, saves as .txt
• image-to-text.json — Larger VLM answering a specific question
• describe-and-regenerate.json — Describe an image, expand the caption, then regenerate it

Composing Text

Assemble one block of text out of parts written once. A multi-shot workflow says the same things about its characters in every shot — who they are, what they are wearing, what their voice sounds like — and the engine deliberately has no string interpolation to splice them in with (see the no-interpolation rule in the workflow guide). Composition is the way round it: a part is a whole value, and compose_text joins parts in order.

{
    "name": "shot_1_prompt",
    "task": {
        "command": "compose_text",
        "arguments": {
            "parts": [
                "variable:character_a_bible",
                "variable:character_a_voice",
                "variable:shot_1_action"
            ],
            "separator": "\n\n"
        }
    }
},
{
    "name": "shot_1",
    "pipeline": { "arguments": { "prompt": "previous_result:shot_1_prompt" } }
}

Argument	Required	Description
parts	Yes	The parts to join, in order — each a whole value, usually a variable:, prompt: or previous_result: reference. Numbers are written out; null is dropped, so an optional part can be a variable left null
separator	No	What goes between the parts (default: a blank line, the paragraph break the prompt formats use)
skip_empty	No	Drop parts that are null or blank (default true). With it off, an empty part still contributes its separator

A part that is neither text nor a number is an error, not a coercion: it means the reference in that position resolved to something other than the text meant.

The parts are positional. A named form ("{bible} says {line}") would be the interpolation the engine does not have, one layer down — so a character bible is a variable named by every shot that needs it, and a voice string written once is checked by being the same value rather than by being compared.

Extracting Sections

Reduce generated text to a known set of labelled sections, dropping anything else:

{
    "task": {
        "command": "extract_sections",
        "arguments": {
            "text": "previous_result:expand",
            "sections": ["integrated_multimodal_description", "overall_soundscape", "non_diegetic_music"]
        }
    },
    "result": { "content_type": "text/plain" }
}

Argument	Required	Description
text	Yes	The generated text, usually a previous_result: reference
sections	Yes	Section labels to keep, in the order they should appear
keep_preamble	No	Keep any text before the first label (default: true)

A section runs from its label: to the end of that paragraph, so a blank line ends one and a single newline does not — a field holding one line per item stays intact. Repeats are dropped, missing sections are skipped, and text with no recognised label is returned unchanged.

This exists because a model asked for a rigid format usually produces it and then keeps going — restating the description, appending a summary, or looping until it runs out of tokens. Prompting against that is unreliable, and at small model sizes adding rules to an already long specification can make adherence worse. Trailing text is not free either: a prompt is conditioning, and a pipeline that does not truncate spends memory and attention on whatever arrives. Keeping the fields that were asked for is deterministic where prompting is not.

The built-in h3_context_ir workflow applies this to its own output, so a workflow delegating to it receives only the fields MiniMax H3 expects.

Text Generation / Prompt Expansion

Generate or expand text using a local language model. Useful for expanding short prompts into detailed image generation prompts, rewriting text, or other text-to-text tasks.

{
    "task": {
        "command": "text_generation",
        "arguments": {
            "prompt": "a cat on a windowsill",
            "system_prompt": "You are a helpful AI assistant that creates detailed prompts for text to image generative AI. When supplied input generate only the prompt, no other text."
        }
    },
    "result": { "content_type": "text/plain" }
}

Argument	Required	Description
prompt	Yes	The user message or short prompt to expand/transform
system_prompt	No	System instruction for the model (e.g., "expand this into a detailed image prompt")
model_name	No	HuggingFace model ID (default: Qwen/Qwen2.5-1.5B-Instruct, or HuggingFaceTB/SmolVLM-256M-Instruct when an image is supplied)
image	No	PIL Image, URL/path, or previous_result: reference — see below
repetition_penalty	No	Vision path only (default: 1.15) — see below
generate_kwargs	No	Anything else to pass to the model's generate() — no_repeat_ngram_size, top_p, min_new_tokens. Merged last, so it overrides the settings above
max_new_tokens	No	Maximum tokens to generate (default: 500)

Writing a prompt from a picture

Supplying image switches the task to a vision-language model, so the generated text describes what is actually in the picture instead of what the prompt guesses is there. model_name must then name a VLM — a text-only model cannot be loaded as one.

{
    "task": {
        "command": "text_generation",
        "arguments": {
            "prompt": "Write a video prompt that starts from this picture.",
            "image": "previous_result:input_image",
            "model_name": "Qwen/Qwen3-VL-4B-Instruct"
        }
    },
    "result": { "content_type": "text/plain" }
}

This matters most ahead of an image-conditioned generation step. Those pipelines pin the supplied picture as the first frame, so a prompt written without seeing it will describe a scene the keyframe contradicts and the two conditionings pull against each other. Pass the same image to both and the prompt agrees with the frame it opens on.

A vision model is large enough to be worth releasing before the generation model loads — see release_models in the workflow guide.

Generation stays greedy so a workflow reproduces, but greedy decoding against a long, rigid format specification makes these models loop — emitting a complete answer and then repeating its closing sections until the token budget runs out. The vision path applies a repetition_penalty of 1.15 to stop that. Measured on Qwen3-VL against the MiniMax H3 prompt spec, 1.05 still looped through the whole budget while 1.15 ended on its own at a length matching the format's own guidance. Raise it if a model still repeats itself, or set 1.0 to disable.

A penalty reins the looping in but does not guarantee the model stops where the format ends; for that, trim the output with extract_sections below.

There is a limit to what a small model will follow. Against the MiniMax H3 spec, neither Qwen3-VL-4B nor 8B produces the <d>[Language]...</d> dialogue tag or the (S1) speaker ids, whether the idea implies speech or supplies the line verbatim; the 8B is worse on layout, capitalising its section labels. Showing a complete worked example does produce them - by copying the example word for word, which is useless - and a placeholder skeleton does not produce them at all. The visual description these models write is grounded and usable; the dialogue markup is not. Write prompts by hand where a subject has to speak.

For anything the arguments above do not cover, generate_kwargs goes straight to generate():

"arguments": {
    "prompt": "a cat on a windowsill",
    "generate_kwargs": { "no_repeat_ngram_size": 25 }
}

It is merged after everything else, so it can override repetition_penalty and the sampling settings as well as add to them.

Examples:

• expand-prompt.json — Expand a short prompt and save as .txt
• expand-prompt.json — Expand prompt, then generate with Flux

Speech Generation

Speak a line of text with a local text-to-speech model. The result is a waveform carrying the rate its model generated at, so it composes with slice_audio, fade_audio and pair_audio directly (concat_videos and dissolve_videos join videos — pair the track onto a video first).

{
    "task": {
        "command": "generate_speech",
        "arguments": {
            "text": "The way ahead is longer still.",
            "voice_preset": "v2/en_speaker_6"
        }
    },
    "result": { "content_type": "audio/wav" }
}

Argument	Required	Description
text	One of text/messages	The line to speak
messages	One of text/messages	Chat-templated input for a model such as VibeVoice that takes a conversation rather than a bare string — a list of {"role": ..., "content": ...} dicts, passed straight through as the pipeline's text_inputs so the model's own chat template applies. A model with no chat template configured (Bark and friends) raises when handed this instead of text
model_name	No	HuggingFace model ID (default: suno/bark-small)
voice_preset	No	The speaker, for a model with presets — v2/en_speaker_0 through v2/en_speaker_9 for Bark. A model with no processor (a single-voice model such as facebook/mms-tts-eng) refuses a voice_preset with an error rather than ignoring it
speaker_embedding	No	A reference audio file (typically an asset: reference) whose voice a SpeechT5 model should speak in. Reduced to an x-vector with speechbrain's spkrec-xvect-voxceleb and injected into forward_params as speaker_embeddings. A model that isn't SpeechT5 refuses it the same way a single-voice model refuses voice_preset
forward_params	No	Passed to the model's forward/generate call
generate_kwargs	No	Ad-hoc generation settings for a generative model — temperature, do_sample

The command's own default is Bark because its voice presets give distinct speakers, which is what two characters in a scene need; facebook/mms-tts-eng is a quarter the size and a good override where one voice will do. generate-speech.json ships with facebook/mms-tts-eng as its own default instead, since a template is usually a single voice. voice_preset is a preprocessing argument — it selects the speaker before generation rather than parameterizing it — so naming it here is what makes it reach the processor. Passed through forward_params it would be dropped and every character would sound the same.

speaker_embedding is the same kind of preprocessing argument for a SpeechT5 model (microsoft/speecht5_tts), which conditions its voice on an x-vector rather than a preset name. A VITS model's speaker is different again — a plain speaker_id int, passed through forward_params unchanged, since it was never a preprocessing argument and needs no argument of its own here.

The result needs no sample_rate. A generated track carries the rate its model produced it at, and that beats the 44100 default; declaring one still wins over both, for a track whose rate was reported wrong. Every TTS model runs at a different rate, so a declared rate that does not match plays the speech at the wrong speed and pitch without ever failing.

Generating a voice to condition on

The role this earns its place in is voice timbre reference, not the track a mouth follows. MiniMax H3 lip-syncs well when it generates the speech itself and poorly when it must follow supplied audio, so its MiniMaxH3AudioReference takes a few seconds of a voice to fix timbre, pitch and delivery while H3 still generates the line. Build the reference with from_previous_result and the clip's own sample rate comes across with it:

"references": [
    {
        "reference_type": "diffusers.modular_pipelines.minimax_h3.MiniMaxH3AudioReference",
        "from_previous_result": "voice"
    }
]

Referencing the same preset in every shot of a scene makes a character's voice a conditioning signal rather than a prose description that has to land identically a dozen times. The other honest uses are a voice that must be matched — a specific delivery H3 will not produce from description alone — and narration over shots where nothing has to lip-sync to it, muxed with pair_audio.

A speech model is worth releasing before a video model loads — set release_models on the step, as in the example below.

Examples:

• generate-speech.json — Speak a line and save it as a .wav
• voice-timbre-reference.json — Generate a voice, then condition H3's <Audio 1> on it

Speech Transcription

Transcribe spoken audio to text with a local Whisper-class model. The word-correctness of a TTS deliverable — a dropped line, a mid-sentence truncation — can only be inferred from duration and timing arithmetic without this; transcribe_audio checks it directly against the text the deliverable was supposed to speak.

{
    "task": {
        "command": "transcribe_audio",
        "arguments": {
            "audio": "previous_result:speak"
        }
    },
    "result": { "content_type": "text/plain" }
}

Argument	Required	Description
audio	Yes	Path or URL of an audio file (or of a video file, whose soundtrack is taken), a video with a soundtrack, or a waveform — usually a previous_result: reference
sample_rate	No	Sample rate of a waveform passed directly
model_name	No	HuggingFace model ID of a Whisper-class ASR model (default: openai/whisper-base)
timestamps	No	"segment" or "word" to get chunk timings instead of plain text (see below)

Multi-channel audio is downmixed to mono and resampled to 16 kHz before transcription, since that is what a Whisper-class model is trained on; the source audio itself is untouched. By default the result is plain text, read with MCP's get_output_text.

Set timestamps to "segment" or "word" to get chunk timings instead — a music video cut to the lyric, or a dialogue shot checked against its line, needs the times Whisper already produces past 30 s rather than the collapsed string. The result becomes {"text": ..., "chunks": [{"start": ..., "end": ..., "text": ...}, ...]}, so the step's result.content_type must be "application/json" rather than "text/plain", and it's read with MCP's get_output_text (JSON results are text). A clip under 30 s asks Whisper for timestamps explicitly when timestamps is set — the 30 s long-form threshold is a separate, unrelated reason to ask.

Example: transcribe-audio.json — Transcribe an audio file to text.

Frame Interpolation

Increase video frame rate using RIFE (Real-Time Intermediate Flow Estimation). Takes a video and inserts intermediate frames between each pair. The result is one video artifact without a soundtrack - the frame count changed, so pair_audio is how the original track comes back. interpolate-frames.json shows the interpolation itself.

{
    "task": {
        "command": "interpolate_frames",
        "arguments": {
            "video": "previous_result:generate_video",
            "multiplier": 2
        }
    },
    "result": { "content_type": "video/mp4", "fps": 60 }
}

Argument	Required	Description
video	Yes	The frames - a frame list, a frame array, or an audio+video pair from a concat or dissolve step (its audio is dropped) - usually a previous_result: reference
multiplier	No	Frame count multiplier: 2, 4, or 8 (default: 2)
model_name	No	HuggingFace repo with RIFE v4.13 weights (default: imaginairy/rife-interpolation)
filename	No	Weights filename within the repo (default: rife-flownet-4.13.2.safetensors)

Uses vendored IFNet v4.13 architecture. Weights are downloaded from HuggingFace Hub on first use.

Example: interpolate-frames.json — Generate video with Mochi, then 2x interpolate from 30fps to 60fps.

Metadata Embedding

Embed generation parameters in saved images. Enable by setting embed_metadata: true in a step's result configuration:

{
    "result": {
        "content_type": "image/png",
        "embed_metadata": true
    }
}

Format	Storage	Notes
PNG	Text chunk (parameters key)	Always available
JPEG/WebP	EXIF UserComment	Requires pip install piexif

Metadata includes step name, model name, and generation arguments (prompt, steps, guidance scale, etc.) as JSON.

Example: embed-metadata.json — Generate with Flux and embed parameters in PNG.

QR Code Generation

{
    "task": {
        "command": "qr_code",
        "arguments": {
            "qr_code_contents": "https://example.com"
        }
    }
}

Argument	Required	Description
qr_code_contents	Yes	Data to encode (URL, text, etc.)
height	No	Used with width to derive output resolution (default: 768)
width	No	Used with height to derive output resolution (default: 768)

The QR code is generated then resampled to max(height, width), aligned to the nearest 64px multiple.

Example: qr-code.json — QR code with artistic ControlNet

Chat/Dict Plumbing

These small tasks glue together multi-step pipelines that mix raw transformers components with task steps — for the cases text_generation does not cover.

format_chat_message

Build a text_inputs chat message list from a system and user message, in the shape a transformers.pipeline text-generation call expects:

{
    "task": {
        "command": "format_chat_message",
        "arguments": {
            "system_prompt": "You are a helpful assistant.",
            "user_message": "variable:prompt"
        }
    }
}

Argument	Required	Description
system_prompt	Yes	System instruction
user_message	Yes	User message content

Returns {"text_inputs": [{"role": "system", ...}, {"role": "user", ...}]}. Pass the result to a transformers.pipeline step's text_inputs argument via previous_result:.

get_dict_value

Extract a single value from a dictionary result (e.g., a transformers pipeline's output) for use in a later step:

{
    "task": {
        "command": "get_dict_value",
        "arguments": {
            "dict": "previous_result:augment_prompt",
            "key": "generated_text"
        }
    }
}

Argument	Required	Description
dict	Yes	Dictionary (or previous_result: reference) to read from
key	Yes	Key to extract

Returns the value at key, or None if the key is absent.

batch_decode_post_process

Decode generated token IDs and run model-specific post-processing (e.g., Florence-2's task-token parsing), using the processor from an earlier pipeline step:

{
    "task": {
        "command": "batch_decode_post_process",
        "pipeline_reference": "describe_image_processor",
        "arguments": {
            "generated_ids": "previous_result:describe_image_model.generated_ids",
            "task": "<DETAILED_CAPTION>"
        }
    }
}

Argument	Required	Description
pipeline_reference	Yes	Name of an earlier pipeline step whose processor to reuse (sibling of command/arguments, not inside arguments)
generated_ids	Yes	Token IDs to decode (e.g., a model step's generated_ids output)
task	Yes	Task token to post-process for (e.g., <DETAILED_CAPTION>)

Calls processor.batch_decode(...) then processor.post_process_generation(..., task=task) and returns parsed_answer[task].

Multi-Step Example

Canny edge detection followed by ControlNet generation:

{
    "steps": [
        {
            "name": "edges",
            "task": {
                "command": "canny",
                "arguments": {
                    "image": {
                        "location": "photo.jpg",
                        "low_threshold": 50,
                        "high_threshold": 200
                    }
                }
            },
            "result": { "content_type": "image/jpeg" }
        },
        {
            "name": "generate",
            "pipeline": {
                "configuration": {
                    "component_type": "FluxControlPipeline",
                    "offload": "sequential"
                },
                "from_pretrained_arguments": {
                    "model_name": "black-forest-labs/FLUX.1-Canny-dev",
                    "torch_dtype": "torch.bfloat16"
                },
                "arguments": {
                    "control_image": "previous_result:edges",
                    "prompt": "a watercolor painting",
                    "num_inference_steps": 50
                }
            },
            "result": { "content_type": "image/jpeg" }
        }
    ]
}

Examples

• controlnet.json — Canny edge ControlNet
• controlnet.json — Depth-guided generation
• qr-code.json — QR code with artistic ControlNet
• upscale-spandrel.json — Spandrel 4x upscale of an existing image
• restore-faces.json — Generate portrait + GFPGAN face restoration
• segment.json — Text-prompted object segmentation
• segment-and-inpaint.json — Segment + inpaint
• image-to-text.json — image captioning with the SmolVLM default
• image-to-text.json — VLM captioning with a specific question
• describe-and-regenerate.json — Describe, expand, then regenerate
• interpolate-frames.json — RIFE frame interpolation
• embed-metadata.json — Embed generation parameters in PNG
• expand-prompt.json — LLM prompt expansion
• expand-prompt.json — Expand prompt + generate image
• upscale-spandrel.json — Spandrel upscale of an existing image
• upscale-diffusion.json — Diffusion upscale of an existing image
• audio-trim-fade.json — Trim a generated track and fade its tail
• generate-speech.json — Speak a line with a local text-to-speech model
• voice-timbre-reference.json — Generate a voice and condition H3's <Audio 1> on it
• dissolve-between-shots.json — Dissolve between supplied shots and mix a score under their own audio