Skip to content

A burned-in overlay is segmented as foreground, and no setting can exclude it #80

Open

Description

@smb

A burned-in overlay is read as foreground and survives into the cutout. On a
Twitch clip with a chat box composited into the bottom-left of the frame, the
segmenter keeps the chat as a hard-edged rectangular block in every frame, so
the exported loop carries a slab of chat text next to the subject. Burned-in
logos, alert boxes and webcam frames have the same shape of problem.

The user has no setting that addresses it. Crop cuts from the frame edges, and
the overlay sits in front of the subject's chest for part of the clip, so
cropping it away takes the subject with it.

What does not work

Keeping only the largest connected alpha component was measured on the clip
above at four timestamps and is not the answer:

  • The chat block touches the subject's shoulder, so at one sample it is 100 % of
    the same component and stays exactly where it was.
  • At two other samples the filter instead deletes the subject's own raised hand,
    which is genuinely detached from the body in the mask.

A share threshold does not rescue it, because the failure is that the wanted and
unwanted regions are one component.

Proposal

An exclusion region: one rectangle the user drags over the overlay, forced to
alpha 0 after segmentation. A burned-in overlay stays in the same place for the
whole clip, so a single rectangle covers the entire video, and the result is
predictable rather than model-dependent.

  • Lives with the other cleanup controls, next to the alpha threshold and
    padding, and is off until a rectangle exists.
  • The rubber-band interaction is already implemented for crop in
    src/matteloop/ui/crop_canvas.py; this should reuse it rather than grow a
    second selection widget.
  • The rectangle belongs in FramingSpec alongside the existing framing values,
    so it takes part in the fingerprint and a change invalidates the cut the same
    way a crop change does.

Automatic detection was considered and rejected for now: an overlay is not
reliably static (chat text scrolls) and a genuinely motionless subject would be
eaten by a variance test.

Activity

  1. self-assigned this
    on Sep 12, 2026
  2. added
    deferredParked: work exists but is on hold, not abandoned
    on Sep 12, 2026
  3. removed their assignment
    on Sep 12, 2026
  4. smb commented on Sep 12, 2026

    @smb
    CollaboratorAuthor

    Parked, not abandoned — labelled deferred and unassigned so nobody assumes it is being worked on.

    The work exists on feat/issue-80-exclusion-regions (PR #166, now a draft) and is green, but does not behave correctly in real use: the cutout does not match what the region appears to select. The mapping itself is measured correct and auto-trim was off, so the two obvious explanations are ruled out; the defect was not found before the pause.

    PR #166 carries the full state, what was measured, and what to do first when resuming.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    deferredParked: work exists but is on hold, not abandonedenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions