Repository navigation
Dolby Vision Profile 5 / 10.0 scrub previews drawn by the system compositor - #840
Merged
Merged
Conversation
…the system compositor A Profile 5 / AV1 Profile 10.0 base layer is IPT-PQ, and no still API applies the RPU, so the engine cannot hand out a correct CGImage of it without its own reshaping (the DolbyVisionStillReshaping trait, off by default). The system compositor colours it the way it colours the playing picture, so the preview now goes there: - CompositorPreviewBuilder cuts the keyframe that opens a resident segment into a standalone one-sample fMP4, with the init rebuilt from the segment's own parameters (dvh1, dvcC, colr, no audio track). No second connection, like scrubThumbnail. - HLSVideoEngine.compositorPreviewFile keeps the last 12 such files per session, keyed by the segment's identity. - AetherScrubPreviewView shows them through its own muted AVPlayer; two players alternate so the outgoing picture stays up until the next is displayable, and requests are latest-wins. - Public: scrubPreviewDrawsInView, showScrubPreview(atSeconds:in:), ScrubPreviewPlacement. Measured with Shrek the Third (4K Profile 5): 26 to 45 ms from request to picture for a new segment on an M1, 1 ms for the one already shown, colours matching the reshaping converter. A tvOS probe showed the same preview next to 4K Profile 5 playback on an Apple TV 4K without disturbing it. An AVPlayerItemVideoOutput on the item switches the Dolby Vision processing off, so none is attached. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WJcXkGQrMiBPixVEFTSBJS
…ough the system compositor The held picture of an audio switch was skipped for Dolby Vision without a displayable base layer in a build without DolbyVisionStillReshaping. It now plays the frame out of the resident segment: - CompositorPreviewBuilder gains a whole-segment mode that remuxes every video sample into a standalone video-only fMP4 and reports the segment's B-frame delay. - HLSVideoEngine.compositorHoldFile builds it for the item time under the playhead; one file at a time. - HeldStillPresenter gets a compositor form: a muted player on the surface seeks frame-exactly, on its own clock (the track's timeRange start, since AVFoundation applies the init's edit list), behind an invisible layer that turns visible once the frame is displayable. Both surfaces implement it. - holdPictureAcrossItemSwap takes that route, within the same 3 s budget the reshaped still had. The offset into the file is item time minus segment start plus the B-frame delay. Measured by sweeping it against the paused outgoing picture on three Profile 5 clips (24 fps with 0.083 s, a second cut 1.2 s off a keyframe, 30 fps with 0.067 s): pixel-identical only there, two frames early without the delay. Up in 165 to 220 ms on an M1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WJcXkGQrMiBPixVEFTSBJS
…he resident window The compositor preview only covered the cached span around the playhead: a position the segment cache did not hold answered nil at once, and a Profile 5 source has no still to fall back to, so scrubbing ahead showed the time alone (device log: every far target nil in 0 ms, every near one shown in 50 ms). Outside the resident window the preview now reads the keyframe that opens the position's plan segment off the source, the way the I-frame rendition reads its entries: a side demuxer seeks to the segment's start and takes one packet, nothing is decoded, and the session's own sample entry, dvcC and colr wrap it. `IFrameRendition.keyframePayload(at:isStale:)` serves it without a neighbour's keyframe standing in, behind the same gates (no read on a sequential, serial or held-connection origin, waits out a seek or restart) and skipping requests a newer one superseded. The builder sits behind a lock that `stop()` closes before it frees the codec parameters. `aetherctl scrubpreview` drives it. On the 4K Profile 5 Patterns of Nature clip with the source throttled to 20 Mbit/s: resident positions 19 to 28 ms, non-resident ones 18 to 25 ms, 6 of 6 shown; the source-read packets are byte-identical to the source's keyframes at 48 s and 64 s, and hvcC, dvcC and colr match the cache-built files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WJcXkGQrMiBPixVEFTSBJS
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A Dolby Vision Profile 5 / AV1 Profile 10.0 base layer is IPT-PQ, and no still API on Apple platforms applies the RPU, so a
CGImageof it is only correct with our own reshaping (DolbyVisionStillReshaping, off by default). The system compositor colours such a picture the same way it colours playback, so the scrub preview now goes through it.How
CompositorPreviewBuildercuts the keyframe that opens a resident segment into a standalone one-sample fMP4. The init is rebuilt from the segment's own parameters (dvh1,dvcC,colr, no audio track). LikescrubThumbnail, it needs no second connection.HLSVideoEngine.compositorPreviewFile(atSeconds:)keeps the last 12 files per session, keyed by segment identity.AetherScrubPreviewViewshows a file through its own mutedAVPlayer. Two players alternate, so the outgoing picture stays up until the next one is displayable. Requests are latest-wins, andclear()releases the decoders.scrubPreviewDrawsInView,showScrubPreview(atSeconds:in:),ScrubPreviewPlacement. Documented indocs/api.md, with a CHANGELOG entry under Unreleased.Measured
AVPlayerItemVideoOutputto the item switches the Dolby Vision processing off for the layer, which then shows raw IPT (green / magenta). The view never attaches one.Tests
CompositorPreviewBuilderTests(2).Held picture across an audio switch (second commit)
For Dolby Vision without a displayable base layer and without the reshaping trait, this hold used to be skipped. It now goes through the compositor too:
timeRangestart, because AVFoundation applies the init's edit list.ItemSwapStillTests' Profile 5 test now expects the compositor route. It passed against a real Profile 5 clip placed locally under the fixture name. The full suite is green again (779 + 4623, strict pool too).Previews outside the resident window (72fff52)
The first device round showed the preview only near the playhead: a position the segment cache did not hold answered nil at once. Those positions now read the keyframe that opens their plan segment off the source with a side demuxer, the way the I-frame rendition reads its entries (one packet, nothing decoded), behind the same gates (no read on a sequential, serial or held-connection origin, waits out a seek or restart, superseded requests skipped).
aetherctl scrubpreview(new): 6 of 6 positions on the 4K Profile 5 clip, 18 to 25 ms for a source read off a local file; the packets are byte-identical to the source's keyframes, hvcC / dvcC / colr match the cache-built files.🤖 Generated with Claude Code
https://claude.ai/code/session_01WJcXkGQrMiBPixVEFTSBJS