Dolby Vision Profile 5 / 10.0 scrub previews drawn by the system compositor

CHANGELOG.md

@@ -10,7 +10,24 @@ the public-API contract.
 
 ## [Unreleased]
 
-_Nothing yet._
+### Added
+
+- **Dolby Vision Profile 5 / 10.0 scrub previews in colour, drawn by the system compositor.**
+  `scrubPreviewDrawsInView`, `showScrubPreview(atSeconds:in:)` and `AetherScrubPreviewView`: the
+  keyframe that opens a resident segment becomes a one-sample fMP4, and a muted `AVPlayer` inside
+  the view shows it, so the colour comes from the same place as the playing picture rather than from
+  a reshaping of our own. Needs no `DolbyVisionStillReshaping` trait. Positions outside the resident
+  window read the segment's keyframe off the source with a side demuxer, as the I-frame rendition
+  does, and get the same gates (no read on a sequential, serial or held-connection origin, waits out a
+  seek, skips superseded requests); before, the preview only covered the cached span around the
+  playhead. 26 to 45 ms per new position on an M1, 18 to 25 ms for one read off a local source,
+  byte-identical to the source's keyframe; verified next to 4K Profile 5 playback on an Apple TV.
+- **The held picture of an audio switch covers Dolby Vision Profile 5 / 10.0 without the reshaping
+  trait.** The resident segment's video becomes a standalone file, and a player on the bound surface
+  seeks it frame-exactly to the outgoing position, so the compositor colours the held frame. It was
+  skipped before. On three Profile 5 clips the held frame is pixel-identical to the paused outgoing
+  picture, up in 165 to 220 ms on an M1. The offset adds the segment's B-frame delay; without it the
+  frame lands two frames early.
 
 ## [7.40.1] - 2026-10-18
 

README.md

@@ -626,7 +626,7 @@ Browse all of this as a searchable site at **[aetherengine.superuser404.de](http
 - **[docs/api.md](docs/api.md)**: every public surface a host consumes, and the contracts that require it to act (how a load ends, `.ended`, the live retune, the audio tap, what must be set before `load`, and which FFmpeg your link hands the engine). A test fails the build when a host-facing public symbol is named nowhere in the docs.
 - **[docs/architecture.md](docs/architecture.md)**: the three playback pipelines, the source-file map, dependencies, the SwiftUI `Menu` pattern.
 - **[docs/formats.md](docs/formats.md)**: codec / container coverage, HDR routing, audio bridging, subtitles, frame extraction, disc playback, live ingest, and known limitations.
-- **[docs/cli.md](docs/cli.md)**: the `aetherctl` repro CLI (twenty-one subcommands).
+- **[docs/cli.md](docs/cli.md)**: the `aetherctl` repro CLI (twenty-four subcommands).
 - **[CHANGELOG.md](CHANGELOG.md)**: per-release index.
 
 ## Stability and versioning

Sources/AetherEngine/AetherEngine+CompositorPreview.swift

@@ -0,0 +1,42 @@
+import Foundation
+
+/// Where a compositor-drawn scrub preview landed (`AetherEngine.showScrubPreview(atSeconds:in:)`).
+public struct ScrubPreviewPlacement: Sendable, Equatable {
+    /// Playlist/output seconds of the keyframe on screen.
+    public let actualSeconds: Double
+    /// The span the same picture stands for: the plan segment that keyframe opens.
+    public let validRange: Range<Double>
+}
+
+extension AetherEngine {
+
+    /// True when this session's scrub previews come through `showScrubPreview(atSeconds:in:)` and an
+    /// `AetherScrubPreviewView` rather than a `CGImage`: a native VOD session of Dolby Vision without
+    /// a displayable base layer (Profile 5, AV1 Profile 10.0), whose colour only the system
+    /// compositor applies. Independent of the `DolbyVisionStillReshaping` trait. Inside the resident
+    /// window the picture comes out of the segment cache, like `scrubThumbnail`; outside it, out of
+    /// the source's own keyframe, read by a side demuxer.
+    public var scrubPreviewDrawsInView: Bool {
+        guard !isLive, let session = nativeVideoSession else { return false }
+        return session.previewNeedsCompositor
+    }
+
+    /// Shows the keyframe that opens the plan segment holding `seconds` (playlist/output axis, as
+    /// `scrubThumbnail` takes it) in `view`: from the segment cache when that segment is resident,
+    /// otherwise read off the source the way the I-frame rendition reads its entries. Nil when
+    /// `scrubPreviewDrawsInView` is false, the segment is not resident and the origin admits no side
+    /// read (sequential, serial or held connection, disc, not reopenable by URL), the read failed, a
+    /// newer request superseded this one, or the session changed meanwhile; the view keeps its
+    /// previous picture in every case.
+    public func showScrubPreview(atSeconds seconds: Double, in view: AetherScrubPreviewView) async -> ScrubPreviewPlacement? {
+        guard seconds.isFinite, scrubPreviewDrawsInView, let session = nativeVideoSession else { return nil }
+        let gen = loadGeneration
+        let file = await BlockingWork.detached(priority: .userInitiated) { [session] in
+            session.compositorPreviewFile(atSeconds: seconds)
+        }.value
+        guard let file, gen == loadGeneration, !Task.isCancelled else { return nil }
+        guard await view.present(file.url), gen == loadGeneration else { return nil }
+        return ScrubPreviewPlacement(actualSeconds: file.startSeconds,
+                                     validRange: file.startSeconds..<(file.startSeconds + file.durationSeconds))
+    }
+}

Sources/AetherEngine/AetherEngine+HeldPicture.swift

@@ -55,16 +55,16 @@ extension AetherEngine {
         let generation = loadGeneration
         let started = ContinuousClock.now
 
+        if let host = nativeSource,
+           Self.heldPictureNeedsReshaping(videoFormat: videoFormat, dolbyVisionProfile: sourceDVProfile),
+           !OptionalComponent.dolbyVisionStillReshaping.isBuiltIn {
+            await holdThroughCompositor(host: host, view: view, generation: generation, started: started)
+            return
+        }
+
         let held: (frame: CVPixelBuffer, isHDR: Bool, route: String)?
         if let host = nativeSource {
             if Self.heldPictureNeedsReshaping(videoFormat: videoFormat, dolbyVisionProfile: sourceDVProfile) {
-                // Without the reshaping there is no correct picture to hold, so nothing is decoded.
-                guard OptionalComponent.dolbyVisionStillReshaping.isBuiltIn else {
-                    heldPictureLastSkip = "Dolby Vision still reshaping not built in"
-                    EngineLog.emit("[AetherEngine] held picture: skipped (Dolby Vision still reshaping not built in, "
-                                   + "DolbyVisionStillReshaping trait off)", category: .engine)
-                    return
-                }
                 held = await reshapedDolbyVisionStill(itemSeconds: host.avPlayer.currentTime().seconds)
                     .map { ($0, false, "Dolby Vision reshaped") }
             } else {
@@ -92,8 +92,58 @@ extension AetherEngine {
             EngineLog.emit("[AetherEngine] held picture: frame could not be wrapped for display", category: .engine)
             return
         }
+        armHeldPicture(on: view, route: held.route, nativeSource: nativeSource)
+        EngineLog.emit(
+            "[AetherEngine] held picture: up after \(Self.milliseconds(ContinuousClock.now - started))ms "
+            + "(\(held.route), \(CVPixelBufferGetWidth(held.frame))x\(CVPixelBufferGetHeight(held.frame)), "
+            + "hdr=\(held.isHDR))",
+            category: .engine)
+    }
+
+    /// Dolby Vision without a displayable base layer and without the still reshaping: the frame under
+    /// the playhead, played out of the resident segment by a player of the surface's own, so the
+    /// system compositor colours it as it colours playback. Waits at most
+    /// `heldPictureDolbyVisionBudgetSeconds`, the outgoing picture paused meanwhile.
+    private func holdThroughCompositor(host: NativeAVPlayerHost, view: any HeldStillSurface,
+                                       generation: UInt64, started: ContinuousClock.Instant) async {
+        let itemSeconds = host.avPlayer.currentTime().seconds
+        guard let session = nativeVideoSession else { return }
+        let built = await BlockingWork.detached(priority: .userInitiated) { [session] in
+            session.compositorHoldFile(atSeconds: itemSeconds)
+        }.value
+        guard let hold = built else {
+            heldPictureLastSkip = "no frame from the outgoing picture"
+            EngineLog.emit("[AetherEngine] held picture: no resident segment for the compositor hold", category: .engine)
+            return
+        }
+        let spent = Double((ContinuousClock.now - started).components.seconds)
+            + Double((ContinuousClock.now - started).components.attoseconds) / 1e18
+        let budget = max(0.2, Self.heldPictureDolbyVisionBudgetSeconds - spent)
+        let shown = await view.showCompositorStill(file: hold.url, pastFirstFrameBy: hold.offsetSeconds,
+                                                   gravity: videoGravity, budgetSeconds: budget)
+        guard shown else {
+            heldPictureLastSkip = "no frame from the outgoing picture"
+            EngineLog.emit("[AetherEngine] held picture: the compositor hold did not become displayable", category: .engine)
+            return
+        }
+        guard loadGeneration == generation, nativeHost === host, (boundStillView ?? boundView) === view else {
+            view.clearStill()
+            heldPictureLastSkip = "superseded while the frame was read"
+            EngineLog.emit("[AetherEngine] held picture: superseded while the frame was read", category: .engine)
+            return
+        }
+        armHeldPicture(on: view, route: "Dolby Vision compositor", nativeSource: host)
+        EngineLog.emit(
+            "[AetherEngine] held picture: up after \(Self.milliseconds(ContinuousClock.now - started))ms "
+            + "(Dolby Vision compositor, item \(String(format: "%.3f", itemSeconds))s)",
+            category: .engine)
+    }
+
+    /// Records the hold and arms its release: the next item's first frame (native), the next host's
+    /// (software), or `heldPictureTimeoutSeconds`.
+    private func armHeldPicture(on view: any HeldStillSurface, route: String, nativeSource: NativeAVPlayerHost?) {
         heldPictureView = view
-        heldPictureLastRoute = held.route
+        heldPictureLastRoute = route
         heldPictureShownAt = .now
         heldPictureToken &+= 1
         let token = heldPictureToken
@@ -113,11 +163,6 @@ extension AetherEngine {
             guard let self, self.heldPictureToken == token, self.heldPictureView != nil else { return }
             self.releaseHeldPicture(reason: "timeout")
         }
-        EngineLog.emit(
-            "[AetherEngine] held picture: up after \(Self.milliseconds(ContinuousClock.now - started))ms "
-            + "(\(held.route), \(CVPixelBufferGetWidth(held.frame))x\(CVPixelBufferGetHeight(held.frame)), "
-            + "hdr=\(held.isHDR))",
-            category: .engine)
     }
 
     /// visionOS has no external playback, and no `isExternalPlaybackActive` to ask.

Sources/AetherEngine/AetherEngine+ScrubThumbnail.swift

@@ -150,7 +150,7 @@ extension AetherEngine {
     /// FrameExtractor (#106). It reports capability, not per-frame availability: transient
     /// nils from `scrubThumbnail` while a segment is still being produced are expected. False as well
     /// on a Dolby Vision Profile 5 / 10.0 session in a build without the `DolbyVisionStillReshaping`
-    /// trait, where no frame can become a correct still.
+    /// trait, where no frame can become a correct still; `scrubPreviewDrawsInView` covers that source.
     public var supportsCacheBackedStills: Bool {
         if let session = nativeVideoSession { return session.yieldsStills }
         guard let softwareHost else { return false }

Sources/AetherEngine/Video/CompositorPreviewBuilder.swift

@@ -0,0 +1,126 @@
+// Sources/AetherEngine/Video/CompositorPreviewBuilder.swift
+import Foundation
+import AetherLibavcodec
+import AetherLibavformat
+import AetherLibavutil
+
+/// Turns a resident segment into a standalone video-only fMP4 that a second AVPlayer can show through
+/// the system compositor. That is the only place a Dolby Vision Profile 5 / 10.0 picture gets its
+/// colour on Apple platforms: no still API applies the RPU, and an `AVPlayerItemVideoOutput` on the
+/// item switches that processing off for the layer too.
+///
+/// The packets come out of the segment the session already muxed, so they carry every conversion the
+/// playing picture got, and the init is rebuilt from the segment's own demuxed parameters: the
+/// sample entry, `dvcC` and `colr` therefore say what the playing item says. The init carries no
+/// audio track, so the item has nothing to wait for.
+enum CompositorPreviewBuilder {
+    enum Content {
+        /// The keyframe that opens the segment, alone, at file time 0 (scrub preview).
+        case firstKeyframe
+        /// Every video sample of the segment, so a seek can land on any frame of it (held picture).
+        /// Where its first frame sits on the player's clock is the player's business: AVFoundation
+        /// honours the init's edit list and starts at 0, ffmpeg reports the composition delay.
+        case wholeSegment
+    }
+
+    struct Output {
+        let file: Data
+        /// How far the segment's first frame sits behind its first decode time (the B-frame delay).
+        /// A frame's offset past the standalone file's first frame is its item time minus the segment's
+        /// start plus this delay: measured pixel-exact against the paused outgoing picture on three
+        /// Profile 5 clips (24 fps with 0.083 s, 30 fps with 0.067 s, and one cut 1.2 s off a keyframe),
+        /// where leaving the delay out lands two frames early. 0 for `.firstKeyframe`.
+        let compositionDelaySeconds: Double
+    }
+
+    static func build(initData: Data, segmentURL: URL, durationSeconds: Double,
+                      codecTagOverride: String?, colorOverride: MP4SegmentMuxer.ColorOverride?,
+                      stagingDir: URL, content: Content = .firstKeyframe) -> Output? {
+        guard let segment = try? Data(contentsOf: segmentURL, options: .alwaysMapped) else { return nil }
+        let demuxer = Demuxer()
+        defer { demuxer.close() }
+        do {
+            try demuxer.open(reader: DataIOReader(parts: [initData, segment]), formatHint: "mp4",
+                             profile: .stillExtraction)
+        } catch {
+            return nil
+        }
+        let videoIdx = demuxer.videoStreamIndex
+        guard videoIdx >= 0, let stream = demuxer.stream(at: videoIdx),
+              let codecpar = stream.pointee.codecpar else { return nil }
+        demuxer.discardAllStreamsExcept([videoIdx])
+        let timeBase = stream.pointee.time_base
+        guard timeBase.num > 0, timeBase.den > 0 else { return nil }
+        let video = MP4SegmentMuxer.VideoConfig(
+            codecpar: UnsafePointer(codecpar), timeBase: timeBase,
+            codecTagOverride: codecTagOverride, doviConfig: .keep, colorOverride: colorOverride)
+
+        switch content {
+        case .firstKeyframe:
+            // The segment opens on its keyframe; a bound keeps a malformed one from walking the file.
+            var payload: Data?
+            for _ in 0..<64 {
+                guard let packet = try? demuxer.readPacket() else { break }
+                defer { av_packet_unref(packet); av_packet_free_safe(packet) }
+                guard packet.pointee.stream_index == videoIdx,
+                      packet.pointee.flags & AV_PKT_FLAG_KEY != 0, let data = packet.pointee.data,
+                      packet.pointee.size > 0 else { continue }
+                payload = Data(bytes: data, count: Int(packet.pointee.size))
+                break
+            }
+            guard let payload,
+                  let output = IFrameFragmentBuilder(video: video, stagingDir: stagingDir)
+                    .build(payload: payload, index: 0, startSeconds: 0,
+                           durationSeconds: max(durationSeconds, 0.04)) else { return nil }
+            return Output(file: output.initSegment + output.fragment, compositionDelaySeconds: 0)
+
+        case .wholeSegment:
+            return remuxWholeSegment(demuxer: demuxer, videoIdx: videoIdx, video: video, stagingDir: stagingDir)
+        }
+    }
+
+    /// Every video packet, shifted so the first decode time is 0, into one fragment.
+    private static func remuxWholeSegment(demuxer: Demuxer, videoIdx: Int32,
+                                          video: MP4SegmentMuxer.VideoConfig, stagingDir: URL) -> Output? {
+        var initBytes = Data()
+        guard let muxer = try? MP4SegmentMuxer(
+            initialSegmentIndex: 0, sessionDir: stagingDir, video: video, audio: nil,
+            onInitCaptured: { initBytes = $0 }) else { return nil }
+        var completed: URL?
+        defer {
+            _ = muxer.finalize()
+            if let completed { try? FileManager.default.removeItem(at: completed) }
+        }
+        var firstDts: Int64?
+        var firstFramePts: Int64?
+        var written = 0
+        // A segment is a few seconds of one GOP or a handful; the bound only stops a malformed file.
+        for _ in 0..<20_000 {
+            guard let packet = try? demuxer.readPacket() else { break }
+            defer { av_packet_unref(packet); av_packet_free_safe(packet) }
+            guard packet.pointee.stream_index == videoIdx, packet.pointee.size > 0 else { continue }
+            let dts = packet.pointee.dts != Int64.min ? packet.pointee.dts : packet.pointee.pts
+            let pts = packet.pointee.pts != Int64.min ? packet.pointee.pts : dts
+            guard dts != Int64.min else { continue }
+            if firstDts == nil {
+                guard packet.pointee.flags & AV_PKT_FLAG_KEY != 0 else { return nil }
+                firstDts = dts
+            }
+            let base = firstDts ?? dts
+            firstFramePts = min(firstFramePts ?? pts, pts)
+            packet.pointee.dts = dts - base
+            packet.pointee.pts = pts - base
+            packet.pointee.stream_index = muxer.videoOutputStreamIndex
+            av_packet_rescale_ts(packet, video.timeBase, muxer.muxerVideoTimeBase)
+            guard muxer.writePacket(packet).rc >= 0 else { return nil }
+            written += 1
+        }
+        guard written > 0, let firstDts, let firstFramePts,
+              case .completed(let path, _) = muxer.cutFragmentForNextSegment(1) else { return nil }
+        completed = path
+        guard let fragment = try? Data(contentsOf: path), !fragment.isEmpty, !initBytes.isEmpty else { return nil }
+        let ticks = Double(video.timeBase.num) / Double(video.timeBase.den)
+        return Output(file: initBytes + fragment,
+                      compositionDelaySeconds: max(0, Double(firstFramePts - firstDts) * ticks))
+    }
+}