diff --git a/CHANGELOG.md b/CHANGELOG.md index d8c710533..c523a4f7f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,24 @@ the public-API contract. ## [Unreleased] -_Nothing yet._ +### Added + +- **Dolby Vision Profile 5 / 10.0 scrub previews in colour, drawn by the system compositor.** + `scrubPreviewDrawsInView`, `showScrubPreview(atSeconds:in:)` and `AetherScrubPreviewView`: the + keyframe that opens a resident segment becomes a one-sample fMP4, and a muted `AVPlayer` inside + the view shows it, so the colour comes from the same place as the playing picture rather than from + a reshaping of our own. Needs no `DolbyVisionStillReshaping` trait. Positions outside the resident + window read the segment's keyframe off the source with a side demuxer, as the I-frame rendition + does, and get the same gates (no read on a sequential, serial or held-connection origin, waits out a + seek, skips superseded requests); before, the preview only covered the cached span around the + playhead. 26 to 45 ms per new position on an M1, 18 to 25 ms for one read off a local source, + byte-identical to the source's keyframe; verified next to 4K Profile 5 playback on an Apple TV. +- **The held picture of an audio switch covers Dolby Vision Profile 5 / 10.0 without the reshaping + trait.** The resident segment's video becomes a standalone file, and a player on the bound surface + seeks it frame-exactly to the outgoing position, so the compositor colours the held frame. It was + skipped before. On three Profile 5 clips the held frame is pixel-identical to the paused outgoing + picture, up in 165 to 220 ms on an M1. The offset adds the segment's B-frame delay; without it the + frame lands two frames early. ## [7.40.1] - 2026-10-18 diff --git a/README.md b/README.md index 52126a816..41f68270b 100644 --- a/README.md +++ b/README.md @@ -626,7 +626,7 @@ Browse all of this as a searchable site at **[aetherengine.superuser404.de](http - **[docs/api.md](docs/api.md)**: every public surface a host consumes, and the contracts that require it to act (how a load ends, `.ended`, the live retune, the audio tap, what must be set before `load`, and which FFmpeg your link hands the engine). A test fails the build when a host-facing public symbol is named nowhere in the docs. - **[docs/architecture.md](docs/architecture.md)**: the three playback pipelines, the source-file map, dependencies, the SwiftUI `Menu` pattern. - **[docs/formats.md](docs/formats.md)**: codec / container coverage, HDR routing, audio bridging, subtitles, frame extraction, disc playback, live ingest, and known limitations. -- **[docs/cli.md](docs/cli.md)**: the `aetherctl` repro CLI (twenty-one subcommands). +- **[docs/cli.md](docs/cli.md)**: the `aetherctl` repro CLI (twenty-four subcommands). - **[CHANGELOG.md](CHANGELOG.md)**: per-release index. ## Stability and versioning diff --git a/Sources/AetherEngine/AetherEngine+CompositorPreview.swift b/Sources/AetherEngine/AetherEngine+CompositorPreview.swift new file mode 100644 index 000000000..f223676b9 --- /dev/null +++ b/Sources/AetherEngine/AetherEngine+CompositorPreview.swift @@ -0,0 +1,42 @@ +import Foundation + +/// Where a compositor-drawn scrub preview landed (`AetherEngine.showScrubPreview(atSeconds:in:)`). +public struct ScrubPreviewPlacement: Sendable, Equatable { + /// Playlist/output seconds of the keyframe on screen. + public let actualSeconds: Double + /// The span the same picture stands for: the plan segment that keyframe opens. + public let validRange: Range +} + +extension AetherEngine { + + /// True when this session's scrub previews come through `showScrubPreview(atSeconds:in:)` and an + /// `AetherScrubPreviewView` rather than a `CGImage`: a native VOD session of Dolby Vision without + /// a displayable base layer (Profile 5, AV1 Profile 10.0), whose colour only the system + /// compositor applies. Independent of the `DolbyVisionStillReshaping` trait. Inside the resident + /// window the picture comes out of the segment cache, like `scrubThumbnail`; outside it, out of + /// the source's own keyframe, read by a side demuxer. + public var scrubPreviewDrawsInView: Bool { + guard !isLive, let session = nativeVideoSession else { return false } + return session.previewNeedsCompositor + } + + /// Shows the keyframe that opens the plan segment holding `seconds` (playlist/output axis, as + /// `scrubThumbnail` takes it) in `view`: from the segment cache when that segment is resident, + /// otherwise read off the source the way the I-frame rendition reads its entries. Nil when + /// `scrubPreviewDrawsInView` is false, the segment is not resident and the origin admits no side + /// read (sequential, serial or held connection, disc, not reopenable by URL), the read failed, a + /// newer request superseded this one, or the session changed meanwhile; the view keeps its + /// previous picture in every case. + public func showScrubPreview(atSeconds seconds: Double, in view: AetherScrubPreviewView) async -> ScrubPreviewPlacement? { + guard seconds.isFinite, scrubPreviewDrawsInView, let session = nativeVideoSession else { return nil } + let gen = loadGeneration + let file = await BlockingWork.detached(priority: .userInitiated) { [session] in + session.compositorPreviewFile(atSeconds: seconds) + }.value + guard let file, gen == loadGeneration, !Task.isCancelled else { return nil } + guard await view.present(file.url), gen == loadGeneration else { return nil } + return ScrubPreviewPlacement(actualSeconds: file.startSeconds, + validRange: file.startSeconds..<(file.startSeconds + file.durationSeconds)) + } +} diff --git a/Sources/AetherEngine/AetherEngine+HeldPicture.swift b/Sources/AetherEngine/AetherEngine+HeldPicture.swift index e023ca5ea..354f4f701 100644 --- a/Sources/AetherEngine/AetherEngine+HeldPicture.swift +++ b/Sources/AetherEngine/AetherEngine+HeldPicture.swift @@ -55,16 +55,16 @@ extension AetherEngine { let generation = loadGeneration let started = ContinuousClock.now + if let host = nativeSource, + Self.heldPictureNeedsReshaping(videoFormat: videoFormat, dolbyVisionProfile: sourceDVProfile), + !OptionalComponent.dolbyVisionStillReshaping.isBuiltIn { + await holdThroughCompositor(host: host, view: view, generation: generation, started: started) + return + } + let held: (frame: CVPixelBuffer, isHDR: Bool, route: String)? if let host = nativeSource { if Self.heldPictureNeedsReshaping(videoFormat: videoFormat, dolbyVisionProfile: sourceDVProfile) { - // Without the reshaping there is no correct picture to hold, so nothing is decoded. - guard OptionalComponent.dolbyVisionStillReshaping.isBuiltIn else { - heldPictureLastSkip = "Dolby Vision still reshaping not built in" - EngineLog.emit("[AetherEngine] held picture: skipped (Dolby Vision still reshaping not built in, " - + "DolbyVisionStillReshaping trait off)", category: .engine) - return - } held = await reshapedDolbyVisionStill(itemSeconds: host.avPlayer.currentTime().seconds) .map { ($0, false, "Dolby Vision reshaped") } } else { @@ -92,8 +92,58 @@ extension AetherEngine { EngineLog.emit("[AetherEngine] held picture: frame could not be wrapped for display", category: .engine) return } + armHeldPicture(on: view, route: held.route, nativeSource: nativeSource) + EngineLog.emit( + "[AetherEngine] held picture: up after \(Self.milliseconds(ContinuousClock.now - started))ms " + + "(\(held.route), \(CVPixelBufferGetWidth(held.frame))x\(CVPixelBufferGetHeight(held.frame)), " + + "hdr=\(held.isHDR))", + category: .engine) + } + + /// Dolby Vision without a displayable base layer and without the still reshaping: the frame under + /// the playhead, played out of the resident segment by a player of the surface's own, so the + /// system compositor colours it as it colours playback. Waits at most + /// `heldPictureDolbyVisionBudgetSeconds`, the outgoing picture paused meanwhile. + private func holdThroughCompositor(host: NativeAVPlayerHost, view: any HeldStillSurface, + generation: UInt64, started: ContinuousClock.Instant) async { + let itemSeconds = host.avPlayer.currentTime().seconds + guard let session = nativeVideoSession else { return } + let built = await BlockingWork.detached(priority: .userInitiated) { [session] in + session.compositorHoldFile(atSeconds: itemSeconds) + }.value + guard let hold = built else { + heldPictureLastSkip = "no frame from the outgoing picture" + EngineLog.emit("[AetherEngine] held picture: no resident segment for the compositor hold", category: .engine) + return + } + let spent = Double((ContinuousClock.now - started).components.seconds) + + Double((ContinuousClock.now - started).components.attoseconds) / 1e18 + let budget = max(0.2, Self.heldPictureDolbyVisionBudgetSeconds - spent) + let shown = await view.showCompositorStill(file: hold.url, pastFirstFrameBy: hold.offsetSeconds, + gravity: videoGravity, budgetSeconds: budget) + guard shown else { + heldPictureLastSkip = "no frame from the outgoing picture" + EngineLog.emit("[AetherEngine] held picture: the compositor hold did not become displayable", category: .engine) + return + } + guard loadGeneration == generation, nativeHost === host, (boundStillView ?? boundView) === view else { + view.clearStill() + heldPictureLastSkip = "superseded while the frame was read" + EngineLog.emit("[AetherEngine] held picture: superseded while the frame was read", category: .engine) + return + } + armHeldPicture(on: view, route: "Dolby Vision compositor", nativeSource: host) + EngineLog.emit( + "[AetherEngine] held picture: up after \(Self.milliseconds(ContinuousClock.now - started))ms " + + "(Dolby Vision compositor, item \(String(format: "%.3f", itemSeconds))s)", + category: .engine) + } + + /// Records the hold and arms its release: the next item's first frame (native), the next host's + /// (software), or `heldPictureTimeoutSeconds`. + private func armHeldPicture(on view: any HeldStillSurface, route: String, nativeSource: NativeAVPlayerHost?) { heldPictureView = view - heldPictureLastRoute = held.route + heldPictureLastRoute = route heldPictureShownAt = .now heldPictureToken &+= 1 let token = heldPictureToken @@ -113,11 +163,6 @@ extension AetherEngine { guard let self, self.heldPictureToken == token, self.heldPictureView != nil else { return } self.releaseHeldPicture(reason: "timeout") } - EngineLog.emit( - "[AetherEngine] held picture: up after \(Self.milliseconds(ContinuousClock.now - started))ms " - + "(\(held.route), \(CVPixelBufferGetWidth(held.frame))x\(CVPixelBufferGetHeight(held.frame)), " - + "hdr=\(held.isHDR))", - category: .engine) } /// visionOS has no external playback, and no `isExternalPlaybackActive` to ask. diff --git a/Sources/AetherEngine/AetherEngine+ScrubThumbnail.swift b/Sources/AetherEngine/AetherEngine+ScrubThumbnail.swift index 5005c2bf8..7ea6f3f22 100644 --- a/Sources/AetherEngine/AetherEngine+ScrubThumbnail.swift +++ b/Sources/AetherEngine/AetherEngine+ScrubThumbnail.swift @@ -150,7 +150,7 @@ extension AetherEngine { /// FrameExtractor (#106). It reports capability, not per-frame availability: transient /// nils from `scrubThumbnail` while a segment is still being produced are expected. False as well /// on a Dolby Vision Profile 5 / 10.0 session in a build without the `DolbyVisionStillReshaping` - /// trait, where no frame can become a correct still. + /// trait, where no frame can become a correct still; `scrubPreviewDrawsInView` covers that source. public var supportsCacheBackedStills: Bool { if let session = nativeVideoSession { return session.yieldsStills } guard let softwareHost else { return false } diff --git a/Sources/AetherEngine/Video/CompositorPreviewBuilder.swift b/Sources/AetherEngine/Video/CompositorPreviewBuilder.swift new file mode 100644 index 000000000..140767c3d --- /dev/null +++ b/Sources/AetherEngine/Video/CompositorPreviewBuilder.swift @@ -0,0 +1,126 @@ +// Sources/AetherEngine/Video/CompositorPreviewBuilder.swift +import Foundation +import AetherLibavcodec +import AetherLibavformat +import AetherLibavutil + +/// Turns a resident segment into a standalone video-only fMP4 that a second AVPlayer can show through +/// the system compositor. That is the only place a Dolby Vision Profile 5 / 10.0 picture gets its +/// colour on Apple platforms: no still API applies the RPU, and an `AVPlayerItemVideoOutput` on the +/// item switches that processing off for the layer too. +/// +/// The packets come out of the segment the session already muxed, so they carry every conversion the +/// playing picture got, and the init is rebuilt from the segment's own demuxed parameters: the +/// sample entry, `dvcC` and `colr` therefore say what the playing item says. The init carries no +/// audio track, so the item has nothing to wait for. +enum CompositorPreviewBuilder { + enum Content { + /// The keyframe that opens the segment, alone, at file time 0 (scrub preview). + case firstKeyframe + /// Every video sample of the segment, so a seek can land on any frame of it (held picture). + /// Where its first frame sits on the player's clock is the player's business: AVFoundation + /// honours the init's edit list and starts at 0, ffmpeg reports the composition delay. + case wholeSegment + } + + struct Output { + let file: Data + /// How far the segment's first frame sits behind its first decode time (the B-frame delay). + /// A frame's offset past the standalone file's first frame is its item time minus the segment's + /// start plus this delay: measured pixel-exact against the paused outgoing picture on three + /// Profile 5 clips (24 fps with 0.083 s, 30 fps with 0.067 s, and one cut 1.2 s off a keyframe), + /// where leaving the delay out lands two frames early. 0 for `.firstKeyframe`. + let compositionDelaySeconds: Double + } + + static func build(initData: Data, segmentURL: URL, durationSeconds: Double, + codecTagOverride: String?, colorOverride: MP4SegmentMuxer.ColorOverride?, + stagingDir: URL, content: Content = .firstKeyframe) -> Output? { + guard let segment = try? Data(contentsOf: segmentURL, options: .alwaysMapped) else { return nil } + let demuxer = Demuxer() + defer { demuxer.close() } + do { + try demuxer.open(reader: DataIOReader(parts: [initData, segment]), formatHint: "mp4", + profile: .stillExtraction) + } catch { + return nil + } + let videoIdx = demuxer.videoStreamIndex + guard videoIdx >= 0, let stream = demuxer.stream(at: videoIdx), + let codecpar = stream.pointee.codecpar else { return nil } + demuxer.discardAllStreamsExcept([videoIdx]) + let timeBase = stream.pointee.time_base + guard timeBase.num > 0, timeBase.den > 0 else { return nil } + let video = MP4SegmentMuxer.VideoConfig( + codecpar: UnsafePointer(codecpar), timeBase: timeBase, + codecTagOverride: codecTagOverride, doviConfig: .keep, colorOverride: colorOverride) + + switch content { + case .firstKeyframe: + // The segment opens on its keyframe; a bound keeps a malformed one from walking the file. + var payload: Data? + for _ in 0..<64 { + guard let packet = try? demuxer.readPacket() else { break } + defer { av_packet_unref(packet); av_packet_free_safe(packet) } + guard packet.pointee.stream_index == videoIdx, + packet.pointee.flags & AV_PKT_FLAG_KEY != 0, let data = packet.pointee.data, + packet.pointee.size > 0 else { continue } + payload = Data(bytes: data, count: Int(packet.pointee.size)) + break + } + guard let payload, + let output = IFrameFragmentBuilder(video: video, stagingDir: stagingDir) + .build(payload: payload, index: 0, startSeconds: 0, + durationSeconds: max(durationSeconds, 0.04)) else { return nil } + return Output(file: output.initSegment + output.fragment, compositionDelaySeconds: 0) + + case .wholeSegment: + return remuxWholeSegment(demuxer: demuxer, videoIdx: videoIdx, video: video, stagingDir: stagingDir) + } + } + + /// Every video packet, shifted so the first decode time is 0, into one fragment. + private static func remuxWholeSegment(demuxer: Demuxer, videoIdx: Int32, + video: MP4SegmentMuxer.VideoConfig, stagingDir: URL) -> Output? { + var initBytes = Data() + guard let muxer = try? MP4SegmentMuxer( + initialSegmentIndex: 0, sessionDir: stagingDir, video: video, audio: nil, + onInitCaptured: { initBytes = $0 }) else { return nil } + var completed: URL? + defer { + _ = muxer.finalize() + if let completed { try? FileManager.default.removeItem(at: completed) } + } + var firstDts: Int64? + var firstFramePts: Int64? + var written = 0 + // A segment is a few seconds of one GOP or a handful; the bound only stops a malformed file. + for _ in 0..<20_000 { + guard let packet = try? demuxer.readPacket() else { break } + defer { av_packet_unref(packet); av_packet_free_safe(packet) } + guard packet.pointee.stream_index == videoIdx, packet.pointee.size > 0 else { continue } + let dts = packet.pointee.dts != Int64.min ? packet.pointee.dts : packet.pointee.pts + let pts = packet.pointee.pts != Int64.min ? packet.pointee.pts : dts + guard dts != Int64.min else { continue } + if firstDts == nil { + guard packet.pointee.flags & AV_PKT_FLAG_KEY != 0 else { return nil } + firstDts = dts + } + let base = firstDts ?? dts + firstFramePts = min(firstFramePts ?? pts, pts) + packet.pointee.dts = dts - base + packet.pointee.pts = pts - base + packet.pointee.stream_index = muxer.videoOutputStreamIndex + av_packet_rescale_ts(packet, video.timeBase, muxer.muxerVideoTimeBase) + guard muxer.writePacket(packet).rc >= 0 else { return nil } + written += 1 + } + guard written > 0, let firstDts, let firstFramePts, + case .completed(let path, _) = muxer.cutFragmentForNextSegment(1) else { return nil } + completed = path + guard let fragment = try? Data(contentsOf: path), !fragment.isEmpty, !initBytes.isEmpty else { return nil } + let ticks = Double(video.timeBase.num) / Double(video.timeBase.den) + return Output(file: initBytes + fragment, + compositionDelaySeconds: max(0, Double(firstFramePts - firstDts) * ticks)) + } +} diff --git a/Sources/AetherEngine/Video/HLSVideoEngine.swift b/Sources/AetherEngine/Video/HLSVideoEngine.swift index 09ba6a639..4001df176 100644 --- a/Sources/AetherEngine/Video/HLSVideoEngine.swift +++ b/Sources/AetherEngine/Video/HLSVideoEngine.swift @@ -257,6 +257,26 @@ public final class HLSVideoEngine: @unchecked Sendable { /// False for a Dolby Vision Profile 5 / 10.0 source in a build without the /// `DolbyVisionStillReshaping` trait: none of its frames can become a correct still. private(set) var yieldsStills = true + /// Dolby Vision without a displayable base layer (Profile 5, AV1 Profile 10.0): only the system + /// compositor shows it in colour, so scrub previews go through `compositorPreviewFile`. + private(set) var previewNeedsCompositor = false + private let compositorPreviewLock = NSLock() + private var compositorPreviewFiles: [(identity: String, file: CompositorPreviewFile)] = [] + private var compositorPreviewSerial = 0 + private var compositorHoldURL: URL? + /// Bumped by every `compositorPreviewFile` call, so a source read queued behind newer requests + /// is skipped instead of making the latest one wait for it. + private var compositorPreviewRequest = 0 + /// Keyframes for previews outside the resident window: the served I-frame rendition when there + /// is one, else a side reader of this session's own, built on the first request that needs it. + /// Guarded by `restartLock`. + private var previewKeyframes: IFrameRendition? + private var previewKeyframeBuilder: PreviewKeyframeBuilder? + /// Whether this session may read preview keyframes off the source: the I-frame rendition's own + /// rule (keyframe-aligned plan, a reopenable URL, no serial or held origin, no disc), decided at + /// start. A Profile 5 / 10.0 session never serves that rendition, so this is asked separately. + private var previewReadsSource = false + nonisolated static let compositorPreviewFileLimit = 12 /// AE#682: the server withheld the JPEG I-frame playlist because AVPlayer never asked for a media /// segment, which is how a refused HDR variant looks next to an SDR I-frame rendition. var iFrameRenditionWithheld: Bool { server?.iFrameRenditionWithheld ?? false } @@ -2241,10 +2261,11 @@ public final class HLSVideoEngine: @unchecked Sendable { let stillsAvailable = FrameDecodeContext.yieldsStill( dolbyVisionNoBaseLayer: FrameDecodeContext.isDVNoBaseLayer(codecpar: codecpar)) yieldsStills = stillsAvailable + previewNeedsCompositor = FrameDecodeContext.isDVNoBaseLayer(codecpar: codecpar) if !stillsAvailable { EngineLog.emit("[HLSVideoEngine] Dolby Vision Profile 5 / 10.0 stills are not built in " - + "(DolbyVisionStillReshaping trait off): no scrub thumbnails, I-frame stills " - + "or held pictures for this source", category: .session) + + "(DolbyVisionStillReshaping trait off): no scrub thumbnails or I-frame stills; " + + "scrub previews and the held picture go through the system compositor", category: .session) } // AE#682: decided before the routing below, because an I-frame rendition is one of the // reasons to serve a master at all. @@ -2259,6 +2280,17 @@ public final class HLSVideoEngine: @unchecked Sendable { isDiscSource: dem.isDiscSource, secondReaderAvailable: sourceReopenableByURL || customIFrameReader != nil, stillsUnavailable: !stillsAvailable)) + // The compositor preview reads by URL only: a custom reader's clone belongs to the rendition. + previewReadsSource = previewNeedsCompositor && IFrameRenditionEligibility.candidate(.init( + requested: true, + isLive: isLiveSession, + planBoundariesClaimRandomAccess: planBoundariesClaimRandomAccess, + sequentialOrigin: sequentialOrigin, + heldSourceConnection: openProfile.avioHeldConnection, + originIsSerial: sourceReopenableByURL + && OriginRequestBudget.shared.requiresSerialRequests(sourceURL), + isDiscSource: dem.isDiscSource, + secondReaderAvailable: sourceReopenableByURL)) == .served let hasNativeSubs = enableNativeSubtitleTrackForSession && !nativeSubtitleCueStoresForSession.isEmpty // AE#187: tvOS HW HEVC needs the codec advertised in a master's CODECS attribute; a bare media // playlist (H.264 is fine media-direct) fails the item with tracks count=0 / -12848. Scope to @@ -2735,6 +2767,199 @@ public final class HLSVideoEngine: @unchecked Sendable { initData: initData, segmentURL: seg.fileURL) } + struct CompositorPreviewFile: Sendable { + let url: URL + /// Output-axis seconds of the keyframe the file shows. + let startSeconds: Double + let durationSeconds: Double + } + + /// A single-keyframe fMP4 for the resident segment holding `seconds` (playlist/output axis), built + /// once per segment identity and kept for the last `compositorPreviewFileLimit` segments. Nil + /// for sources a still covers and live sessions. A segment that is not resident falls through to + /// `sourceKeyframePreviewFile`. + func compositorPreviewFile(atSeconds seconds: Double) -> CompositorPreviewFile? { + guard previewNeedsCompositor, !isLiveSession else { return nil } + compositorPreviewLock.lock() + compositorPreviewRequest &+= 1 + let request = compositorPreviewRequest + compositorPreviewLock.unlock() + guard let source = scrubThumbnailSource(atSeconds: seconds) else { + return sourceKeyframePreviewFile(atSeconds: seconds, request: request) + } + compositorPreviewLock.lock() + if let hit = compositorPreviewFiles.firstIndex(where: { $0.identity == source.identity }) { + let entry = compositorPreviewFiles.remove(at: hit) + compositorPreviewFiles.append(entry) + compositorPreviewLock.unlock() + return entry.file + } + compositorPreviewSerial += 1 + let serial = compositorPreviewSerial + compositorPreviewLock.unlock() + + restartLock.lock() + let directory = cache?.sessionDir.appendingPathComponent("preview", isDirectory: true) + let tag = savedVideoConfig?.codecTagOverride + let color = savedVideoConfig?.colorOverride + restartLock.unlock() + guard let directory else { return nil } + let staging = directory.appendingPathComponent("staging", isDirectory: true) + guard (try? FileManager.default.createDirectory(at: staging, withIntermediateDirectories: true)) != nil, + let built = CompositorPreviewBuilder.build( + initData: source.initData, segmentURL: source.segmentURL, + durationSeconds: source.durationSeconds, codecTagOverride: tag, colorOverride: color, + stagingDir: staging) else { return nil } + let url = directory.appendingPathComponent("seg\(source.segmentIndex)-\(serial).mp4") + guard (try? built.file.write(to: url)) != nil else { return nil } + let file = CompositorPreviewFile(url: url, startSeconds: source.startSeconds, + durationSeconds: source.durationSeconds) + compositorPreviewLock.lock() + compositorPreviewFiles.append((source.identity, file)) + var evicted: [URL] = [] + while compositorPreviewFiles.count > Self.compositorPreviewFileLimit { + evicted.append(compositorPreviewFiles.removeFirst().file.url) + } + compositorPreviewLock.unlock() + for old in evicted { try? FileManager.default.removeItem(at: old) } + return file + } + + /// The plan entry holding `seconds`: the last one starting at or before it. Seconds before the + /// first entry belong to it, a gap past the last one's end to nothing. + nonisolated static func planIndex(containing seconds: Double, in plan: [Segment]) -> Int? { + guard seconds.isFinite, let first = plan.first else { return nil } + if seconds < first.startSeconds { return 0 } + var low = 0, high = plan.count - 1 + while low < high { + let mid = (low + high + 1) / 2 + if plan[mid].startSeconds <= seconds { low = mid } else { high = mid - 1 } + } + let entry = plan[low] + return seconds < entry.startSeconds + max(entry.durationSeconds, 0) || low < plan.count - 1 ? low : nil + } + + /// The preview for a position outside the resident window: the keyframe that opens its plan + /// segment, read off the source by a side demuxer as the I-frame rendition reads it, wrapped + /// with the session's own sample entry, `dvcC` and `colr`. Nothing is decoded; the compositor + /// colours it like the resident preview. Skipped when a newer request arrived meanwhile. + private func sourceKeyframePreviewFile(atSeconds seconds: Double, request: Int) -> CompositorPreviewFile? { + restartLock.lock() + let plan = segmentPlan + let directory = cache?.sessionDir.appendingPathComponent("preview", isDirectory: true) + let allowed = previewReadsSource + restartLock.unlock() + guard allowed, let directory, let index = Self.planIndex(containing: seconds, in: plan) else { return nil } + let identity = "plan:\(index)" + compositorPreviewLock.lock() + if let hit = compositorPreviewFiles.firstIndex(where: { $0.identity == identity }) { + let entry = compositorPreviewFiles.remove(at: hit) + compositorPreviewFiles.append(entry) + compositorPreviewLock.unlock() + return entry.file + } + compositorPreviewLock.unlock() + + guard let (keyframes, builder) = previewKeyframeSource(plan: plan, directory: directory) else { return nil } + let isStale: () -> Bool = { [weak self] in + guard let self else { return true } + self.compositorPreviewLock.lock() + defer { self.compositorPreviewLock.unlock() } + return self.compositorPreviewRequest != request + } + guard let payload = keyframes.keyframePayload(at: index, isStale: isStale) else { return nil } + let segment = plan[index] + guard let built = builder.build(payload: payload, durationSeconds: segment.durationSeconds) else { + return nil + } + + compositorPreviewLock.lock() + compositorPreviewSerial += 1 + let serial = compositorPreviewSerial + compositorPreviewLock.unlock() + let url = directory.appendingPathComponent("plan\(index)-\(serial).mp4") + guard (try? (built.initSegment + built.fragment).write(to: url)) != nil else { return nil } + let file = CompositorPreviewFile(url: url, startSeconds: segment.startSeconds, + durationSeconds: segment.durationSeconds) + compositorPreviewLock.lock() + compositorPreviewFiles.append((identity, file)) + var evicted: [URL] = [] + while compositorPreviewFiles.count > Self.compositorPreviewFileLimit { + evicted.append(compositorPreviewFiles.removeFirst().file.url) + } + compositorPreviewLock.unlock() + for old in evicted { try? FileManager.default.removeItem(at: old) } + return file + } + + /// The side reader and builder behind `sourceKeyframePreviewFile`, made on the first request + /// that needs them and torn down by `stop()`, which drains a read in flight before it frees the + /// codec parameters the builder reads. + private func previewKeyframeSource(plan: [Segment], directory: URL) + -> (IFrameRendition, PreviewKeyframeBuilder)? { + restartLock.lock() + defer { restartLock.unlock() } + if let previewKeyframes, let previewKeyframeBuilder { return (previewKeyframes, previewKeyframeBuilder) } + guard previewReadsSource, let cfg = savedVideoConfig, !plan.isEmpty else { return nil } + let staging = directory.appendingPathComponent("source-staging", isDirectory: true) + guard (try? FileManager.default.createDirectory(at: staging, withIntermediateDirectories: true)) != nil + else { return nil } + let builder = PreviewKeyframeBuilder(IFrameFragmentBuilder( + video: MP4SegmentMuxer.VideoConfig( + codecpar: cfg.codecpar, timeBase: cfg.timeBase, + codecTagOverride: cfg.codecTagOverride, doviConfig: cfg.doviConfig, + colorOverride: cfg.colorOverride, extradataOverride: cfg.extradataOverride, + convertsAnnexBSamples: cfg.convertsAnnexBSamples, + nalFramingLatch: cfg.nalFramingLatch), + stagingDir: staging)) + let entries = plan.map { + IFrameRendition.Entry(startPts: $0.startPts, startSeconds: $0.startSeconds, + durationSeconds: $0.durationSeconds) + } + let keyframes = makeKeyframeRendition( + config: cfg, entries: entries, + cache: IFramePayloadCache(directory: directory.appendingPathComponent("payloads", isDirectory: true)), + format: .keyframes, buildFragment: { _, _, _ in nil }, custom: nil) + previewKeyframes = keyframes + previewKeyframeBuilder = builder + EngineLog.emit("[HLSVideoEngine] compositor preview: source keyframe reader ready " + + "(segments=\(plan.count))", category: .session) + return (keyframes, builder) + } + + /// The whole resident segment holding `seconds` as a video-only fMP4, and how far past the + /// segment's first frame `seconds` lies: what the held picture of an audio switch shows for a + /// source only the compositor colours. One file at a time; building the next removes the previous. + func compositorHoldFile(atSeconds seconds: Double) -> (url: URL, offsetSeconds: Double)? { + guard previewNeedsCompositor, !isLiveSession, seconds.isFinite, + let source = scrubThumbnailSource(atSeconds: seconds) else { return nil } + restartLock.lock() + let directory = cache?.sessionDir.appendingPathComponent("preview", isDirectory: true) + let tag = savedVideoConfig?.codecTagOverride + let color = savedVideoConfig?.colorOverride + restartLock.unlock() + guard let directory else { return nil } + let staging = directory.appendingPathComponent("hold-staging", isDirectory: true) + guard (try? FileManager.default.createDirectory(at: staging, withIntermediateDirectories: true)) != nil, + let built = CompositorPreviewBuilder.build( + initData: source.initData, segmentURL: source.segmentURL, + durationSeconds: source.durationSeconds, codecTagOverride: tag, colorOverride: color, + stagingDir: staging, content: .wholeSegment) else { return nil } + compositorPreviewLock.lock() + compositorPreviewSerial += 1 + let serial = compositorPreviewSerial + let previous = compositorHoldURL + compositorPreviewLock.unlock() + let url = directory.appendingPathComponent("hold\(source.segmentIndex)-\(serial).mp4") + guard (try? built.file.write(to: url)) != nil else { return nil } + compositorPreviewLock.lock() + compositorHoldURL = url + compositorPreviewLock.unlock() + if let previous { try? FileManager.default.removeItem(at: previous) } + let offset = min(max(0, seconds - source.startSeconds + built.compositionDelaySeconds), source.durationSeconds) + return (url, offset) + } + public func stop() { cancelResidentRangesPublish() // Sodalite#32: drop the tap routes first so a pump still draining its last packets no-ops @@ -2756,6 +2981,10 @@ public final class HLSVideoEngine: @unchecked Sendable { server = nil let c = cache cache = nil + compositorPreviewLock.lock() + compositorPreviewFiles.removeAll() + compositorHoldURL = nil + compositorPreviewLock.unlock() let ab = audioBridge audioBridge = nil let d = demuxer @@ -2769,6 +2998,10 @@ public final class HLSVideoEngine: @unchecked Sendable { provider = nil let iFrames = iFrameRendition iFrameRendition = nil + let previewFrames = previewKeyframes + previewKeyframes = nil + let previewBuilder = previewKeyframeBuilder + previewKeyframeBuilder = nil closeUnusedCustomIFrameReader() savedVideoConfig = nil savedAudioConfig = nil @@ -2807,6 +3040,8 @@ public final class HLSVideoEngine: @unchecked Sendable { // (pump accesses them during unwind). ownedParams released last (pump read them). BlockingWork.detached { iFrames?.shutdown() + previewFrames?.shutdown() + previewBuilder?.close() _ = p?.waitForFinish(timeout: 3.0) s?.stop() c?.close() @@ -2867,13 +3102,34 @@ public final class HLSVideoEngine: @unchecked Sendable { startSeconds: entry.startSeconds, durationSeconds: entry.durationSeconds) } } + // The side reader owns the clone from here; `stop()` must not close it a second time. + let custom = customIFrameReader + customIFrameReader = nil + let entries = plan.map { + IFrameRendition.Entry(startPts: $0.startPts, startSeconds: $0.startSeconds, + durationSeconds: $0.durationSeconds) + } + return makeKeyframeRendition( + config: cfg, entries: entries, + cache: IFramePayloadCache(directory: directory.appendingPathComponent("payloads", isDirectory: true)), + format: format, buildFragment: buildFragment, custom: custom) + } + + /// The side-read half every keyframe consumer shares: a demuxer of its own on the source, the + /// P7 conversion the playing samples get, and the gates that keep it off a busy or serial origin. + private func makeKeyframeRendition( + config cfg: HLSSegmentProducer.StreamConfig, + entries: [IFrameRendition.Entry], + cache: IFramePayloadCache, + format: IFrameRenditionFormat, + buildFragment: @escaping (Data, Int, IFrameRendition.Entry) -> IFrameFragmentBuilder.Output?, + custom: (reader: IOReader, formatHint: String?)? + ) -> IFrameRendition { let framing = cfg.nalFramingOverride ?? A53SEIParser.nalFraming( codec: cfg.codecpar.pointee.codec_id == AV_CODEC_ID_HEVC ? .hevc : .h264, extradata: cfg.codecpar.pointee.extradata.map { UnsafePointer($0) }, size: Int(cfg.codecpar.pointee.extradata_size)) - let url = sourceURL, headers = sourceHTTPHeaders, headerStore = requestHeaders, custom = customIFrameReader - // The side reader owns the clone from here; `stop()` must not close it a second time. - customIFrameReader = nil + let url = sourceURL, headers = sourceHTTPHeaders, headerStore = requestHeaders let reader = IFrameSideReader( open: { dem in if let custom { @@ -2888,13 +3144,9 @@ public final class HLSVideoEngine: @unchecked Sendable { cleanup: { custom?.reader.close() }, convertP7ToProfile81: cfg.convertP7ToProfile81, nalFraming: framing) - let entries = plan.map { - IFrameRendition.Entry(startPts: $0.startPts, startSeconds: $0.startSeconds, - durationSeconds: $0.durationSeconds) - } return IFrameRendition( entries: entries, - cache: IFramePayloadCache(directory: directory.appendingPathComponent("payloads", isDirectory: true)), + cache: cache, format: format, readPayload: { reader.payload(startPts: $0.startPts) }, buildFragment: buildFragment, @@ -4569,3 +4821,32 @@ extension HLSVideoEngine { } } } + +/// The fragment builder of the source-keyframe preview, behind a lock: its muxer stages each build +/// under a name taken from the index and every preview builds index 0, and it reads the session's +/// codec parameters, which `stop()` frees once `close()` has returned. +final class PreviewKeyframeBuilder: @unchecked Sendable { + private let builder: IFrameFragmentBuilder + private let lock = NSLock() + private var closed = false + + init(_ builder: IFrameFragmentBuilder) { + self.builder = builder + } + + /// One keyframe at file time 0, as a standalone fMP4's init and fragment. + func build(payload: Data, durationSeconds: Double) -> IFrameFragmentBuilder.Output? { + lock.lock() + defer { lock.unlock() } + guard !closed else { return nil } + return builder.build(payload: payload, index: 0, startSeconds: 0, + durationSeconds: max(durationSeconds, 0.04)) + } + + /// Waits for a build in flight; every later call answers nil. + func close() { + lock.lock() + closed = true + lock.unlock() + } +} diff --git a/Sources/AetherEngine/Video/IFrameRendition.swift b/Sources/AetherEngine/Video/IFrameRendition.swift index 9992f6290..b65f629c7 100644 --- a/Sources/AetherEngine/Video/IFrameRendition.swift +++ b/Sources/AetherEngine/Video/IFrameRendition.swift @@ -88,30 +88,47 @@ final class IFrameRendition: IFrameSegmentSource, @unchecked Sendable { } } + /// The keyframe that opens segment `index`, from the cache or the side reader, and never a + /// neighbour's in its place: a scrub preview names the span it shows, so a stand-in would put + /// the wrong picture over the right time. `isStale` is asked before the source is touched, so a + /// queue of superseded requests costs no reads. + func keyframePayload(at index: Int, isStale: () -> Bool = { false }) -> Data? { + queue.sync { + guard !shutDown, entries.indices.contains(index) else { return nil } + if let cached = cache.payload(for: index) { return cached } + guard !isStale() else { return nil } + return readLocked(at: index) + } + } + + /// Reads and caches one keyframe. nil when the origin has no room for it, teardown interrupted + /// it, or the read failed. + private func readLocked(at index: Int) -> Data? { + // An origin that went serial or started metering mid-session has one slot and a few + // tokens, and the pump needs them; AVKit's overview burst would take them all. + guard sourceReadsAllowed() else { return nil } + waitForLink { self.shutDown } + guard !shutDown else { return nil } + let started = DispatchTime.now() + let read = readPayload(entries[index]) + // A read that teardown aborted is not a miss to paper over. + guard !shutDown, let read else { return nil } + let ms = Double(DispatchTime.now().uptimeNanoseconds - started.uptimeNanoseconds) / 1_000_000 + EngineLog.emit("[IFrameRendition] read idx=\(index) bytes=\(read.count) " + + "ms=\(String(format: "%.0f", ms))", category: .session, level: .verbose) + cache.store(read, for: index) + return read + } + private func buildLocked(at index: Int) -> Data? { guard entries.indices.contains(index) else { return nil } let entry = entries[index] var payload = cache.payload(for: index) if payload == nil { - // An origin that went serial or started metering mid-session has one slot and a few - // tokens, and the pump needs them; AVKit's overview burst would take them all. - var read: Data? - if sourceReadsAllowed() { - waitForLink { self.shutDown } - guard !shutDown else { return nil } - let started = DispatchTime.now() - read = readPayload(entry) - // A read that teardown aborted is not a miss to paper over. - guard !shutDown else { return nil } - if let read { - let ms = Double(DispatchTime.now().uptimeNanoseconds - started.uptimeNanoseconds) / 1_000_000 - EngineLog.emit("[IFrameRendition] read idx=\(index) bytes=\(read.count) " - + "ms=\(String(format: "%.0f", ms))", category: .session, level: .verbose) - } - } - if let read { - cache.store(read, for: index) + if let read = readLocked(at: index) { payload = read + } else if shutDown { + return nil } else if let neighbour = cache.nearestIndex(to: index), let substitute = cache.payload(for: neighbour) { // AVKit keeps the last good picture for a keyframe it cannot fetch, and rate trick diff --git a/Sources/AetherEngine/View/AetherPlayerView.swift b/Sources/AetherEngine/View/AetherPlayerView.swift index df629d1ad..6b857a388 100644 --- a/Sources/AetherEngine/View/AetherPlayerView.swift +++ b/Sources/AetherEngine/View/AetherPlayerView.swift @@ -32,7 +32,7 @@ public final class AetherPlayerView: PlatformBaseView, HeldStillSurface { private var hostedLayer: CALayer? /// AE#711 follow-up: the picture held over an in-place item swap, above `hostedLayer`. private let presenter = HeldStillPresenter() - private var stillLayer: CALayer? { presenter.layer } + private var stillLayer: CALayer? { presenter.layer ?? presenter.playerLayer } /// Engine-internal. The engine this view was last bound to, so a dismantling surface can unbind /// from it synchronously and a second engine binding the view can take it over (AE#536). @@ -147,10 +147,17 @@ public final class AetherPlayerView: PlatformBaseView, HeldStillSurface { return presenter.show(frame, gravity: gravity, isHDR: isHDR, on: host, bounds: bounds) } + func showCompositorStill(file: URL, pastFirstFrameBy offset: Double, gravity: AVLayerVideoGravity, + budgetSeconds: Double) async -> Bool { + let host: CALayer? = layer + return await presenter.show(file: file, pastFirstFrameBy: offset, gravity: gravity, + on: host, bounds: bounds, budgetSeconds: budgetSeconds) + } + /// Engine-internal. Removes the held picture, if any. Idempotent. func clearStill() { presenter.clear() } - var isHoldingStill: Bool { presenter.layer != nil } + var isHoldingStill: Bool { presenter.isShowing } } // MARK: - Platform base view alias diff --git a/Sources/AetherEngine/View/AetherScrubPreviewView.swift b/Sources/AetherEngine/View/AetherScrubPreviewView.swift new file mode 100644 index 000000000..ef2bbbc21 --- /dev/null +++ b/Sources/AetherEngine/View/AetherScrubPreviewView.swift @@ -0,0 +1,135 @@ +import Foundation +import QuartzCore +import AVFoundation + +#if canImport(UIKit) +import UIKit +#elseif canImport(AppKit) +import AppKit +#endif + +/// The scrub preview surface for sources whose pictures only the system compositor can colour: +/// Dolby Vision Profile 5 and AV1 Profile 10.0 (`AetherEngine.scrubPreviewDrawsInView`). Their +/// base layer is IPT-PQ, and no still API applies the RPU, so the engine cannot hand out a correct +/// `CGImage` for them. This view plays a one-keyframe item through its own muted `AVPlayer`, and +/// the compositor draws it the way it draws the playing picture. +/// +/// Fill it with `AetherEngine.showScrubPreview(atSeconds:in:)` and call `clear()` when the preview +/// goes away, which releases its decoder. Two players alternate, so the outgoing picture stays up +/// until the next one is ready instead of blanking between positions. +@MainActor +public final class AetherScrubPreviewView: PlatformBaseView { + + /// How the preview fills the view. + public var videoGravity: AVLayerVideoGravity = .resizeAspect { + didSet { for layer in playerLayers { layer.videoGravity = videoGravity } } + } + + private let players = [AVPlayer(), AVPlayer()] + private lazy var playerLayers: [AVPlayerLayer] = players.map { AVPlayerLayer(player: $0) } + private var front = 0 + private var shownURL: URL? + private var request = 0 + + #if canImport(UIKit) + public override init(frame: CGRect) { + super.init(frame: frame) + setUp() + } + + public required init?(coder: NSCoder) { + super.init(coder: coder) + setUp() + } + + public override func layoutSubviews() { + super.layoutSubviews() + layoutPlayerLayers() + } + #elseif canImport(AppKit) + public override init(frame frameRect: NSRect) { + super.init(frame: frameRect) + wantsLayer = true + setUp() + } + + public required init?(coder: NSCoder) { + super.init(coder: coder) + wantsLayer = true + setUp() + } + + public override func layout() { + super.layout() + layoutPlayerLayers() + } + #endif + + private func setUp() { + #if canImport(UIKit) + backgroundColor = .clear + isUserInteractionEnabled = false + #endif + let host: CALayer? = layer + for (player, layer) in zip(players, playerLayers) { + player.isMuted = true + player.actionAtItemEnd = .pause + #if os(iOS) || os(tvOS) + player.allowsExternalPlayback = false + player.preventsDisplaySleepDuringVideoPlayback = false + #endif + layer.videoGravity = videoGravity + layer.opacity = 0 + host?.addSublayer(layer) + } + } + + private func layoutPlayerLayers() { + CATransaction.begin() + CATransaction.setDisableActions(true) + for layer in playerLayers { layer.frame = bounds } + CATransaction.commit() + } + + /// Drops both items, which releases their decoders, and clears the picture. + public func clear() { + request &+= 1 + shownURL = nil + CATransaction.begin() + CATransaction.setDisableActions(true) + for (player, layer) in zip(players, playerLayers) { + player.replaceCurrentItem(with: nil) + layer.opacity = 0 + } + CATransaction.commit() + } + + /// Shows the one-keyframe file at `url`. False when a newer request superseded this one, or the + /// item did not become displayable within two seconds; the previous picture stays up then. + func present(_ url: URL) async -> Bool { + if url == shownURL { return true } + request &+= 1 + let mine = request + let back = 1 - front + let item = AVPlayerItem(url: url) + players[back].replaceCurrentItem(with: item) + let deadline = ContinuousClock.now + .seconds(2) + while !(item.status == .readyToPlay && playerLayers[back].isReadyForDisplay) { + guard item.status != .failed, ContinuousClock.now < deadline else { + if mine == request { players[back].replaceCurrentItem(with: nil) } + return false + } + try? await Task.sleep(for: .milliseconds(4)) + guard mine == request else { return false } + } + CATransaction.begin() + CATransaction.setDisableActions(true) + playerLayers[back].opacity = 1 + playerLayers[front].opacity = 0 + CATransaction.commit() + players[front].replaceCurrentItem(with: nil) + front = back + shownURL = url + return true + } +} diff --git a/Sources/AetherEngine/View/AetherStillView.swift b/Sources/AetherEngine/View/AetherStillView.swift index e325b6fb4..46da1fe51 100644 --- a/Sources/AetherEngine/View/AetherStillView.swift +++ b/Sources/AetherEngine/View/AetherStillView.swift @@ -62,7 +62,14 @@ public final class AetherStillView: PlatformBaseView, HeldStillSurface { return presenter.show(frame, gravity: videoGravity, isHDR: isHDR, on: host, bounds: bounds) } + func showCompositorStill(file: URL, pastFirstFrameBy offset: Double, gravity: AVLayerVideoGravity, + budgetSeconds: Double) async -> Bool { + let host: CALayer? = layer + return await presenter.show(file: file, pastFirstFrameBy: offset, gravity: videoGravity, + on: host, bounds: bounds, budgetSeconds: budgetSeconds) + } + func clearStill() { presenter.clear() } - var isHoldingStill: Bool { presenter.layer != nil } + var isHoldingStill: Bool { presenter.isShowing } } diff --git a/Sources/AetherEngine/View/HeldStill.swift b/Sources/AetherEngine/View/HeldStill.swift index 5948d85ac..597b3df00 100644 --- a/Sources/AetherEngine/View/HeldStill.swift +++ b/Sources/AetherEngine/View/HeldStill.swift @@ -9,6 +9,10 @@ import AVFoundation protocol HeldStillSurface: AnyObject { /// `gravity` is the engine's; a surface that takes its fill from the host ignores it. func showStill(_ frame: CVPixelBuffer, gravity: AVLayerVideoGravity, isHDR: Bool) -> Bool + /// Shows the frame `offset` seconds past the first frame of a video-only file through a player of + /// its own, so the system compositor colours it (Dolby Vision without a displayable base layer). + func showCompositorStill(file: URL, pastFirstFrameBy offset: Double, gravity: AVLayerVideoGravity, + budgetSeconds: Double) async -> Bool func clearStill() var isHoldingStill: Bool { get } } @@ -19,6 +23,67 @@ protocol HeldStillSurface: AnyObject { @MainActor final class HeldStillPresenter { private(set) var layer: AVSampleBufferDisplayLayer? + private(set) var playerLayer: AVPlayerLayer? + private var player: AVPlayer? + + var isShowing: Bool { layer != nil || playerLayer != nil } + + /// The compositor form of `show`: a muted player seeks frame-exactly to `offset` past the video + /// track's first frame, on the player's own clock (AVFoundation applies the init's edit list, so the + /// file's raw timestamps are not that clock), behind an invisible layer that turns visible once the + /// frame is displayable. False (and nothing shown) + /// when the item fails, a `clear()` intervenes, or `budgetSeconds` runs out first. + func show(file: URL, pastFirstFrameBy offset: Double, gravity: AVLayerVideoGravity, + on host: CALayer?, bounds: CGRect, budgetSeconds: Double) async -> Bool { + clear() + guard let host else { return false } + let player = AVPlayer() + player.isMuted = true + player.actionAtItemEnd = .pause + #if os(iOS) || os(tvOS) + player.allowsExternalPlayback = false + player.preventsDisplaySleepDuringVideoPlayback = false + #endif + let item = AVPlayerItem(url: file) + player.replaceCurrentItem(with: item) + let still = AVPlayerLayer(player: player) + still.videoGravity = gravity + still.opacity = 0 + CATransaction.begin() + CATransaction.setDisableActions(true) + host.addSublayer(still) + still.frame = bounds + CATransaction.commit() + self.player = player + playerLayer = still + + let deadline = ContinuousClock.now + .seconds(budgetSeconds) + func waitFor(_ condition: () -> Bool) async -> Bool { + while !condition() { + guard item.status != .failed, ContinuousClock.now < deadline, playerLayer === still else { return false } + try? await Task.sleep(for: .milliseconds(4)) + } + return playerLayer === still + } + guard await waitFor({ item.status == .readyToPlay }) else { abandon(still); return false } + let track = try? await item.asset.loadTracks(withMediaType: .video).first + let firstFrame = (try? await track?.load(.timeRange))?.start ?? .zero + guard playerLayer === still else { return false } + let target = CMTimeAdd(firstFrame.isNumeric ? firstFrame : .zero, + CMTime(seconds: max(0, offset), preferredTimescale: 90_000)) + guard await player.seek(to: target, toleranceBefore: .zero, toleranceAfter: .zero), + await waitFor({ still.isReadyForDisplay }) else { abandon(still); return false } + CATransaction.begin() + CATransaction.setDisableActions(true) + still.opacity = 1 + CATransaction.commit() + return true + } + + private func abandon(_ still: AVPlayerLayer) { + guard playerLayer === still else { return } + clear() + } /// False when the frame cannot be wrapped for display; nothing is shown then. func show(_ frame: CVPixelBuffer, gravity: AVLayerVideoGravity, isHDR: Bool, @@ -45,6 +110,15 @@ final class HeldStillPresenter { } func clear() { + if let playing = playerLayer { + CATransaction.begin() + CATransaction.setDisableActions(true) + playing.removeFromSuperlayer() + CATransaction.commit() + player?.replaceCurrentItem(with: nil) + player = nil + playerLayer = nil + } guard let still = layer else { return } CATransaction.begin() CATransaction.setDisableActions(true) @@ -56,6 +130,7 @@ final class HeldStillPresenter { func layout(_ bounds: CGRect) { layer?.frame = bounds + playerLayer?.frame = bounds } private static func sample(_ frame: CVPixelBuffer) -> CMSampleBuffer? { diff --git a/Sources/aetherctl/ScrubPreviewCmd.swift b/Sources/aetherctl/ScrubPreviewCmd.swift new file mode 100644 index 000000000..0a2bf91d6 --- /dev/null +++ b/Sources/aetherctl/ScrubPreviewCmd.swift @@ -0,0 +1,81 @@ +import AVFoundation +import Foundation +import AetherEngine + +// MARK: - scrubpreview + +/// Compositor scrub previews (Dolby Vision Profile 5 / 10.0) at a list of positions: for each one, +/// whether a picture came, how long it took, and the span it stands for. Positions inside the +/// resident window come from the segment cache, the rest from the source's own keyframe, so a run +/// against a throttled source exercises both. `--keep ` copies the session's preview files +/// out before teardown, which is how the two kinds of file are compared byte for byte. +func runScrubPreview(url: URL, positions: [Double], warmSeconds: Double, keepDir: String?) -> Int32 { + print(EngineLog.redacted("aetherctl scrubpreview: \(url.absoluteString)")) + let box = UncheckedBox(nil) + Task { @MainActor in + box.value = await scrubPreviewRun(url: url, positions: positions, warmSeconds: warmSeconds, + keepDir: keepDir) + CFRunLoopStop(CFRunLoopGetMain()) + } + CFRunLoopRun() + return box.value ?? 1 +} + +@MainActor +private func scrubPreviewRun(url: URL, positions: [Double], warmSeconds: Double, keepDir: String?) async -> Int32 { + let engine: AetherEngine + do { + engine = try AetherEngine() + try await engine.load(url: url, options: LoadOptions()) + } catch { + print("load failed: \(error.localizedDescription)") + return 1 + } + engine.play() + try? await Task.sleep(for: .seconds(warmSeconds)) + print("backend=\(engine.playbackBackend.rawValue) drawsInView=\(engine.scrubPreviewDrawsInView) " + + String(format: "t=%.1fs duration=%.1fs", engine.currentTime, engine.duration)) + guard engine.scrubPreviewDrawsInView else { + print("FAIL: this source does not draw its previews in a view") + engine.stop() + return 1 + } + let view = AetherScrubPreviewView(frame: CGRect(x: 0, y: 0, width: 320, height: 180)) + var shown = 0 + for target in positions { + let resident = engine.residentRanges.contains { $0.contains(target) } + let started = ContinuousClock.now + let placement = await engine.showScrubPreview(atSeconds: target, in: view) + let ms = Double((ContinuousClock.now - started).components.attoseconds) / 1e15 + + Double((ContinuousClock.now - started).components.seconds) * 1000 + if let placement { shown += 1 } + print(String(format: " t=%7.2f resident=%@ -> %@ in %.0f ms", target, resident ? "yes" : "no ", + placement.map { String(format: "shown %.2f..%.2f", $0.validRange.lowerBound, + $0.validRange.upperBound) } ?? "nil", ms)) + } + if let keepDir { + let root = FileManager.default.temporaryDirectory.appendingPathComponent("aether-segments") + let sessions = (try? FileManager.default.contentsOfDirectory( + at: root, includingPropertiesForKeys: [.contentModificationDateKey])) ?? [] + let newest = sessions.max { + let a = (try? $0.resourceValues(forKeys: [.contentModificationDateKey]).contentModificationDate) ?? .distantPast + let b = (try? $1.resourceValues(forKeys: [.contentModificationDateKey]).contentModificationDate) ?? .distantPast + return a < b + } + if let preview = newest?.appendingPathComponent("preview"), + let files = try? FileManager.default.contentsOfDirectory(atPath: preview.path) { + let out = URL(fileURLWithPath: keepDir, isDirectory: true) + try? FileManager.default.createDirectory(at: out, withIntermediateDirectories: true) + for name in files where name.hasSuffix(".mp4") { + try? FileManager.default.removeItem(at: out.appendingPathComponent(name)) + try? FileManager.default.copyItem(at: preview.appendingPathComponent(name), + to: out.appendingPathComponent(name)) + } + print("kept \(files.filter { $0.hasSuffix(".mp4") }.count) preview files in \(keepDir)") + } + } + view.clear() + engine.stop() + print("\(shown) of \(positions.count) positions showed a picture") + return shown == positions.count ? 0 : 2 +} diff --git a/Sources/aetherctl/main.swift b/Sources/aetherctl/main.swift index fa3a637eb..808f8e080 100644 --- a/Sources/aetherctl/main.swift +++ b/Sources/aetherctl/main.swift @@ -1,6 +1,6 @@ // aetherctl: standalone reproduction harness for AetherEngine on macOS. // -// Twenty-one subcommands, most operating on a media source URL (file:// +// Twenty-four subcommands, most operating on a media source URL (file:// // or http(s)://), a few on built-in synthetic fixtures. The full list // with flags and examples lives in docs/cli.md; the three original modes: // @@ -514,6 +514,26 @@ if first == "feltest" { #endif } +// Compositor scrub previews (Dolby Vision Profile 5 / 10.0), resident and from the source. +if first == "scrubpreview" { + var rest = Array(args.dropFirst(2)) + let at = takeStringFlag("--at", from: &rest) ?? "2,5,30,50,65" + let warm = takeDoubleFlag("--warm", from: &rest) ?? 4 + let keep = takeStringFlag("--keep", from: &rest) + if let kbps = takeIntFlag("--throttle-kbps", from: &rest) { + AetherEngine.setSourceThrottleKbpsForTesting(kbps) + print("[aetherctl] source throttle: \(kbps) kbit/s") + } + guard let urlArg = rest.first(where: { !$0.hasPrefix("--") }) else { + print("Usage: aetherctl scrubpreview [--at s1,s2,...] [--warm S] [--keep DIR] [--throttle-kbps N] ") + exit(64) + } + rest.removeAll { $0 == urlArg } + rejectStrayFlags(rest, subcommand: "scrubpreview") + let positions = at.split(separator: ",").compactMap { Double($0.trimmingCharacters(in: .whitespaces)) } + exit(runScrubPreview(url: parseSourceURL(urlArg), positions: positions, warmSeconds: warm, keepDir: keep)) +} + // Dolby Vision display-metadata survey: picks the pin for the on-device consumption test. if first == "dvsurvey" { var rest = Array(args.dropFirst(2)) diff --git a/Tests/AetherEngineTests/CompositorPreviewBuilderTests.swift b/Tests/AetherEngineTests/CompositorPreviewBuilderTests.swift new file mode 100644 index 000000000..0fe83bf99 --- /dev/null +++ b/Tests/AetherEngineTests/CompositorPreviewBuilderTests.swift @@ -0,0 +1,109 @@ +// Tests/AetherEngineTests/CompositorPreviewBuilderTests.swift +import Foundation +import Testing +import AetherLibavcodec +import AetherLibavformat +import AetherLibavutil +@testable import AetherEngine + +@MainActor +@Suite("Compositor preview builder", .serialized) +struct CompositorPreviewBuilderTests { + private func tempDir() throws -> URL { + let dir = FileManager.default.temporaryDirectory + .appendingPathComponent("compositor-preview-\(UUID().uuidString)", isDirectory: true) + try FileManager.default.createDirectory(at: dir, withIntermediateDirectories: true) + return dir + } + + /// A resident segment the way the session cuts one: a `dvh1` init with the Profile 5 record and + /// a fragment that starts mid-title. + private func residentSegment(in dir: URL) throws -> (initData: Data, segmentURL: URL, keyframe: Data) { + let dem = Demuxer() + defer { dem.close() } + try dem.open(reader: DataIOReader(data: SupersededProbeGeometryTests.dolbyVisionProfile5OverHDR10()), + formatHint: "mp4") + let index = dem.videoStreamIndex + let stream = try #require(dem.stream(at: index)) + var keyframe: Data? + for _ in 0..<64 { + guard let pkt = try dem.readPacket() else { break } + var owned: UnsafeMutablePointer? = pkt + defer { av_packet_free(&owned) } + if pkt.pointee.stream_index == index, pkt.pointee.flags & AV_PKT_FLAG_KEY != 0, let bytes = pkt.pointee.data { + keyframe = Data(bytes: bytes, count: Int(pkt.pointee.size)); break + } + } + let payload = try #require(keyframe) + let staging = dir.appendingPathComponent("segment-staging", isDirectory: true) + try FileManager.default.createDirectory(at: staging, withIntermediateDirectories: true) + let builder = IFrameFragmentBuilder( + video: MP4SegmentMuxer.VideoConfig(codecpar: UnsafePointer(stream.pointee.codecpar), + timeBase: stream.pointee.time_base, codecTagOverride: "dvh1"), + stagingDir: staging) + let out = try #require(builder.build(payload: payload, index: 3, startSeconds: 12.0, durationSeconds: 4.0)) + let url = dir.appendingPathComponent("seg3.m4s") + try out.fragment.write(to: url) + return (out.initSegment, url, payload) + } + + @Test("the first keyframe of a resident segment becomes a standalone one-sample dvh1 file at zero") + func standaloneOneKeyframeFile() throws { + let dir = try tempDir() + defer { try? FileManager.default.removeItem(at: dir) } + let segment = try residentSegment(in: dir) + let staging = dir.appendingPathComponent("preview-staging", isDirectory: true) + try FileManager.default.createDirectory(at: staging, withIntermediateDirectories: true) + + let out = try #require(CompositorPreviewBuilder.build( + initData: segment.initData, segmentURL: segment.segmentURL, durationSeconds: 4.0, + codecTagOverride: "dvh1", colorOverride: nil, stagingDir: staging)) + #expect(IFrameTestBoxes.topLevelTypes(out.file).prefix(2) == ["ftyp", "moov"]) + let fragmentStart = try #require(Self.offsetOfFirstMoof(out.file)) + let fragment = out.file.subdata(in: fragmentStart..? = first + defer { av_packet_free(&owned) } + #expect(first.pointee.flags & AV_PKT_FLAG_KEY != 0) + #expect(Data(bytes: first.pointee.data, count: Int(first.pointee.size)) == segment.keyframe) + #expect(try dem.readPacket() == nil) + } + + @Test("a missing segment file builds nothing") + func missingSegment() throws { + let dir = try tempDir() + defer { try? FileManager.default.removeItem(at: dir) } + let segment = try residentSegment(in: dir) + #expect(CompositorPreviewBuilder.build( + initData: segment.initData, segmentURL: dir.appendingPathComponent("gone.m4s"), durationSeconds: 4, + codecTagOverride: "dvh1", colorOverride: nil, stagingDir: dir) == nil) + } + + private static func fourCC(_ s: String) -> UInt32 { + s.utf8.reversed().reduce(0) { ($0 << 8) | UInt32($1) } + } + + private static func offsetOfFirstMoof(_ d: Data) -> Int? { + var off = 0 + while off + 8 <= d.count { + let size = Int(IFrameTestBoxes.u32(d, off)) + guard size >= 8 else { return nil } + if String(bytes: d[(d.startIndex + off + 4)..<(d.startIndex + off + 8)], encoding: .isoLatin1) == "moof" { return off } + off += size + } + return nil + } +} diff --git a/Tests/AetherEngineTests/CompositorPreviewPlanIndexTests.swift b/Tests/AetherEngineTests/CompositorPreviewPlanIndexTests.swift new file mode 100644 index 000000000..95ea59e87 --- /dev/null +++ b/Tests/AetherEngineTests/CompositorPreviewPlanIndexTests.swift @@ -0,0 +1,27 @@ +// Tests/AetherEngineTests/CompositorPreviewPlanIndexTests.swift +import Foundation +import Testing +@testable import AetherEngine + +/// The plan entry a source-keyframe scrub preview reads for a position outside the resident window. +struct CompositorPreviewPlanIndexTests { + private let plan: [HLSVideoEngine.Segment] = [ + .init(startPts: 0, endPts: 4000, startSeconds: 0, durationSeconds: 4), + .init(startPts: 4000, endPts: 10_000, startSeconds: 4, durationSeconds: 6), + .init(startPts: 10_000, endPts: 12_000, startSeconds: 10, durationSeconds: 2), + ] + + @Test("a position belongs to the entry it falls in, a boundary to the entry it opens", + arguments: [(0.0, 0), (3.99, 0), (4.0, 1), (9.5, 1), (10.0, 2), (11.9, 2)]) + func containing(seconds: Double, expected: Int) { + #expect(HLSVideoEngine.planIndex(containing: seconds, in: plan) == expected) + } + + @Test("before the first entry is the first entry, past the last one's end is nothing") + func edges() { + #expect(HLSVideoEngine.planIndex(containing: -2, in: plan) == 0) + #expect(HLSVideoEngine.planIndex(containing: 12.5, in: plan) == nil) + #expect(HLSVideoEngine.planIndex(containing: .nan, in: plan) == nil) + #expect(HLSVideoEngine.planIndex(containing: 1, in: []) == nil) + } +} diff --git a/Tests/AetherEngineTests/IFrameRenditionTests.swift b/Tests/AetherEngineTests/IFrameRenditionTests.swift index 0fc6896e8..021760c86 100644 --- a/Tests/AetherEngineTests/IFrameRenditionTests.swift +++ b/Tests/AetherEngineTests/IFrameRenditionTests.swift @@ -191,6 +191,42 @@ struct IFrameRenditionTests { #expect(rec.reads == [1000]) } + @Test("a preview keyframe is the entry's own payload, read once, and never a neighbour's") + func previewKeyframeNoStandIn() { + let rec = Recorder() + rec.failing = [3000] + let (r, dir) = make(rec); defer { r.shutdown(); try? FileManager.default.removeItem(at: dir) } + #expect(r.keyframePayload(at: 2) == Data("key2000".utf8)) + #expect(r.keyframePayload(at: 2) == Data("key2000".utf8)) + #expect(r.keyframePayload(at: 3) == nil) + #expect(rec.reads == [2000, 3000]) + } + + @Test("a stale preview request reads nothing but still answers from the cache") + func previewKeyframeStale() { + let rec = Recorder() + let (r, dir) = make(rec); defer { r.shutdown(); try? FileManager.default.removeItem(at: dir) } + #expect(r.keyframePayload(at: 1) == Data("key1000".utf8)) + #expect(r.keyframePayload(at: 4, isStale: { true }) == nil) + #expect(r.keyframePayload(at: 1, isStale: { true }) == Data("key1000".utf8)) + #expect(rec.reads == [1000]) + } + + @Test("preview keyframes respect held source reads, the link wait and shutdown") + func previewKeyframeGates() { + let rec = Recorder() + let (r, dir) = make(rec); defer { try? FileManager.default.removeItem(at: dir) } + rec.readsAllowed = false + #expect(r.keyframePayload(at: 0) == nil) + #expect(rec.reads.isEmpty) + rec.readsAllowed = true + #expect(r.keyframePayload(at: 0) != nil) + #expect(rec.waits == 1) + r.shutdown() + #expect(r.keyframePayload(at: 0) == nil) + #expect(r.keyframePayload(at: 7) == nil) + } + @Test("the failure line is written once per window and counts what it swallowed") func logThrottle() { var throttle = IFrameLogThrottle() diff --git a/Tests/AetherEngineTests/ItemSwapStillTests.swift b/Tests/AetherEngineTests/ItemSwapStillTests.swift index 86c5f4758..e981cac65 100644 --- a/Tests/AetherEngineTests/ItemSwapStillTests.swift +++ b/Tests/AetherEngineTests/ItemSwapStillTests.swift @@ -392,13 +392,14 @@ struct ItemSwapStillTests { #expect(view.isHoldingStill) } #else - /// Without the reshaping a Profile 5 picture cannot be made from the segment, so where the hold - /// would need it (the output presents Dolby Vision) it is skipped with its reason rather than - /// showing the planes read as YCbCr. Where AVPlayer presents the source as HDR10 (a Mac without a - /// Dolby Vision display), the output's own picture is correct and is held as before. - @Test("without the reshaping trait a Profile 5 title never holds a reshaped still", + /// Without the reshaping a Profile 5 picture cannot be made into a still, so where the hold needs + /// one (the output presents Dolby Vision) the engine plays the frame out of the resident segment + /// through a player of the surface's own, and the system compositor colours it. Where AVPlayer + /// presents the source as HDR10 (a Mac without a Dolby Vision display), the output's own picture + /// is correct and is held as before. + @Test("without the reshaping trait a Profile 5 title holds its frame through the compositor", .enabled(if: FileManager.default.fileExists(atPath: ItemSwapStillTests.profile5URL.path))) - func profile5WithoutReshapingHoldsNothingReshaped() async throws { + func profile5WithoutReshapingHoldsThroughCompositor() async throws { let engine = try AetherEngine() defer { engine.stop() } let view = AetherPlayerView(frame: CGRect(x: 0, y: 0, width: 320, height: 180)) @@ -406,14 +407,18 @@ struct ItemSwapStillTests { _ = try await engine.load(url: Self.profile5URL) let host = try #require(engine.nativeHost) try await waitFor { host.isVideoReadyForDisplay } + try await Task.sleep(for: .seconds(1)) engine.pause() - // No scrub still either, and the capability says so up front so a host hides the preview. + // No scrub still either; the compositor preview stands in, and the capabilities say so up front. #expect(!engine.supportsCacheBackedStills) + #expect(engine.scrubPreviewDrawsInView) #expect(await engine.scrubThumbnail(atSeconds: 0.5, maxWidth: 160) == nil) + let started = ContinuousClock.now await engine.holdPictureAcrossItemSwap() + #expect(ContinuousClock.now - started < .seconds(AetherEngine.heldPictureDolbyVisionBudgetSeconds + 0.5)) if AetherEngine.heldPictureNeedsReshaping(videoFormat: engine.videoFormat, dolbyVisionProfile: engine.sourceDVProfile) { - #expect(engine.heldPictureLastSkip == "Dolby Vision still reshaping not built in") - #expect(!view.isHoldingStill) + #expect(engine.heldPictureLastRoute == "Dolby Vision compositor") + #expect(view.isHoldingStill) } else { #expect(engine.heldPictureLastRoute == "native output") } diff --git a/docs/api.md b/docs/api.md index 6c9cd68e0..9e8260f99 100644 --- a/docs/api.md +++ b/docs/api.md @@ -1139,7 +1139,7 @@ say so on the tracker rather than working around it. | Symbol | Notes | | --- | --- | | `videoGravity` | Fill mode of whichever layer is mounted (`AVPlayerLayer` or the software display layer). Settable; this is the aspect-fit / aspect-fill control. | -| `AetherStillView`, `bindStillView(_:)`, `unbindStillView(_:)` | For a host that renders the native path through AVKit rather than `AetherPlayerView`. An audio-track switch swaps the item in place, and `replaceCurrentItem` blanks the player layer until the new item's first frame; the engine holds the outgoing picture over that gap on the bound still view (or, without one, on the bound `AetherPlayerView`). Transparent and input-free; place it above the video and below the host's chrome, e.g. at the top of `AVPlayerViewController.contentOverlayView`, and set its `videoGravity` to the host player's fill. The software path holds its renderer's last picture across the rebuild the same way. Dolby Vision without a displayable base layer (Profile 5, AV1 Profile 10) is held as a 960-wide still decoded from the resident segment through the scrub stills' RPU reshaping (only with the `DolbyVisionStillReshaping` trait; without it the hold is skipped and the outgoing picture is not held), which the switch waits for (up to 3 s, the outgoing picture paused meanwhile); other profiles hold their base layer. Nothing is held in PiP or on external playback. | +| `AetherStillView`, `bindStillView(_:)`, `unbindStillView(_:)` | For a host that renders the native path through AVKit rather than `AetherPlayerView`. An audio-track switch swaps the item in place, and `replaceCurrentItem` blanks the player layer until the new item's first frame; the engine holds the outgoing picture over that gap on the bound still view (or, without one, on the bound `AetherPlayerView`). Transparent and input-free; place it above the video and below the host's chrome, e.g. at the top of `AVPlayerViewController.contentOverlayView`, and set its `videoGravity` to the host player's fill. The software path holds its renderer's last picture across the rebuild the same way. Dolby Vision without a displayable base layer (Profile 5, AV1 Profile 10) cannot be held as the output vends it. Without the `DolbyVisionStillReshaping` trait, the engine remuxes the resident segment's video into a standalone file and a player of the surface's own seeks it frame-exactly to the outgoing position, so the system compositor colours the held frame as it colours playback (pixel-identical to the paused outgoing picture on the clips measured; up in 165 to 220 ms on an M1). With the trait, the frame is a 960-wide still decoded through the scrub stills' RPU reshaping. Either way the switch waits for it (up to 3 s, the outgoing picture paused meanwhile); other profiles hold their base layer. Nothing is held in PiP or on external playback. | | `nativePlayerLayer` | The engine's own `AVPlayerLayer`, for a host building `AVPictureInPictureController` around a layer rather than around `currentAVPlayer`. | | `$softwarePiPSource` | `SoftwarePiPSource` for sample-buffer PiP on the software path: the display layer plus transport answers on the enqueued frames' axis. iOS only in practice; tvOS AVKit does not evaluate sample-buffer content sources (FB9751461). | | `$softwareDisplaySize` | The rectangle the software path's picture presents at: coded size under the decoder's pixel aspect. Mirrored, not latched, so a mid-stream resolution change re-shapes it. nil on every other path. | @@ -1169,8 +1169,9 @@ say so on the tracker rather than working around it. | --- | --- | | `scrubThumbnail(atSeconds:maxWidth:)` | Cache-backed still for the active session, live or VOD. Decodes bytes the session already holds, so it opens no second connection and works on single-connection sources (debrid / torrent links, IPTV accounts capped at one request) where a second demuxer is refused. A native session decodes from its segment cache, a software session from its packet cache (VOD, AE#605) or its DVR packet ring (live, #544). A VOD target outside what is retained returns nil rather than the nearest retained frame. | | `vodScrubThumbnail(atSeconds:maxWidth:)`, `liveScrubThumbnail(atSessionSeconds:maxWidth:)` | The two arms, for callers that know which axis they hold. Both arms also serve software sessions: the live arm out of the DVR packet ring (#544), the VOD arm out of the packet cache its seeks land in (AE#605), keyframe to target, on the same axis `seek(to:)` takes. | -| `supportsCacheBackedStills` | True while the session can serve `scrubThumbnail` without a second connection: any native session, a software live session, and a software VOD session reading a remote source. Gate the scrub-preview affordance on it: it reports capability, not per-frame availability, so a transient nil from `scrubThumbnail` while a segment is still being produced, or for a position the cache does not hold, is expected and means "time only, no image". False on a software session playing a local file, which keeps no cache because re-reading the file is free; `makeFrameExtractor()` serves that case. Before AE#605 this was false on every software session, live included. | +| `supportsCacheBackedStills` | True while the session can serve `scrubThumbnail` without a second connection: any native session, a software live session, and a software VOD session reading a remote source. Gate the scrub-preview affordance on it: it reports capability, not per-frame availability, so a transient nil from `scrubThumbnail` while a segment is still being produced, or for a position the cache does not hold, is expected and means "time only, no image". False on a software session playing a local file, which keeps no cache because re-reading the file is free; `makeFrameExtractor()` serves that case. Before AE#605 this was false on every software session, live included. A Dolby Vision Profile 5 / 10.0 source without the `DolbyVisionStillReshaping` trait reads false here; `scrubPreviewDrawsInView` is its preview. | | `scrubPreviewFrame(atSeconds:refined:maxWidth:isCancelled:)` | The timestamped form of `scrubThumbnail`, for a preview that labels the frame it shows rather than the position asked for. Reads only resident native segments or software packets, never seeks playback and never opens a second source, and returns a `ScrubFrame`: the image, the measured `actualSeconds` on the same axis `seek(to:)` takes, `refined`, and the resident `validRange` the frame's run covers. `refined: false` stops at the first decoded frame (cached per segment at the same width); `true` decodes toward the target, and `refined` in the result means that decode reached it, not that the two times are equal. Missing, cancelled, replaced or stale work returns nil. Decode is bounded at 750 ms and 900 packets per request. | +| `scrubPreviewDrawsInView`, `showScrubPreview(atSeconds:in:)`, `AetherScrubPreviewView`, `ScrubPreviewPlacement` | The scrub preview for Dolby Vision without a displayable base layer (Profile 5, AV1 Profile 10.0) on a native VOD session. Its base layer is IPT-PQ and no still API applies the RPU, so no `CGImage` of it can be correct; only the system compositor colours it. `scrubPreviewDrawsInView` is true for such a session, independent of the `DolbyVisionStillReshaping` trait. `showScrubPreview(atSeconds:in:)` takes the same playlist/output seconds as `scrubThumbnail`, cuts the keyframe that opens the plan segment holding them into a standalone one-sample fMP4 and shows it in the `AetherScrubPreviewView` through that view's own muted `AVPlayer`. Inside the resident window the keyframe comes out of the segment cache with no second connection. Outside it, the engine reads that keyframe off the source with a side demuxer of its own, the way the I-frame rendition reads its entries (seek to the segment's start, one packet, nothing decoded), and wraps it with the session's own sample entry, `dvcC` and `colr`: byte-identical to the source's keyframe, and the same picture the cache would give. That read is withheld where the I-frame rendition would be (a plan whose boundaries are not keyframes, a sequential, serial or held-connection origin, a disc, a source not reopenable by URL), waits out a seek or producer restart, and skips a request a newer one has superseded; those cases answer nil. The result is a `ScrubPreviewPlacement`: `actualSeconds` of the keyframe on screen and the `validRange` of its segment. Requests on one view are latest-wins, and the outgoing picture stays up until the next one is displayable (two players alternate), so a superseded or failed request leaves the previous picture. Call `clear()` when the preview is hidden, which releases the view's decoders. Measured: 26 to 45 ms from request to picture on an M1 for a new segment, 1 ms for the one already shown, and a 4K preview next to 4K Profile 5 playback on an Apple TV 4K (3rd generation) without disturbing it. Never attach an `AVPlayerItemVideoOutput` to a player meant to show such a picture: AVFoundation switches the Dolby Vision processing off for the layer as soon as one is attached. | | `scrubPreviewSourceGeneration` | A fresh `UUID` on every teardown, distinct across engine instances, so a host can fence a preview that resolved after its source went away. | | `clearResidentPreviewFrames()` | Cancels in-flight resident previews and drops their cached approximations; playback media is untouched. | diff --git a/docs/cli.md b/docs/cli.md index 3bb45fbcc..ca674b08c 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -15,6 +15,7 @@ swift run aetherctl dvsurvey # survey per-frame Dolby Vision L1 / L2 swift run aetherctl pktdump # dump raw demuxer packet timing (dts/pts/keyframe) per open profile swift run aetherctl dualsubs ... # dual subtitle-track render probe (--primary / --secondary stream index) swift run aetherctl extract # FrameExtractor still-image extraction + leak testing +swift run aetherctl scrubpreview # Dolby Vision Profile 5 / 10.0 compositor scrub previews, resident and from the source swift run aetherctl audio [--seconds N] # audio-only pipeline smoke test (default 10 s) swift run aetherctl audiotap # decode the PCM audio tap headless, write mono 48 kHz WAV (#95) swift run aetherctl bgaudio # SW-path background-audio keepalive probe (iOS background behavior) @@ -29,7 +30,7 @@ swift run aetherctl smbtest # play a file off an SMB2/3 share via t swift run aetherctl # alias for serve (backwards compat) ``` -Twenty-three subcommands plus the bare-URL `serve` alias. +Twenty-four subcommands plus the bare-URL `serve` alias. ## probe @@ -520,6 +521,14 @@ swift run aetherctl pktdump --at 660 --count 300 --profile stillExtraction The profile differential is the diagnostic: a `video_delay=0` plus NOPTS or non-monotonic dts under one profile while the other is clean means that profile's open path cannot reconstruct decode-order dts for B-frame content (the #93 post-recovery judder root cause). Backed by the public `PacketTimingProbe.run(url:seekSeconds:packetCount:profileName:)`. +## scrubpreview + +Loads a Dolby Vision Profile 5 / 10.0 source on the native path, plays it for `--warm S` seconds (default 4), then asks `showScrubPreview(atSeconds:in:)` for each position in `--at s1,s2,...` and prints whether a picture came, how long it took, the span it stands for, and whether the position was resident. A position inside the resident window is cut from the segment cache, one outside it is read off the source as the keyframe that opens its plan segment, so `--throttle-kbps N` (which keeps the producer close to the playhead) is how one run covers both. `--keep DIR` copies the session's preview files out before teardown: `seg-*.mp4` came from the cache, `plan-*.mp4` from the source, and comparing them with `ffprobe -show_data_hash` against the source's own keyframes is the byte-for-byte check. Exit codes: `0` every position showed a picture, `1` load failed or the source does not draw its previews in a view, `2` at least one position showed none. + +```bash +swift run aetherctl scrubpreview --throttle-kbps 20000 --at 2,30,50,65 --keep /tmp/previews +``` + ## extract Opens a `FrameExtractor` against the source and pulls a still frame. Thumbnail mode (default) snaps to the nearest keyframe and downscales to `--width` (default 320); `--snapshot` decodes frame-accurately at full resolution. `--at ` sets the seek position (default 60.0). The first frame is written to `aetherctl-extract-.png` in aetherctl's private per-user temporary directory (the path is printed). `--loops N` repeats the extraction across eight cycling positions, which pairs with `leaks --atExit` to validate the decode-context teardown is clean: