From 2986230aca8d4a6152b0695987723ee62a4d041f Mon Sep 17 00:00:00 2001 From: yipengfei Date: Tue, 13 Oct 2026 11:04:55 +0800 Subject: [PATCH 001/132] feat(disc): play a DVD-Video VIDEO_TS folder over HTTP A DiscDirectoryReader whose listing holds VIDEO_TS/ files is now read as DVD-Video instead of Blu-ray. The title-set code of wrapDVD moves into dvdTitleSet, which takes a listing and a way to read one whole file, so an image and a folder share the VMGI filtering, durations, chapters, languages and the selected title's VTS IFO. wrapDVDFolder concatenates the selected title set's VOBs in part order with MultiFileConcatIOReader; that is the same run of bytes as their sectors on a disc, so the cell spans and the VTS time map apply unchanged. A listing without VIDEO_TS/ is read as Blu-ray, as before. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 9 +++ Sources/AetherEngine/Disc/DiscDirectory.swift | 49 ++++++++++++--- Sources/AetherEngine/Disc/DiscReader.swift | 49 ++++++++------- .../DVDCellTimelineTests.swift | 2 +- Tests/AetherEngineTests/DVDFolderTests.swift | 59 +++++++++++++++++++ docs/api.md | 2 +- docs/formats.md | 2 + 7 files changed, 143 insertions(+), 29 deletions(-) create mode 100644 Tests/AetherEngineTests/DVDFolderTests.swift diff --git a/CHANGELOG.md b/CHANGELOG.md index 62041fd06..9c3d161e5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,15 @@ the public-API contract. ## [Unreleased] +### Added + +- **A DVD-Video ripped as a `VIDEO_TS` folder plays over HTTP, as a `BDMV` folder does.** A + `DiscDirectoryReader` (or `HTTPDiscDirectoryReader`) whose listing holds `VIDEO_TS/` files is now read + as DVD-Video: the title sets, VMGI filtering, durations, chapters and languages come from the same code + as an image's (`DiscReader.dvdTitleSet`, now shared), the selected title's VOBs play back to back in + part order, and the cell timeline and VTS time map apply as on an image, since the VOBs in order are + the same run of bytes as their sectors. A listing without `VIDEO_TS/` is read as Blu-ray, as before. + ### Fixed - **The surround-downmix warning no longer fires on 5.1 inside a Dolby MAT carrier.** After an APAC diff --git a/Sources/AetherEngine/Disc/DiscDirectory.swift b/Sources/AetherEngine/Disc/DiscDirectory.swift index b42dfc86f..8015b2e91 100644 --- a/Sources/AetherEngine/Disc/DiscDirectory.swift +++ b/Sources/AetherEngine/Disc/DiscDirectory.swift @@ -1,15 +1,16 @@ import Foundation -/// A Blu-ray disc that a host serves as a FOLDER (the `BDMV` tree, one address per file) rather than -/// as a single image. Hand one to `load(source: .custom(reader))`: the engine reads the playlists, -/// selects the title and concatenates its clips the same way it does for a Blu-ray ISO, so titles, -/// chapters, track languages and the multi-clip timeline (AE#105) all behave as they do there. +/// A disc that a host serves as a FOLDER (the `BDMV` or `VIDEO_TS` tree, one address per file) rather +/// than as a single image. Hand one to `load(source: .custom(reader))`: the engine selects the title and +/// concatenates its clips or VOBs the same way it does for an ISO, so titles, chapters, track languages, +/// the multi-clip and multi-cell timelines and the seek tables all behave as they do there. A listing +/// with `VIDEO_TS/` in it is read as DVD-Video, anything else as Blu-ray. /// /// A folder carries no bytes of its own: `read` is never called on it. The engine recognizes the /// conformance before it reads anything and opens the files it needs through `openDiscFile(_:)`. public protocol DiscDirectoryReader: IOReader { /// Every file of the disc that can be read: its path relative to the disc root, with `/` as the - /// separator (`BDMV/STREAM/00001.m2ts`), and its size in bytes. + /// separator (`BDMV/STREAM/00001.m2ts`, `VIDEO_TS/VTS_01_1.VOB`), and its size in bytes. var discFiles: [(path: String, size: Int64)] { get } /// A new, independent reader over one file of `discFiles` (matched case-insensitively), or nil when @@ -18,11 +19,11 @@ public protocol DiscDirectoryReader: IOReader { /// The `.mpls` file name of the title to play (`00800.mpls`), when the host already knows it. The /// engine then reads only that playlist and publishes it as the only title; nil reads every playlist - /// and selects the main title by the engine's own rules, as on an ISO. + /// and selects the main title by the engine's own rules, as on an ISO. Blu-ray only. var preferredPlaylist: String? { get } } -/// A Blu-ray folder over HTTP: one URL per file, each read through its own range reader, so an origin +/// A disc folder over HTTP: one URL per file, each read through its own range reader, so an origin /// serves the disc as plain files and never remuxes it. public final class HTTPDiscDirectoryReader: DiscDirectoryReader { public struct File: Sendable { @@ -239,6 +240,40 @@ final class MultiFileConcatIOReader: IOReader, SourceTransferCounting, @unchecke } extension DiscReader { + /// A DVD-Video folder: the image path (`wrapDVD`) with files in place of sector ranges. A title set's + /// VOBs follow each other in part order, which is the same run of bytes as their sectors on a disc, so + /// the cell spans and the time map from its VTS IFO apply unchanged. + static func wrapDVDFolder(_ folder: DiscDirectoryReader, selectTitleID: Int?) -> DiscInfo? { + // Only VIDEO_TS's own files; a folder has no sectors, so each file is listed by name and size. + let entries = folder.discFiles.filter { $0.path.uppercased().hasPrefix("VIDEO_TS/") } + var byName: [String: (path: String, size: Int64)] = [:] + let files = entries.compactMap { entry -> DiscFile? in + let name = String(entry.path.split(separator: "/").last ?? "") + guard !name.isEmpty, entry.size > 0, entry.size <= Int64(Int.max) else { return nil } + byName[name.uppercased()] = entry + return DiscFile(name: name, startSector: 0, length: Int(entry.size)) + } + let readFile: (DiscFile) -> [UInt8] = { file in + guard let entry = byName[file.name.uppercased()], let reader = folder.openDiscFile(entry.path) else { return [] } + defer { reader.close() } + return readAll(reader, [(offset: 0, length: entry.size)]) + } + guard let set = dvdTitleSet(files: files, selectTitleID: selectTitleID, readFile: readFile) else { + EngineLog.emit("[disc] VIDEO_TS folder: no title VOBs among \(entries.count) files", category: .demux) + return nil + } + let segments = set.vobs.compactMap { vob in + byName[vob.name.uppercased()].map { MultiFileConcatIOReader.Segment(path: $0.path, size: $0.size) } + } + guard !segments.isEmpty else { return nil } + let (cellTimeline, timeMap) = set.selectedIFO.map(dvdCellTimeline) ?? ([], nil) + EngineLog.emit("[disc] DVD-Video folder recognized: \(set.titles.count) title(s), selected \(set.selectedIndex) " + + "VOBs=\(segments.map(\.path)) bytes=\(segments.reduce(0) { $0 + $1.size })", category: .demux) + return DiscInfo(reader: MultiFileConcatIOReader(segments: segments, opener: { folder.openDiscFile($0) }), + formatHint: "mpeg", titles: set.titles, selectedTitleIndex: set.selectedIndex, + clipTimeline: cellTimeline, dvdTimeMap: timeMap) + } + /// A Blu-ray folder: the image path (`wrapBluRay`) with files in place of UDF extents. The playlist scan /// and the clip assembly are the same bounded code, so a folder cannot make recognition do more than /// an image can (audit NET-103). diff --git a/Sources/AetherEngine/Disc/DiscReader.swift b/Sources/AetherEngine/Disc/DiscReader.swift index 5b0aa7de1..033aa440f 100644 --- a/Sources/AetherEngine/Disc/DiscReader.swift +++ b/Sources/AetherEngine/Disc/DiscReader.swift @@ -276,6 +276,9 @@ enum DiscReader { // A disc folder is a set of files, not one byte range, so it is recognized before the cache, // which rebuilds a reader from extents of a single image. if let folder = reader as? DiscDirectoryReader { + if folder.discFiles.contains(where: { $0.path.uppercased().hasPrefix("VIDEO_TS/") }) { + return wrapDVDFolder(folder, selectTitleID: selectTitleID) + } return wrapBluRayFolder(folder, selectTitleID: selectTitleID) } if let cacheKey, let cached = DiscRecognitionCache.lookup(key: cacheKey, selectTitleID: selectTitleID) { @@ -343,6 +346,26 @@ enum DiscReader { /// listing holds no title VOBs. private static func wrapDVD(_ reader: IOReader, files: [DiscFile], sectorSize: Int, selectTitleID: Int?, cacheKey: String?) -> DiscInfo? { + let readFile: (DiscFile) -> [UInt8] = { file in + readAll(reader, [(offset: Int64(file.startSector * sectorSize), length: Int64(file.length))]) + } + guard let set = dvdTitleSet(files: files, selectTitleID: selectTitleID, readFile: readFile) else { return nil } + let extents = set.vobs.map { + (offset: Int64($0.startSector * sectorSize), length: Int64($0.length)) + } + let (cellTimeline, timeMap) = set.selectedIFO.map(dvdCellTimeline) ?? ([], nil) + storeRecognition(cacheKey: cacheKey, selectTitleID: selectTitleID, + formatHint: "mpeg", titles: set.titles, selectedIndex: set.selectedIndex, extents: extents, + clipTimeline: cellTimeline, dvdTimeMap: timeMap) + return DiscInfo(reader: ConcatIOReader(base: reader, extents: extents), + formatHint: "mpeg", titles: set.titles, selectedTitleIndex: set.selectedIndex, + clipTimeline: cellTimeline, dvdTimeMap: timeMap) + } + + /// The titles of a `VIDEO_TS` listing and the selected one's VOBs (in part order) and VTS IFO, whatever + /// holds the bytes: `readFile` reads one whole file. nil when the listing holds no title VOBs. + static func dvdTitleSet(files: [DiscFile], selectTitleID: Int?, readFile: (DiscFile) -> [UInt8]) + -> (titles: [DiscTitle], selectedIndex: Int, vobs: [DiscFile], selectedIFO: [UInt8]?)? { let groups = DVDTitleSelector.enumerateTitleVOBGroups(files) guard !groups.isEmpty else { return nil } // VIDEO_TS.IFO's TT_SRPT names which title sets are real titles; filter the VOB groups to those so @@ -350,19 +373,12 @@ enum DiscReader { // falls back to the full VOB-grouped set, so a disc with an unreadable VMGI still plays multi-title. var orderedGroups = groups let filesByName = Dictionary(files.map { ($0.name.uppercased(), $0) }, uniquingKeysWith: { first, _ in first }) - if let ifoFile = filesByName["VIDEO_TS.IFO"] { - let ifoBytes = readAll(reader, [(offset: Int64(ifoFile.startSector * sectorSize), - length: Int64(ifoFile.length))]) - if let ifoTitles = DVDIFOParser.parseTitles(ifoBytes) { - let titleVTSNs = Set(ifoTitles.map(\.vtsn)) - let filtered = groups.filter { titleVTSNs.contains($0.vtsn) } - if !filtered.isEmpty { orderedGroups = filtered } - } + if let ifoFile = filesByName["VIDEO_TS.IFO"], let ifoTitles = DVDIFOParser.parseTitles(readFile(ifoFile)) { + let titleVTSNs = Set(ifoTitles.map(\.vtsn)) + let filtered = groups.filter { titleVTSNs.contains($0.vtsn) } + if !filtered.isEmpty { orderedGroups = filtered } } let selectedIndex = selectTitleID.flatMap { orderedGroups.indices.contains($0) ? $0 : nil } ?? 0 - let extents = orderedGroups[selectedIndex].vobs.map { - (offset: Int64($0.startSector * sectorSize), length: Int64($0.length)) - } // Whole-VTS titles. Each VTS_NN_0.IFO's main PGC gives the title duration and chapter starts; a disc // with an unreadable VTS IFO keeps duration 0 / no chapters but still plays. dvdVTSN keeps the // title -> title-set mapping. @@ -376,8 +392,7 @@ enum DiscReader { let nn = g.vtsn < 10 ? "0\(g.vtsn)" : "\(g.vtsn)" let ifoName = "VTS_\(nn)_0.IFO" if let vtsIFO = filesByName[ifoName] { - let bytes = readAll(reader, [(offset: Int64(vtsIFO.startSector * sectorSize), - length: Int64(vtsIFO.length))]) + let bytes = readFile(vtsIFO) if idx == selectedIndex { selectedIFO = bytes } if let detail = DVDIFOParser.parseTitleDetail(bytes) { durationTicks = detail.durationTicks @@ -392,13 +407,7 @@ enum DiscReader { streamLanguages: streamLanguages, dvdSubpictureStreamIDs: subpictureStreamIDs) } - let (cellTimeline, timeMap) = selectedIFO.map(dvdCellTimeline) ?? ([], nil) - storeRecognition(cacheKey: cacheKey, selectTitleID: selectTitleID, - formatHint: "mpeg", titles: titles, selectedIndex: selectedIndex, extents: extents, - clipTimeline: cellTimeline, dvdTimeMap: timeMap) - return DiscInfo(reader: ConcatIOReader(base: reader, extents: extents), - formatHint: "mpeg", titles: titles, selectedTitleIndex: selectedIndex, - clipTimeline: cellTimeline, dvdTimeMap: timeMap) + return (titles, selectedIndex, orderedGroups[selectedIndex].vobs, selectedIFO) } /// The selected DVD title's cells as AE#105 clip spans, and its time map. diff --git a/Tests/AetherEngineTests/DVDCellTimelineTests.swift b/Tests/AetherEngineTests/DVDCellTimelineTests.swift index aa99c71bd..a116fb2f6 100644 --- a/Tests/AetherEngineTests/DVDCellTimelineTests.swift +++ b/Tests/AetherEngineTests/DVDCellTimelineTests.swift @@ -27,7 +27,7 @@ struct DVDCellTimelineTests { } /// A PGC whose cell playback table holds `cells`, one program per cell. - private static func pgc(_ cells: [CellSpec]) -> [UInt8] { + static func pgc(_ cells: [CellSpec]) -> [UInt8] { var pgc = [UInt8](repeating: 0, count: 0xEC) pgc[2] = UInt8(cells.count) pgc[3] = UInt8(cells.count) diff --git a/Tests/AetherEngineTests/DVDFolderTests.swift b/Tests/AetherEngineTests/DVDFolderTests.swift new file mode 100644 index 000000000..2ccebea0c --- /dev/null +++ b/Tests/AetherEngineTests/DVDFolderTests.swift @@ -0,0 +1,59 @@ +import Foundation +import Testing +@testable import AetherEngine + +/// A DVD-Video served as a `VIDEO_TS` folder (`DiscDirectoryReader`): the same title sets, cell spans +/// and time map as an image, over files instead of sectors. +@Suite("DVD-Video folder sources") +struct DVDFolderTests { + typealias Folder = BluRayFolderTests.MemoryFolder + + static func bytes(_ seed: UInt8, _ count: Int) -> Data { + Data((0.. Folder { + Folder([ + ("VIDEO_TS/VIDEO_TS.IFO", bytes(1, 2048)), + ("VIDEO_TS/VIDEO_TS.VOB", bytes(2, 2048)), + ("VIDEO_TS/VTS_01_0.IFO", vts1IFO.map { Data($0) } ?? bytes(3, 2048)), + ("VIDEO_TS/VTS_01_0.VOB", bytes(4, 2048)), + ("VIDEO_TS/VTS_01_1.VOB", bytes(5, 7000)), + ("VIDEO_TS/VTS_01_2.VOB", bytes(6, 4000)), + ("VIDEO_TS/VTS_02_1.VOB", bytes(7, 3000)), + ]) + } + + @Test("the main title set's VOBs play back to back in part order, and the other title is selectable") + func titleSetsFromAFolder() throws { + let folder = Self.folder() + let info = try #require(try DiscReader.wrap(folder)) + #expect(info.formatHint == "mpeg") + #expect(info.titles.map(\.dvdVTSN) == [1, 2]) + #expect(BluRayFolderTests.readToEnd(info.reader) == Self.bytes(5, 7000) + Self.bytes(6, 4000)) + #expect(info.reader.seek(offset: 0, whence: 65536) == 11000) + // The menu VOBs are never opened, and no Blu-ray path is tried. + #expect(!folder.openedPaths.contains("VIDEO_TS/VTS_01_0.VOB")) + + let second = try #require(try DiscReader.wrap(Self.folder(), selectTitleID: 1)) + #expect(BluRayFolderTests.readToEnd(second.reader) == Self.bytes(7, 3000)) + } + + @Test("the selected title's cells and time map apply to the folder's concatenated VOBs") + func cellTimelineFromAFolder() throws { + let ifo = DVDCellTimelineTests.vtsIFO(pgcs: [DVDCellTimelineTests.pgc(DVDCellTimelineTests.threeCells)], + timeMaps: [(unit: 4, sectors: [10, 20])]) + let info = try #require(try DiscReader.wrap(Self.folder(vts1IFO: ifo))) + #expect(info.clipTimeline.map(\.concatByteStart) == [0, 1000 * 2048, 3000 * 2048]) + #expect(info.dvdTimeMap?.byteOffsets == [10 * 2048, 20 * 2048]) + #expect(info.titles.first?.durationTicks == UInt64(2100 * discTickRate)) + } + + @Test("a VIDEO_TS folder without title VOBs is not a disc") + func noTitleVOBs() throws { + let folder = Folder([("VIDEO_TS/VIDEO_TS.IFO", Self.bytes(1, 2048)), ("VIDEO_TS/VIDEO_TS.VOB", Self.bytes(2, 2048))]) + #expect(try DiscReader.wrap(folder) == nil) + } +} diff --git a/docs/api.md b/docs/api.md index ecf343240..8db9b8b72 100644 --- a/docs/api.md +++ b/docs/api.md @@ -442,7 +442,7 @@ try await player.reloadAtCurrentPosition() | --- | --- | | `load(url:startPosition:options:audioSourceStreamIndex:discTitleID:)` | `async throws -> SourceProbe?`. Discardable. Tears down any running session first. | | `load(source:startPosition:options:audioSourceStreamIndex:discTitleID:)` | Same, for `MediaSource.url` or `.custom(IOReader, formatHint:)`. A custom source whose initial probe fails throws, since it cannot be reopened by URL. | -| `DiscDirectoryReader`, `HTTPDiscDirectoryReader` | A Blu-ray ripped as a `BDMV` folder, for `.custom(reader)`: `discFiles` lists the files (root-relative path and size), `openDiscFile(_:)` opens one, `preferredPlaylist` optionally names the title's `.mpls`. `HTTPDiscDirectoryReader(files:preferredPlaylist:httpHeaders:)` serves each `File` (path, size, url) from its own `Range` origin. See [Blu-ray folders](formats.md#disc-dvd--blu-ray-iso). | +| `DiscDirectoryReader`, `HTTPDiscDirectoryReader` | A Blu-ray ripped as a `BDMV` folder, or a DVD-Video as a `VIDEO_TS` folder, for `.custom(reader)`: `discFiles` lists the files (root-relative path and size), `openDiscFile(_:)` opens one, `preferredPlaylist` optionally names the title's `.mpls`. `HTTPDiscDirectoryReader(files:preferredPlaylist:httpHeaders:)` serves each `File` (path, size, url) from its own `Range` origin. See [Blu-ray folders](formats.md#disc-dvd--blu-ray-iso). | | `reloadAtCurrentPosition()` | `async throws`. Background reopen at the current position, preserving options. Session-preserving: it finishes an installed audio tap and keeps the native host where it can. It also preserves the session's TRANSPORT rather than replaying `autoplay`, so a session that was playing comes back playing and one that was paused comes back paused, whatever the mount was given (AE#464 round 2). A `play()` or `pause()` that arrives while it runs is the newer word and decides how it comes back. The one exception is the resume after a background teardown, which has no transport left to read and is the host's call, so there the mount flag still decides. | | `prepareForItemReplacement()` | One-shot. Keeps the current native `AVPlayerItem` attached until the next `load()` replaces it atomically, for a host that mounts the engine's own player layer and would otherwise show a black layer across the nil-item gap of a foreground episode or playlist change. Consumed by the next `load()`, cancelled by `stop()`, no effect when the outgoing session is not native. PiP hosts do not need it: an active PiP window already forces the handover (AE#158). | | `stop(resetDisplayCriteria:finalTeardown:)` | Ends the session, `state` becomes `.idle`, `startupProgress` becomes nil. `resetDisplayCriteria: false` keeps the panel in its current mode across an item handoff. | diff --git a/docs/formats.md b/docs/formats.md index 7e916c61b..10b0e973c 100644 --- a/docs/formats.md +++ b/docs/formats.md @@ -510,6 +510,8 @@ A remote `.iso` / `.img` / `.udf` needs an origin that honours HTTP `Range`, bec **Blu-ray folders.** A disc that was ripped as a `BDMV` folder rather than an image plays the same way through `MediaSource.custom` with a `DiscDirectoryReader`: the host lists the folder's files (path relative to the disc root, and size) and opens any one of them on request. `HTTPDiscDirectoryReader` is the ready-made one, with one URL per file, so an origin serves the disc as plain files over `Range` and never remuxes it. Recognition is the image path with files in place of UDF extents: the same bounded playlist scan and title selection, the selected title's `.m2ts` files concatenated in playlist order as one MPEG-TS, the same multi-clip timeline, chapters and STN track languages, and the same seek by the clips' EP maps when the folder carries `BDMV/CLIPINF`. A host that already knows the main playlist names it in `preferredPlaylist`, and only that playlist is read and published as the title. The concatenating reader opens a clip's file when the read reaches it and keeps at most three open. A controlled probe (`limits` or `cancellation`) does not recognize a folder yet; an uncontrolled `probe(source:)` and `load(source:)` do. +**DVD-Video folders.** A listing that holds `VIDEO_TS/` files is read as DVD-Video instead, by the same title-set code as an image: the selected title set's VOBs are concatenated in part order, which is the same run of bytes as their sectors on a disc, so the cell timeline and the VTS time map apply unchanged. `preferredPlaylist` does not apply. + **Title selection.** `engine.discTitles` (`@Published [TitleInfo]`) lists the disc's titles (id, name, duration, chapter count) and `engine.selectedDiscTitle` is the active one; `engine.selectTitle(id:)` switches title, rebuilding the pipeline from the new title's head. The selection survives audio-track switches and background-resume reloads, and a fresh `load` defaults to the main title (an out-of-range id clamps to it). Blu-ray enumerates all playlists; a DVD enumerates its title sets (the VMGI TT_SRPT title list, resolved whole-VTS, with the duration read from each VTS's main program chain; per-cell / episodic splitting is deferred). **Chapters.** `engine.discChapters` (`@Published [ChapterInfo]`) carries the selected title's chapters; `engine.selectChapter(id:)` seeks to one (a thin `seek` wrapper, no pipeline rebuild). For Blu-ray they come from the playlist's PlayListMark entries (entry marks only; link points dropped), each mark's timestamp on its clip's STC offset by the clip's in_time and the cumulative duration of preceding play items. For DVD they come from the main program chain's program map plus the cumulative cell playback times. Chapter starts are title-relative (0-based); `selectChapter` adds the title's content-start base (the native playlist shift, or the software path's container start PTS) so the seek lands on the source-PTS playback axis. From 3bf8acc850411b9960db9fd48ff26df701658ca9 Mon Sep 17 00:00:00 2001 From: orut34iop Date: Tue, 13 Oct 2026 15:29:16 +0800 Subject: [PATCH 002/132] fix(cache): publish retained software VOD resident ranges --- CHANGELOG.md | 6 + .../AetherEngine/AetherEngine+Loading.swift | 7 + Sources/AetherEngine/AetherEngine.swift | 3 +- .../Native/SoftwarePacketCoverage.swift | 26 +++ .../Native/SoftwarePacketDiskFIFO.swift | 1 + .../Native/SoftwarePacketReadAhead.swift | 89 ++++++++ .../Native/SoftwarePlaybackHost.swift | 62 ++++++ .../Native/SoftwareVideoPacketCoverage.swift | 6 + .../SoftwareResidentRangesTests.swift | 204 ++++++++++++++++++ docs/api.md | 19 +- 10 files changed, 420 insertions(+), 3 deletions(-) create mode 100644 Tests/AetherEngineTests/SoftwareResidentRangesTests.swift diff --git a/CHANGELOG.md b/CHANGELOG.md index 73d956cdd..d02c73bff 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,12 @@ the public-API contract. ### Fixed +- **Software VOD exposes retained cache ranges to host timelines.** `residentRanges` now covers + compressed video packets together with the selected audio stream, including retained history, + gaps and disk eviction. Cached seeks keep the same spans; cold seeks and teardown clear them. + Sampling continues while paused and runs off the main actor, with stale seek/stop results + discarded. Hosts no longer need to infer a cache band's start from the moving playhead. + - **A host with its own Objective-C and C++ links again (#752).** Since 7.34.0 LibTrueHD and LibDovi each brought a `rust_eh_personality`, and with the host's `__objc_personality_v0` and `__gxx_personality_v0` that made four personality routines, one more than compact unwind encodes diff --git a/Sources/AetherEngine/AetherEngine+Loading.swift b/Sources/AetherEngine/AetherEngine+Loading.swift index c6a71dc6b..f88276bbf 100644 --- a/Sources/AetherEngine/AetherEngine+Loading.swift +++ b/Sources/AetherEngine/AetherEngine+Loading.swift @@ -1941,6 +1941,13 @@ extension AetherEngine { // #353: the settled picture size, wired next to the frame times because a host laying out an // overlay needs the rectangle as well as the clock, and both come off this renderer. mirrorSoftwareDisplaySize(from: host.$videoDisplaySize, storeIn: &softwareCancellables) + host.$residentRanges + .sink { [weak self, weak host] ranges in + guard let self, let host, self.softwareHost === host, + self.loadGeneration == generation, !self.isLive else { return } + self.residentRanges = ranges + } + .store(in: &softwareCancellables) // SW-PiP: publish the bridge once the session owns its layer (the layer object is stable for // the session; the host attaches it to the view and, on PiP start, to the system window). softwarePiPSource = SoftwarePiPSource(layer: host.displayLayer, isLive: isLive, engine: self) diff --git a/Sources/AetherEngine/AetherEngine.swift b/Sources/AetherEngine/AetherEngine.swift index 295e5c0e7..8c22f2a17 100644 --- a/Sources/AetherEngine/AetherEngine.swift +++ b/Sources/AetherEngine/AetherEngine.swift @@ -83,7 +83,8 @@ public final class AetherEngine: ObservableObject { didSet { recomputePlaybackPhase() } } - /// Where the loopback segment cache holds picture right now, on the same 0-based display axis as + /// Where the loopback segment cache or software VOD packet cache holds picture right now, + /// with selected audio coverage required for software A/V, on the same 0-based display axis as /// `currentTime`, `duration` and `seek(to:)`. An empty array is the nil-equivalent, and live /// sessions publish one always: their rewind depth is `clock.seekableLiveRange`, a different /// contract. Residency is not a promise that a seek inside a range will be instant; the player may diff --git a/Sources/AetherEngine/Native/SoftwarePacketCoverage.swift b/Sources/AetherEngine/Native/SoftwarePacketCoverage.swift index bd1e2998c..20408a1e1 100644 --- a/Sources/AetherEngine/Native/SoftwarePacketCoverage.swift +++ b/Sources/AetherEngine/Native/SoftwarePacketCoverage.swift @@ -22,6 +22,32 @@ struct SoftwarePacketCoverage: Sendable { var rangeCount: Int { ranges.count } var isFull: Bool { ranges.count >= maximumRangeCount } + /// Immutable presentation spans for a background residency snapshot. Never round over a gap. + func rangesSeconds(timeBaseNumerator: Int32, timeBaseDenominator: Int32) -> [ClosedRange] { + guard timeBaseNumerator > 0, timeBaseDenominator > 0 else { return [] } + return ranges.compactMap { range in + guard Int64(exactly: Double(range.lowerBound)) == range.lowerBound, + Int64(exactly: Double(range.upperBound)) == range.upperBound else { return nil } + let start = Double(range.lowerBound) * Double(timeBaseNumerator) / Double(timeBaseDenominator) + let end = Double(range.upperBound) * Double(timeBaseNumerator) / Double(timeBaseDenominator) + guard start.isFinite, end.isFinite, start < end else { return nil } + return start...end + } + } + + /// Linear intersection of sorted, disjoint spans; selected audio is required even when muted. + static func intersection(_ video: [ClosedRange], _ audio: [ClosedRange]) -> [ClosedRange] { + var result: [ClosedRange] = [] + var v = 0, a = 0 + while v < video.count, a < audio.count { + let start = max(video[v].lowerBound, audio[a].lowerBound) + let end = min(video[v].upperBound, audio[a].upperBound) + if start < end { result.append(start...end) } + if video[v].upperBound < audio[a].upperBound { v += 1 } else { a += 1 } + } + return result + } + /// Adds [pts, pts + duration). Invalid or capacity-exceeding input leaves existing coverage /// unchanged. Rejecting a sparse extension is conservative: it never claims a missing packet. @discardableResult diff --git a/Sources/AetherEngine/Native/SoftwarePacketDiskFIFO.swift b/Sources/AetherEngine/Native/SoftwarePacketDiskFIFO.swift index 2126cbf78..c665727fe 100644 --- a/Sources/AetherEngine/Native/SoftwarePacketDiskFIFO.swift +++ b/Sources/AetherEngine/Native/SoftwarePacketDiskFIFO.swift @@ -57,6 +57,7 @@ final class SoftwarePacketDiskFIFO: @unchecked Sendable { let staleSweepResult: StaleSweepResult private let chunkTargetBytes: Int private let retainConsumed: Bool + var retainsConsumedPackets: Bool { retainConsumed } private let lock = NSLock() private var writer: FileHandle? private var reader: FileHandle? diff --git a/Sources/AetherEngine/Native/SoftwarePacketReadAhead.swift b/Sources/AetherEngine/Native/SoftwarePacketReadAhead.swift index 3a7ee53d2..5d4e533cf 100644 --- a/Sources/AetherEngine/Native/SoftwarePacketReadAhead.swift +++ b/Sources/AetherEngine/Native/SoftwarePacketReadAhead.swift @@ -82,6 +82,7 @@ final class SoftwarePacketReadAhead: @unchecked Sendable { } private var keyframes: [Keyframe] = [] private let maximumKeyframes = 65_536 + private var residency = SoftwarePacketResidency() /// Construct only off-main: creating the FIFO touches the temporary volume. init(video: Stream, audio: Stream?, byteBudget: Int, forwardSeconds: Double, @@ -134,6 +135,35 @@ final class SoftwarePacketReadAhead: @unchecked Sendable { cacheSeekMisses: cacheSeekMisses) } + /// Called only by a background sampler. Copy bounded metadata under the short state lock; + /// timestamp conversion/intersection and keyframe selection happen after releasing it. No I/O. + var residentRanges: [ClosedRange] { + condition.lock() + guard !closed, !sourceRepositioning, !resetPending, failure == nil, + fifo.retainsConsumedPackets else { condition.unlock(); return [] } + let videoCoverage = self.videoCoverage + let presentationCoverage = self.presentationCoverage + let audioCoverage = self.audioCoverage + let keyframes = self.keyframes + let videoFloor = residency.videoFloor + let audioFloor = residency.audioFloor + condition.unlock() + let videoRanges = presentationCoverage?.rangesSeconds ?? videoCoverage.rangesSeconds( + timeBaseNumerator: video.numerator, timeBaseDenominator: video.denominator) + let ranges: [ClosedRange] + if let audio { + ranges = SoftwarePacketCoverage.intersection(videoRanges, audioCoverage.rangesSeconds( + timeBaseNumerator: audio.numerator, timeBaseDenominator: audio.denominator)) + } else { ranges = videoRanges } + guard let keyframe = keyframes.lazy.map(\.seconds) + .filter({ $0 >= videoFloor }).min() else { return [] } + let floor = max(keyframe, audio == nil ? -.infinity : audioFloor) + return ranges.compactMap { range in + let start = max(range.lowerBound, floor) + return start < range.upperBound ? start...range.upperBound : nil + } + } + /// Main-thread safe: metadata only. Decode/render backpressure still belongs to the old loop. func updatePlayhead(_ seconds: Double) { guard seconds.isFinite else { return } @@ -405,6 +435,7 @@ final class SoftwarePacketReadAhead: @unchecked Sendable { videoCoverage.reset(); audioCoverage.reset(); presentationCoverage?.reset() storedVideoSeconds = nil; consumedVideoSeconds = nil keyframes.removeAll(keepingCapacity: true) + residency = SoftwarePacketResidency() } /// Presentation seconds of a stored packet, or nil when this stream cannot express them. @@ -420,6 +451,7 @@ final class SoftwarePacketReadAhead: @unchecked Sendable { residentBytes = state.residentByteCount if let floor = state.oldestRetainedChunkID { keyframes.removeAll { $0.cursor.chunkID < floor } + residency.evict(before: floor) } else { keyframes.removeAll(keepingCapacity: true) } } @@ -497,6 +529,12 @@ final class SoftwarePacketReadAhead: @unchecked Sendable { condition.lock() if token == sourceEpoch, !closed, !sourceRepositioning { copyDiskStateLocked(state) + if packet.streamIndex == video.index || packet.streamIndex == audio?.index { + let stream = packet.streamIndex == video.index ? video : audio! + residency.insert(chunkID: cursor.chunkID, pts: packet.pts, + duration: packet.duration, numerator: stream.numerator, + denominator: stream.denominator, isVideo: packet.streamIndex == video.index) + } if packet.streamIndex == video.index { if presentationCoverage != nil { if presentationCoverage?.isFull == true, let tick = playheadTick(video) { @@ -640,3 +678,54 @@ final class SoftwarePacketReadAhead: @unchecked Sendable { + "qos=\(QoSClass.name(reportedProducerQoS))" } } + +/// Constant-size summaries per retained disk chunk, never per packet. The cumulative discarded +/// high-water marks also cover reordered PTS: clipping at the first packet in the new chunk alone +/// could otherwise claim an evicted B picture or audio packet still exists. Metadata pressure may +/// conservatively shorten a displayed range; it never changes disk retention or seek policy. +struct SoftwarePacketResidency: Sendable { + private struct Chunk: Sendable { + let id: UInt64 + var videoEnd = -Double.infinity + var audioEnd = -Double.infinity + } + private var chunks: [Chunk] = [] + private(set) var videoFloor = -Double.infinity + private(set) var audioFloor = -Double.infinity + private let maximumChunks: Int + + init(maximumChunks: Int = 4096) { self.maximumChunks = max(1, maximumChunks) } + + mutating func insert(chunkID: UInt64, pts: Int64, duration: Int64, + numerator: Int32, denominator: Int32, isVideo: Bool) { + if chunks.last?.id != chunkID { + if chunks.count == maximumChunks { discardFirst(1) } + chunks.append(Chunk(id: chunkID)) + } + // A successor-duration video with no packet duration still excludes the evicted PTS. + let (end, overflow) = pts.addingReportingOverflow(max(1, duration)) + let seconds: Double + if pts == Int64.min || overflow || numerator <= 0 || denominator <= 0 + || Int64(exactly: Double(end)) != end { + seconds = .infinity // unknown discarded timestamps cannot prove later residency + } else { + seconds = Double(end) * Double(numerator) / Double(denominator) + } + let index = chunks.count - 1 + if isVideo { chunks[index].videoEnd = max(chunks[index].videoEnd, seconds) } + else { chunks[index].audioEnd = max(chunks[index].audioEnd, seconds) } + } + + mutating func evict(before chunkID: UInt64) { + discardFirst(chunks.prefix { $0.id < chunkID }.count) + } + + private mutating func discardFirst(_ count: Int) { + guard count > 0 else { return } + for chunk in chunks.prefix(count) { + videoFloor = max(videoFloor, chunk.videoEnd) + audioFloor = max(audioFloor, chunk.audioEnd) + } + chunks.removeFirst(count) + } +} diff --git a/Sources/AetherEngine/Native/SoftwarePlaybackHost.swift b/Sources/AetherEngine/Native/SoftwarePlaybackHost.swift index a383b4d29..58199214d 100644 --- a/Sources/AetherEngine/Native/SoftwarePlaybackHost.swift +++ b/Sources/AetherEngine/Native/SoftwarePlaybackHost.swift @@ -138,6 +138,8 @@ final class SoftwarePlaybackHost { private var vodPacketReadAhead: SoftwarePacketReadAhead? /// #687: this host's entries in the process-wide retention ledger, released on `stop()`. private var retentionClaims: [RetentionClaims.Claim] = [] + @Published private(set) var residentRanges: [ClosedRange] = [] + private let residentRangePublication = SoftwareResidentRangePublication() /// The same public buffered-position axis as the live and native hosts, but backed by /// actual compressed A/V packet coverage. nil when no continuous cache span contains the clock. @@ -1473,6 +1475,7 @@ final class SoftwarePlaybackHost { // reposition can tell on `seekQueue` whether a newer seek has already taken over. bumpSeekGeneration() let generation = seekGeneration + residentRangePublication.invalidate() didReachEnd = false didParkClockAtEnd = false didEmitParkedDiag = false @@ -1538,6 +1541,7 @@ final class SoftwarePlaybackHost { switch preparation { case .success(let hit): cacheHit = hit + if !hit { residentRanges = [] } EngineLog.emit( "[SWHost] packet cache seek generation=\(generation) " + "result=\(hit ? "hit" : "miss") " @@ -1553,6 +1557,7 @@ final class SoftwarePlaybackHost { // a normal cache miss. Leave playback parked and publish an explicit failure. seekInFlight = false packetSource.close() + residentRanges = [] noteSeekSettled(generation) EngineLog.emit("[SWHost] packet cache seek generation=\(generation) result=error", category: .swPlayback) @@ -1842,6 +1847,8 @@ final class SoftwarePlaybackHost { noteSeekSettled(seekGeneration) timeTimer?.cancel() timeTimer = nil + residentRangePublication.invalidate() + residentRanges = [] vodPacketReadAhead?.close() vodPacketReadAhead = nil renderer.subtitleCompositor.reset() @@ -3285,6 +3292,32 @@ final class SoftwarePlaybackHost { // MARK: - Time updates + nonisolated static func displayResidentRanges(_ ranges: [ClosedRange], + sessionZeroSeconds: Double) -> [ClosedRange] { + guard sessionZeroSeconds.isFinite else { return [] } + let zero = max(0, sessionZeroSeconds) + return ranges.compactMap { range in + let start = max(0, range.lowerBound - zero), end = range.upperBound - zero + return start.isFinite && end.isFinite && start < end ? start...end : nil + } + } + + /// Also runs while paused, independently of rendered-time events. At most one bounded + /// metadata calculation is pending. Cached seeks keep the last valid spans; a miss clears + /// them at the actual discard decision. Seek/stop invalidate any pre-operation result. + private func sampleResidentRanges() { + guard !isLive, !stopRequested, !seekInFlight, + let source = vodPacketReadAhead else { return } + let zero = clockSessionZero + residentRangePublication.sample { + Self.displayResidentRanges(source.residentRanges, sessionZeroSeconds: zero) + } publish: { [weak self] ranges in + guard let self, self.vodPacketReadAhead === source, !self.stopRequested, + self.clockSessionZero == zero else { return } + if self.residentRanges != ranges { self.residentRanges = ranges } + } + } + private func startTimeUpdates() { timeTimer = Timer.publish(every: 0.25, on: .main, in: .common) .autoconnect() @@ -3292,6 +3325,7 @@ final class SoftwarePlaybackHost { guard let self else { return } // Independent of the clock below: a seek in flight must not defer the one check. self.checkSurfaceVisibilityIfDue() + self.sampleResidentRanges() guard let aOut = self.audioOutput else { return } // #254: a reposition in flight holds `currentTime` at its target; the synchronizer is // still on the pre-seek anchor and would drag the published position backwards. @@ -3576,3 +3610,31 @@ extension SoftwarePlaybackHost: LiveRecordingHost { ) } } + +/// A single background snapshot at a time. Invalidating at seek/stop prevents a delayed snapshot +/// from restoring a discarded range or from clearing a newer request. Main never waits for work. +@MainActor +final class SoftwareResidentRangePublication { + private var task: Task? + private var revision: UInt64 = 0 + + func invalidate() { + revision &+= 1 + task?.cancel() + task = nil + } + + @discardableResult + func sample(_ read: @escaping @Sendable () async -> [ClosedRange], + publish: @escaping @MainActor ([ClosedRange]) -> Void) -> Task? { + guard task == nil else { return nil } + let token = revision + task = Task { [weak self] in + let ranges = await Task.detached(priority: .utility, operation: read).value + guard let self, !Task.isCancelled, self.revision == token else { return } + self.task = nil + publish(ranges) + } + return task + } +} diff --git a/Sources/AetherEngine/Native/SoftwareVideoPacketCoverage.swift b/Sources/AetherEngine/Native/SoftwareVideoPacketCoverage.swift index 136b77b1a..ae44a14e9 100644 --- a/Sources/AetherEngine/Native/SoftwareVideoPacketCoverage.swift +++ b/Sources/AetherEngine/Native/SoftwareVideoPacketCoverage.swift @@ -48,6 +48,12 @@ struct SoftwareVideoPacketCoverage: Sendable { var rangeCount: Int { coverage.rangeCount } var isFull: Bool { coverage.isFull } + var rangesSeconds: [ClosedRange] { + guard !isInvalidated else { return [] } + return coverage.rangesSeconds(timeBaseNumerator: timeBaseNumerator, + timeBaseDenominator: timeBaseDenominator) + } + /// Duration/DTS are deliberately not inputs. Duplicate pending PTS do not advance the reorder /// watermark. At most depth + 1 timestamps are held transiently, then the minimum is emitted. @discardableResult diff --git a/Tests/AetherEngineTests/SoftwareResidentRangesTests.swift b/Tests/AetherEngineTests/SoftwareResidentRangesTests.swift new file mode 100644 index 000000000..62e0573aa --- /dev/null +++ b/Tests/AetherEngineTests/SoftwareResidentRangesTests.swift @@ -0,0 +1,204 @@ +import Foundation +import Testing +@testable import AetherEngine + +@Suite("Software VOD resident ranges", .timeLimit(.minutes(2))) +struct SoftwareResidentRangesTests { + private final class Input: @unchecked Sendable { + private let lock = NSLock() + private var packets: [SoftwareStoredPacket] + private var index = 0 + init(_ packets: [SoftwareStoredPacket]) { self.packets = packets } + func next() -> SoftwareStoredPacket? { + lock.lock(); defer { lock.unlock() } + guard index < packets.count else { return nil } + defer { index += 1 }; return packets[index] + } + } + private func packet(_ pts: Int64, stream: Int32 = 0, duration: Int64 = 40, + key: Bool = true) -> SoftwareStoredPacket { + .init(pts: pts, dts: pts, duration: duration, position: 0, streamIndex: stream, + flags: key ? 1 : 0, timeBaseNumerator: 1, timeBaseDenominator: 1000, + bytes: Data(repeating: 7, count: 100), sideData: []) + } + private func source(_ packets: [SoftwareStoredPacket], audio: Bool = false, + reorder: Int? = nil, budget: Int = 1_000_000, + chunkBytes: Int = 1024) throws -> SoftwarePacketReadAhead { + let input = Input(packets) + let fifo = try SoftwarePacketDiskFIFO(chunkTargetBytes: chunkBytes, retainConsumed: true) + let source = SoftwarePacketReadAhead(video: .init(index: 0, numerator: 1, denominator: 1000), + audio: audio ? .init(index: 1, numerator: 1, denominator: 1000) : nil, + byteBudget: budget, forwardSeconds: 3600, initialSourceClock: 0, + fifo: fifo, videoReorderDepth: reorder) { _ in input.next() } + source.start() + return source + } + + @Test("Backward and forward cache hits preserve the same spans through begin, prepare and settle") + func cachedSeeks() async throws { + let source = try source((0..<100).map { packet(Int64($0) * 40) }) + defer { source.close() } + try await waitFor { source.snapshot.sourceEnded } + #expect(source.residentRanges == [0...4]) + for target in [3.0, 1, 2.5, 0.5] { + source.updatePlayhead(target) + let token = source.beginSeek(to: target) + #expect(source.residentRanges == [0...4]) + #expect(try source.prepareSeek(token, to: target)) + #expect(source.residentRanges == [0...4]) + source.endSeek(token, sourceClock: target) + #expect(source.residentRanges == [0...4]) + } + let miss = source.beginSeek(to: 8) + #expect(try !source.prepareSeek(miss, to: 8)) + #expect(source.residentRanges.isEmpty) + source.close() + #expect(source.residentRanges.isEmpty) + } + + @Test("Required audio intersects video, keeps holes, and missing selected audio proves nothing") + func audioIntersection() async throws { + let video = (0..<100).map { packet(Int64($0) * 40) } + let audio = [packet(500, stream: 1, duration: 500), packet(2000, stream: 1, duration: 1000)] + let source = try source(video + audio, audio: true) + defer { source.close() } + try await waitFor { source.snapshot.sourceEnded } + #expect(source.residentRanges == [0.5...1, 2...3]) + let missing = try self.source(video, audio: true) + defer { missing.close() } + try await waitFor { missing.snapshot.sourceEnded } + #expect(missing.residentRanges.isEmpty) + } + + @Test("Successor PTS retains reorder semantics, a discontinuity, and no invented final frame") + func reorderedVideo() async throws { + let pts: [Int64] = [0, 120, 40, 80, 160, 2000, 2120, 2040, 2080] + let source = try source(pts.map { packet($0, duration: 0) }, reorder: 4) + defer { source.close() } + try await waitFor { source.snapshot.sourceEnded } + #expect(source.residentRanges == [0...0.16, 2...2.12]) + let invalid = try self.source(([Int64](stride(from: 0, to: 2000, by: 40)) + [0]) + .map { packet($0) }, reorder: 4) + defer { invalid.close() } + try await waitFor { invalid.snapshot.sourceEnded } + #expect(invalid.residentRanges.isEmpty) + } + + @Test("Real FIFO eviction shortens backward coverage without changing the read bytes") + func diskEviction() async throws { + let packets = (0..<100).map { packet(Int64($0) * 40) } + let source = try source(packets, budget: 1400, chunkBytes: 512) + defer { source.close() } + let retained = try await withTaskCancellationHandler { + try await withCheckedThrowingContinuation { (continuation: CheckedContinuation<[SoftwareStoredPacket], Error>) in + Thread.detachNewThread { + do { + var values: [SoftwareStoredPacket] = [] + while let packet = try source.read() { + values.append(packet) + source.updatePlayhead(Double(packet.pts) / 1000) + } + continuation.resume(returning: values) + } catch { continuation.resume(throwing: error) } + } + } + } onCancel: { source.close() } + #expect(retained == packets) + try await waitFor { source.snapshot.sourceEnded } + let spans = source.residentRanges + #expect(!spans.isEmpty) + #expect((spans.first?.lowerBound ?? 0) > 3) + #expect(spans.last?.upperBound == 4) + let miss = source.beginSeek(to: 0.1) + #expect(try !source.prepareSeek(miss, to: 0.1)) + #expect(source.residentRanges.isEmpty) + } + + @Test("Evicted reordered timestamps and metadata pressure only shorten coverage") + func residencyWatermarks() { + var value = SoftwarePacketResidency(maximumChunks: 2) + value.insert(chunkID: 0, pts: 900, duration: 40, numerator: 1, denominator: 1000, isVideo: true) + value.insert(chunkID: 1, pts: 100, duration: 40, numerator: 1, denominator: 1000, isVideo: true) + value.insert(chunkID: 1, pts: 1100, duration: 40, numerator: 1, denominator: 1000, isVideo: false) + value.evict(before: 1) + #expect(value.videoFloor == 0.94) + #expect(value.audioFloor == -.infinity) + value.insert(chunkID: 2, pts: 1200, duration: 0, numerator: 1, denominator: 1000, isVideo: true) + value.insert(chunkID: 3, pts: 1300, duration: 0, numerator: 1, denominator: 1000, isVideo: true) + #expect(value.videoFloor == 0.94) + #expect(value.audioFloor == 1.14) + value.evict(before: 4) + #expect(value.videoFloor == 1.301) + } + + @Test("Invalid discarded timestamps fail closed and reset begins a separate residency lifetime") + func invalidWatermark() { + var value = SoftwarePacketResidency() + value.insert(chunkID: 0, pts: .max, duration: 40, numerator: 1, denominator: 1000, isVideo: true) + value.evict(before: 1) + #expect(value.videoFloor == .infinity) + value = SoftwarePacketResidency() + #expect(value.videoFloor == -.infinity) + } + + @Test("Source origin mapping preserves gaps, clips negatives and rejects unknown origins") + func displayMapping() { + let spans = [90.0...105, 110...120] + #expect(SoftwarePlaybackHost.displayResidentRanges(spans, sessionZeroSeconds: 100) == [0...5, 10...20]) + #expect(SoftwarePlaybackHost.displayResidentRanges(spans, sessionZeroSeconds: .nan).isEmpty) + #expect(SoftwarePlaybackHost.displayResidentRanges(spans, sessionZeroSeconds: 200).isEmpty) + } + + @Test("Large inexact timestamps do not manufacture display ranges") + func exactness() { + var coverage = SoftwarePacketCoverage() + coverage.insert(pts: 9_007_199_254_740_993, duration: 4) + #expect(coverage.rangesSeconds(timeBaseNumerator: 1, timeBaseDenominator: 1).isEmpty) + } +} + +@MainActor +@Suite("Software resident range publication", .timeLimit(.minutes(2))) +struct SoftwareResidentRangePublicationTests { + private actor Gate { + private(set) var entered = false + private var waiter: CheckedContinuation? + func wait() async { + await withCheckedContinuation { waiter = $0; entered = true } + } + func release() { waiter?.resume(); waiter = nil } + } + + @Test("A delayed sampler leaves main responsive, coalesces ticks, and cannot publish after seek/stop") + func delayedSnapshots() async throws { + let publisher = SoftwareResidentRangePublication() + let gate = Gate() + var received: [[ClosedRange]] = [] + let old = publisher.sample { + await gate.wait() + return [0...100] + } publish: { received.append($0) } + #expect(publisher.sample({ [0...999] }, publish: { received.append($0) }) == nil) + try await waitFor { await gate.entered } + // This main-actor turn can run while the background read is suspended. A new seek or stop + // invalidates it; a new session/seek can immediately publish without waiting for that read. + publisher.invalidate() + let replacement = publisher.sample({ [200...220] }, publish: { received.append($0) }) + await replacement?.value + #expect(received == [[200...220]]) + await gate.release() + await old?.value + #expect(received == [[200...220]]) + } + + @Test("Paused growth, eviction and a reset publish without a clock tick") + func pausedUpdates() async { + let publisher = SoftwareResidentRangePublication() + var received: [[ClosedRange]] = [] + for spans: [ClosedRange] in [[10...30], [10...60], [20...60], [], [80...90]] { + let work = publisher.sample({ spans }, publish: { received.append($0) }) + await work?.value + } + #expect(received == [[10...30], [10...60], [20...60], [], [80...90]]) + } +} diff --git a/docs/api.md b/docs/api.md index ecf343240..7ecfb3ae4 100644 --- a/docs/api.md +++ b/docs/api.md @@ -585,10 +585,25 @@ Time lives on `player.clock`, a separate `ObservableObject`, so ~10 Hz ticks nev | `clock.$progress` | `currentTime / duration` | | `clock.$bufferedPosition` | source-axis position buffered ahead | | `clock.$liveEdgeTime`, `clock.$seekableLiveRange`, `clock.$behindLiveSeconds`, `clock.$isAtLiveEdge` | live-window surfaces. `seekableLiveRange` is the intersection of the DVR window (policy) and what the segment cache actually holds and can play forward from (fact), so it is honest to scale a rewind strip on and `seek(to:)` clamps to the same floor (AE#441). The two diverge for the whole first `dvrWindowSeconds` of a session and again whenever retention evicts faster than the window slides. Software live sessions hold a packet ring instead, which evicts by the same byte budget as the native cache (a quarter of the volume's free space, at most 2 GiB, at least 64 MiB), so the range's floor follows the ring once eviction has moved it past the window. | -| `$residentRanges` | where the loopback segment cache holds picture right now, as disjoint ascending spans on the `currentTime` axis. This is the cache's own truth, not AVPlayer's `loadedTimeRanges`: a measured session held 64 segments over four minutes across several islands while AVPlayer exposed roughly twelve seconds around the playhead and forgot a seeked-ahead island as soon as the playhead left it. Empty is the nil-equivalent, and a live session publishes empty always (its rewind depth is `seekableLiveRange`, which answers a different question). Residency is not a promise that a seek inside a span is instant: the player may still re-anchor and decode at the target, and a segment can start mid-GOP (AE#412). Coalesced to at most four updates a second, cleared on `load()` and teardown. | +| `$residentRanges` | where the loopback segment cache or software VOD packet cache holds picture right now, as disjoint ascending spans on the `currentTime` axis. This is the cache's own truth, not AVPlayer's `loadedTimeRanges`: a measured session held 64 segments over four minutes across several islands while AVPlayer exposed roughly twelve seconds around the playhead and forgot a seeked-ahead island as soon as the playhead left it. Empty is the nil-equivalent, and a live session publishes empty always (its rewind depth is `seekableLiveRange`, which answers a different question). Residency is not a promise that a seek inside a span is instant: the player may still re-anchor and decode at the target, and a segment can start mid-GOP (AE#412). Coalesced to at most four updates a second, cleared on `load()` and teardown. | | `player.currentTime`, `sourceTime`, `sourceTimeFollowsPicture`, `progress`, `bufferedPosition`, `liveEdgeTime`, `seekableLiveRange`, `behindLiveSeconds`, `isAtLiveEdge` | non-published mirrors of the same values for one-shot reads | | `$duration` | seconds; a `LoadOptions.declaredDurationSeconds` outranks the container's | +For software VOD, `residentRanges` intersects video presentation coverage with the selected audio +stream, even if muted; a video-only source does not require audio. H.264/HEVC use verified +successor-PTS intervals, so the final picture with no known successor does not acquire an invented +end. Ranges are clipped against retained keyframes and discarded packet timestamps; unknown or +invalid coverage stays empty. Metadata limits can conservatively omit old spans rather than claim +packets that no longer exist. + +The software sampler computes an immutable snapshot in the background and publishes on the main +actor, including while paused. Cached seeks preserve retained spans; a cache miss clears them when +the old store is discarded. Seek, replacement and teardown guards reject a late snapshot. Display +these spans directly on the `currentTime` axis: `bufferedPosition` alone is a forward frontier and +cannot establish a backward cache start. Live playback continues to expose its separate seekable +window rather than VOD resident ranges. + + **`sourceTime` on `nativeRemoteHLS` (AE#616).** The engine sees no segment on that route, so the clock is AVPlayer's item time. An origin whose playlist places a segment at its slot while the segment starts at the keyframe before it (a Jellyfin transcode restarted by `-ss -noaccurate_seek -copyts`) makes @@ -1128,7 +1143,7 @@ All flags default to safe values; the table is the full set. Depth for the media | `deinterlaceFieldRate` | `.field` | A `DeinterlaceFieldRate`: the hardware deinterlacer emits one frame per field (25i to 50p) or per frame. The software fallback is always frame rate, because doubling a CPU bwdif is the wrong trade and a fallback should not change cost class. | | `probesize`, `maxAnalyzeDuration` | nil | Caller-bounded open-time probe budget (defaults 50 MB / 60 s). They fail **open**: an over-tight budget loads with late-resolving tracks silently missing rather than throwing, so validate track presence if you tighten them. Do not pass `0` for `maxAnalyzeDuration`; FFmpeg maps it to a shorter heuristic. | | `progressiveSegmentDelivery` | false | Serve each VOD loopback segment while it is being written: the muxer flushes a fragment about every half second and the loopback server sends each one as it lands, so AVPlayer shows and starts on a segment's first fragments instead of waiting for its cut. Changes nothing on a source fast enough to cut segments before AVPlayer asks; on a slow link it removes a whole segment's download from startup and from a seek outside the buffer. A segment the producer abandons (restart, failed cut) ends its connection without the final chunk, so AVPlayer asks for it again. VOD only; ignored on live, `nativeRemoteHLS` and the software path. | -| `forwardBufferSegments` | nil (10, about 40 s) | How far the producer may race ahead and how much the cache keeps resident. Clamped to 4...2700; past the historical 150 the real bound is the session's disk budget, so a "buffer without limit" option can pass `Int.max`. Ignored on `nativeRemoteHLS`. Native HLS retention is readable as `$residentRanges` (see Time). Software VOD uses the same forward-window and volume-budget policy for compressed packet read-ahead; its continuous selected A/V frontier is `bufferedPosition`, not native `residentRanges`. | +| `forwardBufferSegments` | nil (10, about 40 s) | How far the producer may race ahead and how much the cache keeps resident. Clamped to 4...2700; past the historical 150 the real bound is the session's disk budget, so a "buffer without limit" option can pass `Int.max`. Ignored on `nativeRemoteHLS`. Native HLS retention is readable as `$residentRanges` (see Time). Software VOD uses the same forward-window and volume-budget policy for compressed packet read-ahead; its continuous selected A/V frontier is `bufferedPosition`, and its retained spans (including history and gaps) are `$residentRanges`. | | `sequentialOrigin` | false | Declare an origin that fabricates range answers: one long-lived unranged GET, no ranged probes, non-seekable pb. **Seeking is unavailable**; re-request the archive at a shifted start instead. A URL source the reader finds forward-only by itself (`Range` ignored, so it can only stream front to back; the open confirms a 200 answer with a one byte range, because some CDNs answer a cache miss that way) is served this way without the declaration when its container states a duration, which a stated `Content-Length` lets an MPEG-TS source estimate; `loadedOptions` then reads `true` with that duration declared, and without a duration it stays on the software path. An AirPlay hop on such a session swaps the item onto the LAN address instead of reopening the source, since a reopen could only start again from byte 0. | | `declaredDurationSeconds` | nil | Trusted duration, overriding the container's. Required alongside `sequentialOrigin` on VOD, where the tail read is gone. | | `maxConcurrentSourceRequests` | nil | Most requests the reader may have open against this origin at once, across every path it fetches on (pump ranges, detour blocks, size probes, tail prefetch, subtitle side reader, and the HLS live ingest's playlist, segment and key fetches, whose prefetch window narrows to it, AE#678). nil counts without capping and lowers the ceiling on its own after a 429/503/509. Set it when the provider states a limit; `1` also switches off the speculative parallel paths, which exist only to overlap with the pump. Counts **requests**, not TCP connections, because over HTTP/2 a session multiplexes every request onto one connection while the origin still counts each one (AE#377). It is also the only ceiling: several engines playing from one origin are bounded by this value and by what the origin refuses, not by a transport pool underneath it (AE#450). | From 761b031bcf406ada527d82fad251a25bb2415567 Mon Sep 17 00:00:00 2001 From: orut34iop Date: Tue, 13 Oct 2026 15:43:56 +0800 Subject: [PATCH 003/132] fix(cache): use upstream blocking executor for residency sampling --- Sources/AetherEngine/Native/SoftwarePlaybackHost.swift | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Sources/AetherEngine/Native/SoftwarePlaybackHost.swift b/Sources/AetherEngine/Native/SoftwarePlaybackHost.swift index 58199214d..82f584abd 100644 --- a/Sources/AetherEngine/Native/SoftwarePlaybackHost.swift +++ b/Sources/AetherEngine/Native/SoftwarePlaybackHost.swift @@ -3630,7 +3630,7 @@ final class SoftwareResidentRangePublication { guard task == nil else { return nil } let token = revision task = Task { [weak self] in - let ranges = await Task.detached(priority: .utility, operation: read).value + let ranges = await BlockingWork.detached(priority: .utility, operation: read).value guard let self, !Task.isCancelled, self.revision == token else { return } self.task = nil publish(ranges) From 4a5eae0eefb750923355152f9326df33f5f94f08 Mon Sep 17 00:00:00 2001 From: orut34iop Date: Tue, 13 Oct 2026 17:08:55 +0800 Subject: [PATCH 004/132] test(cache): guard residency sampling executor --- .../SoftwareResidentRangesTests.swift | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/Tests/AetherEngineTests/SoftwareResidentRangesTests.swift b/Tests/AetherEngineTests/SoftwareResidentRangesTests.swift index 62e0573aa..8cd47c854 100644 --- a/Tests/AetherEngineTests/SoftwareResidentRangesTests.swift +++ b/Tests/AetherEngineTests/SoftwareResidentRangesTests.swift @@ -160,6 +160,25 @@ struct SoftwareResidentRangesTests { @MainActor @Suite("Software resident range publication", .timeLimit(.minutes(2))) struct SoftwareResidentRangePublicationTests { + private nonisolated static func isMainThread() -> Bool { + Thread.isMainThread + } + + @Test("Residency sampling leaves both main and the cooperative executor") + @available(visionOS 2, *) + func samplingExecutor() async { + let publisher = SoftwareResidentRangePublication() + var received: [ClosedRange] = [] + let work = publisher.sample { + let label = String(cString: __dispatch_queue_get_label(nil)) + #expect(!Self.isMainThread()) + #expect(!label.contains("cooperative"), "residency read ran on \(label)") + return [1...2] + } publish: { received = $0 } + await work?.value + #expect(received == [1...2]) + } + private actor Gate { private(set) var entered = false private var waiter: CheckedContinuation? From 02e1f022473722e53fabc4f7e65a330eb8ab2754 Mon Sep 17 00:00:00 2001 From: orut34iop Date: Tue, 13 Oct 2026 17:56:40 +0800 Subject: [PATCH 005/132] fix(demux): repair variable-cadence H264 heads and hierarchical partial offsets --- CHANGELOG.md | 6 ++ .../test-h264-partial-composition-controls.sh | 14 ++++ ...4PartialCompositionRuntimeStandalone.swift | 36 +++++++- .../H264PartialCompositionStandalone.swift | 66 +++++++++++++++ Scripts/tests/make-missing-head-fixture.py | 77 +++++++++++++++++ .../Video/H264CompositionOffsetRepair.swift | 22 +++++ .../Video/H264PartialCompositionRepair.swift | 80 ++++++++++++++++-- .../H264PartialCompositionRepairSession.swift | 83 ++++++++++++++++--- .../H264CompositionOffsetRepairTests.swift | 34 ++++++++ docs/formats.md | 25 ++++-- 10 files changed, 412 insertions(+), 31 deletions(-) create mode 100644 Scripts/tests/make-missing-head-fixture.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 751e60c90..a118685f0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,6 +21,12 @@ the public-API contract. ### Fixed +- **H.264 MP4s with missing composition offsets keep display order across cadence changes and + hierarchical B pictures.** A corroborated missing-offset head with a changing decode cadence now + reuses the existing timestamp slots. Later zero-offset regions can prove a forward-coded anchor + farther ahead than the declared backward reorder delay, using a bounded complete-prefix check. + Original DTS, audio, payload and index remain unchanged; unsupported sequences still pass through. + - **A host with its own Objective-C and C++ links again (#752).** Since 7.34.0 LibTrueHD and LibDovi each brought a `rust_eh_personality`, and with the host's `__objc_personality_v0` and `__gxx_personality_v0` that made four personality routines, one more than compact unwind encodes diff --git a/Scripts/test-h264-partial-composition-controls.sh b/Scripts/test-h264-partial-composition-controls.sh index 9e2c82177..4c9bdf6b0 100644 --- a/Scripts/test-h264-partial-composition-controls.sh +++ b/Scripts/test-h264-partial-composition-controls.sh @@ -32,3 +32,17 @@ ffmpeg -hide_banner -loglevel error -n \ ruby "$TASK_ROOT/Scripts/tests/make-partial-ctts-fixture.rb" \ "$TASK_TMP/healthy-scenecut.mp4" "$TASK_TMP/variable.mp4" tail 300 1200 1200 bash "$TASK_ROOT/Scripts/test-h264-partial-composition-runtime.sh" "$TASK_TMP/variable.mp4" 0 12 0 + +# Entire missing-offset head with a decode cadence change, plus its untouched healthy control. +python3 "$TASK_ROOT/Scripts/tests/make-missing-head-fixture.py" "$TASK_TMP" +bash "$TASK_ROOT/Scripts/test-h264-partial-composition-runtime.sh" "$TASK_TMP/missing-head.mp4" 0 1.5 0 20 40 +bash "$TASK_ROOT/Scripts/test-h264-partial-composition-runtime.sh" "$TASK_TMP/healthy-head.mp4" 0 + +# Hierarchical B pictures: forward distance 3, declared backward depth 2. The healthy twin +# supplies an exact PTS oracle for both a damaged tail and a damaged middle with recovery. +python3 "$TASK_ROOT/Scripts/tests/make-missing-head-fixture.py" "$TASK_TMP" --partial-hierarchical +AETHER_TIMESTAMP_HEALTHY_TWIN="$TASK_TMP/healthy-b3.mp4" \ + bash "$TASK_ROOT/Scripts/test-h264-partial-composition-runtime.sh" "$TASK_TMP/tail-b3.mp4" 0 12 0 6 22 +AETHER_TIMESTAMP_HEALTHY_TWIN="$TASK_TMP/healthy-b3.mp4" \ + bash "$TASK_ROOT/Scripts/test-h264-partial-composition-runtime.sh" "$TASK_TMP/middle-b3.mp4" 0 12 0 6 18 22 +bash "$TASK_ROOT/Scripts/test-h264-partial-composition-runtime.sh" "$TASK_TMP/healthy-b3.mp4" 0 diff --git a/Scripts/tests/H264PartialCompositionRuntimeStandalone.swift b/Scripts/tests/H264PartialCompositionRuntimeStandalone.swift index c11ea29ed..bec0e4c3b 100644 --- a/Scripts/tests/H264PartialCompositionRuntimeStandalone.swift +++ b/Scripts/tests/H264PartialCompositionRuntimeStandalone.swift @@ -54,6 +54,7 @@ struct H264PartialCompositionRuntimeTests { enum Failure: Error { case open, decoder, demux, decode, session, boundedRead } static func main() throws { guard CommandLine.arguments.count >= 2 else { throw Failure.open } + EngineLog.handler = { FileHandle.standardError.write(Data(($0 + "\n").utf8)) } av_log_set_level(AV_LOG_QUIET) var format: UnsafeMutablePointer? guard avformat_open_input(&format, CommandLine.arguments[1], nil, nil) >= 0, let format else { throw Failure.open } @@ -62,6 +63,7 @@ struct H264PartialCompositionRuntimeTests { let index = av_find_best_stream(format, AVMEDIA_TYPE_VIDEO, -1, -1, nil, 0) guard index >= 0, let stream = format.pointee.streams[Int(index)], let par = stream.pointee.codecpar, par.pointee.codec_id == AV_CODEC_ID_H264 else { throw Failure.open } + let twinPTS = try ProcessInfo.processInfo.environment["AETHER_TIMESTAMP_HEALTHY_TWIN"].map(referenceTimes) let name = format.pointee.iformat.flatMap { $0.pointee.name }.map(String.init(cString:)) ?? "" let matroska = name.contains("matroska") guard !matroska else { throw Failure.session } @@ -96,6 +98,9 @@ struct H264PartialCompositionRuntimeTests { precondition(actual.duration == expected.duration && actual.flags == expected.flags) precondition(actual.position == expected.position && actual.streamIndex == expected.streamIndex) precondition(actual.dts == expected.dts, "partial repair must preserve the published decode/index axis") + if let twinPTS { + precondition(twinPTS[actual.dts] == actual.pts, "repaired video PTS must exactly match its healthy twin") + } try decode(fixed, packet: packet, into: &fixedPTS) } else { precondition(actual == expected) } delivered += 1 @@ -120,7 +125,8 @@ struct H264PartialCompositionRuntimeTests { try decode(raw, packet: packet, into: &rawPTS) } if !session.ingest(packet) { try emit(packet) } - observedRepair = observedRepair || session.summary.contains("confirmed_partial_composition_offsets") + observedRepair = observedRepair || (session.summary.contains("confirmed_partial_composition_offsets") || session.summary.contains("confirmed_missing_composition_offsets")) + } session.endOfStream() while let packet = session.dequeue() { try emit(packet) } @@ -130,7 +136,7 @@ struct H264PartialCompositionRuntimeTests { let rawRegressions = zip(rawPTS, rawPTS.dropFirst()).filter { $1 <= $0 }.count let fixedRegressions = zip(fixedPTS, fixedPTS.dropFirst()).filter { $1 <= $0 }.count let diagnostic = session.summary - observedRepair = observedRepair || diagnostic.contains("confirmed_partial_composition_offsets") + observedRepair = observedRepair || (diagnostic.contains("confirmed_partial_composition_offsets") || diagnostic.contains("confirmed_missing_composition_offsets")) FileHandle.standardError.write(Data("MEASURE seek=\(position) raw=\(rawRegressions) fixed=\(fixedRegressions) offset=\(session.decodeTimestampOffset ?? 0) state=\(diagnostic) lead=\(0) shift=\(0)\n".utf8)) precondition(rawPTS.count == fixedPTS.count && !fixedPTS.isEmpty) precondition(fixedRegressions == 0) @@ -145,6 +151,29 @@ struct H264PartialCompositionRuntimeTests { } } + /// Optional generated healthy twin checks exact timestamps, beyond monotonic decoded output. + static func referenceTimes(_ path: String) throws -> [Int64: Int64] { + var input: UnsafeMutablePointer? + guard avformat_open_input(&input, path, nil, nil) >= 0, let input else { throw Failure.open } + defer { var owned: UnsafeMutablePointer? = input; avformat_close_input(&owned) } + guard avformat_find_stream_info(input, nil) >= 0 else { throw Failure.open } + let index = av_find_best_stream(input, AVMEDIA_TYPE_VIDEO, -1, -1, nil, 0) + guard index >= 0 else { throw Failure.open } + var result: [Int64: Int64] = [:] + while true { + guard let packet = trackedPacketAlloc() else { throw Failure.demux } + defer { av_packet_free_safe(packet) } + let status = av_read_frame(input, packet) + if status == -541478725 { break } + guard status >= 0 else { throw Failure.demux } + if packet.pointee.stream_index == index { + precondition(result.count < 8000 && result[packet.pointee.dts] == nil) + result[packet.pointee.dts] = packet.pointee.pts + } + } + return result + } + static func lifecycleChecks(session: H264CompositionOffsetRepairSession, format: UnsafeMutablePointer, stream: UnsafeMutablePointer, index: Int32, position: Double) throws { @@ -172,7 +201,7 @@ struct H264PartialCompositionRuntimeTests { var confirmed = false for _ in 0..<4000 { try read() - if session.summary.contains("confirmed_partial_composition_offsets") { confirmed = true; break } + if (session.summary.contains("confirmed_partial_composition_offsets") || session.summary.contains("confirmed_missing_composition_offsets")) { confirmed = true; break } } precondition(confirmed) guard let packet = trackedPacketAlloc() else { throw Failure.demux } @@ -181,6 +210,7 @@ struct H264PartialCompositionRuntimeTests { packet.pointee.pts = malformed.pts if !session.ingest(packet) { av_packet_free_safe(packet) } while let packet = session.dequeue() { av_packet_free_safe(packet) } + precondition(session.summary.contains("partial_composition_sequence_unproven")) precondition(PacketBalanceTracker.alive == 0, "a refusal still owns every packet it took") // A refused sequence streams through the session rather than out of its queue, so // count both ways a packet can come back. diff --git a/Scripts/tests/H264PartialCompositionStandalone.swift b/Scripts/tests/H264PartialCompositionStandalone.swift index bb914f09f..f7088400c 100644 --- a/Scripts/tests/H264PartialCompositionStandalone.swift +++ b/Scripts/tests/H264PartialCompositionStandalone.swift @@ -32,7 +32,73 @@ struct H264PartialCompositionTests { return (times, deepest, !queue.isEmpty) } + static func missingHeadChecks() { + let ticks: [Int64] = [-48000, 0, 48000, 96000, 144000, 192000, 240000, 288000, 336000, 384000, 424040, 464080] + let order: [Int64] = [0, 6, 2, 4, 12, 8, 10, 18, 14, 16, 22, 20] + let keys = [true] + Array(repeating: false, count: 11) + func proof(_ dts: [Int64] = ticks, pts: [Int64]? = nil, pocs: [Int64] = order, + start: Int64 = 0, origin: Int64 = -48000, depth: Int = 1) -> Int? { + Policy.missingHeadForwardDepth(dts: dts, pts: pts ?? dts, pocs: pocs, keys: keys, + streamStart: start, ladderStart: origin, reorderDepth: depth) + } + precondition(proof() == 2) // Forward-coded P differs from backward reorder lead. + precondition(proof(start: -48000) == nil) + precondition(proof(start: 1) == nil) + precondition(proof(origin: 0) == nil) + precondition(proof(depth: 0) == nil) + precondition(proof(depth: 17) == nil) + precondition(proof(pocs: Array(stride(from: Int64(0), to: 24, by: 2))) == nil) + for bad: Int64 in [-1, 3, 6, 1000, Int64.max] { + var changed = order; changed[4] = bad + precondition(proof(pocs: changed) == nil) + } + var offset = ticks; offset[3] += 1 + precondition(proof(pts: offset) == nil) + var flat = ticks; flat[3] = flat[2] + precondition(proof(flat) == nil) + var missing = ticks; missing[3] = Int64.min + precondition(proof(missing) == nil) + precondition(proof(Array(ticks.prefix(11))) == nil) + print("PASS missing head: corroborated lead, variable slots, bounded forward reorder; invalid/ambiguous evidence refused") + } + + static func forwardProofChecks() { + let ranks = [0, 4, 2, 1, 3, 8, 6, 5, 7, 12, 10, 9] + let dts = (0..<12).map { Int64($0) * 3600 } + func proof(_ order: [Int] = ranks, slots: [Int64] = dts, lead: Int64 = 7200, + depth: Int = 2) -> Int? { + Policy.provenForwardDepth(dts: slots, ranks: order, lead: lead, reorderDepth: depth) + } + precondition(proof() == 3) + precondition(proof(slots: dts.map { $0 + 63_000_000 }) == 3) + precondition(proof(lead: 3600) == nil) + precondition(proof(depth: 1) == nil) + precondition(proof(Array(ranks.prefix(11)), slots: Array(dts.prefix(11))) == nil) + precondition(proof(Array(0..<12)) == nil) + for bad in [-1, 0, 1000, Int.max] { + var order = ranks; order[4] = bad + precondition(proof(order) == nil) + } + // Unique bounded ranks can still omit a display slot: no complete prefix, no proof. + precondition(proof([0, 5, 3, 2, 4, 9, 7, 6, 8, 13, 11, 10]) == nil) + var repeated = dts; repeated[4] = repeated[3] + precondition(proof(slots: repeated) == nil) + var missing = dts; missing[4] = Int64.min + precondition(proof(slots: missing) == nil) + precondition(proof(slots: dts.map { Int64.max - 40000 + $0 }) == nil) + var ladder = Policy.SlotLadder() + for value in dts { precondition(ladder.append(dts: value)) } + precondition(ladder.claim(rank: 4, decodeIndex: 1, lead: 7200, reorderDepth: 2) == .refuse) + precondition(ladder.refusalReason == "forward_wait_depth_exceeded") + precondition(ladder.claim(rank: 4, decodeIndex: 1, lead: 7200, reorderDepth: 2, forwardDepth: 3) == .time(21600)) + precondition(ladder.claim(rank: 4, decodeIndex: 1, lead: 7200, reorderDepth: 2, forwardDepth: 3) == .refuse) + precondition(ladder.refusalReason == "duplicate_display_rank") + print("PASS partial forward proof: complete prefix, asymmetric depth, exact slot, invalid evidence refused") + } + static func main() { + missingHeadChecks() + forwardProofChecks() let expected = pocs.map { slots[Int($0 / 2)] + lead } let plain = run() precondition(!plain.refused) diff --git a/Scripts/tests/make-missing-head-fixture.py b/Scripts/tests/make-missing-head-fixture.py new file mode 100644 index 000000000..a1546ee41 --- /dev/null +++ b/Scripts/tests/make-missing-head-fixture.py @@ -0,0 +1,77 @@ +#!/usr/bin/env python3 +"""Generate a public variable-cadence H.264 twin, never consume real media. + +Run the timestamp runtime on missing-head.mp4 at 0 1.5 0 20 40, and on +healthy-head.mp4 at 0. The former must remove frame-PTS regressions while +preserving DTS/audio/payload; the latter must remain byte-for-byte unchanged. +This fixture proves display ordering, not recovery of an arbitrary writer's +lost original composition timestamps or physical A/V synchronization. +""" +import argparse +from pathlib import Path +import struct +import subprocess + +parser = argparse.ArgumentParser(description=__doc__) +parser.add_argument('output', type=Path) +parser.add_argument('--partial-hierarchical', action='store_true', help='Generate healthy/tail/middle B3 twins for the partial-region path') +args = parser.parse_args() +args.output.mkdir(parents=True, exist_ok=True) +if args.partial_hierarchical: + healthy = args.output / 'healthy-b3.mp4' + subprocess.run([ + 'ffmpeg', '-hide_banner', '-loglevel', 'error', '-n', + '-f', 'lavfi', '-i', 'color=c=blue:s=96x64:r=25:d=24', + '-f', 'lavfi', '-i', 'sine=frequency=440:sample_rate=48000:duration=24', + '-c:v', 'libx264', '-preset', 'fast', '-pix_fmt', 'yuv420p', '-g', '49', '-bf', '3', + '-x264-params', 'scenecut=0:b-adapt=0', '-c:a', 'aac', '-video_track_timescale', '90000', + '-metadata', 'comment=Aether synthetic timestamp fixture', '-movflags', '+faststart', str(healthy), + ], check=True) + # 49-picture GOPs align the IDR boundaries with ctts runs; mutate offsets only. + for mode in ('tail', 'middle'): + subprocess.run(['ruby', str(Path(__file__).with_name('make-partial-ctts-fixture.rb')), + str(healthy), str(args.output / (mode + '-b3.mp4')), mode, '196', '392', '600'], check=True) + raise SystemExit(0) + +healthy = args.output / 'healthy-head.mp4' +missing = args.output / 'missing-head.mp4' +if healthy.exists() or missing.exists(): + raise SystemExit('Refusing to replace an existing fixture') +subprocess.run([ + 'ffmpeg', '-hide_banner', '-loglevel', 'error', '-n', + '-f', 'lavfi', '-i', 'testsrc2=s=96x64:r=25:d=60', + '-f', 'lavfi', '-i', 'sine=frequency=440:sample_rate=48000:duration=60', + '-vf', "settb=1/1200000,setpts='if(lt(N,8),N*48000,384000+(N-8)*40040)'", + '-fps_mode', 'passthrough', '-enc_time_base', '1/1200000', + '-c:v', 'libx264', '-preset', 'fast', '-pix_fmt', 'yuv420p', + '-g', '60', '-bf', '2', '-x264-params', 'scenecut=0:b-adapt=0:b-pyramid=none', + '-c:a', 'aac', '-video_track_timescale', '1200000', + '-metadata', 'comment=Aether synthetic timestamp fixture', + '-movflags', '+faststart', str(healthy), +], check=True) +data = bytearray(healthy.read_bytes()) +changed = 0 + + +def zero_offsets(start, end): + global changed + while start < end: + size, kind = struct.unpack_from('>I4s', data, start) + assert size >= 8 and start + size <= end + if kind in (b'moov', b'trak', b'mdia', b'minf', b'stbl'): + zero_offsets(start + 8, start + size) + elif kind == b'ctts': + count = struct.unpack_from('>I', data, start + 12)[0] + assert size == 16 + 8 * count + for index in range(count): + offset = start + 16 + 8 * index + changed += struct.unpack_from('>I', data, offset)[0] + struct.pack_into('>I', data, offset + 4, 0) + start += size + + +zero_offsets(0, len(data)) +assert changed == 1500 +with missing.open('xb') as output: + output.write(data) +print('PASS generated missing-head twin: 1500 frames, retained edit/index/audio/payload') diff --git a/Sources/AetherEngine/Video/H264CompositionOffsetRepair.swift b/Sources/AetherEngine/Video/H264CompositionOffsetRepair.swift index f718bd9d2..1f0329007 100644 --- a/Sources/AetherEngine/Video/H264CompositionOffsetRepair.swift +++ b/Sources/AetherEngine/Video/H264CompositionOffsetRepair.swift @@ -622,6 +622,7 @@ final class H264CompositionOffsetRepairSession { private let partialRepairCandidate: H264PartialCompositionRepairSession? private var monitorsPartialOffsets = false private var samples: [H264CompositionOffsetRepair.Sample] = [] + private var sampledFramePictures = true private var held: [(packet: UnsafeMutablePointer, pictureOrderCount: Int64?)] = [] private var heldBytes = 0 private var verdictDescription = "sampling" @@ -701,6 +702,7 @@ final class H264CompositionOffsetRepairSession { var pictureOrderCount: Int64? if packet.pointee.stream_index == streamIndex { pictureOrderCount = reader?.pictureOrderCount(for: packet) + sampledFramePictures = sampledFramePictures && reader?.isFramePicture == true samples.append( H264CompositionOffsetRepair.Sample( dts: packet.pointee.dts, @@ -801,6 +803,7 @@ final class H264CompositionOffsetRepairSession { case .sampling: // The sample restarts where the source now stands. samples.removeAll(keepingCapacity: true) + sampledFramePictures = true case .repairing: rewriter?.noteSeek() reader?.reset() @@ -882,6 +885,25 @@ final class H264CompositionOffsetRepairSession { verdictDescription = "healthy" disarm() case .inconclusive(let reason): + if reason == "decode ladder is not uniform", sampledFramePictures, + let candidate = partialRepairCandidate, + let forwardDepth = H264PartialCompositionRepair.missingHeadForwardDepth( + dts: samples.map(\.dts), pts: samples.map(\.pts), + pocs: samples.map(\.pictureOrderCount), keys: samples.map(\.isKeyframe), + streamStart: streamStartTime, ladderStart: ladderStart, reorderDepth: videoDelay), + let firstPacket = held.first(where: { $0.packet.pointee.stream_index == streamIndex })?.packet, + candidate.enableMissingHead(forwardDepth: forwardDepth, firstPacket: firstPacket) { + // Reuse #513's streaming slots. Hand over each interleaved packet once; + // leading non-video packets remain ahead of the candidate's queue. + monitorsPartialOffsets = true + disarm() + let input = held + held.removeAll(keepingCapacity: false) + heldBytes = 0 + for entry in input where !candidate.ingest(entry.packet) { held.append(entry) } + samples.removeAll(keepingCapacity: false) + return + } verdictDescription = "inconclusive (\(reason))" // Only worth a line when the stream looked like a candidate: a file that simply carries // composition offsets is the normal case and says nothing. diff --git a/Sources/AetherEngine/Video/H264PartialCompositionRepair.swift b/Sources/AetherEngine/Video/H264PartialCompositionRepair.swift index 6f5f887c9..4ed6cb1e0 100644 --- a/Sources/AetherEngine/Video/H264PartialCompositionRepair.swift +++ b/Sources/AetherEngine/Video/H264PartialCompositionRepair.swift @@ -9,8 +9,8 @@ import Foundation /// slot waits exactly the mini-GOP the reorder created, so nothing is held for the length of a /// sequence and no sequence is too long to repair. enum H264PartialCompositionRepair { - /// A picture cannot sit further from its own slot than the reorder delay the container - /// declares. `video_delay` is the honest bound; this is only the ceiling on believing it. + /// Backward displacement must fit the declared reorder delay. Forward-coded anchors may + /// be further ahead (#700); a separate bounded prefix proof must establish that distance. static let maximumReorderDepth = 16 /// A sequence past this is not a GOP any more, and the ladder is one Int64 per picture. static let maximumPictures = 1 << 16 @@ -23,6 +23,59 @@ enum H264PartialCompositionRepair { /// that reorders at all shows it within its reorder delay; this is slack, not a policy. static let maximumProofPictures = 64 + /// A missing-offset head has no healthy packet to corroborate its lead. Require + /// the container start/index distance to equal the observed reorder head, then + /// dry-run the existing slots. Never fit an average rate to variable timestamps. + static func missingHeadForwardDepth( + dts: [Int64], pts: [Int64], pocs: [Int64], keys: [Bool], + streamStart: Int64, ladderStart: Int64, reorderDepth: Int + ) -> Int? { + guard dts.count >= 12, dts.count <= maximumProofPictures, + pts.count == dts.count, pocs.count == dts.count, keys.count == dts.count, + (1...maximumReorderDepth).contains(reorderDepth), reorderDepth < dts.count, + streamStart != Int64.min, ladderStart != Int64.min, + dts.first == ladderStart, keys.first == true, pocs.first == 0, + !keys.dropFirst().contains(true), pts == dts else { return nil } + let (lead, overflow) = streamStart.subtractingReportingOverflow(ladderStart) + let (observed, headOverflow) = dts[reorderDepth].subtractingReportingOverflow(ladderStart) + guard !overflow, !headOverflow, lead > 0, lead == observed else { return nil } + var ranks: [Int] = [] + for poc in pocs { + guard poc >= 0, poc % 2 == 0, poc / 2 < Int64(maximumPictures) else { return nil } + ranks.append(Int(poc / 2)) + } + return provenForwardDepth(dts: dts, ranks: ranks, lead: lead, reorderDepth: reorderDepth) + } + + /// Reuse #700's closed decode-prefix evidence for #513's original timestamp slots. + /// Uniqueness plus [0...prefix.count-1] coverage proves a complete mini-GOP prefix; + /// the declared backward depth and every observed slot/lead still have to agree. + static func provenForwardDepth(dts: [Int64], ranks: [Int], lead: Int64, + reorderDepth: Int) -> Int? { + guard (12...maximumProofPictures).contains(dts.count), ranks.count == dts.count, + ranks.first == 0, lead > 0, (1...maximumReorderDepth).contains(reorderDepth), + ranks.allSatisfy({ $0 >= 0 && $0 < maximumPictures }), + Set(ranks).count == ranks.count, + zip(ranks, ranks.dropFirst()).contains(where: { $1 < $0 }) else { return nil } + var ladder = SlotLadder() + for timestamp in dts { guard ladder.append(dts: timestamp) else { return nil } } + var high = 0 + var closedPrefix = false + var forward = reorderDepth + for (index, rank) in ranks.enumerated() { + high = max(high, rank) + if index >= 8, high == index { closedPrefix = true } + guard index - rank <= reorderDepth, rank - index <= maximumReorderDepth else { return nil } + forward = max(forward, rank - index) + } + guard closedPrefix else { return nil } + for (index, rank) in ranks.enumerated() { + if ladder.claim(rank: rank, decodeIndex: index, lead: lead, + reorderDepth: reorderDepth, forwardDepth: forward) == .refuse { return nil } + } + return forward + } + enum Claim: Equatable { /// The slot is still ahead of the reader. The picture waits, and so does everything /// behind it, because the container's own order is what leaves this demuxer. @@ -36,6 +89,7 @@ enum H264PartialCompositionRepair { struct SlotLadder { private var slots: [Int64] = [] private var claimed: [Bool] = [] + private(set) var refusalReason = "none" var count: Int { slots.count } var isEmpty: Bool { slots.isEmpty } @@ -52,14 +106,26 @@ enum H264PartialCompositionRepair { /// The presentation time the picture at `decodeIndex` owns, given its display `rank`. /// `.wait` while its slot is still ahead of the reader, which is the mini-GOP. - mutating func claim(rank: Int, decodeIndex: Int, lead: Int64, reorderDepth: Int) -> Claim { + mutating func claim(rank: Int, decodeIndex: Int, lead: Int64, reorderDepth: Int, + forwardDepth: Int? = nil) -> Claim { + refusalReason = "none" + let forward = forwardDepth ?? reorderDepth guard lead > 0, rank >= 0, decodeIndex >= 0, decodeIndex < slots.count, - abs(rank - decodeIndex) <= reorderDepth, rank < Self.limit else { return .refuse } + (1...H264PartialCompositionRepair.maximumReorderDepth).contains(forward), + rank < Self.limit else { refusalReason = "invalid_slot_claim"; return .refuse } + guard decodeIndex - rank <= reorderDepth else { + refusalReason = "backward_reorder_depth_exceeded"; return .refuse + } + guard rank - decodeIndex <= forward else { + refusalReason = "forward_wait_depth_exceeded"; return .refuse + } guard rank < slots.count else { return .wait } - // One slot, one picture. A rank claimed twice is a sequence this policy cannot own. - guard !claimed[rank] else { return .refuse } + guard !claimed[rank] else { refusalReason = "duplicate_display_rank"; return .refuse } let (pts, overflow) = slots[rank].addingReportingOverflow(lead) - guard !overflow, pts >= slots[decodeIndex] else { return .refuse } + guard !overflow else { refusalReason = "presentation_time_overflow"; return .refuse } + guard pts >= slots[decodeIndex] else { + refusalReason = "presentation_before_decode"; return .refuse + } claimed[rank] = true return .time(pts) } diff --git a/Sources/AetherEngine/Video/H264PartialCompositionRepairSession.swift b/Sources/AetherEngine/Video/H264PartialCompositionRepairSession.swift index b69941ec3..825d831be 100644 --- a/Sources/AetherEngine/Video/H264PartialCompositionRepairSession.swift +++ b/Sources/AetherEngine/Video/H264PartialCompositionRepairSession.swift @@ -24,6 +24,12 @@ final class H264PartialCompositionRepairSession { private let timeBase: AVRational private let lead: Int64 private let reorderDepth: Int + private var missingHead = false + private var forwardDepth: Int? + // Numeric evidence only, capped independently of interleaved packet/byte budgets. + private var proofDTS: [Int64] = [] + private var proofRanks: [Int] = [] + private var forwardProofs = 0 private let framing: VideoNALFraming private let reader: H264PictureOrderReader private var pending: [Entry] = [] @@ -55,7 +61,21 @@ final class H264PartialCompositionRepairSession { } deinit { releasePackets() } - var hasDecision: Bool { reason != "composition_offsets_present" } + var hasDecision: Bool { missingHead || reason != "composition_offsets_present" } + + /// Only the parent's head proof enables this. Healthy-head behavior is unchanged. + func enableMissingHead(forwardDepth: Int, firstPacket: UnsafeMutablePointer) -> Bool { + precondition(!inSequence && pending.isEmpty && ready.isEmpty) + guard isIDR(firstPacket) else { return false } + missingHead = true + self.forwardDepth = forwardDepth + reason = "missing_composition_offsets_pending" + return true + } + + private var confirmedReason: String { + missingHead ? "confirmed_missing_composition_offsets" : "confirmed_partial_composition_offsets" + } private func isIDR(_ packet: UnsafeMutablePointer) -> Bool { guard packet.pointee.flags & AV_PKT_FLAG_KEY != 0, @@ -96,16 +116,20 @@ final class H264PartialCompositionRepairSession { inSequence = true } if abandoned { return handBack(packet) } - guard !missing, ladder.append(dts: packet.pointee.dts) else { return failOpen(with: packet) } + guard !missing, ladder.append(dts: packet.pointee.dts) else { return failOpen(with: packet, cause: "missing_or_nonincreasing_dts") } let poc = reader.pictureOrderCount(for: packet) // Complete, closed, progressive sequences only. Fields, open-GOP leading pictures and a // parser that cannot read an order are not guessed at. guard let poc, reader.isFramePicture, poc >= 0, poc % 2 == 0, decodeIndex > 0 || poc == 0, packet.pointee.flags & AV_PKT_FLAG_KEY == 0 || (idr && poc == 0) - else { return failOpen(with: packet) } + else { return failOpen(with: packet, cause: "unsupported_picture_order") } let rank = Int(poc / 2) if rank != decodeIndex { sequenceReordered = true } + if !missingHead && proofRanks.count < H264PartialCompositionRepair.maximumProofPictures { + proofDTS.append(packet.pointee.dts) + proofRanks.append(rank) + } append(packet, rank: rank) decodeIndex += 1 drain() @@ -119,21 +143,43 @@ final class H264PartialCompositionRepairSession { guard confirmed || sequenceReordered else { return } while let first = pending.first { guard let rank = first.rank else { promoteFirst(); continue } - switch ladder.claim(rank: rank, decodeIndex: first.decodeIndex, lead: lead, reorderDepth: reorderDepth) { + if !missingHead, forwardDepth == nil, rank - first.decodeIndex > reorderDepth { + // Keep the existing fast path for ordinary mini-GOPs. Only an anchor beyond the + // old symmetric allowance waits for bounded, complete-prefix evidence. + if let depth = H264PartialCompositionRepair.provenForwardDepth( + dts: proofDTS, ranks: proofRanks, lead: lead, reorderDepth: reorderDepth), + rank - first.decodeIndex <= depth { + forwardDepth = depth + forwardProofs += 1 + if forwardProofs <= 8 { + EngineLog.emit("[Demuxer] H264 slot forward wait proven: rank=\(rank)" + + " decode_index=\(first.decodeIndex) samples=\(proofRanks.count)" + + " reorder_depth=\(reorderDepth) forward_depth=\(depth)", category: .demux) + } + } else if proofRanks.count < H264PartialCompositionRepair.maximumProofPictures { + deepestWait = max(deepestWait, pending.count) + return + } else { + failOpen(cause: "forward_prefix_unproven") + return + } + } + switch ladder.claim(rank: rank, decodeIndex: first.decodeIndex, lead: lead, + reorderDepth: reorderDepth, forwardDepth: forwardDepth) { case .wait: deepestWait = max(deepestWait, pending.count) return case .refuse: - failOpen() + failOpen(cause: ladder.refusalReason) return case .time(let pts): first.packet.pointee.pts = pts repairedCount += 1 if !confirmed { confirmed = true - reason = "confirmed_partial_composition_offsets" + reason = confirmedReason EngineLog.emit( - "[Demuxer] partial H264 composition offsets confirmed: reorder_depth=\(reorderDepth)" + "[Demuxer] H264 slot composition offsets confirmed: kind=\(confirmedReason) reorder_depth=\(reorderDepth)" + " presentation_lead=\(lead) decode_offset=0 time_base=\(timeBase.num)/\(timeBase.den)", category: .demux ) @@ -160,18 +206,25 @@ final class H264PartialCompositionRepairSession { if pending.count >= H264PartialCompositionRepair.maximumHeldPackets || bytes >= H264PartialCompositionRepair.maximumHeldBytes || (!confirmed && !sequenceReordered && decodeIndex > H264PartialCompositionRepair.maximumProofPictures) { - failOpen() + failOpen(cause: "packet_byte_or_proof_limit") } } /// Hands every held packet back untouched and lets the rest of this sequence stream through. /// The next IDR starts a new candidate; nothing about the session is given up. @discardableResult - private func failOpen(with packet: UnsafeMutablePointer? = nil) -> Bool { + private func failOpen(with packet: UnsafeMutablePointer? = nil, cause: String) -> Bool { if !abandoned, inSequence { abandoned = true abandonedSequences += 1 - if reason == "confirmed_partial_composition_offsets" { + // One bounded diagnostic per refused sequence; never log payloads or source URLs. + if abandonedSequences <= 8 { + EngineLog.emit("[Demuxer] H264 slot repair refused: cause=\(cause)" + + " rank=\(pending.first?.rank ?? -1) decode_index=\(pending.first?.decodeIndex ?? decodeIndex)" + + " samples=\(decodeIndex) reorder_depth=\(reorderDepth) forward_depth=\(forwardDepth ?? reorderDepth)" + + " presentation_lead=\(lead) held_packets=\(pending.count) held_bytes=\(bytes)", category: .demux) + } + if reason == confirmedReason { reason = "partial_composition_sequence_unproven_after_repair" } else { reason = "partial_composition_sequence_unproven" @@ -192,7 +245,7 @@ final class H264PartialCompositionRepairSession { private func closeSequence() { drain() // Anything still waiting is waiting for a slot this sequence never carried. - if !pending.isEmpty { failOpen() } + if !pending.isEmpty { failOpen(cause: "incomplete_sequence") } resetSequence() } @@ -202,6 +255,9 @@ final class H264PartialCompositionRepairSession { inSequence = false sequenceReordered = false abandoned = false + proofDTS.removeAll(keepingCapacity: true) + proofRanks.removeAll(keepingCapacity: true) + if !missingHead { forwardDepth = nil } } private func publishPending() { @@ -227,7 +283,7 @@ final class H264PartialCompositionRepairSession { reader.reset() resetSequence() confirmed = false - reason = "composition_offsets_present" + reason = missingHead ? "missing_composition_offsets_pending" : "composition_offsets_present" } private func releasePackets() { @@ -239,7 +295,8 @@ final class H264PartialCompositionRepairSession { var summary: String { "reason=\(reason) repaired=\(repairedCount) abandoned_sequences=\(abandonedSequences)" - + " deepest_wait=\(deepestWait) reorder_depth=\(reorderDepth)" + + " deepest_wait=\(deepestWait) reorder_depth=\(reorderDepth) forward_proofs=\(forwardProofs)" + " presentation_lead=\(lead) decode_offset=0" } + } diff --git a/Tests/AetherEngineTests/H264CompositionOffsetRepairTests.swift b/Tests/AetherEngineTests/H264CompositionOffsetRepairTests.swift index d01688edd..9612b7df7 100644 --- a/Tests/AetherEngineTests/H264CompositionOffsetRepairTests.swift +++ b/Tests/AetherEngineTests/H264CompositionOffsetRepairTests.swift @@ -68,6 +68,40 @@ struct H264CompositionOffsetRepairTests { #expect(verdict(samples) == .inconclusive("decode ladder is not uniform")) } + @Test("a hierarchical anchor proves forward distance separately from backward reorder depth") + func partialForwardPrefix() { + let ranks = [0, 4, 2, 1, 3, 8, 6, 5, 7, 12, 10, 9] + let dts = (0..<12).map { Int64($0) * 3600 } + #expect(H264PartialCompositionRepair.provenForwardDepth( + dts: dts, ranks: ranks, lead: 7200, reorderDepth: 2) == 3) + #expect(H264PartialCompositionRepair.provenForwardDepth( + dts: dts, ranks: ranks, lead: 3600, reorderDepth: 2) == nil) + #expect(H264PartialCompositionRepair.provenForwardDepth( + dts: dts, ranks: ranks, lead: 7200, reorderDepth: 1) == nil) + #expect(H264PartialCompositionRepair.provenForwardDepth( + dts: dts, ranks: [0, 5, 3, 2, 4, 9, 7, 6, 8, 13, 11, 10], + lead: 7200, reorderDepth: 2) == nil) + } + + @Test("a variable-cadence missing head requires an exactly corroborated container lead") + func variableCadenceHeadProof() { + let dts: [Int64] = [-96000, -48000, 0, 48000, 96000, 144000, + 192000, 240000, 288000, 328040, 368080, 408120] + let pocs: [Int64] = [0, 6, 2, 4, 12, 8, 10, 18, 14, 16, 24, 20] + let keys = [true] + Array(repeating: false, count: 11) + #expect(H264PartialCompositionRepair.missingHeadForwardDepth( + dts: dts, pts: dts, pocs: pocs, keys: keys, + streamStart: 0, ladderStart: -96000, reorderDepth: 2) == 2) + #expect(H264PartialCompositionRepair.missingHeadForwardDepth( + dts: dts, pts: dts, pocs: pocs, keys: keys, + streamStart: 1, ladderStart: -96000, reorderDepth: 2) == nil) + var healthy = dts + healthy[0] = 0 + #expect(H264PartialCompositionRepair.missingHeadForwardDepth( + dts: dts, pts: healthy, pocs: pocs, keys: keys, + streamStart: 0, ladderStart: -96000, reorderDepth: 2) == nil) + } + @Test("a sample that does not start on a picture-order origin cannot be anchored") func mustStartOnOrigin() { var samples = malformedSamples() diff --git a/docs/formats.md b/docs/formats.md index 9d8f1e5df..b154e40e6 100644 --- a/docs/formats.md +++ b/docs/formats.md @@ -57,10 +57,16 @@ zero). A source is only repaired when every sampled pair is equal, the decode la picture order regresses, and the ranks it produces are distinct and fill the sampled window (or a decode-order prefix of at least nine pictures fills its own range exactly, which covers a hierarchical mini-GOP longer than the reorder delay, #699). Anything -short of that (variable frame timing, a picture order that does not advance one rank per picture, a +short of that (a picture order that does not advance one rank per picture, a sample that starts nowhere it can be anchored) is delivered exactly as the container wrote it. Reported by @orut34iop; the HEVC case (#699) by @ijuniorfu. +For H.264 only, a changing decode cadence can instead use the original-slot policy described below. +This fallback requires a progressive IDR head with equal PTS/DTS, a container start/index lead that +exactly matches the observed reorder head, and a complete, distinct picture-order prefix whose +timestamps pass the slot policy. It assigns existing slots plus that lead rather than fitting a +frame rate; DTS, audio and the keyframe index remain unchanged. An ambiguous head stays untouched. + ### Matroska with presentation slots in coding order Matroska block timestamps are presentation timestamps by specification, and the format has no @@ -104,10 +110,9 @@ offsets. Seeking into that region can produce persistent judder despite normal aggregate FPS and sufficient network buffering. When a healthy origin picture corroborates the container's edit/index lead, the -demuxer keeps a zero-hold healthy path and watches for zero-offset IDRs. One -bounded, complete IDR-to-IDR progressive sequence is parsed for picture order. -If every selected packet has valid equal PTS/DTS and its distinct even POC fills -the complete sequence, a proven permutation assigns its original DTS slots plus +demuxer keeps a zero-hold healthy path and watches for zero-offset IDRs. Progressive +pictures are parsed as they arrive. Valid equal PTS/DTS and distinct even POC let +the streaming slot ladder assign original DTS slots plus the corroborated presentation lead by display rank. Original DTS, audio and the already published keyframe index never move. Actual timestamp slots, rather than an average-FPS clock, preserve interval changes and quantization within a sequence. @@ -115,8 +120,11 @@ an average-FPS clock, preserve interval changes and quantization within a sequen The slot a picture needs is a packet away, not a plan away, exactly as in the Matroska policy above: a picture coded ahead of its own slot waits the mini-GOP the reorder created, so nothing waits for the end of a sequence and no sequence is too long to -repair. The wait is bounded by the reorder delay the container declares, believed up -to **16 pictures**, and the packets held behind it by **1024 interleaved packets and +repair. Backward displacement remains bounded by the declared reorder delay. A forward-coded +anchor farther ahead needs a bounded, complete decode-prefix proof, as in #700: distinct ranks +fill a prefix, the backward delay still fits, and the observed slots plus lead pass a dry run. +Only the measured forward distance is then allowed, limited to **16 pictures**; that proof resets +at each candidate IDR. The packets held behind it remain bounded by **1024 interleaved packets and 32 MiB**, which is a ceiling on a container's interleaving rather than on its GOP. A candidate sequence has to show its reordering within **64 pictures** before a single picture is rewritten. @@ -127,7 +135,8 @@ not guessed at. **Every refusal hands the held packets back exactly as they arri and lets the rest of that sequence stream through, so a shape this policy does not own costs the repair and never the session; the next IDR is a fresh candidate. Seek and teardown release both unpublished input and pending output. No decoder-route or host-UI -change is needed. The existing whole-file missing-offset policy above remains separate. +change is needed. The uniform whole-file policy above is unchanged; its strictly proven +variable-cadence H.264 fallback reuses this same slot ladder. See the [partial-composition regression and reproduction](partial-composition-regression.md) for generated fixtures, original numeric evidence and verification limits. From b76b1153523760d6df2d1331d3d0a40e69af3fbe Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Oskar=20Sch=C3=B6ldstr=C3=B6m?= Date: Tue, 13 Oct 2026 07:38:45 -0300 Subject: [PATCH 006/132] fix(demux): reconstruct clamped Matroska H264 timestamps Co-Authored-By: OpenAI Codex --- .../Video/H264MatroskaSlotPermutation.swift | 206 ++++++++++++++++-- .../Issue511MatroskaCodingOrderTests.swift | 67 ++++++ 2 files changed, 258 insertions(+), 15 deletions(-) diff --git a/Sources/AetherEngine/Video/H264MatroskaSlotPermutation.swift b/Sources/AetherEngine/Video/H264MatroskaSlotPermutation.swift index 53720e956..6e6adf862 100644 --- a/Sources/AetherEngine/Video/H264MatroskaSlotPermutation.swift +++ b/Sources/AetherEngine/Video/H264MatroskaSlotPermutation.swift @@ -59,8 +59,8 @@ enum H264MatroskaSlotPermutation { typealias Sample = H264CompositionOffsetRepair.Sample /// Video pictures to sample before deciding, and the same held-packet ceilings the ISO-BMFF - /// sample pays. A well-formed MKV with B-frames never reaches any of them: its first stepped-back - /// slot ends the sampling, and that is normally the third packet. + /// sample pays. Streams without an exact whole-tick cadence still exit on their first + /// stepped-back slot, normally the third packet. static let sampleTarget = H264CompositionOffsetRepair.sampleTarget static let minimumSamples = H264CompositionOffsetRepair.minimumSamples static let sampleByteBudget = H264CompositionOffsetRepair.sampleByteBudget @@ -71,6 +71,12 @@ enum H264MatroskaSlotPermutation { /// wait nobody should pay. static let lookaheadCeiling: Int64 = 32 + private static func timestamp(anchor: Int64, index: Int64, step: Int64) -> Int64? { + let (offset, productOverflow) = index.multipliedReportingOverflow(by: step) + let (value, sumOverflow) = anchor.addingReportingOverflow(offset) + return productOverflow || sumOverflow ? nil : value + } + enum Verdict: Equatable, Sendable { /// The container presents in display order, which is what the format says it should do. case healthy @@ -78,29 +84,25 @@ enum H264MatroskaSlotPermutation { case inconclusive(String) /// Ticks a picture order count advances per displayed picture, measured rather than assumed. case permute(pocStep: Int64) + /// A CFR ladder whose B-picture timestamps were clamped forward onto their decode slots. + case reconstruct(step: Int64, pocStep: Int64) } /// Fail-closed decision over the sampled head. Anything unproven returns `.inconclusive`, which /// leaves the stream exactly as the container delivered it. - static func classify(samples: [Sample], videoDelay: Int) -> Verdict { + static func classify(samples: [Sample], videoDelay: Int, nominalStep: Int64? = nil) -> Verdict { guard videoDelay > 0, videoDelay <= 16 else { return .inconclusive("reorder delay \(videoDelay) outside 1...16") } - // One stepped-back slot is the container doing its job, and it ends the sampling before the - // window fills. `PTS != DTS` proves nothing here: libavformat synthesizes a decode ladder - // from a rising presentation one just as readily as from a reordered one, which is why the - // ISO-BMFF eligibility test cannot be reused. let slots = samples.map(\.pts) - if zip(slots, slots.dropFirst()).contains(where: { $1 < $0 }) { return .healthy } + if nominalStep == nil, + zip(slots, slots.dropFirst()).contains(where: { $1 < $0 }) { return .healthy } guard samples.count >= minimumSamples else { return .inconclusive("only \(samples.count) sampled pictures") } guard slots.allSatisfy({ $0 != Int64.min }) else { return .inconclusive("presentation slots are not all stated") } - guard !zip(slots, slots.dropFirst()).contains(where: { $1 == $0 }) else { - return .inconclusive("two pictures share a presentation slot") - } guard let first = samples.first, first.isKeyframe, first.pictureOrderCount == 0 else { return .inconclusive("sample does not start on a picture-order origin") } @@ -141,6 +143,64 @@ enum H264MatroskaSlotPermutation { highest - lowest + 1 <= Int64(samples.count + videoDelay + 1) else { return .inconclusive("display ranks do not fill the sampled window") } + + // Some Matroska writers preserve most CFR presentation slots but clamp a B picture whose + // true PTS precedes its synthesized DTS forward onto that DTS. The result has paired duplicate + // PTS/DTS slots and paired holes: decoded output alternates 0 and 2 frame intervals forever. + // Reconstruct only when the container's declared rate is an exact whole-tick cadence, every + // picture-order rank lands either on that lattice or on this precise forward clamp, and at + // least two clamps corroborate the shape. A VFR ladder, one damaged packet, and #511's + // coding-order-slot defect all fail closed. + if let step = nominalStep, step > 0 { + let anchor = first.pts + var clamped = 0 + var reconstructable = true + for (decodeIndex, sample) in samples.enumerated() { + let rank = sample.pictureOrderCount / pocStep + guard let expectedPTS = timestamp(anchor: anchor, index: rank, step: step), + let expectedDTS = timestamp( + anchor: anchor, + index: Int64(decodeIndex) - Int64(videoDelay), step: step) + else { + reconstructable = false + break + } + if sample.pts != expectedPTS { + let (forward, overflow) = sample.pts.subtractingReportingOverflow(expectedPTS) + guard !overflow, sample.pts == sample.dts, forward == step, + !slots.contains(expectedPTS), + slots.filter({ $0 == sample.pts }).count == 2 else { + reconstructable = false + break + } + clamped += 1 + } + if sample.dts == Int64.min { + guard decodeIndex < videoDelay else { + reconstructable = false + break + } + } else if sample.dts != expectedDTS { + let (forward, overflow) = sample.dts.subtractingReportingOverflow(expectedDTS) + guard !overflow, sample.pts != expectedPTS, sample.dts == sample.pts, + forward == step else { + reconstructable = false + break + } + } + } + if reconstructable, clamped >= 2 { + return .reconstruct(step: step, pocStep: pocStep) + } + } + + // One stepped-back slot is the container doing its job. `PTS != DTS` proves nothing here: + // libavformat synthesizes a decode ladder from a rising presentation one just as readily as + // from a reordered one, which is why the ISO-BMFF eligibility test cannot be reused. + if zip(slots, slots.dropFirst()).contains(where: { $1 < $0 }) { return .healthy } + guard !zip(slots, slots.dropFirst()).contains(where: { $1 == $0 }) else { + return .inconclusive("two pictures share a presentation slot") + } // The muxer invariant, checked on the window before a single packet is moved: a picture may // sit at most its own reorder delay behind the slot it owns, or its repaired presentation // time would land before the decode time the container states for it. @@ -151,6 +211,64 @@ enum H264MatroskaSlotPermutation { return .permute(pocStep: pocStep) } + struct CFRRewriter { + let step: Int64 + let pocStep: Int64 + let videoDelay: Int + private(set) var brokenReason: String? + private var anchorPTS: Int64? + private var decodeIndex: Int64 = 0 + + mutating func noteSeek() { + anchorPTS = nil + decodeIndex = 0 + brokenReason = nil + } + + mutating func rewrite( + pts: Int64, dts: Int64, pictureOrderCount: Int64?, isKeyframe: Bool + ) -> (pts: Int64, dts: Int64)? { + guard brokenReason == nil else { return nil } + if isKeyframe, pictureOrderCount == 0 { + anchorPTS = nil + decodeIndex = 0 + if pts != Int64.min { anchorPTS = pts } + } + guard let anchorPTS else { + return nil + } + defer { decodeIndex += 1 } + guard let pictureOrderCount, pictureOrderCount >= 0, pocStep > 0, + pictureOrderCount % pocStep == 0 else { + brokenReason = "a picture carries no usable picture order" + return nil + } + guard let expectedPTS = H264MatroskaSlotPermutation.timestamp( + anchor: anchorPTS, index: pictureOrderCount / pocStep, step: step), + let expectedDTS = H264MatroskaSlotPermutation.timestamp( + anchor: anchorPTS, index: decodeIndex - Int64(videoDelay), step: step) + else { + brokenReason = "timestamp reconstruction overflowed" + return nil + } + + let ptsForward = pts.subtractingReportingOverflow(expectedPTS) + let dtsForward = dts.subtractingReportingOverflow(expectedDTS) + let unsetDTS = dts == Int64.min + let ptsFits = pts == expectedPTS + || (!ptsForward.overflow && ptsForward.partialValue == step) + let dtsFits = dts == expectedDTS + || unsetDTS + || (!dtsForward.overflow && dtsForward.partialValue == step) + guard ptsFits, dtsFits else { + brokenReason = "the timestamp ladder changed shape " + + "(pts=\(pts)/\(expectedPTS) dts=\(dts)/\(expectedDTS))" + return nil + } + return (expectedPTS, expectedDTS) + } + } + /// One coded video sequence, as the container hands it out. Values only, so the permutation is /// testable without FFmpeg. struct Sequence { @@ -257,7 +375,7 @@ enum H264MatroskaSlotPermutation { /// interleave waits with it, which is one mini-GOP and not one sequence. final class H264MatroskaSlotPermutationSession: H264TimestampRepairSession { - enum Phase: Equatable { case sampling, permuting, off } + enum Phase: Equatable { case sampling, permuting, reconstructing, off } private struct Entry { let packet: UnsafeMutablePointer @@ -279,6 +397,8 @@ final class H264MatroskaSlotPermutationSession: H264TimestampRepairSession { private var ready: [UnsafeMutablePointer] = [] private var pendingBytes = 0 private var sequence: H264MatroskaSlotPermutation.Sequence? + private var cfrRewriter: H264MatroskaSlotPermutation.CFRRewriter? + private let nominalStep: Int64? private var verdictDescription = "sampling" /// nil unless this stream is the shape the defect needs: Matroska, H.264, and a bitstream that @@ -299,6 +419,12 @@ final class H264MatroskaSlotPermutationSession: H264TimestampRepairSession { self.streamIndex = streamIndex self.videoDelay = Int(codecpar.pointee.video_delay) self.reader = reader + let rate = stream.pointee.avg_frame_rate.num > 0 && stream.pointee.avg_frame_rate.den > 0 + ? stream.pointee.avg_frame_rate : stream.pointee.r_frame_rate + let numerator = Int64(stream.pointee.time_base.den) * Int64(rate.den) + let denominator = Int64(stream.pointee.time_base.num) * Int64(rate.num) + nominalStep = denominator > 0 && numerator > 0 && numerator % denominator == 0 + ? numerator / denominator : nil } deinit { releasePending() } @@ -329,11 +455,29 @@ final class H264MatroskaSlotPermutationSession: H264TimestampRepairSession { if samples.count >= H264MatroskaSlotPermutation.sampleTarget || pendingBytes >= H264MatroskaSlotPermutation.sampleByteBudget || pending.count >= H264MatroskaSlotPermutation.heldPacketCeiling - || earlyHealthyVerdict { + || (nominalStep == nil && earlyHealthyVerdict) { decide() } return true } + if phase == .reconstructing { + if isPicture, var rewriter = cfrRewriter { + applyCFRRepair( + to: packet, pictureOrderCount: pictureOrderCount, + using: &rewriter + ) + cfrRewriter = rewriter + if let reason = rewriter.brokenReason { + verdictDescription = "stopped (\(reason))" + EngineLog.emit( + "[Demuxer] Matroska H.264 CFR reconstruction stopped on stream " + + "\(streamIndex): \(reason)", category: .demux) + phase = .off + cfrRewriter = nil + } + } + return false + } let rank = isPicture ? admit(packet, pictureOrderCount: pictureOrderCount) : nil append(packet, isPicture: isPicture, pictureOrderCount: pictureOrderCount, rank: rank) drain() @@ -432,7 +576,7 @@ final class H264MatroskaSlotPermutationSession: H264TimestampRepairSession { } /// A well-formed MKV with B-frames proves itself on the packet that steps back, which is normally - /// the third one, so it never pays the sample window. + /// the third one. Exact whole-tick CFR candidates keep sampling so duplicate slots can be checked. private var earlyHealthyVerdict: Bool { guard samples.count >= 2 else { return false } let last = samples[samples.count - 1] @@ -441,7 +585,8 @@ final class H264MatroskaSlotPermutationSession: H264TimestampRepairSession { } private func decide() { - let verdict = H264MatroskaSlotPermutation.classify(samples: samples, videoDelay: videoDelay) + let verdict = H264MatroskaSlotPermutation.classify( + samples: samples, videoDelay: videoDelay, nominalStep: nominalStep) switch verdict { case .permute(let pocStep): verdictDescription = "permute pocStep=\(pocStep)" @@ -463,6 +608,23 @@ final class H264MatroskaSlotPermutationSession: H264TimestampRepairSession { pending[index].rank = rank } drain() + case .reconstruct(let step, let pocStep): + verdictDescription = "reconstruct step=\(step) pocStep=\(pocStep)" + phase = .reconstructing + var rewriter = H264MatroskaSlotPermutation.CFRRewriter( + step: step, pocStep: pocStep, videoDelay: videoDelay) + for entry in pending where entry.isPicture { + applyCFRRepair( + to: entry.packet, pictureOrderCount: entry.pictureOrderCount, + using: &rewriter + ) + } + cfrRewriter = rewriter + flushPendingUntouched() + EngineLog.emit( + "[Demuxer] Matroska H.264 clamped CFR timestamps confirmed on stream " + + "\(streamIndex): \(verdictDescription) samples=\(samples.count)", + category: .demux) case .healthy: verdictDescription = "healthy" disarm() @@ -502,6 +664,7 @@ final class H264MatroskaSlotPermutationSession: H264TimestampRepairSession { reader.reset() if phase == .sampling { samples.removeAll(keepingCapacity: true) } sequence?.noteSeek() + cfrRewriter?.noteSeek() } private func releasePending() { @@ -527,4 +690,17 @@ final class H264MatroskaSlotPermutationSession: H264TimestampRepairSession { + "repaired=\(sequence?.repairedPictures ?? 0) " + "unrepaired=\(sequence?.unrepairedPictures ?? 0)" } + + private func applyCFRRepair( + to packet: UnsafeMutablePointer, + pictureOrderCount: Int64?, + using rewriter: inout H264MatroskaSlotPermutation.CFRRewriter + ) { + if let repaired = rewriter.rewrite( + pts: packet.pointee.pts, dts: packet.pointee.dts, + pictureOrderCount: pictureOrderCount, isKeyframe: isKeyframe(packet)) { + packet.pointee.pts = repaired.pts + packet.pointee.dts = repaired.dts + } + } } diff --git a/Tests/AetherEngineTests/Issue511MatroskaCodingOrderTests.swift b/Tests/AetherEngineTests/Issue511MatroskaCodingOrderTests.swift index f63122715..09d0f1c0f 100644 --- a/Tests/AetherEngineTests/Issue511MatroskaCodingOrderTests.swift +++ b/Tests/AetherEngineTests/Issue511MatroskaCodingOrderTests.swift @@ -167,6 +167,73 @@ struct Issue511MatroskaCodingOrderTests { #expect(Permutation.classify(samples: Array(healthy.prefix(3)), videoDelay: 1) == .healthy) } + @Test("clamped CFR B-picture timestamps are reconstructed from picture order") + func reconstructsClampedCFR() throws { + let n = Int64.min + let samples = [ + Sample(dts: n, pts: 0, pictureOrderCount: 0, isKeyframe: true), + Sample(dts: n, pts: 200, pictureOrderCount: 10, isKeyframe: false), + Sample(dts: 0, pts: 80, pictureOrderCount: 4, isKeyframe: false), + Sample(dts: 80, pts: 80, pictureOrderCount: 2, isKeyframe: false), + Sample(dts: 80, pts: 120, pictureOrderCount: 6, isKeyframe: false), + Sample(dts: 120, pts: 160, pictureOrderCount: 8, isKeyframe: false), + Sample(dts: 160, pts: 440, pictureOrderCount: 22, isKeyframe: false), + Sample(dts: 200, pts: 320, pictureOrderCount: 16, isKeyframe: false), + Sample(dts: 280, pts: 280, pictureOrderCount: 12, isKeyframe: false), + Sample(dts: 280, pts: 280, pictureOrderCount: 14, isKeyframe: false), + Sample(dts: 320, pts: 360, pictureOrderCount: 18, isKeyframe: false), + Sample(dts: 360, pts: 400, pictureOrderCount: 20, isKeyframe: false), + ] + #expect(Permutation.classify(samples: samples, videoDelay: 2, nominalStep: 40) + == .reconstruct(step: 40, pocStep: 2)) + + let repairedPTS: [Int64] = [0, 200, 80, 40, 120, 160, 440, 320, 240, 280, 360, 400] + let repairedDTS: [Int64] = [-80, -40, 0, 40, 80, 120, 160, 200, 240, 280, 320, 360] + let healthy = zip(samples.indices, samples).map { index, sample in + Sample( + dts: repairedDTS[index], pts: repairedPTS[index], + pictureOrderCount: sample.pictureOrderCount, isKeyframe: sample.isKeyframe) + } + #expect(Permutation.classify(samples: healthy, videoDelay: 2, nominalStep: 40) == .healthy) + #expect(Permutation.classify(samples: samples, videoDelay: 2, nominalStep: Int64.max) + == .healthy) + var oneClamp = samples + oneClamp[8] = Sample( + dts: 240, pts: 240, pictureOrderCount: 12, isKeyframe: false) + #expect(Permutation.classify(samples: oneClamp, videoDelay: 2, nominalStep: 40) + == .healthy) + + var rewriter = Permutation.CFRRewriter(step: 40, pocStep: 2, videoDelay: 2) + let repaired = samples.map { + rewriter.rewrite( + pts: $0.pts, dts: $0.dts, pictureOrderCount: $0.pictureOrderCount, + isKeyframe: $0.isKeyframe) + } + #expect(try repaired.map { try #require($0).pts } == repairedPTS) + #expect(try repaired.map { try #require($0).dts } == repairedDTS) + + #expect(rewriter.rewrite( + pts: n, dts: n, pictureOrderCount: 0, isKeyframe: true) == nil) + #expect(rewriter.rewrite( + pts: 480, dts: 400, pictureOrderCount: 2, isKeyframe: false) == nil) + #expect(rewriter.rewrite( + pts: 1_000, dts: n, pictureOrderCount: 0, isKeyframe: true)?.pts == 1_000) + _ = rewriter.rewrite( + pts: 1_200, dts: n, pictureOrderCount: 10, isKeyframe: false) + _ = rewriter.rewrite( + pts: 1_080, dts: 1_000, pictureOrderCount: 4, isKeyframe: false) + #expect(rewriter.rewrite( + pts: 1_160, dts: n, pictureOrderCount: 6, isKeyframe: false)?.pts == 1_120) + #expect(rewriter.rewrite( + pts: 1_200, dts: 1_120, pictureOrderCount: 8, isKeyframe: false)?.dts == 1_080) + + var broken = Permutation.CFRRewriter(step: 40, pocStep: 2, videoDelay: 2) + _ = broken.rewrite(pts: 0, dts: n, pictureOrderCount: 0, isKeyframe: true) + #expect(broken.rewrite(pts: 200, dts: n, pictureOrderCount: nil, isKeyframe: false) == nil) + #expect(broken.brokenReason != nil) + #expect(broken.rewrite(pts: 80, dts: 0, pictureOrderCount: 4, isKeyframe: false) == nil) + } + @Test("shapes this permutation does not describe are left alone") func failsClosed() { func verdict(slots: [Int64], pocs: [Int64], videoDelay: Int = 1) -> Permutation.Verdict { From 523b90471a5570d4b43e3614ee390bd7b85ae129 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Oskar=20Sch=C3=B6ldstr=C3=B6m?= Date: Tue, 13 Oct 2026 07:47:18 -0300 Subject: [PATCH 007/132] docs(changelog): note Matroska H264 cadence repair Co-Authored-By: OpenAI Codex --- CHANGELOG.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 751e60c90..bdc77b362 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,6 +21,14 @@ the public-API contract. ### Fixed +- **A 25 fps H.264 Matroska file with forward-clamped B-picture timestamps plays at a steady + cadence.** Its packet timestamps repeated display slots and left the preceding slots empty, so + decoded output alternated 0 ms and 80 ms frame intervals instead of 40 ms. The demuxer now + recognizes that shape only from an exact CFR lattice, complete picture-order ranks and two + corroborating clamps, then reconstructs PTS and DTS from the IDR anchor, picture order and decode + order. A seek starts from a new IDR, and overflow or any later timestamp outside the proven shape + disables the repair and leaves that packet untouched. + - **A host with its own Objective-C and C++ links again (#752).** Since 7.34.0 LibTrueHD and LibDovi each brought a `rust_eh_personality`, and with the host's `__objc_personality_v0` and `__gxx_personality_v0` that made four personality routines, one more than compact unwind encodes From 93d2be8b72db9f0f2d61b5c57eb55d8fb7b4595c Mon Sep 17 00:00:00 2001 From: Pavel Kajaba <15980400+pkajaba@users.noreply.github.com> Date: Wed, 7 Oct 2026 10:29:07 +0100 Subject: [PATCH 008/132] feat(dovi): Metal FEL composer and Dolby Vision display metadata (not routed) Adds the pieces that rebuild a Profile 7 FEL picture, with nothing selecting them yet. - DoviFelComposer (Metal): rebuilds the 12-bit picture from base layer, enhancement layer and RPU: reshaping curves (polynomial and MMR), inverse NLQ, MMR chroma, bilinear EL upsampling, ordered dither down to 10-bit, output in a pooled P010 buffer, composeAsync bounded at maxFramesInFlight. DoviMetalParams is the parameter ABI, DoviShaderSource the kernels. The output pool survives a seek flush (flush() trims idle buffers, releaseOutputPool() drops it on teardown): dropping it made post-seek frames allocate a second full pool beside the buffers the renderer still held, about +260 MB at 4K for ~0.6 s per seek; measured end to end with the decoder in the next PRs, peak 674 -> 500 MB over 12 seeks. - DoviDisplayMetadata: attaches the Profile 8.1 RPU as DolbyVisionRPUData plus DolbyCompatibilityID (a CFString: a CFNumber aborts the render server). Both keys are undocumented; they are defined in this one file and UndocumentedDolbyVisionKeysTests fails if either literal appears in another source file. - DoviRpuConverter.separateDualLayers splits one packet into base layer, enhancement-layer packet, converted RPU and composer parameters; convertPacketToProfile81 is now that walk with the EL dropped. The RPU attached to composed frames is converted with mode 2 (the composer has already applied the reshaping and added the residual, so the display side must not reshape again); DoviComposedFrameRpuTests pins this. Nothing calls the composer, so no behaviour changes for any source. Public API: none. Not in this PR: the dual-layer decoder, the routing flag and host wiring, the aetherctl tooling, and dolby-vision-fel.md (comments here cite its section numbers; the document arrives with the route PR). Tests: GPU curve sweep, edge cases and adversarial inputs, layout probe (the GPU reads each parameter sub-block back at the offset Swift wrote it), shader/ABI sync, dither, enhancement upsampler, FEL math, FEL and CRC-valid-mutation fuzzers, converter and guard tests, async compose, look-ahead pool sizing. The real-clip tests skip without DV_FEL_TESTS_DIR. --- .../Video/DoviDisplayMetadata.swift | 122 ++ .../AetherEngine/Video/DoviRpuConverter.swift | 305 +++- .../Video/Metal/DoviFelComposer.swift | 1452 +++++++++++++++++ .../Video/Metal/DoviMetalParams.swift | 315 ++++ .../Video/Metal/DoviShaderSource.swift | 722 ++++++++ .../DoviComposeAsyncTests.swift | 283 ++++ .../DoviComposedFrameRpuTests.swift | 305 ++++ .../DoviCrcValidFuzzTests.swift | 150 ++ .../DoviDisplayMetadataTests.swift | 99 ++ Tests/AetherEngineTests/DoviDitherTests.swift | 291 ++++ .../DoviElBitDepthTests.swift | 100 ++ .../DoviEnhancementUpsamplerTests.swift | 62 + .../AetherEngineTests/DoviFelFuzzTests.swift | 252 +++ .../AetherEngineTests/DoviFelMathTests.swift | 323 ++++ .../DoviFelRealFileTests.swift | 46 + .../DoviGpuAdversarialTests.swift | 210 +++ .../DoviGpuCurveSweepTests.swift | 393 +++++ .../DoviGpuEdgeCaseTests.swift | 492 ++++++ .../DoviLayoutProbeTests.swift | 240 +++ .../DoviRpuConverterTests.swift | 194 +++ .../AetherEngineTests/DoviRpuGuardTests.swift | 7 + .../DoviShaderSyncTests.swift | 81 + .../SampleBufferRendererLookaheadTests.swift | 5 + .../Support/DoviTestSupport.swift | 31 + .../UndocumentedDolbyVisionKeysTests.swift | 33 + docs/architecture.md | 9 +- 26 files changed, 6453 insertions(+), 69 deletions(-) create mode 100644 Sources/AetherEngine/Video/DoviDisplayMetadata.swift create mode 100644 Sources/AetherEngine/Video/Metal/DoviFelComposer.swift create mode 100644 Sources/AetherEngine/Video/Metal/DoviMetalParams.swift create mode 100644 Sources/AetherEngine/Video/Metal/DoviShaderSource.swift create mode 100644 Tests/AetherEngineTests/DoviComposeAsyncTests.swift create mode 100644 Tests/AetherEngineTests/DoviComposedFrameRpuTests.swift create mode 100644 Tests/AetherEngineTests/DoviCrcValidFuzzTests.swift create mode 100644 Tests/AetherEngineTests/DoviDisplayMetadataTests.swift create mode 100644 Tests/AetherEngineTests/DoviDitherTests.swift create mode 100644 Tests/AetherEngineTests/DoviElBitDepthTests.swift create mode 100644 Tests/AetherEngineTests/DoviEnhancementUpsamplerTests.swift create mode 100644 Tests/AetherEngineTests/DoviFelFuzzTests.swift create mode 100644 Tests/AetherEngineTests/DoviFelMathTests.swift create mode 100644 Tests/AetherEngineTests/DoviFelRealFileTests.swift create mode 100644 Tests/AetherEngineTests/DoviGpuAdversarialTests.swift create mode 100644 Tests/AetherEngineTests/DoviGpuCurveSweepTests.swift create mode 100644 Tests/AetherEngineTests/DoviGpuEdgeCaseTests.swift create mode 100644 Tests/AetherEngineTests/DoviLayoutProbeTests.swift create mode 100644 Tests/AetherEngineTests/DoviShaderSyncTests.swift create mode 100644 Tests/AetherEngineTests/Support/DoviTestSupport.swift create mode 100644 Tests/AetherEngineTests/UndocumentedDolbyVisionKeysTests.swift diff --git a/Sources/AetherEngine/Video/DoviDisplayMetadata.swift b/Sources/AetherEngine/Video/DoviDisplayMetadata.swift new file mode 100644 index 000000000..5443014b1 --- /dev/null +++ b/Sources/AetherEngine/Video/DoviDisplayMetadata.swift @@ -0,0 +1,122 @@ +import Foundation +import CoreVideo + +/// Per-frame Dolby Vision display metadata carried on a `CVPixelBuffer`. +/// +/// The FEL composer reconstructs the 12-bit master on the GPU, but reconstruction only +/// restores *pixels*. The display side of Dolby Vision — L1 frame statistics, L2 creative +/// trims, L5 active area, L8/L9/L11 — is metadata that has to travel alongside the frame +/// so the compositor can drive the panel (or hand it to the TV over HDMI). On the +/// `AVPlayer` path VideoToolbox does this itself; on the `AVSampleBufferDisplayLayer` path +/// the buffers are ours, so the metadata is ours to attach. +/// +/// ## The channel +/// CoreVideo and IOSurface both expose this channel under plain string keys +/// (`kCVImageBufferDolbyVisionRPUDataKey`, `kIOSurfaceDolbyVisionRPUData`), so it can be +/// driven with `CVBufferSetAttachment` with no private symbol linkage. It is still +/// SPI in spirit: the keys are not in any header, and nothing contractual says a +/// given OS release will consume them. +/// +/// ## The payload, established empirically +/// Decoding a Profile 8.1 asset through `AVAssetReader` and dumping every attachment on the +/// output buffers shows VideoToolbox setting exactly two keys per frame — `DolbyVisionRPUData` +/// (a 188/189-byte `CFData` that changes frame to frame) and `DolbyCompatibilityID` (`1`) — +/// on both the `CVBuffer` and its backing `IOSurface`. Comparing those payloads byte-for-byte +/// against the source bitstream identifies them exactly: **the `unspec62` NAL with its 2-byte +/// HEVC NAL header (`0x7C 0x01`) removed and emulation-prevention bytes retained.** The +/// EPB-stripped RBSP is 176 bytes and does not match, which rules out the other plausible +/// reading. +/// +/// That is precisely what `DoviRpuConverter.separateDualLayers` already produces: libdovi +/// converts the stream's Profile 7 RPU to Profile 8.1 with +/// `dovi_convert_rpu_with_mode(rpu, 2)`, which preserves the full DM payload (L1/L2/L5/L8/L9/L11) +/// and only rewrites the profile-specific container. The consumer on Apple platforms is the +/// private `HDRProcessing.framework` (`DolbyVisionComposer`, `DolbyVisionDisplayManagement`, +/// destination display types `…DolbyVisionTV` / `…LowLatencyDolbyVisionTV`). +/// +/// ## What is verified +/// Verified on macOS: VideoToolbox *produces* these attachments in this exact format, and a +/// hand-set attachment propagates to the backing `IOSurface` and survives into `CMSampleBuffer`. +/// +/// Verified on an Apple TV 4K (tvOS 27.0): the compositor **does** consume them from a +/// client-supplied buffer on the `AVSampleBufferDisplayLayer` path. The proof arrived the hard +/// way — as a `backboardd` crash report — but it is unambiguous, because the whole stack is in +/// the trace: +/// +/// ``` +/// CA::WindowServer::IOMFBServer::vsync_callback (thread com.apple.coreanimation.display.external-0) +/// CA::Render::ImageQueue::forward_dm +/// CA::HDRProcessorInternal::create_surface_with_forward_dm +/// CA::HDRProcessorInternal::tonemap_surface_internal(__IOSurface*, …, __CFData const*, …) +/// -[HDRProcessor generateMSRColorConfigWithOperation:inputSurface:outputSurface:metadata:…] +/// -[HDRProcessor processFrameInternalWithLayer0:layer1:outout:metadata:…] +/// -[HDRProcessor applyDoVi81PolicyWithInput:WithRPU:] +/// ``` +/// +/// That is the *external display* CoreAnimation thread, on the vsync callback, pulling forward +/// display management off our `IOSurface` and applying Dolby Vision 8.1 policy with our RPU. +/// Nothing else puts `HDRProcessor` on that thread with a `__CFData` in hand. The attachment +/// channel works, and it reaches the HDMI output. +enum DoviDisplayMetadata { + + // MARK: Undocumented attachment keys + // + // These two strings are the ONLY undocumented platform contract the FEL route depends on, and + // this is the only place they are spelled (`UndocumentedDolbyVisionKeysTests` fails the build if + // either literal appears anywhere else in `Sources/`). Neither is in a public header; both are + // what VideoToolbox itself attaches with `kVTDecompressionPropertyKey_PropagatePerFrameHDRDisplayMetadata`, + // and what tvOS's compositor reads back. No OS release promises to keep honouring them. If one + // stops (the dv-pulse test in `Probes/DolbyVisionPulse` shows it: the pulse goes flat, like its "S" arm), + // this is the one spot to change, and `preferDualLayerDolbyVision = false` is the fallback. + + /// `kCVImageBufferDolbyVisionRPUDataKey` / `kIOSurfaceDolbyVisionRPUData`. Undocumented. + static let rpuDataKey = "DolbyVisionRPUData" + /// `kCVImageBufferDolbyCompatibilityIDKey`. Undocumented. + static let compatibilityIDKey = "DolbyCompatibilityID" + + /// Profile 8 compatibility ID 1 — HDR10-compatible base layer, which is what + /// `dovi_convert_rpu_with_mode(rpu, 2)` targets and what the composed frame is. + /// + /// A **string**, deliberately. VideoToolbox emits this attachment as a `CFString` `"1"` + /// (`NSTaggedPointerString`), not as a number, and tvOS depends on that: inside + /// `-[HDRProcessor applyDoVi81PolicyWithInput:WithRPU:]` the value is parsed with + /// `CFStringGetIntValue`, which messages `-length` on whatever it is handed. A `CFNumber` + /// here does not respond to `-length`, so it raises `NSInvalidArgumentException` on + /// backboardd's `com.apple.coreanimation.display.external-0` thread and aborts the render + /// server — which takes this app down with it, roughly a third of a second after the first + /// composed frame reaches the display. Confirmed twice on an Apple TV 4K (tvOS 27.0) and + /// diagnosed from the resulting `backboardd-*.ips`; the type was then read back off + /// VideoToolbox's own output to be sure. Do not "simplify" this to an integer. + static let profile81CompatibilityID = "1" + + /// Attaches (or clears) the per-frame RPU on `buffer`. + /// + /// Pass `nil` to clear. Clearing matters: decoder output buffers are recycled from a + /// CoreVideo pool, so a frame that has no RPU must actively remove any attachment left + /// behind by a previous tenant rather than silently inherit stale trims. + /// + /// - Parameter nal: the Profile 8.1 `unspec62` NAL *including* its 2-byte header, as + /// returned by `dovi_write_unspec62_nalu`. The header is stripped here. + static func attach(rpuNAL nal: [UInt8]?, to buffer: CVPixelBuffer) { + guard let payload = payload(fromNAL: nal) else { + CVBufferRemoveAttachment(buffer, rpuDataKey as CFString) + CVBufferRemoveAttachment(buffer, compatibilityIDKey as CFString) + return + } + CVBufferSetAttachment(buffer, rpuDataKey as CFString, payload as CFData, .shouldPropagate) + CVBufferSetAttachment(buffer, compatibilityIDKey as CFString, profile81CompatibilityID as CFString, .shouldPropagate) + } + + /// Strips the `0x7C 0x01` unspec62 NAL header, leaving the RPU payload the attachment wants. + /// Emulation-prevention bytes are deliberately left in place — that is the form VideoToolbox + /// itself emits. + static func payload(fromNAL nal: [UInt8]?) -> Data? { + guard let nal, nal.count > 2 else { return nil } + if nal[0] == 0x7C && nal[1] == 0x01 { + return Data(nal.dropFirst(2)) + } + // Already header-less (payload starts with the DV RPU prefix byte 0x19). + if nal[0] == 0x19 { return Data(nal) } + return nil + } +} diff --git a/Sources/AetherEngine/Video/DoviRpuConverter.swift b/Sources/AetherEngine/Video/DoviRpuConverter.swift index a99b55b8c..62b972dd0 100644 --- a/Sources/AetherEngine/Video/DoviRpuConverter.swift +++ b/Sources/AetherEngine/Video/DoviRpuConverter.swift @@ -20,49 +20,156 @@ public enum DoviRpuConverter { } } + struct DualLayerPacketResult: @unchecked Sendable { + let elPacket: UnsafeMutablePointer? + let rpuData: [UInt8]? + /// The Profile 8.1 RPU libdovi produced from `rpuData`, as a complete unspec62 NAL + /// (2-byte HEVC NAL header included). On the remux route this is what goes back into + /// the packet; on the FEL compose route it is the per-frame display metadata payload + /// (L1 statistics, L2 trims, L5/L8/L9/L11), which `DoviDisplayMetadata` attaches to the + /// composed pixel buffer. Nil when there was no RPU or the conversion failed. + let convertedRpu: [UInt8]? + let rpuParams: DoviMetalParams? + let isFEL: Bool + let isMEL: Bool + let success: Bool + /// `use_prev_vdr_rpu_flag`: the RPU carries no mapping and means "keep the previous one". + let usesPreviousMapping: Bool + /// This RPU's own `disable_residual_flag`. The header is read even when no mapping follows, so + /// a `use_prev_vdr_rpu_flag` frame can still say whether its residual applies. + let disableResidual: Bool + + init(elPacket: UnsafeMutablePointer?, rpuData: [UInt8]?, convertedRpu: [UInt8]? = nil, rpuParams: DoviMetalParams? = nil, isFEL: Bool, isMEL: Bool = false, success: Bool, usesPreviousMapping: Bool = false, disableResidual: Bool = false) { + self.elPacket = elPacket + self.rpuData = rpuData + self.convertedRpu = convertedRpu + self.rpuParams = rpuParams + self.isFEL = isFEL + self.isMEL = isMEL + self.success = success + self.usesPreviousMapping = usesPreviousMapping + self.disableResidual = disableResidual + } + } + + /// Stream classification from one RPU (dolby-vision-fel.md §5.2, invariant I9): a property of the + /// encode, decided once. Only `el_type` feeds it — a per-frame `disable_residual_flag` + /// rides to the kernels in `DoviMetalParams.flags` (`.disableResidual`) and never + /// changes the classification. + enum StreamKind: Sendable, Equatable { + case undetermined + case fel + case mel + } + + /// Classifies one `separateDualLayers` result. Pure over the result so the decoder's + /// one-shot latch and routing can share it. + static func classifyStream(_ result: DualLayerPacketResult) -> StreamKind { + if result.isMEL { return .mel } + if result.isFEL { return .fel } + return .undetermined + } + + /// A piece of the rebuilt packet: points into the source packet (a NAL passed through) or into a + /// libdovi buffer (a rewritten RPU). Audit PERF-111: what survives is described, not copied, and the + /// one rebuild is the only copy. + private struct Piece { + let base: UnsafePointer + let count: Int + } + + /// Writes `pieces` into a new padded buffer, each behind its start code or 4-byte length. + /// nil on an allocation failure. `total` is what `Self.framedSize` returned for the same pieces. + private static func buildPacketBuffer( + _ pieces: [Piece], total: Int, emitsStartCodes: Bool + ) -> UnsafeMutablePointer? { + let pad = Int(AV_INPUT_BUFFER_PADDING_SIZE) + guard let ref = av_buffer_alloc(total + pad) else { return nil } + guard let dst = ref.pointee.data else { + var r: UnsafeMutablePointer? = ref + av_buffer_unref(&r) + return nil + } + var w = 0 + for piece in pieces { + let n = piece.count + if emitsStartCodes { + dst[w + 0] = 0; dst[w + 1] = 0; dst[w + 2] = 0; dst[w + 3] = 1 + } else { + dst[w + 0] = UInt8((n >> 24) & 0xFF) + dst[w + 1] = UInt8((n >> 16) & 0xFF) + dst[w + 2] = UInt8((n >> 8) & 0xFF) + dst[w + 3] = UInt8(n & 0xFF) + } + w += lengthPrefixSize + if n > 0 { + memcpy(dst + w, piece.base, n) + } + w += n + } + memset(dst + total, 0, pad) // decoders read past size + return ref + } + + private static func framedSize(_ pieces: [Piece]) -> Int { + pieces.reduce(0) { $0 + lengthPrefixSize + $1.count } + } + /// The Profile 8.1 form of a parsed RPU (`dovi_convert_rpu_with_mode`, mode 2) written back as an /// unspec62 NAL unit, or nil when libdovi refuses either step. The caller frees the result with /// `dovi_data_free`. This is the pair of calls the library can abort in, so everything that feeds it /// goes through `DoviRpuGuard.parse` first, and the tests drive it through this one entry point. + /// + /// Mode 2 and not 5, on the dual-layer route too: by the time a frame reaches the display layer the + /// Metal composer has already applied the RPU's luma/chroma reshaping and added the residual, so the + /// picture is finished. Mode 2 sets both curves to no-op (and drops the EL/NLQ part) while keeping + /// L1/L2 and the rest of the display metadata; an RPU that kept the mapping would make the display + /// side reshape a second time. `DoviComposedFrameRpuTests` pins this on the RPU attached to composed frames. static func convertedUnspec62(_ rpu: OpaquePointer) -> UnsafePointer? { guard dovi_convert_rpu_with_mode(rpu, 2) == 0 else { return nil } return dovi_write_unspec62_nalu(rpu) } - /// Returns `false` when a libdovi conversion fails (or on an internal allocation failure). - /// On a libdovi failure the offending RPU (and EL) are dropped so the frame degrades to the - /// clean HDR10 base, rather than shipping a P7 RPU inside a container already declared 8.1 - /// (mixed-profile hazard, #135). Packets with no RPU/EL are left untouched and return `true`. + /// Splits a Profile 7 packet into its layers in one walk. /// - /// `framing` must be the framing of THIS packet, and the rewritten packet is emitted in the same - /// one: the mp4 muxer decides whether to convert samples from the extradata, so a packet that - /// changes framing under it is a second defect (#365). An Annex-B packet walked as length- - /// prefixed reads `00 00 01 40` as a 320-byte NAL and finds no RPU at all, which is how a P7 - /// source shipped its RPU and EL untouched inside a container already rewritten to 8.1. - public static func convertPacketToProfile81( + /// The base-layer packet is rewritten in place to hold the base layer and the Profile 8.1 RPU (mode 2), + /// unless `useCustomBlockSource` leaves it untouched for a decoder that skips the unspec NALs itself. + /// With `extractEL` the enhancement layer comes back as a separate packet of its own (the 2-byte + /// unspec63 header stripped), stamped like the source; with `extractParams` the RPU is also parsed into + /// the composition parameters. This is the walk `convertPacketToProfile81` runs with both off. + /// + /// A libdovi failure drops the offending RPU, reports `success == false` and leaves the rest of the + /// packet usable (#135). The RPU is parsed through `DoviRpuGuard`, which refuses the CRC-valid inputs + /// that make libdovi abort. + static func separateDualLayers( _ packet: UnsafeMutablePointer, - framing: VideoNALFraming = .lengthPrefixed(size: 4) - ) -> Bool { - guard let data = packet.pointee.data, packet.pointee.size > 0 else { return true } - guard packet.pointee.size > 4 else { return true } - guard Self.canRebuild(framing) else { return true } + framing: VideoNALFraming = .lengthPrefixed(size: 4), + extractEL: Bool = true, + useCustomBlockSource: Bool = false, + extractParams: Bool = true + ) -> DualLayerPacketResult { + let untouched = DualLayerPacketResult(elPacket: nil, rpuData: nil, isFEL: false, success: true) + guard let data = packet.pointee.data, packet.pointee.size > 4 else { return untouched } + guard Self.canRebuild(framing) else { return untouched } let size = Int(packet.pointee.size) - // Audit PERF-111: what survives is described, not copied. Each piece points into the source - // packet (passed through) or into a libdovi buffer (a rewritten RPU), and the one rebuild below - // is the only copy. The source stays alive until that copy is made, and the libdovi buffers - // until this function returns. - struct Piece { - let base: UnsafePointer - let count: Int - } + // The libdovi buffers a piece points into live until this function returns. var pieces: [Piece] = [] pieces.reserveCapacity(16) + var elPieces: [Piece] = [] var rewritten: [UnsafePointer] = [] defer { for buffer in rewritten { dovi_data_free(buffer) } } + let keepRpuBytes = extractEL || extractParams + var rawRpuBytes: [UInt8]? + var convertedRpuBytes: [UInt8]? + var rpuParams: DoviMetalParams? var converted = false - var droppedEL = false + var isFEL = false + var isMEL = false + var hadEL = false var degraded = false + var usesPrevious = false + var disableResidual = false A53SEIParser.forEachNAL(data, size, framing) { nal, len in // HEVC NAL type: bits 1..6 of byte 0. @@ -70,6 +177,10 @@ public enum DoviRpuConverter { switch nalType { case nalTypeRPU: + // Everything this RPU contributes is collected in locals and committed together once it has + // converted, so a packet with several RPUs never mixes bytes from one with parameters or + // a conversion from another: the outputs are those of the last RPU that fully succeeded. + let thisRaw: [UInt8]? = keepRpuBytes ? [UInt8](UnsafeBufferPointer(start: nal, count: len)) : nil // Any libdovi failure drops the RPU (degraded) instead of muxing a stale P7 RPU // inside a container already declared 8.1 (mixed-profile hazard, #135). guard let rpu = DoviRpuGuard.parse(nal, len) else { @@ -77,6 +188,23 @@ public enum DoviRpuConverter { return } defer { dovi_rpu_free(rpu) } + let thisParams: DoviMetalParams? = extractParams ? DoviFelComposer.parseRpuParameters(rpu: rpu) : nil + var thisPrevious = false, thisDisableResidual = false, thisFEL = false, thisMEL = false + if let hdr = dovi_rpu_get_header(rpu) { + if hdr.pointee.use_prev_vdr_rpu_flag { thisPrevious = true } + thisDisableResidual = hdr.pointee.disable_residual_flag + if let elTypePtr = hdr.pointee.el_type { + switch String(cString: elTypePtr) { + case "FEL": thisFEL = true + case "MEL": thisMEL = true + default: break + } + } + // `disable_residual_flag` is per-RPU, not per-stream: it reaches the kernels as + // `DoviComposerFlags.disableResidual` (set by `parseRpuParameters` above) and must + // not flip the stream classification. + dovi_rpu_free_header(hdr) + } guard let out = Self.convertedUnspec62(rpu) else { degraded = true return @@ -87,64 +215,107 @@ public enum DoviRpuConverter { return } rewritten.append(out) - pieces.append(Piece(base: outData, count: out.pointee.len)) + if !useCustomBlockSource { + pieces.append(Piece(base: outData, count: out.pointee.len)) + } converted = true + rawRpuBytes = thisRaw + convertedRpuBytes = keepRpuBytes ? [UInt8](UnsafeBufferPointer(start: outData, count: out.pointee.len)) : nil + rpuParams = thisParams + usesPrevious = thisPrevious + disableResidual = thisDisableResidual + isFEL = thisFEL + isMEL = thisMEL + case nalTypeEL: - droppedEL = true + hadEL = true + // Strip the 2-byte unspec63 NAL header (0x7E, 0x01) to reveal the raw HEVC NAL. + if extractEL && len > 2 { + elPieces.append(Piece(base: nal + 2, count: len - 2)) + } default: - pieces.append(Piece(base: nal, count: len)) + if !useCustomBlockSource { + pieces.append(Piece(base: nal, count: len)) + } } } - if !converted && !droppedEL && !degraded { - return true - } - - var total = 0 - for piece in pieces { - total += lengthPrefixSize + piece.count - } - // Degenerate: all NALs were RPU/EL. Leave the packet untouched rather than emit a zero-length video packet. - guard total > 0 else { return true } - - let pad = Int(AV_INPUT_BUFFER_PADDING_SIZE) - guard let newRef = av_buffer_alloc(total + pad) else { - return false - } - guard let dst = newRef.pointee.data else { - var ref: UnsafeMutablePointer? = newRef - av_buffer_unref(&ref) - return false - } + var allocationFailed = false - let emitsStartCodes = framing == .annexB - var w = 0 - for piece in pieces { - let n = piece.count - if emitsStartCodes { - dst[w + 0] = 0; dst[w + 1] = 0; dst[w + 2] = 0; dst[w + 3] = 1 + // The EL packet is built BEFORE the base layer is rebuilt: its pieces point into the original buffer. + var producedElPacket: UnsafeMutablePointer? + let elTotal = framedSize(elPieces) + if extractEL, elTotal > 0 { + var elPktOpt = trackedPacketAlloc() + if let elPkt = elPktOpt, + let elRef = buildPacketBuffer(elPieces, total: elTotal, emitsStartCodes: framing == .annexB) { + elPkt.pointee.pts = packet.pointee.pts + elPkt.pointee.dts = packet.pointee.dts + elPkt.pointee.duration = packet.pointee.duration + elPkt.pointee.flags = packet.pointee.flags + elPkt.pointee.stream_index = packet.pointee.stream_index + elPkt.pointee.buf = elRef + elPkt.pointee.data = elRef.pointee.data + elPkt.pointee.size = Int32(elTotal) + producedElPacket = elPkt } else { - dst[w + 0] = UInt8((n >> 24) & 0xFF) - dst[w + 1] = UInt8((n >> 16) & 0xFF) - dst[w + 2] = UInt8((n >> 8) & 0xFF) - dst[w + 3] = UInt8(n & 0xFF) + // The packet shell allocated but its buffer did not: free the shell. + if elPktOpt != nil { trackedPacketFree(&elPktOpt) } + allocationFailed = true } - w += lengthPrefixSize - if n > 0 { - memcpy(dst + w, piece.base, n) + } + + // Rebuild the base layer (not with a custom block source: the FEL decoder needs the original + // packet untouched so VT can skip unspec62/63 NALs via the base-layer format description). + if (converted || hadEL || degraded), !useCustomBlockSource { + let total = framedSize(pieces) + // Degenerate: all NALs were RPU/EL. Leave the packet untouched rather than emit a zero-length video packet. + if total > 0 { + if let newRef = buildPacketBuffer(pieces, total: total, emitsStartCodes: framing == .annexB) { + av_buffer_unref(&packet.pointee.buf) + packet.pointee.buf = newRef + packet.pointee.data = newRef.pointee.data + packet.pointee.size = Int32(total) + } else { + allocationFailed = true + } } - w += n } - memset(dst + total, 0, pad) // decoders read past size - av_buffer_unref(&packet.pointee.buf) - packet.pointee.buf = newRef - packet.pointee.data = newRef.pointee.data - packet.pointee.size = Int32(total) + // A dropped-RPU degrade still produces a valid HDR10 packet, but reports failure so callers // can log it and the probe can flag the source for dovi_tool inspection. - return !degraded + return DualLayerPacketResult( + elPacket: producedElPacket, + rpuData: rawRpuBytes, + convertedRpu: convertedRpuBytes, + rpuParams: rpuParams, + isFEL: isFEL && !isMEL, + isMEL: isMEL, + success: !degraded && !allocationFailed, + usesPreviousMapping: usesPrevious, + disableResidual: disableResidual + ) + } + + /// Returns `false` when a libdovi conversion fails (or on an internal allocation failure). + /// On a libdovi failure the offending RPU (and EL) are dropped so the frame degrades to the + /// clean HDR10 base, rather than shipping a P7 RPU inside a container already declared 8.1 + /// (mixed-profile hazard, #135). Packets with no RPU/EL are left untouched and return `true`. + /// + /// `framing` must be the framing of THIS packet, and the rewritten packet is emitted in the same + /// one: the mp4 muxer decides whether to convert samples from the extradata, so a packet that + /// changes framing under it is a second defect (#365). An Annex-B packet walked as length- + /// prefixed reads `00 00 01 40` as a 320-byte NAL and finds no RPU at all, which is how a P7 + /// source shipped its RPU and EL untouched inside a container already rewritten to 8.1. + /// + /// This is `separateDualLayers` with the enhancement layer dropped and no composition parameters. + public static func convertPacketToProfile81( + _ packet: UnsafeMutablePointer, + framing: VideoNALFraming = .lengthPrefixed(size: 4) + ) -> Bool { + separateDualLayers(packet, framing: framing, extractEL: false, extractParams: false).success } /// Enhancement-layer type ("FEL" or "MEL") of the first unspec62 RPU in a P7 packet, or nil if diff --git a/Sources/AetherEngine/Video/Metal/DoviFelComposer.swift b/Sources/AetherEngine/Video/Metal/DoviFelComposer.swift new file mode 100644 index 000000000..18a5eae25 --- /dev/null +++ b/Sources/AetherEngine/Video/Metal/DoviFelComposer.swift @@ -0,0 +1,1452 @@ +import Foundation +import Metal +import CoreVideo +import CoreMedia +import Dovi +import os + +/// Hardware-accelerated Metal compute engine for Dolby Vision Profile 7 Full +/// Enhancement Layer (FEL). +/// +/// Reconstructs the VDR master from a 10-bit base layer and a 10-bit enhancement +/// layer: each component's base-layer sample goes through its piecewise +/// reshaping curve (polynomial or MMR), the inverse-NLQ residual decoded from the +/// enhancement layer is added, and the result is quantised back down to a 10-bit +/// video-range P010 CVPixelBuffer. +final class DoviFelComposer: @unchecked Sendable { + + /// `os_signpost` interval from "slot acquired, about to encode" to "GPU + /// retired, handing off to `OrderedDelivery`" (dolby-vision-fel.md §5.11), readable in + /// Instruments alongside `OrderedDelivery`'s own "Delivery" interval. + private static let signpostLog = OSLog(subsystem: EngineLog.subsystem, category: "dovi.compose") + + private let device: MTLDevice + private let commandQueue: MTLCommandQueue + private var textureCache: CVMetalTextureCache? + + /// Lazily compiled and process-shared compute pipelines (dolby-vision-fel.md §5.6). + private var pipelines: SharedPipelines? + /// Whether `dispatchThreads` is available. It needs non-uniform threadgroup + /// support (`MTLGPUFamily.apple4`, i.e. A11 and later; the first-generation + /// Apple TV 4K is A10X and is not); everything older has to dispatch whole + /// threadgroups and rely on the kernels' bounds guard. + private let supportsNonUniformThreadgroups: Bool + + private var pixelBufferPool: CVPixelBufferPool? + private var poolWidth: Int = 0 + private var poolHeight: Int = 0 + private var poolBuffers: [WeakPoolBuffer] = [] + private var poolBufferHighWaterMark = 0 + /// Set by ``releaseOutputPool()``. A composition arriving after teardown (a VideoToolbox callback still in + /// flight when the decoder closed) is refused instead of quietly building a new pool nobody will release. + private var tornDown = false + private let lock = NSLock() + + private final class WeakPoolBuffer { + weak var value: CVPixelBuffer? + init(_ value: CVPixelBuffer) { self.value = value } + } + + /// Preallocated Metal parameter buffers (one per in-flight slot) to avoid per-frame allocations (dolby-vision-fel.md §5.6). + private let paramBuffers: [MTLBuffer] + /// Preallocated 1024-entry luma prediction LUT buffers (dolby-vision-fel.md §5.6). + private let lumaLutBuffers: [MTLBuffer] + private var freeParamSlotIndices: [Int] + + /// Output pixel buffer pool status for 1 Hz summary diagnostics (dolby-vision-fel.md §5.11). + var allocatedPoolBuffers: Int { + lock.lock() + defer { lock.unlock() } + return poolBufferHighWaterMark + } + + /// Number of `uint` slots `dovi_layout_probe` writes. + static let layoutProbeSlots = 36 + 3 * 20 + + /// Largest `coefficient_log2_denom` the fixed-point path accepts. The RPU + /// syntax caps it at 23 (0x17), so a + /// larger value means a misparse rather than an exotic stream. + private static let maxCoefLog2Denom = 23 + + /// One-shot latch for the `linear_deadzone_threshold != 0` diagnostic. + /// + /// The inverse NLQ adds the threshold to the scaled magnitude before the shift, + /// as the spec's pseudo-code and DoViBaker do; see `dovi_eval_inv_nlq`. Every + /// stream available to test against carries a zero threshold, so a non-zero one + /// is the first content to actually exercise that term and is worth saying so + /// out loud exactly once. + nonisolated(unsafe) private static var loggedNonZeroDeadzone = false + private static let deadzoneLogLock = NSLock() + + /// Lines already logged by ``logOnce(_:)``. Bounded: a stream that varies the numbers in a message stops + /// being logged after `maxOnceLines` distinct lines instead of growing the set. + nonisolated(unsafe) private static var onceLines = Set() + private static let onceLock = NSLock() + private static let maxOnceLines = 64 + + /// Logs a parse-time rejection once per distinct line. `parseRpuParameters` runs for every RPU, so a stream + /// that trips one of these would otherwise repeat it on every frame. + private static func logOnce(_ line: String, category: EngineLog.Category = .swPlayback) { + onceLock.lock() + let fresh = onceLines.count < maxOnceLines && onceLines.insert(line).inserted + onceLock.unlock() + if fresh { EngineLog.emit(line, category: category) } + } + + /// Test seam: counts how many times `parseRpuParameters` has been called + /// (i.e., how many RPUs have been parsed). Reset with + /// `resetParseRpuCountForTesting()` between measurements. + /// Per thread, so concurrently running tests cannot disturb each other's count. + private static let parseCountKey = "DoviFelComposer.parseRpuCount" + private static var _parseRpuCount: Int { + get { Thread.current.threadDictionary[parseCountKey] as? Int ?? 0 } + set { Thread.current.threadDictionary[parseCountKey] = newValue } + } + static var parseRpuCountForTesting: Int { _parseRpuCount } + static func resetParseRpuCountForTesting() { _parseRpuCount = 0 } + + enum ComposerError: Error, Sendable { + case missingKernelFunction + } + + /// Process-shared compute pipelines compiled once per device (dolby-vision-fel.md §5.6). + final class SharedPipelines: @unchecked Sendable { + let lumaPipeline: MTLComputePipelineState + let chromaPipeline: MTLComputePipelineState + let layoutProbePipeline: MTLComputePipelineState? + + init( + luma: MTLComputePipelineState, + chroma: MTLComputePipelineState, + probe: MTLComputePipelineState? + ) { + self.lumaPipeline = luma + self.chromaPipeline = chroma + self.layoutProbePipeline = probe + } + + /// Per-device results, failures included: a shader that does not compile will not compile on the next + /// frame either, so the failure is kept and every later call fails fast instead of recompiling. + final class Cache: @unchecked Sendable { + private let lock = NSLock() + private var entries: [ObjectIdentifier: Result] = [:] + + func resolve(device: MTLDevice, compile: () throws -> SharedPipelines) throws -> SharedPipelines { + lock.lock() + defer { lock.unlock() } + let key = ObjectIdentifier(device) + if let entry = entries[key] { return try entry.get() } + let result = Result { try compile() } + entries[key] = result + return try result.get() + } + + func removeAll() { + lock.lock() + entries.removeAll() + lock.unlock() + } + } + + private static let shared = Cache() + + static func getOrCreate(device: MTLDevice) throws -> SharedPipelines { + try shared.resolve(device: device) { + do { + return try compile(device: device, source: DoviShaderSource.source) + } catch { + EngineLog.emit("[DoviFelComposer] Metal shader compilation error: \(error)", category: .swPlayback) + throw error + } + } + } + + static func compile(device: MTLDevice, source: String) throws -> SharedPipelines { + let library = try device.makeLibrary(source: source, options: nil) + guard let lumaFn = library.makeFunction(name: "dovi_compose_luma"), + let chromaFn = library.makeFunction(name: "dovi_compose_chroma") else { + throw ComposerError.missingKernelFunction + } + let luma = try device.makeComputePipelineState(function: lumaFn) + let chroma = try device.makeComputePipelineState(function: chromaFn) + let probe: MTLComputePipelineState? + if let probeFn = library.makeFunction(name: "dovi_layout_probe") { + probe = try? device.makeComputePipelineState(function: probeFn) + } else { + probe = nil + } + + return SharedPipelines(luma: luma, chroma: chroma, probe: probe) + } + + #if DEBUG + static func resetCacheForTesting() { + shared.removeAll() + } + #endif + } + + /// Lazily resolves or compiles the process-shared Metal compute pipelines (dolby-vision-fel.md §5.6). + private func ensurePipelines() -> SharedPipelines? { + // Under `lock` on every read: `pipelines` is written from whichever thread composes first, and a + // failed compile is cached by `SharedPipelines.Cache`, so a broken shader costs one compile, once. + lock.lock() + defer { lock.unlock() } + if let pipelines { return pipelines } + guard let p = try? SharedPipelines.getOrCreate(device: device) else { return nil } + self.pipelines = p + return p + } + + #if DEBUG + var lumaPipelineForTesting: MTLComputePipelineState? { + ensurePipelines()?.lumaPipeline + } + + var chromaPipelineForTesting: MTLComputePipelineState? { + ensurePipelines()?.chromaPipeline + } + #endif + + init?() { + guard let dev = MTLCreateSystemDefaultDevice(), + let queue = dev.makeCommandQueue() else { + return nil + } + self.device = dev + self.commandQueue = queue + self.supportsNonUniformThreadgroups = dev.supportsFamily(.apple4) || dev.supportsFamily(.mac2) + + var cache: CVMetalTextureCache? + guard CVMetalTextureCacheCreate(kCFAllocatorDefault, nil, dev, nil, &cache) == kCVReturnSuccess, + let createdCache = cache else { + return nil + } + self.textureCache = createdCache + + var pBuffers: [MTLBuffer] = [] + var lutBuffers: [MTLBuffer] = [] + for _ in 0...stride, options: .storageModeShared) else { + return nil + } + pBuffers.append(buf) + lutBuffers.append(lut) + } + self.paramBuffers = pBuffers + self.lumaLutBuffers = lutBuffers + self.freeParamSlotIndices = Array(0.. CVPixelBufferPool? { + lock.lock() + defer { lock.unlock() } + + // Checked under the same lock `releaseOutputPool()` takes, so a teardown landing between a caller's + // earlier look and this point cannot leave a fresh pool behind that nothing will release. + guard !tornDown else { return nil } + + if let pool = pixelBufferPool, poolWidth == width, poolHeight == height { + return pool + } + + let poolAttributes: [CFString: Any] = [ + kCVPixelBufferPoolMinimumBufferCountKey: 4, + kCVPixelBufferPoolMaximumBufferAgeKey: 1.0, + kCVPixelBufferPoolAllocationThresholdKey: DoviFelComposer.outputPoolAllocationThreshold + ] + + let pixelBufferAttributes: [CFString: Any] = [ + kCVPixelBufferPixelFormatTypeKey: kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange, + kCVPixelBufferWidthKey: width, + kCVPixelBufferHeightKey: height, + kCVPixelBufferIOSurfacePropertiesKey: [:] as [String: Any], + kCVPixelBufferMetalCompatibilityKey: true + ] + + var newPool: CVPixelBufferPool? + let status = CVPixelBufferPoolCreate( + kCFAllocatorDefault, + poolAttributes as CFDictionary, + pixelBufferAttributes as CFDictionary, + &newPool + ) + + guard status == kCVReturnSuccess, let pool = newPool else { + return nil + } + + self.pixelBufferPool = pool + self.poolWidth = width + self.poolHeight = height + poolBuffers.removeAll() + poolBufferHighWaterMark = 0 + return pool + } + + private func recordPoolBuffer(_ buffer: CVPixelBuffer, pool: CVPixelBufferPool) { + lock.lock() + defer { lock.unlock() } + guard pool === pixelBufferPool else { return } + poolBuffers.removeAll { $0.value == nil } + if !poolBuffers.contains(where: { $0.value === buffer }) { + poolBuffers.append(WeakPoolBuffer(buffer)) + } + poolBufferHighWaterMark = max(poolBufferHighWaterMark, poolBuffers.count) + } + + #if DEBUG + func ensurePoolForTesting(width: Int, height: Int) -> CVPixelBufferPool? { + ensurePool(width: width, height: height) + } + + /// One allocation from the output pool: `reserve: false` stops at the threshold, `true` draws on the reserve + /// above it, as `encodeCompose` does once the threshold refuses. + func createOutputBufferForTesting(pool: CVPixelBufferPool, reserve: Bool = false) -> (status: CVReturn, buffer: CVPixelBuffer?) { + var buf: CVPixelBuffer? + let status = CVPixelBufferPoolCreatePixelBufferWithAuxAttributes( + kCFAllocatorDefault, + pool, + reserve ? Self.poolReserveAuxAttributes : Self.poolAuxAttributes, + &buf + ) + if status == kCVReturnSuccess, let buf { recordPoolBuffer(buf, pool: pool) } + return (status, buf) + } + #endif + + // MARK: - Texture Extraction + + private func makeTexture( + from pixelBuffer: CVPixelBuffer, + plane: Int, + format: MTLPixelFormat, + width: Int, + height: Int, + keepAlive: inout [CVMetalTexture] + ) -> MTLTexture? { + guard let cache = textureCache else { return nil } + var cvTex: CVMetalTexture? + let status = CVMetalTextureCacheCreateTextureFromImage( + kCFAllocatorDefault, + cache, + pixelBuffer, + nil, + format, + width, + height, + plane, + &cvTex + ) + guard status == kCVReturnSuccess, let validCvTex = cvTex else { + return nil + } + keepAlive.append(validCvTex) + return CVMetalTextureGetTexture(validCvTex) + } + + // MARK: - Frame Composition + + /// Reconstructs the VDR master from BL and EL pixel buffers, returning a + /// 10-bit video-range P010 CVPixelBuffer. + /// + /// Blocks the calling thread until the GPU is done. That is what an offline + /// caller wants — a probe, a unit test, `aetherctl` — but it is the wrong + /// shape for playback, where the wait lands on the decode thread and stops + /// decode(N+1) from overlapping GPU(N). Playback uses + /// ``composeAsync(baseBuffer:enhancementBuffer:params:frameIndex:completion:)``. + func compose( + baseBuffer: CVPixelBuffer, + enhancementBuffer: CVPixelBuffer, + params: DoviMetalParams, + frameIndex: UInt32 = 0 + ) -> CVPixelBuffer? { + // Takes an in-flight slot like `composeAsync`, so a parameter slot is never taken outside the semaphore + // that bounds them. Only tests, `aetherctl` and probes call this; playback uses `composeAsync`. + acquireSlot() + defer { releaseInFlightSlot() } + guard let work = encodeCompose(baseBuffer: baseBuffer, + enhancementBuffer: enhancementBuffer, + params: params, + frameIndex: frameIndex) else { + return nil + } + defer { returnParamSlot(work.paramSlot) } + work.commandBuffer.commit() + work.commandBuffer.waitUntilCompleted() + if let failure = Self.commandBufferFailure(work.commandBuffer, forced: forcedCommandBufferFailure.withLock { $0 }) { + // The GPU did not finish writing the destination: it is not a frame, so no frame is returned. + EngineLog.emit("[DoviFelComposer] Command buffer error: \(failure)", category: .swPlayback) + return nil + } + _ = work.keepAlive + // Colour attachments are copied in `encodeCompose`, at encode time on + // this calling thread — nothing left to do here. + return work.destination + } + + /// Test seam: makes every command buffer count as failed, so a test can drive the failure path without + /// a GPU fault. Read once per command buffer; a plain unfair lock, never held across anything. + let forcedCommandBufferFailure = OSAllocatedUnfairLock(initialState: false) + + /// The command buffer's error, as text, or nil when it completed cleanly. + private static func commandBufferFailure(_ cb: MTLCommandBuffer, forced: Bool) -> String? { + forced ? "forced failure (test)" : cb.error?.localizedDescription + } + + /// Same composition, without blocking the caller on the GPU. + /// + /// `completion` is invoked exactly once — off the calling thread once the + /// command buffer retires, or inline with nil if the frame could not be set + /// up at all. The GPU's writes are complete before it fires, so the buffer + /// it hands over is safe to enqueue for display. + /// + /// At most ``maxFramesInFlight`` compositions are outstanding on the GPU; a + /// caller that runs further ahead than that blocks here until one retires. + /// That is the backpressure that keeps the pixel buffer pool from growing + /// without bound, and it is deliberately the decode thread that absorbs it. + /// The slot is released when the command buffer retires rather than when + /// `completion` returns, so a slow consumer can leave at most + /// ``maxFramesInFlight`` further frames queued for delivery behind it. That + /// asymmetry is required, not incidental: releasing the slot from the + /// delivery side would put the signal behind the caller's own lock. + func composeAsync( + baseBuffer: CVPixelBuffer, + enhancementBuffer: CVPixelBuffer, + params: DoviMetalParams, + frameIndex: UInt32 = 0, + completion: @escaping @Sendable (CVPixelBuffer?) -> Void + ) { + acquireSlot() + composeAsyncWithAcquiredSlot( + baseBuffer: baseBuffer, + enhancementBuffer: enhancementBuffer, + params: params, + frameIndex: frameIndex, + completion: completion + ) + } + + /// Blocks until one of ``maxFramesInFlight`` compositions is free. + /// + /// dolby-vision-fel.md §5.6: the decoder calls this with no lock held, on the + /// VideoToolbox output thread that produced the `compose` action, so the + /// backpressure lands on that thread instead of stalling `deliver`, the + /// other layer's callback, `flush()` and `close()` behind the decoder lock. + func acquireSlot() { + inFlight.wait() + slotLock.lock() + activeSlotsInUse += 1 + if activeSlotsInUse > peakSlotsInUse { + peakSlotsInUse = activeSlotsInUse + } + slotLock.unlock() + } + + /// Releases one in-flight slot. Only for callers that acquired with + /// ``acquireSlot()`` and then failed before reaching + /// ``composeAsyncWithAcquiredSlot(baseBuffer:enhancementBuffer:params:frameIndex:completion:)``. + func releaseSlot() { + releaseInFlightSlot() + } + + private func releaseInFlightSlot() { + slotLock.lock() + if activeSlotsInUse > 0 { + activeSlotsInUse -= 1 + } + slotLock.unlock() + inFlight.signal() + } + + /// Reads and resets the peak slot count for the 1 Hz summary (dolby-vision-fel.md §5.11). + func takePeakSlotsInUse() -> Int { + slotLock.lock() + defer { slotLock.unlock() } + let peak = peakSlotsInUse + peakSlotsInUse = activeSlotsInUse + return peak + } + + private func takeParamSlot() -> Int? { + slotLock.lock() + defer { slotLock.unlock() } + guard !freeParamSlotIndices.isEmpty else { return nil } + return freeParamSlotIndices.removeLast() + } + + private func returnParamSlot(_ slot: Int) { + slotLock.lock() + freeParamSlotIndices.append(slot) + slotLock.unlock() + } + + /// Same composition, assuming the caller already holds a slot from + /// ``acquireSlot()``. The slot is released when the command buffer retires + /// (or inline with `completion(nil)` when the frame cannot be set up). + func composeAsyncWithAcquiredSlot( + baseBuffer: CVPixelBuffer, + enhancementBuffer: CVPixelBuffer, + params: DoviMetalParams, + frameIndex: UInt32 = 0, + completion: @escaping @Sendable (CVPixelBuffer?) -> Void + ) { + let signpostID = OSSignpostID(log: Self.signpostLog) + os_signpost(.begin, log: Self.signpostLog, name: "Compose", signpostID: signpostID, + "frame=%{public}u", frameIndex) + guard let work = encodeCompose(baseBuffer: baseBuffer, + enhancementBuffer: enhancementBuffer, + params: params, + frameIndex: frameIndex) else { + os_signpost(.end, log: Self.signpostLog, name: "Compose", signpostID: signpostID, "encode-failed") + releaseInFlightSlot() + completion(nil) + return + } + // `textures` holds the CVMetalTextures; the source buffers are held + // alongside them because a CVMetalTexture is not documented to keep the + // pixel buffer it was made from alive, and the GPU reads both until the + // command buffer retires. Boxed because CoreVideo's types are not + // `Sendable` and this crosses two queues: CoreVideo buffers are + // themselves thread-safe, and only one queue touches the box at a time. + let frame = ComposedFrame(destination: work.destination, + source: baseBuffer, + enhancement: enhancementBuffer, + textures: work.keepAlive, + cache: textureCache) + // Captured as a value, not through `self?.`, before the handler below is + // even built: the whole point of `deliveryQueue` is that calling + // `completion` never depends on the composer still being alive, so there + // is no `self?.deliveryQueue ?? .global()`-shaped fallback to reach for + // if it were. `DispatchQueue` is a plain `Sendable` + // reference, so this costs nothing. + let deliveryQueue = self.deliveryQueue + let paramSlot = work.paramSlot + // Held strongly so it outlives a composer that is released mid-flight: libdispatch traps + // if a semaphore is destroyed below its initial count, and `self` is weak in the handler. + let inFlightSemaphore = self.inFlight + let forcedFailure = self.forcedCommandBufferFailure + work.commandBuffer.addCompletedHandler { [weak self] cb in + // Metal runs these handlers on ONE serial queue per command queue, so + // this block must never take a lock the caller above might be holding + // and must never block (dolby-vision-fel.md §4.3). The caller is inside + // `inFlight.wait()` while it holds the decoder's lock, and a `completion` + // that reached for that same lock would block here, which would + // stop every LATER handler from running, and one of those is the + // signal the caller is waiting for. Signalling first only defers that + // by a frame - it deadlocked on device after ~10s of playback. + // + // So: release the slot inline — nothing below this line can block — + // then hop to `deliveryQueue` for everything else, `completion` + // included, so Metal's shared queue never waits on it no matter what + // it does. In production `completion` is `DualLayerVideoDecoder. + // runCompose`'s closure, itself just a `queue.async` into + // `OrderedDelivery.fulfil`, so this hop is normally a hand-off from + // one non-blocking queue to another; `DoviComposeAsyncTests. + // doesNotDeadlockAgainstTheCallersLock` is what this hop exists for — + // it hands `composeAsync` a `completion` that deliberately blocks on + // a lock the caller holds, and only survives because that block runs + // off Metal's queue. + // + // This event marks the instant + // Metal actually invoked this handler — i.e. "GPU retired" — + // distinct from the "Compose" interval's `.end`, which only + // fires later, after the hop below has run. Without it, nothing + // in the signposts brackets the hand-off from Metal's queue to + // `deliveryQueue` on its own: Instruments could see "Compose" + // ran long but not whether that was GPU time or queueing delay + // hopping to the relay. It has to be the very first thing here, + // ahead of even `inFlight.signal()`, so it costs nothing on the + // must-never-block path (`os_signpost` does not block) and is + // not itself delayed by anything below it. + os_signpost(.event, log: Self.signpostLog, name: "Compose", signpostID: signpostID, "metal-completion-fired") + if let strongSelf = self { + strongSelf.returnParamSlot(paramSlot) + strongSelf.releaseInFlightSlot() + } else { + inFlightSemaphore.signal() + } + // Everything the hop needs pulled out of `cb` here, as plain + // Sendable values: `cb` itself is `any MTLCommandBuffer`, not + // Sendable, so it cannot cross into `deliveryQueue.async`'s + // closure directly, and `self` is captured once, strongly, into a + // fresh local rather than referenced from inside the nested + // closure (which is what the "captured var 'self' in + // concurrently-executing code" warning was about). + let errorDescription = Self.commandBufferFailure(cb, forced: forcedFailure.withLock { $0 }) + let gpuMs = (cb.gpuEndTime - cb.gpuStartTime) * 1000 + let composer = self + deliveryQueue.async { + if let errorDescription { + EngineLog.emit("[DoviFelComposer] Command buffer error: \(errorDescription)", category: .swPlayback) + } + composer?.recordGPUTime(ms: gpuMs) + os_signpost(.end, log: Self.signpostLog, name: "Compose", signpostID: signpostID) + // A command buffer that errored leaves the destination partly written: deliver a failure, + // so the caller falls back to the base layer instead of showing garbage. + completion(errorDescription == nil ? frame.destination : nil) + frame.release() + } + } + work.commandBuffer.commit() + } + + /// Compositions allowed to be outstanding at once. Sized against the pool's + /// `kCVPixelBufferPoolMinimumBufferCountKey` so a full pipeline still draws + /// from the preallocated set rather than growing the pool. + static let maxFramesInFlight = 3 + /// Buffers the output pool allocates in normal running; past it `encodeCompose` draws on + /// `outputPoolReserve`, and only with that spent too does a frame bypass to the base layer + /// (dolby-vision-fel.md §5.6, §5.10). + /// + /// Derived from the renderer's look-ahead cap, not a second constant to keep in step: when decode + /// outruns the clock the gate holds the look-ahead at its limit, and the pool then holds that many + /// frames plus the one on screen, the one the display layer retains, and the compositions in flight. + /// + /// The margin is built from those holders: the one on screen, the one the layer retains, + /// `maxFramesInFlight` compositions, and one spare. A smaller margin let a burst of in-flight compositions + /// on top of a full look-ahead exhaust the pool (CoreVideo -6689) and fall back to the base layer. + /// A 4K P010 buffer is about 25 MB, so the threshold is about 375 MB. + private static let residualSwitch = OSAllocatedUnfairLock(initialState: true) + + /// Whether the enhancement-layer residual (EL fetch + NLQ) is applied. Off, the composer still maps the + /// base layer through the RPU curves, which is what a minimal enhancement layer (MEL) stream decodes + /// to; on, it is the full FEL result. Exists for A/B viewing (`aetherctl play --cycle-el`); it takes + /// effect from the next composed frame and applies to every composer in the process. Default on. + static var residualEnabled: Bool { + get { residualSwitch.withLock { $0 } } + set { residualSwitch.withLock { $0 = newValue } } + } + + static let outputPoolAllocationThreshold = SampleBufferRenderer.felVideoLookaheadFrames + 2 + maxFramesInFlight + 1 + nonisolated(unsafe) private static let poolAuxAttributes: CFDictionary = [ + kCVPixelBufferPoolAllocationThresholdKey: outputPoolAllocationThreshold + ] as CFDictionary + /// Buffers the pool may grow by when the threshold refuses one. A refused buffer means a base-layer + /// frame, a visible brightness step on real titles (`fallbackFrameDelta`); a burst composes into one of + /// these instead. They return to the pool like any other and `kCVPixelBufferPoolMaximumBufferAgeKey` + /// frees them after a second unused, so the extra memory (about 25 MB each at 4K) lasts only as long as the + /// burst. Sized like the in-flight limit, the holder a burst's shortfall came from on the titles measured. + static let outputPoolReserve = maxFramesInFlight + /// With the pool and its reserve both refused, the composer retries this many times, sleeping + /// `outputPoolRetryInterval` between attempts (never after the last): at most 9 x 4 ms, about 36 ms. + static let outputPoolAttempts = 10 + static let outputPoolRetryInterval: TimeInterval = 0.004 + nonisolated(unsafe) private static let poolReserveAuxAttributes: CFDictionary = [ + kCVPixelBufferPoolAllocationThresholdKey: outputPoolAllocationThreshold + outputPoolReserve + ] as CFDictionary + private let inFlight = DispatchSemaphore(value: DoviFelComposer.maxFramesInFlight) + private let slotLock = NSLock() + private var activeSlotsInUse: Int = 0 + private var peakSlotsInUse: Int = 0 + + /// Exists for exactly one reason (dolby-vision-fel.md §4.3): Metal runs every + /// `addCompletedHandler` block for a command queue on **one serial queue**, + /// so a `completion` closure that blocks — on a caller's lock, or on + /// anything else — would stall every later handler too, including the + /// `inFlight.signal()` a different, currently-blocked caller is waiting on. + /// `DoviComposeAsyncTests.doesNotDeadlockAgainstTheCallersLock` reproduces + /// that shape directly against `composeAsync`, independent of anything + /// `DualLayerVideoDecoder`/`OrderedDelivery` promise about their own + /// callback never blocking — this queue is what makes the composer safe + /// against a slow or badly-behaved `completion` on its own terms. + /// + /// This is not a second ordering mechanism: this queue carries no + /// ordering responsibility at all (`OrderedDelivery` reorders by ticket + /// regardless of which thread calls `fulfil`), it only gets `completion` + /// off Metal's queue. It runs at `qos: .userInteractive`, and it is + /// captured as a plain value below, never through `self?.`, so there is + /// no unordered `.global()` fallback to reach for. + private let deliveryQueue = DispatchQueue(label: "de.superuser404.dovi.compose-delivery", qos: .userInteractive) + + /// One composition's CoreVideo objects, kept alive from encode until the + /// completion has run. `@unchecked Sendable` for the reason given at its + /// construction site in `composeAsync`. + private final class ComposedFrame: @unchecked Sendable { + let destination: CVPixelBuffer + private var source_: CVPixelBuffer? + private var enhancement_: CVPixelBuffer? + private var textures_: [CVMetalTexture]? + /// A `CVMetalTexture` calls back into its cache when it finalizes and does not + /// retain it, so the cache has to outlive the textures. Without this a composer + /// released while a command buffer was in flight crashed in + /// `CVMetalTextureCache::bufferBackingNotInUse`. + private var cache_: CVMetalTextureCache? + + var source: CVPixelBuffer { source_! } + + init(destination: CVPixelBuffer, source: CVPixelBuffer, + enhancement: CVPixelBuffer, textures: [CVMetalTexture], + cache: CVMetalTextureCache?) { + self.destination = destination + self.cache_ = cache + self.source_ = source + self.enhancement_ = enhancement + self.textures_ = textures + } + + /// Drops the inputs once the consumer has the output. Explicit so the + /// lifetime is a statement rather than a `_ =` that a later edit can + /// delete without noticing what it was holding up. + func release() { + source_ = nil + enhancement_ = nil + textures_ = nil + cache_ = nil + } + } + + private struct EncodedFrame { + let commandBuffer: MTLCommandBuffer + let destination: CVPixelBuffer + let keepAlive: [CVMetalTexture] + let paramSlot: Int + } + + /// Everything up to (not including) `commit`: pool rental, texture binding, + /// parameter upload and the two dispatches. + private func encodeCompose( + baseBuffer: CVPixelBuffer, + enhancementBuffer: CVPixelBuffer, + params: DoviMetalParams, + frameIndex: UInt32 + ) -> EncodedFrame? { + let blW = CVPixelBufferGetWidth(baseBuffer) + let blH = CVPixelBufferGetHeight(baseBuffer) + let elW = CVPixelBufferGetWidth(enhancementBuffer) + let elH = CVPixelBufferGetHeight(enhancementBuffer) + + guard blW > 0, blH > 0, elW > 0, elH > 0 else { return nil } + guard let pool = ensurePool(width: blW, height: blH) else { return nil } + + // A base-layer frame is NOT close to the composed one on real titles: measured on the three + // real FEL files its 10-bit luma differs by a mean of ~8 codes (p99 up to 230) on two and ~79 + // on clip C (`fallbackFrameDelta`), a visible brightness step. So when the threshold + // refuses, take a buffer from the reserve above it rather than wait. Only with the reserve + // spent too, wait briefly, but never long: on device a 300 ms wait stalled the + // VideoToolbox callback, then the demux loop, starving audio into 7 rebuffers in a minute. + var outBuffer: CVPixelBuffer? + var poolStatus = kCVReturnError + for attempt in 0.., params: DoviMetalParams) { + let poly = params.polyY + let blMax = Float(params.blMax) + 1 // normalise by 2^bits, as the kernels do + guard poly.numPivots >= 2 else { + for code in 0..<1024 { + ptr[code] = Float(code) / blMax + } + return + } + + let numPivots = Int(poly.numPivots) + let lastPiece = numPivots - 2 + let lo = Int(poly.pivots[0]) + let hi = Int(poly.pivots[numPivots - 1]) + + for code in 0..<1024 { + var piece = 0 + for i in 0...lastPiece { + if code >= Int(poly.pivots[i]) { + piece = i + } + } + let p = max(0, min(piece, lastPiece)) + let method = (p < DoviPolynomialParams.maxPieces && p < poly.method.count) ? poly.method[p] : DoviMapping.polynomial.rawValue + if method != DoviMapping.polynomial.rawValue { + ptr[code] = Float(code) / blMax + continue + } + + let clamped = min(max(code, lo), hi) + let x = Float(clamped) / blMax + let shift = (p < poly.coefShift.count) ? Int32(poly.coefShift[p]) : 0 + let inv = ldexpf(1.0, -shift) + let coefs = (p < poly.coef.count) ? poly.coef[p] : [0, 0, 0] + let c0 = (coefs.count > 0) ? Float(coefs[0]) : 0 + let c1 = (coefs.count > 1) ? Float(coefs[1]) : 0 + let c2 = (coefs.count > 2) ? Float(coefs[2]) : 0 + + var pred = c0 * inv + let order = (p < poly.order.count) ? poly.order[p] : 0 + if order >= 1 { + pred = fmaf(c1 * inv, x, pred) + } + if order >= 2 { + pred = fmaf(c2 * inv, x * x, pred) + } + ptr[code] = pred + } + } + + /// Sizes threadgroups using the pipeline's execution width (dolby-vision-fel.md §5.6). + private func threadgroupSize(for pipeline: MTLComputePipelineState) -> MTLSize { + let w = pipeline.threadExecutionWidth + let maxThreads = pipeline.maxTotalThreadsPerThreadgroup + if w >= 32 { + let h = max(1, min(maxThreads / w, 8)) + return MTLSize(width: w, height: h, depth: 1) + } else { + return MTLSize(width: 16, height: 16, depth: 1) + } + } + + private func propagateColorAttachments(from baseBuffer: CVPixelBuffer, to dst: CVPixelBuffer) { + CVBufferPropagateAttachments(baseBuffer, dst) + } + + // MARK: - GPU Timing + + /// Mean GPU time per composition over the last `gpuTimeWindow` frames, in + /// milliseconds, or nil before the first window closes. + /// + /// The wall-clock cost of a frame is GPU time plus texture binding, pool + /// rental and encode; only this separates them, which is what says whether + /// the shaders need work or the CPU side does. + var meanGPUTimeMs: Double? { + gpuTimeLock.lock(); defer { gpuTimeLock.unlock() } + if let gpuTimeMean { return gpuTimeMean } + if gpuTimeCount > 0 { return gpuTimeAccum / Double(gpuTimeCount) } + return nil + } + + private static let gpuTimeWindow = 120 + private let gpuTimeLock = NSLock() + private var gpuTimeAccum: Double = 0 + private var gpuTimeCount: Int = 0 + private var gpuTimeMean: Double? + + private func recordGPUTime(ms: Double) { + guard ms.isFinite, ms >= 0 else { return } + gpuTimeLock.lock() + gpuTimeAccum += ms + gpuTimeCount += 1 + var closed: Double? + if gpuTimeCount >= Self.gpuTimeWindow { + closed = gpuTimeAccum / Double(gpuTimeCount) + gpuTimeMean = closed + gpuTimeAccum = 0 + gpuTimeCount = 0 + } + gpuTimeLock.unlock() + if let closed { + EngineLog.emit(String(format: "[DoviFelComposer] gpu %.1fms/frame over %d frames", + closed, Self.gpuTimeWindow), category: .swPlayback) + } + } + + // MARK: - Flush & Invalidation + + /// Flushes the texture cache, releasing cached IOSurface-backed Metal textures (dolby-vision-fel.md §5.6). + func flushTextureCache() { + lock.lock() + defer { lock.unlock() } + if let cache = textureCache { + CVMetalTextureCacheFlush(cache, 0) + } + } + + /// Seek/flush: keeps the output pool. Dropping it here made the next frames allocate a second pool while + /// the renderer still held the old one's buffers (the displayed image and the look-ahead the display layer + /// had not yet let go of), so the footprint doubled for the half second a seek took to settle (about + /// +260 MB at 4K, where the pool holds up to 15 buffers of 25 MB). The same pool is capped by the allocation threshold, + /// so the old buffers return to it as the renderer releases them and the new frames reuse them. Idle + /// buffers beyond the minimum are released at once. + func flush() { + lock.lock() + defer { lock.unlock() } + if let cache = textureCache { + CVMetalTextureCacheFlush(cache, 0) + } + if let pool = pixelBufferPool { + CVPixelBufferPoolFlush(pool, .excessBuffers) + } + } + + /// Teardown: drops the pool so its idle buffers go with it, and marks the composer as torn down: every + /// later composition is refused (`compose` returns nil, `composeAsync` completes with nil and gives its + /// slot back), so no pool is rebuilt after this. One composer serves one decoder, which is closed once. + func releaseOutputPool() { + lock.lock() + defer { lock.unlock() } + tornDown = true + pixelBufferPool = nil + poolBuffers.removeAll() + poolBufferHighWaterMark = 0 + poolWidth = 0 + poolHeight = 0 + } + + // MARK: - ABI Layout Probe + + /// Uploads `params` exactly the way ``compose(baseBuffer:enhancementBuffer:params:frameIndex:)`` + /// does and returns the sentinels `dovi_layout_probe` reads back out of it. + /// + /// Metal has no `static_assert` and the offline `metal` compiler is not always + /// installed, so this is how the shader's view of the parameter block gets + /// checked against ``DoviMetalParams/serialize()`` rather than by hand. + /// Returns nil when the probe kernel is unavailable. + func probeLayout(params: DoviMetalParams) -> [UInt32]? { + guard let pipelines = ensurePipelines(), let pipeline = pipelines.layoutProbePipeline else { return nil } + + let slots = Self.layoutProbeSlots + guard let outBuffer = device.makeBuffer(length: slots * 4, options: .storageModeShared) else { + return nil + } + memset(outBuffer.contents(), 0, slots * 4) + + guard let commandBuffer = commandQueue.makeCommandBuffer(), + let encoder = commandBuffer.makeComputeCommandEncoder() else { + return nil + } + encoder.setComputePipelineState(pipeline) + let paramData = params.serialize() + paramData.withUnsafeBytes { raw in + if let ptr = raw.baseAddress { + encoder.setBytes(ptr, length: raw.count, index: 0) + } + } + encoder.setBuffer(outBuffer, offset: 0, index: 1) + dispatch(encoder, width: 1, height: 1, + threadgroup: MTLSize(width: 1, height: 1, depth: 1)) + encoder.endEncoding() + commandBuffer.commit() + commandBuffer.waitUntilCompleted() + guard commandBuffer.error == nil else { return nil } + + let ptr = outBuffer.contents().bindMemory(to: UInt32.self, capacity: slots) + return Array(UnsafeBufferPointer(start: ptr, count: slots)) + } + + /// Covers exactly `width` x `height` threads. + /// + /// `dispatchThreads` takes the thread count and splits the edge threadgroups + /// itself, so the grid is the plane. Rounding the grid up to a multiple of the + /// threadgroup is the `dispatchThreadgroups` convention, and that is the path + /// taken where non-uniform threadgroups are unavailable: there the extra + /// threads are real and the kernels' `gid >= bl_width` guard is what discards + /// them. + private func dispatch( + _ encoder: MTLComputeCommandEncoder, + width: Int, + height: Int, + threadgroup: MTLSize + ) { + guard width > 0, height > 0 else { return } + if supportsNonUniformThreadgroups { + encoder.dispatchThreads( + MTLSize(width: width, height: height, depth: 1), + threadsPerThreadgroup: threadgroup + ) + } else { + encoder.dispatchThreadgroups( + MTLSize( + width: (width + threadgroup.width - 1) / threadgroup.width, + height: (height + threadgroup.height - 1) / threadgroup.height, + depth: 1 + ), + threadsPerThreadgroup: threadgroup + ) + } + } + + // MARK: - RPU Parsing via libdovi + + /// Whether the base and enhancement layers are the 10-bit codes the kernels assume. The kernels read and + /// write 10-in-16 P010 planes, so a base layer of any other depth would need different texture formats, not + /// just a new divisor; the enhancement layer is bound as a 10-in-16 plane and the NLQ constants assume a + /// 10-bit code, so an EL of any other depth would be read at the wrong scale. libdovi 3.4 already rejects + /// both at parse time, which is why this is separate from the parse and tested directly: the check keeps the + /// composer safe if a later libdovi stops enforcing it. + static func layerBitDepthsSupported(bl: Int, el: Int) -> Bool { + guard bl == 10 else { + logOnce("[DoviFelComposer] bl_bit_depth \(bl) unsupported; refusing to compose", category: .swPlayback) + return false + } + guard el == 10 else { + logOnce("[DoviFelComposer] el_bit_depth \(el) unsupported; refusing to compose", category: .swPlayback) + return false + } + return true + } + + /// Parses an unspec62 RPU NAL into composer parameters. + /// + /// Returns nil when the RPU cannot be trusted at all (no header, a + /// coefficient encoding the kernels do not implement, a bit depth they cannot + /// address), so the caller presents the base layer rather than a guess. A + /// single unusable component instead degrades to an identity curve for that + /// component, which is visually the base layer for that channel and never + /// flat mid-grey. + static func parseRpuParameters(rpu: OpaquePointer) -> DoviMetalParams? { + _parseRpuCount += 1 + + if let err = dovi_rpu_get_error(rpu) { + logOnce("[DoviFelComposer] libdovi error: \(String(cString: err))", category: .swPlayback) + } + + guard let mapping = dovi_rpu_get_data_mapping(rpu) else { + logOnce("[DoviFelComposer] dovi_rpu_get_data_mapping returned nil", category: .swPlayback) + return nil + } + defer { dovi_rpu_free_data_mapping(mapping) } + + guard let header = dovi_rpu_get_header(rpu) else { + logOnce("[DoviFelComposer] dovi_rpu_get_header returned nil; refusing to guess bit depths", category: .swPlayback) + return nil + } + defer { dovi_rpu_free_header(header) } + + var params = DoviMetalParams() + + // coefficient_data_type selects fixed point (0) or 32-bit float (1). Every + // coefficient below is decoded as (int << denom) + frac, which is only the + // fixed-point reading, so refuse the float form instead of misreading it. + let coefDataType = header.pointee.coefficient_data_type + guard coefDataType == 0 else { + logOnce("[DoviFelComposer] coefficient_data_type \(coefDataType) is not fixed point; refusing to compose", category: .swPlayback) + return nil + } + + let coefLog2DenomRaw = header.pointee.coefficient_log2_denom + guard coefLog2DenomRaw <= UInt64(maxCoefLog2Denom) else { + logOnce("[DoviFelComposer] coefficient_log2_denom \(coefLog2DenomRaw) out of range; refusing to compose", category: .swPlayback) + return nil + } + let coefLog2Denom = Int(coefLog2DenomRaw) + + let blBitDepth = Int(header.pointee.bl_bit_depth_minus8) + 8 + let vdrBitDepth = Int(header.pointee.vdr_bit_depth_minus8) + 8 + let elBitDepth = Int(header.pointee.el_bit_depth_minus8) + 8 + guard layerBitDepthsSupported(bl: blBitDepth, el: elBitDepth) else { return nil } + guard params.setBitDepths(bl: blBitDepth, vdr: vdrBitDepth) else { + logOnce("[DoviFelComposer] bit depth pair bl=\(blBitDepth) vdr=\(vdrBitDepth) unsupported; refusing to compose", category: .swPlayback) + return nil + } + + if header.pointee.disable_residual_flag { + params.flags.insert(.disableResidual) + } + if let elType = header.pointee.el_type, strcmp(elType, "MEL") == 0 { + // MEL carries no residual by construction: the enhancement layer is a + // constant mid-grey and only the RPU matters. + params.flags.insert(.disableResidual) + } + + // The inverse NLQ product is Q(coefficient_log2_denom) of a normalised + // residual; this shift brings it into VDR code units. A denominator below + // the VDR depth would make the shift negative and the whole derivation + // meaningless, so that disables the residual rather than shifting the + // other way. + let nlqShift = UInt32(max(0, coefLog2Denom - vdrBitDepth)) + let nlqMethodIdc = mapping.pointee.nlq_method_idc + var nlqMethod: DoviNlqMethod = .none + if coefLog2Denom < vdrBitDepth { + logOnce("[DoviFelComposer] coefficient_log2_denom \(coefLog2Denom) below vdr_bit_depth \(vdrBitDepth); dropping residual", category: .swPlayback) + } else if nlqMethodIdc == 0 { + nlqMethod = .linear + } else if nlqMethodIdc > 0 { + logOnce("[DoviFelComposer] nlq_method_idc \(nlqMethodIdc) not implemented; dropping residual", category: .swPlayback) + } + + // Predicted NLQ pivots are delta-coded like the reshaping pivots. + var nlqPivots: [UInt32] = [0, params.blMax] + if mapping.pointee.nlq_pred_pivot_value.len >= 2, + let pivotData = mapping.pointee.nlq_pred_pivot_value.data { + let p0 = UInt32(pivotData[0]) + let p1 = p0 + UInt32(pivotData[1]) + nlqPivots = [p0, p1] + } + + let curves = [mapping.pointee.curves.0, mapping.pointee.curves.1, mapping.pointee.curves.2] + let labels = ["Y", "Cb", "Cr"] + + for c in 0..<3 { + var poly = DoviPolynomialParams() + var mmr = DoviMmrParams() + if !parseReshapingCurve( + curves[c], + coefLog2Denom: coefLog2Denom, + blMax: params.blMax, + polyOut: &poly, + mmrOut: &mmr + ) { + // Identity for this component: the base layer passes through + // unreshaped, which is wrong but recognisable, where a partially + // filled curve would put a flat value across the plane. + logOnce("[DoviFelComposer] \(labels[c]) reshaping curve rejected; falling back to identity", category: .swPlayback) + poly = .identity + mmr = DoviMmrParams() + } + + let nlq = parseNlqParams( + nlq: mapping.pointee.nlq, + component: c, + label: labels[c], + method: nlqMethod, + coefLog2Denom: coefLog2Denom, + nlqShift: nlqShift, + pivots: nlqPivots + ) + + switch c { + case 0: + params.polyY = poly + params.mmrY = mmr + params.nlqY = nlq + case 1: + params.polyCb = poly + params.mmrCb = mmr + params.nlqCb = nlq + default: + params.polyCr = poly + params.mmrCr = mmr + params.nlqCr = nlq + } + } + + if params.polyY.usesMmr { + // Tells the luma kernel to fetch the base-layer chroma planes it + // otherwise skips. No observed profile 7 stream does this (libdovi would + // report mapping_idc = 1 for component 0), so it is worth a log. + logOnce("[DoviFelComposer] luma curve uses MMR; enabling cross-component luma path", category: .swPlayback) + params.flags.insert(.lumaMmr) + } + + return params + } + + /// Combines an RPU coefficient's signed integer part and unsigned fraction + /// into one Q(denom) value, the way FFmpeg's `get_se_coef` does. + /// + /// Returns nil rather than saturating: a value that does not fit is a misparse + /// or an unsupported stream, and clamping it would quietly install a curve + /// that is wrong in a way nothing downstream can detect. + private static func fixedPoint(int intPart: Int64, frac: UInt64, denom: Int) -> Int32? { + guard let wide = fixedPointWide(int: intPart, frac: frac, denom: denom) else { return nil } + guard wide >= Int64(Int32.min), wide <= Int64(Int32.max) else { return nil } + return Int32(wide) + } + + /// As ``fixedPoint(int:frac:denom:)`` but keeps the full 64-bit result, for + /// values that are shifted down again before they have to fit in 32 bits. + private static func fixedPointWide(int intPart: Int64, frac: UInt64, denom: Int) -> Int64? { + guard denom >= 0, denom <= maxCoefLog2Denom else { return nil } + let scale = Int64(1) << Int64(denom) + guard frac < UInt64(scale) else { return nil } + let (shifted, shiftOverflow) = intPart.multipliedReportingOverflow(by: scale) + guard !shiftOverflow else { return nil } + let (sum, addOverflow) = shifted.addingReportingOverflow(Int64(frac)) + guard !addOverflow else { return nil } + return sum + } + + /// Fills `polyOut` (and `mmrOut` for MMR pieces) from one component's + /// reshaping curve. Returns false if anything about the curve is inconsistent, + /// leaving the outputs partially written for the caller to discard. + private static func parseReshapingCurve( + _ curve: DoviReshapingCurve, + coefLog2Denom: Int, + blMax: UInt32, + polyOut: inout DoviPolynomialParams, + mmrOut: inout DoviMmrParams + ) -> Bool { + let numPivots = Int(curve.num_pivots_minus2) + 2 + guard numPivots >= 2, numPivots <= DoviPolynomialParams.maxPivots else { return false } + guard let pivotsData = curve.pivots.data, curve.pivots.len >= numPivots else { return false } + + // Pivots are delta-coded; prefix-sum them into absolute BL code values. + var running: UInt64 = 0 + for i in 0.. polyOut.pivots[0] else { return false } + polyOut.numPivots = UInt32(numPivots) + + let pieces = numPivots - 1 + guard pieces >= 1, pieces <= DoviPolynomialParams.maxPieces else { return false } + + switch curve.mapping_idc { + case 0: + guard let poly = curve.polynomial else { return false } + guard let orderData = poly.pointee.poly_order_minus1.data, + poly.pointee.poly_order_minus1.len >= pieces else { return false } + guard let intPieces = poly.pointee.poly_coef_int.list, + poly.pointee.poly_coef_int.len >= pieces, + let fracPieces = poly.pointee.poly_coef.list, + poly.pointee.poly_coef.len >= pieces else { return false } + + for p in 0..= 1, order <= 2 else { return false } + polyOut.method[p] = DoviMapping.polynomial.rawValue + polyOut.order[p] = UInt32(order) + polyOut.coefShift[p] = UInt32(coefLog2Denom) + + guard let intData = intPieces[p]?.pointee, let intValues = intData.data, + let fracData = fracPieces[p]?.pointee, let fracValues = fracData.data else { return false } + let count = order + 1 + guard intData.len >= count, fracData.len >= count else { return false } + for j in 0..= pieces else { return false } + guard let constInt = mmr.pointee.mmr_constant_int.data, + mmr.pointee.mmr_constant_int.len >= pieces, + let constFrac = mmr.pointee.mmr_constant.data, + mmr.pointee.mmr_constant.len >= pieces else { return false } + guard let intPieces = mmr.pointee.mmr_coef_int.list, + mmr.pointee.mmr_coef_int.len >= pieces, + let fracPieces = mmr.pointee.mmr_coef.list, + mmr.pointee.mmr_coef.len >= pieces else { return false } + + for p in 0..= 1, order <= 3 else { return false } + polyOut.method[p] = DoviMapping.mmr.rawValue + polyOut.order[p] = UInt32(order) + + guard let constant = fixedPoint(int: constInt[p], frac: constFrac[p], denom: coefLog2Denom) else { + return false + } + mmrOut.constant[p] = constant + + guard let intOrders = intPieces[p]?.pointee, let intOrderList = intOrders.list, + let fracOrders = fracPieces[p]?.pointee, let fracOrderList = fracOrders.list, + intOrders.len >= order, fracOrders.len >= order else { return false } + + for o in 0..= 7, fracTerms.len >= 7 else { return false } + for k in 0..<7 { + guard let coef = fixedPoint(int: intValues[k], frac: fracValues[k], denom: coefLog2Denom) else { + return false + } + mmrOut.coef[p][o * 7 + k] = coef + } + } + } + mmrOut.coefShift = UInt32(coefLog2Denom) + return true + + default: + return false + } + } + + /// Reads one component's inverse-NLQ parameters. Any value that does not fit + /// the fixed-point ABI disables the residual for that component instead of + /// being clamped, while keeping the decoded shift and pivots so a probe can + /// still report what the RPU said. + private static func parseNlqParams( + nlq: UnsafePointer?, + component c: Int, + label: String, + method: DoviNlqMethod, + coefLog2Denom: Int, + nlqShift: UInt32, + pivots: [UInt32] + ) -> DoviNlqParams { + var disabled = DoviNlqParams.disabled + disabled.nlqPivots = pivots + disabled.nlqShift = nlqShift + + guard method == .linear, let nlq = nlq else { return disabled } + + let offset: UInt16 + let slopeInt: UInt64 + let slopeFrac: UInt64 + let thrInt: UInt64 + let thrFrac: UInt64 + let maxInt: UInt64 + let maxFrac: UInt64 + + switch c { + case 0: + offset = nlq.pointee.nlq_offset.0 + slopeInt = nlq.pointee.linear_deadzone_slope_int.0 + slopeFrac = nlq.pointee.linear_deadzone_slope.0 + thrInt = nlq.pointee.linear_deadzone_threshold_int.0 + thrFrac = nlq.pointee.linear_deadzone_threshold.0 + maxInt = nlq.pointee.vdr_in_max_int.0 + maxFrac = nlq.pointee.vdr_in_max.0 + case 1: + offset = nlq.pointee.nlq_offset.1 + slopeInt = nlq.pointee.linear_deadzone_slope_int.1 + slopeFrac = nlq.pointee.linear_deadzone_slope.1 + thrInt = nlq.pointee.linear_deadzone_threshold_int.1 + thrFrac = nlq.pointee.linear_deadzone_threshold.1 + maxInt = nlq.pointee.vdr_in_max_int.1 + maxFrac = nlq.pointee.vdr_in_max.1 + default: + offset = nlq.pointee.nlq_offset.2 + slopeInt = nlq.pointee.linear_deadzone_slope_int.2 + slopeFrac = nlq.pointee.linear_deadzone_slope.2 + thrInt = nlq.pointee.linear_deadzone_threshold_int.2 + thrFrac = nlq.pointee.linear_deadzone_threshold.2 + maxInt = nlq.pointee.vdr_in_max_int.2 + maxFrac = nlq.pointee.vdr_in_max.2 + } + + guard let slopeIntSigned = Int64(exactly: slopeInt), + let thrIntSigned = Int64(exactly: thrInt), + let maxIntSigned = Int64(exactly: maxInt), + let scale = fixedPoint(int: slopeIntSigned, frac: slopeFrac, denom: coefLog2Denom), + let threshold = fixedPoint(int: thrIntSigned, frac: thrFrac, denom: coefLog2Denom), + let maxFp = fixedPointWide(int: maxIntSigned, frac: maxFrac, denom: coefLog2Denom) else { + logOnce("[DoviFelComposer] \(label) NLQ parameters do not fit the fixed-point ABI; dropping residual", category: .swPlayback) + return disabled + } + + // vdr_in_max is a normalised ceiling in Q(coefficient_log2_denom); the + // same shift that converts the residual converts the ceiling. + let scaledMax = maxFp >> Int64(nlqShift) + guard scaledMax >= 0, scaledMax <= Int64(Int32.max) else { + logOnce("[DoviFelComposer] \(label) vdr_in_max \(scaledMax) out of range; dropping residual", category: .swPlayback) + return disabled + } + + if threshold != 0 { + deadzoneLogLock.lock() + let first = !loggedNonZeroDeadzone + loggedNonZeroDeadzone = true + deadzoneLogLock.unlock() + if first { + EngineLog.emit("[DoviFelComposer] \(label) linear_deadzone_threshold = \(threshold) (non-zero); " + + "the inverse NLQ adds it to the scaled magnitude, as the spec's pseudo-code " + + "does, but no surveyed stream carries a non-zero threshold. If highlights look " + + "crushed or lifted, this is the first thing to question.", category: .swPlayback) + } + } + + var params = DoviNlqParams() + params.nlqMethod = .linear + params.nlqPivots = pivots + params.nlqShift = nlqShift + params.nlqOffset = Int32(offset) + params.nlqScale = scale + params.nlqThreshold = threshold + params.vdrInMax = Int32(scaledMax) + return params + } +} diff --git a/Sources/AetherEngine/Video/Metal/DoviMetalParams.swift b/Sources/AetherEngine/Video/Metal/DoviMetalParams.swift new file mode 100644 index 000000000..04287391f --- /dev/null +++ b/Sources/AetherEngine/Video/Metal/DoviMetalParams.swift @@ -0,0 +1,315 @@ +import Foundation + +/// How one reshaping piece is evaluated (`mapping_idc`). +enum DoviMapping: UInt32, Sendable { + case polynomial = 0 + case mmr = 1 +} + +/// `nlq_method_idc`. Anything the composer cannot invert maps to ``none``, +/// which yields no residual rather than a wrong one. +enum DoviNlqMethod: UInt32, Sendable { + case linear = 0 + case none = 0xFFFF_FFFF +} + +/// Output quantisation mode. +enum DoviDither: UInt32, Sendable { + case none = 0 + case bayer8x8 = 1 + /// Test only: planes hold the un-quantised 12-bit VDR code shifted left 4, not P010. + case debugRawVDR = 2 + /// Undithered, but the VDR bits below the 10-bit code are kept in P010's unused low 6 bits, + /// so a consumer that reads the full 16-bit word sees 12-bit (or finer) precision while one + /// that reads the top 10 bits sees the truncated code. Not validated on a display path yet. + case fullPrecision = 3 +} + +/// `DoviMetalParams.flags` bits. +struct DoviComposerFlags: OptionSet, Sendable { + let rawValue: UInt32 + init(rawValue: UInt32) { self.rawValue = rawValue } + + /// `disable_residual_flag`, or the RPU declares `el_type == MEL`. + static let disableResidual = DoviComposerFlags(rawValue: 0x1) + /// The luma curve carries at least one MMR piece, so the luma kernel has to + /// read the base-layer chroma planes as well. + static let lumaMmr = DoviComposerFlags(rawValue: 0x2) +} + +/// Piecewise reshaping curve for one component; mirrors +/// `struct DoviPolynomialParams` in `DoviShaderSource`. +/// +/// `numPivots < 2` means identity passthrough, which is the fail-safe the parser +/// installs whenever a curve cannot be trusted. +struct DoviPolynomialParams: Sendable { + static let maxPivots = 9 + static let maxPieces = 8 + /// Serialized size in bytes, asserted against the C struct by `_Static_assert`. + /// num_pivots + pivots + method + order + coef + coef_shift. + static let stride = 4 * (1 + maxPivots + maxPieces + maxPieces + maxPieces * 3 + maxPieces) + + var numPivots: UInt32 = 0 + var pivots: [UInt32] = Array(repeating: 0, count: DoviPolynomialParams.maxPivots) + var method: [UInt32] = Array(repeating: 0, count: DoviPolynomialParams.maxPieces) + var order: [UInt32] = Array(repeating: 0, count: DoviPolynomialParams.maxPieces) + var coef: [[Int32]] = Array(repeating: [0, 0, 0], count: DoviPolynomialParams.maxPieces) + var coefShift: [UInt32] = Array(repeating: 0, count: DoviPolynomialParams.maxPieces) + + init() {} + + /// Identity passthrough: the base layer reaches the output unreshaped. + static var identity: DoviPolynomialParams { DoviPolynomialParams() } + + var isIdentity: Bool { numPivots < 2 } + + /// A pivot count the fixed-size arrays and the kernels cannot hold is a corrupt curve; it takes the + /// same identity fail-safe as a curve the parser could not trust, instead of indexing past `pivots`. + mutating func clampToCapacity() { + if numPivots > UInt32(Self.maxPivots) { numPivots = 0 } + for i in coefShift.indices where coefShift[i] > 31 { coefShift[i] = 31 } + if pivots.count < Self.maxPivots { pivots += Array(repeating: 0, count: Self.maxPivots - pivots.count) } + } + + /// True when any live piece is evaluated as MMR, which is what decides + /// whether the kernel needs the cross-component inputs. + var usesMmr: Bool { + guard numPivots >= 2 else { return false } + let pieces = min(Int(numPivots) - 1, Self.maxPieces) + for p in 0.. Bool { + guard bl >= 8, bl <= 16, vdr >= bl, vdr <= 16 else { return false } + blMax = UInt32((1 << bl) - 1) + vdrMax = UInt32((1 << vdr) - 1) + vdrScale = UInt32(1 << (vdr - bl)) + return true + } + + /// Little-endian sequential writer to a raw buffer pointer. Appending in declaration order and + /// checking each sub-block's length is what keeps the layout honest; writes directly into + /// preallocated Metal buffers without intermediate Data or array allocations. + private struct BufferWriter { + let buffer: UnsafeMutableRawBufferPointer + var offset: Int = 0 + + init(buffer: UnsafeMutableRawBufferPointer) { + precondition(buffer.count >= DoviMetalParams.serializedSize, + "Buffer too small for serialized DoviMetalParams") + self.buffer = buffer + } + + mutating func u32(_ value: UInt32) { + let v = value.littleEndian + buffer.storeBytes(of: v, toByteOffset: offset, as: UInt32.self) + offset += 4 + } + + mutating func i32(_ value: Int32) { + let v = value.littleEndian + buffer.storeBytes(of: v, toByteOffset: offset, as: Int32.self) + offset += 4 + } + + mutating func u32s(_ values: [UInt32], count: Int) { + for i in 0.. 0 ? nlq.nlqPivots[0] : 0) + u32(nlq.nlqPivots.count > 1 ? nlq.nlqPivots[1] : 0) + precondition(offset - start == DoviNlqParams.stride, + "DoviNlqParams serialization is not stride bytes") + } + } + + /// Serializes directly into a preallocated raw buffer pointer without allocating Data (dolby-vision-fel.md §5.6). + func serialize(into buffer: UnsafeMutableRawBufferPointer) { + var w = BufferWriter(buffer: buffer) + + w.u32(blWidth) + w.u32(blHeight) + w.u32(elWidth) + w.u32(elHeight) + w.u32(targetMode) + w.u32(ditherMode.rawValue) + w.u32(frameIndex) + w.u32(flags.rawValue) + + w.append(polyY) + w.append(mmrY) + w.append(nlqY) + + w.append(polyCb) + w.append(mmrCb) + w.append(nlqCb) + + w.append(polyCr) + w.append(mmrCr) + w.append(nlqCr) + + w.u32(minPq) + w.u32(maxPq) + w.u32(avgPq) + w.u32(blMax) + w.u32(vdrMax) + w.u32(vdrScale) + + precondition(w.offset == Self.serializedSize, + "DoviMetalParams serialization is not serializedSize bytes") + } + + /// Serializes to the exact C/Metal memory layout. + func serialize() -> Data { + var data = Data(count: Self.serializedSize) + data.withUnsafeMutableBytes { raw in + serialize(into: raw) + } + return data + } +} diff --git a/Sources/AetherEngine/Video/Metal/DoviShaderSource.swift b/Sources/AetherEngine/Video/Metal/DoviShaderSource.swift new file mode 100644 index 000000000..ededc06ef --- /dev/null +++ b/Sources/AetherEngine/Video/Metal/DoviShaderSource.swift @@ -0,0 +1,722 @@ +import Foundation + +/// Canonical Metal source for the Dolby Vision FEL composer. +/// +/// `DoviFelComposer` compiles ``source`` at runtime with +/// `MTLDevice.makeLibrary(source:)` on demand. +enum DoviShaderSource { + + /// Metal-side spelling of the fixed-width typedefs the shared block uses. + static let preamble = """ +#include +using namespace metal; + +typedef uint dovi_u32; +typedef int dovi_i32; +""" + + /// Definitions of the parameters and structs consumed by the Metal kernels. + static let sharedStructs = """ +/* DOVI_SHARED_STRUCTS_BEGIN */ +/* ---- Target output modes (informational; the composer always emits + video-range 10-bit P010, so this does not currently switch clamping) ---- */ +#define DOVI_TARGET_MODE_PROFILE_8_1 0 /* Rec.2020 YCbCr PQ 10-bit limited range */ +#define DOVI_TARGET_MODE_PROFILE_5 1 /* Dolby IPTPQc2 10-bit full range */ + +/* ---- Dither modes ---- */ +#define DOVI_DITHER_NONE 0 /* round to nearest */ +#define DOVI_DITHER_BAYER_8X8 1 /* ordered dither, inverted on odd frames */ +#define DOVI_DITHER_FULL_PRECISION 3 /* keep the VDR bits below the 10-bit code in P010's low 6 bits */ +#define DOVI_DEBUG_RAW_VDR 2 /* test only: write the un-quantised VDR code << 4 (12-bit in 16) */ + +/* ---- nlq_method_idc (ETSI GS CCM 001). Anything other than LINEAR yields no + residual rather than a wrong inverse, degrading to base-layer output. ---- */ +#define DOVI_NLQ_METHOD_LINEAR 0 +#define DOVI_NLQ_METHOD_NONE 0xFFFFFFFFu + +/* ---- mapping_idc, per reshaping piece ---- */ +#define DOVI_MAPPING_POLYNOMIAL 0 +#define DOVI_MAPPING_MMR 1 + +/* ---- flags ---- */ +#define DOVI_FLAG_DISABLE_RESIDUAL 0x1u /* disable_residual_flag, or el_type == MEL */ +#define DOVI_FLAG_LUMA_MMR 0x2u /* luma curve has at least one MMR piece */ + +/* Piecewise reshaping curve for one component. + * + * num_pivots < 2 means "identity passthrough"; the parser installs that as the + * fail-safe whenever a curve cannot be trusted, so a malformed RPU degrades to + * the base layer instead of to black. + * + * pivots are absolute (prefix-summed) BL code values, ascending, and pivots[0] + * / pivots[num_pivots - 1] bound the evaluation domain. method[i] selects how + * piece i is evaluated. The syntax lets a component mix polynomial and MMR + * pieces, but libdovi keeps one mapping_idc per component (the last piece's) and + * one coefficient list per method, so a mixed curve cannot be read back + * through its C API, and DoviRpuGuard refuses it before it gets here: in + * practice every piece of a component has the same method. order[i] is + * the polynomial order for a polynomial piece and the MMR order for an MMR + * piece. coef[i][j] is Q(coef_shift[i]) and is evaluated against inputs + * normalised by dividing the BL code value by 2^bl_bit_depth (bl_max + 1), the + * fixed-point convention of the spec's pseudo-code. */ +struct DoviPolynomialParams { + dovi_u32 num_pivots; + dovi_u32 pivots[9]; + dovi_u32 method[8]; + dovi_u32 order[8]; + dovi_i32 coef[8][3]; + dovi_u32 coef_shift[8]; +}; + +/* Multiple-model regression coefficients, one set per reshaping piece. + * + * The piece count, the per-piece MMR order and the piece selection all live in + * the matching DoviPolynomialParams; this block only carries the coefficients, + * laid out as coef[piece][order * 7 + term] over the seven cross terms + * [y, u, v, yu, yv, uv, yuv]. Order o contributes those terms raised to the + * power o + 1. Every coefficient is Q(coef_shift) against normalised inputs. */ +struct DoviMmrParams { + dovi_i32 mmr_constant[8]; + dovi_i32 coef[8][21]; + dovi_u32 coef_shift; +}; + +/* Inverse non-linear quantisation for one component. + * + * nlq_scale and nlq_threshold are Q(coefficient_log2_denom) in the normalised + * residual domain; nlq_shift = coefficient_log2_denom - vdr_bit_depth converts + * that product into VDR code units. vdr_in_max is the ceiling already reduced + * to VDR code units by the parser. */ +struct DoviNlqParams { + dovi_u32 nlq_method; + dovi_i32 nlq_offset; + dovi_i32 nlq_scale; + dovi_i32 nlq_threshold; + dovi_i32 vdr_in_max; + dovi_u32 nlq_shift; + dovi_u32 nlq_pivots[2]; +}; + +struct DoviMetalParams { + dovi_u32 bl_width; + dovi_u32 bl_height; + dovi_u32 el_width; + dovi_u32 el_height; + + dovi_u32 target_mode; + dovi_u32 dither_mode; + dovi_u32 frame_index; + dovi_u32 flags; + + struct DoviPolynomialParams poly_y; + struct DoviMmrParams mmr_y; + struct DoviNlqParams nlq_y; + + struct DoviPolynomialParams poly_cb; + struct DoviMmrParams mmr_cb; + struct DoviNlqParams nlq_cb; + + struct DoviPolynomialParams poly_cr; + struct DoviMmrParams mmr_cr; + struct DoviNlqParams nlq_cr; + + dovi_u32 min_pq; + dovi_u32 max_pq; + dovi_u32 avg_pq; + + /* Bit-depth derived constants, so the kernels carry no literal 1023/4095. + bl_max = (1 << bl_bit_depth) - 1 + vdr_max = (1 << vdr_bit_depth) - 1 + vdr_scale = 1 << (vdr_bit_depth - bl_bit_depth) + Curve inputs are normalised by 2^bl_bit_depth (bl_max + 1) and a normalised + prediction is denormalised by (bl_max + 1) * vdr_scale = 2^vdr_bit_depth, the + same scale both ways, so an identity curve is bit-exact and a neutral chroma + (code 512) maps to exactly 2048 (DoViBaker agrees; scaling by bl_max instead + puts it at 2049, and up to 2041 through an MMR constant). */ + dovi_u32 bl_max; + dovi_u32 vdr_max; + dovi_u32 vdr_scale; +}; +/* DOVI_SHARED_STRUCTS_END */ +""" + + /// Kernel bodies and their helpers: the only place the composition math is + /// written down, shared by both consumers unchanged. + static let kernels = """ + +/* ============================================================================ + Ordered dither + ========================================================================= */ + +constant float dovi_bayer_8x8[64] = { + 0.0f/64.0f, 32.0f/64.0f, 8.0f/64.0f, 40.0f/64.0f, 2.0f/64.0f, 34.0f/64.0f, 10.0f/64.0f, 42.0f/64.0f, + 48.0f/64.0f, 16.0f/64.0f, 56.0f/64.0f, 24.0f/64.0f, 50.0f/64.0f, 18.0f/64.0f, 58.0f/64.0f, 26.0f/64.0f, + 12.0f/64.0f, 44.0f/64.0f, 4.0f/64.0f, 36.0f/64.0f, 14.0f/64.0f, 46.0f/64.0f, 6.0f/64.0f, 38.0f/64.0f, + 60.0f/64.0f, 28.0f/64.0f, 52.0f/64.0f, 20.0f/64.0f, 62.0f/64.0f, 30.0f/64.0f, 54.0f/64.0f, 22.0f/64.0f, + 3.0f/64.0f, 35.0f/64.0f, 11.0f/64.0f, 43.0f/64.0f, 1.0f/64.0f, 33.0f/64.0f, 9.0f/64.0f, 41.0f/64.0f, + 51.0f/64.0f, 19.0f/64.0f, 59.0f/64.0f, 27.0f/64.0f, 49.0f/64.0f, 17.0f/64.0f, 57.0f/64.0f, 25.0f/64.0f, + 15.0f/64.0f, 47.0f/64.0f, 7.0f/64.0f, 39.0f/64.0f, 13.0f/64.0f, 45.0f/64.0f, 5.0f/64.0f, 37.0f/64.0f, + 63.0f/64.0f, 31.0f/64.0f, 55.0f/64.0f, 23.0f/64.0f, 61.0f/64.0f, 29.0f/64.0f, 53.0f/64.0f, 21.0f/64.0f +}; + +// Ordered dither offset in [0.5/64, 63.5/64], whose mean over the cell is +// exactly 0.5 -- the same rounding bias the undithered branch applies. +// +// The raw Bayer table holds k/64 for k in 0..63 and averages 31.5/64 = 0.4921875, +// 1/128 of a code low. That bias is invisible at any bit depth a Dolby Vision +// stream can ask for: the value reaching the rounding step has a fraction of +// j/vdr_scale, and for vdr_scale <= 64 the raw and centred tables select exactly +// the same cells at every j, so 10-bit BL into 12-bit VDR (vdr_scale = 4) is +// bit-identical either way. It becomes visible only at the vdr_scale >= 128 that +// setBitDepths still accepts, where the raw table's worst per-level error is +// 0.0117 codes against the centred table's 0.0078. Centring costs one add, makes +// the mean exactly 0.5 as claimed, and removes the case where it would matter. +// +// phase translates the cell so that two components written from the same value do +// not receive the same pattern. Not every translation does that: this matrix +// satisfies M[(y + 4) % 8][(x + 4) % 8] == M[y][x] +/- 1, so the (4, 4) diagonal +// is its own near-symmetry (a Cr phase there leaves Cb and Cr disagreeing on +// an average of 1 pixel in 64 -- at the 50% level, where the selected set is a +// checkerboard in (x + y) parity and a (4, 4) shift preserves parity, on zero +// pixels). Translations of odd parity are the ones that decorrelate, averaging 32 +// pixels in 64. Parity is additive, so three mutually odd offsets do not exist; +// (0, 0) / (1, 0) / (1, 1) makes the two chroma components and the luma-Cb pair +// odd and leaves luma-Cr, which is also the pair that differs in plane and in +// resolution, at 16 in 64. +static inline float dovi_dither(uint2 gid, uint frame_index, uint2 phase) { + uint dx = (gid.x + phase.x) % 8u; + uint dy = (gid.y + phase.y) % 8u; + float d = dovi_bayer_8x8[dy * 8u + dx]; + if (frame_index & 1u) { + // Inversion is over the raw table, so the frame pair still covers every + // cell exactly once; the half-step is added to both frames afterwards. + d = (63.0f / 64.0f) - d; + } + return d + (0.5f / 64.0f); +} + +/* ============================================================================ + Reshaping curve evaluation + ========================================================================= */ + +// Index of the piece owning bl_val: the last pivot that bl_val reaches. +// Deliberately last-match-wins rather than first, because adjacent pivots can +// repeat (a zero-width piece) and the spec assigns the sample to the later one. +static inline int dovi_select_piece(constant struct DoviPolynomialParams &curve, int bl_val) { + int last_piece = int(curve.num_pivots) - 2; + int piece = 0; + for (int i = 0; i <= last_piece; ++i) { + if (bl_val >= int(curve.pivots[i])) { + piece = i; + } + } + return clamp(piece, 0, max(last_piece, 0)); +} + +// Polynomial piece, evaluated on the input normalised by `bl_max`. Callers pass +// bl_max + 1 (= 2^bl_bit_depth), the same normaliser as the y/u/v inputs. +// Returns the prediction in normalised units; dovi_denormalize floors it at 0 before the residual is +// added (DoViBaker clamps `vv`/`rr` at 0 there), and the sum is clamped again afterwards. +static inline float dovi_eval_poly( + constant struct DoviPolynomialParams &curve, + int piece, + int bl_val, + uint bl_max) +{ + int lo = int(curve.pivots[0]); + int hi = int(curve.pivots[curve.num_pivots - 1]); + float x = float(clamp(bl_val, lo, hi)) / float(bl_max); + + float inv = ldexp(1.0f, -int(curve.coef_shift[piece])); + float pred = float(curve.coef[piece][0]) * inv; + uint order = curve.order[piece]; + if (order >= 1u) { + pred = fma(float(curve.coef[piece][1]) * inv, x, pred); + } + if (order >= 2u) { + pred = fma(float(curve.coef[piece][2]) * inv, x * x, pred); + } + return pred; +} + +// An MMR input clamped to the pivot range of its own component (luma to poly_y's, Cb to poly_cb's, +// Cr to poly_cr's), as DoViBaker's `mmrMapping` does before it forms the cross terms. A component with +// no usable curve (num_pivots < 2) has no range and is left as is. +static inline float dovi_clamp_mmr_input(constant struct DoviPolynomialParams &curve, float norm, float scale) { + if (curve.num_pivots < 2u) { + return norm; + } + float lo = float(curve.pivots[0]) / scale; + float hi = float(curve.pivots[curve.num_pivots - 1u]) / scale; + return clamp(norm, lo, hi); +} + +// MMR piece: a regression over the seven cross terms of all three normalised +// base-layer components, with order o contributing those terms raised to o + 1. +static inline float dovi_eval_mmr( + constant struct DoviPolynomialParams &curve, + constant struct DoviMmrParams &mmr, + int piece, + float y_norm, float u_norm, float v_norm) +{ + // y_norm, u_norm and v_norm are already clamped to their components' pivot ranges. + float inv = ldexp(1.0f, -int(mmr.coef_shift)); + float s = float(mmr.mmr_constant[piece]) * inv; + + float base[7] = { + y_norm, + u_norm, + v_norm, + y_norm * u_norm, + y_norm * v_norm, + u_norm * v_norm, + y_norm * u_norm * v_norm + }; + float t[7]; + for (int k = 0; k < 7; ++k) { + t[k] = base[k]; + } + + uint order = min(curve.order[piece], 3u); + for (uint o = 0u; o < order; ++o) { + for (int k = 0; k < 7; ++k) { + s = fma(float(mmr.coef[piece][o * 7u + uint(k)]) * inv, t[k], s); + } + if (o + 1u < order) { + for (int k = 0; k < 7; ++k) { + t[k] *= base[k]; + } + } + } + return s; +} + +// One component's reshaping prediction, in normalised [0, 1] units. +// num_pivots < 2 is the parser's fail-safe for a curve it could not trust, and +// means identity: the base layer passes through untouched. +static inline float dovi_eval_curve( + constant struct DoviPolynomialParams &curve, + constant struct DoviMmrParams &mmr, + constant struct DoviPolynomialParams &poly_y, + constant struct DoviPolynomialParams &poly_cb, + constant struct DoviPolynomialParams &poly_cr, + int bl_val, + float y_norm, float u_norm, float v_norm, + uint bl_max) +{ + if (curve.num_pivots < 2u) { + return float(bl_val) / float(bl_max); + } + int piece = dovi_select_piece(curve, bl_val); + if (curve.method[piece] == DOVI_MAPPING_MMR) { + float scale = float(bl_max); + return dovi_eval_mmr(curve, mmr, piece, + dovi_clamp_mmr_input(poly_y, y_norm, scale), + dovi_clamp_mmr_input(poly_cb, u_norm, scale), + dovi_clamp_mmr_input(poly_cr, v_norm, scale)); + } + return dovi_eval_poly(curve, piece, bl_val, bl_max); +} + +// Normalised prediction back into VDR code units. Callers pass bl_max + 1, so the +// scale is (bl_max + 1) * vdr_scale = 2^vdr_bit_depth rather than vdr_max, which is +// what makes an identity curve bit-exact end to end. +static inline int dovi_denormalize(float pred_norm, uint bl_max, uint vdr_scale) { + // The prediction is floored at 0 before the residual is added, as DoViBaker does for both the + // polynomial and the MMR result, so a negative prediction cannot be pulled back up by the residual. + return max(int(rint(pred_norm * float(bl_max) * float(vdr_scale))), 0); +} + +/* ============================================================================ + Inverse NLQ + ========================================================================= */ + +// Inverse linear-deadzone quantiser, in VDR code units. +// +// mag_fp is Q(coefficient_log2_denom) of the normalised residual, so the shift +// by nlq_shift = coefficient_log2_denom - vdr_bit_depth lands it in VDR code +// units. That reading is corroborated by the RPUs themselves: both test streams +// carry linear_deadzone_slope = 2048 and vdr_in_max = 0.125 at denom 23, and +// 512 * 2048 >> 11 is exactly 512, i.e. the slope is tuned so the enhancement +// layer's full +/-512 swing lands precisely on vdr_in_max. Shifting by denom +// instead (the convention the vc4 composer uses) collapses every residual to +// zero, and no other shift makes the two fields agree. +// +// The threshold is ADDED to the magnitude, in the same Q(coefficient_log2_denom) domain as +// (|r| - 1/2) * slope, and there is no early zero: that is the spec's pseudo-code, and DoViBaker +// implements it (`dq += T * sign`). Every real stream carries linear_deadzone_threshold == 0 (RPU +// survey, 9,816 RPUs), where all readings agree. Subtracting it, or zeroing anything inside the +// deadzone, would be a different reading and does not match a second implementation. +static inline int dovi_eval_inv_nlq(constant struct DoviNlqParams &nlq, int el_val) { + if (nlq.nlq_method != DOVI_NLQ_METHOD_LINEAR || nlq.nlq_scale == 0) { + return 0; + } + int r = el_val - nlq.nlq_offset; + if (r == 0) { + return 0; + } + // Reconstruction at the middle of the quantisation step, (|r| - 1/2) * scale, in doubled fixed + // point so the half step stays exact. + long mag_fp = (2L * long(abs(r)) - 1L) * long(nlq.nlq_scale); + long threshold2 = 2L * long(nlq.nlq_threshold); + long mag = (mag_fp + threshold2) >> (nlq.nlq_shift + 1u); + if (nlq.vdr_in_max > 0 && mag > long(nlq.vdr_in_max)) { + mag = long(nlq.vdr_in_max); + } + return (r < 0) ? -int(mag) : int(mag); +} + +/* ============================================================================ + Output quantisation + ========================================================================= */ + +// Full-precision sample: v/vdr_scale in 10-bit codes with 6 fractional bits, clamped to the +// video-range limits. The top 10 bits are the truncated code; the rest is what P010 leaves unused. +static inline int dovi_full_precision(int vdr_val, uint vdr_scale, int lo, int hi) { + int v = (vdr_val * 64 + int(vdr_scale >> 1)) / int(vdr_scale); + return clamp(v, lo << 6, hi << 6); +} + +// VDR code value down to the 10-bit output plane, dithered or rounded. +static inline int dovi_quantize( + int vdr_val, + uint vdr_scale, + uint dither_mode, + float dither, + int lo, + int hi) +{ + float v = float(vdr_val) / float(vdr_scale); + // Both branches have mean 0.5, so enabling dither redistributes the rounding + // error without moving the average level. + float bias = (dither_mode == DOVI_DITHER_NONE) ? 0.5f : dither; + return clamp(int(floor(v + bias)), lo, hi); +} + +/* ============================================================================ + Enhancement layer upsampling + ========================================================================= */ + +// EL upsampling: one hardware bilinear fetch. +// +// el_spatial_resampling_filter_flag advertises that the encoder used its own +// separable filter, whose taps the RPU does not carry, so no bit-exact inverse +// is available from the stream. Against the DoViBaker oracle on a fine-detail set (gratings, zone plate, +// band-limited noise) a 4-fetch Catmull-Rom bought about 0.7% of per-pixel error sd over bilinear, with chroma +// unchanged, for four times the EL reads. `tex_size` is unused by bilinear; it is +// kept so a wider filter can replace this one without touching the callers. +static inline float dovi_sample_el_1d( + texture2d tex, + sampler s, + float2 uv, + float2 tex_size) +{ + return tex.sample(s, uv).r; +} + +static inline float2 dovi_sample_el_2d( + texture2d tex, + sampler s, + float2 uv, + float2 tex_size) +{ + return tex.sample(s, uv).rg; +} + +// UNORM [0, 1] from a 10-in-16 MSB-aligned P010 plane back to a code value. +// 65535 / 64 is exact: the hardware divides the stored ushort by 65535, and the +// code value is that ushort >> 6. +constant float dovi_unorm_to_code = 65535.0f / 64.0f; + +/* ============================================================================ + Composition kernels + ========================================================================= */ + +// Luma: BL luma + reshaping curve + inverse-NLQ residual -> 10-bit P010 plane. +// +// bl_uv_tex is bound only because a luma curve may carry MMR pieces, which need +// all three components. No observed profile 7 stream has an MMR luma curve (libdovi +// would report it as mapping_idc = 1 for component 0), so the DOVI_FLAG_LUMA_MMR gate +// normally skips the fetch entirely. When it does fire, the chroma pair is taken +// nearest-neighbour from the 4:2:0 plane: the reshaping model is defined on 4:4:4 +// input, and reconstructing chroma to full resolution here would cost a separate +// pass for a case that does not occur in practice. +kernel void dovi_compose_luma( + texture2d bl_y_tex [[texture(0)]], + texture2d bl_uv_tex [[texture(1)]], + texture2d el_y_tex [[texture(2)]], + texture2d out_y_tex [[texture(3)]], + constant struct DoviMetalParams& params [[buffer(0)]], + constant float* luma_lut [[buffer(1)]], + uint2 gid [[thread_position_in_grid]]) +{ + if (gid.x >= params.bl_width || gid.y >= params.bl_height) { + return; + } + + int bl_y = int(bl_y_tex.read(gid).r >> 6); + + float pred; + if ((params.flags & DOVI_FLAG_LUMA_MMR) == 0u) { + pred = luma_lut[bl_y]; + } else { + float y_norm = float(bl_y) / float(params.bl_max + 1u); + ushort4 bl_uv = bl_uv_tex.read(uint2(gid.x >> 1, gid.y >> 1)); + float u_norm = float(int(bl_uv.r >> 6)) / float(params.bl_max + 1u); + float v_norm = float(int(bl_uv.g >> 6)) / float(params.bl_max + 1u); + pred = dovi_eval_curve(params.poly_y, params.mmr_y, + params.poly_y, params.poly_cb, params.poly_cr, bl_y, + y_norm, u_norm, v_norm, params.bl_max + 1u); + } + int vdr_y = dovi_denormalize(pred, params.bl_max + 1u, params.vdr_scale); + + if ((params.flags & DOVI_FLAG_DISABLE_RESIDUAL) == 0u && + params.nlq_y.nlq_method == DOVI_NLQ_METHOD_LINEAR && + params.nlq_y.nlq_scale != 0) { + constexpr sampler s(coord::normalized, filter::linear, address::clamp_to_edge); + float2 uv = (float2(gid) + 0.5f) / float2(params.bl_width, params.bl_height); + float el = dovi_sample_el_1d(el_y_tex, s, uv, + float2(params.el_width, params.el_height)) * dovi_unorm_to_code; + vdr_y += dovi_eval_inv_nlq(params.nlq_y, int(rint(el))); + } + + vdr_y = clamp(vdr_y, 0, int(params.vdr_max)); + + int out_y = dovi_quantize(vdr_y, params.vdr_scale, params.dither_mode, + dovi_dither(gid, params.frame_index, uint2(0u, 0u)), 64, 940); + if (params.dither_mode == DOVI_DEBUG_RAW_VDR) { + out_y_tex.write(ushort4(ushort(vdr_y << 4), 0, 0, 0), gid); + return; + } + if (params.dither_mode == DOVI_DITHER_FULL_PRECISION) { + out_y_tex.write(ushort4(ushort(dovi_full_precision(vdr_y, params.vdr_scale, 64, 940)), 0, 0, 0), gid); + return; + } + out_y_tex.write(ushort4(ushort(out_y << 6), 0, 0, 0), gid); +} + +// Chroma: both chroma planes in one pass, since an MMR curve for Cb and for Cr +// reads the same three base-layer components. +kernel void dovi_compose_chroma( + texture2d bl_y_tex [[texture(0)]], + texture2d bl_uv_tex [[texture(1)]], + texture2d el_uv_tex [[texture(2)]], + texture2d out_uv_tex [[texture(3)]], + constant struct DoviMetalParams& params [[buffer(0)]], + uint2 gid [[thread_position_in_grid]]) +{ + uint uv_width = params.bl_width / 2u; + uint uv_height = params.bl_height / 2u; + + if (gid.x >= uv_width || gid.y >= uv_height) { + return; + } + + // Base-layer luma at the chroma site, for the cross-component terms: [1 2 1] + // around the even column in each of the cell's two rows, then the rows + // averaged, in integers, edges clamped. This is DoViBaker's reading, and + // Dolby-encoded streams agree with it: on one with order-3 MMR chroma the + // coded residual matches it at about half the error of the plain vertical + // 2-tap this kernel used before. + uint lx = min(gid.x * 2u, params.bl_width - 1u); + uint lxl = lx > 0u ? lx - 1u : 0u; + uint lxr = min(lx + 1u, params.bl_width - 1u); + int bl_y = 0; + for (uint r = 0u; r < 2u; ++r) { + uint ly = min(gid.y * 2u + r, params.bl_height - 1u); + int row = int(bl_y_tex.read(uint2(lxl, ly)).r >> 6) + + 2 * int(bl_y_tex.read(uint2(lx, ly)).r >> 6) + + int(bl_y_tex.read(uint2(lxr, ly)).r >> 6); + bl_y += (row + 2) >> 2; + } + bl_y = (bl_y + 1) >> 1; + + ushort4 bl_uv = bl_uv_tex.read(gid); + int bl_cb = int(bl_uv.r >> 6); + int bl_cr = int(bl_uv.g >> 6); + + float y_norm = float(bl_y) / float(params.bl_max + 1u); + float u_norm = float(bl_cb) / float(params.bl_max + 1u); + float v_norm = float(bl_cr) / float(params.bl_max + 1u); + + float pred_cb = dovi_eval_curve(params.poly_cb, params.mmr_cb, + params.poly_y, params.poly_cb, params.poly_cr, bl_cb, + y_norm, u_norm, v_norm, params.bl_max + 1u); + float pred_cr = dovi_eval_curve(params.poly_cr, params.mmr_cr, + params.poly_y, params.poly_cb, params.poly_cr, bl_cr, + y_norm, u_norm, v_norm, params.bl_max + 1u); + + int vdr_cb = dovi_denormalize(pred_cb, params.bl_max + 1u, params.vdr_scale); + int vdr_cr = dovi_denormalize(pred_cr, params.bl_max + 1u, params.vdr_scale); + + if ((params.flags & DOVI_FLAG_DISABLE_RESIDUAL) == 0u) { + bool cb_active = (params.nlq_cb.nlq_method == DOVI_NLQ_METHOD_LINEAR && params.nlq_cb.nlq_scale != 0); + bool cr_active = (params.nlq_cr.nlq_method == DOVI_NLQ_METHOD_LINEAR && params.nlq_cr.nlq_scale != 0); + if (cb_active || cr_active) { + constexpr sampler s(coord::normalized, filter::linear, address::clamp_to_edge); + float2 uv = (float2(gid) + 0.5f) / float2(uv_width, uv_height); + float2 el = dovi_sample_el_2d(el_uv_tex, s, uv, + float2(params.el_width / 2u, params.el_height / 2u)) * dovi_unorm_to_code; + if (cb_active) { + vdr_cb += dovi_eval_inv_nlq(params.nlq_cb, int(rint(el.r))); + } + if (cr_active) { + vdr_cr += dovi_eval_inv_nlq(params.nlq_cr, int(rint(el.g))); + } + } + } + + vdr_cb = clamp(vdr_cb, 0, int(params.vdr_max)); + vdr_cr = clamp(vdr_cr, 0, int(params.vdr_max)); + + int out_cb = dovi_quantize(vdr_cb, params.vdr_scale, params.dither_mode, + dovi_dither(gid, params.frame_index, uint2(1u, 0u)), 64, 960); + int out_cr = dovi_quantize(vdr_cr, params.vdr_scale, params.dither_mode, + dovi_dither(gid, params.frame_index, uint2(1u, 1u)), 64, 960); + + if (params.dither_mode == DOVI_DEBUG_RAW_VDR) { + out_uv_tex.write(ushort4(ushort(vdr_cb << 4), ushort(vdr_cr << 4), 0, 0), gid); + return; + } + if (params.dither_mode == DOVI_DITHER_FULL_PRECISION) { + out_uv_tex.write(ushort4(ushort(dovi_full_precision(vdr_cb, params.vdr_scale, 64, 960)), + ushort(dovi_full_precision(vdr_cr, params.vdr_scale, 64, 960)), 0, 0), gid); + return; + } + out_uv_tex.write(ushort4(ushort(out_cb << 6), ushort(out_cr << 6), 0, 0), gid); +} + +/* ============================================================================ + Layout probe + ========================================================================= */ + +// Reads sentinels back out of the parameter block the host actually uploaded, +// so the Swift serialiser and this shader's struct definitions are checked +// against each other on the GPU rather than by hand. Metal has no static_assert +// and the offline metal compiler is not always installed, so slot 0 carries +// sizeof(DoviMetalParams), slots 1-35 the scalars and the last element of each +// sub-block, and slots 36 onwards, per component, the first element of every array +// and every NLQ scalar: any offset drift or swapped neighbours moves a sentinel +// and the test fails. +kernel void dovi_layout_probe( + constant struct DoviMetalParams& params [[buffer(0)]], + device uint* out [[buffer(1)]], + uint tid [[thread_position_in_grid]]) +{ + if (tid != 0u) { + return; + } + + out[0] = uint(sizeof(struct DoviMetalParams)); + + out[1] = params.bl_width; + out[2] = params.bl_height; + out[3] = params.el_width; + out[4] = params.el_height; + out[5] = params.target_mode; + out[6] = params.dither_mode; + out[7] = params.frame_index; + out[8] = params.flags; + + out[9] = params.poly_y.num_pivots; + out[10] = params.poly_y.pivots[8]; + out[11] = params.poly_y.method[7]; + out[12] = params.poly_y.order[7]; + out[13] = uint(params.poly_y.coef[7][2]); + out[14] = params.poly_y.coef_shift[7]; + out[15] = uint(params.mmr_y.mmr_constant[7]); + out[16] = uint(params.mmr_y.coef[7][20]); + out[17] = params.mmr_y.coef_shift; + out[18] = uint(params.nlq_y.vdr_in_max); + out[19] = params.nlq_y.nlq_pivots[1]; + + out[20] = params.poly_cb.num_pivots; + out[21] = params.poly_cb.coef_shift[7]; + out[22] = uint(params.mmr_cb.coef[7][20]); + out[23] = params.mmr_cb.coef_shift; + out[24] = params.nlq_cb.nlq_pivots[1]; + + out[25] = params.poly_cr.num_pivots; + out[26] = params.poly_cr.coef_shift[7]; + out[27] = uint(params.mmr_cr.coef[7][20]); + out[28] = params.mmr_cr.coef_shift; + out[29] = params.nlq_cr.nlq_pivots[1]; + + out[30] = params.min_pq; + out[31] = params.max_pq; + out[32] = params.avg_pq; + out[33] = params.bl_max; + out[34] = params.vdr_max; + out[35] = params.vdr_scale; + + // --- y: first element of every array and every NLQ scalar --- + out[36] = params.poly_y.pivots[0]; + out[37] = params.poly_y.pivots[1]; + out[38] = params.poly_y.method[0]; + out[39] = params.poly_y.order[0]; + out[40] = uint(params.poly_y.coef[0][0]); + out[41] = uint(params.poly_y.coef[0][1]); + out[42] = uint(params.poly_y.coef[1][0]); + out[43] = params.poly_y.coef_shift[0]; + out[44] = uint(params.mmr_y.mmr_constant[0]); + out[45] = uint(params.mmr_y.coef[0][0]); + out[46] = uint(params.mmr_y.coef[0][1]); + out[47] = uint(params.mmr_y.coef[1][0]); + out[48] = params.nlq_y.nlq_method; + out[49] = uint(params.nlq_y.nlq_offset); + out[50] = uint(params.nlq_y.nlq_scale); + out[51] = uint(params.nlq_y.nlq_threshold); + out[52] = params.nlq_y.nlq_shift; + out[53] = uint(params.nlq_y.vdr_in_max); + out[54] = params.nlq_y.nlq_pivots[0]; + out[55] = params.nlq_y.nlq_pivots[1]; + // --- cb: first element of every array and every NLQ scalar --- + out[56] = params.poly_cb.pivots[0]; + out[57] = params.poly_cb.pivots[1]; + out[58] = params.poly_cb.method[0]; + out[59] = params.poly_cb.order[0]; + out[60] = uint(params.poly_cb.coef[0][0]); + out[61] = uint(params.poly_cb.coef[0][1]); + out[62] = uint(params.poly_cb.coef[1][0]); + out[63] = params.poly_cb.coef_shift[0]; + out[64] = uint(params.mmr_cb.mmr_constant[0]); + out[65] = uint(params.mmr_cb.coef[0][0]); + out[66] = uint(params.mmr_cb.coef[0][1]); + out[67] = uint(params.mmr_cb.coef[1][0]); + out[68] = params.nlq_cb.nlq_method; + out[69] = uint(params.nlq_cb.nlq_offset); + out[70] = uint(params.nlq_cb.nlq_scale); + out[71] = uint(params.nlq_cb.nlq_threshold); + out[72] = params.nlq_cb.nlq_shift; + out[73] = uint(params.nlq_cb.vdr_in_max); + out[74] = params.nlq_cb.nlq_pivots[0]; + out[75] = params.nlq_cb.nlq_pivots[1]; + // --- cr: first element of every array and every NLQ scalar --- + out[76] = params.poly_cr.pivots[0]; + out[77] = params.poly_cr.pivots[1]; + out[78] = params.poly_cr.method[0]; + out[79] = params.poly_cr.order[0]; + out[80] = uint(params.poly_cr.coef[0][0]); + out[81] = uint(params.poly_cr.coef[0][1]); + out[82] = uint(params.poly_cr.coef[1][0]); + out[83] = params.poly_cr.coef_shift[0]; + out[84] = uint(params.mmr_cr.mmr_constant[0]); + out[85] = uint(params.mmr_cr.coef[0][0]); + out[86] = uint(params.mmr_cr.coef[0][1]); + out[87] = uint(params.mmr_cr.coef[1][0]); + out[88] = params.nlq_cr.nlq_method; + out[89] = uint(params.nlq_cr.nlq_offset); + out[90] = uint(params.nlq_cr.nlq_scale); + out[91] = uint(params.nlq_cr.nlq_threshold); + out[92] = params.nlq_cr.nlq_shift; + out[93] = uint(params.nlq_cr.vdr_in_max); + out[94] = params.nlq_cr.nlq_pivots[0]; + out[95] = params.nlq_cr.nlq_pivots[1]; +} +""" + + /// Runtime-compiled form: typedefs, struct block and kernels in one string. + static var source: String { + preamble + "\n" + sharedStructs + "\n" + kernels + "\n" + } +} diff --git a/Tests/AetherEngineTests/DoviComposeAsyncTests.swift b/Tests/AetherEngineTests/DoviComposeAsyncTests.swift new file mode 100644 index 000000000..8d0a4e194 --- /dev/null +++ b/Tests/AetherEngineTests/DoviComposeAsyncTests.swift @@ -0,0 +1,283 @@ +import Testing +import CoreVideo +import Foundation +import Metal +@testable import AetherEngine + +/// `composeAsync` is the playback path: it must produce exactly what the blocking +/// `compose` produces, fire its completion exactly once per call, and never let +/// more than `maxFramesInFlight` compositions be outstanding. +@Suite(.enabled(if: MTLCreateSystemDefaultDevice() != nil, "needs a Metal device")) +struct DoviComposeAsyncTests { + + private static func makeBuffer(width: Int, height: Int, fill: UInt16) -> CVPixelBuffer? { + var out: CVPixelBuffer? + let status = CVPixelBufferCreate(kCFAllocatorDefault, width, height, + kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange, + [kCVPixelBufferMetalCompatibilityKey: true, + kCVPixelBufferIOSurfacePropertiesKey: [:] as CFDictionary] as CFDictionary, + &out) + guard status == kCVReturnSuccess, let buf = out else { return nil } + CVPixelBufferLockBaseAddress(buf, []) + defer { CVPixelBufferUnlockBaseAddress(buf, []) } + for plane in 0.. [UInt16] { + CVPixelBufferLockBaseAddress(buffer, .readOnly) + defer { CVPixelBufferUnlockBaseAddress(buffer, .readOnly) } + guard let base = CVPixelBufferGetBaseAddressOfPlane(buffer, 0) else { return [] } + let rows = CVPixelBufferGetHeightOfPlane(buffer, 0) + let stride = CVPixelBufferGetBytesPerRowOfPlane(buffer, 0) + var out: [UInt16] = [] + out.reserveCapacity(rows * (stride / 2)) + for r in 0.. DoviMetalParams { + var p = DoviMetalParams() + p.blWidth = UInt32(blW) + p.blHeight = UInt32(blH) + p.elWidth = UInt32(elW) + p.elHeight = UInt32(elH) + return p + } + + @Test("a command buffer that fails delivers a failure, not the half-written destination") + func gpuErrorIsAFailure() async throws { + let composer = try #require(DoviFelComposer()) + guard let bl = Self.makeBuffer(width: 256, height: 256, fill: 0x8000), + let el = Self.makeBuffer(width: 128, height: 128, fill: 0x4000) else { return } + let params = Self.identityParams(blW: 256, blH: 256, elW: 128, elH: 128) + composer.forcedCommandBufferFailure.withLock { $0 = true } + + #expect(composer.compose(baseBuffer: bl, enhancementBuffer: el, params: params, frameIndex: 1) == nil) + let delivered: Bool = await withCheckedContinuation { cont in + composer.composeAsync(baseBuffer: bl, enhancementBuffer: el, params: params, frameIndex: 1) { out in + cont.resume(returning: out != nil) + } + } + #expect(!delivered) + + // The slot and the pool were given back: composing works again once the fault is gone. + composer.forcedCommandBufferFailure.withLock { $0 = false } + #expect(composer.compose(baseBuffer: bl, enhancementBuffer: el, params: params, frameIndex: 2) != nil) + } + + @Test("after releaseOutputPool a composition is refused, its slot is returned and no pool is rebuilt", + .timeLimit(.minutes(1))) + func composeAfterTeardownIsRefused() async throws { + let composer = try #require(DoviFelComposer()) + let bl = try #require(Self.makeBuffer(width: 256, height: 256, fill: 0x8000)) + let el = try #require(Self.makeBuffer(width: 128, height: 128, fill: 0x4000)) + let params = Self.identityParams(blW: 256, blH: 256, elW: 128, elH: 128) + #expect(composer.compose(baseBuffer: bl, enhancementBuffer: el, params: params, frameIndex: 1) != nil) + + composer.releaseOutputPool() + #expect(composer.allocatedPoolBuffers == 0) + + #expect(composer.compose(baseBuffer: bl, enhancementBuffer: el, params: params, frameIndex: 2) == nil) + // More refusals than there are in-flight slots: a leaked slot would block here for good. + for frame in 0..<(DoviFelComposer.maxFramesInFlight + 3) { + let delivered: Bool = await withCheckedContinuation { cont in + composer.composeAsync(baseBuffer: bl, enhancementBuffer: el, params: params, frameIndex: UInt32(frame)) { out in + cont.resume(returning: out != nil) + } + } + #expect(!delivered) + } + #expect(composer.allocatedPoolBuffers == 0, "a pool was rebuilt after teardown") + } + + @Test("pool creation is refused once torn down, whatever the caller saw before", .timeLimit(.minutes(1))) + func poolCreationAfterTeardownIsRefused() throws { + let composer = try #require(DoviFelComposer()) + #expect(composer.ensurePoolForTesting(width: 256, height: 256) != nil) + composer.releaseOutputPool() + // The state a racing compose is in when teardown lands after its own look and before its pool request. + #expect(composer.ensurePoolForTesting(width: 256, height: 256) == nil) + #expect(composer.ensurePoolForTesting(width: 512, height: 512) == nil) + #expect(composer.allocatedPoolBuffers == 0) + } + + @Test("the blocking compose waits for an in-flight slot like composeAsync does", .timeLimit(.minutes(1))) + func blockingComposeTakesAnInFlightSlot() throws { + final class Box: @unchecked Sendable { let done = DispatchSemaphore(value: 0); var result: Bool? } + let composer = try #require(DoviFelComposer()) + let bl = try #require(Self.makeBuffer(width: 256, height: 256, fill: 0x8000)) + let el = try #require(Self.makeBuffer(width: 128, height: 128, fill: 0x4000)) + let params = Self.identityParams(blW: 256, blH: 256, elW: 128, elH: 128) + for _ in 0.. [UInt8] { [UInt8(type << 1), 0x01] + payload } + + private static func avcc(_ nals: [[UInt8]]) -> UnsafeMutablePointer { + var bytes: [UInt8] = [] + for n in nals { + let c = n.count + bytes += [UInt8((c >> 24) & 0xFF), UInt8((c >> 16) & 0xFF), UInt8((c >> 8) & 0xFF), UInt8(c & 0xFF)] + bytes += n + } + let pkt = av_packet_alloc()! + _ = av_new_packet(pkt, Int32(bytes.count)) + bytes.withUnsafeBytes { _ = memcpy(pkt.pointee.data, $0.baseAddress, bytes.count) } + return pkt + } + + /// What `separateDualLayers` does with one packet, flags included, and what it returns: the raw RPU + /// NAL and the converted one that a composed frame carries. + static func composedFrameRpu(source nal: [UInt8]) -> (raw: [UInt8], attached: [UInt8])? { + let pkt = avcc([hevcNAL(type: 1, payload: [0xAA, 0xBB]), nal, hevcNAL(type: 63, payload: [0xCC, 0xDD])]) + defer { var p: UnsafeMutablePointer? = pkt; av_packet_free(&p) } + return composedFrameRpu(packet: pkt) + } + + static func composedFrameRpu(packet: UnsafeMutablePointer) -> (raw: [UInt8], attached: [UInt8])? { + let r = DoviRpuConverter.separateDualLayers(packet, extractEL: true, useCustomBlockSource: true, extractParams: true) + var el = r.elPacket + trackedPacketFree(&el) + guard let raw = r.rpuData, let attached = r.convertedRpu else { return nil } + return (raw, attached) + } + + /// The same conversion with another libdovi mode, for the guard test. + static func converted(_ nal: [UInt8], mode: UInt8) -> [UInt8]? { + nal.withUnsafeBufferPointer { buf -> [UInt8]? in + guard let rpu = DoviRpuGuard.parse(buf.baseAddress!, buf.count) else { return nil } + defer { dovi_rpu_free(rpu) } + guard dovi_convert_rpu_with_mode(rpu, mode) == 0, let out = dovi_write_unspec62_nalu(rpu) else { return nil } + defer { dovi_data_free(out) } + return [UInt8](UnsafeBufferPointer(start: out.pointee.data, count: out.pointee.len)) + } + } + + // MARK: - Snapshot of what libdovi reads back + + struct Curve: Equatable { + var mappingIdc: UInt8 + var pivots: [UInt16] + var hasPolynomial: Bool + var hasMMR: Bool + var polyOrderMinus1: [UInt64] + var polyCoefInt: [[Int64]] + var polyCoef: [[UInt64]] + } + + struct Snapshot: Equatable { + var profile: UInt8 + var elType: String? + var rpuFormat: UInt16 + var disableResidual: Bool + var curves: [Curve] + var nlqMethodIdc: Int32 + var hasNLQ: Bool + /// Scalar fields of the VDR DM data (everything but the extension blocks). + var dmScalars: [String] + /// Extension blocks by level, every field of every block. + var blocks: [String: [String]] + + /// Both reshaping curves of every component are the identity polynomial: order 1, y = x. + var isIdentityMapping: Bool { + curves.count == 3 && curves.allSatisfy { c in + c.mappingIdc == 0 && c.hasPolynomial && !c.hasMMR + && c.polyOrderMinus1.allSatisfy { $0 == 0 } + && c.polyCoefInt.allSatisfy { $0 == [0, 1] } + && c.polyCoef.allSatisfy { $0 == [0, 0] } + } + } + + /// No residual: not a dual-layer RPU any more. + var carriesNoEnhancementLayer: Bool { !hasNLQ && elType == nil } + } + + private static func dump(_ v: T, skip: Set = []) -> [String] { + Mirror(reflecting: v).children.compactMap { c in + guard let l = c.label, !skip.contains(l) else { return nil } + return "\(l)=\(c.value)" + } + } + + private static func list(_ list: UnsafePointer?>?, _ n: Int) -> [[String]] { + guard let list else { return [] } + return (0.. Snapshot? { + nal.withUnsafeBufferPointer { buf -> Snapshot? in + guard let rpu = DoviRpuGuard.parse(buf.baseAddress!, buf.count) else { return nil } + defer { dovi_rpu_free(rpu) } + guard let hdr = dovi_rpu_get_header(rpu), let map = dovi_rpu_get_data_mapping(rpu), + let dm = dovi_rpu_get_vdr_dm_data(rpu) else { return nil } + defer { dovi_rpu_free_header(hdr); dovi_rpu_free_data_mapping(map); dovi_rpu_free_vdr_dm_data(dm) } + + var curves: [Curve] = [] + for c in [map.pointee.curves.0, map.pointee.curves.1, map.pointee.curves.2] { + var poly = Curve(mappingIdc: c.mapping_idc, + pivots: c.pivots.data.map { Array(UnsafeBufferPointer(start: $0, count: c.pivots.len)) } ?? [], + hasPolynomial: c.polynomial != nil, hasMMR: c.mmr != nil, + polyOrderMinus1: [], polyCoefInt: [], polyCoef: []) + if let p = c.polynomial?.pointee { + poly.polyOrderMinus1 = p.poly_order_minus1.data.map { Array(UnsafeBufferPointer(start: $0, count: p.poly_order_minus1.len)) } ?? [] + if let l = p.poly_coef_int.list { + poly.polyCoefInt = (0.. 0 { blocks["L2"] = list(d.level2.list, d.level2.len).flatMap { $0 } } + if let b = d.level3 { blocks["L3"] = dump(b.pointee) } + if let b = d.level4 { blocks["L4"] = dump(b.pointee) } + if let b = d.level5 { blocks["L5"] = dump(b.pointee) } + if let b = d.level6 { blocks["L6"] = dump(b.pointee) } + if d.level8.len > 0 { blocks["L8"] = list(d.level8.list, d.level8.len).flatMap { $0 } } + if let b = d.level9 { blocks["L9"] = dump(b.pointee) } + if d.level10.len > 0 { blocks["L10"] = list(d.level10.list, d.level10.len).flatMap { $0 } } + if let b = d.level11 { blocks["L11"] = dump(b.pointee) } + if let b = d.level254 { blocks["L254"] = dump(b.pointee) } + if let b = d.level255 { blocks["L255"] = dump(b.pointee) } + + return Snapshot(profile: hdr.pointee.guessed_profile, + elType: hdr.pointee.el_type.map { String(cString: $0) }, + rpuFormat: hdr.pointee.rpu_format, + disableResidual: hdr.pointee.disable_residual_flag, + curves: curves, nlqMethodIdc: map.pointee.nlq_method_idc, + hasNLQ: map.pointee.nlq != nil, + dmScalars: dump(dm.pointee, skip: ["dm_data"]), blocks: blocks) + } + } + + /// The assertions every composed-frame RPU has to satisfy against the source it came from. + static func expectPinned(source: Snapshot, attached: Snapshot, label: String, + sourceLocation: SourceLocation = #_sourceLocation) { + #expect(attached.profile == 8, "\(label): attached RPU is not Profile 8", sourceLocation: sourceLocation) + #expect(attached.carriesNoEnhancementLayer, "\(label): attached RPU still carries an EL/NLQ part", sourceLocation: sourceLocation) + #expect(attached.isIdentityMapping, "\(label): attached RPU keeps a non-identity mapping (a second reshape on the display side)", sourceLocation: sourceLocation) + #expect(Set(attached.blocks.keys) == Set(source.blocks.keys), + "\(label): extension levels changed, source \(source.blocks.keys.sorted()) attached \(attached.blocks.keys.sorted())", sourceLocation: sourceLocation) + for (level, fields) in source.blocks { + #expect(attached.blocks[level] == fields, "\(label): \(level) differs after conversion", sourceLocation: sourceLocation) + } + #expect(attached.dmScalars == source.dmScalars, "\(label): VDR DM scalars differ after conversion", sourceLocation: sourceLocation) + } + + // MARK: - Synthetic + + @Test("The fixture is a Profile 7 FEL RPU with L1/L2 display metadata") + func fixtureIsFel() throws { + let s = try #require(Self.snapshot(ofNAL: Self.felNAL)) + #expect(s.profile == 7) + #expect(s.elType == "FEL") + #expect(s.hasNLQ) + #expect(!s.isIdentityMapping) + #expect(s.blocks["L1"] != nil) + #expect(s.blocks["L2"] != nil) + } + + @Test("The RPU attached to a composed frame has no-op curves, no EL, and the source's display metadata") + func attachedRpuIsMode2() throws { + let source = try #require(Self.snapshot(ofNAL: Self.felNAL)) + let (raw, attached) = try #require(Self.composedFrameRpu(source: Self.felNAL)) + #expect(raw == Self.felNAL) + #expect(attached.prefix(2).elementsEqual([0x7C, 0x01]), "complete unspec62 NAL, as DoviDisplayMetadata.attach expects") + let out = try #require(Self.snapshot(ofNAL: attached)) + Self.expectPinned(source: source, attached: out, label: "fel_orig") + } + + // MARK: - Real file (CM v4.0) + + static let cmv40Clip = DoviFelRealFileTests.fixtures[2] + static let packetsToCheck = 240 + + /// The mapping of this clip's RPUs is already the identity polynomial, so it proves the display metadata + /// survives (CM v4.0 L3/L8/L9/L254 included) but not the mode-2-vs-5 difference; the fixture above does. + /// The RPUs of the first `packetsToCheck` video packets of the clip, through the production path. + @Test("Real CM v4.0 clip: every composed-frame RPU is no-op mapped and keeps L1/L2/L3/L8/L9/L254 (and every level the source has)", + .enabled(if: DoviFelRealFileTests.available(DoviComposedFrameRpuTests.cmv40Clip), DoviFelRealFileTests.needsClips)) + func realClipKeepsCmv40() throws { + let demuxer = Demuxer() + try demuxer.open(url: DoviFelRealFileTests.url(Self.cmv40Clip)) + defer { demuxer.close() } + let idx = Int32(demuxer.videoStreamIndex) + + var checked = 0, nonIdentitySources = 0 + var levelsBefore = Set(), levelsAfter = Set() + var packets = 0 + while packets < Self.packetsToCheck, let pkt = try demuxer.readPacket() { + defer { + av_packet_unref(pkt) + var o: UnsafeMutablePointer? = pkt + trackedPacketFree(&o) + } + guard pkt.pointee.stream_index == idx else { continue } + packets += 1 + guard let (raw, attached) = Self.composedFrameRpu(packet: pkt) else { continue } + let source = try #require(Self.snapshot(ofNAL: raw)) + let out = try #require(Self.snapshot(ofNAL: attached)) + Self.expectPinned(source: source, attached: out, label: "packet \(packets)") + levelsBefore.formUnion(source.blocks.keys) + levelsAfter.formUnion(out.blocks.keys) + if !source.isIdentityMapping { nonIdentitySources += 1 } + checked += 1 + } + print("DOVI-REAL packets=\(packets) rpus=\(checked) sourceNonIdentity=\(nonIdentitySources) levelsBefore=\(levelsBefore.sorted()) levelsAfter=\(levelsAfter.sorted())") + #expect(checked > 0) + // `expectPinned` already compares every level the source has. This pins that the clip really is + // CM v4.0 (so the check is not vacuous) and that those levels came out the other side. The clip + // has no L10/L11 (nothing to drop); `levelsAfter == levelsBefore` is asserted for whatever it has. + for level in ["L1", "L2", "L3", "L8", "L9", "L254"] { + #expect(levelsBefore.contains(level), "clip no longer carries \(level); the check would be vacuous") + #expect(levelsAfter.contains(level), "\(level) did not survive") + } + #expect(levelsAfter == levelsBefore) + } + + // MARK: - Guard: the assertions can tell mode 2 from modes that keep the mapping + + @Test("Mode 5 keeps the mapping and the EL, so the pinned assertions would catch a regression to it") + func mode5FailsThePin() throws { + let source = try #require(Self.snapshot(ofNAL: Self.felNAL)) + let m5 = try #require(Self.converted(Self.felNAL, mode: 5)) + let kept = try #require(Self.snapshot(ofNAL: m5)) + #expect(kept.profile == 7) + #expect(!kept.carriesNoEnhancementLayer) + #expect(!kept.isIdentityMapping, "mode 5 must not satisfy the identity-mapping assertion, or the pin proves nothing") + // The DM survives mode 5 as well, so the mapping assertions are what separate the two modes. + #expect(kept.blocks == source.blocks) + } + + @Test("Mode 0 (no conversion) fails the pin too") + func mode0FailsThePin() throws { + let m0 = try #require(Self.converted(Self.felNAL, mode: 0)) + let kept = try #require(Self.snapshot(ofNAL: m0)) + #expect(!kept.isIdentityMapping) + #expect(!kept.carriesNoEnhancementLayer) + } +} diff --git a/Tests/AetherEngineTests/DoviCrcValidFuzzTests.swift b/Tests/AetherEngineTests/DoviCrcValidFuzzTests.swift new file mode 100644 index 000000000..47396bdce --- /dev/null +++ b/Tests/AetherEngineTests/DoviCrcValidFuzzTests.swift @@ -0,0 +1,150 @@ +import Testing +import Foundation +import Dovi +@testable import AetherEngine + +/// The existing RPU fuzz mostly dies at the CRC check. This one repairs the CRC after mutating the +/// payload (and re-applies emulation prevention), so the mutated RPU actually reaches libdovi's +/// parser, which is built panic=abort: any upstream panic here would kill the process. +struct DoviCrcValidFuzzTests { + + private static let crcTable: [UInt32] = (0..<256).map { i in + var c = UInt32(i) << 24 + for _ in 0..<8 { c = c & 0x8000_0000 != 0 ? (c << 1) ^ 0x04C1_1DB7 : c << 1 } + return c + } + + static func escape(_ u: [UInt8]) -> [UInt8] { + var out = [UInt8](), zeros = 0 + for (i, b) in u.enumerated() { + if zeros >= 2 && b <= 3 && i >= 2 { out.append(3); zeros = 0 } + out.append(b) + zeros = b == 0 ? zeros + 1 : 0 + } + return out + } + + /// Mutates payload bytes of an unescaped RPU and fixes up the CRC. + static func mutateValid(_ u: [UInt8], _ rng: inout DoviFelFuzzTests.RNG) -> [UInt8] { + var b = u + let tailAt = b.count - 5 + guard tailAt > 4 else { return b } + switch rng.below(4) { + case 0: for _ in 0..<(1 + rng.below(4)) { b[3 + rng.below(tailAt - 3)] ^= UInt8(1 << rng.below(8)) } + case 1: b[3 + rng.below(tailAt - 3)] = UInt8(truncatingIfNeeded: rng.next()) + case 2: // concentrate on the header/mapping region, where the syntax branches + for _ in 0..<(1 + rng.below(3)) { b[3 + rng.below(min(40, tailAt - 3))] ^= UInt8(1 << rng.below(8)) } + default: + let start = 3 + rng.below(tailAt - 3), len = 1 + rng.below(8) + for i in start..> 24) ^ UInt32(b[i]))] } + b[tailAt] = UInt8(crc >> 24); b[tailAt + 1] = UInt8((crc >> 16) & 0xFF) + b[tailAt + 2] = UInt8((crc >> 8) & 0xFF); b[tailAt + 3] = UInt8(crc & 0xFF) + return b + } + + /// Found by `crcValidMutations` (fixture 1): a CRC-valid RPU whose partition count is a 41-zero exp-Golomb + /// code. libdovi parses it and then reads a linear-interp flag (abort); the guard must follow the code past 32 zeros. + static let crcBytesReachLinearFlag = "7c011908090840613650ae20002008020080200802007f801ffc00ffc000ff00ff00000300000300d012811f6bf799ff6bf740000033fed7ee80000067fdafdd00000300cffb5fba000003019ff6bf74000003033fed7ee8000003029cdbf713e290d652833715954c126341284d15cc741520f02a42e5370f5ae1a762d5a57ca0430473c965dc73e4d29f0f297d65394bee5e94a7d40d35adc59e1a36d3f34ee2eb989b55a704d10a2f93177b914f4592097648f189d4419cd595ec49e0ecb7c041984ff7b4f97998e4d1ef509fe7b00ee182b2675ba33e73521f46cfaf3860d29087e92b30b43b2437b1006000004010004000004018000010040010000010060000040100040000072566000035ea2566f9fceb1c256644ca0000030100000300080000030008000003001c36224301860a5e308e051400000301a63e5affff000003000003000003000060207dce0151c030080002300e80c028217c58006d48008351000180580f0010010011a73442000404044129100a000003000460230024183e8000c828340ab0b02e561a80" + + @Test("the guard reads exp-Golomb codes as long as libdovi does") + func guardCoversCrcBytes() throws { + let bytes = stride(from: 0, to: Self.crcBytesReachLinearFlag.count, by: 2).map { + UInt8(Self.crcBytesReachLinearFlag.dropFirst($0).prefix(2), radix: 16)! + } + #expect(bytes.withUnsafeBufferPointer { DoviRpuGuard.crcIsValid($0.baseAddress!, $0.count) }) + #expect(bytes.withUnsafeBufferPointer { DoviRpuGuard.parse($0.baseAddress!, $0.count) } == nil) + } + + /// RPUs built in the test source (`DoviRpuGuardTests.craft`), so the fuzz runs wherever the suite does: + /// a Profile 7 FEL RPU with a two-piece luma curve and MMR chroma, a MEL one whose curves are all + /// polynomial, and a MEL one with MMR chroma. A MEL RPU keeps its mapping through the mode 2 + /// conversion, so these reach the writer's curve code; an FEL RPU's mapping is dropped there. + static let syntheticSeeds: [(name: String, nal: [UInt8])] = [ + ("synthetic FEL", DoviRpuGuardTests.craft(pieces: [[.polynomial, .polynomial], [.mmr], [.mmr]], residual: .fel)), + ("synthetic MEL, polynomial", DoviRpuGuardTests.craft(pieces: [[.polynomial], [.polynomial], [.polynomial]], residual: .mel)), + ("synthetic MEL, MMR chroma", DoviRpuGuardTests.craft(pieces: [[.polynomial], [.mmr], [.mmr]], residual: .mel)), + ] + + /// `AETHER_FUZZ_N` overrides the iteration count; the default is what a CI run can afford per seed + /// for the synthetic seeds and what an unattended soak is worth for a clip. + static func iterations(default fallback: Int) -> Int { + Int(ProcessInfo.processInfo.environment["AETHER_FUZZ_N"] ?? "") ?? fallback + } + + /// Mutates `nal` with the CRC repaired every time, and pushes whatever the guard lets through the + /// way production does: parse, the composer's parameter walk, then the mode 2 conversion and the + /// write (`DoviRpuConverter.convertedUnspec62`). libdovi aborts the process on a panic, so + /// getting to the end is the assertion. + static func fuzz(seed nal: [UInt8], label: String, rngSeed: UInt64, iterations: Int) throws { + try DoviFelFuzzTests.expectLibdoviAccepts(nal) + let u = try #require(nal.withUnsafeBufferPointer { DoviRpuGuard.verifiedUnescaped($0.baseAddress!, $0.count) }) + let env = ProcessInfo.processInfo.environment + var rng = DoviFelFuzzTests.RNG(state: rngSeed + (UInt64(env["AETHER_FUZZ_SEED"] ?? "") ?? 0) << 16) + var reached = 0, guarded = 0, parsed = 0, written = 0 + for _ in 0.. iterations * 3 / 4) + #expect(written > 0, "no mutant survived to the writer: the seed is not exercising it") + } + + @Test("a CRC-valid mutated synthetic RPU never aborts libdovi in parse, convert or write", + .timeLimit(.minutes(5)), arguments: DoviCrcValidFuzzTests.syntheticSeeds.indices) + func crcValidMutationsOfSyntheticSeeds(index: Int) throws { + let seed = Self.syntheticSeeds[index] + try Self.fuzz(seed: seed.nal, label: seed.name, rngSeed: 0x5E0 + UInt64(index), + iterations: Self.iterations(default: 20000)) + } + + static func fuzzClip(_ fixture: Int) throws { + let seed = try #require(try DoviFelFuzzTests.seed(index: fixture)) + try fuzz(seed: seed.rpu, label: "fixture \(fixture)", rngSeed: 0xC4C + UInt64(fixture), + iterations: iterations(default: 20000)) + } + + @Test("a CRC-valid mutated RPU from the first clip never aborts libdovi in parse, convert or write", + .enabled(if: DoviFelRealFileTests.available(DoviFelRealFileTests.fixtures[0]), DoviFelRealFileTests.needsClips), + .timeLimit(.minutes(30))) + func crcValidMutationsClip0() throws { try Self.fuzzClip(0) } + + @Test("a CRC-valid mutated RPU from the second clip never aborts libdovi in parse, convert or write", + .enabled(if: DoviFelRealFileTests.available(DoviFelRealFileTests.fixtures[1]), DoviFelRealFileTests.needsClips), + .timeLimit(.minutes(30))) + func crcValidMutationsClip1() throws { try Self.fuzzClip(1) } + + @Test("a CRC-valid mutated RPU from the third clip never aborts libdovi in parse, convert or write", + .enabled(if: DoviFelRealFileTests.available(DoviFelRealFileTests.fixtures[2]), DoviFelRealFileTests.needsClips), + .timeLimit(.minutes(30))) + func crcValidMutationsClip2() throws { try Self.fuzzClip(2) } +} + +extension DoviCrcValidFuzzTests { + @Test("replay a dumped RPU", .enabled(if: ProcessInfo.processInfo.environment["AETHER_FUZZ_REPLAY"] != nil)) + func replay() throws { + let d = [UInt8](try Data(contentsOf: URL(fileURLWithPath: ProcessInfo.processInfo.environment["AETHER_FUZZ_REPLAY"]!))) + let u = d.withUnsafeBufferPointer { DoviRpuGuard.verifiedUnescaped($0.baseAddress!, $0.count) } + print("[replay] crc ok \(u != nil), guard says unimplemented: \(u.map { DoviRpuGuard.reachesUnimplementedPath($0) } ?? false)") + let r = d.withUnsafeBufferPointer { DoviRpuGuard.parse($0.baseAddress!, $0.count) } + print("[replay] parsed \(r != nil)") + } +} diff --git a/Tests/AetherEngineTests/DoviDisplayMetadataTests.swift b/Tests/AetherEngineTests/DoviDisplayMetadataTests.swift new file mode 100644 index 000000000..30b447c4d --- /dev/null +++ b/Tests/AetherEngineTests/DoviDisplayMetadataTests.swift @@ -0,0 +1,99 @@ +import Testing +import Foundation +import CoreVideo +import IOSurface +@testable import AetherEngine + +/// `DoviDisplayMetadata`: the RPU payload form, and the two attachments tvOS's compositor reads, on the +/// CVBuffer and on its backing IOSurface (where tvOS actually reads them). +struct DoviDisplayMetadataTests { + + private static let nal: [UInt8] = [0x7C, 0x01, 0x19, 0x08, 0x09, 0xAA, 0xBB] + + private static func makeBuffer() throws -> CVPixelBuffer { + var buffer: CVPixelBuffer? + let status = CVPixelBufferCreate( + kCFAllocatorDefault, 64, 64, kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange, + [kCVPixelBufferIOSurfacePropertiesKey: [:] as CFDictionary] as CFDictionary, &buffer) + #expect(status == kCVReturnSuccess) + let b = try #require(buffer) + #expect(CVPixelBufferGetIOSurface(b) != nil, "the buffer must be IOSurface-backed") + return b + } + + private static func surfaceValue(_ b: CVPixelBuffer, _ key: String) -> CFTypeRef? { + guard let s = CVPixelBufferGetIOSurface(b)?.takeUnretainedValue() else { return nil } + return IOSurfaceCopyValue(s, key as CFString) + } + + // MARK: payload(fromNAL:) + + @Test("payload strips the 0x7C 0x01 NAL header and leaves emulation-prevention bytes alone") + func payloadStripsHeader() { + let withEpb: [UInt8] = [0x7C, 0x01, 0x19, 0x00, 0x00, 0x03, 0x01] + #expect(DoviDisplayMetadata.payload(fromNAL: withEpb) == Data([0x19, 0x00, 0x00, 0x03, 0x01])) + #expect(DoviDisplayMetadata.payload(fromNAL: Self.nal) == Data([0x19, 0x08, 0x09, 0xAA, 0xBB])) + } + + @Test("payload passes a header-less RPU (leading 0x19) through unchanged") + func payloadPassesThroughHeaderless() { + let raw: [UInt8] = [0x19, 0x08, 0x09, 0x01] + #expect(DoviDisplayMetadata.payload(fromNAL: raw) == Data(raw)) + } + + @Test("payload returns nil for nil, for too-short input and for anything else") + func payloadRejectsOthers() { + #expect(DoviDisplayMetadata.payload(fromNAL: nil) == nil) + #expect(DoviDisplayMetadata.payload(fromNAL: []) == nil) + #expect(DoviDisplayMetadata.payload(fromNAL: [0x7C, 0x01]) == nil) + #expect(DoviDisplayMetadata.payload(fromNAL: [0x40, 0x01, 0x19, 0x00]) == nil) + #expect(DoviDisplayMetadata.payload(fromNAL: [0x7C, 0x02, 0x19, 0x00]) == nil) + #expect(DoviDisplayMetadata.payload(fromNAL: [0x18, 0x00, 0x00]) == nil) + } + + // MARK: attach + + @Test("attach sets the RPU as CFData and the compatibility ID as a CFString, on the CVBuffer and the IOSurface") + func attachSetsBothKeys() throws { + let b = try Self.makeBuffer() + DoviDisplayMetadata.attach(rpuNAL: Self.nal, to: b) + + let rpu = try #require(CVBufferCopyAttachment(b, DoviDisplayMetadata.rpuDataKey as CFString, nil)) + #expect(CFGetTypeID(rpu) == CFDataGetTypeID()) + #expect((rpu as? Data) == Data([0x19, 0x08, 0x09, 0xAA, 0xBB])) + + // tvOS parses this with CFStringGetIntValue: a CFNumber here aborts backboardd. + let id = try #require(CVBufferCopyAttachment(b, DoviDisplayMetadata.compatibilityIDKey as CFString, nil)) + #expect(CFGetTypeID(id) == CFStringGetTypeID(), "compatibility ID must be a CFString") + #expect((id as? String) == "1") + + // Mirrored onto the backing surface, with the same types. + let surfaceRpu = try #require(Self.surfaceValue(b, DoviDisplayMetadata.rpuDataKey)) + #expect(CFGetTypeID(surfaceRpu) == CFDataGetTypeID()) + let surfaceId = try #require(Self.surfaceValue(b, DoviDisplayMetadata.compatibilityIDKey)) + #expect(CFGetTypeID(surfaceId) == CFStringGetTypeID()) + } + + @Test("attach(nil) clears both keys from the CVBuffer and from the backing IOSurface") + func attachNilClearsEverywhere() throws { + let b = try Self.makeBuffer() + DoviDisplayMetadata.attach(rpuNAL: Self.nal, to: b) + #expect(Self.surfaceValue(b, DoviDisplayMetadata.rpuDataKey) != nil, "precondition: mirrored to the surface") + + DoviDisplayMetadata.attach(rpuNAL: nil, to: b) + + #expect(CVBufferCopyAttachment(b, DoviDisplayMetadata.rpuDataKey as CFString, nil) == nil) + #expect(CVBufferCopyAttachment(b, DoviDisplayMetadata.compatibilityIDKey as CFString, nil) == nil) + #expect(Self.surfaceValue(b, DoviDisplayMetadata.rpuDataKey) == nil, "stale RPU left on the IOSurface") + #expect(Self.surfaceValue(b, DoviDisplayMetadata.compatibilityIDKey) == nil, "stale compatibility ID left on the IOSurface") + } + + @Test("an unusable NAL clears a previous tenant's attachments too") + func attachUnusableClears() throws { + let b = try Self.makeBuffer() + DoviDisplayMetadata.attach(rpuNAL: Self.nal, to: b) + DoviDisplayMetadata.attach(rpuNAL: [0x00, 0x00, 0x00], to: b) + #expect(CVBufferCopyAttachment(b, DoviDisplayMetadata.rpuDataKey as CFString, nil) == nil) + #expect(Self.surfaceValue(b, DoviDisplayMetadata.rpuDataKey) == nil) + } +} diff --git a/Tests/AetherEngineTests/DoviDitherTests.swift b/Tests/AetherEngineTests/DoviDitherTests.swift new file mode 100644 index 000000000..ac2b64f2b --- /dev/null +++ b/Tests/AetherEngineTests/DoviDitherTests.swift @@ -0,0 +1,291 @@ +import Testing +import Foundation +import CoreVideo +import Metal +@testable import AetherEngine + +/// S4 from ``DoviFelMathTests``' list: +/// the ordered dither is unbiased at the output, and Cb and Cr do not share a +/// dither pattern. +/// +/// These run the real `dovi_compose_luma` / `dovi_compose_chroma` kernels and +/// read the planes back, so they fail if the shader stops matching the claim. +/// +/// What they deliberately do *not* lock is the half-step centring in +/// `dovi_dither`. At `vdr_scale <= 64` the raw table `k/64` and the centred table +/// `(k + 0.5)/64` select exactly the same cells, so at the 10-bit-into-12-bit +/// scale of 4 that every Dolby Vision stream uses, the two are bit-identical and +/// no end-to-end test can tell them apart. The centring matters only above that, +/// and the reason it is there is written out in the shader. +struct DoviDitherTests { + + static let width = 64 + static let height = 64 + + static func makeBuffer(width: Int, height: Int) -> CVPixelBuffer? { + var buffer: CVPixelBuffer? + CVPixelBufferCreate( + kCFAllocatorDefault, width, height, + kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange, + [ + kCVPixelBufferMetalCompatibilityKey: true, + kCVPixelBufferIOSurfacePropertiesKey: [:] as CFDictionary + ] as CFDictionary, + &buffer + ) + return buffer + } + + static func fillPlane(_ buffer: CVPixelBuffer, plane: Int, components: Int, value: Int) { + CVPixelBufferLockBaseAddress(buffer, []) + defer { CVPixelBufferUnlockBaseAddress(buffer, []) } + guard let base = CVPixelBufferGetBaseAddressOfPlane(buffer, plane) else { return } + let bytesPerRow = CVPixelBufferGetBytesPerRowOfPlane(buffer, plane) + let w = CVPixelBufferGetWidthOfPlane(buffer, plane) + let h = CVPixelBufferGetHeightOfPlane(buffer, plane) + for row in 0.. [Int] { + CVPixelBufferLockBaseAddress(buffer, .readOnly) + defer { CVPixelBufferUnlockBaseAddress(buffer, .readOnly) } + guard let base = CVPixelBufferGetBaseAddressOfPlane(buffer, plane) else { return [] } + let bytesPerRow = CVPixelBufferGetBytesPerRowOfPlane(buffer, plane) + var out: [Int] = [] + out.reserveCapacity(64) + for row in 0..<8 { + let rowPtr = (base + row * bytesPerRow).assumingMemoryBound(to: UInt16.self) + for column in 0..<8 { + out.append(Int(rowPtr[column * components + component] >> 6)) + } + } + return out + } + + /// A flat curve: `pred_norm` is the constant `c0`, whatever the input code. + /// Used to drive one exact VDR value onto the whole plane, so the only thing + /// varying across the cell is the dither. + static func constantCurve(c0: Int32) -> DoviPolynomialParams { + var poly = DoviPolynomialParams() + poly.numPivots = 2 + poly.pivots[0] = 0 + poly.pivots[1] = 1023 + poly.method[0] = DoviMapping.polynomial.rawValue + poly.order[0] = 1 + poly.coef[0] = [c0, 0, 0] + poly.coefShift[0] = 23 + return poly + } + + /// Smallest `c0` whose denormalised VDR value is exactly `vdr`, and the value + /// the kernel will actually compute from it. + static func curveHitting(vdr: Int, blMax: Int, vdrScale: Int) -> (curve: DoviPolynomialParams, vdr: Int) { + // Curves are normalised by 2^bits (blMax + 1) both ways. + let blNorm = blMax + 1 + let norm = Double(vdr) / (Double(blNorm) * Double(vdrScale)) + let fixed = Int32((norm * Double(1 << 23)).rounded()) + let exact = Double(fixed) / Double(1 << 23) + let produced = Int((exact * Double(blNorm) * Double(vdrScale)).rounded(.toNearestOrEven)) + return (constantCurve(c0: fixed), produced) + } + + static func compose( + lumaCurve: DoviPolynomialParams, + cbCurve: DoviPolynomialParams, + crCurve: DoviPolynomialParams, + dither: DoviDither, + frameIndex: UInt32 + ) -> CVPixelBuffer? { + guard let composer = DoviFelComposer(), + let bl = makeBuffer(width: width, height: height), + let el = makeBuffer(width: width / 2, height: height / 2) else { return nil } + fillPlane(bl, plane: 0, components: 1, value: 512) + fillPlane(bl, plane: 1, components: 2, value: 512) + fillPlane(el, plane: 0, components: 1, value: 512) + fillPlane(el, plane: 1, components: 2, value: 512) + + var params = DoviMetalParams() + params.ditherMode = dither + params.flags = [.disableResidual] + params.polyY = lumaCurve + params.polyCb = cbCurve + params.polyCr = crCurve + return composer.compose(baseBuffer: bl, enhancementBuffer: el, params: params, frameIndex: frameIndex) + } + + // MARK: - Unbiased + + @Test("dither is unbiased: the 8x8 cell mean is the exact pre-rounding value, at every quarter-code residue") + func ditherMeanMatchesExactValue() throws { + // vdr_scale is 4, so the value reaching the rounding step is k + r/4. + // r = 0 is the case where rounding is exact and dither must not move it; + // r = 2 is the case a flat 0.5 bias resolves entirely the wrong way for + // half the levels, which is what dither exists to fix. + for r in 0..<4 { + let target = 500 * 4 + r + let (curve, vdr) = Self.curveHitting(vdr: target, blMax: 1023, vdrScale: 4) + #expect(vdr == target, "curve construction missed the target VDR value: \(vdr) != \(target)") + + guard let dst = Self.compose( + lumaCurve: curve, + cbCurve: Self.constantCurve(c0: 0), + crCurve: Self.constantCurve(c0: 0), + dither: .bayer8x8, + frameIndex: 0 + ) else { + Issue.record("Composer returned nil") + return + } + + let cell = Self.readCell(dst, plane: 0, components: 1, component: 0) + #expect(cell.count == 64) + let mean = Double(cell.reduce(0, +)) / 64.0 + let exact = Double(vdr) / 4.0 + #expect(abs(mean - exact) < 1e-9, + "residue \(r): dithered cell mean \(mean) is not the exact value \(exact)") + + // Only two output levels may appear, and they must bracket the value. + let levels = Set(cell).sorted() + #expect(levels.count <= 2, "residue \(r): dither produced \(levels.count) levels: \(levels)") + for level in levels { + #expect(abs(Double(level) - exact) < 1.0, + "residue \(r): level \(level) is more than a code from \(exact)") + } + } + } + + @Test("the per-frame inversion does not bias either frame of the pair") + func frameInversionIsUnbiased() throws { + // Residue 1: three quarters of the cell stays low, so an inversion that + // drew from a shifted set of offsets would show up as a mean shift here. + let (curve, vdr) = Self.curveHitting(vdr: 500 * 4 + 1, blMax: 1023, vdrScale: 4) + var means: [Double] = [] + for frame in [UInt32(0), UInt32(1)] { + guard let dst = Self.compose( + lumaCurve: curve, + cbCurve: Self.constantCurve(c0: 0), + crCurve: Self.constantCurve(c0: 0), + dither: .bayer8x8, + frameIndex: frame + ) else { + Issue.record("Composer returned nil") + return + } + let cell = Self.readCell(dst, plane: 0, components: 1, component: 0) + means.append(Double(cell.reduce(0, +)) / 64.0) + } + let exact = Double(vdr) / 4.0 + #expect(abs(means[0] - exact) < 1e-9, "frame 0 mean \(means[0]) != \(exact)") + #expect(abs(means[1] - exact) < 1e-9, "frame 1 mean \(means[1]) != \(exact)") + } + + // MARK: - Decorrelated + + @Test("Cb and Cr use different dither phases, so a flat field does not get a shared pattern") + func chromaPhasesAreDecorrelated() throws { + // Residue 2 splits the cell evenly, which is the residue with the most + // pattern to compare. + let (curve, vdr) = Self.curveHitting(vdr: 500 * 4 + 2, blMax: 1023, vdrScale: 4) + + guard let dst = Self.compose( + lumaCurve: Self.constantCurve(c0: 0), + cbCurve: curve, + crCurve: curve, + dither: .bayer8x8, + frameIndex: 0 + ) else { + Issue.record("Composer returned nil") + return + } + + let cb = Self.readCell(dst, plane: 1, components: 2, component: 0) + let cr = Self.readCell(dst, plane: 1, components: 2, component: 1) + #expect(cb.count == 64 && cr.count == 64) + + let exact = Double(vdr) / 4.0 + #expect(abs(Double(cb.reduce(0, +)) / 64.0 - exact) < 1e-9, "Cb cell mean is biased") + #expect(abs(Double(cr.reduce(0, +)) / 64.0 - exact) < 1e-9, "Cr cell mean is biased") + + // Same curve, same input, same frame, so any difference is the phase. + // At the 50% level the selected set is a checkerboard in (x + y) parity, + // and the Cb/Cr offsets differ by (0, 1), which inverts that parity: the + // two patterns come out exact complements, 64 of 64. A weak offset is the + // failure worth catching here rather than a missing one -- the (4, 4) + // diagonal this replaced is a near-symmetry of the matrix and scored 0 of + // 64 at this level, so `cb != cr` alone would not have been enough. + let differing = zip(cb, cr).filter { $0 != $1 }.count + #expect(differing >= 48, + "Cb and Cr dither patterns differ on only \(differing) of 64 pixels; the phase offset is gone or is a symmetry of the Bayer matrix") + } + + // MARK: - Dither actually changes something + + @Test("with dither off the same field collapses to one level, so the cell spread above is the dither's") + func unditheredIsFlat() throws { + let (curve, _) = Self.curveHitting(vdr: 500 * 4 + 2, blMax: 1023, vdrScale: 4) + guard let dst = Self.compose( + lumaCurve: curve, + cbCurve: Self.constantCurve(c0: 0), + crCurve: Self.constantCurve(c0: 0), + dither: .none, + frameIndex: 0 + ) else { + Issue.record("Composer returned nil") + return + } + let cell = Self.readCell(dst, plane: 0, components: 1, component: 0) + #expect(Set(cell).count == 1, "undithered output is not flat: \(Set(cell).sorted())") + } + + /// The 8x8 ordered-dither matrix as a literal table (`k` in `k/64`). + static let bayerGolden: [Int] = [ + 0, 32, 8, 40, 2, 34, 10, 42, + 48, 16, 56, 24, 50, 18, 58, 26, + 12, 44, 4, 36, 14, 46, 6, 38, + 60, 28, 52, 20, 62, 30, 54, 22, + 3, 35, 11, 43, 1, 33, 9, 41, + 51, 19, 59, 27, 49, 17, 57, 25, + 15, 47, 7, 39, 13, 45, 5, 37, + 63, 31, 55, 23, 61, 29, 53, 21, + ] + + /// Recovers every cell of the dither matrix from the GPU and compares it with the golden table. + /// A 16-bit VDR into a 10-bit plane gives `vdr_scale = 64`, so a constant luma of `500*64 + j` + /// rounds up in a cell exactly when `j + k >= 64`; the first `j` at which a cell rises gives `k = 64 - j`. + @Test("the dither matrix the kernel applies is the golden Bayer 8x8, cell for cell") + func ditherMatrixIsGolden() throws { + let composer = try #require(DoviFelComposer()) + var recovered = [Int](repeating: 0, count: 64) + var rose = [Bool](repeating: false, count: 64) + for j in 0..<64 { + var p = DoviMetalParams() + let depthsOK = p.setBitDepths(bl: 10, vdr: 16) + #expect(depthsOK) + p.ditherMode = .bayer8x8 + p.flags = [.disableResidual] + p.frameIndex = 0 + p.polyY = DoviGpuCurveSweepTests.identityCurve() + p.polyCb = DoviGpuCurveSweepTests.identityCurve() + p.polyCr = DoviGpuCurveSweepTests.identityCurve() + p.polyY.numPivots = 2 + p.polyY.pivots[0] = 0; p.polyY.pivots[1] = 1023 + p.polyY.method[0] = DoviMapping.polynomial.rawValue + p.polyY.order[0] = 0 + p.polyY.coefShift[0] = 23 + p.polyY.coef[0] = [Int32((Double(500 * 64 + j) / (1024.0 * 64.0) * 8388608.0).rounded()), 0, 0] + let out = try DoviGpuEdgeCaseTests.composeConstant(composer, blWidth: 16, blHeight: 16, + bl: (500, 512, 512), el: (512, 512, 512), params: p) + let rows = (0..<8).map { DoviGpuCurveSweepTests.readPlane(out, plane: 0, components: 1, row: $0) } + for y in 0..<8 { for x in 0..<8 where !rose[y * 8 + x] && rows[y][x][0] == 501 { + rose[y * 8 + x] = true; recovered[y * 8 + x] = 64 - j + } } + } + #expect(recovered == Self.bayerGolden, "recovered \(recovered)") + } +} diff --git a/Tests/AetherEngineTests/DoviElBitDepthTests.swift b/Tests/AetherEngineTests/DoviElBitDepthTests.swift new file mode 100644 index 000000000..36df3cf37 --- /dev/null +++ b/Tests/AetherEngineTests/DoviElBitDepthTests.swift @@ -0,0 +1,100 @@ +import Testing +import Foundation +import Dovi +@testable import AetherEngine + +/// An RPU whose enhancement layer is not 10-bit is refused (the composer then falls back to the base layer), +/// the same as one whose base layer is not 10-bit. +struct DoviElBitDepthTests { + + private static let crcTable: [UInt32] = (0..<256).map { i in + var c = UInt32(i) << 24 + for _ in 0..<8 { c = c & 0x8000_0000 != 0 ? (c << 1) ^ 0x04C1_1DB7 : c << 1 } + return c + } + + private static func unescape(_ b: [UInt8]) -> [UInt8] { + var out = [UInt8](), zeros = 0 + for x in b { + if zeros >= 2 && x == 3 { zeros = 0; continue } + out.append(x) + zeros = x == 0 ? zeros + 1 : 0 + } + return out + } + + /// Rewrites the CRC of an unescaped RPU (header 7C 01, payload, CRC32, 0x80) after a mutation. + private static func repairCrc(_ u: inout [UInt8]) { + let tailAt = u.count - 5 + var crc: UInt32 = 0xFFFF_FFFF + for i in 3..> 24) ^ UInt32(u[i]))] } + u[tailAt] = UInt8(crc >> 24); u[tailAt + 1] = UInt8((crc >> 16) & 0xFF) + u[tailAt + 2] = UInt8((crc >> 8) & 0xFF); u[tailAt + 3] = UInt8(crc & 0xFF) + } + + private static func header(_ nal: [UInt8]) -> (bl: Int, el: Int, vdr: Int)? { + // Through the guard: a mutated stream can reach paths in libdovi that abort the process. + guard let rpu = nal.withUnsafeBufferPointer({ DoviRpuGuard.parse($0.baseAddress!, $0.count) }) else { return nil } + defer { dovi_rpu_free(rpu) } + guard dovi_rpu_get_error(rpu) == nil, let h = dovi_rpu_get_header(rpu) else { return nil } + defer { dovi_rpu_free_header(h) } + return (Int(h.pointee.bl_bit_depth_minus8) + 8, Int(h.pointee.el_bit_depth_minus8) + 8, Int(h.pointee.vdr_bit_depth_minus8) + 8) + } + + private static func bytes(_ hex: String) -> [UInt8] { + stride(from: 0, to: hex.count, by: 2).map { UInt8(hex.dropFirst($0).prefix(2), radix: 16)! } + } + + /// The header codes el_bit_depth_minus8 = 2 as the exp-Golomb string 011; flipping its last bit gives 010, + /// which is el_bit_depth_minus8 = 1 (9-bit) in the same number of bits, so nothing after it shifts. Find that + /// bit by trying the first bytes and keeping the flip libdovi reports as an el_bit_depth error. + private static func rpuWithNineBitEl(from hex: String) -> [UInt8]? { + let base = unescape(bytes(hex)) + guard let original = header(bytes(hex)), original.el == 10 else { return nil } + for byte in 3.. Int) throws -> [Int] { + let composer = try #require(DoviFelComposer()) + var p = DoviGpuEdgeCaseTests.params(nlqY: DoviGpuEdgeCaseTests.nlq(vdrInMax: 4095)) + p.ditherMode = .debugRawVDR + let bl = try #require(H.makeBuffer(width: 64, height: 8)) + let el = try #require(H.makeBuffer(width: 32, height: 4)) + H.fillPlane(bl, plane: 0, components: 1) { _, _, _ in 512 } + H.fillPlane(bl, plane: 1, components: 2) { _, _, _ in 512 } + H.fillPlane(el, plane: 0, components: 1) { x, _, _ in elColumn(x) } + H.fillPlane(el, plane: 1, components: 2) { _, _, _ in 512 } + let out = try #require(composer.compose(baseBuffer: bl, enhancementBuffer: el, params: p)) + return Array(DoviTestSupport.rawVDR(out)[0][0..<64]) + } + + /// Residual in VDR codes for an EL code, from the DoViBaker formula (positive side): ((2r - 1) * 2048) >> 12. + private func residual(_ el: Int) -> Double { + Double(((2 * (el - 512) - 1) * 2048) >> 12) + } + + @Test("a linear EL ramp is reproduced between EL texel centres, one bilinear step per base-layer pixel") + func rampIsLinearlyInterpolated() throws { + // EL column x is 600 + 4x. BL pixel gx samples EL coordinate gx / 2 - 0.25 (centres at x + 0.5). + let row = try composeLuma { 600 + 4 * $0 } + for gx in 2..<62 { + let elValue = 600.0 + 4.0 * (Double(gx) / 2.0 - 0.25) + let expected = 2048.0 + (elValue - 512.0) - 0.5 // linear in el: |r| - 1/2 + #expect(abs(Double(row[gx]) - expected) <= 1.5, "gx \(gx): \(row[gx]) vs \(expected)") + } + } + + @Test("a step edge in the EL is blended, never overshot, and the plateaus are exact") + func stepHasNoRinging() throws { + let low = 600, high = 700 + let row = try composeLuma { $0 < 16 ? low : high } + let lo = 2048 + Int(residual(low)), hi = 2048 + Int(residual(high)) + for gx in 0..<64 { + #expect(row[gx] >= lo && row[gx] <= hi, "gx \(gx) = \(row[gx]) leaves [\(lo), \(hi)]: the filter rings") + } + #expect(row[10] == lo) + #expect(row[53] == hi) + // The edge sits between EL columns 15 and 16, i.e. BL pixels 31 and 32: one pixel each side is strictly inside. + let inside = (28..<36).filter { row[$0] > lo && row[$0] < hi } + #expect(inside.count >= 2 && inside.count <= 4, "edge transition spans \(inside.count) pixels") + for gx in 1..<64 { #expect(row[gx] >= row[gx - 1], "monotone at \(gx)") } + } +} diff --git a/Tests/AetherEngineTests/DoviFelFuzzTests.swift b/Tests/AetherEngineTests/DoviFelFuzzTests.swift new file mode 100644 index 000000000..0d9c8a0e7 --- /dev/null +++ b/Tests/AetherEngineTests/DoviFelFuzzTests.swift @@ -0,0 +1,252 @@ +import Testing +import Foundation +import AetherLibavformat +import AetherLibavcodec +import AetherLibavutil +import Dovi +@testable import AetherEngine + +/// Corrupt and hostile input must degrade (nil parameters, dropped EL, base layer +/// only), never crash or hang. Seeds are real packets/RPUs from the FEL fixtures; +/// mutations are deterministic so a failure reproduces. +struct DoviFelFuzzTests { + + /// SplitMix64: deterministic, no dependence on the system RNG. + struct RNG { + var state: UInt64 + mutating func next() -> UInt64 { + state &+= 0x9E3779B97F4A7C15 + var z = state + z = (z ^ (z >> 30)) &* 0xBF58476D1CE4E5B9 + z = (z ^ (z >> 27)) &* 0x94D049BB133111EB + return z ^ (z >> 31) + } + mutating func below(_ n: Int) -> Int { n <= 0 ? 0 : Int(next() % UInt64(n)) } + } + + /// First video packet that contains an RPU, as raw bytes, plus one RPU NAL. + static func seed(index: Int = 2) throws -> (packet: [UInt8], rpu: [UInt8])? { + let name = DoviFelRealFileTests.fixtures[index] + guard DoviFelRealFileTests.available(name) else { return nil } + let demuxer = Demuxer() + try demuxer.open(url: DoviFelRealFileTests.url(name)) + defer { demuxer.close() } + let idx = Int32(demuxer.videoStreamIndex) + while let pkt = try demuxer.readPacket() { + defer { + av_packet_unref(pkt) + var o: UnsafeMutablePointer? = pkt + trackedPacketFree(&o) + } + guard pkt.pointee.stream_index == idx, let data = pkt.pointee.data else { continue } + let bytes = [UInt8](UnsafeBufferPointer(start: data, count: Int(pkt.pointee.size))) + let copy = makePacket(bytes) + defer { var o: UnsafeMutablePointer? = copy; trackedPacketFree(&o) } + let r = DoviRpuConverter.separateDualLayers(copy, extractEL: false, useCustomBlockSource: true, extractParams: true) + if let raw = r.rpuData { return (bytes, raw) } + } + return nil + } + + /// Owns its buffer, padded the way FFmpeg packets are. + static func makePacket(_ bytes: [UInt8]) -> UnsafeMutablePointer { + let pkt = trackedPacketAlloc()! + let pad = Int(AV_INPUT_BUFFER_PADDING_SIZE) + let ref = av_buffer_alloc(bytes.count + pad)! + memset(ref.pointee.data, 0, bytes.count + pad) + bytes.withUnsafeBufferPointer { memcpy(ref.pointee.data, $0.baseAddress!, bytes.count) } + pkt.pointee.buf = ref + pkt.pointee.data = ref.pointee.data + pkt.pointee.size = Int32(bytes.count) + pkt.pointee.pts = 0 + pkt.pointee.dts = 0 + return pkt + } + + /// Mutations only probe the parser if they start from an RPU libdovi really accepts. libdovi + /// hands back a handle even when parsing fails, so the error, not the handle, is the verdict. + static func expectLibdoviAccepts(_ nal: [UInt8], sourceLocation: SourceLocation = #_sourceLocation) throws { + let rpu = try #require(nal.withUnsafeBufferPointer { DoviRpuGuard.parse($0.baseAddress!, $0.count) }, + sourceLocation: sourceLocation) + defer { dovi_rpu_free(rpu) } + #expect(dovi_rpu_get_error(rpu) == nil, "\(dovi_rpu_get_error(rpu).map { String(cString: $0) } ?? "")", + sourceLocation: sourceLocation) + #expect(DoviFelComposer.parseRpuParameters(rpu: rpu) != nil, sourceLocation: sourceLocation) + } + + static func mutate(_ input: [UInt8], _ rng: inout RNG) -> [UInt8] { + var b = input + switch rng.below(6) { + case 0: // flip a few bits + for _ in 0..<(1 + rng.below(8)) where !b.isEmpty { b[rng.below(b.count)] ^= UInt8(1 << rng.below(8)) } + case 1: // truncate + b = Array(b.prefix(rng.below(b.count + 1))) + case 2: // overwrite a run with 0xFF / 0x00 + if !b.isEmpty { + let start = rng.below(b.count), len = 1 + rng.below(16) + let v: UInt8 = rng.below(2) == 0 ? 0xFF : 0x00 + for i in start.. 4 { + let start = rng.below(b.count - 1), len = 1 + rng.below(min(32, b.count - start)) + b.insert(contentsOf: b[start..<(start + len)], at: start) + } + default: // extend with junk + for _ in 0.. [UInt8] { + func framed(_ nal: [UInt8]) -> [UInt8] { + let n = UInt32(nal.count) + return [UInt8(n >> 24), UInt8(n >> 16 & 0xFF), UInt8(n >> 8 & 0xFF), UInt8(n & 0xFF)] + nal + } + let base: [UInt8] = [0x02, 0x01] + (0..<48).map { UInt8(truncatingIfNeeded: $0 &* 7 &+ 1) } + let el: [UInt8] = [0x7E, 0x01, 0x02, 0x01] + (0..<24).map { UInt8(truncatingIfNeeded: $0 &* 5 &+ 3) } + return framed(base) + framed(el) + framed(rpu) + } + + static func checkMutatedRpuParsing(seedRPU: [UInt8], label: String, rngState: UInt64, iterations: Int) throws { + #expect(seedRPU.withUnsafeBufferPointer { DoviRpuGuard.crcIsValid($0.baseAddress!, $0.count) }, "a seed RPU must pass the CRC guard") + try expectLibdoviAccepts(seedRPU) + var rng = RNG(state: rngState) + var parsed = 0, rejected = 0 + for _ in 0..= 2 { + let pieces = Int(poly.numPivots) - 1 + for i in 0..= 1) + for n in [p.nlqY, p.nlqCb, p.nlqCr] { + #expect(n.nlqShift <= 23) + #expect(n.vdrInMax >= 0) + } + } else { + rejected += 1 + } + } + print("[fuzz] RPU \(label): \(parsed) parsed, \(rejected) rejected of \(iterations)") + #expect(parsed + rejected > iterations * 3 / 4) + } + + @Test("a mutated synthetic RPU never crashes libdovi or the parameter parser; the result is nil or self-consistent", + .timeLimit(.minutes(2)), arguments: DoviCrcValidFuzzTests.syntheticSeeds.indices) + func mutatedSyntheticRpuParsing(index: Int) throws { + let seed = DoviCrcValidFuzzTests.syntheticSeeds[index] + try Self.checkMutatedRpuParsing(seedRPU: seed.nal, label: seed.name, rngState: 0xFE1 + UInt64(index), iterations: 4000) + } + + @Test("a mutated RPU from a real clip never crashes libdovi or the parameter parser; the result is nil or self-consistent", + .enabled(if: DoviFelRealFileTests.available(DoviFelRealFileTests.fixtures[2]), DoviFelRealFileTests.needsClips), + .timeLimit(.minutes(2))) + func mutatedRpuParsing() throws { + let seed = try #require(try Self.seed()) + try Self.checkMutatedRpuParsing(seedRPU: seed.rpu, label: "clip", rngState: 0xFE1, iterations: 4000) + } + + static func checkMutatedPacketSplitting(seedPacket: [UInt8], useCustomBlockSource: Bool, iterations: Int) { + var rng = RNG(state: useCustomBlockSource ? 0xA11 : 0xB22) + for i in 0..= 0 && off + 4 + n <= size, "iteration \(i): EL NAL overruns its packet") + if off + 4 + n > size { break } + off += 4 + n + } + #expect(off == size || size - off < 4, "iteration \(i): trailing bytes in EL packet") + var o: UnsafeMutablePointer? = el + trackedPacketFree(&o) + } + var o: UnsafeMutablePointer? = pkt + trackedPacketFree(&o) + } + } + + @Test("a mutated or truncated synthetic access unit is split without crashing, and never reads past its buffer", + .timeLimit(.minutes(3)), + arguments: DoviCrcValidFuzzTests.syntheticSeeds.indices, [true, false]) + func mutatedSyntheticPacketSplitting(index: Int, useCustomBlockSource: Bool) { + let packet = Self.syntheticPacket(rpu: DoviCrcValidFuzzTests.syntheticSeeds[index].nal) + Self.checkMutatedPacketSplitting(seedPacket: packet, useCustomBlockSource: useCustomBlockSource, iterations: 1500) + } + + @Test("a mutated or truncated access unit from a real clip is split without crashing, and never reads past its buffer", + .enabled(if: DoviFelRealFileTests.available(DoviFelRealFileTests.fixtures[2]), DoviFelRealFileTests.needsClips), + .timeLimit(.minutes(3)), + arguments: [true, false]) + func mutatedPacketSplitting(useCustomBlockSource: Bool) throws { + let seed = try #require(try Self.seed()) + Self.checkMutatedPacketSplitting(seedPacket: seed.packet, useCustomBlockSource: useCustomBlockSource, iterations: 1500) + } + + @Test("length prefixes that lie about their size are contained by the NAL walker", + arguments: [0xFFFF_FFFF, 0x7FFF_FFFF, 0x8000_0000, 0x0000_1000, 0x0000_0004, 0]) + func lyingLengthPrefix(length: UInt32) throws { + // [len][2-byte NAL header + 6 bytes] — the prefix claims `length`. + var bytes: [UInt8] = [ + UInt8(length >> 24 & 0xFF), UInt8(length >> 16 & 0xFF), UInt8(length >> 8 & 0xFF), UInt8(length & 0xFF), + 0x7E, 0x01, 1, 2, 3, 4, 5, 6, + ] + for variant in 0..<3 { + if variant == 1 { bytes += bytes } // a second, equally wrong, NAL + if variant == 2 { bytes = Array(bytes.prefix(6)) } // cut inside the header + let pkt = Self.makePacket(bytes) + let r = DoviRpuConverter.separateDualLayers(pkt, extractEL: true, useCustomBlockSource: false, extractParams: true) + if let el = r.elPacket { var o: UnsafeMutablePointer? = el; trackedPacketFree(&o) } + var o: UnsafeMutablePointer? = pkt + trackedPacketFree(&o) + } + } + + @Test("empty, tiny and null-data packets are handled") + func degeneratePackets() { + for size in [0, 1, 2, 3, 4, 5] { + let pkt = Self.makePacket([UInt8](repeating: 0x7E, count: size)) + pkt.pointee.size = Int32(size) + let r = DoviRpuConverter.separateDualLayers(pkt, extractEL: true, useCustomBlockSource: false, extractParams: true) + #expect(r.elPacket == nil) + var o: UnsafeMutablePointer? = pkt + trackedPacketFree(&o) + } + let empty = trackedPacketAlloc()! // data == nil + let r = DoviRpuConverter.separateDualLayers(empty, extractEL: true, useCustomBlockSource: true, extractParams: true) + #expect(r.elPacket == nil && r.rpuData == nil) + var o: UnsafeMutablePointer? = empty + trackedPacketFree(&o) + } +} diff --git a/Tests/AetherEngineTests/DoviFelMathTests.swift b/Tests/AetherEngineTests/DoviFelMathTests.swift new file mode 100644 index 000000000..bffd2ed3a --- /dev/null +++ b/Tests/AetherEngineTests/DoviFelMathTests.swift @@ -0,0 +1,323 @@ +import Testing +import Foundation +import CoreVideo +import Metal +import Dovi +@testable import AetherEngine + +/// Numerical checks of the Dolby Vision Profile 7 FEL reconstruction, all driven through the production +/// `DoviFelComposer.compose` with `.debugRawVDR` (12-bit VDR codes, no quantisation) and the real RPU parser. +/// Nothing here re-implements the kernel's arithmetic: expectations are worked out from the DoViBaker +/// formulas (erazortt/DoViBaker, `DoViProcessor.cpp`) and written down as numbers. +struct DoviFelMathTests { + + // Raw 376-byte unspec62 RPU NAL extracted from fel-clip-a.mkv + static let clipARpuHex = "7c011908090840613650ae20002008020080200802007f801ffc00ffc001fffa0000030100000300d0000008000006800000400000340000030200000301a000001000000d0000030080000068000004000003034000002000000a800000800000e18618800000800000800000800000800000800000b0c30c800000800000800000800000800000800000986186800000800000800000800000800000540000040000040000070c30c4000004000004000004000004000004000005861864000004000004000004000004000004000004c30c34000004000004000004000004012000010010010000030100480000400400400000401200001001001000001c95980000d7a8959be7f3ac7095991328000004000003002000000300200000030070d8890c06182978c238145000000698f96bfffc00000300000300000300018081f73805463008015faa4900c0282183180081872b94210002020000030008050000030002261140120c1f4000641f2e1276ffb6cfda80" + + // Raw 225-byte unspec62 RPU NAL extracted from fel-clip-c.mp4 + static let clipCRpuHex = "7c011908090840613650af003ff801ffc00ffc001fffa000001000000d0000030080000068000004000003020090000080080080000080240000200200200000200900000800800800000d12b300001af512b37cfe758e12b3226500000300800000040000030004000003000e1b112180c3052f1847028a00000300d31f2d7fff800000030000030000030030103ee700a8c0300800410a6680c02821800800800800800400020200000300080500000300000300000300120c07d0000201c000f650301c0040038680b080180080080080080080041204ff80008041ac0ad580" + + private static func hexToBytes(_ hex: String) -> [UInt8] { + var bytes: [UInt8] = [] + var i = hex.startIndex + while i < hex.endIndex { + let next = hex.index(i, offsetBy: 2) + if let byte = UInt8(hex[i.. DoviPolynomialParams { + var poly = DoviPolynomialParams() + poly.numPivots = 2 + poly.pivots[0] = 0; poly.pivots[1] = 1023 + poly.method[0] = DoviMapping.polynomial.rawValue + poly.order[0] = order + poly.coefShift[0] = 23 + poly.coef[0] = [Int32(c0 * 8388608.0), Int32(c1 * 8388608.0), Int32(c2 * 8388608.0)] + return poly + } + + private static func rawParams(residual: Bool) -> DoviMetalParams { + var p = DoviGpuEdgeCaseTests.params(flags: residual ? [] : [.disableResidual]) + p.ditherMode = .debugRawVDR + return p + } + + // MARK: - D1 & S2: inverse NLQ + + /// DoViBaker `nonLinearInverseQuantization`: rr = e - offset; rr = 2 rr - sign; dq = rr * S, then a shift + /// that lands in VDR codes; for S = 2048 at denom 23 that is (2|r| - 1) * 2048 >> 12. The kernel reconstructs + /// the magnitude and applies the sign afterwards, so a negative residual truncates toward zero where + /// DoViBaker's arithmetic shift floors: the two differ by at most one code on negative half-steps. + @Test("inverse NLQ through the composer: el = 1023 gives +510 (not 511), el = 0 gives -511, centre gives 0", + arguments: [ + (1023, 2558), // r = +511: (1021 * 2048) >> 12 = 510 + (0, 1537), // r = -512: (1023 * 2048) >> 12 = 511 -> -511 (DoViBaker floors to -512) + (512, 2048), // r = 0 + (768, 2303), // r = +256: (511 * 2048) >> 12 = 255 + (256, 1793), // r = -256: -255 (DoViBaker -256) + (513, 2048), // r = +1: (1 * 2048) >> 12 = 0 + (514, 2049), // r = +2: (3 * 2048) >> 12 = 1 + ]) + func inverseNlqThroughComposer(el: Int, expectedVdr: Int) throws { + let composer = try #require(DoviFelComposer()) + var p = Self.rawParams(residual: true) + p.nlqY = DoviGpuEdgeCaseTests.nlq(vdrInMax: 4095) // not binding: isolate the quantiser + let out = try DoviGpuEdgeCaseTests.composeConstant(composer, bl: (512, 512, 512), el: (el, 512, 512), params: p) + #expect(Set(DoviTestSupport.rawVDR(out)[0]) == [expectedVdr], "el=\(el)") + } + + // MARK: - D3: delta-coded pivots + + @Test("D3: the parser prefix-sums delta-coded pivots into absolute breakpoints") + func pivotPrefixSumAccumulation() { + let clipABytes = Self.hexToBytes(Self.clipARpuHex) + guard let rpu = dovi_parse_unspec62_nalu(clipABytes, clipABytes.count) else { + Issue.record("Failed to parse clip A RPU") + return + } + defer { dovi_rpu_free(rpu) } + // libdovi returns a handle even when parsing fails; only the error says it was accepted. + #expect(dovi_rpu_get_error(rpu) == nil, "\(dovi_rpu_get_error(rpu).map { String(cString: $0) } ?? "")") + guard let params = DoviFelComposer.parseRpuParameters(rpu: rpu) else { + Issue.record("Failed to parse clip A RPU") + return + } + + // Luma pivots from clip A are delta-coded: 0 128 128 128 128 128 128 128 127 + let expectedLumaPivots: [UInt32] = [0, 128, 256, 384, 512, 640, 768, 896, 1023] + #expect(params.polyY.numPivots == 9) + for i in 0..<9 { + #expect(params.polyY.pivots[i] == expectedLumaPivots[i], Comment("Pivot \(i) expected \(expectedLumaPivots[i]), got \(params.polyY.pivots[i])")) + } + } + + /// DoViBaker `getPivotIndex`: the first piece whose upper pivot the sample is below, and the last piece for + /// anything at or above the top pivot. Pieces here are [0,128), [128,256), [256,1023] and each one's constant + /// is distinct, so the output names the piece that was selected. + @Test("piece selection on the GPU: boundaries belong to the upper piece and the top code selects the last piece") + func pieceSelectionOnGpu() throws { + let composer = try #require(DoviFelComposer()) + var poly = DoviPolynomialParams() + poly.numPivots = 4 + poly.pivots[0] = 0; poly.pivots[1] = 128; poly.pivots[2] = 256; poly.pivots[3] = 1023 + let levels = [0.25, 0.5, 0.75] + for piece in 0..<3 { + poly.method[piece] = DoviMapping.polynomial.rawValue + poly.order[piece] = 0 + poly.coefShift[piece] = 23 + poly.coef[piece] = [Int32(levels[piece] * 8388608.0), 0, 0] + } + var p = Self.rawParams(residual: false) + p.polyY = poly + + let bl = try #require(H.makeBuffer(width: 1024, height: 4)) + let el = try #require(H.makeBuffer(width: 512, height: 2)) + H.fillPlane(bl, plane: 0, components: 1) { c, _, _ in min(c, 1023) } + H.fillPlane(bl, plane: 1, components: 2) { _, _, _ in 512 } + H.fillPlane(el, plane: 0, components: 1) { _, _, _ in 512 } + H.fillPlane(el, plane: 1, components: 2) { _, _, _ in 512 } + let out = try #require(composer.compose(baseBuffer: bl, enhancementBuffer: el, params: p)) + let row = Array(DoviTestSupport.rawVDR(out)[0][0..<1024]) + let expected = [(0, 1024), (127, 1024), (128, 2048), (255, 2048), (256, 3072), (1000, 3072), (1023, 3072)] + for (code, vdr) in expected { + #expect(row[code] == vdr, "BL code \(code)") + } + } + + // MARK: - D4 & N1: polynomial coefficients and denominator + + /// DoViBaker `polynompialMapping`: the input is the BL code over 2^bl_bit_depth (1024, not 1023) and the + /// result is in 12-bit VDR codes after 2^vdr_bit_depth (4096). + @Test("polynomial reshaping on the GPU matches the closed form over all 1024 base-layer codes") + func polynomialOverAllCodes() throws { + let composer = try #require(DoviFelComposer()) + let c0 = 0.02, c1 = 0.95, c2 = 0.10 + var p = Self.rawParams(residual: false) + p.polyY = Self.polynomial(c0, c1, c2, order: 2) + // The coefficients the kernel actually receives, so the comparison is not about Int32 truncation. + let q = p.polyY.coef[0].map { Double($0) / 8388608.0 } + + let bl = try #require(H.makeBuffer(width: 1024, height: 4)) + let el = try #require(H.makeBuffer(width: 512, height: 2)) + H.fillPlane(bl, plane: 0, components: 1) { c, _, _ in c } + H.fillPlane(bl, plane: 1, components: 2) { _, _, _ in 512 } + H.fillPlane(el, plane: 0, components: 1) { _, _, _ in 512 } + H.fillPlane(el, plane: 1, components: 2) { _, _, _ in 512 } + let out = try #require(composer.compose(baseBuffer: bl, enhancementBuffer: el, params: p)) + let row = DoviTestSupport.rawVDR(out)[0] + var worst = 0.0 + for code in 0..<1024 { + let x = Double(code) / 1024.0 + let exact = min(max((q[0] + q[1] * x + q[2] * x * x) * 4096.0, 0), 4095) + worst = max(worst, abs(Double(row[code]) - exact)) + } + #expect(worst <= 1.0, "worst deviation \(worst) VDR codes") + } + + // MARK: - N3: prediction overshoot survives until the residual is added + + @Test("N3: a prediction above the VDR range keeps its overshoot until the residual is added") + func predictionOvershootKeptBeforeResidual() throws { + let composer = try #require(DoviFelComposer()) + // Prediction 1.02 -> 4178 12-bit codes (above vdr_max 4095); EL 312 -> r = -200 -> (399 * 2048) >> 12 = 199 -> -199. + // Clamping the prediction first would give 4095 - 199 = 3896; adding first gives 4178 - 199 = 3979. + var p = Self.rawParams(residual: true) + p.polyY = Self.polynomial(1.02, order: 0) + p.nlqY = DoviGpuEdgeCaseTests.nlq(vdrInMax: 4095) + let out = try DoviGpuEdgeCaseTests.composeConstant(composer, bl: (512, 512, 512), el: (312, 512, 512), params: p) + #expect(Set(DoviTestSupport.rawVDR(out)[0]) == [3979]) + } + + // MARK: - D2: chroma MMR on a real RPU + + @Test("D2: clip A order-3 chroma MMR parses, and maps mid-grey to mid-grey and a Cb sweep to the pinned curve") + func chromaMmrOnRealRpu() throws { + let clipABytes = Self.hexToBytes(Self.clipARpuHex) + let rpu = try #require(dovi_parse_unspec62_nalu(clipABytes, clipABytes.count)) + defer { dovi_rpu_free(rpu) } + #expect(dovi_rpu_get_error(rpu) == nil, "\(dovi_rpu_get_error(rpu).map { String(cString: $0) } ?? "")") + var params = try #require(DoviFelComposer.parseRpuParameters(rpu: rpu)) + + #expect(params.polyCb.usesMmr, "Cb curve must be MMR on clip A") + #expect(params.polyCr.usesMmr, "Cr curve must be MMR on clip A") + #expect(params.polyCb.order[0] == 3, Comment("Cb MMR order must be 3, got \(params.polyCb.order[0])")) + #expect(params.polyCr.order[0] == 3, Comment("Cr MMR order must be 3, got \(params.polyCr.order[0])")) + #expect(params.mmrCb.coefShift == 23, "MMR coef shift must be 23") + + params.ditherMode = .debugRawVDR + params.flags.insert(.disableResidual) + let composer = try #require(DoviFelComposer()) + + // Chroma plane is 1024 wide for a 2048-wide base layer: column c carries Cb = c, with Y and Cr at mid-grey. + let bl = try #require(H.makeBuffer(width: 2048, height: 4)) + let el = try #require(H.makeBuffer(width: 1024, height: 2)) + H.fillPlane(bl, plane: 0, components: 1) { _, _, _ in 512 } + H.fillPlane(bl, plane: 1, components: 2) { c, _, comp in comp == 0 ? c : 512 } + H.fillPlane(el, plane: 0, components: 1) { _, _, _ in 512 } + H.fillPlane(el, plane: 1, components: 2) { _, _, _ in 512 } + let out = try #require(composer.compose(baseBuffer: bl, enhancementBuffer: el, params: params)) + let planes = DoviTestSupport.rawVDR(out, chromaOnly: true) + let cb = Array(planes[1][0..<1024]).map { Double($0) / 4.0 } // 10-bit equivalent codes + let cr = Array(planes[2][0..<1024]).map { Double($0) / 4.0 } + + // Mid-grey in, mid-grey out: the property the flat-64 green collapse violated. + #expect(abs(cb[512] - 512.0) < 2.0, "Cb(512) must be ~512, got \(cb[512])") + #expect(abs(cr[512] - 512.0) < 2.0, "Cr at mid-grey input must be ~512, got \(cr[512])") + + let span = (cb.max() ?? 0) - (cb.min() ?? 0) + #expect(span > 500.0, "Cb curve is nearly flat over the full input range (span \(span) codes)") + var violations = 0 + for code in 1..<1024 where cb[code] < cb[code - 1] - 1.0 { violations += 1 } + #expect(violations == 0, "Cb curve must be non-decreasing in its own component, \(violations) violations") + + // Regression anchors measured off this RPU through the composer (not independently derived; mid-grey above + // is the derived check). They catch a piece, an order or a coefficient index shifting. + for (code, anchor) in Self.chromaAnchors { + #expect(abs(cb[code] - anchor) < 1.0, "Cb(\(code)) drifted: \(cb[code]), pinned \(anchor)") + } + } + + private static let chromaAnchors: [(Int, Double)] = [(0, 0.0), (256, 222.5), (768, 886.75), (1023, 1023.75)] + + // MARK: - Metal GPU Composition & Pixel Sanity Assertions + + @Test("Metal GPU Composition: Real RPU reconstructs 10-bit P010 without flat-64 green chroma") + func metalGpuCompositionSanity() throws { + guard let composer = DoviFelComposer() else { + Issue.record("Failed to initialize DoviFelComposer") + return + } + + let clipABytes = Self.hexToBytes(Self.clipARpuHex) + guard let rpu = dovi_parse_unspec62_nalu(clipABytes, clipABytes.count) else { + Issue.record("Failed to parse clip A RPU") + return + } + defer { dovi_rpu_free(rpu) } + // libdovi returns a handle even when parsing fails; only the error says it was accepted. + #expect(dovi_rpu_get_error(rpu) == nil, "\(dovi_rpu_get_error(rpu).map { String(cString: $0) } ?? "")") + guard let params = DoviFelComposer.parseRpuParameters(rpu: rpu) else { + Issue.record("Failed to parse clip A RPU") + return + } + + let w = 256, h = 256 + let pixelFormat = kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange + + var blBufferOpt: CVPixelBuffer? + CVPixelBufferCreate(kCFAllocatorDefault, w, h, pixelFormat, [ + kCVPixelBufferMetalCompatibilityKey: true, + kCVPixelBufferIOSurfacePropertiesKey: [:] as CFDictionary + ] as CFDictionary, &blBufferOpt) + + var elBufferOpt: CVPixelBuffer? + CVPixelBufferCreate(kCFAllocatorDefault, w / 2, h / 2, pixelFormat, [ + kCVPixelBufferMetalCompatibilityKey: true, + kCVPixelBufferIOSurfacePropertiesKey: [:] as CFDictionary + ] as CFDictionary, &elBufferOpt) + + guard let blBuffer = blBufferOpt, let elBuffer = elBufferOpt else { + Issue.record("Failed to allocate CVPixelBuffers") + return + } + + // Mid-grey base layer (luma 512, Cb/Cr 512) and a neutral enhancement + // layer (512, the NLQ offset, so the residual is zero). + // + // Both go through the stride-aware helper. The previous version walked each + // plane as `width * height` contiguous samples, which ignores row padding + // and, for the enhancement layer's chroma plane, computed twice the + // sample count the plane holds: it wrote 16 KiB past the end and faulted + // whenever the pages behind the IOSurface happened not to be mapped. + DoviGpuCurveSweepTests.fillPlane(blBuffer, plane: 0, components: 1) { _, _, _ in 512 } + DoviGpuCurveSweepTests.fillPlane(blBuffer, plane: 1, components: 2) { _, _, _ in 512 } + DoviGpuCurveSweepTests.fillPlane(elBuffer, plane: 0, components: 1) { _, _, _ in 512 } + DoviGpuCurveSweepTests.fillPlane(elBuffer, plane: 1, components: 2) { _, _, _ in 512 } + + // Execute Metal GPU composition + guard let dst = composer.compose( + baseBuffer: blBuffer, + enhancementBuffer: elBuffer, + params: params, + frameIndex: 1 + ) else { + Issue.record("Composer returned nil") + return + } + + // Read back output pixels and assert that chroma is NOT flat 64 + CVPixelBufferLockBaseAddress(dst, .readOnly) + defer { CVPixelBufferUnlockBaseAddress(dst, .readOnly) } + + guard let outUvPtr = CVPixelBufferGetBaseAddressOfPlane(dst, 1) else { + Issue.record("No output chroma plane") + return + } + + let outUv = outUvPtr.assumingMemoryBound(to: UInt16.self) + let sampleCb = Int(outUv[0] >> 6) + let sampleCr = Int(outUv[1] >> 6) + + // Mid-gray in -> mid-gray out (~512). A flat 64 would be the green-frame failure. + #expect(sampleCb > 400 && sampleCb < 600, "Output Cb sample must be near mid-gray (~512), was \(sampleCb) (D2 flat 64 regression)") + #expect(sampleCr > 400 && sampleCr < 600, "Output Cr sample must be near mid-gray (~512), was \(sampleCr) (D2 flat 64 regression)") + + guard let outYPtr = CVPixelBufferGetBaseAddressOfPlane(dst, 0) else { + Issue.record("No output luma plane") + return + } + let outY = outYPtr.assumingMemoryBound(to: UInt16.self) + let sampleY = Int(outY[0] >> 6) + #expect(sampleY > 400 && sampleY < 600, "Output Y sample must be near mid-gray (~512), was \(sampleY)") + } +} diff --git a/Tests/AetherEngineTests/DoviFelRealFileTests.swift b/Tests/AetherEngineTests/DoviFelRealFileTests.swift new file mode 100644 index 000000000..992f4bd41 --- /dev/null +++ b/Tests/AetherEngineTests/DoviFelRealFileTests.swift @@ -0,0 +1,46 @@ +import Testing +import Foundation +@testable import AetherEngine + +/// End-to-end FEL checks on real Profile 7 files, where the synthetic suites +/// cannot reach: container framing quirks (the Matroska `packet->data != buf->data` +/// case that produced zero frames), real RPUs, real reorder, seeks and teardown. +/// +/// The fixtures (`fel-clip-a.mkv`, `fel-clip-b.mkv`, `fel-clip-c.mp4`) live in the +/// directory named by `DV_FEL_TESTS_DIR`, or in `DV FEL Tests/` next to the package; +/// they are not checked in, and every test skips when its file is absent. +struct DoviFelRealFileTests { + + static let fixtures = [ + "fel-clip-a.mkv", + "fel-clip-b.mkv", + "fel-clip-c.mp4", + ] + + static func url(_ name: String) -> URL { + if let dir = ProcessInfo.processInfo.environment["DV_FEL_TESTS_DIR"] { + return URL(fileURLWithPath: dir).appendingPathComponent(name) + } + return URL(fileURLWithPath: #filePath) + .deletingLastPathComponent() // AetherEngineTests + .deletingLastPathComponent() // Tests + .deletingLastPathComponent() // AetherEngine + .deletingLastPathComponent() // repo root + .appendingPathComponent("DV FEL Tests") + .appendingPathComponent(name) + } + + static func available(_ name: String) -> Bool { + FileManager.default.fileExists(atPath: url(name).path) + } + + /// The `.enabled(if:)` reason of every test that needs a clip: the clips are not checked in, so + /// those tests show as skipped without them rather than passing without having run. + static let needsClips: Comment = "needs the DV FEL test clips (not checked in): set DV_FEL_TESTS_DIR" + + /// The fixtures present, for `arguments:` of a test that runs once per clip. + static var availableFixtures: [String] { fixtures.filter(available) } + + /// The first two clips, when present: the ones long enough to seek into. + static var firstTwoClips: [String] { [fixtures[0], fixtures[1]].filter(available) } +} diff --git a/Tests/AetherEngineTests/DoviGpuAdversarialTests.swift b/Tests/AetherEngineTests/DoviGpuAdversarialTests.swift new file mode 100644 index 000000000..e39d0f257 --- /dev/null +++ b/Tests/AetherEngineTests/DoviGpuAdversarialTests.swift @@ -0,0 +1,210 @@ +import Testing +import Foundation +import CoreVideo +@testable import AetherEngine + +/// Hostile parameter blocks and content thrown at the composition kernels. The RPU guard and the +/// parser should never let such values through, but the kernels are the last line: whatever arrives, +/// they must neither hang nor write outside the legal output range, and must be deterministic. +struct DoviGpuAdversarialTests { + + typealias H = DoviGpuCurveSweepTests + + struct RNG: RandomNumberGenerator { + var s: UInt64 + mutating func next() -> UInt64 { s = s &* 6364136223846793005 &+ 1442695040888963407; return s ^ (s >> 29) } + } + + static let extremes: [Int32] = [0, 1, -1, 2, 1 << 15, -(1 << 15), 1 << 23, -(1 << 23), 1 << 30, Int32.max, Int32.min, Int32.min + 1] + + static func i32(_ r: inout RNG) -> Int32 { + r.next() % 3 == 0 ? extremes[Int(r.next() % UInt64(extremes.count))] : Int32(truncatingIfNeeded: r.next()) + } + static func u32(_ r: inout RNG, _ cap: UInt32) -> UInt32 { + r.next() % 4 == 0 ? [0, 1, cap, cap + 1, UInt32.max][Int(r.next() % 5)] : UInt32(r.next() % UInt64(cap + 1)) + } + + static func hostileCurve(_ r: inout RNG) -> DoviPolynomialParams { + var c = DoviPolynomialParams() + c.numPivots = u32(&r, 12) + for i in 0.. DoviMmrParams { + var m = DoviMmrParams() + for i in 0.. DoviNlqParams { + var n = DoviNlqParams() + n.nlqMethod = r.next() % 4 == 0 ? .none : .linear + n.nlqOffset = i32(&r); n.nlqScale = i32(&r); n.nlqThreshold = i32(&r); n.vdrInMax = i32(&r) + n.nlqShift = u32(&r, 40) + return n + } + + static func hostile(_ r: inout RNG, w: Int, h: Int) -> DoviMetalParams { + var p = DoviMetalParams() + p.blWidth = UInt32(w); p.blHeight = UInt32(h); p.elWidth = UInt32(w / 2); p.elHeight = UInt32(h / 2) + p.ditherMode = [.none, .bayer8x8, .fullPrecision][Int(r.next() % 3)] + p.frameIndex = UInt32(truncatingIfNeeded: r.next()) + var f = DoviComposerFlags() + if r.next() % 4 == 0 { f.insert(.disableResidual) } + if r.next() % 2 == 0 { f.insert(.lumaMmr) } + p.flags = f + p.polyY = hostileCurve(&r); p.polyCb = hostileCurve(&r); p.polyCr = hostileCurve(&r) + p.mmrY = hostileMmr(&r); p.mmrCb = hostileMmr(&r); p.mmrCr = hostileMmr(&r) + p.nlqY = hostileNlq(&r); p.nlqCb = hostileNlq(&r); p.nlqCr = hostileNlq(&r) + // Bit depths straight from the parser's validated set, plus one the setter refuses. + let depths = [(10, 12), (10, 10), (8, 12), (10, 16), (12, 12), (16, 16)] + let d = depths[Int(r.next() % UInt64(depths.count))] + _ = p.setBitDepths(bl: d.0, vdr: d.1) + if r.next() % 8 == 0 { p.blMax = u32(&r, 70000); p.vdrMax = u32(&r, 70000); p.vdrScale = u32(&r, 70000) } + return p + } + + static func words(_ b: CVPixelBuffer, plane: Int) -> [UInt16] { + CVPixelBufferLockBaseAddress(b, .readOnly); defer { CVPixelBufferUnlockBaseAddress(b, .readOnly) } + let base = CVPixelBufferGetBaseAddressOfPlane(b, plane)! + let bpr = CVPixelBufferGetBytesPerRowOfPlane(b, plane) + let w = CVPixelBufferGetWidthOfPlane(b, plane) * (plane == 0 ? 1 : 2), hh = CVPixelBufferGetHeightOfPlane(b, plane) + var out: [UInt16] = [] + for y in 0.. Int { full ? Int(s.next() & 0xFFFF) >> 6 : Int(s.next() % 1024) } + H.fillPlane(bl, plane: 0, components: 1) { _, _, _ in code(&s) } + H.fillPlane(bl, plane: 1, components: 2) { _, _, _ in code(&s) } + H.fillPlane(el, plane: 0, components: 1) { _, _, _ in code(&s) } + H.fillPlane(el, plane: 1, components: 2) { _, _, _ in code(&s) } + + let a = try #require(composer.compose(baseBuffer: bl, enhancementBuffer: el, params: params, frameIndex: 0), + "seed \(seed) iter \(iter) refused a well-formed buffer pair") + let aY = Self.words(a, plane: 0), aC = Self.words(a, plane: 1) + let b = try #require(composer.compose(baseBuffer: bl, enhancementBuffer: el, params: params, frameIndex: 0)) + #expect(aY == Self.words(b, plane: 0) && aC == Self.words(b, plane: 1), "seed \(seed) iter \(iter) is not deterministic") + + let bad = aY.filter { ($0 >> 6) < 64 || ($0 >> 6) > 940 }.count + aC.filter { ($0 >> 6) < 64 || ($0 >> 6) > 960 }.count + #expect(bad == 0, "seed \(seed) iter \(iter): \(bad) samples outside video range (dither \(params.ditherMode), blMax \(params.blMax), vdrMax \(params.vdrMax), vdrScale \(params.vdrScale))") + } + } + + @Test("parameters that lie about the buffer size cannot read or write out of bounds") + func lyingGeometry() throws { + let composer = try #require(DoviFelComposer()) + let bl = try #require(H.makeBuffer(width: 64, height: 32)) + let el = try #require(H.makeBuffer(width: 32, height: 16)) + H.fillPlane(bl, plane: 0, components: 1) { x, y, _ in (x * 7 + y * 13) & 1023 } + H.fillPlane(bl, plane: 1, components: 2) { x, y, _ in (x * 5 + y * 3) & 1023 } + H.fillPlane(el, plane: 0, components: 1) { x, y, _ in (x * 11 + y) & 1023 } + H.fillPlane(el, plane: 1, components: 2) { x, y, _ in (x + y * 9) & 1023 } + let cases: [(UInt32, UInt32, UInt32, UInt32)] = [ + (0, 0, 0, 0), (1, 1, 1, 1), (64, 32, 0, 0), (64, 32, 1, 1), (64, 32, 4096, 4096), + (4096, 4096, 32, 16), (UInt32.max, UInt32.max, UInt32.max, UInt32.max), (64, 32, 31, 15), (65, 33, 32, 16), + ] + for (bw, bh, ew, eh) in cases { + var p = DoviGpuEdgeCaseTests.params() + p.blWidth = bw; p.blHeight = bh; p.elWidth = ew; p.elHeight = eh + // Either refused or composed at the real buffer geometry; never a crash or a hang. + if let out = composer.compose(baseBuffer: bl, enhancementBuffer: el, params: p, frameIndex: 0) { + #expect(CVPixelBufferGetWidth(out) == 64 && CVPixelBufferGetHeight(out) == 32, "output must follow the real base buffer, not the claimed size") + let y = Self.words(out, plane: 0) + #expect(y.allSatisfy { ($0 >> 6) >= 64 && ($0 >> 6) <= 940 }) + } + } + } +} + +extension DoviGpuAdversarialTests { + /// Production drives the composer through the async API, which owns the in-flight slot accounting; + /// the synchronous `compose` is single-threaded by contract. + @Test("concurrent async composes at changing resolutions neither deadlock nor corrupt each other") + func concurrentComposeWithResolutionChanges() async throws { + let composer = try #require(DoviFelComposer()) + let sizes = [(64, 16), (128, 32), (96, 48), (256, 64)] + let results = await withTaskGroup(of: Bool.self, returning: [Bool].self) { group in + for t in 0..<6 { + group.addTask { + var ok = true + for i in 0..<60 { + let (w, h) = sizes[(t + i / 5) % sizes.count] + guard let bl = H.makeBuffer(width: w, height: h), let el = H.makeBuffer(width: w / 2, height: h / 2) else { return false } + H.fillPlane(bl, plane: 0, components: 1) { _, _, _ in 512 + t * 10 } + H.fillPlane(bl, plane: 1, components: 2) { _, _, _ in 512 } + H.fillPlane(el, plane: 0, components: 1) { _, _, _ in 612 } + H.fillPlane(el, plane: 1, components: 2) { _, _, _ in 512 } + let good: Bool = await withCheckedContinuation { c in + composer.composeAsync(baseBuffer: bl, enhancementBuffer: el, + params: DoviGpuEdgeCaseTests.params(), frameIndex: UInt32(i)) { out in + guard let out else { c.resume(returning: false); return } + c.resume(returning: CVPixelBufferGetWidth(out) == w && Set(Self.words(out, plane: 0)).count == 1) + } + } + ok = ok && good + } + return ok + } + } + var r: [Bool] = [] + for await v in group { r.append(v) } + return r + } + #expect(results.count == 6 && results.allSatisfy { $0 }) + composer.acquireSlot(); composer.releaseSlot() + } +} + +extension DoviGpuAdversarialTests { + /// The spec's fixed-point pseudo-code and the DoViBaker oracle normalise by 2^bits both + /// ways, so a curve value of exactly one half is exactly the middle of the VDR range. Normalising by + /// 2^bits - 1 put the same neutral chroma at 2049 (clip A) and as low as 2041 (clip B). + @Test("a curve value of one half lands on exactly 2048, through polynomial and MMR constants", + arguments: [false, true]) + func halfIsExactlyMidRange(mmr: Bool) throws { + let composer = try #require(DoviFelComposer()) + var p = DoviGpuEdgeCaseTests.params(flags: [.disableResidual]) + p.ditherMode = .debugRawVDR + var curve = DoviPolynomialParams() + curve.numPivots = 2; curve.pivots[0] = 0; curve.pivots[1] = 1023 + curve.order[0] = 1; curve.coefShift[0] = 23 + if mmr { + curve.method[0] = DoviMapping.mmr.rawValue + var m = DoviMmrParams(); m.constant[0] = 1 << 22; m.coefShift = 23 + p.mmrCb = m; p.mmrCr = m + } else { + curve.method[0] = DoviMapping.polynomial.rawValue + curve.coef[0] = [1 << 22, 0, 0] + } + p.polyCb = curve; p.polyCr = curve + let out = try DoviGpuEdgeCaseTests.composeConstant(composer, bl: (512, 700, 300), el: (512, 512, 512), params: p) + for plane in DoviTestSupport.rawVDR(out)[1...] { #expect(Set(plane) == [2048]) } + } +} diff --git a/Tests/AetherEngineTests/DoviGpuCurveSweepTests.swift b/Tests/AetherEngineTests/DoviGpuCurveSweepTests.swift new file mode 100644 index 000000000..d142ab989 --- /dev/null +++ b/Tests/AetherEngineTests/DoviGpuCurveSweepTests.swift @@ -0,0 +1,393 @@ +import Testing +import Foundation +import CoreVideo +import Metal +@testable import AetherEngine + +/// Runs the real composition kernels over every base-layer code and compares the +/// result against ``DolbyVisionStillConverter/ReshapeCurve`` in Swift. +/// +/// The rest of the suite re-implements the shader's arithmetic in Swift and tests +/// the re-implementation, so it would pass unchanged if the Metal source diverged +/// from it. These tests execute `dovi_compose_luma` and `dovi_compose_chroma` and +/// read the output planes back, which is the only way the two can disagree and be +/// caught. They are built around the cases a single-piece test cannot see: +/// per-piece order, per-piece coefficient denominators, and per-piece MMR +/// coefficients. +struct DoviGpuCurveSweepTests { + + static let blWidth = 1024 + static let blHeight = 4 + + // MARK: - Buffer helpers + + static func makeBuffer(width: Int, height: Int) -> CVPixelBuffer? { + var buffer: CVPixelBuffer? + CVPixelBufferCreate( + kCFAllocatorDefault, width, height, + kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange, + [ + kCVPixelBufferMetalCompatibilityKey: true, + kCVPixelBufferIOSurfacePropertiesKey: [:] as CFDictionary + ] as CFDictionary, + &buffer + ) + return buffer + } + + /// Writes one 10-bit code per (plane, column, row) through `value`, honouring + /// the plane's own stride. P010 is MSB-aligned, hence the shift by 6. + static func fillPlane( + _ buffer: CVPixelBuffer, + plane: Int, + components: Int, + _ value: (_ column: Int, _ row: Int, _ component: Int) -> Int + ) { + CVPixelBufferLockBaseAddress(buffer, []) + defer { CVPixelBufferUnlockBaseAddress(buffer, []) } + guard let base = CVPixelBufferGetBaseAddressOfPlane(buffer, plane) else { return } + let bytesPerRow = CVPixelBufferGetBytesPerRowOfPlane(buffer, plane) + let width = CVPixelBufferGetWidthOfPlane(buffer, plane) + let height = CVPixelBufferGetHeightOfPlane(buffer, plane) + for row in 0.. [[Int]] { + CVPixelBufferLockBaseAddress(buffer, .readOnly) + defer { CVPixelBufferUnlockBaseAddress(buffer, .readOnly) } + guard let base = CVPixelBufferGetBaseAddressOfPlane(buffer, plane) else { return [] } + let bytesPerRow = CVPixelBufferGetBytesPerRowOfPlane(buffer, plane) + let width = CVPixelBufferGetWidthOfPlane(buffer, plane) + let rowPtr = (base + row * bytesPerRow).assumingMemoryBound(to: UInt16.self) + var out: [[Int]] = [] + out.reserveCapacity(width) + for column in 0..> 6)) + } + out.append(sample) + } + return out + } + + // MARK: - Reference output stage + + /// The quantisation the kernels apply after the curve: denormalise by + /// `bl_max * vdr_scale`, clamp to the VDR range, then round down to the output + /// plane and clamp to video range. Dither is off for these tests, so the + /// kernel's bias is a flat 0.5 and the result is exactly reproducible. + static func quantize(predNorm: Double, blMax: Int, vdrScale: Int, vdrMax: Int, lo: Int, hi: Int) -> Int { + let vdr = min(max(Int((predNorm * Double(blMax) * Double(vdrScale)).rounded(.toNearestOrEven)), 0), vdrMax) + let scaled = Double(vdr) / Double(vdrScale) + return min(max(Int(floor(scaled + 0.5)), lo), hi) + } + + /// Exact value of an RPU-style Q(shift) coefficient, so the reference and the + /// shader are compared on the same number rather than on the intended one. + static func fp(_ value: Double, shift: UInt32) -> (fixed: Int32, exact: Double) { + let scale = Double(1 << shift) + let fixed = Int32(value * scale) + return (fixed, Double(fixed) / scale) + } + + static func identityCurve() -> DoviPolynomialParams { + var poly = DoviPolynomialParams() + poly.numPivots = 2 + poly.pivots[0] = 0 + poly.pivots[1] = 1023 + poly.method[0] = DoviMapping.polynomial.rawValue + poly.order[0] = 1 + poly.coef[0] = [0, 1 << 23, 0] + poly.coefShift[0] = 23 + return poly + } + + // MARK: - Luma: multi-piece polynomial + + @Test("GPU luma sweep: a 3-piece polynomial with per-piece order and denominator matches the reference curve") + func lumaMultiPiecePolynomialSweep() throws { + guard let composer = DoviFelComposer() else { + Issue.record("Failed to initialize DoviFelComposer") + return + } + + let blMax = 1023 + let pivotCodes: [UInt32] = [0, 256, 700, 1023] + + // Piece 1 deliberately uses a different coefficient denominator and a + // quadratic term: a kernel that read piece 0's order or shift for every + // piece would pass a single-piece test and fail here. + let shifts: [UInt32] = [23, 20, 23] + let intended: [[Double]] = [ + [0.08, 0.90, 0.00], + [0.05, 0.70, 0.20], + [0.30, 0.55, 0.00] + ] + let orders: [UInt32] = [1, 2, 1] + + var poly = DoviPolynomialParams() + poly.numPivots = UInt32(pivotCodes.count) + for (i, pivot) in pivotCodes.enumerated() { poly.pivots[i] = pivot } + + var exact: [[Double]] = [] + for piece in 0..<3 { + poly.method[piece] = DoviMapping.polynomial.rawValue + poly.order[piece] = orders[piece] + poly.coefShift[piece] = shifts[piece] + var row: [Double] = [] + for j in 0..<3 { + let c = Self.fp(intended[piece][j], shift: shifts[piece]) + // Coefficients above the piece's order are not read by the kernel, + // so the reference must zero them rather than trust the table. + let live = UInt32(j) <= orders[piece] + poly.coef[piece][j] = live ? c.fixed : 0 + row.append(live ? c.exact : 0) + } + exact.append(row) + } + + var params = DoviMetalParams() + params.ditherMode = .none + params.flags = [.disableResidual] // this test is about the curve, not the residual + params.polyY = poly + params.polyCb = Self.identityCurve() + params.polyCr = Self.identityCurve() + + guard let bl = Self.makeBuffer(width: Self.blWidth, height: Self.blHeight), + let el = Self.makeBuffer(width: Self.blWidth / 2, height: Self.blHeight / 2) else { + Issue.record("Failed to allocate CVPixelBuffers") + return + } + // One base-layer code per column, so a single row of output covers the + // whole 10-bit input domain. + Self.fillPlane(bl, plane: 0, components: 1) { column, _, _ in column } + Self.fillPlane(bl, plane: 1, components: 2) { _, _, _ in 512 } + Self.fillPlane(el, plane: 0, components: 1) { _, _, _ in 512 } + Self.fillPlane(el, plane: 1, components: 2) { _, _, _ in 512 } + + guard let dst = composer.compose(baseBuffer: bl, enhancementBuffer: el, params: params, frameIndex: 0) else { + Issue.record("Composer returned nil") + return + } + + let reference = DolbyVisionStillConverter.ReshapeCurve( + pivots: pivotCodes.map { Double($0) / Double(blMax) }, + isMMR: [false, false, false], + poly: exact, + mmrOrder: [0, 0, 0], + mmrConst: [0, 0, 0], + mmrCoef: [[[]], [[]], [[]]] + ) + + let output = Self.readPlane(dst, plane: 0, components: 1, row: 0) + #expect(output.count == Self.blWidth) + + var worst = 0 + var worstCode = -1 + for code in 0.. worst { + worst = diff + worstCode = code + } + } + #expect(worst <= 1, + "GPU luma output diverges from the reference curve by \(worst) codes at BL \(worstCode)") + + // A curve that landed on a single piece everywhere, or collapsed to + // identity, would still satisfy a tolerance check, so pin two interior + // points that only the correct piece can produce. + // BL 100 sits in piece 0: 0.08 + 0.9 * 100/1023 = 0.1680 -> 172. + // BL 900 sits in piece 2: 0.30 + 0.55 * 900/1023 = 0.7839 -> 802. + #expect(abs(output[100][0] - 172) <= 1, "piece 0 not applied at BL 100: got \(output[100][0])") + #expect(abs(output[900][0] - 802) <= 1, "piece 2 not applied at BL 900: got \(output[900][0])") + } + + // MARK: - Chroma: per-piece MMR + + @Test("GPU chroma sweep: MMR coefficients are read per reshaping piece, not only from piece 0") + func chromaPerPieceMmrSweep() throws { + guard let composer = DoviFelComposer() else { + Issue.record("Failed to initialize DoviFelComposer") + return + } + + let blMax = 1023 + let shift: UInt32 = 23 + let pivotCodes: [UInt32] = [0, 512, 1023] + + // Two MMR pieces whose coefficients differ enough that reading piece 0 for + // the whole range is off by ~160 codes at the top of the sweep, well past + // any float tolerance. Piece 1 is order 2, so it also catches an + // implementation that reuses piece 0's order. + let constants = [Self.fp(0.02, shift: shift), Self.fp(-0.05, shift: shift)] + let piece0Order0 = [0.10, 0.80, 0.05, 0, 0, 0, 0] + let piece1Order0 = [0.05, 1.10, 0.02, 0, 0, 0, 0] + let piece1Order1 = [0.00, -0.35, 0.00, 0, 0, 0, 0] + + var poly = DoviPolynomialParams() + poly.numPivots = UInt32(pivotCodes.count) + for (i, pivot) in pivotCodes.enumerated() { poly.pivots[i] = pivot } + poly.method[0] = DoviMapping.mmr.rawValue + poly.method[1] = DoviMapping.mmr.rawValue + poly.order[0] = 1 + poly.order[1] = 2 + + var mmr = DoviMmrParams() + mmr.coefShift = shift + mmr.constant[0] = constants[0].fixed + mmr.constant[1] = constants[1].fixed + + var exactCoef: [[[Double]]] = [ + [Array(repeating: 0.0, count: 7), Array(repeating: 0.0, count: 7), Array(repeating: 0.0, count: 7)], + [Array(repeating: 0.0, count: 7), Array(repeating: 0.0, count: 7), Array(repeating: 0.0, count: 7)] + ] + for (k, value) in piece0Order0.enumerated() { + let c = Self.fp(value, shift: shift) + mmr.coef[0][k] = c.fixed + exactCoef[0][0][k] = c.exact + } + for (k, value) in piece1Order0.enumerated() { + let c = Self.fp(value, shift: shift) + mmr.coef[1][k] = c.fixed + exactCoef[1][0][k] = c.exact + } + for (k, value) in piece1Order1.enumerated() { + let c = Self.fp(value, shift: shift) + mmr.coef[1][7 + k] = c.fixed + exactCoef[1][1][k] = c.exact + } + + var params = DoviMetalParams() + params.ditherMode = .none + params.flags = [.disableResidual] + params.polyY = Self.identityCurve() + params.polyCb = poly + params.mmrCb = mmr + params.polyCr = Self.identityCurve() + + guard let bl = Self.makeBuffer(width: Self.blWidth, height: Self.blHeight), + let el = Self.makeBuffer(width: Self.blWidth / 2, height: Self.blHeight / 2) else { + Issue.record("Failed to allocate CVPixelBuffers") + return + } + // Constant luma so the cross terms are a known y, and two codes of Cb per + // chroma column so the 512-wide plane still sweeps the full input range. + let lumaCode = 512 + Self.fillPlane(bl, plane: 0, components: 1) { _, _, _ in lumaCode } + Self.fillPlane(bl, plane: 1, components: 2) { column, _, component in + component == 0 ? min(column * 2, 1023) : 512 + } + Self.fillPlane(el, plane: 0, components: 1) { _, _, _ in 512 } + Self.fillPlane(el, plane: 1, components: 2) { _, _, _ in 512 } + + guard let dst = composer.compose(baseBuffer: bl, enhancementBuffer: el, params: params, frameIndex: 0) else { + Issue.record("Composer returned nil") + return + } + + let reference = DolbyVisionStillConverter.ReshapeCurve( + pivots: pivotCodes.map { Double($0) / Double(blMax) }, + isMMR: [true, true], + poly: [[0, 0, 0], [0, 0, 0]], + mmrOrder: [1, 2], + mmrConst: [constants[0].exact, constants[1].exact], + mmrCoef: exactCoef + ) + + let output = Self.readPlane(dst, plane: 1, components: 2, row: 0) + #expect(output.count == Self.blWidth / 2) + + let y = Double(lumaCode) / Double(blMax) + let v = Double(512) / Double(blMax) + + var worst = 0 + var worstColumn = -1 + for column in 0.. worst { + worst = diff + worstColumn = column + } + // Cr is identity, so it also shows whether the two curves were mixed up. + #expect(abs(output[column][1] - 512) <= 1, + "Cr must pass through at 512, got \(output[column][1]) at column \(column)") + } + #expect(worst <= 2, + "GPU chroma output diverges from the reference MMR curve by \(worst) codes at column \(worstColumn)") + + // Explicit N4 guard: at the top of the range piece 1 must be in force. + // Piece 1 at u = 1022/1023 gives ~0.7372 -> ~754 codes; evaluating piece 0 + // there would give ~0.9207 -> ~942, so the two readings cannot be confused. + let top = output.count - 1 + #expect(output[top][0] < 800, + "the top of the sweep used piece 0's MMR coefficients: got \(output[top][0])") + } + + // MARK: - Identity is bit-exact + + @Test("GPU identity curve is bit-exact: every base-layer code survives the round trip") + func identityCurveIsBitExact() throws { + guard let composer = DoviFelComposer() else { + Issue.record("Failed to initialize DoviFelComposer") + return + } + + var params = DoviMetalParams() + params.ditherMode = .none + params.flags = [.disableResidual] + params.polyY = Self.identityCurve() + params.polyCb = Self.identityCurve() + params.polyCr = Self.identityCurve() + + guard let bl = Self.makeBuffer(width: Self.blWidth, height: Self.blHeight), + let el = Self.makeBuffer(width: Self.blWidth / 2, height: Self.blHeight / 2) else { + Issue.record("Failed to allocate CVPixelBuffers") + return + } + Self.fillPlane(bl, plane: 0, components: 1) { column, _, _ in column } + Self.fillPlane(bl, plane: 1, components: 2) { column, _, _ in min(column * 2, 1023) } + Self.fillPlane(el, plane: 0, components: 1) { _, _, _ in 512 } + Self.fillPlane(el, plane: 1, components: 2) { _, _, _ in 512 } + + guard let dst = composer.compose(baseBuffer: bl, enhancementBuffer: el, params: params, frameIndex: 0) else { + Issue.record("Composer returned nil") + return + } + + // Denormalising by bl_max * vdr_scale (1023 * 4) rather than vdr_max is + // what makes this exact; * 4095 would drift by up to 3 codes at the top. + let output = Self.readPlane(dst, plane: 0, components: 1, row: 0) + for code in 64...940 { + #expect(output[code][0] == code, + "identity curve changed BL code \(code) into \(output[code][0])") + } + // Below and above the video range the output saturates instead. + #expect(output[0][0] == 64) + #expect(output[1023][0] == 940) + } +} diff --git a/Tests/AetherEngineTests/DoviGpuEdgeCaseTests.swift b/Tests/AetherEngineTests/DoviGpuEdgeCaseTests.swift new file mode 100644 index 000000000..30bbc0334 --- /dev/null +++ b/Tests/AetherEngineTests/DoviGpuEdgeCaseTests.swift @@ -0,0 +1,492 @@ +import Testing +import Foundation +import CoreVideo +import Metal +@testable import AetherEngine + +/// Edge cases for the composition kernels, executed on the GPU with expected +/// values worked out by hand rather than by re-running the shader's arithmetic. +/// +/// Conventions used throughout: identity curves, denominator 23, 10-bit BL into +/// 12-bit VDR (vdr_scale 4), and an NLQ with `scale = 2048`, `shift = 11`, which is +/// what the real P7 streams carry, so that one EL code of residual is exactly one +/// VDR code. Dither is off so every expectation is exact. +struct DoviGpuEdgeCaseTests { + + typealias H = DoviGpuCurveSweepTests + + // MARK: - Fixtures + + static func nlq( + offset: Int32 = 512, + scale: Int32 = 2048, + threshold: Int32 = 0, + vdrInMax: Int32 = 512, + method: DoviNlqMethod = .linear + ) -> DoviNlqParams { + var n = DoviNlqParams() + n.nlqMethod = method + n.nlqOffset = offset + n.nlqScale = scale + n.nlqThreshold = threshold + n.vdrInMax = vdrInMax + n.nlqShift = 11 + return n + } + + static func params( + flags: DoviComposerFlags = [], + nlqY: DoviNlqParams = nlq(), + nlqCb: DoviNlqParams = nlq(), + nlqCr: DoviNlqParams = nlq() + ) -> DoviMetalParams { + var p = DoviMetalParams() + p.ditherMode = .none + p.flags = flags + p.polyY = H.identityCurve() + p.polyCb = H.identityCurve() + p.polyCr = H.identityCurve() + p.nlqY = nlqY + p.nlqCb = nlqCb + p.nlqCr = nlqCr + return p + } + + /// Constant-field compose: BL (y, cb, cr) at `blWidth x blHeight`, EL + /// (y, cb, cr) at half that, returning the output buffer. + static func composeConstant( + _ composer: DoviFelComposer, + blWidth: Int = 64, blHeight: Int = 16, + bl: (y: Int, cb: Int, cr: Int), + el: (y: Int, cb: Int, cr: Int), + params: DoviMetalParams + ) throws -> CVPixelBuffer { + let blBuf = try #require(H.makeBuffer(width: blWidth, height: blHeight)) + let elBuf = try #require(H.makeBuffer(width: blWidth / 2, height: blHeight / 2)) + H.fillPlane(blBuf, plane: 0, components: 1) { _, _, _ in bl.y } + H.fillPlane(blBuf, plane: 1, components: 2) { _, _, c in c == 0 ? bl.cb : bl.cr } + H.fillPlane(elBuf, plane: 0, components: 1) { _, _, _ in el.y } + H.fillPlane(elBuf, plane: 1, components: 2) { _, _, c in c == 0 ? el.cb : el.cr } + return try #require(composer.compose(baseBuffer: blBuf, enhancementBuffer: elBuf, params: params, frameIndex: 0)) + } + + /// Every sample of a plane. + static func allSamples(_ buffer: CVPixelBuffer, plane: Int, components: Int) -> [[Int]] { + let height = CVPixelBufferGetHeightOfPlane(buffer, plane) + return (0.. Set { + Set(allSamples(buffer, plane: 0, components: 1).map { $0[0] }) + } + + // MARK: - Residual (inverse NLQ) on the GPU + + /// (EL luma code, expected output luma code) for a BL of 512. + /// vdr = 512 * 4 + residual; out = floor(vdr / 4 + 0.5). The residual is reconstructed at the + /// middle of the quantisation step, (|r| - 1/2) * scale, truncated: |r| = 1 -> 0, 2 -> 1, 3 -> 2. + @Test("GPU residual: positive, negative and zero residual land on the hand-computed code", + arguments: [ + (612, 537), // +100 -> 99 -> 2147 / 4 = 536.75 -> 537 + (412, 487), // -100 -> -99 -> 1949 / 4 = 487.25 -> 487 + (512, 512), // el == offset: no residual, and not zeroed out + (513, 512), // +1 -> 0 -> 2048 / 4 = 512 + (514, 512), // +2 -> 1 -> 2049 / 4 = 512.25 -> 512 + (515, 513), // +3 -> 2 -> 2050 / 4 = 512.5 -> rounds up to 513 + (511, 512), // -1 -> 0 -> 512 + ]) + func residualSigns(elY: Int, expected: Int) throws { + let composer = try #require(DoviFelComposer()) + let out = try Self.composeConstant(composer, bl: (512, 512, 512), el: (elY, 512, 512), params: Self.params()) + #expect(Self.lumaValues(out) == [expected], "el=\(elY)") + } + + @Test("GPU residual: vdr_in_max clamps the magnitude symmetrically") + func residualClampedByVdrInMax() throws { + let composer = try #require(DoviFelComposer()) + let p = Self.params(nlqY: Self.nlq(vdrInMax: 256)) + // +511 -> clamped to +256 -> (2048 + 256) / 4 = 576 ; -512 -> -256 -> 448 + let hi = try Self.composeConstant(composer, bl: (512, 512, 512), el: (1023, 512, 512), params: p) + let lo = try Self.composeConstant(composer, bl: (512, 512, 512), el: (0, 512, 512), params: p) + #expect(Self.lumaValues(hi) == [576]) + #expect(Self.lumaValues(lo) == [448]) + } + + @Test("GPU residual: a non-zero deadzone threshold is added to the magnitude, as in the reference pseudo-code") + func residualThresholdAdded() throws { + // DoViBaker: dq = (2|r| - 1) * S + 2 * T, all >> (shift + 1). With S = 2048 (one code per EL code), shift 11 and + // T = 50 codes: |r| = 100 -> (99.5 + 50) = 149.5 -> 149; |r| = 30 -> 29.5 + 50 = 79.5 -> 79; negative mirrors. + let composer = try #require(DoviFelComposer()) + let p = Self.params(nlqY: Self.nlq(threshold: 50 * 2048)) + let big = try Self.composeConstant(composer, bl: (512, 512, 512), el: (612, 512, 512), params: p) + let small = try Self.composeConstant(composer, bl: (512, 512, 512), el: (542, 512, 512), params: p) + let neg = try Self.composeConstant(composer, bl: (512, 512, 512), el: (412, 512, 512), params: p) + #expect(Self.lumaValues(big) == [(2048 + 149) / 4]) // 549.25 -> 549 + #expect(Self.lumaValues(small) == [532]) // (2048 + 79) / 4 = 531.75, rounded + #expect(Self.lumaValues(neg) == [475]) // (2048 - 149) / 4 = 474.75, rounded + } + + @Test("GPU residual: zero scale, method none and disable_residual each yield the bare base layer") + func residualOff() throws { + let composer = try #require(DoviFelComposer()) + let cases: [(String, DoviMetalParams)] = [ + ("zero scale", Self.params(nlqY: Self.nlq(scale: 0))), + ("method none", Self.params(nlqY: Self.nlq(method: .none))), + ("disable flag", Self.params(flags: [.disableResidual])), + ("disabled helper", Self.params(nlqY: .disabled)), + ] + for (name, p) in cases { + let out = try Self.composeConstant(composer, bl: (512, 512, 512), el: (900, 100, 900), params: p) + #expect(Self.lumaValues(out) == [512], "\(name)") + } + } + + @Test("GPU residual: Cb and Cr are inverted independently and with their own parameters") + func chromaResidualIndependent() throws { + let composer = try #require(DoviFelComposer()) + // Cr gets half the slope of Cb, so the same +100 EL swing gives +100 and +50. + let p = Self.params(nlqCb: Self.nlq(scale: 2048), nlqCr: Self.nlq(scale: 1024)) + let out = try Self.composeConstant(composer, bl: (512, 400, 600), el: (512, 612, 612), params: p) + let uv = Self.allSamples(out, plane: 1, components: 2) + // Cb: (1600 + 99) / 4 = 424.75 -> 425 ; Cr: (2400 + 49) / 4 = 612.25 -> 612 + #expect(Set(uv.map { $0[0] }) == [425]) + #expect(Set(uv.map { $0[1] }) == [612]) + } + + @Test("GPU residual: only Cb active leaves Cr at its base value") + func chromaOneComponentActive() throws { + let composer = try #require(DoviFelComposer()) + let p = Self.params(nlqCr: Self.nlq(scale: 0)) + let out = try Self.composeConstant(composer, bl: (512, 400, 600), el: (512, 612, 612), params: p) + let uv = Self.allSamples(out, plane: 1, components: 2) + #expect(Set(uv.map { $0[0] }) == [425]) + #expect(Set(uv.map { $0[1] }) == [600]) + } + + // MARK: - Output range + + @Test("GPU clamps to video range: luma 64...940, chroma 64...960") + func outputClamps() throws { + let composer = try #require(DoviFelComposer()) + let p = Self.params(flags: [.disableResidual]) + let low = try Self.composeConstant(composer, bl: (0, 0, 0), el: (512, 512, 512), params: p) + let high = try Self.composeConstant(composer, bl: (1023, 1023, 1023), el: (512, 512, 512), params: p) + #expect(Self.lumaValues(low) == [64]) + #expect(Set(Self.allSamples(low, plane: 1, components: 2).flatMap { $0 }) == [64]) + #expect(Self.lumaValues(high) == [940]) + #expect(Set(Self.allSamples(high, plane: 1, components: 2).flatMap { $0 }) == [960]) + } + + @Test("GPU residual pushing past the VDR range saturates instead of wrapping") + func residualSaturates() throws { + let composer = try #require(DoviFelComposer()) + let p = Self.params() + // BL 1023 + 511 residual -> 4092 + 511 > 4095 -> clamp -> out 1024 -> luma cap 940. + let hi = try Self.composeConstant(composer, bl: (1023, 512, 512), el: (1023, 512, 512), params: p) + // BL 0 - 512 residual -> negative -> clamp 0 -> luma floor 64. + let lo = try Self.composeConstant(composer, bl: (0, 512, 512), el: (0, 512, 512), params: p) + #expect(Self.lumaValues(hi) == [940]) + #expect(Self.lumaValues(lo) == [64]) + } + + // MARK: - Geometry + + @Test("GPU writes every pixel for sizes that are not a multiple of the threadgroup or of 8", + arguments: [(4, 4), (6, 4), (34, 10), (130, 18), (514, 6), (1030, 10)]) + func oddGeometryCoversEveryPixel(width: Int, height: Int) throws { + let composer = try #require(DoviFelComposer()) + // Residual of +100 over the whole frame: an unwritten pixel would read 0, + // and a border pixel that mis-sampled the EL would miss 537. + let out = try Self.composeConstant( + composer, blWidth: width, blHeight: height, + bl: (512, 512, 512), el: (612, 612, 412), params: Self.params() + ) + #expect(Self.lumaValues(out) == [537], "luma \(width)x\(height)") + let uv = Self.allSamples(out, plane: 1, components: 2) + #expect(uv.count == (width / 2) * (height / 2)) + #expect(Set(uv.map { $0[0] }) == [537], "Cb \(width)x\(height)") + #expect(Set(uv.map { $0[1] }) == [487], "Cr \(width)x\(height)") + } + + @Test("a frame whose enhancement chroma plane would be empty is refused, not composed from garbage", + arguments: [(2, 2), (6, 2)]) + func tooSmallForChromaIsRefused(width: Int, height: Int) throws { + let composer = try #require(DoviFelComposer()) + let blBuf = try #require(H.makeBuffer(width: width, height: height)) + let elBuf = try #require(H.makeBuffer(width: width / 2, height: height / 2)) + #expect(composer.compose(baseBuffer: blBuf, enhancementBuffer: elBuf, params: Self.params()) == nil) + // and the failed attempt must not have leaked a parameter slot + let ok = try Self.composeConstant(composer, bl: (512, 512, 512), el: (612, 512, 512), params: Self.params()) + #expect(Self.lumaValues(ok) == [537]) + } + + @Test("GPU enhancement-layer upsampling preserves a constant field to the last row and column") + func upsamplingPartitionOfUnity() throws { + let composer = try #require(DoviFelComposer()) + // EL is 1/4 the BL area. A weight set that does not sum to one, or one that + // reads past the edge, shows up as a border that differs from the interior. + for (w, h) in [(64, 16), (66, 18), (130, 34)] { + let out = try Self.composeConstant( + composer, blWidth: w, blHeight: h, + bl: (512, 512, 512), el: (700, 512, 512), params: Self.params() + ) + // +188 is clamped to vdr_in_max 512 so is not clipped: (2048 + 188)/4 = 559. + #expect(Self.lumaValues(out) == [559], "\(w)x\(h)") + } + } + + // MARK: - Curve shape + + @Test("GPU zero-width piece is never selected; the later piece owns the shared pivot") + func zeroWidthPiece() throws { + let composer = try #require(DoviFelComposer()) + var poly = DoviPolynomialParams() + poly.numPivots = 4 + poly.pivots[0] = 0; poly.pivots[1] = 500; poly.pivots[2] = 500; poly.pivots[3] = 1023 + for piece in 0..<3 { + poly.method[piece] = DoviMapping.polynomial.rawValue + poly.order[piece] = 1 + poly.coefShift[piece] = 23 + } + poly.coef[0] = [0, 1 << 23, 0] // identity + poly.coef[1] = [Int32(0.9 * 8388608.0), 0, 0] // would output ~0.9 if ever chosen + poly.coef[2] = [0, 1 << 23, 0] // identity + var p = Self.params(flags: [.disableResidual]) + p.polyY = poly + + let bl = try #require(H.makeBuffer(width: 1024, height: 4)) + let el = try #require(H.makeBuffer(width: 512, height: 2)) + H.fillPlane(bl, plane: 0, components: 1) { c, _, _ in min(c, 1023) } + H.fillPlane(bl, plane: 1, components: 2) { _, _, _ in 512 } + H.fillPlane(el, plane: 0, components: 1) { _, _, _ in 512 } + H.fillPlane(el, plane: 1, components: 2) { _, _, _ in 512 } + let out = try #require(composer.compose(baseBuffer: bl, enhancementBuffer: el, params: p)) + let row = H.readPlane(out, plane: 0, components: 1, row: 0).map { $0[0] } + for code in [100, 499, 500, 501, 900] { + #expect(row[code] == min(max(code, 64), 940), "code \(code)") + } + } + + @Test("GPU luma MMR path reads the base-layer chroma, not just luma") + func lumaMmrUsesChroma() throws { + let composer = try #require(DoviFelComposer()) + var poly = DoviPolynomialParams() + poly.numPivots = 2 + poly.pivots[0] = 0; poly.pivots[1] = 1023 + poly.method[0] = DoviMapping.mmr.rawValue + poly.order[0] = 1 + poly.coefShift[0] = 23 + var mmr = DoviMmrParams() + mmr.coefShift = 23 + // Terms are [y, u, v, yu, yv, uv, yuv]; select u only, so luma out = BL Cb. + mmr.coef[0][1] = 1 << 23 + + var p = Self.params(flags: [.disableResidual, .lumaMmr]) + p.polyY = poly + p.mmrY = mmr + let out = try Self.composeConstant(composer, bl: (800, 300, 700), el: (512, 512, 512), params: p) + #expect(Self.lumaValues(out) == [300]) + } + + @Test("GPU luma MMR with a constant term and order 2 matches the hand-computed value") + func lumaMmrOrderTwo() throws { + let composer = try #require(DoviFelComposer()) + var poly = DoviPolynomialParams() + poly.numPivots = 2 + poly.pivots[0] = 0; poly.pivots[1] = 1023 + poly.method[0] = DoviMapping.mmr.rawValue + poly.order[0] = 2 + poly.coefShift[0] = 23 + var mmr = DoviMmrParams() + mmr.coefShift = 23 + mmr.constant[0] = Int32(0.1 * 8388608.0) + mmr.coef[0][0] = Int32(0.5 * 8388608.0) // order 1, y + mmr.coef[0][7] = Int32(0.2 * 8388608.0) // order 2, y^2 + var p = Self.params(flags: [.disableResidual, .lumaMmr]) + p.polyY = poly + p.mmrY = mmr + // y = 600/1023: 0.1 + 0.5 y + 0.2 y^2 + let y = 600.0 / 1023.0 + let pred = 0.1 + 0.5 * y + 0.2 * y * y + let expected = Int(floor(Double(Int((pred * 1023.0 * 4.0).rounded(.toNearestOrEven))) / 4.0 + 0.5)) + let out = try Self.composeConstant(composer, bl: (600, 512, 512), el: (512, 512, 512), params: p) + #expect(Self.lumaValues(out) == [expected]) + } + + // MARK: - Dither + + @Test("GPU dither keeps the mean level and only moves values by one code") + func ditherStaysWithinOneCode() throws { + let composer = try #require(DoviFelComposer()) + var p = Self.params() + p.ditherMode = .bayer8x8 + // Residual +3 (reconstructed as +2) on a 512 base: exactly 512.5, so dither splits 512/513. + let blBuf = try #require(H.makeBuffer(width: 64, height: 64)) + let elBuf = try #require(H.makeBuffer(width: 32, height: 32)) + H.fillPlane(blBuf, plane: 0, components: 1) { _, _, _ in 512 } + H.fillPlane(blBuf, plane: 1, components: 2) { _, _, _ in 512 } + H.fillPlane(elBuf, plane: 0, components: 1) { _, _, _ in 515 } + H.fillPlane(elBuf, plane: 1, components: 2) { _, _, _ in 512 } + for frame: UInt32 in [0, 1] { + let out = try #require(composer.compose(baseBuffer: blBuf, enhancementBuffer: elBuf, params: p, frameIndex: frame)) + let samples = Self.allSamples(out, plane: 0, components: 1).map { $0[0] } + #expect(samples.allSatisfy { $0 == 512 || $0 == 513 }) + let mean = Double(samples.reduce(0, +)) / Double(samples.count) + #expect(abs(mean - 512.5) < 0.01, "frame \(frame) mean \(mean)") + } + } + + // MARK: - Robustness of the composer itself + + @Test("compose refuses degenerate buffers instead of crashing") + func degenerateBuffers() throws { + let composer = try #require(DoviFelComposer()) + // A non-planar buffer cannot give the six textures the kernels need. + var bgra: CVPixelBuffer? + CVPixelBufferCreate(kCFAllocatorDefault, 64, 16, kCVPixelFormatType_32BGRA, + [kCVPixelBufferMetalCompatibilityKey: true] as CFDictionary, &bgra) + let good = try #require(H.makeBuffer(width: 64, height: 16)) + let half = try #require(H.makeBuffer(width: 32, height: 8)) + if let bgra { + #expect(composer.compose(baseBuffer: bgra, enhancementBuffer: half, params: Self.params()) == nil) + #expect(composer.compose(baseBuffer: good, enhancementBuffer: bgra, params: Self.params()) == nil) + } + } + + @Test("repeated composes at a changing resolution recycle the pool without leaking slots") + func resolutionChangeMidStream() throws { + let composer = try #require(DoviFelComposer()) + for (w, h) in [(64, 16), (128, 32), (64, 16), (256, 64), (64, 16)] { + for _ in 0..<8 { + let out = try Self.composeConstant( + composer, blWidth: w, blHeight: h, + bl: (512, 512, 512), el: (612, 512, 512), params: Self.params() + ) + #expect(CVPixelBufferGetWidth(out) == w) + #expect(CVPixelBufferGetHeight(out) == h) + } + } + // All in-flight slots must have been returned: a leak would deadlock here. + composer.acquireSlot() + composer.releaseSlot() + #expect(composer.takePeakSlotsInUse() >= 1) + } +} + +extension DoviGpuEdgeCaseTests { + private static func rawWord(_ b: CVPixelBuffer, plane: Int) -> Int { + CVPixelBufferLockBaseAddress(b, .readOnly); defer { CVPixelBufferUnlockBaseAddress(b, .readOnly) } + let stride = CVPixelBufferGetBytesPerRowOfPlane(b, plane) / 2 + return Int(CVPixelBufferGetBaseAddressOfPlane(b, plane)!.assumingMemoryBound(to: UInt16.self)[3 * stride + 3]) + } + + @Test("full-precision output keeps the two VDR bits the 10-bit code drops") + func fullPrecisionKeepsLowBits() throws { + let composer = try #require(DoviFelComposer()) + // BL 500 -> VDR 2000; +2 residual (reconstructed as +1) -> 2001 = 500.25 in 10-bit codes. + var p = Self.params(); p.ditherMode = .fullPrecision + let out = try Self.composeConstant(composer, bl: (500, 512, 512), el: (514, 512, 512), params: p) + #expect(Self.rawWord(out, plane: 0) == 2001 * 16) + #expect(Self.rawWord(out, plane: 0) >> 6 == 500, "the top 10 bits stay a valid P010 code") + var r = Self.params(); r.ditherMode = .none + let rounded = try Self.composeConstant(composer, bl: (500, 512, 512), el: (514, 512, 512), params: r) + #expect(Self.rawWord(rounded, plane: 0) == 500 << 6) + } +} + +// MARK: - DoViBaker input and prediction clamps + +extension DoviGpuEdgeCaseTests { + /// A one-piece MMR curve over `[lo, hi]` whose only non-zero term is `term` of `[y, u, v, yu, yv, uv, yuv]`. + private static func mmrPassThrough(term: Int, lo: UInt32 = 0, hi: UInt32 = 1023) -> (DoviPolynomialParams, DoviMmrParams) { + var poly = DoviPolynomialParams() + poly.numPivots = 2 + poly.pivots[0] = lo; poly.pivots[1] = hi + poly.method[0] = DoviMapping.mmr.rawValue + poly.order[0] = 1 + poly.coefShift[0] = 23 + var mmr = DoviMmrParams() + mmr.coefShift = 23 + mmr.coef[0][term] = 1 << 23 + return (poly, mmr) + } + + private static func rawParams() -> DoviMetalParams { + var p = params(flags: [.disableResidual]) + p.ditherMode = .debugRawVDR + return p + } + + /// DoViBaker `mmrMapping` clamps s0 to pivot_value[0], s1 to pivot_value[1], s2 to pivot_value[2] (each + /// component's own range) before forming the cross terms. With a pass-through MMR the output is the + /// clamped input, in 12-bit VDR codes (x4). + @Test("MMR inputs are clamped to their own component's pivot range: Cb to poly_cb, Cr to poly_cr, luma to poly_y") + func mmrInputsClampedToOwnComponentRange() throws { + let composer = try #require(DoviFelComposer()) + + // Cb term (u) with poly_cb range [64, 940]: BL Cb 1000 -> 940, BL Cb 30 -> 64. + var p = Self.rawParams() + (p.polyCb, p.mmrCb) = Self.mmrPassThrough(term: 1, lo: 64, hi: 940) + var out = try Self.composeConstant(composer, bl: (512, 1000, 512), el: (512, 512, 512), params: p) + #expect(Set(DoviTestSupport.rawVDR(out)[1]) == [940 * 4]) + out = try Self.composeConstant(composer, bl: (512, 30, 512), el: (512, 512, 512), params: p) + #expect(Set(DoviTestSupport.rawVDR(out)[1]) == [64 * 4]) + + // Cb output reading the v term: clamped by poly_cr's range [100, 900], not poly_cb's. + p = Self.rawParams() + (p.polyCb, p.mmrCb) = Self.mmrPassThrough(term: 2) + p.polyCr = { + var c = H.identityCurve(); c.pivots[0] = 100; c.pivots[1] = 900; return c + }() + out = try Self.composeConstant(composer, bl: (512, 512, 1000), el: (512, 512, 512), params: p) + #expect(Set(DoviTestSupport.rawVDR(out)[1]) == [900 * 4]) + out = try Self.composeConstant(composer, bl: (512, 512, 20), el: (512, 512, 512), params: p) + #expect(Set(DoviTestSupport.rawVDR(out)[1]) == [100 * 4]) + + // Cr output reading the y term: clamped by poly_y's range [64, 940]. + p = Self.rawParams() + (p.polyCr, p.mmrCr) = Self.mmrPassThrough(term: 0) + p.polyY = { + var c = H.identityCurve(); c.pivots[0] = 64; c.pivots[1] = 940; return c + }() + out = try Self.composeConstant(composer, bl: (1000, 512, 512), el: (512, 512, 512), params: p) + #expect(Set(DoviTestSupport.rawVDR(out)[2]) == [940 * 4]) + + // Luma MMR reading Cb: clamped by poly_cb's range. + p = Self.rawParams() + p.flags.insert(.lumaMmr) + (p.polyY, p.mmrY) = Self.mmrPassThrough(term: 1) + p.polyCb = { + var c = H.identityCurve(); c.pivots[0] = 200; c.pivots[1] = 800; return c + }() + out = try Self.composeConstant(composer, bl: (512, 1000, 512), el: (512, 512, 512), params: p) + #expect(Set(DoviTestSupport.rawVDR(out)[0]) == [800 * 4]) + } + + /// DoViBaker floors the polynomial and the MMR result at 0 (`vv = (vv < 0) ? 0 : vv`) before the residual + /// is added, so a negative prediction is not pulled back up by it: 0 + 99, not -410 + 99 -> clamp 0. + @Test("a negative prediction is floored at 0 before the residual is added") + func negativePredictionFlooredBeforeResidual() throws { + let composer = try #require(DoviFelComposer()) + var neg = DoviPolynomialParams() + neg.numPivots = 2; neg.pivots[0] = 0; neg.pivots[1] = 1023 + neg.method[0] = DoviMapping.polynomial.rawValue + neg.order[0] = 0; neg.coefShift[0] = 23 + neg.coef[0] = [Int32(-0.1 * 8388608.0), 0, 0] // -0.1 * 4096 = -410 VDR codes + + var p = Self.params() + p.ditherMode = .debugRawVDR + p.polyY = neg + let out = try Self.composeConstant(composer, bl: (512, 512, 512), el: (612, 512, 512), params: p) + #expect(Set(DoviTestSupport.rawVDR(out)[0]) == [99]) // +100 EL codes -> +99 residual + + // The same through MMR (chroma): constant -0.1, residual +99. + var m = DoviMmrParams(); m.coefShift = 23; m.constant[0] = Int32(-0.1 * 8388608.0) + var mp = DoviPolynomialParams() + mp.numPivots = 2; mp.pivots[1] = 1023 + mp.method[0] = DoviMapping.mmr.rawValue; mp.order[0] = 1; mp.coefShift[0] = 23 + p.polyCb = mp; p.mmrCb = m + let c = try Self.composeConstant(composer, bl: (512, 512, 512), el: (512, 612, 512), params: p) + #expect(Set(DoviTestSupport.rawVDR(c)[1]) == [99]) + } +} diff --git a/Tests/AetherEngineTests/DoviLayoutProbeTests.swift b/Tests/AetherEngineTests/DoviLayoutProbeTests.swift new file mode 100644 index 000000000..b60c8eeb0 --- /dev/null +++ b/Tests/AetherEngineTests/DoviLayoutProbeTests.swift @@ -0,0 +1,240 @@ +import Testing +import Foundation +import Metal +@testable import AetherEngine + +/// Checks the parameter ABI on the GPU rather than by hand. +/// +/// Metal has no `static_assert`, and `xcrun metal` is not installed on every +/// machine that runs this suite, so the shader's view of `DoviMetalParams` cannot +/// be asserted at compile time. Instead `dovi_layout_probe` reads the last field +/// of each sub-block back out of the buffer the host actually uploaded: any offset +/// drift between ``DoviMetalParams/serialize()`` and the shader's struct +/// definitions moves a sentinel and fails a comparison below. +struct DoviLayoutProbeTests { + + /// Distinct value per probed field, so a shifted read lands on a value that + /// belongs to a different field rather than on a plausible one. + static func sentinelParams() -> DoviMetalParams { + var p = DoviMetalParams() + + p.blWidth = 101 + p.blHeight = 102 + p.elWidth = 103 + p.elHeight = 104 + p.targetMode = 105 + p.ditherMode = .bayer8x8 + p.frameIndex = 107 + p.flags = [.disableResidual, .lumaMmr] + + p.polyY.numPivots = 9 + p.polyY.pivots[8] = 1023 + p.polyY.method[7] = DoviMapping.mmr.rawValue + p.polyY.order[7] = 2 + p.polyY.coef[7][2] = -12345 // negative, to prove the sign survives + p.polyY.coefShift[7] = 23 + p.mmrY.constant[7] = 777 + p.mmrY.coef[7][20] = 888 + p.mmrY.coefShift = 23 + + p.polyCb.numPivots = 5 + p.polyCb.coefShift[7] = 17 + p.mmrCb.coef[7][20] = 1234 + p.mmrCb.coefShift = 19 + + p.polyCr.numPivots = 3 + p.polyCr.coefShift[7] = 13 + p.mmrCr.coef[7][20] = -4321 + p.mmrCr.coefShift = 11 + + Self.applyComponentSentinels(&p) + + p.minPq = 1 + p.maxPq = 2 + p.avgPq = 3 + p.blMax = 1023 + p.vdrMax = 4095 + p.vdrScale = 4 + + return p + } + + + // MARK: - Per-component sentinels + + /// Fields probed per component, in slot order from `firstComponentSlot`; mirrors the shader's probe kernel. + static let componentFieldNames = [ + "poly.pivots[0]", "poly.pivots[1]", "poly.method[0]", "poly.order[0]", + "poly.coef[0][0]", "poly.coef[0][1]", "poly.coef[1][0]", "poly.coef_shift[0]", + "mmr.constant[0]", "mmr.coef[0][0]", "mmr.coef[0][1]", "mmr.coef[1][0]", + "nlq.method", "nlq.offset", "nlq.scale", "nlq.threshold", "nlq.shift", "nlq.vdr_in_max", + "nlq.pivots[0]", "nlq.pivots[1]", + ] + static let firstComponentSlot = 36 + + /// Distinct non-zero sentinel for component `c`, field `k`; fields that are signed alternate sign so a + /// sign-lost read is caught too. `nlq.method` is an enum with two legal values, so it uses those. + static func sentinel(_ c: Int, _ k: Int) -> UInt32 { + let magnitude = UInt32(1000 * (c + 1) + 10 * k + 1) + let signed = [4, 5, 8, 10, 15] // coef[0][0], coef[0][1], mmr.constant[0], mmr.coef[0][1], nlq.threshold + if signed.contains(k) { return UInt32(bitPattern: -Int32(magnitude)) } + if k == 12 { return [DoviNlqMethod.none, .linear, .none][c].rawValue } + return magnitude + } + + static func slot(_ c: Int, _ k: Int) -> Int { firstComponentSlot + c * componentFieldNames.count + k } + + /// Writes the per-component sentinels over `p`. + static func applyComponentSentinels(_ p: inout DoviMetalParams) { + let polys: [WritableKeyPath] = [\.polyY, \.polyCb, \.polyCr] + let mmrs: [WritableKeyPath] = [\.mmrY, \.mmrCb, \.mmrCr] + let nlqs: [WritableKeyPath] = [\.nlqY, \.nlqCb, \.nlqCr] + for c in 0..<3 { + func u(_ k: Int) -> UInt32 { sentinel(c, k) } + func i(_ k: Int) -> Int32 { Int32(bitPattern: sentinel(c, k)) } + p[keyPath: polys[c]].pivots[0] = u(0) + p[keyPath: polys[c]].pivots[1] = u(1) + p[keyPath: polys[c]].method[0] = u(2) + p[keyPath: polys[c]].order[0] = u(3) + p[keyPath: polys[c]].coef[0][0] = i(4) + p[keyPath: polys[c]].coef[0][1] = i(5) + p[keyPath: polys[c]].coef[1][0] = i(6) + p[keyPath: polys[c]].coefShift[0] = u(7) + p[keyPath: mmrs[c]].constant[0] = i(8) + p[keyPath: mmrs[c]].coef[0][0] = i(9) + p[keyPath: mmrs[c]].coef[0][1] = i(10) + p[keyPath: mmrs[c]].coef[1][0] = i(11) + p[keyPath: nlqs[c]].nlqMethod = DoviNlqMethod(rawValue: u(12))! + p[keyPath: nlqs[c]].nlqOffset = i(13) + p[keyPath: nlqs[c]].nlqScale = i(14) + p[keyPath: nlqs[c]].nlqThreshold = i(15) + p[keyPath: nlqs[c]].nlqShift = u(16) + p[keyPath: nlqs[c]].vdrInMax = i(17) + p[keyPath: nlqs[c]].nlqPivots = [u(18), u(19)] + } + } + + @Test("The shader reads every parameter sub-block at the offset the Swift serializer wrote it to") + func layoutProbeRoundTrip() throws { + let composer = try #require(DoviFelComposer()) + let params = Self.sentinelParams() + let slots = try #require(composer.probeLayout(params: params), + "dovi_layout_probe is unavailable, so the parameter ABI is unverified") + #expect(slots.count == DoviFelComposer.layoutProbeSlots) + + /// Compares one slot, naming the field so a failure says which offset moved. + func expect(_ slot: Int, _ value: UInt32, _ field: String) { + #expect(slots[slot] == value, + "slot \(slot) (\(field)) read \(slots[slot]), expected \(value)") + } + func expect(_ slot: Int, signed value: Int32, _ field: String) { + expect(slot, UInt32(bitPattern: value), field) + } + + // sizeof on the GPU side, against the size the serializer produced. + expect(0, UInt32(DoviMetalParams.serializedSize), "sizeof(DoviMetalParams)") + #expect(params.serialize().count == DoviMetalParams.serializedSize) + + expect(1, 101, "bl_width") + expect(2, 102, "bl_height") + expect(3, 103, "el_width") + expect(4, 104, "el_height") + expect(5, 105, "target_mode") + expect(6, DoviDither.bayer8x8.rawValue, "dither_mode") + expect(7, 107, "frame_index") + expect(8, DoviComposerFlags([.disableResidual, .lumaMmr]).rawValue, "flags") + + expect(9, 9, "poly_y.num_pivots") + expect(10, 1023, "poly_y.pivots[8]") + expect(11, DoviMapping.mmr.rawValue, "poly_y.method[7]") + expect(12, 2, "poly_y.order[7]") + expect(13, signed: -12345, "poly_y.coef[7][2]") + expect(14, 23, "poly_y.coef_shift[7]") + expect(15, 777, "mmr_y.mmr_constant[7]") + expect(16, 888, "mmr_y.coef[7][20]") + expect(17, 23, "mmr_y.coef_shift") + expect(18, Self.sentinel(0, 17), "nlq_y.vdr_in_max") + expect(19, Self.sentinel(0, 19), "nlq_y.nlq_pivots[1]") + + expect(20, 5, "poly_cb.num_pivots") + expect(21, 17, "poly_cb.coef_shift[7]") + expect(22, 1234, "mmr_cb.coef[7][20]") + expect(23, 19, "mmr_cb.coef_shift") + expect(24, Self.sentinel(1, 19), "nlq_cb.nlq_pivots[1]") + + expect(25, 3, "poly_cr.num_pivots") + expect(26, 13, "poly_cr.coef_shift[7]") + expect(27, signed: -4321, "mmr_cr.coef[7][20]") + expect(28, 11, "mmr_cr.coef_shift") + expect(29, Self.sentinel(2, 19), "nlq_cr.nlq_pivots[1]") + + expect(30, 1, "min_pq") + expect(31, 2, "max_pq") + expect(32, 3, "avg_pq") + expect(33, 1023, "bl_max") + expect(34, 4095, "vdr_max") + expect(35, 4, "vdr_scale") + + // First element of every array and every NLQ scalar, per component, each with its own sentinel. + let names = ["y", "cb", "cr"] + for c in 0..<3 { + for (k, field) in Self.componentFieldNames.enumerated() { + expect(Self.slot(c, k), Self.sentinel(c, k), "\(names[c]).\(field)") + } + } + } + + @Test("Bit depths derive the normalisation constants, and reject pairs the kernels cannot express") + func bitDepthDerivation() { + var p = DoviMetalParams() + + let profile7 = p.setBitDepths(bl: 10, vdr: 12) + #expect(profile7) + #expect(p.blMax == 1023) + #expect(p.vdrMax == 4095) + #expect(p.vdrScale == 4) + + let eightBit = p.setBitDepths(bl: 8, vdr: 8) + #expect(eightBit) + #expect(p.blMax == 255) + #expect(p.vdrMax == 255) + #expect(p.vdrScale == 1, "equal depths must not scale the residual") + + // A VDR depth below the base-layer depth has no meaning in the + // reconstruction, and would make vdr_scale a fraction. + let vdrBelowBl = p.setBitDepths(bl: 12, vdr: 10) + let vdrTooDeep = p.setBitDepths(bl: 10, vdr: 20) + let blTooShallow = p.setBitDepths(bl: 4, vdr: 12) + #expect(!vdrBelowBl) + #expect(!vdrTooDeep) + #expect(!blTooShallow) + #expect(p.blMax == 255, "a rejected pair must leave the previous values alone") + } + + @Test("A zeroed NLQ block is neutral, matching what a memset gives the C side") + func zeroedNlqIsNeutral() { + let zeroed = DoviNlqParams() + // nlq_method 0 is LINEAR, so neutrality has to come from the zero slope: + // the kernel returns no residual when nlq_scale is zero. That is what lets + // a memset-zeroed C struct and a default-initialised Swift value agree. + #expect(zeroed.nlqMethod == .linear) + #expect(zeroed.nlqScale == 0) + + let disabled = DoviNlqParams.disabled + #expect(disabled.nlqMethod == .none) + #expect(disabled.nlqScale == 0) + } + + @Test("serialize(into:) writes byte-for-byte identical output to serialize() -> Data") + func serializeIntoRawBufferMatchesSerializeData() { + let params = Self.sentinelParams() + let expectedData = params.serialize() + + var rawBytes = [UInt8](repeating: 0, count: DoviMetalParams.serializedSize) + rawBytes.withUnsafeMutableBytes { raw in + params.serialize(into: raw) + } + + #expect(Data(rawBytes) == expectedData) + } +} diff --git a/Tests/AetherEngineTests/DoviRpuConverterTests.swift b/Tests/AetherEngineTests/DoviRpuConverterTests.swift index 7e33a1789..da8ab3d33 100644 --- a/Tests/AetherEngineTests/DoviRpuConverterTests.swift +++ b/Tests/AetherEngineTests/DoviRpuConverterTests.swift @@ -7,6 +7,7 @@ import Dovi /// Successful real-RPU conversion and the FEL/MEL string value are validated against /// dovi_tool ground truth via `aetherctl dovitest` and on device; these guard the /// pure byte-walk branches (degrade-on-failure, EL drop, no-op) from regressing. +@Suite(.serialized) struct DoviRpuConverterTests { /// 2-byte HEVC NAL header (type in bits 1..6 of byte 0, layer 0, temporal_id_plus1 = 1) + payload. @@ -165,6 +166,15 @@ struct DoviRpuConverterTests { #expect(DoviRpuConverter.enhancementLayerType(pkt) == nil) } + @Test("enhancementLayerType does not hand libdovi an RPU the guard rejects (it would abort the process)") + func elTypeSkipsAnRPUThatWouldAbortLibdovi() { + // CRC-valid, but a linear-interpolation polynomial piece, which libdovi panics on. + let rpu = DoviRpuGuardTests.craft(linearOnComponent: 0) + let pkt = avccPacket([hevcNAL(type: 1, payload: [0xAA]), rpu]) + defer { free(pkt) } + #expect(DoviRpuConverter.enhancementLayerType(pkt) == nil) + } + // MARK: - #365: framing is given, not assumed /// Pack NALs Annex B, the framing a Matroska remux with Annex-B CodecPrivate delivers. @@ -436,4 +446,188 @@ struct DoviRpuConverterTests { #expect(DoviRpuConverter.convertPacketToProfile81(pkt) == true) #expect(packetBytes(pkt) == input) } + + // MARK: - Single parse per RPU + + /// Each RPU must be parsed exactly once. Before the fix, `separateDualLayers` + /// parsed via `DoviFelComposer.parseRpuParameters` (internally calling + /// `dovi_parse_unspec62_nalu`) and then again via a direct call, and + /// `DualLayerVideoDecoder` added a third. After the fix: one parse per RPU. + @Test("separateDualLayers parses each RPU exactly once") + func singleParsePerRPUPerformance() { + // Build a P7 packet from the real RPU fixture: BL + RPU + EL. + // clipARpuHex already includes the 2-byte HEVC NAL header (0x7C, 0x01 = type 62). + let bl = hevcNAL(type: 1, payload: [0xAA, 0xBB]) + let rpu = hexToBytes(DoviFelMathTests.clipARpuHex) + let el = hevcNAL(type: 63, payload: [0xCC]) + let pkt = avccPacket([bl, rpu, el]) + defer { free(pkt) } + + // Baseline: no prior parses. + DoviFelComposer.resetParseRpuCountForTesting() + #expect(DoviFelComposer.parseRpuCountForTesting == 0) + + let result = DoviRpuConverter.separateDualLayers(pkt) + + // Exactly one parse happened (via parseRpuParameters). + #expect(DoviFelComposer.parseRpuCountForTesting == 1, + "expected exactly 1 parse per RPU, got \(DoviFelComposer.parseRpuCountForTesting)") + #expect(result.rpuParams != nil) + #expect(result.convertedRpu != nil) + #expect(result.isFEL == true) + #expect(result.isMEL == false) + #expect(result.success == true) + } + + // MARK: - Several RPUs in one packet + + /// The raw bytes, the parameters and the converted RPU of a result must all come from one RPU. + /// Clip A and clip C differ in pivots and in size, so each output names its source. + @Test("with two valid RPUs in a packet, raw bytes, parameters and converted RPU all come from the last one") + func twoValidRpusComeFromOneRpu() throws { + let first = hexToBytes(DoviFelMathTests.clipARpuHex) + let second = hexToBytes(DoviFelMathTests.clipCRpuHex) + let pkt = avccPacket([hevcNAL(type: 1, payload: [0xAA]), first, second, hevcNAL(type: 63, payload: [0xCC])]) + defer { free(pkt) } + let result = DoviRpuConverter.separateDualLayers(pkt) + + #expect(result.success) + #expect(result.rpuData == second) + // Parameters of the second RPU: its luma curve is not the first's 9-pivot one. + let expected = try #require(Self.parameters(of: second)) + let got = try #require(result.rpuParams) + #expect(got.polyY.numPivots == expected.polyY.numPivots) + #expect(got.polyY.pivots == expected.polyY.pivots) + #expect(got.polyCb.coef == expected.polyCb.coef) + #expect(got.mmrCb.coef == expected.mmrCb.coef) + let other = try #require(Self.parameters(of: first)) + #expect(got.polyY.pivots != other.polyY.pivots || got.mmrCb.coef != other.mmrCb.coef, "fixture RPUs must differ") + #expect(result.convertedRpu == Self.converted(second)) + } + + @Test("a second RPU that fails the guard leaves the first RPU's bytes, parameters and conversion intact") + func failingSecondRpuKeepsFirstConsistent() throws { + let first = hexToBytes(DoviFelMathTests.clipARpuHex) + var bad = hexToBytes(DoviFelMathTests.clipCRpuHex) + bad[bad.count - 6] ^= 0xFF // inside the CRC-covered payload: the guard refuses it + let pkt = avccPacket([hevcNAL(type: 1, payload: [0xAA]), first, bad, hevcNAL(type: 63, payload: [0xCC])]) + defer { free(pkt) } + let result = DoviRpuConverter.separateDualLayers(pkt) + + #expect(!result.success, "the dropped RPU is still reported") + #expect(result.rpuData == first) + let expected = try #require(Self.parameters(of: first)) + let got = try #require(result.rpuParams) + #expect(got.polyY.pivots == expected.polyY.pivots) + #expect(got.mmrCb.coef == expected.mmrCb.coef) + #expect(result.convertedRpu == Self.converted(first)) + #expect(result.isFEL) + } + + private static func parameters(of nal: [UInt8]) -> DoviMetalParams? { + guard let rpu = dovi_parse_unspec62_nalu(nal, nal.count) else { return nil } + defer { dovi_rpu_free(rpu) } + return DoviFelComposer.parseRpuParameters(rpu: rpu) + } + + private static func converted(_ nal: [UInt8]) -> [UInt8]? { + guard let rpu = dovi_parse_unspec62_nalu(nal, nal.count) else { return nil } + defer { dovi_rpu_free(rpu) } + guard let out = DoviRpuConverter.convertedUnspec62(rpu) else { return nil } + defer { dovi_data_free(out) } + return [UInt8](UnsafeBufferPointer(start: out.pointee.data, count: out.pointee.len)) + } + + @Test("separateDualLayers skips parameter extraction when extractParams is false") + func separateDualLayersSkipsParameterExtractionWhenRequested() { + let bl = hevcNAL(type: 1, payload: [0xAA, 0xBB]) + let rpu = hexToBytes(DoviFelMathTests.clipARpuHex) + let el = hevcNAL(type: 63, payload: [0xCC]) + let pkt = avccPacket([bl, rpu, el]) + defer { free(pkt) } + + DoviFelComposer.resetParseRpuCountForTesting() + #expect(DoviFelComposer.parseRpuCountForTesting == 0) + + let result = DoviRpuConverter.separateDualLayers(pkt, extractParams: false) + + #expect(DoviFelComposer.parseRpuCountForTesting == 0, + "expected 0 parses when extractParams is false, got \(DoviFelComposer.parseRpuCountForTesting)") + #expect(result.rpuParams == nil) + #expect(result.convertedRpu != nil) + #expect(result.isFEL == true) + #expect(result.isMEL == false) + #expect(result.success == true) + } + + @Test("convertPacketToProfile81 skips parameter extraction") + func convertPacketToProfile81SkipsParameterExtraction() { + let bl = hevcNAL(type: 1, payload: [0xAA, 0xBB]) + let rpu = hexToBytes(DoviFelMathTests.clipARpuHex) + let el = hevcNAL(type: 63, payload: [0xCC]) + let pkt = avccPacket([bl, rpu, el]) + defer { free(pkt) } + + DoviFelComposer.resetParseRpuCountForTesting() + #expect(DoviFelComposer.parseRpuCountForTesting == 0) + + let success = DoviRpuConverter.convertPacketToProfile81(pkt) + + #expect(DoviFelComposer.parseRpuCountForTesting == 0, + "expected 0 parses in convertPacketToProfile81, got \(DoviFelComposer.parseRpuCountForTesting)") + #expect(success == true) + } + + // MARK: - Untouched Base Layer with CustomBlockSource + + @Test("separateDualLayers with useCustomBlockSource leaves packet untouched while extracting EL and RPU") + func separateDualLayersWithCustomBlockSourceLeavesPacketUntouched() { + let bl = hevcNAL(type: 1, payload: [0xAA, 0xBB, 0xCC, 0xDD]) + let rpu = hexToBytes(DoviFelMathTests.clipARpuHex) + let el = hevcNAL(type: 63, payload: [0xEE, 0xFF]) + let pkt = avccPacket([bl, rpu, el]) + defer { free(pkt) } + + let originalData = pkt.pointee.data + let originalSize = pkt.pointee.size + let originalBuf = pkt.pointee.buf + + let result = DoviRpuConverter.separateDualLayers( + pkt, + extractEL: true, + useCustomBlockSource: true, + extractParams: true + ) + defer { + var elOpt: UnsafeMutablePointer? = result.elPacket + trackedPacketFree(&elOpt) + } + + // Base-layer packet pointer, size, and AVBufferRef must be COMPLETELY UNTOUCHED + #expect(pkt.pointee.data == originalData) + #expect(pkt.pointee.size == originalSize) + #expect(pkt.pointee.buf == originalBuf) + + // EL packet, RPU params, and converted RPU must still be correctly extracted + #expect(result.elPacket != nil) + #expect(result.rpuParams != nil) + #expect(result.convertedRpu != nil) + #expect(result.isFEL == true) + #expect(result.isMEL == false) + #expect(result.success == true) + } + + /// Hex-string decoder for RPU fixtures. + private func hexToBytes(_ hex: String) -> [UInt8] { + var bytes: [UInt8] = [] + var i = hex.startIndex + while i < hex.endIndex { + let next = hex.index(i, offsetBy: 2) + if let byte = UInt8(hex[i.. Bool { SampleBufferRenderer.mayEnqueueAnotherFrame( diff --git a/Tests/AetherEngineTests/Support/DoviTestSupport.swift b/Tests/AetherEngineTests/Support/DoviTestSupport.swift new file mode 100644 index 000000000..a854947b1 --- /dev/null +++ b/Tests/AetherEngineTests/Support/DoviTestSupport.swift @@ -0,0 +1,31 @@ +import Foundation +import CoreVideo + +/// Helpers shared by the Dolby Vision composer tests. +enum DoviTestSupport { + + /// `media/` next to the package checkout, where local test titles live (git-ignored; tests that + /// need one skip when it is absent). + static var mediaDir: URL { + URL(fileURLWithPath: #filePath) + .deletingLastPathComponent().deletingLastPathComponent() + .deletingLastPathComponent().deletingLastPathComponent() + .deletingLastPathComponent() + .appendingPathComponent("media") + } + + /// Raw planes from a `.debugRawVDR` compose: 12-bit VDR codes, Cb and Cr de-interleaved. + static func rawVDR(_ b: CVPixelBuffer, chromaOnly: Bool = false) -> [[Int]] { + CVPixelBufferLockBaseAddress(b, .readOnly); defer { CVPixelBufferUnlockBaseAddress(b, .readOnly) } + var out = [[Int]]() + let w = CVPixelBufferGetWidthOfPlane(b, 0), h = CVPixelBufferGetHeightOfPlane(b, 0) + let ys = CVPixelBufferGetBytesPerRowOfPlane(b, 0) / 2 + let yb = CVPixelBufferGetBaseAddressOfPlane(b, 0)!.assumingMemoryBound(to: UInt16.self) + out.append(chromaOnly ? [] : (0..<(w * h)).map { Int(yb[($0 / w) * ys + $0 % w] >> 4) }) + let cw = CVPixelBufferGetWidthOfPlane(b, 1), ch = CVPixelBufferGetHeightOfPlane(b, 1) + let cs = CVPixelBufferGetBytesPerRowOfPlane(b, 1) / 2 + let cb = CVPixelBufferGetBaseAddressOfPlane(b, 1)!.assumingMemoryBound(to: UInt16.self) + for c in 0..<2 { out.append((0..<(cw * ch)).map { Int(cb[($0 / cw) * cs + ($0 % cw) * 2 + c] >> 4) }) } + return out + } +} diff --git a/Tests/AetherEngineTests/UndocumentedDolbyVisionKeysTests.swift b/Tests/AetherEngineTests/UndocumentedDolbyVisionKeysTests.swift new file mode 100644 index 000000000..b125c2455 --- /dev/null +++ b/Tests/AetherEngineTests/UndocumentedDolbyVisionKeysTests.swift @@ -0,0 +1,33 @@ +import XCTest +@testable import AetherEngine + +/// The FEL route hands tvOS its per-frame Dolby Vision metadata under two attachment keys that are in +/// no public header. If a tvOS release stops honouring them there must be exactly one place to change, +/// so the literals may be spelled in `DoviDisplayMetadata.swift` and nowhere else in `Sources/`; +/// everything else goes through `DoviDisplayMetadata.rpuDataKey` / `.compatibilityIDKey`. +final class UndocumentedDolbyVisionKeysTests: XCTestCase { + + func testTheUndocumentedKeysAreSpelledInOneFileOnly() throws { + let sources = URL(fileURLWithPath: #filePath) + .deletingLastPathComponent() + .deletingLastPathComponent() + .deletingLastPathComponent() + .appendingPathComponent("Sources") + let owner = "DoviDisplayMetadata.swift" + let literals = ["\"DolbyVisionRPUData\"", "\"DolbyCompatibilityID\""] + let files = try XCTUnwrap(FileManager.default.enumerator(at: sources, includingPropertiesForKeys: nil)) + var offenders: [String] = [] + var ownerSpellsBoth = false + for case let url as URL in files where url.pathExtension == "swift" { + let text = try String(contentsOf: url, encoding: .utf8) + let hits = literals.filter { text.contains($0) } + if url.lastPathComponent == owner { + ownerSpellsBoth = hits.count == literals.count + } else if !hits.isEmpty { + offenders.append("\(url.lastPathComponent): \(hits.joined(separator: ", "))") + } + } + XCTAssertTrue(ownerSpellsBoth, "\(owner) should define both keys") + XCTAssertEqual(offenders, [], "use DoviDisplayMetadata.rpuDataKey / .compatibilityIDKey instead") + } +} diff --git a/docs/architecture.md b/docs/architecture.md index 51cadd49a..3746316d2 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -540,8 +540,13 @@ Sources/AetherEngine/ │ ├── HLSVideoEngine+LiveReopen.swift Native path: live source-loss recovery (capped-backoff reopen on the same timeline); VOD backpressure-wedge re-anchor + consumer re-engage nudge, which re-reads the rendered position at nudge time so the zero-tolerance seek never lands behind the on-screen frame (#115) │ ├── CodecRoutePolicy.swift Native path: DV / HDR / codec routing decisions (track types, CODECS strings, VIDEO-RANGE) │ ├── AnnexBSampleConverter.swift Native path: Annex-B HEVC samples rewritten to length-prefixed NALs by the session muxer, in-band VPS/SPS/PPS kept. movenc's own conversion under `hvc1` drops them, so a stream that sends a new PPS mid-title stopped decoding at the change (PR #703) -│ ├── DoviRpuConverter.swift Native path: per-packet DV Profile 7 → 8.1 RPU conversion via libdovi (NAL surgery: convert type-62 RPU, drop type-63 EL) +│ ├── DoviRpuConverter.swift libdovi RPU surgery for both routes. Native path: per-packet Profile 7 → 8.1 rewrite (convert type-62 RPU, drop type-63 EL). SW FEL path: `separateDualLayers` also lifts the EL into its own packet and returns the composer's reshaping parameters plus the 8.1 RPU attached as display metadata │ ├── DoviRpuConverter+Probe.swift Diagnostic DV-conversion probe (`doviConvertProbe` / `DoviConvertProbeResult`), backs `aetherctl dovitest` +│ ├── DoviDisplayMetadata.swift SW FEL path: sets `DolbyVisionRPUData` (the Profile 8.1 RPU minus its NAL header) and `DolbyCompatibilityID` (the CFString "1", never a number) on every emitted buffer, or clears both; tvOS consumes them in backboardd +│ ├── Metal/ +│ │ ├── DoviFelComposer.swift BL + EL + RPU composition: runtime-compiled pipelines, pooled P010 output, `composeAsync` bounded at `maxFramesInFlight` with completions on its own serial queue; RPU → `DoviMetalParams` parsing +│ │ ├── DoviShaderSource.swift Luma / chroma compose kernels (reshaping, inverse NLQ, bilinear EL upsampling, ordered dither) and the ABI layout probe, compiled at runtime +│ │ └── DoviMetalParams.swift Swift mirror of the kernel parameter structs and their byte-exact serializer │ ├── Issue65LivelockBreakers.swift Pure backpressure-wedge detection (`BackpressureWedgeDetector`) breaking the VOD HLS scrub-burst livelock (#65); `seekIsWedged` starvation check + `SeekResumeGuard` single-resume latch for the deadline-bounded seek │ ├── Issue99MuxerFailureRevive.swift Bounded revive for a VOD pump that died with `muxerFailed` (#99): the first cut firing before any bridged audio reached the muxer (dec3 box) no longer ends the session │ ├── SlowServeSignal.swift One-shot slow-serve timer arming the server's early chunked header (keeps TTFB under AVPlayer's ~3.5 s -12889 window, #93 round 3) @@ -588,7 +593,7 @@ The `aetherctl` CLI target (`Sources/aetherctl/`) is documented separately in [d | Package | License | Purpose | | --- | --- | --- | | [FFmpegBuild](https://github.com/superuser404notfound/FFmpegBuild) | LGPL-2.1-or-later | Slim FFmpeg 8.1 (avcodec / avformat / avutil / swresample / swscale / avfilter + zimg) for demux + HLS-fMP4 mux + AudioBridge FLAC encode + SW-path dav1d decode + sws_scale YUV → NV12 / P010. avfilter ships a trimmed filter set: zscale + tonemap + colorspace (HDR → SDR still extraction), bwdif + yadif + yadif_videotoolbox + hwupload (SW + GPU deinterlacing). Bundles libzvbi for the DVB teletext decoder (#107). Shipped as dynamically linked frameworks, no GPL components | -| [LibDovi](https://github.com/superuser404notfound/LibDovi) | MIT / Apache-2.0 | libdovi (the `dolby_vision` crate's C API) for live Dolby Vision Profile 7 to single-layer 8.1 RPU conversion (`dovi_convert_rpu_with_mode`, mode 2), so the Apple TV engages real DV on dual-layer UHD-BD remuxes instead of plain HDR10. Prebuilt xcframework, no Rust at the consumer's build time | +| [LibDovi](https://github.com/superuser404notfound/LibDovi) | MIT / Apache-2.0 | libdovi (the `dolby_vision` crate's C API) for live Dolby Vision Profile 7 to single-layer 8.1 RPU conversion (`dovi_convert_rpu_with_mode`, mode 2), so the Apple TV engages real DV on dual-layer UHD-BD remuxes instead of plain HDR10. The software FEL path uses the same library to read the reshaping and NLQ parameters for the Metal composer and to produce the per-frame 8.1 RPU it attaches as display metadata. Prebuilt xcframework, no Rust at the consumer's build time | | [SMBClient](https://github.com/kishikawakatsumi/SMBClient) | MIT | Pure-Swift SMB2 client over `NWConnection`, backing the opt-in `AetherEngineSMB` product only (never linked by the core engine). NTLMv2 / guest, SMB 2.0.2 / 2.1. Replaced AMSMB2/libsmb2, which `EPERM`s on tvOS / iOS | | VideoToolbox | System | Native path video decode (HW where available, Apple's bundled SW dav1d on iOS / macOS) | | AVFoundation | System | AVPlayer + AVDisplayManager (native path); AVSampleBufferDisplayLayer + AVSampleBufferRenderSynchronizer (SW path) | From 36c28ec2a97916c8d88d520d21bbdc59390c1664 Mon Sep 17 00:00:00 2001 From: Pavel Kajaba <15980400+pkajaba@users.noreply.github.com> Date: Wed, 7 Oct 2026 10:31:18 +0100 Subject: [PATCH 009/132] feat(dovi): picture-order pairing and delivery building blocks for the dual-layer decoder MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four self-contained pieces the dual-layer Profile 7 decoder is built from, none of them wired to anything yet: - HEVCPictureOrderCount: the H.265 picture order count (§8.3.1) of one layer, read from the parameter sets and each picture's first slice segment header. No decoding. Anything it cannot follow (unknown PPS/SPS, truncated header, pictures before the first IRAP) answers nil rather than a guess. - DualLayerPictureAlignment: derives each layer's display slot from its POC and gives each enhancement-layer packet the presentation time of the base-layer picture in the same slot, so a stream whose enhancement layer has B-frames over a base layer without them pairs the right pictures. Falls back to access-unit pairing when a layer's POC is unavailable or the slots disagree by more than 16. - FrameSideDataRing: fixed-capacity (32) ring of per-frame side data keyed by PTS, looked up within +-300 ticks. - OrderedDelivery: hands composed frames across the VideoToolbox / Metal queue boundary in decode order. Also exposes A53SEIParser.unescapeRBSP and parseHVCCParameterSets (DoviRpuConverter+Probe) at module scope, which the POC reader and its test use. No behaviour change: nothing calls these types. Not in this PR: the decoder that uses them, the routing, the docs. Tests: DualLayerPictureAlignmentTests, FrameSideDataRingTests, OrderedDeliveryTests. --- .../AetherEngine/Decoder/A53SEIParser.swift | 2 +- .../Decoder/DualLayerPictureAlignment.swift | 173 ++++++++++ .../Decoder/FrameSideDataRing.swift | 164 +++++++++ .../Decoder/OrderedDelivery.swift | 326 ++++++++++++++++++ .../Video/DoviRpuConverter+Probe.swift | 2 +- .../Video/HEVCPictureOrderCount.swift | 155 +++++++++ ...lLayerPictureAlignmentSyntheticTests.swift | 170 +++++++++ .../DualLayerPictureAlignmentTests.swift | 137 ++++++++ .../FrameSideDataRingTests.swift | 152 ++++++++ .../HEVCPictureOrderCountTests.swift | 321 +++++++++++++++++ .../HEVCSyntheticBitstream.swift | 127 +++++++ .../OrderedDeliveryTests.swift | 247 +++++++++++++ 12 files changed, 1974 insertions(+), 2 deletions(-) create mode 100644 Sources/AetherEngine/Decoder/DualLayerPictureAlignment.swift create mode 100644 Sources/AetherEngine/Decoder/FrameSideDataRing.swift create mode 100644 Sources/AetherEngine/Decoder/OrderedDelivery.swift create mode 100644 Sources/AetherEngine/Video/HEVCPictureOrderCount.swift create mode 100644 Tests/AetherEngineTests/DualLayerPictureAlignmentSyntheticTests.swift create mode 100644 Tests/AetherEngineTests/DualLayerPictureAlignmentTests.swift create mode 100644 Tests/AetherEngineTests/FrameSideDataRingTests.swift create mode 100644 Tests/AetherEngineTests/HEVCPictureOrderCountTests.swift create mode 100644 Tests/AetherEngineTests/HEVCSyntheticBitstream.swift create mode 100644 Tests/AetherEngineTests/OrderedDeliveryTests.swift diff --git a/Sources/AetherEngine/Decoder/A53SEIParser.swift b/Sources/AetherEngine/Decoder/A53SEIParser.swift index 2b34f6371..7c2d47933 100644 --- a/Sources/AetherEngine/Decoder/A53SEIParser.swift +++ b/Sources/AetherEngine/Decoder/A53SEIParser.swift @@ -116,7 +116,7 @@ enum A53SEIParser { /// Strip emulation-prevention bytes (00 00 03 -> 00 00). In a valid stream, a 0x03 after two /// zeros is always an EPB, so the unconditional drop matches FFmpeg's h2645 unescape. - private static func unescapeRBSP(_ p: UnsafePointer, _ n: Int) -> [UInt8] { + static func unescapeRBSP(_ p: UnsafePointer, _ n: Int) -> [UInt8] { var out = [UInt8]() out.reserveCapacity(n) var zeros = 0 diff --git a/Sources/AetherEngine/Decoder/DualLayerPictureAlignment.swift b/Sources/AetherEngine/Decoder/DualLayerPictureAlignment.swift new file mode 100644 index 000000000..80a27660e --- /dev/null +++ b/Sources/AetherEngine/Decoder/DualLayerPictureAlignment.swift @@ -0,0 +1,173 @@ +import Foundation + +/// Gives each enhancement-layer packet the presentation time of the base-layer picture it belongs to. +/// +/// A profile 7 access unit carries one picture of each layer, and the container stamps the unit with +/// the base layer's presentation time. The two pictures of one unit need not be the same frame: the +/// enhancement layer may have its own GOP (B-frames where the base layer has none, IDRs at other cuts), +/// so its picture in a given unit can be a frame or two earlier or later. A spec decoder composes each +/// layer's pictures in that layer's display order; pairing by access unit instead puts a neighbouring +/// frame's residual onto moving detail. +/// +/// Each layer's display order is followed causally as a display slot counted in access units from the +/// start (or the last seek): a picture that opens a sequence (IDR, BLA, a CRA after a seek) is shown +/// after every picture decoded before it, so its slot is the number of units before it; later pictures +/// of that sequence sit at slot + (POC - its POC). Streams whose layers share a GOP (every UHD Blu-ray +/// title seen so far) get identical slots in both layers, and their packets keep their unit's time. +/// +/// Two limits, both of which end in the access-unit pairing a stream without this type would get: +/// - A sequence opener takes the slot "units before it", which assumes the previous sequence's slots +/// all lie below that count. If the previous sequence's POC numbering ran ahead of its unit count +/// (gaps in its POCs), the opener and its first followers share slots with its last pictures, and +/// the time kept for a slot is the newer one, so a restamp into that range can land on the wrong +/// picture. It needs a mid-stream sequence change with such gaps; a seek resets all slots. +/// - A layer whose slots drift further than `maxSlotDistance` from the unit position stops being +/// placed until its next IDR or BLA (the only mid-stream pictures that open a sequence; a CRA +/// opens one only at the start or after a seek). A stream that cuts GOPs with CRAs (open GOP) +/// therefore stays on access-unit pairing from that point until the next seek. +struct DualLayerPictureAlignment { + + /// A slot further than this from the unit's own position is further than any DPB allows, so the + /// layer's sequence is not trusted (dolby-vision-fel.md §5.3). + static let maxSlotDistance = 16 + + /// How many slots earlier than its own unit an enhancement picture may be restamped. A base-layer + /// frame waits for its partner only until K newer base frames have arrived (`DualLayerVideoDecoder. + /// maxLayerSkewFrames`), so an enhancement picture restamped to a frame further back than that + /// finds its partner already bypassed and is dropped: the pair is lost without anyone noticing. + /// Beyond this distance the unit is treated like one whose slot is unknown (access-unit pairing, + /// counted in `unabsorbableRestamps`). The owner sets it from its own skew bound; the default is + /// the DPB bound, which absorbs anything the slots can say. + var maxBackwardRestamp = DualLayerPictureAlignment.maxSlotDistance + /// Units whose enhancement picture sat further back than `maxBackwardRestamp`. + private(set) var unabsorbableRestamps = 0 + + /// One layer's display slots. + private struct Layer { + var counter = HEVCPictureOrderCount() + /// slot = POC + offset in the current sequence; nil until a sequence opens, and for the rest of + /// a sequence whose slots cannot be trusted. + var offset: Int? + + mutating func slot(_ p: HEVCPictureOrderCount.Picture, unit: Int) -> Int? { + if p.startsSequence { + offset = unit - p.poc + return unit + } + guard let offset else { return nil } + // A leading picture of the opener (RADL, POC below the IRAP's) is shown before the IRAP + // but decoded after it. That does not break the numbering: POC still steps one per + // picture, so it keeps the sequence's offset and lands before the opener's slot. + // (RASL pictures of a sequence-opening CRA never get here: `HEVCPictureOrderCount` does + // not report them, since a decoder starting at the CRA does not output them.) + let s = p.poc + offset + // Further from decode order than any DPB allows: POC is not stepping one per frame. + if abs(s - unit) > DualLayerPictureAlignment.maxSlotDistance { self.offset = nil; return nil } + return s + } + } + + private var base = Layer() + private var enhancement = Layer() + /// Access units since the start or the last seek. + private var units = 0 + /// Presentation time of recent base-layer pictures, by slot. + private var baseTime: [Int: Int64] = [:] + /// First base-layer slot seen since the reset: with the latest it gives the time per slot. + private var origin: (slot: Int, pts: Int64)? + private var latest: (slot: Int, pts: Int64)? + + /// Units whose enhancement-layer picture was moved to another frame's time. + private(set) var reorderedUnits = 0 + /// Whether the last unit's enhancement-layer picture was placed by its display slot (as opposed + /// to keeping its unit's time for want of one). + private(set) var lastEnhancementPlaced = false + + mutating func seedBaseParameterSets(_ nals: [[UInt8]]) { + base.counter.feedParameterSets(nals) + } + + /// After a seek both layers restart at their next IRAP. + mutating func reset() { + base.counter.reset(); base.offset = nil + enhancement.counter.reset(); enhancement.offset = nil + units = 0 + baseTime.removeAll() + origin = nil + latest = nil + } + + /// Reads both layers' pictures from one access unit (base-layer NALs, and enhancement-layer NALs + /// wrapped in unspec63) and returns the time to stamp on its enhancement-layer packet. `pts` is the + /// unit's presentation time, in the stream's time base. It comes back unchanged when both pictures + /// are the same frame, and whenever the enhancement picture's slot is not known. + mutating func enhancementPTS(_ data: UnsafePointer, _ size: Int, framing: VideoNALFraming, pts: Int64) -> Int64 { + var bl: HEVCPictureOrderCount.Picture? + var el: HEVCPictureOrderCount.Picture? + A53SEIParser.forEachNAL(data, size, framing) { nal, len in + let type = (nal[0] >> 1) & 0x3F + if type == 63 { + if len > 4, let p = enhancement.counter.feed(nal + 2, len - 2), el == nil { el = p } + } else if type < 48 { + if let p = base.counter.feed(nal, len), bl == nil { bl = p } + } + } + let unit = units + units += 1 + lastEnhancementPlaced = false + + let blSlot = bl.flatMap { base.slot($0, unit: unit) } + var elSlot = el.flatMap { enhancement.slot($0, unit: unit) } + if let slot = elSlot, (blSlot ?? unit) - slot > maxBackwardRestamp { + unabsorbableRestamps += 1 + elSlot = nil + } + guard pts != Int64.min else { return pts } + if let blSlot { + baseTime[blSlot] = pts + if baseTime.count > 64 { + for s in baseTime.keys where s < unit - 32 { baseTime.removeValue(forKey: s) } + } + if origin == nil { origin = (blSlot, pts) } + latest = (blSlot, pts) + } + + guard let elSlot else { return pts } + lastEnhancementPlaced = true + if elSlot == blSlot { return pts } + if let known = baseTime[elSlot] { + reorderedUnits += 1 + return known + } + // That base-layer picture is still to come: extrapolate from the time per slot. + guard let origin, let latest, latest.slot != origin.slot, + let result = Self.extrapolate(origin: origin, latest: latest, slot: elSlot) else { + lastEnhancementPlaced = false; return pts + } + reorderedUnits += 1 + return result + } + + /// Time of a slot beyond the latest one, from the time per slot between two known ones. Timestamps + /// come from the stream, so every step is overflow-checked: a crafted value makes this decline + /// (nil), never trap. + static func extrapolate(origin: (slot: Int, pts: Int64), latest: (slot: Int, pts: Int64), slot: Int) -> Int64? { + guard let span = checkedSub(latest.pts, origin.pts), + let slotSpan = checkedSub(latest.slot, origin.slot), slotSpan != 0, + let ahead = checkedSub(slot, latest.slot) else { return nil } + let perSlot = Double(span) / Double(slotSpan) + guard perSlot > 0, perSlot.isFinite else { return nil } + let offset = (Double(ahead) * perSlot).rounded() + // 2^63 is exactly representable; anything at or beyond it, or NaN, cannot convert. + guard offset.isFinite, offset > -9.223372036854775807e18, offset < 9.223372036854775807e18 else { return nil } + return checkedAdd(latest.pts, Int64(offset)) + } + + private static func checkedSub(_ a: T, _ b: T) -> T? { + let (v, o) = a.subtractingReportingOverflow(b); return o ? nil : v + } + + private static func checkedAdd(_ a: T, _ b: T) -> T? { + let (v, o) = a.addingReportingOverflow(b); return o ? nil : v + } +} diff --git a/Sources/AetherEngine/Decoder/FrameSideDataRing.swift b/Sources/AetherEngine/Decoder/FrameSideDataRing.swift new file mode 100644 index 000000000..51310d5e1 --- /dev/null +++ b/Sources/AetherEngine/Decoder/FrameSideDataRing.swift @@ -0,0 +1,164 @@ +import Foundation + +/// Fixed-capacity ring buffer for Dolby Vision per-frame side data (dolby-vision-fel.md §5.3). +/// +/// Stores `{ params, dmNAL }` keyed by canonical PTS. +/// Fixed capacity of 32 entries (reorder depth plus maximum layer skew with margin). +/// Insert overwrites the oldest entry when full. +/// Lookup scans up to 32 entries for the nearest key within ±300 ticks (~3.3 ms). +/// No heap reallocation on insert, no sorting, no dictionary overhead. +struct FrameSideDataRing: Sendable { + static let capacity = 32 + static let jitterToleranceTicks: Int64 = 300 + + /// Distance between two canonical keys, or nil when either is the missing-PTS key (`Int64.min`) or the + /// distance does not fit. A frame without a timestamp has no neighbour, so nothing matches it; and + /// the trapping `abs` of a plain subtraction overflows on exactly that key. + static func distance(_ a: Int64, _ b: Int64) -> UInt64? { + guard a != Int64.min, b != Int64.min else { return nil } + let (d, overflow) = a.subtractingReportingOverflow(b) + return overflow ? nil : d.magnitude + } + + /// True when both keys carry a timestamp and are within the pairing jitter of each other. + static func isWithinJitter(_ a: Int64, _ b: Int64) -> Bool { + guard let d = distance(a, b) else { return false } + return d <= UInt64(jitterToleranceTicks) + } + + struct Entry: Sendable { + var key: Int64 + var params: DoviMetalParams? + var dmNAL: [UInt8]? + + init(key: Int64, params: DoviMetalParams? = nil, dmNAL: [UInt8]? = nil) { + self.key = key + self.params = params + self.dmNAL = dmNAL + } + } + + private var buffer: [Entry?] + private var writeIndex: Int = 0 + private var entryCount: Int = 0 + + init() { + self.buffer = Array(repeating: nil, count: Self.capacity) + } + + var count: Int { entryCount } + var isEmpty: Bool { entryCount == 0 } + + var paramsCount: Int { + var result = 0 + for i in 0.. Int? { + var bestIndex: Int? + var minDiff = UInt64.max + + for i in 0.. Entry? { + guard let idx = findNearestIndex(key: key) else { return nil } + return buffer[idx] + } + + /// Removes and returns `dmNAL` for `key` (or nearest within ±300 ticks). + mutating func removeDisplayMetadata(key: Int64) -> [UInt8]? { + var bestIndex: Int? + var minDiff = UInt64.max + + for i in 0.. DoviMetalParams? { + var bestIndex: Int? + var minDiff = UInt64.max + + for i in 0..= nextToEmit else { + os_signpost(.end, log: Self.signpostLog, name: "Delivery", signpostID: signpostID, "dropped") + return + } + pending[ticket] = PendingEntry(frame: frame, signpostID: signpostID) + updatePendingCounts() + drainReady() + } + } + + /// Bumps the epoch and fast-forwards the head of the line to `nextTicket`, + /// dropping anything still pending behind it. Enqueues the work + /// (`queue.async`) rather than waiting for it, so this is safe to call + /// while holding a lock — see the type doc's "resetAsync/barrier" + /// section and ``barrier()``. + /// + /// The caller is responsible for the ordering guarantee this depends on: + /// nothing that could produce a ticket for `newEpoch` may be able to run + /// until *after* the call to this method has returned (in + /// `DualLayerVideoDecoder`, that is what holding `lock` across this call + /// provides — pairing cannot issue a new-epoch ticket until the same + /// critical section that calls this has unlocked). Given that, FIFO + /// ordering on `queue` guarantees this reset always runs before any + /// `fulfil` for the epoch it installs, however soon that `fulfil` is + /// called after this method returns. + func resetAsync(epoch newEpoch: UInt64, nextTicket: UInt64) { + queue.async { [self] in + for entry in pending.values { + os_signpost(.end, log: Self.signpostLog, name: "Delivery", signpostID: entry.signpostID, "reset") + } + epoch = newEpoch + nextToEmit = nextTicket + pending.removeAll() + stallLogPending = false + pendingCountLock.lock() + pendingCount = 0 + peakPendingCount = 0 + pendingCountLock.unlock() + } + } + + /// Blocks until everything enqueued on `queue` ahead of this call — + /// every `fulfil` and `resetAsync` already running or waiting, in + /// particular the one a caller just enqueued — has finished running. + /// + /// This is `reset`'s wait, split out so `resetAsync` can be enqueued + /// under a lock and this can be called after releasing it: waiting here + /// while still holding `DualLayerVideoDecoder.lock` would be exactly the + /// blocking-wait-under-a-lock shape I8 forbids. The precondition catches + /// the one way to defeat the wait itself: calling this from inside + /// `onFrame`, which would deadlock the queue against its own + /// synchronous wait. + func barrier() { + dispatchPrecondition(condition: .notOnQueue(queue)) + queue.sync {} + } + + /// `resetAsync` followed immediately by `barrier()`: bumps the epoch and + /// fast-forwards the head of the line to `nextTicket`, and does not + /// return until that has actually happened — nothing after this call + /// returns can reach `onFrame` under the old epoch (invariant I3, + /// dolby-vision-fel.md §5.8). For callers with no lock to hold across the two halves (tests, + /// mainly); `DualLayerVideoDecoder.flush()`/`close()` call `resetAsync` + /// and `barrier()` separately so the wait happens outside `lock`. + func reset(epoch newEpoch: UInt64, nextTicket: UInt64) { + resetAsync(epoch: newEpoch, nextTicket: nextTicket) + barrier() + } + + /// Emits every ticket that has become the head of the line, in order, + /// applying the stall guard when the head is missing and a bound number + /// of frames have backed up behind it. Runs on `queue`. + private func drainReady() { + defer { updatePendingCounts() } + while true { + if let entry = pending.removeValue(forKey: nextToEmit) { + nextToEmit &+= 1 + stallLogPending = false + emit(entry.frame) + os_signpost(.end, log: Self.signpostLog, name: "Delivery", signpostID: entry.signpostID) + continue + } + guard pending.count > stallThreshold else { break } + // The ticket at the head is not coming: skip it so the frames + // already backed up behind it are not stalled forever + // (dolby-vision-fel.md §5.7). This costs exactly the one frame at `nextToEmit`. + if !stallLogPending { + EngineLog.emit( + "[OrderedDelivery] stall guard: ticket \(nextToEmit) never arrived with \(pending.count) frames pending; skipping", + category: .swPlayback) + stallLogPending = true + } + nextToEmit &+= 1 + } + } + + /// Attaches the Dolby Vision display metadata and hands the frame to the + /// host (see `dolby-vision-fel.md §4.1`). Runs on `queue`, holding no lock of this + /// type's. `onFrame` may take the decoder's lock for a moment; that cannot deadlock because + /// no caller of `fulfil`/`resetAsync` holds a lock while waiting on `queue` (only `barrier()` + /// and `reset()` wait, and they are called with no decoder lock held). + private func emit(_ frame: DeliveredFrame) { + #if DEBUG + dispatchPrecondition(condition: .onQueue(queue)) + #endif + DoviDisplayMetadata.attach(rpuNAL: frame.dmNAL, to: frame.buffer.value) + onFrame?(frame.buffer.value, frame.pts, frame.hdr10Data) + } + + private func updatePendingCounts() { + pendingCountLock.lock() + let count = pending.count + pendingCount = count + if count > peakPendingCount { + peakPendingCount = count + } + pendingCountLock.unlock() + } + + /// Peak count of pending frames waiting in the delivery queue since the last read. + /// Resets the peak to the current pending count for the next interval (dolby-vision-fel.md §5.11). + func takePeakPendingCount() -> Int { + pendingCountLock.lock() + defer { pendingCountLock.unlock() } + let peak = peakPendingCount + peakPendingCount = pendingCount + return peak + } + + /// Current count of pending frames waiting in the delivery queue without blocking `queue`. + var currentPendingCount: Int { + pendingCountLock.lock() + defer { pendingCountLock.unlock() } + return pendingCount + } + + // MARK: - Test seams + + /// `queue.sync` reads for `DualLayerVideoDecoder.stateSnapshotForTesting` + /// and its `onFrame` getter (never on a hot path — nothing in production + /// reads the handler back, only assigns it). + /// + /// Every accessor below is + /// `queue.sync`, same as `reset`/`barrier`. Reading any of them — in + /// particular `currentOnFrame`, which backs `DualLayerVideoDecoder. + /// onFrame`'s getter — from inside a closure already running on `queue` + /// (i.e. from inside `emit`, or a host `onFrame` implementation that + /// reads the decoder's own `onFrame` back) deadlocks, the same + /// `dispatchPrecondition(.notOnQueue)` violation `reset`/`barrier` guard + /// against explicitly. Nothing in this codebase does that today — the + /// getter exists for the host to read what it itself assigned, not for + /// `onFrame`'s body to call back into — so there is no precondition + /// here, only this note. + var pendingCountForTesting: Int { queue.sync { pending.count } } + var nextToEmitForTesting: UInt64 { queue.sync { nextToEmit } } + var hasOnFrameForTesting: Bool { queue.sync { onFrame != nil } } + var currentOnFrame: DecodedFrameHandler? { queue.sync { onFrame } } +} diff --git a/Sources/AetherEngine/Video/DoviRpuConverter+Probe.swift b/Sources/AetherEngine/Video/DoviRpuConverter+Probe.swift index c724255af..89c8eb5ae 100644 --- a/Sources/AetherEngine/Video/DoviRpuConverter+Probe.swift +++ b/Sources/AetherEngine/Video/DoviRpuConverter+Probe.swift @@ -4,7 +4,7 @@ import AetherLibavcodec import AetherLibavutil /// Extract VPS/SPS/PPS NALs from hvcC extradata (22-byte header + numOfArrays arrays). Returns raw NAL bytes without length prefix or start code. -private func parseHVCCParameterSets(_ ed: UnsafePointer, _ size: Int) -> [[UInt8]] { +func parseHVCCParameterSets(_ ed: UnsafePointer, _ size: Int) -> [[UInt8]] { guard size > 23 else { return [] } var out: [[UInt8]] = [] let numArrays = Int(ed[22]) diff --git a/Sources/AetherEngine/Video/HEVCPictureOrderCount.swift b/Sources/AetherEngine/Video/HEVCPictureOrderCount.swift new file mode 100644 index 000000000..49dfa5ff1 --- /dev/null +++ b/Sources/AetherEngine/Video/HEVCPictureOrderCount.swift @@ -0,0 +1,155 @@ +import Foundation + +/// H.265 picture order count (ITU-T H.265 §8.3.1) for one layer, read from the parameter sets and +/// the first slice segment header of each picture. Nothing is decoded: the POC is what puts a +/// picture in display order, and the two layers of a Dolby Vision profile 7 stream are paired by it. +/// +/// Feed every NAL of the layer in decode order. Anything it cannot follow (a slice whose PPS or SPS +/// it has not seen, a truncated header, pictures before the first IRAP) answers nil rather than a guess. +struct HEVCPictureOrderCount { + + struct Picture: Equatable { + let poc: Int + /// An IRAP with NoRaslOutputFlag = 1 (IDR, BLA, or a CRA that opens the bitstream, follows an + /// end of sequence, or follows a seek): POC numbering restarts here. + let startsSequence: Bool + } + + private struct SPS { let pocLsbBits: Int; let separateColourPlane: Bool } + private struct PPS { let sps: Int; let outputFlagPresent: Bool; let extraSliceHeaderBits: Int } + + private var sps: [Int: SPS] = [:] + private var pps: [Int: PPS] = [:] + private var prevTid0POC = 0 + /// True until an IRAP has opened a sequence; set again by `reset()` and by an end of sequence. + private var awaitingIRAP = true + /// True while the most recent IRAP had NoRaslOutputFlag = 1 and can have RASL pictures: those + /// reference pictures before the IRAP that a decoder starting here never had, so they are not + /// output (§8.1.3) and get no POC here. A CRA that follows a picture in the same sequence has + /// NoRaslOutputFlag = 0, and its RASL pictures are ordinary output pictures. + private var skippingRASL = false + + /// After a seek: the next IRAP is handled as the first picture of the bitstream (HandleCraAsBlaFlag) + /// and pictures before it get no POC. Parameter sets are kept. + mutating func reset() { + awaitingIRAP = true + } + + /// Feeds one NAL (2-byte header included, emulation prevention bytes intact). Returns the POC when + /// the NAL is the first slice segment of a picture. + mutating func feed(_ nal: UnsafePointer, _ len: Int) -> Picture? { + guard len > 2 else { return nil } + let type = Int((nal[0] >> 1) & 0x3F) + switch type { + case 33: parseSPS(nal, len); return nil + case 34: parsePPS(nal, len); return nil + case 36, 37: awaitingIRAP = true; return nil // end of sequence / end of bitstream + case 0...9, 16...21: return parseSlice(type: type, temporalId: Int(nal[1] & 7) - 1, nal, len) + default: return nil + } + } + + /// Parameter sets carried outside the stream (the hvcC arrays). + mutating func feedParameterSets(_ nals: [[UInt8]]) { + for nal in nals { + nal.withUnsafeBufferPointer { b in + if let base = b.baseAddress { _ = feed(base, b.count) } + } + } + } + + // MARK: - Parsing + + private static func reader(_ nal: UnsafePointer, _ len: Int, limit: Int) -> H264SPS.BitReader { + // The fields read here sit well inside the first bytes; cap the unescape at `limit`. + H264SPS.BitReader(A53SEIParser.unescapeRBSP(nal + 2, min(len, limit) - 2)) + } + + private mutating func parseSPS(_ nal: UnsafePointer, _ len: Int) { + var r = Self.reader(nal, len, limit: 256) + guard r.u(4) != nil, let subLayers = r.u(3), r.u(1) != nil, + r.skip(88), r.skip(8) else { return } // general profile_tier_level, level_idc + var present: [(profile: Bool, level: Bool)] = [] + for _ in 0.. 0 { guard r.skip(2 * (8 - subLayers)) else { return } } + for p in present { + guard r.skip(p.profile ? 88 : 0), r.skip(p.level ? 8 : 0) else { return } + } + guard let id = r.ue(), let chroma = r.ue() else { return } + var separate = false + if chroma == 3 { guard let f = r.flag() else { return }; separate = f } + guard r.ue() != nil, r.ue() != nil, let window = r.flag() else { return } // width, height + if window { for _ in 0..<4 { guard r.ue() != nil else { return } } } + guard r.ue() != nil, r.ue() != nil, let lsb = r.ue(), lsb <= 12 else { return } // bit depths + sps[id] = SPS(pocLsbBits: lsb + 4, separateColourPlane: separate) + } + + private mutating func parsePPS(_ nal: UnsafePointer, _ len: Int) { + var r = Self.reader(nal, len, limit: 32) + guard let id = r.ue(), let spsID = r.ue(), r.flag() != nil, // dependent_slice_segments_enabled + let outputFlag = r.flag(), let extra = r.u(3) else { return } + pps[id] = PPS(sps: spsID, outputFlagPresent: outputFlag, extraSliceHeaderBits: extra) + } + + private mutating func parseSlice(type: Int, temporalId: Int, _ nal: UnsafePointer, _ len: Int) -> Picture? { + let irap = (16...23).contains(type) + var r = Self.reader(nal, len, limit: 64) + guard r.flag() == true else { return nil } // first_slice_segment_in_pic_flag + if irap { guard r.flag() != nil else { return nil } } // no_output_of_prior_pics_flag + guard let ppsID = r.ue(), let p = pps[ppsID], let s = sps[p.sps] else { return nil } + guard r.skip(p.extraSliceHeaderBits), r.ue() != nil else { return nil } // slice_reserved_flag, slice_type + if p.outputFlagPresent { guard r.flag() != nil else { return nil } } + if s.separateColourPlane { guard r.skip(2) else { return nil } } + + let idr = type == 19 || type == 20 + var lsb = 0 + if !idr { guard let v = r.u(s.pocLsbBits) else { return nil }; lsb = v } + + if awaitingIRAP && !irap { return nil } // not decodable before an IRAP + let startsSequence = idr || (16...18).contains(type) || (irap && awaitingIRAP) + // NoRaslOutputFlag of an IRAP is 1 for IDR, BLA and a CRA that starts the sequence; an IDR + // has no RASL pictures at all. RASL_N / RASL_R (8, 9) of such an IRAP are skipped. + if irap { skippingRASL = startsSequence && !idr } + if skippingRASL && (type == 8 || type == 9) { return nil } + let msb: Int + if startsSequence { + msb = 0 + awaitingIRAP = false + } else { + let maxLsb = 1 << s.pocLsbBits + let prevLsb = prevTid0POC & (maxLsb - 1) + let prevMsb = prevTid0POC - prevLsb + if lsb < prevLsb && prevLsb - lsb >= maxLsb / 2 { + msb = prevMsb + maxLsb + } else if lsb > prevLsb && lsb - prevLsb > maxLsb / 2 { + msb = prevMsb - maxLsb + } else { + msb = prevMsb + } + } + let poc = msb + lsb + // prevTid0Pic: TemporalId 0 and not a RASL, RADL or sub-layer non-reference picture. + let leading = (6...9).contains(type) + let subLayerNonReference = type <= 14 && type % 2 == 0 + if temporalId == 0 && !leading && !subLayerNonReference { + prevTid0POC = poc + } + return Picture(poc: poc, startsSequence: startsSequence) + } +} + +extension H264SPS.BitReader { + /// Skips `n` bits; false when that runs past the end. + mutating func skip(_ n: Int) -> Bool { + var left = n + while left > 0 { + let step = min(left, 32) + guard u(step) != nil else { return false } + left -= step + } + return true + } +} diff --git a/Tests/AetherEngineTests/DualLayerPictureAlignmentSyntheticTests.swift b/Tests/AetherEngineTests/DualLayerPictureAlignmentSyntheticTests.swift new file mode 100644 index 000000000..5fdb46267 --- /dev/null +++ b/Tests/AetherEngineTests/DualLayerPictureAlignmentSyntheticTests.swift @@ -0,0 +1,170 @@ +import Testing +import Foundation +@testable import AetherEngine + +/// `DualLayerPictureAlignment` on hand-built access units, so the leading-picture rules run without +/// the real clips `DualLayerPictureAlignmentTests` needs. Each access unit holds one base-layer +/// slice and one enhancement-layer slice (wrapped in NAL type 63, as in a profile 7 packet); the +/// enhancement layer's picture order is chosen to differ from the base layer's, so a restamp shows +/// as a changed time. +struct DualLayerPictureAlignmentSyntheticTests { + + typealias B = HEVCSyntheticBitstream + typealias T = HEVCSyntheticBitstream.T + + /// One access unit: base slice, then the enhancement layer's slice (and, when `parameterSets`, + /// its SPS/PPS, which have no other way in). + private static func unit(bl: [UInt8], el: [UInt8], parameterSets: Bool = false) -> [UInt8] { + var nals = [bl] + if parameterSets { + nals.append(B.wrappedEL(B.sps(pocLsbBits: 8))) + nals.append(B.wrappedEL(B.pps())) + } + nals.append(B.wrappedEL(el)) + return B.accessUnit(nals) + } + + private static func slice(_ type: Int, poc: Int) -> [UInt8] { + B.slice(type: type, pocLsb: poc & 0xFF, pocLsbBits: 8) + } + + private struct Result: Equatable { let pts: Int64; let placed: Bool } + + private static func run(_ a: inout DualLayerPictureAlignment, _ data: [UInt8], pts: Int64) -> Result { + let out = data.withUnsafeBufferPointer { + a.enhancementPTS($0.baseAddress!, $0.count, framing: .lengthPrefixed(size: 4), pts: pts) + } + return Result(pts: out, placed: a.lastEnhancementPlaced) + } + + private static func seeded() -> DualLayerPictureAlignment { + var a = DualLayerPictureAlignment() + a.seedBaseParameterSets([B.sps(pocLsbBits: 8), B.pps()]) + return a + } + + // MARK: - After a seek onto a CRA + + /// x265's default open GOP: a seek lands on a CRA followed by RASL pictures that reference what + /// came before it. A decoder starting there does not output them. They used to knock the layer's + /// offset out ("a picture below the opener's POC"), and with it the alignment, for the rest of the + /// session. + @Test("a seek onto a CRA with RASL pictures keeps the alignment: the RASL pictures are skipped, the rest still restamp") + func craWithRaslAfterSeek() { + var a = Self.seeded() + a.reset() + // Display order: RASL 6, RASL 7, CRA 8, 9, 10. PTS follows display order, one frame = 1000. + let u0 = Self.run(&a, Self.unit(bl: Self.slice(T.cra, poc: 8), el: Self.slice(T.cra, poc: 8), parameterSets: true), pts: 2000) + #expect(u0 == Result(pts: 2000, placed: true)) + + // The RASL pictures are not output: no slot for either layer, and the unit keeps its own time + let u1 = Self.run(&a, Self.unit(bl: Self.slice(T.raslN, poc: 6), el: Self.slice(T.raslN, poc: 6)), pts: 0) + let u2 = Self.run(&a, Self.unit(bl: Self.slice(T.raslR, poc: 7), el: Self.slice(T.raslR, poc: 7)), pts: 1000) + #expect(u1 == Result(pts: 0, placed: false)) + #expect(u2 == Result(pts: 1000, placed: false)) + + // The enhancement layer codes picture 10 before 9 (B-frames); the base layer does not. + // Slot of POC p is p - 8 for both layers. EL picture 10 pairs with the base picture of POC 10, + // which has not arrived yet: its time is extrapolated from the time per slot (1000). + let u3 = Self.run(&a, Self.unit(bl: Self.slice(T.trailR, poc: 9), el: Self.slice(T.trailR, poc: 10)), pts: 3000) + #expect(u3 == Result(pts: 4000, placed: true), "alignment must survive the RASL pictures") + // EL picture 9 pairs with the base picture of POC 9, which came out one unit earlier + let u4 = Self.run(&a, Self.unit(bl: Self.slice(T.trailR, poc: 10), el: Self.slice(T.trailR, poc: 9)), pts: 4000) + #expect(u4 == Result(pts: 3000, placed: true)) + #expect(a.reorderedUnits == 2) + } + + @Test("a CRA in the middle of a stream keeps its RASL pictures in the numbering") + func craWithRaslMidStream() { + var a = Self.seeded() + _ = Self.run(&a, Self.unit(bl: Self.slice(T.idrNLP, poc: 0), el: Self.slice(T.idrNLP, poc: 0), parameterSets: true), pts: 0) + for i in 1...4 { + _ = Self.run(&a, Self.unit(bl: Self.slice(T.trailR, poc: i), el: Self.slice(T.trailR, poc: i)), pts: Int64(i) * 1000) + } + // CRA at POC 7 with RASL 5 and 6 (decoded after it, shown before it): all output pictures. + // Slots are POC (offset 0). The EL shows 6 where the BL has 5 and vice versa. + _ = Self.run(&a, Self.unit(bl: Self.slice(T.cra, poc: 7), el: Self.slice(T.cra, poc: 7)), pts: 7000) + let r5 = Self.run(&a, Self.unit(bl: Self.slice(T.raslR, poc: 5), el: Self.slice(T.raslR, poc: 6)), pts: 5000) + #expect(r5.placed) + // slot 6 is on the unit's own display side of the CRA: the base layer's POC 6 comes next + let r6 = Self.run(&a, Self.unit(bl: Self.slice(T.raslR, poc: 6), el: Self.slice(T.raslR, poc: 5)), pts: 6000) + #expect(r6 == Result(pts: 5000, placed: true)) + } + + // MARK: - IDR with RADL + + @Test("RADL pictures, whose POC is below their IDR's, keep the layer's offset instead of discarding it") + func idrWithRadl() { + var a = Self.seeded() + a.reset() + // Display order: RADL -2, RADL -1, IDR 0, 1, 2. PTS follows display order. + let u0 = Self.run(&a, Self.unit(bl: Self.slice(T.idrWRADL, poc: 0), el: Self.slice(T.idrWRADL, poc: 0), parameterSets: true), pts: 2000) + #expect(u0 == Result(pts: 2000, placed: true)) + let u1 = Self.run(&a, Self.unit(bl: Self.slice(T.radlR, poc: -2), el: Self.slice(T.radlR, poc: -2)), pts: 0) + let u2 = Self.run(&a, Self.unit(bl: Self.slice(T.radlR, poc: -1), el: Self.slice(T.radlR, poc: -1)), pts: 1000) + #expect(u1 == Result(pts: 0, placed: true), "same frame in both layers: unchanged, but placed by its slot") + #expect(u2 == Result(pts: 1000, placed: true)) + + let u3 = Self.run(&a, Self.unit(bl: Self.slice(T.trailR, poc: 1), el: Self.slice(T.trailR, poc: 2)), pts: 3000) + #expect(u3 == Result(pts: 4000, placed: true), "alignment must survive the RADL pictures") + let u4 = Self.run(&a, Self.unit(bl: Self.slice(T.trailR, poc: 2), el: Self.slice(T.trailR, poc: 1)), pts: 4000) + #expect(u4 == Result(pts: 3000, placed: true)) + } + + // MARK: - Baselines the fix must not change + + @Test("layers with the same GOP keep every unit's own time") + func sameGopUnchanged() { + var a = Self.seeded() + a.reset() + _ = Self.run(&a, Self.unit(bl: Self.slice(T.idrNLP, poc: 0), el: Self.slice(T.idrNLP, poc: 0), parameterSets: true), pts: 0) + for i in 1...20 { + let r = Self.run(&a, Self.unit(bl: Self.slice(T.trailR, poc: i), el: Self.slice(T.trailR, poc: i)), pts: Int64(i) * 1000) + #expect(r == Result(pts: Int64(i) * 1000, placed: true)) + } + #expect(a.reorderedUnits == 0) + } + + @Test("POC that does not step one per frame is not trusted: the layer falls back to access-unit pairing") + func slotFurtherThanDpbIsNotTrusted() { + var a = Self.seeded() + a.reset() + _ = Self.run(&a, Self.unit(bl: Self.slice(T.idrNLP, poc: 0), el: Self.slice(T.idrNLP, poc: 0), parameterSets: true), pts: 0) + // The EL jumps 40 pictures ahead in one unit: further than any DPB + let r = Self.run(&a, Self.unit(bl: Self.slice(T.trailR, poc: 1), el: Self.slice(T.trailR, poc: 41)), pts: 1000) + #expect(r == Result(pts: 1000, placed: false)) + } + + // MARK: - Restamp distance the pairing can absorb + + @Test("an enhancement picture restamped further back than the owner's skew bound is not placed; one within it is") + func restampBeyondSkewBoundIsNotPlaced() { + var a = Self.seeded() + a.maxBackwardRestamp = 2 // DualLayerVideoDecoder.maxLayerSkewFrames (3) - 1 + a.reset() + _ = Self.run(&a, Self.unit(bl: Self.slice(T.idrNLP, poc: 0), el: Self.slice(T.idrNLP, poc: 0), parameterSets: true), pts: 0) + for i in 1...4 { + _ = Self.run(&a, Self.unit(bl: Self.slice(T.trailR, poc: i), el: Self.slice(T.trailR, poc: i)), pts: Int64(i) * 1000) + } + // Unit 5, EL shows picture 3: two back, the partner (base picture 3) is still waiting + let within = Self.run(&a, Self.unit(bl: Self.slice(T.trailR, poc: 5), el: Self.slice(T.trailR, poc: 3)), pts: 5000) + #expect(within == Result(pts: 3000, placed: true)) + #expect(a.unabsorbableRestamps == 0) + // Unit 6, EL shows picture 3 again: three back, its partner has been bypassed by then + let beyond = Self.run(&a, Self.unit(bl: Self.slice(T.trailR, poc: 6), el: Self.slice(T.trailR, poc: 3)), pts: 6000) + #expect(beyond == Result(pts: 6000, placed: false)) + #expect(a.unabsorbableRestamps == 1) + } + + @Test("the default bound is the DPB bound: restamps up to 16 back are placed") + func defaultBoundAbsorbsDpbDepth() { + var a = Self.seeded() + a.reset() + _ = Self.run(&a, Self.unit(bl: Self.slice(T.idrNLP, poc: 0), el: Self.slice(T.idrNLP, poc: 0), parameterSets: true), pts: 0) + for i in 1...20 { + _ = Self.run(&a, Self.unit(bl: Self.slice(T.trailR, poc: i), el: Self.slice(T.trailR, poc: i)), pts: Int64(i) * 1000) + } + let r = Self.run(&a, Self.unit(bl: Self.slice(T.trailR, poc: 21), el: Self.slice(T.trailR, poc: 6)), pts: 21000) + #expect(r == Result(pts: 6000, placed: true)) + } +} diff --git a/Tests/AetherEngineTests/DualLayerPictureAlignmentTests.swift b/Tests/AetherEngineTests/DualLayerPictureAlignmentTests.swift new file mode 100644 index 000000000..dcd78572f --- /dev/null +++ b/Tests/AetherEngineTests/DualLayerPictureAlignmentTests.swift @@ -0,0 +1,137 @@ +import Testing +import Foundation +import AetherLibavformat +import AetherLibavcodec +import AetherLibavutil +@testable import AetherEngine + +/// The enhancement-layer restamp on real profile 7 files. One of the test clips codes its EL with B-frames +/// and its BL without, so 2822 of its access units carry an EL picture of another frame +/// (counting from the decoded POCs gives 2823: that also counts the last unit, whose EL is a lone IDR +/// that pairs with its own BL picture but ranks two places off because two EL-less units precede it); +/// the others share one GOP. +struct DualLayerPictureAlignmentTests { + + struct Walk { + var units: [(pts: Int64, el: Int64?)] = [] + var reordered = 0 + } + + static func walk(_ name: String, seekTo seconds: Double? = nil, limit: Int = .max) throws -> Walk { + let demuxer = Demuxer() + try demuxer.open(url: DoviFelRealFileTests.url(name)) + defer { demuxer.close() } + let index = demuxer.videoStreamIndex + let par = try #require(demuxer.stream(at: index)?.pointee.codecpar?.pointee) + var alignment = DualLayerPictureAlignment() + if let ed = par.extradata, par.extradata_size > 23, ed[0] == 1 { + alignment.seedBaseParameterSets(parseHVCCParameterSets(ed, Int(par.extradata_size))) + } + if let seconds { + try #require(demuxer.seek(to: seconds), "seek failed") + alignment.reset() + } + var out = Walk() + while out.units.count < limit, let pkt = try demuxer.readPacket() { + defer { av_packet_unref(pkt); var o: UnsafeMutablePointer? = pkt; trackedPacketFree(&o) } + guard pkt.pointee.stream_index == Int32(index), let data = pkt.pointee.data else { continue } + var hasEL = false + A53SEIParser.forEachNAL(data, Int(pkt.pointee.size), .lengthPrefixed(size: 4)) { nal, _ in + if (nal[0] >> 1) & 0x3F == 63 { hasEL = true } + } + let el = alignment.enhancementPTS(data, Int(pkt.pointee.size), framing: .lengthPrefixed(size: 4), pts: pkt.pointee.pts) + out.units.append((pkt.pointee.pts, hasEL ? el : nil)) + } + out.reordered = alignment.reorderedUnits + return out + } + + /// Every restamped EL picture lands on exactly one base-layer picture's time (within 3 ms). + static func unmatched(_ w: Walk, base basePTS: [Int64]? = nil) -> (missing: Int, shared: Int) { + let base = (basePTS ?? w.units.map(\.pts)).sorted() + var used = Set() + var missing = 0, shared = 0 + for case let el? in w.units.map(\.el) { + var lo = 0, hi = base.count + while lo < hi { let mid = (lo + hi) / 2; if base[mid] < el { lo = mid + 1 } else { hi = mid } } + let near = [lo - 1, lo].filter { base.indices.contains($0) }.min { abs(base[$0] - el) < abs(base[$1] - el) } + guard let i = near, abs(base[i] - el) <= 3 else { missing += 1; continue } + if !used.insert(i).inserted { shared += 1 } + } + return (missing, shared) + } + + /// The two clips whose layers share one GOP, when present. + static let sharedGOPClips = [ + "fel-clip-b.mkv", + "fel-clip-c.mp4", + ].filter(DoviFelRealFileTests.available) + + @Test("an EL coded with B-frames over a BL without is paired by POC", + .enabled(if: DoviFelRealFileTests.available(DoviFelRealFileTests.fixtures[0]), DoviFelRealFileTests.needsClips)) + func reorderedEnhancementLayer() throws { + let name = DoviFelRealFileTests.fixtures[0] + let w = try Self.walk(name) + #expect(w.reordered == 2822) + let moved = w.units.compactMap { u in u.el.map { $0 - u.pts } }.filter { $0 != 0 } + #expect(moved.count == 2822) + #expect(moved.allSatisfy { abs($0) <= 3 * 42 }, "offsets beyond two frames") + let m = Self.unmatched(w) + #expect(m.missing == 0 && m.shared == 0, "\(m)") + } + + @Test("after a seek the layers realign at the first IRAP", + .enabled(if: DoviFelRealFileTests.available(DoviFelRealFileTests.fixtures[0]), DoviFelRealFileTests.needsClips)) + func realignsAfterSeek() throws { + let name = DoviFelRealFileTests.fixtures[0] + let w = try Self.walk(name, seekTo: 90, limit: 600) + #expect(w.reordered > 0) + // Until the EL's first sequence-opening IRAP after the seek, each EL picture keeps its own unit's + // time (access-unit pairing), and the first restamped pictures can then land on base slots those + // units already took (on this clip, two). That stretch is short. From the first restamp on, every EL + // picture lands on exactly one base picture. The last few may belong to base frames past the window. + let firstRestamp = try #require(w.units.firstIndex { $0.el != nil && $0.el != $0.pts }) + #expect(firstRestamp < 30, "access-unit pairing lasted \(firstRestamp) units") + #expect(w.units.prefix(firstRestamp).allSatisfy { $0.el == nil || $0.el == $0.pts }) + let realigned = Walk(units: Array(w.units[firstRestamp...].dropLast(3)), reordered: 0) + let m = Self.unmatched(realigned, base: w.units.map(\.pts)) + #expect(m.missing == 0 && m.shared == 0, "\(m)") + // That clip's EL opens a sequence roughly every 100 units, so by the second half of the window it has + // realigned: from there each stamp is the one an unbroken walk from the start gives the same unit. + let linear = Dictionary(try Self.walk(name).units.map { ($0.pts, $0.el) }, uniquingKeysWith: { a, _ in a }) + let late = w.units.dropFirst(300).dropLast(3) + #expect(late.count > 200, "too few units left to compare") + let differing = late.filter { linear[$0.pts] != .some($0.el) }.count + #expect(differing == 0, "\(differing) of \(late.count) units stamped unlike the linear walk") + } + + @Test("layers with one GOP keep their access unit's time", + .enabled(if: !DualLayerPictureAlignmentTests.sharedGOPClips.isEmpty, DoviFelRealFileTests.needsClips), + arguments: DualLayerPictureAlignmentTests.sharedGOPClips) + func sharedGOPUntouched(name: String) throws { + let w = try Self.walk(name) + #expect(w.units.contains { $0.el != nil }, "no EL found") + #expect(w.reordered == 0) + #expect(w.units.allSatisfy { $0.el == nil || $0.el == $0.pts }) + } + + @Test("extrapolating from crafted timestamps declines instead of trapping") + func extremeTimestampsDoNotOverflow() { + // Ordinary case still extrapolates: 10 ticks per slot, three slots ahead. + #expect(DualLayerPictureAlignment.extrapolate(origin: (0, 0), latest: (4, 40), slot: 7) == 70) + + let big = Int64.max + let small = Int64.min + 1 + // latest.pts - origin.pts overflows. + #expect(DualLayerPictureAlignment.extrapolate(origin: (0, small), latest: (4, big), slot: 6) == nil) + // The slot distances overflow. + #expect(DualLayerPictureAlignment.extrapolate(origin: (Int.min, 0), latest: (Int.max, 10), slot: 0) == nil) + #expect(DualLayerPictureAlignment.extrapolate(origin: (0, 0), latest: (1, 10), slot: Int.min) == nil) + // The extrapolated offset does not fit an Int64, and neither does the sum. + #expect(DualLayerPictureAlignment.extrapolate(origin: (0, 0), latest: (1, big / 2), slot: Int.max) == nil) + #expect(DualLayerPictureAlignment.extrapolate(origin: (0, 0), latest: (1, big - 5), slot: 3) == nil) + // Time running backwards or standing still has no rate to extrapolate. + #expect(DualLayerPictureAlignment.extrapolate(origin: (0, 100), latest: (4, 40), slot: 6) == nil) + #expect(DualLayerPictureAlignment.extrapolate(origin: (3, 0), latest: (3, 40), slot: 6) == nil) + } +} diff --git a/Tests/AetherEngineTests/FrameSideDataRingTests.swift b/Tests/AetherEngineTests/FrameSideDataRingTests.swift new file mode 100644 index 000000000..4ca82368b --- /dev/null +++ b/Tests/AetherEngineTests/FrameSideDataRingTests.swift @@ -0,0 +1,152 @@ +import Testing +import Foundation +@testable import AetherEngine + +@Suite("FrameSideDataRing unit tests") +struct FrameSideDataRingTests { + + @Test("matches within ±300 ticks") + func matchesWithinTolerance() { + var ring = FrameSideDataRing() + let dmNAL: [UInt8] = [0x7C, 0x01, 0x01] + ring.insert(key: 10_000, params: DoviMetalParams(), dmNAL: dmNAL) + + // Exact match + #expect(ring.find(key: 10_000) != nil) + #expect(ring.find(key: 10_000)?.key == 10_000) + + // Positive boundary: +300 ticks + #expect(ring.find(key: 10_300) != nil) + #expect(ring.find(key: 10_300)?.key == 10_000) + + // Negative boundary: -300 ticks + #expect(ring.find(key: 9_700) != nil) + #expect(ring.find(key: 9_700)?.key == 10_000) + + // Intermediate offset: +150 ticks + #expect(ring.find(key: 10_150) != nil) + #expect(ring.find(key: 10_150)?.key == 10_000) + } + + @Test("misses at 301 ticks") + func missesAt301() { + var ring = FrameSideDataRing() + ring.insert(key: 10_000, params: DoviMetalParams(), dmNAL: [0x01]) + + // +301 ticks: strictly outside ±300 tolerance + #expect(ring.find(key: 10_301) == nil) + + // -301 ticks: strictly outside ±300 tolerance + #expect(ring.find(key: 9_699) == nil) + + // Far away + #expect(ring.find(key: 20_000) == nil) + } + + @Test("overwrites the oldest entry when capacity 32 is exceeded") + func overwritesOldest() { + var ring = FrameSideDataRing() + #expect(FrameSideDataRing.capacity == 32) + + // Insert 32 items, keys 1_000, 2_000, ..., 32_000 (well separated, > 300 apart) + for i in 1...32 { + ring.insert(key: Int64(i * 1_000), params: DoviMetalParams(), dmNAL: [UInt8(i)]) + } + + #expect(ring.count == 32) + #expect(ring.find(key: 1_000) != nil) + #expect(ring.find(key: 32_000) != nil) + + // Insert 33rd item: key 33_000 + ring.insert(key: 33_000, params: DoviMetalParams(), dmNAL: [33]) + + // Total count stays capped at 32 + #expect(ring.count == 32) + + // The oldest entry (key 1_000) was overwritten and is now gone + #expect(ring.find(key: 1_000) == nil) + + // Key 2_000 is now the oldest surviving entry + #expect(ring.find(key: 2_000) != nil) + + // Key 33_000 is present + #expect(ring.find(key: 33_000) != nil) + } + + @Test("selects nearest entry when multiple entries are within ±300 ticks") + func selectsNearest() { + var ring = FrameSideDataRing() + ring.insert(key: 1_000, params: DoviMetalParams(), dmNAL: [1]) + ring.insert(key: 1_200, params: DoviMetalParams(), dmNAL: [2]) + + // Query 1_050: distance to 1_000 is 50, distance to 1_200 is 150 -> selects 1_000 + #expect(ring.find(key: 1_050)?.key == 1_000) + + // Query 1_180: distance to 1_200 is 20, distance to 1_000 is 180 -> selects 1_200 + #expect(ring.find(key: 1_180)?.key == 1_200) + + // Query 1_100: exact midpoint (distance 100 to both) -> picks first encountered + #expect(ring.find(key: 1_100) != nil) + } + + @Test("display metadata and parameters can be removed independently") + func independentRemoval() { + var ring = FrameSideDataRing() + let nal: [UInt8] = [0x7C, 0x01, 0xAA] + ring.insert(key: 5_000, params: DoviMetalParams(), dmNAL: nal) + + #expect(ring.dmNALCount == 1) + #expect(ring.paramsCount == 1) + + // Remove display metadata first + let extractedNAL = ring.removeDisplayMetadata(key: 5_050) + #expect(extractedNAL == nal) + #expect(ring.dmNALCount == 0) + #expect(ring.paramsCount == 1) + + // Second removal of display metadata returns nil + #expect(ring.removeDisplayMetadata(key: 5_000) == nil) + + // Parameters are still intact and can be removed + let extractedParams = ring.removeParams(key: 4_950) + #expect(extractedParams != nil) + #expect(ring.paramsCount == 0) + + // Second removal of parameters returns nil + #expect(ring.removeParams(key: 5_000) == nil) + } + + @Test("inserting for identical key updates in place without consuming a slot") + func inPlaceUpdate() { + var ring = FrameSideDataRing() + ring.insert(key: 5_000, params: DoviMetalParams(), dmNAL: nil) + #expect(ring.count == 1) + #expect(ring.paramsCount == 1) + #expect(ring.dmNALCount == 0) + + let nal: [UInt8] = [0x7C, 0x01, 0xBB] + ring.insert(key: 5_000, params: nil, dmNAL: nal) + #expect(ring.count == 1) + #expect(ring.paramsCount == 1) + #expect(ring.dmNALCount == 1) + + #expect(ring.removeDisplayMetadata(key: 5_000) == nal) + #expect(ring.removeParams(key: 5_000) != nil) + } + + @Test("removeAll clears all entries") + func removeAll() { + var ring = FrameSideDataRing() + ring.insert(key: 1_000, params: DoviMetalParams(), dmNAL: [1]) + ring.insert(key: 2_000, params: DoviMetalParams(), dmNAL: [2]) + #expect(!ring.isEmpty) + #expect(ring.count == 2) + + ring.removeAll() + #expect(ring.isEmpty) + #expect(ring.count == 0) + #expect(ring.paramsCount == 0) + #expect(ring.dmNALCount == 0) + #expect(ring.find(key: 1_000) == nil) + } +} diff --git a/Tests/AetherEngineTests/HEVCPictureOrderCountTests.swift b/Tests/AetherEngineTests/HEVCPictureOrderCountTests.swift new file mode 100644 index 000000000..0826c231e --- /dev/null +++ b/Tests/AetherEngineTests/HEVCPictureOrderCountTests.swift @@ -0,0 +1,321 @@ +import Testing +import Foundation +@testable import AetherEngine + +/// `HEVCPictureOrderCount` against hand-built parameter sets and slice headers: every expected POC +/// below is worked out from H.265 §8.3.1 by hand, not by running the reader. (The real-clip tests +/// in `DualLayerPictureAlignmentTests` need files that are not in the repository.) +struct HEVCPictureOrderCountTests { + + typealias B = HEVCSyntheticBitstream + typealias T = HEVCSyntheticBitstream.T + + /// A counter with SPS 0 / PPS 0 loaded, 4-bit POC LSB (MaxPicOrderCntLsb = 16) unless told otherwise. + private static func counter(pocLsbBits: Int = 4, pps: [UInt8]? = nil, sps: [UInt8]? = nil) -> HEVCPictureOrderCount { + var c = HEVCPictureOrderCount() + c.feedParameterSets([sps ?? B.sps(pocLsbBits: pocLsbBits), pps ?? B.pps()]) + return c + } + + private static func feed(_ c: inout HEVCPictureOrderCount, _ nal: [UInt8]) -> HEVCPictureOrderCount.Picture? { + nal.withUnsafeBufferPointer { c.feed($0.baseAddress!, $0.count) } + } + + private static func slice(_ type: Int, lsb: Int = 0, tid: Int = 0, bits: Int = 4) -> [UInt8] { + B.slice(type: type, temporalId: tid, pocLsb: lsb, pocLsbBits: bits) + } + + // MARK: - MSB wrap + + @Test("POC MSB wraps forward: LSB falling by half the range or more adds MaxPicOrderCntLsb") + func msbWrapsForward() { + var c = Self.counter() + #expect(Self.feed(&c, Self.slice(T.idrNLP)) == .init(poc: 0, startsSequence: true)) + #expect(Self.feed(&c, Self.slice(T.trailR, lsb: 5))?.poc == 5) + #expect(Self.feed(&c, Self.slice(T.trailR, lsb: 10))?.poc == 10) + #expect(Self.feed(&c, Self.slice(T.trailR, lsb: 15))?.poc == 15) + // prevLsb 15, lsb 4: 15 - 4 = 11 >= 8, so MSB += 16 -> 16 + 4 + #expect(Self.feed(&c, Self.slice(T.trailR, lsb: 4))?.poc == 20) + // And a second wrap + #expect(Self.feed(&c, Self.slice(T.trailR, lsb: 9))?.poc == 25) + #expect(Self.feed(&c, Self.slice(T.trailR, lsb: 14))?.poc == 30) + #expect(Self.feed(&c, Self.slice(T.trailR, lsb: 3))?.poc == 35) + } + + @Test("POC MSB wraps backward: LSB rising by more than half the range subtracts MaxPicOrderCntLsb") + func msbWrapsBackward() { + var c = Self.counter() + _ = Self.feed(&c, Self.slice(T.idrNLP)) + _ = Self.feed(&c, Self.slice(T.trailR, lsb: 6)) + _ = Self.feed(&c, Self.slice(T.trailR, lsb: 12)) + // prevTid0 = 12 (lsb 12, msb 0); lsb 4 -> 12 - 4 = 8 >= 8: wraps forward to 20 + #expect(Self.feed(&c, Self.slice(T.trailR, lsb: 4))?.poc == 20) + // prevTid0 = 20 (lsb 4, msb 16); lsb 15: 15 - 4 = 11 > 8 -> MSB = 16 - 16 = 0 -> 15 (a picture + // displayed before the one at 20, as a B-frame under a wrap would be) + #expect(Self.feed(&c, Self.slice(T.trailR, lsb: 15))?.poc == 15) + } + + @Test("a half-range step is a wrap one way and not the other (the spec's strict and non-strict comparisons)") + func halfRangeBoundary() { + var c = Self.counter() + _ = Self.feed(&c, Self.slice(T.idrNLP)) + _ = Self.feed(&c, Self.slice(T.trailR, lsb: 8)) + // lsb 0, prev 8: prev - lsb = 8 >= 8, wraps forward -> 16 + #expect(Self.feed(&c, Self.slice(T.trailR, lsb: 0))?.poc == 16) + // lsb 8, prev 0 (of 16): lsb - prev = 8, not > 8, so no wrap -> 24 + #expect(Self.feed(&c, Self.slice(T.trailR, lsb: 8))?.poc == 24) + } + + // MARK: - Sequence starts + + @Test("IDR and BLA start a sequence at MSB 0; BLA keeps its LSB") + func idrAndBlaStartSequences() { + var c = Self.counter() + _ = Self.feed(&c, Self.slice(T.idrWRADL)) + _ = Self.feed(&c, Self.slice(T.trailR, lsb: 7)) + let bla = Self.feed(&c, Self.slice(T.blaWLP, lsb: 9)) + #expect(bla == .init(poc: 9, startsSequence: true)) + // POC numbering restarts: the next picture is relative to the BLA, not to the IDR's sequence + #expect(Self.feed(&c, Self.slice(T.trailR, lsb: 10))?.poc == 10) + } + + @Test("a CRA in the middle of a sequence continues the numbering; after reset() it starts one (CRA handled as BLA)") + func craAsBlaAfterReset() { + var c = Self.counter() + _ = Self.feed(&c, Self.slice(T.idrNLP)) + _ = Self.feed(&c, Self.slice(T.trailR, lsb: 7)) + _ = Self.feed(&c, Self.slice(T.trailR, lsb: 15)) + // Mid-sequence CRA: NoRaslOutputFlag = 0, POC continues across the wrap (16 + 3) + #expect(Self.feed(&c, Self.slice(T.cra, lsb: 3)) == .init(poc: 19, startsSequence: false)) + + c.reset() + // Pictures before the first IRAP after a seek are not decodable: no POC + #expect(Self.feed(&c, Self.slice(T.trailR, lsb: 4)) == nil) + // The CRA opens a sequence (HandleCraAsBlaFlag): MSB 0, so POC = LSB + #expect(Self.feed(&c, Self.slice(T.cra, lsb: 3)) == .init(poc: 3, startsSequence: true)) + #expect(Self.feed(&c, Self.slice(T.trailR, lsb: 5))?.poc == 5) + } + + @Test("end of sequence and end of bitstream NALs make the next IRAP a sequence start and hide what precedes it") + func endOfSequenceAndBitstream() { + for eosType in [T.eos, T.eob] { + var c = Self.counter() + _ = Self.feed(&c, Self.slice(T.idrNLP)) + _ = Self.feed(&c, Self.slice(T.trailR, lsb: 5)) + #expect(Self.feed(&c, B.nal(type: eosType, rbsp: [0x80])) == nil) + #expect(Self.feed(&c, Self.slice(T.trailR, lsb: 6)) == nil, "no picture is decodable between the end of sequence and the next IRAP") + // A CRA after the end of sequence has NoRaslOutputFlag = 1 + #expect(Self.feed(&c, Self.slice(T.cra, lsb: 11)) == .init(poc: 11, startsSequence: true)) + #expect(Self.feed(&c, Self.slice(T.trailR, lsb: 12))?.poc == 12) + } + } + + // MARK: - prevTid0Pic + + /// prevTid0Pic is the previous TemporalId-0 picture that is not RASL, RADL or a sub-layer + /// non-reference picture. Each excluded kind is fed with an LSB that would change the result + /// if it had been taken as the reference. + @Test("RASL, RADL, sub-layer non-reference pictures and TemporalId > 0 are not prevTid0Pic", + arguments: [T.trailN, T.tsaN, T.radlN, T.radlR, T.raslN, T.raslR]) + func excludedFromPrevTid0(type: Int) { + var c = Self.counter() + _ = Self.feed(&c, Self.slice(T.idrNLP)) + #expect(Self.feed(&c, Self.slice(T.trailR, lsb: 2))?.poc == 2) // prevTid0 = 2 + // The excluded picture: POC 9 relative to prevTid0 = 2 (9 - 2 = 7, no wrap) + #expect(Self.feed(&c, Self.slice(type, lsb: 9))?.poc == 9) + // If it had become prevTid0 (9), lsb 14 would be poc 14. Relative to 2 it is 14 - 2 = 12 > 8, + // a backward wrap: 14 - 16 = -2. + #expect(Self.feed(&c, Self.slice(T.trailR, lsb: 14))?.poc == -2) + } + + @Test("a picture with TemporalId > 0 is not prevTid0Pic") + func higherTemporalLayerExcluded() { + var c = Self.counter() + _ = Self.feed(&c, Self.slice(T.idrNLP)) + _ = Self.feed(&c, Self.slice(T.trailR, lsb: 2)) + #expect(Self.feed(&c, Self.slice(T.trailR, lsb: 9, tid: 1))?.poc == 9) + #expect(Self.feed(&c, Self.slice(T.trailR, lsb: 14))?.poc == -2) + } + + @Test("a reference picture at TemporalId 0 is prevTid0Pic") + func referencePictureIsPrevTid0() { + var c = Self.counter() + _ = Self.feed(&c, Self.slice(T.idrNLP)) + _ = Self.feed(&c, Self.slice(T.trailR, lsb: 2)) + _ = Self.feed(&c, Self.slice(T.trailR, lsb: 9)) + // prevTid0 = 9: lsb 14 is 14 - 9 = 5 ahead, no wrap + #expect(Self.feed(&c, Self.slice(T.trailR, lsb: 14))?.poc == 14) + } + + // MARK: - Leading pictures after a sequence-opening CRA + + @Test("RASL pictures of a CRA that opens a sequence get no POC; RADL and trailing pictures do") + func raslSkippedAfterOpeningCra() { + for rasl in [T.raslN, T.raslR] { + var c = Self.counter() + c.reset() + #expect(Self.feed(&c, Self.slice(T.cra, lsb: 8))?.startsSequence == true) + #expect(Self.feed(&c, Self.slice(rasl, lsb: 6)) == nil, "RASL of a CRA with NoRaslOutputFlag = 1 is not output") + #expect(Self.feed(&c, Self.slice(T.radlR, lsb: 7))?.poc == 7, "RADL is decodable and output before the CRA") + #expect(Self.feed(&c, Self.slice(T.trailR, lsb: 9))?.poc == 9) + } + } + + @Test("RASL pictures of a CRA in the middle of a sequence are ordinary pictures") + func raslKeptAfterMidStreamCra() { + var c = Self.counter() + _ = Self.feed(&c, Self.slice(T.idrNLP)) + _ = Self.feed(&c, Self.slice(T.trailR, lsb: 4)) + #expect(Self.feed(&c, Self.slice(T.cra, lsb: 8))?.startsSequence == false) + #expect(Self.feed(&c, Self.slice(T.raslR, lsb: 6))?.poc == 6) + } + + @Test("an IDR clears the RASL skip a previous CRA set") + func idrClearsRaslSkip() { + var c = Self.counter() + c.reset() + _ = Self.feed(&c, Self.slice(T.cra, lsb: 8)) + _ = Self.feed(&c, Self.slice(T.idrWRADL)) + // Not a legal stream (an IDR has no RASL), but the skip must not outlive its IRAP + #expect(Self.feed(&c, Self.slice(T.raslN, lsb: 1))?.poc == 1) + } + + @Test("the RASL skip ends at the next IRAP; BLA also opens it") + func blaSkipsRasl() { + var c = Self.counter() + _ = Self.feed(&c, Self.slice(T.idrNLP)) + _ = Self.feed(&c, Self.slice(T.blaWLP, lsb: 8)) + #expect(Self.feed(&c, Self.slice(T.raslN, lsb: 6)) == nil) + _ = Self.feed(&c, Self.slice(T.cra, lsb: 12)) // mid-sequence CRA: RASL are output again + #expect(Self.feed(&c, Self.slice(T.raslN, lsb: 10))?.poc == 10) + } + + // MARK: - Header syntax + + @Test("emulation prevention bytes inside the parameter sets and the slice header are removed before parsing") + func emulationPrevention() { + // The SPS carries 6 zero bytes of constraint flags, which always needs escaping + let sps = B.sps(pocLsbBits: 16) + #expect(sps.contains(0x03)) + #expect(Array(sps).windows3.contains { $0 == [0, 0, 3] }) + // Slice: first_slice 1, no_output 0, pps ue(0) = 1, two extra bits 00, slice_type ue(2) = 011 + // makes byte 0 = 10100011; a 16-bit POC LSB of 0 then gives two zero bytes, and the following + // 00000001 byte makes the sequence 00 00 01, which must be escaped to 00 00 03 01. + let pps = B.pps(extraSliceHeaderBits: 2) + let cra = B.slice(type: T.cra, extraBits: 2, pocLsb: 0, pocLsbBits: 16, afterPOC: (1, 8)) + #expect(cra.windows3.contains { $0 == [0, 0, 3] }, "the slice must actually contain an escape") + var c = HEVCPictureOrderCount() + c.feedParameterSets([sps, pps]) + c.reset() + #expect(Self.feed(&c, cra) == .init(poc: 0, startsSequence: true)) + let trail = B.slice(type: T.trailR, extraBits: 2, pocLsb: 0x0003, pocLsbBits: 16) + #expect(Self.feed(&c, trail)?.poc == 3) + } + + @Test("num_extra_slice_header_bits shifts where slice_type and the POC LSB sit") + func extraSliceHeaderBits() { + for extra in [0, 1, 3, 7] { + var c = HEVCPictureOrderCount() + c.feedParameterSets([B.sps(pocLsbBits: 6), B.pps(extraSliceHeaderBits: extra)]) + _ = Self.feed(&c, B.slice(type: T.idrNLP, extraBits: extra, extraBitsValue: (1 << extra) - 1, pocLsbBits: 6)) + let p = Self.feed(&c, B.slice(type: T.trailR, extraBits: extra, extraBitsValue: (1 << extra) - 1, + pocLsb: 20, pocLsbBits: 6)) + #expect(p?.poc == 20, "extra bits = \(extra)") + } + } + + @Test("pic_output_flag is read when output_flag_present_flag is set, and only then") + func outputFlagPresent() { + var with = HEVCPictureOrderCount() + with.feedParameterSets([B.sps(pocLsbBits: 8), B.pps(outputFlagPresent: true)]) + _ = Self.feed(&with, B.slice(type: T.idrNLP, outputFlag: true, pocLsbBits: 8)) + // A picture with pic_output_flag = 0 still has a POC, and still moves prevTid0Pic + #expect(Self.feed(&with, B.slice(type: T.trailR, outputFlag: false, pocLsb: 77, pocLsbBits: 8))?.poc == 77) + #expect(Self.feed(&with, B.slice(type: T.trailR, outputFlag: true, pocLsb: 78, pocLsbBits: 8))?.poc == 78) + + // Without the flag in the PPS, a slice that does not carry it parses the same LSB + var without = HEVCPictureOrderCount() + without.feedParameterSets([B.sps(pocLsbBits: 8), B.pps(outputFlagPresent: false)]) + _ = Self.feed(&without, B.slice(type: T.idrNLP, pocLsbBits: 8)) + #expect(Self.feed(&without, B.slice(type: T.trailR, pocLsb: 77, pocLsbBits: 8))?.poc == 77) + } + + @Test("colour_plane_id is read when separate_colour_plane_flag is set (4:4:4 coded as three planes)") + func separateColourPlane() { + var c = HEVCPictureOrderCount() + c.feedParameterSets([B.sps(pocLsbBits: 8, chromaFormatIdc: 3, separateColourPlane: true), B.pps()]) + _ = Self.feed(&c, B.slice(type: T.idrNLP, colourPlane: 0, pocLsbBits: 8)) + #expect(Self.feed(&c, B.slice(type: T.trailR, colourPlane: 1, pocLsb: 41, pocLsbBits: 8))?.poc == 41) + + // 4:4:4 with the planes together has no colour_plane_id + var joint = HEVCPictureOrderCount() + joint.feedParameterSets([B.sps(pocLsbBits: 8, chromaFormatIdc: 3, separateColourPlane: false), B.pps()]) + _ = Self.feed(&joint, B.slice(type: T.idrNLP, pocLsbBits: 8)) + #expect(Self.feed(&joint, B.slice(type: T.trailR, pocLsb: 41, pocLsbBits: 8))?.poc == 41) + } + + @Test("sub-layer profile/level flags in the SPS are skipped correctly") + func spsWithSubLayers() { + var c = HEVCPictureOrderCount() + c.feedParameterSets([B.sps(pocLsbBits: 9, subLayers: 3), B.pps()]) + _ = Self.feed(&c, B.slice(type: T.idrNLP, pocLsbBits: 9)) + #expect(Self.feed(&c, B.slice(type: T.trailR, pocLsb: 200, pocLsbBits: 9))?.poc == 200) + } + + // MARK: - Missing or unusable input + + @Test("a slice whose PPS or SPS has not been seen has no POC; it gets one once they arrive") + func missingParameterSets() { + var c = HEVCPictureOrderCount() + let idr = B.slice(type: T.idrNLP, pocLsbBits: 8) + #expect(Self.feed(&c, idr) == nil, "no PPS, no SPS") + _ = Self.feed(&c, B.pps(id: 0, spsID: 0)) + #expect(Self.feed(&c, idr) == nil, "PPS refers to an SPS that has not been seen") + _ = Self.feed(&c, B.sps(id: 0, pocLsbBits: 8)) + #expect(Self.feed(&c, B.slice(type: T.idrNLP, ppsID: 5, pocLsbBits: 8)) == nil, "PPS id 5 was never sent") + #expect(Self.feed(&c, idr) == .init(poc: 0, startsSequence: true)) + } + + @Test("a picture that is not the first slice segment is ignored and does not disturb state") + func nonFirstSliceIgnored() { + var c = Self.counter() + _ = Self.feed(&c, Self.slice(T.idrNLP)) + #expect(Self.feed(&c, B.slice(type: T.trailR, firstSliceSegment: false, pocLsb: 9, pocLsbBits: 4)) == nil) + // prevTid0 is still 0: lsb 9 is 9 > 8 ahead, a backward wrap -> -7. Had the second slice + // been taken as a picture (prevTid0 = 9) it would read 9 -> 9. + #expect(Self.feed(&c, Self.slice(T.trailR, lsb: 9))?.poc == -7) + } + + @Test("pictures before the first IRAP, truncated headers and junk NALs answer nil") + func rejectsWhatItCannotFollow() { + var c = Self.counter() + #expect(Self.feed(&c, Self.slice(T.trailR, lsb: 3)) == nil, "no IRAP yet") + #expect(Self.feed(&c, Array(Self.slice(T.idrNLP).prefix(2))) == nil, "a bare NAL header") + #expect(Self.feed(&c, [0x02]) == nil) + #expect(Self.feed(&c, B.nal(type: 39, rbsp: [0x00, 0x01])) == nil, "an SEI is not a picture") + #expect(Self.feed(&c, B.nal(type: 32, rbsp: [0x0C, 0x01])) == nil, "nor is a VPS") + // The decoder still works afterwards + #expect(Self.feed(&c, Self.slice(T.idrNLP))?.poc == 0) + // A slice header cut off inside the POC LSB, after an IRAP so that the truncation is the only + // reason to answer nil: the same header complete is a picture + let trail = Self.slice(T.trailR, lsb: 5) + #expect(Self.feed(&c, Array(trail.prefix(3))) == nil) + #expect(Self.feed(&c, trail)?.poc == 5) + } + + @Test("an SPS with an out-of-range POC LSB width is not stored") + func rejectsOversizedLsbWidth() { + var c = HEVCPictureOrderCount() + // log2_max_pic_order_cnt_lsb_minus4 = 13 (> 12) is outside what the spec allows + c.feedParameterSets([B.sps(pocLsbBits: 17), B.pps()]) + #expect(Self.feed(&c, B.slice(type: T.idrNLP, pocLsbBits: 17)) == nil) + } +} + +private extension Array where Element == UInt8 { + /// Every run of three consecutive bytes. + var windows3: [[UInt8]] { + guard count >= 3 else { return [] } + return (0...(count - 3)).map { Array(self[$0..<$0 + 3]) } + } +} diff --git a/Tests/AetherEngineTests/HEVCSyntheticBitstream.swift b/Tests/AetherEngineTests/HEVCSyntheticBitstream.swift new file mode 100644 index 000000000..3e0e8fffb --- /dev/null +++ b/Tests/AetherEngineTests/HEVCSyntheticBitstream.swift @@ -0,0 +1,127 @@ +import Foundation +@testable import AetherEngine + +/// Hand-built HEVC parameter sets and slice segment headers, just deep enough for +/// `HEVCPictureOrderCount` and `DualLayerPictureAlignment` to read: no picture data, no real +/// profile. Every field the reader consumes is written from a named argument, so a test states +/// the stream it means (POC LSB width, extra slice header bits, output flag, colour planes) and +/// the expected POC follows from the spec, not from running the code under test. +enum HEVCSyntheticBitstream { + + /// MSB-first bit writer with Exp-Golomb. + struct BitWriter { + private(set) var bits: [UInt8] = [] + + mutating func u(_ value: Int, _ count: Int) { + for i in stride(from: count - 1, through: 0, by: -1) { bits.append(UInt8((value >> i) & 1)) } + } + mutating func flag(_ on: Bool) { bits.append(on ? 1 : 0) } + mutating func ue(_ value: Int) { + let v = value + 1 + let width = Int.bitWidth - v.leadingZeroBitCount + u(0, width - 1) + u(v, width) + } + /// The bytes written so far, padded with a stop bit and zeros to a byte boundary + /// (rbsp_trailing_bits), followed by `tail` verbatim. + func rbsp(tail: [UInt8] = []) -> [UInt8] { + var all = bits + all.append(1) + while all.count % 8 != 0 { all.append(0) } + var out: [UInt8] = [] + for i in stride(from: 0, to: all.count, by: 8) { + out.append(all[i.. [UInt8] { + var out: [UInt8] = [] + var zeros = 0 + for b in rbsp { + if zeros >= 2 && b <= 3 { out.append(3); zeros = 0 } + out.append(b) + zeros = b == 0 ? zeros + 1 : 0 + } + return out + } + + /// NAL unit: 2-byte header, then the escaped payload. + static func nal(type: Int, temporalId: Int = 0, rbsp: [UInt8]) -> [UInt8] { + [UInt8(type << 1), UInt8(temporalId + 1)] + escape(rbsp) + } + + static func sps(id: Int = 0, pocLsbBits: Int = 8, chromaFormatIdc: Int = 1, + separateColourPlane: Bool = false, subLayers: Int = 1) -> [UInt8] { + var w = BitWriter() + w.u(0, 4) // sps_video_parameter_set_id + w.u(subLayers - 1, 3) // sps_max_sub_layers_minus1 + w.flag(true) // temporal_id_nesting + w.u(0, 2); w.flag(false); w.u(2, 5) // profile_space, tier, profile_idc (Main 10) + w.u(0x20000000, 32) // compatibility flags + w.u(0, 48) // constraint flags: a long run of zero bytes, which is what + // makes emulation prevention show up in every SPS + w.u(120, 8) // general_level_idc + for _ in 0..<(subLayers - 1) { w.flag(false); w.flag(false) } // sub_layer profile/level absent + if subLayers > 1 { w.u(0, 2 * (8 - (subLayers - 1))) } + w.ue(id) + w.ue(chromaFormatIdc) + if chromaFormatIdc == 3 { w.flag(separateColourPlane) } + w.ue(3840); w.ue(2160) + w.flag(false) // conformance_window_flag + w.ue(2); w.ue(2) // bit depths (10) + w.ue(pocLsbBits - 4) // log2_max_pic_order_cnt_lsb_minus4 + return nal(type: 33, rbsp: w.rbsp(tail: [0x40])) + } + + static func pps(id: Int = 0, spsID: Int = 0, outputFlagPresent: Bool = false, + extraSliceHeaderBits: Int = 0) -> [UInt8] { + var w = BitWriter() + w.ue(id); w.ue(spsID) + w.flag(false) // dependent_slice_segments_enabled_flag + w.flag(outputFlagPresent) + w.u(extraSliceHeaderBits, 3) + return nal(type: 34, rbsp: w.rbsp(tail: [0x40])) + } + + /// First (or later) slice segment of a picture. `pocLsb` is ignored for IDR (19, 20), which + /// carry none. `tail` stands in for the rest of the header and the slice data. + static func slice(type: Int, temporalId: Int = 0, firstSliceSegment: Bool = true, + ppsID: Int = 0, extraBits: Int = 0, extraBitsValue: Int = 0, + outputFlag: Bool? = nil, colourPlane: Int? = nil, + pocLsb: Int = 0, pocLsbBits: Int = 8, afterPOC: (value: Int, bits: Int)? = nil, + tail: [UInt8] = [0xAA]) -> [UInt8] { + var w = BitWriter() + w.flag(firstSliceSegment) + if (16...23).contains(type) { w.flag(false) } // no_output_of_prior_pics_flag + w.ue(ppsID) + w.u(extraBitsValue, extraBits) // slice_reserved_flag[] + w.ue(2) // slice_type + if let outputFlag { w.flag(outputFlag) } + if let colourPlane { w.u(colourPlane, 2) } + if type != 19 && type != 20 { w.u(pocLsb, pocLsbBits) } + if let afterPOC { w.u(afterPOC.value, afterPOC.bits) } // whatever header field follows + return nal(type: type, temporalId: temporalId, rbsp: w.rbsp(tail: tail)) + } + + /// NAL type numbers used by the tests. + enum T { + static let trailN = 0, trailR = 1, tsaN = 2, radlN = 6, radlR = 7, raslN = 8, raslR = 9 + static let blaWLP = 16, blaWRADL = 17, blaNLP = 18, idrWRADL = 19, idrNLP = 20, cra = 21 + static let eos = 36, eob = 37 + } + + // MARK: - Access units for DualLayerPictureAlignment + + /// Length-prefixed (4 bytes) access unit. + static func accessUnit(_ nals: [[UInt8]]) -> [UInt8] { + nals.flatMap { n in + [UInt8((n.count >> 24) & 0xFF), UInt8((n.count >> 16) & 0xFF), UInt8((n.count >> 8) & 0xFF), UInt8(n.count & 0xFF)] + n + } + } + + /// An enhancement-layer NAL as it rides in the base-layer packet: wrapped in NAL type 63. + static func wrappedEL(_ inner: [UInt8]) -> [UInt8] { [UInt8(63 << 1), 0x01] + inner } +} diff --git a/Tests/AetherEngineTests/OrderedDeliveryTests.swift b/Tests/AetherEngineTests/OrderedDeliveryTests.swift new file mode 100644 index 000000000..03958ebc5 --- /dev/null +++ b/Tests/AetherEngineTests/OrderedDeliveryTests.swift @@ -0,0 +1,247 @@ +import Testing +import CoreMedia +import CoreVideo +import Foundation +@testable import AetherEngine + +/// `OrderedDelivery` is one serial `.userInteractive` queue, +/// owned by the decoder, that both composed and bypassed frames go through on +/// their way to `onFrame`. These tests drive it directly — no decoder, no +/// GPU — to pin its three load-bearing guarantees: ticket order survives any +/// arrival order, an epoch bump (`reset`) is what makes flush/close's "nothing +/// from before this returns after" invariant (I3) hold, and a ticket that +/// never arrives costs exactly the one frame at the stall-guard threshold, +/// not the rest of the session. +@Suite(.serialized, .timeLimit(.minutes(1))) +struct OrderedDeliveryTests { + + private static func makeBuffer() -> CVPixelBuffer? { + var out: CVPixelBuffer? + let status = CVPixelBufferCreate(kCFAllocatorDefault, 4, 4, + kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange, + [:] as CFDictionary, &out) + return status == kCVReturnSuccess ? out : nil + } + + /// Encodes an identifying number into a frame's PTS so a test can tell + /// which ticket a delivery corresponds to without `DeliveredFrame` + /// carrying the ticket itself (it deliberately does not — `emit` has no + /// use for it). + private static func frame(_ buffer: CVPixelBuffer, tag: Int64) -> DeliveredFrame { + DeliveredFrame(buffer: SendableBuffer(value: buffer), pts: CMTime(value: tag, timescale: 1), hdr10Data: nil, dmNAL: nil) + } + + private final class Recorder: @unchecked Sendable { + private let lock = NSLock() + private var tags: [Int64] = [] + func record(_ pts: CMTime) { lock.lock(); tags.append(pts.value); lock.unlock() } + var snapshot: [Int64] { lock.lock(); defer { lock.unlock() }; return tags } + var count: Int { lock.lock(); defer { lock.unlock() }; return tags.count } + } + + @Test("frames reach onFrame in ticket order no matter what order fulfil is called in") + func deliversInTicketOrder() async throws { + let buffer = try #require(Self.makeBuffer()) + let delivery = OrderedDelivery(maxFramesInFlight: 3, maxLayerSkew: 3, label: "test.ordered-delivery.order") + let recorder = Recorder() + delivery.setOnFrame { _, pts, _ in recorder.record(pts) } + + // Deliberately out of arrival order — this is what a composition that + // finishes on the GPU out of order, or a bypass that races ahead of a + // slower composition, looks like from `fulfil`'s point of view. + for ticket in [3, 1, 0, 4, 2] { + delivery.fulfil(ticket: UInt64(ticket), epoch: 0, Self.frame(buffer, tag: Int64(ticket))) + } + + try await waitFor { recorder.count == 5 } + #expect(recorder.snapshot == [0, 1, 2, 3, 4]) + } + + @Test("bypass-shaped and compose-shaped fulfil calls interleave through the same order, indistinguishably") + func bypassAndComposeShareOneOrder() async throws { + let buffer = try #require(Self.makeBuffer()) + let delivery = OrderedDelivery(maxFramesInFlight: 3, maxLayerSkew: 3, label: "test.ordered-delivery.mixed") + let recorder = Recorder() + delivery.setOnFrame { _, pts, _ in recorder.record(pts) } + + // A "bypass" is just a `fulfil` called synchronously on the pairing + // thread; a "compose" is the same call from Metal's completion queue, + // later. `OrderedDelivery` has no notion of which is which, so there is + // one ordering mechanism, not two. + delivery.fulfil(ticket: 0, epoch: 0, Self.frame(buffer, tag: 0)) // bypass, ticket 0 + delivery.fulfil(ticket: 2, epoch: 0, Self.frame(buffer, tag: 2)) // bypass, ticket 2, arrives immediately + // Both have been handled before the "compose" for ticket 1 finishes: ticket 2 is pending. + delivery.barrier() + let lateFrame = Self.frame(buffer, tag: 1) // built outside the closure: `DeliveredFrame` is Sendable, `CVPixelBuffer` is not + // A dedicated thread, not the global pool: blocking tests elsewhere in a full run can + // starve GCD's workers for seconds, which made this test time out rather than reorder. + Thread.detachNewThread { + // "Compose" for ticket 1 finishes late, off a different thread. + delivery.fulfil(ticket: 1, epoch: 0, lateFrame) + } + + try await waitFor { recorder.count == 3 } + #expect(recorder.snapshot == [0, 1, 2], "ticket 2 must wait behind the late ticket 1, whichever path produced it") + } + + @Test("a ticket that is never fulfilled costs exactly one frame (the stall guard)") + func stallGuardCostsExactlyOneFrame() async throws { + let buffer = try #require(Self.makeBuffer()) + // Threshold = maxFramesInFlight + maxLayerSkew + 2 = 3 + 3 + 2 = 8, + // matching production (`DoviFelComposer.maxFramesInFlight`, + // `DualLayerVideoDecoder.maxLayerSkewFrames`). + let delivery = OrderedDelivery(maxFramesInFlight: 3, maxLayerSkew: 3, label: "test.ordered-delivery.stall") + let recorder = Recorder() + delivery.setOnFrame { _, pts, _ in recorder.record(pts) } + + // Ticket 0 is never fulfilled — the composition it belonged to failed + // to call back at all, the one failure mode a ticket, rather than a + // bypass, cannot self-report. Tickets 1...12 arrive normally. + for ticket in 1...12 { + delivery.fulfil(ticket: UInt64(ticket), epoch: 0, Self.frame(buffer, tag: Int64(ticket))) + } + + try await waitFor { recorder.count == 12 } + #expect(recorder.count == 12, "every ticket but the missing one must still be delivered") + #expect(recorder.snapshot == Array(1...12), "delivery order is unaffected beyond the one skipped ticket") + } + + @Test("reset drops a stale-epoch delivery that is already pending when reset runs") + func resetDropsStaleEpoch() async throws { + let buffer = try #require(Self.makeBuffer()) + let delivery = OrderedDelivery(maxFramesInFlight: 3, maxLayerSkew: 3, label: "test.ordered-delivery.epoch") + let recorder = Recorder() + delivery.setOnFrame { _, pts, _ in recorder.record(pts) } + + // Ticket 0 never arrives (as if its base layer were still waiting for + // a partner when a seek lands), so tickets 1 and 2 — tagged with the + // pre-seek epoch — sit pending behind it. + delivery.fulfil(ticket: 1, epoch: 0, Self.frame(buffer, tag: 1)) + delivery.fulfil(ticket: 2, epoch: 0, Self.frame(buffer, tag: 2)) + delivery.barrier() // both are now in `pending` + + // The seek: bump the epoch and fast-forward past every ticket issued + // so far. `reset` is `queue.sync`, so this line does not return until + // both `fulfil` calls above have already run to completion (they + // deliver nothing, since ticket 0 is still missing and below the new + // stall threshold) — this is invariant I3's mechanism, exercised + // directly rather than through `DualLayerVideoDecoder.flush()`. + delivery.reset(epoch: 1, nextTicket: 10) + + // A composition for the OLD epoch retiring late — e.g. a GPU command + // buffer that was already committed before the seek — must never + // reach `onFrame`, even though its ticket number (11) is not below + // the new base ticket: only the epoch tells it from a live one. + delivery.fulfil(ticket: 11, epoch: 0, Self.frame(buffer, tag: 11)) + // A fresh, post-seek ticket at `nextTicket` with the NEW epoch must + // deliver normally. Ticket 11 would follow it if the epoch were ignored. + delivery.fulfil(ticket: 10, epoch: 1, Self.frame(buffer, tag: 10)) + + delivery.barrier() + #expect(recorder.snapshot == [10], "only the post-reset ticket may reach onFrame; tickets 1 and 2 predate the reset and ticket 11 carries the old epoch") + } + + @Test("a new-epoch ticket fulfilled before reset is (visibly) delivered is not lost") + func newEpochTicketFulfilledBeforeResetIsDeliveredIsNotLost() async throws { + // The sequence `fulfil(0...4, e0)`, `fulfil(5, e1)`, `reset(e1, 5)`, + // `fulfil(6...13, e1)` loses ticket 5 and stalls 6...13 until the + // stall guard skips it if `flush()` calls `reset` (synchronously, + // `queue.sync`) only after a concurrent caller could already have + // reached `fulfil` for the new epoch. + // + // `DualLayerVideoDecoder.flush()`/`close()` (see the comment atop + // `flush()`) therefore call `resetAsync` — the non-blocking + // half of `reset` — from INSIDE the same lock that makes issuing a + // new-epoch ticket possible in the first place, so the reset is + // always enqueued before any ticket that could race it. This test + // drives that exact ordering directly against `OrderedDelivery`, + // with no decoder involved: `resetAsync` is called first, and + // `fulfil(5, epoch: 1, ...)` is called immediately after — before + // anything has waited for `resetAsync`'s block to actually run, i.e. + // "fulfilled before reset is delivered" — and ticket 5 must still + // arrive, with nothing behind it held back either. + // + // Tickets 0...4 (epoch 0) are expected to drain and reach `onFrame` + // normally before any of this runs — they were enqueued strictly + // before `resetAsync`, so FIFO order alone (with no epoch mismatch + // to drop them) delivers them first. The property under + // test is only about ticket 5 onward. + let delivery = OrderedDelivery(maxFramesInFlight: 3, maxLayerSkew: 3, label: "test.ordered-delivery.post-flush-ticket") + let recorder = Recorder() + delivery.setOnFrame { _, pts, _ in recorder.record(pts) } + let buffer = try #require(Self.makeBuffer()) + + for ticket in 0...4 { + delivery.fulfil(ticket: UInt64(ticket), epoch: 0, Self.frame(buffer, tag: Int64(ticket))) + } + delivery.resetAsync(epoch: 1, nextTicket: 5) + delivery.fulfil(ticket: 5, epoch: 1, Self.frame(buffer, tag: 5)) + for ticket in 6...13 { + delivery.fulfil(ticket: UInt64(ticket), epoch: 1, Self.frame(buffer, tag: Int64(ticket))) + } + + try await waitFor { recorder.count == 14 } + #expect(recorder.snapshot == Array(0...13), "ticket 5 must not be lost, and 6...13 must not be held back waiting for it") + } + + @Test("exactly the stall threshold's worth of pending frames does not trigger the stall guard") + func stallGuardBoundaryDoesNotSkipAtExactlyTheThreshold() async throws { + let buffer = try #require(Self.makeBuffer()) + // Threshold = maxFramesInFlight + maxLayerSkew + 2 = 3 + 3 + 2 = 8. + // `drainReady`'s guard is `pending.count > stallThreshold` (strictly + // greater), so exactly 8 pending behind the missing head-of-line + // ticket must not be enough to skip it. + let delivery = OrderedDelivery(maxFramesInFlight: 3, maxLayerSkew: 3, label: "test.ordered-delivery.stall-boundary-no-skip") + let recorder = Recorder() + delivery.setOnFrame { _, pts, _ in recorder.record(pts) } + + for ticket in 1...8 { + delivery.fulfil(ticket: UInt64(ticket), epoch: 0, Self.frame(buffer, tag: Int64(ticket))) + } + // Every arrival has been handled, so ticket 0 would already have been (wrongly) skipped + // while draining 1...8; there is no ninth arrival to trigger it. + delivery.barrier() + #expect(recorder.count == 0, "nothing may deliver while ticket 0, the head of the line, is still missing") + + delivery.fulfil(ticket: 0, epoch: 0, Self.frame(buffer, tag: 0)) + try await waitFor { recorder.count == 9 } + #expect(recorder.snapshot == Array(0...8), "ticket 0 must not have been skipped at exactly 8 pending behind it") + } + + @Test("a ticket that arrives late, but before the stall threshold is exceeded, still delivers in order") + func lateTicketWithinThresholdDeliversInOrder() async throws { + let buffer = try #require(Self.makeBuffer()) + let delivery = OrderedDelivery(maxFramesInFlight: 3, maxLayerSkew: 3, label: "test.ordered-delivery.late-ticket-in-order") + let recorder = Recorder() + delivery.setOnFrame { _, pts, _ in recorder.record(pts) } + + // 7 pending behind the missing ticket 0 — one short of the 8-pending + // stall-guard threshold — then ticket 0 arrives late, well inside + // the window the guard tolerates. + for ticket in 1...7 { + delivery.fulfil(ticket: UInt64(ticket), epoch: 0, Self.frame(buffer, tag: Int64(ticket))) + } + delivery.barrier() + delivery.fulfil(ticket: 0, epoch: 0, Self.frame(buffer, tag: 0)) + + try await waitFor { recorder.count == 8 } + #expect(recorder.snapshot == Array(0...7), "a late ticket within the stall threshold must still deliver, in order") + } + + @Test("DEBUG: emit only ever runs on OrderedDelivery's own queue") + func emitRunsOnItsOwnQueue() async throws { + // `emit`'s `dispatchPrecondition(.onQueue(queue))` (DEBUG-only) traps + // if this is ever violated. Driving a normal delivery through to + // completion without crashing, from a suite that runs this alongside + // every other `fulfil` call above, is the running proof that it never + // fires — there is no path to `emit` except through `queue.async`/ + // `queue.sync` inside this class. + let buffer = try #require(Self.makeBuffer()) + let delivery = OrderedDelivery(maxFramesInFlight: 1, maxLayerSkew: 0, label: "test.ordered-delivery.onqueue") + let recorder = Recorder() + delivery.setOnFrame { _, pts, _ in recorder.record(pts) } + delivery.fulfil(ticket: 0, epoch: 0, Self.frame(buffer, tag: 0)) + try await waitFor { recorder.count == 1 } + #expect(recorder.snapshot == [0]) + } +} From bf5716ccc3b2e253893e1f66cbb24108f93f85a5 Mon Sep 17 00:00:00 2001 From: Pavel Kajaba <15980400+pkajaba@users.noreply.github.com> Date: Wed, 7 Oct 2026 10:25:09 +0100 Subject: [PATCH 010/132] fix(dovi): refuse hostile RPUs before libdovi can abort the process libdovi is built panic=abort, so a single panic ends the host process. A CRC-valid but hostile Dolby Vision RPU could reach one on the native Profile 7 -> 8.1 conversion. Adds DoviRpuGuard: it checks the RPU CRC and pre-scans the payload for the inputs that make libdovi 3.4 panic or allocate without bound (linear interpolation pieces, polynomial/MMR mixes that break the writer, pivot and block counts the payload cannot hold, undefined display-management block lengths). The existing paths now check each RPU with it before parsing: the Profile 8.1 conversion in DoviRpuConverter (convertPacketToProfile81), the enhancement-layer-type probe, and the DolbyVisionRecordAudit RPU read. The mode 2 convert -> write pair on a parsed RPU is factored into DoviRpuConverter.convertedUnspec62 so the tests drive production's own entry point. A rejected RPU degrades the frame to the HDR10 base exactly as a libdovi failure already did (#135). This hardens the existing native 8.1 conversion; it changes no routing and adds no Dolby Vision FEL code. The guard knows the paths listed above; it does not prove there are no others. Tests: DoviRpuGuardTests (crafted RPUs, verdicts, conversion round trip) and a DolbyVisionRecordAudit case. The CRC-valid mutation fuzz and the real-file guard test arrive with the composer, whose parameter walk they also exercise. --- CHANGELOG.md | 1 + .../Video/DolbyVisionRecordAudit.swift | 3 +- .../AetherEngine/Video/DoviRpuConverter.swift | 17 +- Sources/AetherEngine/Video/DoviRpuGuard.swift | 252 +++++++++ .../DolbyVisionRecordAuditTests.swift | 8 + .../DoviRpuConverterTests.swift | 57 +- .../AetherEngineTests/DoviRpuGuardTests.swift | 531 ++++++++++++++++++ 7 files changed, 863 insertions(+), 6 deletions(-) create mode 100644 Sources/AetherEngine/Video/DoviRpuGuard.swift create mode 100644 Tests/AetherEngineTests/DoviRpuGuardTests.swift diff --git a/CHANGELOG.md b/CHANGELOG.md index c97a6da1d..189c9f9fc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -436,6 +436,7 @@ the public-API contract. - **A play or pause pressed during any session rebuild decides how it comes back.** An audio or disc-title switch, `reloadAtCurrentPosition()` or an engine-raised rebuild came back in the transport state it started from, so a pause pressed while it ran was lost. `play()` and `pause()` now write the transport the rebuild settles from, `transportIntentUnderReconstruction` (#464 round 3), while one runs. #711 had added this for audio picks in a field of its own; the two are one now. - **Video routing asks for the AV1 hardware decoder only for AV1 sources (#708).** It registered the supplemental decoder on every first load, H.264 and HEVC included. Contributed by yucelokan. - **The #243 fetch-tally test tolerates a parallel session start.** A session start zeroes the process-wide tally, so an engine test loading in parallel failed it on CI. Test-only. +- **A CRC-valid but hostile Dolby Vision RPU no longer aborts the process on any path found so far, in parse, convert or write.** libdovi is built panic=abort, so one panic ends the process. `DoviRpuGuard` checks the CRC and pre-scans each RPU for the inputs that make libdovi 3.4 panic or allocate unbounded: a polynomial piece with linear interpolation (`unimplemented!` in the parser and the writer), a reshaping curve that mixes polynomial and MMR pieces (it parses, and a MEL or residual-free RPU keeps its mapping through the mode 2 conversion, so `dovi_write_unspec62_nalu` then indexes out of bounds), pivot and block counts the payload cannot hold, and display-management block lengths the spec does not define (`unreachable!`). It also refuses an Exp-Golomb value of 2^32 - 1 or more anywhere it reads one (libdovi reads 64 leading zeros differently from a plain shift, so a crafted `mapping_idc` could slip past, and its writer allocates about 0.5 GB to write such an id back). The guard knows these paths; it does not prove there are no others. ## [7.28.3] - 2026-10-07 diff --git a/Sources/AetherEngine/Video/DolbyVisionRecordAudit.swift b/Sources/AetherEngine/Video/DolbyVisionRecordAudit.swift index bbfbabc50..386de2e58 100644 --- a/Sources/AetherEngine/Video/DolbyVisionRecordAudit.swift +++ b/Sources/AetherEngine/Video/DolbyVisionRecordAudit.swift @@ -31,7 +31,8 @@ public enum DolbyVisionRecordAudit { var result: Int? A53SEIParser.forEachNAL(data, size, framing) { nal, len in guard result == nil, (nal[0] >> 1) & 0x3F == nalTypeRPU else { return } - guard let rpu = dovi_parse_unspec62_nalu(nal, len) else { return } + // Behind the CRC and panic-path checks: libdovi aborts the process on some malformed RPUs. + guard let rpu = DoviRpuGuard.parse(nal, len) else { return } defer { dovi_rpu_free(rpu) } guard let hdr = dovi_rpu_get_header(rpu) else { return } defer { dovi_rpu_free_header(hdr) } diff --git a/Sources/AetherEngine/Video/DoviRpuConverter.swift b/Sources/AetherEngine/Video/DoviRpuConverter.swift index 2dd68515e..a99b55b8c 100644 --- a/Sources/AetherEngine/Video/DoviRpuConverter.swift +++ b/Sources/AetherEngine/Video/DoviRpuConverter.swift @@ -20,6 +20,15 @@ public enum DoviRpuConverter { } } + /// The Profile 8.1 form of a parsed RPU (`dovi_convert_rpu_with_mode`, mode 2) written back as an + /// unspec62 NAL unit, or nil when libdovi refuses either step. The caller frees the result with + /// `dovi_data_free`. This is the pair of calls the library can abort in, so everything that feeds it + /// goes through `DoviRpuGuard.parse` first, and the tests drive it through this one entry point. + static func convertedUnspec62(_ rpu: OpaquePointer) -> UnsafePointer? { + guard dovi_convert_rpu_with_mode(rpu, 2) == 0 else { return nil } + return dovi_write_unspec62_nalu(rpu) + } + /// Returns `false` when a libdovi conversion fails (or on an internal allocation failure). /// On a libdovi failure the offending RPU (and EL) are dropped so the frame degrades to the /// clean HDR10 base, rather than shipping a P7 RPU inside a container already declared 8.1 @@ -63,13 +72,12 @@ public enum DoviRpuConverter { case nalTypeRPU: // Any libdovi failure drops the RPU (degraded) instead of muxing a stale P7 RPU // inside a container already declared 8.1 (mixed-profile hazard, #135). - guard let rpu = dovi_parse_unspec62_nalu(nal, len) else { + guard let rpu = DoviRpuGuard.parse(nal, len) else { degraded = true return } defer { dovi_rpu_free(rpu) } - guard dovi_convert_rpu_with_mode(rpu, 2) == 0, - let out = dovi_write_unspec62_nalu(rpu) else { + guard let out = Self.convertedUnspec62(rpu) else { degraded = true return } @@ -153,7 +161,8 @@ public enum DoviRpuConverter { var result: String? A53SEIParser.forEachNAL(data, size, framing) { nal, len in guard result == nil, (nal[0] >> 1) & 0x3F == nalTypeRPU else { return } - guard let rpu = dovi_parse_unspec62_nalu(nal, len) else { return } + // Behind the CRC and panic-path checks: libdovi aborts the process on some malformed RPUs. + guard let rpu = DoviRpuGuard.parse(nal, len) else { return } defer { dovi_rpu_free(rpu) } guard let hdr = dovi_rpu_get_header(rpu) else { return } defer { dovi_rpu_free_header(hdr) } diff --git a/Sources/AetherEngine/Video/DoviRpuGuard.swift b/Sources/AetherEngine/Video/DoviRpuGuard.swift new file mode 100644 index 000000000..76fecb89b --- /dev/null +++ b/Sources/AetherEngine/Video/DoviRpuGuard.swift @@ -0,0 +1,252 @@ +import Foundation +import Dovi + +/// libdovi is built panic=abort and parses before it checks the CRC, so a corrupt RPU that +/// reaches an `unimplemented!()` / `unreachable!()` branch (libdovi 3.4 has no `todo!()` on these +/// paths) kills the process. Every RPU is CRC-checked here first; a +/// payload with a bad CRC never reaches the Rust parser. A crafted RPU with a valid CRC can still +/// reach a panic in the parser or, after `dovi_convert_rpu_with_mode`, in the writer; +/// `reachesUnimplementedPath` refuses the ones that are known (see there). +enum DoviRpuGuard { + + private static let crcTable: [UInt32] = (0..<256).map { i in + var c = UInt32(i) << 24 + for _ in 0..<8 { c = c & 0x8000_0000 != 0 ? (c << 1) ^ 0x04C1_1DB7 : c << 1 } + return c + } + + /// `nal` starts at the 2-byte HEVC NAL header (type 62), emulation prevention still in place. + /// Returns the unescaped RPU (header included) when its CRC32 is right, else nil. + /// Layout after unescaping: header(2) 0x19 payload... crc32(4) 0x80. + static func verifiedUnescaped(_ nal: UnsafePointer, _ len: Int) -> [UInt8]? { + guard len >= 8 else { return nil } + var u = [UInt8]() + u.reserveCapacity(len) + var zeros = 0 + for i in 0..= 2 && b == 3 { zeros = 0; continue } + u.append(b) + zeros = b == 0 ? zeros + 1 : 0 + } + while let last = u.last, last == 0 { u.removeLast() } // cabac_zero_words / trailing zeros + guard u.count >= 8, u[2] == 0x19, u[u.count - 1] == 0x80 else { return nil } + let tailAt = u.count - 5 + var crc: UInt32 = 0xFFFF_FFFF + for i in 3..> 24) ^ UInt32(u[i]))] } + let stored = (UInt32(u[tailAt]) << 24) | (UInt32(u[tailAt + 1]) << 16) + | (UInt32(u[tailAt + 2]) << 8) | UInt32(u[tailAt + 3]) + return crc == stored ? u : nil + } + + static func crcIsValid(_ nal: UnsafePointer, _ len: Int) -> Bool { + verifiedUnescaped(nal, len) != nil + } + + /// MSB-first bit reader. Every read returns nil past the end instead of trapping. + private struct Bits { + let bytes: [UInt8] + let end: Int // in bits + var pos: Int + + mutating func bit() -> UInt64? { + guard pos < end else { return nil } + let v = (bytes[pos >> 3] >> UInt8(7 - (pos & 7))) & 1 + pos += 1 + return UInt64(v) + } + mutating func bits(_ n: Int) -> UInt64? { + guard n >= 0, n <= 64, pos + n <= end else { return nil } + var v: UInt64 = 0 + for _ in 0.. UInt64? { + var zeros = 0 + while true { + guard let b = bit() else { return nil } // out of bits: libdovi errors on the same read + if b == 1 { break } + zeros += 1 + if zeros >= Self.hostileLeadingZeros { hostile = true; return nil } + } + guard let rest = bits(zeros) else { return nil } + return ((UInt64(1) << UInt64(zeros)) - 1) + rest // zeros < 32: no overflow + } + mutating func skipSE() -> Bool { ue() != nil } + } + + /// True when the RPU steers libdovi 3.4 into a panic, which aborts the process because the library is + /// built panic=abort. Mirrors `RpuDataHeader::parse`, `RpuDataMapping::parse` and the DM payload far + /// enough to see these, each of which parses and survives to the writer or aborts in the parser: + /// + /// - a polynomial piece with `poly_order_minus1 == 0 && linear_interp_flag` (`unimplemented!()` in + /// both the parser and the writer); + /// - a curve whose pieces do not all share one `mapping_idc`. The parser accepts the mix and keeps + /// one coefficient list per method, but the writer emits the curve's last `mapping_idc` for every + /// piece and indexes the polynomial list per piece, which is shorter than the piece count: + /// an out-of-bounds panic in `dovi_write_unspec62_nalu`. Mode 2 conversion drops the mapping only + /// for an FEL stream, so a MEL or residual-free RPU reaches the writer with the mix intact; + /// - a pivot or block count the payload cannot hold (an allocation abort), and a level 8, 9 or 10 + /// block of a length the spec does not define (`unreachable!()` in `required_bits`). + /// + /// An Exp-Golomb code with 32 or more leading zeros anywhere the scan reads one is hostile (see + /// `Bits.hostile`) and answers true. + /// + /// Otherwise conservative, and every `false` is on a path where libdovi itself returns an error + /// before reaching any panic: running out of bits (libdovi's reads fail on the same field, the + /// scan mirrors its read order), a value libdovi's header or mapping parser rejects with `ensure!` + /// or `bail!` (`rpu_type`, `coefficient_data_type`, the bit depths and denominator checked by + /// `RpuDataHeader::validate`, `mapping_idc` other than 0 or 1, `poly_order_minus1` above 1, + /// `mmr_order_minus1` above 2, a non-zero `nlq_method_idc`), or a DM block level the CM version + /// does not allow. + static func reachesUnimplementedPath(_ u: [UInt8]) -> Bool { + // libdovi's reader runs to the end of the trimmed data, CRC and final byte included, so a mutated + // piece count can make it read the linear flag out of the CRC bytes. Mirror that bound. + var r = Bits(bytes: u, end: u.count * 8, pos: 3 * 8) // after NAL header + 0x19 + return scan(&r) || r.hostile + } + + private static func scan(_ r: inout Bits) -> Bool { + guard r.bits(6) == 2, let format = r.bits(11) else { return false } + guard r.bits(4) != nil, r.bits(4) != nil, let seqInfo = r.bit() else { return false } + guard seqInfo == 1 else { return false } + guard r.bit() != nil, let dataType = r.bits(2) else { return false } + var denom: UInt64 = 0 + if dataType == 0 { guard let d = r.ue() else { return false }; denom = d } + guard r.bits(2) != nil, r.bit() != nil else { return false } + var blDepth: UInt64 = 0, elDepth: UInt64 = 0, reserved3: UInt64 = 0 + var disableResidual = false + if format & 0x700 == 0 { + guard let bl = r.ue(), let el = r.ue(), r.ue() != nil, r.bit() != nil, let res = r.bits(3), + r.bit() != nil, let dr = r.bit() else { return false } + blDepth = bl + elDepth = el & 0xFF + reserved3 = res + disableResidual = dr == 1 + } + let coefLen: Int + if dataType == 0 { guard denom <= 32 else { return false }; coefLen = Int(denom) } + else if dataType == 1 { coefLen = 32 } else { return false } + // libdovi rejects anything but 10-bit base layers and denom > 23 before reading the mapping. + guard blDepth == 2, denom <= 23 else { return false } + guard let dmPresent = r.bit(), let usePrev = r.bit() else { return false } + if usePrev == 1 { + // libdovi reads prev_vdr_rpu_id and skips only the mapping (curves and NLQ); the DM payload + // is still parsed when vdr_dm_metadata_present_flag is set, so an invalid block must be + // caught here too. + guard r.ue() != nil, dmPresent == 1 else { return false } + return reachesInvalidDmBlock(&r, compressed: reserved3 == 1) + } + guard r.ue() != nil, r.ue() != nil, r.ue() != nil else { return false } + let depth = Int(blDepth) + 8 + var pieces = [Int](repeating: 0, count: 3) + for c in 0..<3 { + guard let n = r.ue() else { return false } + // libdovi sizes its pivot and piece vectors from this count before reading them, so a count the + // payload cannot hold aborts on allocation instead of failing: reject it here. + if n > UInt64(r.end - r.pos) / UInt64(depth) { return true } + pieces[c] = Int(n) + 1 + for _ in 0..<(Int(n) + 2) { guard r.bits(depth) != nil else { return false } } + } + if format & 0x700 == 0 && !disableResidual { + guard r.bits(3) == 0 else { return false } + for _ in 0..<2 { guard r.bits(depth) != nil else { return false } } // NLQ_NUM_PIVOTS == 2 (guarded by the bounds above) + } + guard r.ue() != nil, r.ue() != nil else { return false } + for c in 0..<3 { + var curveIdc: UInt64? + for _ in 0.. Bool { + guard r.ue() != nil, r.ue() != nil, r.ue() != nil else { return false } + if !compressed { // fixed fields of VdrDmData::parse + guard r.pos + 509 <= r.end else { return false } + r.pos += 509 + } // fixed fields of VdrDmData::parse + for section in 0..<2 { + if section == 1 && r.end - r.pos < 56 { return false } // DM_DATA_PAYLOAD2_MIN_BITS + guard let count = r.ue() else { return false } + if count > UInt64(r.end - r.pos) / 9 { return true } // allocated up front by libdovi, like the pivot count + while r.pos & 7 != 0 { guard r.bit() != nil else { return false } } + for _ in 0.., _ len: Int) -> OpaquePointer? { + guard let u = verifiedUnescaped(nal, len), !reachesUnimplementedPath(u) else { return nil } + return dovi_parse_unspec62_nalu(nal, len) + } +} diff --git a/Tests/AetherEngineTests/DolbyVisionRecordAuditTests.swift b/Tests/AetherEngineTests/DolbyVisionRecordAuditTests.swift index 2d7afba61..55e88dce7 100644 --- a/Tests/AetherEngineTests/DolbyVisionRecordAuditTests.swift +++ b/Tests/AetherEngineTests/DolbyVisionRecordAuditTests.swift @@ -101,6 +101,14 @@ struct DolbyVisionRecordAuditTests { #expect(DolbyVisionRecordAudit.rpuProfile(pkt) == nil) } + @Test("an RPU the guard rejects is never handed to libdovi, which would abort the process") + func rpuThatWouldAbortLibdoviSaysNothing() { + let rpu = DoviRpuGuardTests.craft(linearOnComponent: 1) + let pkt = Self.avccPacket([Self.hevcNAL(type: 1, payload: [0xAA]), rpu]) + defer { Self.free(pkt) } + #expect(DolbyVisionRecordAudit.rpuProfile(pkt) == nil) + } + // MARK: - When the RPU is worth reading @Test("a Profile 5 record over a BT.2020 YCbCr PQ VUI is worth auditing") diff --git a/Tests/AetherEngineTests/DoviRpuConverterTests.swift b/Tests/AetherEngineTests/DoviRpuConverterTests.swift index 2942a60d4..7e33a1789 100644 --- a/Tests/AetherEngineTests/DoviRpuConverterTests.swift +++ b/Tests/AetherEngineTests/DoviRpuConverterTests.swift @@ -71,6 +71,61 @@ struct DoviRpuConverterTests { #expect(nalTypes(pkt) == [1]) } + /// CRC-valid RPUs that reach a panic or an unbounded allocation in libdovi, and one with a wrong CRC. + /// Handing any of them to libdovi aborts the test process, so passing is the point. + typealias Hostile = (name: String, rpu: [UInt8]) + static let hostileRPUs: [Hostile] = { + typealias G = DoviRpuGuardTests + let poly: [[G.Piece]] = [[.polynomial], [.polynomial], [.polynomial]] + return [ + ("linear interpolation", G.craft(linearOnComponent: 1)), + ("polynomial and MMR mix, no residual", G.craft(pieces: [[.polynomial, .mmr], [.polynomial], [.polynomial]])), + ("polynomial and MMR mix, MEL", G.craft(pieces: [[.mmr, .polynomial], [.polynomial], [.polynomial]], residual: .mel)), + ("polynomial and MMR mix, FEL", G.craft(pieces: [[.polynomial], [.polynomial], [.polynomial, .mmr]], residual: .fel)), + ("pivot count beyond the payload", G.craft(pieces: poly, pivotCount: (UInt64(1) << 32) - 2)), + ("mapping_idc with 64 leading zeros", G.craft(pieces: [[.linear], [.polynomial], [.polynomial]], + firstMappingIdc: { $0.ueRaw(zeros: 64, rest: 0) })), + ("vdr_rpu_id with 40 leading zeros", G.craft(pieces: poly, vdrRpuId: { $0.ueRaw(zeros: 40, rest: G.maxRest(40)) })), + ("undefined level 8 block length", G.craftUsePrev(cmv40Block: (11, 8))), + ("bad CRC", G.withBadCRC(G.craft(linearOnComponent: nil), crcByte: 1)), + ] + }() + + @Test("A hostile or corrupt RPU is refused on the real conversion paths: HDR10 fallback, no abort", + arguments: Array(DoviRpuConverterTests.hostileRPUs.indices), [VideoNALFraming.lengthPrefixed(size: 4), .annexB]) + func hostileRPUDegradesOnEveryEntryPoint(index: Int, framing: VideoNALFraming) { + let (name, rpu) = Self.hostileRPUs[index] + let nals = [hevcNAL(type: 1, payload: [0xAA, 0xBB]), rpu, hevcNAL(type: 63, payload: [0xCC])] + let bytes = framed(nals, framing) + + // The probes read the packet and must come back empty-handed. + let probe = packet(from: bytes) + defer { free(probe) } + #expect(DoviRpuConverter.enhancementLayerType(probe, framing: framing) == nil, Comment(rawValue: name)) + #expect(DolbyVisionRecordAudit.rpuProfile(probe, framing: framing) == nil, Comment(rawValue: name)) + + // The conversion reports the failure and drops the RPU and the EL, keeping the base layer. + let pkt = packet(from: bytes) + defer { free(pkt) } + #expect(DoviRpuConverter.convertPacketToProfile81(pkt, framing: framing) == false, Comment(rawValue: name)) + let expected = framed([nals[0]], framing) + #expect(packetBytes(pkt) == expected, Comment(rawValue: name)) + } + + @Test("A well-formed crafted RPU takes the same paths and converts, so the refusals above are the guard's") + func craftedRPUConvertsOnTheSamePaths() { + let rpu = DoviRpuGuardTests.craft(pieces: [[.polynomial], [.mmr], [.mmr]], residual: .fel) + let bytes = framed([hevcNAL(type: 1, payload: [0xAA, 0xBB]), rpu, hevcNAL(type: 63, payload: [0xCC])], + .lengthPrefixed(size: 4)) + let probe = packet(from: bytes) + defer { free(probe) } + #expect(DoviRpuConverter.enhancementLayerType(probe) == "FEL") + let pkt = packet(from: bytes) + defer { free(pkt) } + #expect(DoviRpuConverter.convertPacketToProfile81(pkt) == true) + #expect(nalTypes(pkt) == [1, 62]) + } + @Test("A non-DV packet is left untouched") func leavesNonDVUntouched() { let bl = hevcNAL(type: 1, payload: [0xAA, 0xBB]) @@ -258,7 +313,7 @@ struct DoviRpuConverterTests { A53SEIParser.forEachNAL(buf.baseAddress!, buf.count, framing) { nal, len in switch (nal[0] >> 1) & 0x3F { case 62: - guard let rpu = dovi_parse_unspec62_nalu(nal, len) else { degraded = true; return } + guard let rpu = DoviRpuGuard.parse(nal, len) else { degraded = true; return } // as production: the guard first, libdovi is panic=abort if dovi_convert_rpu_with_mode(rpu, 2) != 0 { dovi_rpu_free(rpu); degraded = true; return } guard let out = dovi_write_unspec62_nalu(rpu) else { dovi_rpu_free(rpu); degraded = true; return } guard let data = out.pointee.data, out.pointee.len > 0 else { diff --git a/Tests/AetherEngineTests/DoviRpuGuardTests.swift b/Tests/AetherEngineTests/DoviRpuGuardTests.swift new file mode 100644 index 000000000..2f5616a7d --- /dev/null +++ b/Tests/AetherEngineTests/DoviRpuGuardTests.swift @@ -0,0 +1,531 @@ +import Testing +import Foundation +import Dovi +@testable import AetherEngine + +/// libdovi aborts the process on a polynomial piece with linear interpolation. The guard must +/// see that coming for an RPU whose CRC is perfectly valid, and must not flag real ones. +struct DoviRpuGuardTests { + + struct BitWriter { + var bits: [Bool] = [] + mutating func put(_ v: UInt64, _ n: Int) { for i in stride(from: n - 1, through: 0, by: -1) { bits.append((v >> UInt64(i)) & 1 == 1) } } + mutating func ue(_ v: UInt64) { + let x = v + 1 + let len = 64 - x.leadingZeroBitCount + put(0, len - 1); put(x, len) + } + mutating func se(_ v: Int64) { ue(v > 0 ? UInt64(2 * v - 1) : UInt64(-2 * v)) } + /// An Exp-Golomb code with exactly `zeros` leading zeros and `rest` as its suffix, however large + /// the value that makes (`ue` cannot write above 2^64 - 2). libdovi reads it as `rest + 2^zeros - 1`, + /// wrapping: its release build masks a shift by 64, so 64 zeros read as just `rest`. + mutating func ueRaw(zeros: Int, rest: UInt64) { + put(0, zeros); put(1, 1); put(rest, zeros) + } + var bytes: [UInt8] { + var out = [UInt8](repeating: 0, count: (bits.count + 7) / 8) + for (i, b) in bits.enumerated() where b { out[i >> 3] |= UInt8(0x80 >> (i & 7)) } + return out + } + } + + static func crc(_ data: ArraySlice) -> UInt32 { + var c: UInt32 = 0xFFFF_FFFF + for byte in data { + c ^= UInt32(byte) << 24 + for _ in 0..<8 { c = c & 0x8000_0000 != 0 ? (c << 1) ^ 0x04C1_1DB7 : c << 1 } + } + return c + } + + enum Piece { + case polynomial, mmr + /// A polynomial piece with `linear_interp_flag` set, which libdovi 3.4 cannot represent. + case linear + } + + enum Residual { + /// Profile 8 shape: `disable_residual_flag` set, no NLQ. + case none + /// Profile 7 with the minimal enhancement layer's NLQ values. + case mel + /// Profile 7 with non-trivial NLQ values. + case fel + } + + /// A structurally valid RPU with a valid CRC, one entry of `pieces` per component (luma, Cb, Cr), + /// each listing that curve's pieces in order. No DM metadata, so the mapping is all there is to parse + /// and to write back. Residual-free by default (profile 8 shape); `residual` makes it a Profile 7 RPU. + /// One display-management payload: the ids, the optional fixed fields, then the CM v2.9 list and the + /// CM v4.0 list. Each list holds at most one block (zero payload bytes) followed, in the v4.0 list, by + /// the mandatory level 254 block. + struct DMPayload { + /// `reserved_zero_3bits == 1`: only the three ids precede the block lists. Otherwise the 509-bit + /// fixed field set of `VdrDmData::parse` follows them. + var compressed = true + var currentDmMetadataId: ((inout BitWriter) -> Void)? + /// The declared CM v2.9 block count, used only when `cmv29Block` is nil (no block follows it). + var cmv29Count: UInt64 = 0 + var cmv29Block: (length: UInt64, level: UInt64)? + var cmv40Block: (length: UInt64, level: UInt64)? + } + + static func writeDM(_ w: inout BitWriter, _ dm: DMPayload) { + w.ue(0) // affected_dm_metadata_id + if let id = dm.currentDmMetadataId { id(&w) } else { w.ue(0) } + w.ue(0) // scene_refresh_flag + if !dm.compressed { // 509 bits, valid per libdovi's validate + for _ in 0..<9 { w.put(0, 16) } // ycc_to_rgb_coef + for _ in 0..<3 { w.put(0, 32) } // ycc_to_rgb_offset + for _ in 0..<9 { w.put(0, 16) } // rgb_to_lms_coef + w.put(65535, 16); w.put(0, 16); w.put(0, 16); w.put(0, 32) // signal_eotf (65535 with no params) + w.put(10, 5); w.put(0, 2); w.put(0, 2); w.put(0, 2) // depth, color space, chroma, range + w.put(0, 12); w.put(0, 12); w.put(0, 10) // min/max pq, diagonal + } + if let block = dm.cmv29Block { + w.ue(1) + while w.bits.count % 8 != 0 { w.bits.append(false) } + w.ue(block.length); w.put(block.level, 8) + for _ in 0.. [UInt8] { + var body: [UInt8] = [0x19] + payload + let crc = crc(body[1...]) + body += [UInt8(crc >> 24), UInt8(crc >> 16 & 0xFF), UInt8(crc >> 8 & 0xFF), UInt8(crc & 0xFF), 0x80] + return escape(body) + } + + /// `body` is 0x19, the payload, the CRC and the final byte; the NAL header, emulation prevention and + /// the padding are added. + static func escape(_ body: [UInt8]) -> [UInt8] { + var out: [UInt8] = [0x7C, 0x01] + var zeros = 0 + for b in body { + if zeros >= 2 && b <= 3 { out.append(3); zeros = 0 } + out.append(b) + zeros = b == 0 ? zeros + 1 : 0 + } + // libdovi refuses an input under 25 bytes before it looks at anything else; trailing zero bytes + // are trimmed by both it and the guard, so they pad a short RPU without changing it. + while out.count < 26 { out.append(0) } + return out + } + + /// `coefficientDataType` 1 (32-bit coefficients, no integer part, no denominator) is residual-free only. + /// `pivotCount` replaces luma's `num_pivots_minus2` (the pivots written stay the real ones), + /// `firstMappingIdc` the encoding of luma's first `mapping_idc`, `vdrRpuId` the mapping's id and `dm` + /// appends a display-management payload after the mapping and NLQ. + static func craft(pieces: [[Piece]], residual: Residual = .none, coefficientDataType: Int = 0, + pivotCount: UInt64? = nil, firstMappingIdc: ((inout BitWriter) -> Void)? = nil, + vdrRpuId: ((inout BitWriter) -> Void)? = nil, dm: DMPayload? = nil) -> [UInt8] { + precondition(coefficientDataType == 0 || residual == .none) + var w = BitWriter() + w.put(2, 6); w.put(18, 11) // rpu_type, rpu_format (libdovi checks the 0x08 0x09 start) + w.put(1, 4); w.put(0, 4) // vdr_rpu_profile 1, level 0 + w.put(1, 1) // vdr_seq_info_present + w.put(0, 1); w.put(UInt64(coefficientDataType), 2) // chroma filter flag, coefficient_data_type + if coefficientDataType == 0 { w.ue(23) } // denom + w.put(0, 2); w.put(0, 1) // normalized idc, full range + w.ue(2); w.ue(2); w.ue(4) // bl, el, vdr depth - 8 + w.put(0, 1) // spatial resampling flag + w.put(dm?.compressed == true ? 1 : 0, 3) // reserved_zero_3bits, 1 selects compressed DM data + w.put(residual == .none ? 0 : 1, 1) // el_spatial_resampling_filter_flag (with the residual: profile 7) + w.put(residual == .none ? 1 : 0, 1) // disable_residual + w.put(dm == nil ? 0 : 1, 1) // vdr_dm_metadata_present + w.put(0, 1) // use_prev_vdr_rpu + if let vdrRpuId { vdrRpuId(&w) } else { w.ue(0) } + w.ue(0); w.ue(0) // color space, chroma format + for (c, curve) in pieces.enumerated() { + if c == 0, let pivotCount { w.ue(pivotCount) } else { w.ue(UInt64(curve.count) - 1) } // num_pivots_minus2 + for k in 0...curve.count { w.put(UInt64(k * 1023 / curve.count), 10) } + } + if residual != .none { w.put(0, 3); w.put(0, 10); w.put(1023, 10) } // NLQ method, pivots + w.ue(0); w.ue(0) // partitions + /// One coefficient: signed integer part and 23 fraction bits, or (type 1) 32 plain bits. + func coef(_ w: inout BitWriter, _ int: Int64) { + if coefficientDataType == 0 { w.se(int); w.put(0, 23) } else { w.put(0, 32) } + } + for (c, curve) in pieces.enumerated() { + for (k, piece) in curve.enumerated() { + let idc = { (w: inout BitWriter, v: UInt64) in + if c == 0 && k == 0, let firstMappingIdc { firstMappingIdc(&w) } else { w.ue(v) } + } + switch piece { + case .mmr: + idc(&w, 1); w.put(0, 2) // MMR, order 1 + coef(&w, 0) + for _ in 0..<7 { coef(&w, 0) } + case .linear: + idc(&w, 0); w.ue(0); w.put(1, 1) + case .polynomial: + idc(&w, 0) // polynomial + w.ue(0) // poly_order_minus1 + w.put(0, 1) // linear_interp_flag + for k in 0..<2 { coef(&w, k == 1 ? 1 : 0) } + } + } + } + if residual != .none { + for c in 0..<3 { // NLQ parameters; all zero but vdr_in_max_int is the MEL signature + let fel = residual == .fel + w.put(fel ? UInt64(40 + 8 * c) : 0, 10) + w.ue(fel ? 0 : 1); w.put(fel ? 1 << 20 : 0, 23) + w.ue(0); w.put(fel ? 3 << 18 : 0, 23) + w.ue(0); w.put(fel ? 1 << 19 : 0, 23) + } + } + if let dm { writeDM(&w, dm) } + while w.bits.count % 8 != 0 { w.bits.append(false) } + return seal(w.bytes) + } + + /// One piece per component, polynomial unless named: the shape most tests here build. + static func craft(linearOnComponent: Int?, mmrOnComponent: Int? = nil) -> [UInt8] { + craft(pieces: (0..<3).map { c in + [c == linearOnComponent ? Piece.linear : c == mmrOnComponent ? .mmr : .polynomial] + }) + } + + /// `nal` with one bit of its stored CRC32 flipped (`crcByte` 0 is the first CRC byte) or, for + /// `payloadByte`, one bit of the payload flipped under the CRC that was computed for the original. + static func withBadCRC(_ nal: [UInt8], crcByte: Int? = nil, payloadByte: Int? = nil) -> [UInt8] { + var u: [UInt8] = [] + var zeros = 0 + for b in nal { + if zeros >= 2 && b == 3 { zeros = 0; continue } + u.append(b) + zeros = b == 0 ? zeros + 1 : 0 + } + while u.last == 0 { u.removeLast() } + var body = Array(u.dropFirst(2)) // 0x19, payload, crc32(4), 0x80 + if let crcByte { body[body.count - 5 + crcByte] ^= 0x01 } + if let payloadByte { body[1 + payloadByte] ^= 0x10 } + return escape(body) + } + + static func verdict(_ nal: [UInt8]) -> (crc: Bool, flagged: Bool) { + nal.withUnsafeBufferPointer { p in + guard let u = DoviRpuGuard.verifiedUnescaped(p.baseAddress!, p.count) else { return (false, false) } + return (true, DoviRpuGuard.reachesUnimplementedPath(u)) + } + } + + @Test("a crafted structurally valid RPU parses in libdovi, so the scanner is tracking a real layout") + func craftedBaselineParses() { + let nal = Self.craft(linearOnComponent: nil) + #expect(Self.verdict(nal) == (true, false)) + let rpu = nal.withUnsafeBufferPointer { DoviRpuGuard.parse($0.baseAddress!, $0.count) } + #expect(rpu != nil, "libdovi should accept the crafted RPU") + if let rpu { + // A handle comes back even on a parse failure; the error is what says it was accepted. + #expect(dovi_rpu_get_error(rpu) == nil, "\(dovi_rpu_get_error(rpu).map { String(cString: $0) } ?? "")") + dovi_rpu_free(rpu) + } + } + + @Test("an RPU whose CRC is wrong is refused before libdovi sees it", + arguments: [(0 as Int?, nil as Int?), (3, nil), (nil, 0), (nil, 10)]) + func badCRCIsRefused(crcByte: Int?, payloadByte: Int?) { + let good = Self.craft(linearOnComponent: nil) + Self.expectAccepted(good) + let bad = Self.withBadCRC(good, crcByte: crcByte, payloadByte: payloadByte) + #expect(bad != good) + #expect(Self.verdict(bad) == (false, false)) + #expect(bad.withUnsafeBufferPointer { DoviRpuGuard.parse($0.baseAddress!, $0.count) } == nil) + // The CRC is checked before the scan, so a hostile RPU with a wrong CRC is refused on the CRC. + let hostile = Self.withBadCRC(Self.craft(linearOnComponent: 1), crcByte: 2) + #expect(Self.verdict(hostile) == (false, false)) + #expect(hostile.withUnsafeBufferPointer { DoviRpuGuard.parse($0.baseAddress!, $0.count) } == nil) + } + + @Test("an MMR piece is walked past without a false positive") + func mmrIsSkipped() { + for c in 0..<3 { + let nal = Self.craft(linearOnComponent: nil, mmrOnComponent: c) + #expect(Self.verdict(nal) == (true, false), "MMR on component \(c)") + } + } + + @Test("linear interpolation on any component is caught before libdovi sees it", + arguments: [0, 1, 2]) + func linearInterpIsRejected(component: Int) { + let nal = Self.craft(linearOnComponent: component) + #expect(Self.verdict(nal) == (true, true)) + let rpu = nal.withUnsafeBufferPointer { DoviRpuGuard.parse($0.baseAddress!, $0.count) } + #expect(rpu == nil) + } + + @Test("linear interpolation after an MMR piece is still caught") + func linearAfterMmr() { + // MMR on component 0, linear on component 2 + let nal = Self.craft(linearOnComponent: 2, mmrOnComponent: 0) + #expect(Self.verdict(nal) == (true, true)) + #expect(nal.withUnsafeBufferPointer { DoviRpuGuard.parse($0.baseAddress!, $0.count) } == nil) + } + + /// A Profile 8-shaped RPU with `use_prev_vdr_rpu_flag` set and DM metadata: no mapping follows the + /// header, only the DM payload. `cmv40Block` is one (length, level) extended block followed by the + /// mandatory level 254 block, `cmv29Count` the declared CM v2.9 block count (no blocks follow it, so only 0 parses). + /// `prevRpuId` replaces the `prev_vdr_rpu_id` encoding. + static func craftUsePrev(cmv29Count: UInt64 = 0, cmv29Block: (length: UInt64, level: UInt64)? = nil, + cmv40Block: (length: UInt64, level: UInt64)?, compressed: Bool = true, + currentDmMetadataId: ((inout BitWriter) -> Void)? = nil, + prevRpuId: ((inout BitWriter) -> Void)? = nil) -> [UInt8] { + var w = BitWriter() + w.put(2, 6); w.put(18, 11) + w.put(1, 4); w.put(0, 4) + w.put(1, 1) + w.put(0, 1); w.put(0, 2) + w.ue(23) + w.put(0, 2); w.put(0, 1) + w.ue(2); w.ue(2); w.ue(4) + w.put(0, 1); w.put(compressed ? 1 : 0, 3); w.put(0, 1) // reserved_zero_3bits == 1 selects compressed DM data + w.put(1, 1) // disable_residual + w.put(1, 1) // vdr_dm_metadata_present + w.put(1, 1) // use_prev_vdr_rpu + if let prevRpuId { prevRpuId(&w) } else { w.ue(0) } // prev_vdr_rpu_id + writeDM(&w, DMPayload(compressed: compressed, currentDmMetadataId: currentDmMetadataId, + cmv29Count: cmv29Count, cmv29Block: cmv29Block, cmv40Block: cmv40Block)) + return seal(w.bytes) + } + + @Test("a use_prev RPU with valid DM blocks parses in libdovi and is not flagged") + func usePrevBaselineParses() { + let nal = Self.craftUsePrev(cmv40Block: (10, 8)) + #expect(Self.verdict(nal) == (true, false)) + let rpu = nal.withUnsafeBufferPointer { DoviRpuGuard.parse($0.baseAddress!, $0.count) } + #expect(rpu != nil) + if let rpu { + #expect(dovi_rpu_get_error(rpu) == nil, "\(dovi_rpu_get_error(rpu).map { String(cString: $0) } ?? "")") + dovi_rpu_free(rpu) + } + } + + @Test("use_prev does not skip the DM check: an invalid level 8 length is caught") + func usePrevInvalidLevel8Length() { + let nal = Self.craftUsePrev(cmv40Block: (11, 8)) + #expect(Self.verdict(nal) == (true, true)) + let rpu = nal.withUnsafeBufferPointer { DoviRpuGuard.parse($0.baseAddress!, $0.count) } + #expect(rpu == nil) + } + + @Test("use_prev does not skip the DM check: an oversized extended block count is caught") + func usePrevHugeBlockCount() { + let nal = Self.craftUsePrev(cmv29Count: 1 << 30, cmv40Block: nil) + #expect(Self.verdict(nal) == (true, true)) + let rpu = nal.withUnsafeBufferPointer { DoviRpuGuard.parse($0.baseAddress!, $0.count) } + #expect(rpu == nil) + } + + // MARK: - Hostile encodings and the display-management walk + + /// The guard refuses `nal`: CRC-valid, flagged, and `parse` answers nil without calling libdovi. + /// Never call libdovi directly on an RPU passed here: it is built panic=abort and the ones these + /// tests build reach a panic or an allocation abort in it. + static func expectRejected(_ nal: [UInt8], _ note: String = "", sourceLocation: SourceLocation = #_sourceLocation) { + #expect(verdict(nal) == (true, true), Comment(rawValue: note), sourceLocation: sourceLocation) + #expect(nal.withUnsafeBufferPointer { DoviRpuGuard.parse($0.baseAddress!, $0.count) } == nil, + Comment(rawValue: note), sourceLocation: sourceLocation) + } + + /// The guard lets `nal` through and libdovi parses it without an error (parsing alone is safe for + /// these; they are not hostile). + static func expectAccepted(_ nal: [UInt8], _ note: String = "", sourceLocation: SourceLocation = #_sourceLocation) { + #expect(verdict(nal) == (true, false), Comment(rawValue: note), sourceLocation: sourceLocation) + let rpu = nal.withUnsafeBufferPointer { DoviRpuGuard.parse($0.baseAddress!, $0.count) } + #expect(rpu != nil, Comment(rawValue: note), sourceLocation: sourceLocation) + if let rpu { + #expect(dovi_rpu_get_error(rpu) == nil, + Comment(rawValue: "\(note) \(dovi_rpu_get_error(rpu).map { String(cString: $0) } ?? "")"), + sourceLocation: sourceLocation) + dovi_rpu_free(rpu) + } + } + + /// The largest suffix an Exp-Golomb code with `zeros` leading zeros can carry. + static func maxRest(_ zeros: Int) -> UInt64 { zeros >= 64 ? .max : (1 << UInt64(zeros)) - 1 } + + @Test("a mapping_idc coded with 64 leading zeros is rejected, not misread as rest - 1") + func mappingIdcWithSixtyFourLeadingZeros() { + // libdovi's release-mode `read_ue` masks the shift `1 << 64` to `1 << 0`, so it reads this code as + // rest = 0, i.e. mapping_idc 0 (polynomial), then poly_order_minus1 0 and the linear flag: the + // `unimplemented!()` abort. A plain Swift shift gives 0 and `0 - 1` wrapped is not 0 or 1, so a + // scan that trusted it walked away from the RPU and let it through. + let nal = Self.craft(pieces: [[.linear], [.polynomial], [.polynomial]], + firstMappingIdc: { $0.ueRaw(zeros: 64, rest: 0) }) + Self.expectRejected(nal) + } + + @Test("an Exp-Golomb code of 32 or more leading zeros is rejected wherever the scan reads one", + arguments: [32, 33, 48, 64]) + func oversizeExpGolombIsRejected(zeros: Int) { + let rest = Self.maxRest(zeros) + // current_dm_metadata_id is written back as is by the conversion: write_ue(u64::MAX) unary-codes + // 2^32 - 1 zeros, a ~0.5 GB allocation. prev_vdr_rpu_id (header) and vdr_rpu_id (mapping) are + // written the same way and, like the DM id, not range-checked by libdovi. + Self.expectRejected(Self.craftUsePrev(cmv40Block: nil, currentDmMetadataId: { $0.ueRaw(zeros: zeros, rest: rest) }), + "current_dm_metadata_id") + Self.expectRejected(Self.craftUsePrev(cmv40Block: nil, prevRpuId: { $0.ueRaw(zeros: zeros, rest: rest) }), + "prev_vdr_rpu_id") + Self.expectRejected(Self.craft(pieces: [[.polynomial], [.polynomial], [.polynomial]], vdrRpuId: { $0.ueRaw(zeros: zeros, rest: rest) }), + "vdr_rpu_id") + Self.expectRejected(Self.craft(pieces: [[.polynomial], [.polynomial], [.polynomial]], + firstMappingIdc: { $0.ueRaw(zeros: zeros, rest: rest) }), + "mapping_idc") + } + + @Test("an Exp-Golomb code just under the bound (31 leading zeros) is still read normally") + func largestInBoundExpGolombIsAccepted() { + let nal = Self.craftUsePrev(cmv40Block: nil, currentDmMetadataId: { $0.ueRaw(zeros: 31, rest: Self.maxRest(31)) }) + Self.expectAccepted(nal) + } + + @Test("a pivot count the payload cannot hold is rejected before libdovi sizes its vectors from it", + arguments: [UInt64(1) << 31, (UInt64(1) << 32) - 2]) + func pivotCountBeyondPayload(count: UInt64) { + // libdovi allocates `vec![0; num_pivots_minus2 + 2]` per curve before it reads one pivot. + Self.expectRejected(Self.craft(pieces: [[.polynomial], [.polynomial], [.polynomial]], pivotCount: count)) + } + + @Test("an uncompressed DM payload is skipped by its 509 fixed bits and the blocks behind it are checked") + func uncompressedDmPayload() { + // VdrDmData::parse reads three ids then 509 bits of fixed fields; the guard must land exactly on + // the CM v2.9 block count behind them, or the level 8 block below is never seen. + Self.expectAccepted(Self.craftUsePrev(cmv40Block: (10, 8), compressed: false), "valid level 8, uncompressed") + Self.expectRejected(Self.craftUsePrev(cmv40Block: (11, 8), compressed: false), "level 8 length 11, uncompressed") + } + + @Test("a non-empty CM v2.9 block list is walked to the CM v4.0 list behind it") + func cmv29BlockWalk() { + // Level 1 is a 5-byte CM v2.9 block. + Self.expectAccepted(Self.craftUsePrev(cmv29Block: (5, 1), cmv40Block: (10, 8)), "valid") + Self.expectRejected(Self.craftUsePrev(cmv29Block: (5, 1), cmv40Block: (11, 8)), "level 8 length 11 behind a level 1 block") + Self.expectRejected(Self.craftUsePrev(cmv29Block: (5, 1), cmv40Block: (11, 8), compressed: false), "same, uncompressed") + } + + @Test("level 9 and level 10 blocks of an undefined length are rejected, the defined ones are not", + arguments: [(1, 9, true), (17, 9, true), (0, 9, false), (2, 9, false), (16, 9, false), (18, 9, false), + (5, 10, true), (21, 10, true), (0, 10, false), (4, 10, false), (6, 10, false), (20, 10, false)]) + func level9And10Lengths(length: Int, level: Int, valid: Bool) { + // `required_bits()` is `unreachable!()` for any other length, and the parser only compares the + // length with the block's own, so libdovi aborts after a clean parse. + let nal = Self.craftUsePrev(cmv40Block: (UInt64(length), UInt64(level))) + if valid { + Self.expectAccepted(nal, "level \(level) length \(length)") + } else { + Self.expectRejected(nal, "level \(level) length \(length)") + } + } + + @Test("level 8 lengths: the five defined ones pass, their neighbours are rejected", + arguments: [(10, true), (12, true), (13, true), (19, true), (25, true), + (0, false), (11, false), (14, false), (20, false), (24, false), (26, false)]) + func level8Lengths(length: Int, valid: Bool) { + let nal = Self.craftUsePrev(cmv40Block: (UInt64(length), 8)) + if valid { Self.expectAccepted(nal, "length \(length)") } else { Self.expectRejected(nal, "length \(length)") } + } + + @Test("coefficient_data_type 1 (32-bit coefficients, no integer parts): the walk stays in step") + func coefficientDataTypeOne() { + // With type 1 libdovi reads 32 bits per coefficient and no signed integer part; a scan that kept + // the type-0 layout would drift and miss the linear piece on the last component. + let ok = Self.craft(pieces: [[.polynomial], [.mmr], [.mmr]], coefficientDataType: 1) + Self.expectAccepted(ok) + for component in 0..<3 { + var pieces: [[Piece]] = [[.polynomial], [.mmr], [.mmr]] + pieces[component] = [.linear] + Self.expectRejected(Self.craft(pieces: pieces, coefficientDataType: 1), "linear on component \(component)") + } + Self.expectRejected(Self.craft(pieces: [[.polynomial, .mmr], [.mmr], [.mmr]], coefficientDataType: 1), "mixed curve") + } + + @Test("a Profile 7 RPU's NLQ parameters are walked to the DM payload behind them", + arguments: [Residual.mel, .fel], [true, false]) + func nlqFollowedByDm(residual: Residual, compressed: Bool) throws { + let pieces: [[Piece]] = [[.polynomial], [.mmr], [.mmr]] + func dm(_ length: UInt64) -> DMPayload { DMPayload(compressed: compressed, cmv40Block: (length, 8)) } + let good = Self.craft(pieces: pieces, residual: residual, dm: dm(10)) + Self.expectAccepted(good, "\(residual) compressed \(compressed)") + // The DM survives the conversion and write like production does. + let written = try #require(Self.convertedBytes(good)) + #expect(!written.isEmpty) + Self.expectRejected(Self.craft(pieces: pieces, residual: residual, dm: dm(11)), + "level 8 length 11, \(residual) compressed \(compressed)") + } + + // MARK: - Write path + + /// What production does with an RPU the guard let through: `DoviRpuConverter.convertedUnspec62`, the + /// mode 2 conversion and the write. nil when the guard or libdovi refuses; reaching the end at all is + /// the assertion, because libdovi is built panic=abort and a panic here kills the test process. + static func convertedBytes(_ nal: [UInt8]) -> [UInt8]? { + guard let rpu = nal.withUnsafeBufferPointer({ DoviRpuGuard.parse($0.baseAddress!, $0.count) }) else { return nil } + defer { dovi_rpu_free(rpu) } + guard let out = DoviRpuConverter.convertedUnspec62(rpu) else { return nil } + defer { dovi_data_free(out) } + guard let data = out.pointee.data else { return nil } + return [UInt8](UnsafeBufferPointer(start: data, count: out.pointee.len)) + } + + @Test("a curve that mixes polynomial and MMR pieces is rejected: the writer indexes the polynomial list per piece", + arguments: [0, 1, 2], [true, false]) + func mixedCurveIsRejected(component: Int, polynomialFirst: Bool) { + var pieces = [[Piece]](repeating: [.polynomial], count: 3) + pieces[component] = polynomialFirst ? [.polynomial, .mmr] : [.mmr, .polynomial] + for residual in [Residual.none, .mel, .fel] { + let nal = Self.craft(pieces: pieces, residual: residual) + #expect(Self.verdict(nal) == (true, true), "component \(component), residual \(residual)") + #expect(nal.withUnsafeBufferPointer { DoviRpuGuard.parse($0.baseAddress!, $0.count) } == nil) + } + } + + @Test("the same pieces without the mix are accepted, and convert and write like production does", + arguments: [Residual.none, .mel, .fel]) + func unmixedMultiPieceCurvesConvertAndWrite(residual: Residual) throws { + for pieces in [ + [[Piece.polynomial, .polynomial], [.polynomial, .polynomial], [.polynomial, .polynomial]], + [[.polynomial, .polynomial], [.mmr, .mmr], [.mmr, .mmr]], + [[.polynomial], [.mmr], [.mmr]], + ] { + let nal = Self.craft(pieces: pieces, residual: residual) + #expect(Self.verdict(nal) == (true, false), "\(pieces)") + let written = try #require(Self.convertedBytes(nal), "\(pieces)") + // The output is a Profile 8.1 RPU libdovi itself reads back. + let reread = written.withUnsafeBufferPointer { dovi_parse_unspec62_nalu($0.baseAddress!, $0.count) } + defer { reread.map { dovi_rpu_free($0) } } + #expect(reread != nil && dovi_rpu_get_error(reread) == nil, "\(pieces)") + } + } + + @Test("a crafted Profile 7 RPU is classified from its NLQ values and converts to 8.1", + arguments: [Residual.mel, .fel]) + func craftedProfile7Classification(residual: Residual) throws { + let nal = Self.craft(pieces: [[.polynomial], [.mmr], [.mmr]], residual: residual) + let rpu = try #require(nal.withUnsafeBufferPointer { DoviRpuGuard.parse($0.baseAddress!, $0.count) }) + defer { dovi_rpu_free(rpu) } + #expect(dovi_rpu_get_error(rpu) == nil) + let header = try #require(dovi_rpu_get_header(rpu)) + defer { dovi_rpu_free_header(header) } + let elType = String(cString: try #require(header.pointee.el_type)) + #expect(elType == (residual == .fel ? "FEL" : "MEL")) + #expect(Self.convertedBytes(nal) != nil) + } +} From 242a12f11ca47f0bf0ec284ac707e4f0383f649f Mon Sep 17 00:00:00 2001 From: Pavel Kajaba <15980400+pkajaba@users.noreply.github.com> Date: Wed, 7 Oct 2026 10:26:24 +0100 Subject: [PATCH 011/132] feat(renderer): bounded video look-ahead gate and display-order PTS for reordered streams Adds the SampleBufferRenderer pieces the dual-layer Dolby Vision path needs for pacing, as inert, opt-in additions: - felVideoLookaheadFrames (9) and mayEnqueueVideoFrame / setFELLookahead: a gate that runs BEFORE decode and refuses a compressed frame more than nine frame intervals ahead of the playback clock, so composed 4K frames (about 25 MB each) cannot pile up behind Apple's readiness gate alone. - displayOrderPTS: estimates a packet's display-order PTS in decode order from the largest amount a later packet fell below the highest PTS seen, so B-frame streams do not swing the cap with the position in the GOP. A gap larger than HEVC's 16-picture reorder ceiling is treated as a timestamp discontinuity and does not poison the learned lead. Nothing calls the gate yet and it is off unless setFELLookahead(enabled:) is called, so non-FEL pacing is unchanged. The decoder and the host wiring come in later PRs; the composer-side pool sizing that follows this cap (DoviFelComposer.outputPoolAllocationThreshold) is pinned in the composer PR. Tests: SampleBufferRendererLookaheadTests, SampleBufferRendererReorderDepthTests. --- .../Renderer/SampleBufferRenderer.swift | 95 +++++++++ .../SampleBufferRendererLookaheadTests.swift | 189 ++++++++++++++++++ 2 files changed, 284 insertions(+) create mode 100644 Tests/AetherEngineTests/SampleBufferRendererLookaheadTests.swift diff --git a/Sources/AetherEngine/Renderer/SampleBufferRenderer.swift b/Sources/AetherEngine/Renderer/SampleBufferRenderer.swift index 22ec80131..1a9df1de3 100644 --- a/Sources/AetherEngine/Renderer/SampleBufferRenderer.swift +++ b/Sources/AetherEngine/Renderer/SampleBufferRenderer.swift @@ -61,6 +61,99 @@ final class SampleBufferRenderer: @unchecked Sendable { return reorderBuffer.count } + /// How many frame intervals of compressed video may be admitted ahead of the playback clock on + /// the FEL path. The gate runs BEFORE decode, so the decode + EL + compose latency (a few frames) + /// comes out of this budget: 5 left a displayed lead of ~0 and the layer dropped 9 frames/s + /// on device, against 0.5 s and no drops with no cap. A composed 4K frame costs ~25 MB, and + /// Apple's readiness gate alone allowed ~12 of them. Includes frames held in our reorder buffer, which is + /// `hardwareDecoderReorderDepth` deep on this path too: the held frames sit inside the window, so a + /// composer output pool sized from this cap does not grow with the reorder depth, but they + /// are frames the display layer has not been handed, so the lead it holds is that much smaller. + static let felVideoLookaheadFrames = 9 + private var felFrameRate: Double? // guarded by reorderLock; nil leaves non-FEL pacing unchanged + /// Highest compressed-packet PTS the gate has seen, and the largest amount any later packet's PTS + /// fell below it. Packets arrive in decode order, so with B-frames a packet's PTS runs ahead of the + /// frames still waiting to be displayed; measured on the real fixtures that lead is up to 7 frames + /// (a hierarchical-B GOP of 8), 2 to 4 on other clips, 0 on a third. The gate subtracts it so the cap bounds + /// display-order look-ahead rather than swinging with the position in the GOP. Guarded by + /// `reorderLock`; reset on flush. + private var gateMaxPacketPTS: Double? + private var gateReorderLead: Double = 0 + /// HEVC's ceiling on `sps_max_num_reorder_pics`; a bigger PTS gap than this many frames is not reordering. + static let maxReorderPictures: Double = 16 + /// Ceiling on the learned reorder lead, in frames. A hierarchical-B GOP of 8 makes a packet trail the + /// highest PTS seen by up to 7 frames in decode order (measured on a UHD Blu-ray FEL remux; other + /// clips measured 2 to 4); 8 leaves one frame to spare. It is unrelated to `hardwareDecoderReorderDepth`, + /// which sizes the buffer behind the decoder. Without a ceiling one backward timestamp glitch inside + /// the 16-picture window would raise the lead, and so the admitted look-ahead, for the whole session. + static let maxLearnedReorderFrames: Double = 8 + + /// Display-order PTS estimate for a packet in decode order. Pure; the state is passed in. + /// + /// A gap below the highest PTS seen that is larger than `maxLead` cannot be B-frame reordering + /// (HEVC allows at most 16 pictures of it), so it is a timestamp discontinuity: the highest PTS + /// restarts from this packet and the learned lead is left alone, instead of absorbing the jump + /// into the lead and switching the look-ahead cap off for the rest of the session. + /// + /// The learned lead is also capped at `maxLearnedLead`: a backward step inside the 16-picture + /// window that is still a glitch (a 12-frame one, say) would otherwise raise the lead for the whole + /// session, and with it the look-ahead the gate admits. The gate passes `maxLearnedReorderFrames`. + static func displayOrderPTS(packetPTS: Double, maxSeenPTS: Double?, reorderLead: Double, + maxLead: Double = .infinity, maxLearnedLead: Double = .infinity) + -> (pts: Double, maxSeen: Double, lead: Double) { + let seen = maxSeenPTS ?? packetPTS + if seen - packetPTS > maxLead { + return (packetPTS - reorderLead, packetPTS, reorderLead) + } + let maxSeen = max(seen, packetPTS) + let lead = min(maxLearnedLead, max(reorderLead, maxSeen - packetPTS)) + return (packetPTS - lead, maxSeen, lead) + } + + func setFELLookahead(enabled: Bool, frameRate: Double?) { + reorderLock.lock() + // Container avg_frame_rate -> r_frame_rate; use 24 fps if both are unavailable. + felFrameRate = enabled ? (frameRate.flatMap { $0.isFinite && $0 > 0 ? $0 : nil } ?? 24) : nil + // Whatever was learned belonged to the previous stream or setting. + gateMaxPacketPTS = nil + gateReorderLead = 0 + reorderLock.unlock() + } + + /// Combined queue-readiness and look-ahead gate, called BEFORE decode so excess frames stay compressed. + func mayEnqueueVideoFrame(ptsSeconds: Double, clockSeconds: Double, clockArmed: Bool) -> Bool { + reorderLock.lock() + let rate = felFrameRate + let frontier = _newestEnqueuedPtsSeconds + var gatePTS = ptsSeconds.isFinite ? ptsSeconds : frontier ?? .nan + if rate != nil, ptsSeconds.isFinite { + let d = Self.displayOrderPTS(packetPTS: ptsSeconds, maxSeenPTS: gateMaxPacketPTS, + reorderLead: gateReorderLead, + maxLead: Self.maxReorderPictures / (rate ?? 24), + maxLearnedLead: Self.maxLearnedReorderFrames / (rate ?? 24)) + gateMaxPacketPTS = d.maxSeen + gateReorderLead = d.lead + gatePTS = d.pts + } + reorderLock.unlock() + return Self.mayEnqueueAnotherFrame( + rendererReady: isReadyForMoreMediaData, felFrameRate: rate, + framePTS: gatePTS, + clockSeconds: clockSeconds, clockArmed: clockArmed) + } + + /// Pure admission decision. Before clock arming there is no meaningful lead: preserve startup + /// and the existing broken-audio arming fallback. Late frames remain eligible for catch-up. + static func mayEnqueueAnotherFrame( + rendererReady: Bool, felFrameRate: Double?, framePTS: Double, + clockSeconds: Double, clockArmed: Bool + ) -> Bool { + guard rendererReady else { return false } + guard let rate = felFrameRate, rate.isFinite, rate > 0, + clockArmed, clockSeconds.isFinite, framePTS.isFinite else { return true } + return framePTS - clockSeconds <= Double(felVideoLookaheadFrames) / rate + } + /// Drop frames before this PTS after a seek (prevents keyframe-to-target fast-forward). Cleared after the first passing frame. private var skipUntilPTS: CMTime? @@ -435,6 +528,8 @@ final class SampleBufferRenderer: @unchecked Sendable { // backward seek would keep reporting the pre-seek timestamp and read as a cushion of // however far the seek travelled. _newestEnqueuedPtsSeconds = nil + gateMaxPacketPTS = nil + gateReorderLead = 0 // a seek relearns the lead from the new position // #311: everything reported before this point describes frames that are now gone. _flushGeneration = SampleBufferRenderer.flushGenerations.next() // Invalidate the format description cache; the next load() may open a stream with different colorimetry at the same resolution. diff --git a/Tests/AetherEngineTests/SampleBufferRendererLookaheadTests.swift b/Tests/AetherEngineTests/SampleBufferRendererLookaheadTests.swift new file mode 100644 index 000000000..e7257b402 --- /dev/null +++ b/Tests/AetherEngineTests/SampleBufferRendererLookaheadTests.swift @@ -0,0 +1,189 @@ +import Testing +@testable import AetherEngine + +@Suite("SampleBufferRenderer FEL look-ahead") +struct SampleBufferRendererLookaheadTests { + @Test("FEL admits the capped number of frame intervals, parks the next, and resumes as the clock advances", + arguments: [24.0, 24000.0 / 1001.0, 30.0, 60.0]) + func frameCap(rate: Double) { + let n = Double(SampleBufferRenderer.felVideoLookaheadFrames) + #expect(SampleBufferRenderer.felVideoLookaheadFrames == 9) + #expect(mayEnqueue(rate: rate, pts: n / rate, clock: 0)) + #expect(!mayEnqueue(rate: rate, pts: (n + 1) / rate, clock: 0)) + #expect(mayEnqueue(rate: rate, pts: (n + 1) / rate, clock: 2 / rate)) + #expect(mayEnqueue(rate: rate, pts: 9, clock: 10)) + } + + @Test("Non-FEL pacing depends only on queue readiness") + func nonFEL() { + #expect(mayEnqueue(rate: nil, pts: 100, clock: 0)) + #expect(!mayEnqueue(rate: nil, pts: 0, clock: 0, ready: false)) + } + + @Test("FEL still respects Apple's queue readiness") + func queueFull() { + #expect(!mayEnqueue(rate: 24, pts: 0, clock: 0, ready: false)) + #expect(!mayEnqueue(rate: 24, pts: 0, clock: 0, ready: false, armed: false)) + } + + @Test("Startup and unavailable timing do not prevent clock arming") + func startup() { + #expect(mayEnqueue(rate: 24, pts: 100, clock: 0, armed: false)) + #expect(mayEnqueue(rate: 24, pts: 100, clock: .nan)) + #expect(mayEnqueue(rate: 24, pts: .nan, clock: 0)) + } + + @Test("Lead uses the source axis, including after a backward seek") + func sourceAxis() { + #expect(mayEnqueue(rate: 24, pts: 100.3, clock: 100)) // 9 frames = 0.375 s + #expect(!mayEnqueue(rate: 24, pts: 100.4, clock: 100)) + #expect(mayEnqueue(rate: 24, pts: 10.3, clock: 10)) + #expect(!mayEnqueue(rate: 24, pts: 10.4, clock: 10)) + } + + @Test("Enabling FEL applies the cap and disabling it restores queue-only pacing") + @MainActor func configuration() throws { + let renderer = SampleBufferRenderer() + // The negative checks below only mean something while Apple's readiness gate is open. + try #require(renderer.isReadyForMoreMediaData) + renderer.setFELLookahead(enabled: true, frameRate: 60) + #expect(!renderer.mayEnqueueVideoFrame(ptsSeconds: 0.2, clockSeconds: 0, clockArmed: true)) // 9 frames at 60 fps = 0.15 s + renderer.setFELLookahead(enabled: false, frameRate: 60) + #expect(renderer.mayEnqueueVideoFrame(ptsSeconds: 100, clockSeconds: 0, clockArmed: true) + == renderer.isReadyForMoreMediaData) + renderer.setFELLookahead(enabled: true, frameRate: nil) + #expect(renderer.mayEnqueueVideoFrame(ptsSeconds: 0.37, clockSeconds: 0, clockArmed: true) + == renderer.isReadyForMoreMediaData) // nil rate falls back to 24 fps: 0.375 s + #expect(!renderer.mayEnqueueVideoFrame(ptsSeconds: 0.4, clockSeconds: 0, clockArmed: true)) + } + + @Test("Decode-order packet PTS is corrected by the measured reorder lead") + func displayOrder() { + // I P B B: PTS 0, 4, 2, 1 (in frames) arriving in decode order, 3-deep lead. + var maxSeen: Double? = nil + var lead = 0.0 + var out: [Double] = [] + for pts in [0.0, 4, 2, 1, 8, 6, 5] { + let d = SampleBufferRenderer.displayOrderPTS(packetPTS: pts, maxSeenPTS: maxSeen, reorderLead: lead) + maxSeen = d.maxSeen; lead = d.lead; out.append(d.pts) + } + #expect(lead == 3) + // Once the lead is known, the reference packet (PTS 8) is gated as the frame 5 ahead it must wait behind. + #expect(out[4] == 5) + // Monotonic streams are untouched. + let m = SampleBufferRenderer.displayOrderPTS(packetPTS: 3, maxSeenPTS: 2, reorderLead: 0) + #expect(m.pts == 3 && m.lead == 0) + } + + @Test("A glitch inside the reorder window cannot grow the learned lead past the reorder depth") + func glitchLeadIsBounded() { + let frame = 1.0 / 24 + let bound = SampleBufferRenderer.maxLearnedReorderFrames * frame + let maxLead = SampleBufferRenderer.maxReorderPictures * frame + // One packet 12 frames below the highest PTS: inside the 16-picture window, so not a discontinuity. + let d = SampleBufferRenderer.displayOrderPTS(packetPTS: 100 - 12 * frame, maxSeenPTS: 100, reorderLead: 0, + maxLead: maxLead, maxLearnedLead: bound) + #expect(abs(d.lead - bound) < 1e-9) + // The gated PTS is therefore at most `bound` below the packet's own: the admitted look-ahead + // grows by at most 8 frames, not 12 (at the 24 fps used here). + #expect(abs(d.pts - (100 - 12 * frame - bound)) < 1e-9) + // And a later, ordinary packet does not raise it further. + let n = SampleBufferRenderer.displayOrderPTS(packetPTS: 100.5, maxSeenPTS: d.maxSeen, reorderLead: d.lead, + maxLead: maxLead, maxLearnedLead: bound) + #expect(abs(n.lead - bound) < 1e-9) + } + + @Test("The gate itself stays bounded after a 12-frame backward glitch") + @MainActor func gateGlitchIsBounded() throws { + let renderer = SampleBufferRenderer() + try #require(renderer.isReadyForMoreMediaData) + renderer.setFELLookahead(enabled: true, frameRate: 24) + let frame = 1.0 / 24 + #expect(renderer.mayEnqueueVideoFrame(ptsSeconds: 100, clockSeconds: 100, clockArmed: true)) + #expect(renderer.mayEnqueueVideoFrame(ptsSeconds: 100 - 12 * frame, clockSeconds: 100, clockArmed: true)) + // The glitch teaches 8 frames (the bound, not 12): 9 + 8 = 17 frames ahead is admitted, 18 is not. + #expect(renderer.mayEnqueueVideoFrame(ptsSeconds: 100 + 17 * frame, clockSeconds: 100, clockArmed: true)) + #expect(!renderer.mayEnqueueVideoFrame(ptsSeconds: 100 + 18 * frame, clockSeconds: 100, clockArmed: true)) + } + + /// Teaches the gate a 4-frame lead with the highest packet PTS at 100, then runs `interrupt`. If the + /// lead or the highest PTS survived it, the first packet below 100 would be read as reordering and + /// the 10-frame check would pass (stale lead 4 or 3: gated 6 or 7 frames ahead); with both forgotten + /// it is the first packet of a new stream and a frame 10 ahead is refused. + @MainActor private func expectGateForgotten(after interrupt: (SampleBufferRenderer) -> Void) throws { + let renderer = SampleBufferRenderer() + try #require(renderer.isReadyForMoreMediaData) + let frame = 1.0 / 24 + renderer.setFELLookahead(enabled: true, frameRate: 24) + _ = renderer.mayEnqueueVideoFrame(ptsSeconds: 100, clockSeconds: 100, clockArmed: true) + _ = renderer.mayEnqueueVideoFrame(ptsSeconds: 100 - 4 * frame, clockSeconds: 100, clockArmed: true) // learns 4 frames + interrupt(renderer) + let clock = 100 - 3 * frame + // 3 frames below the old highest PTS: a stale highest PTS would teach a lead of 3 here. + #expect(renderer.mayEnqueueVideoFrame(ptsSeconds: clock, clockSeconds: clock, clockArmed: true)) + #expect(renderer.mayEnqueueVideoFrame(ptsSeconds: clock + 9 * frame, clockSeconds: clock, clockArmed: true)) + #expect(!renderer.mayEnqueueVideoFrame(ptsSeconds: clock + 10 * frame, clockSeconds: clock, clockArmed: true)) + } + + @Test("Disabling and re-enabling the look-ahead forgets the learned lead and highest PTS") + @MainActor func disableResetsGateState() throws { + try expectGateForgotten { renderer in + renderer.setFELLookahead(enabled: false, frameRate: nil) + renderer.setFELLookahead(enabled: true, frameRate: 24) + } + } + + @Test("A seek (flush) forgets the learned lead and highest PTS") + @MainActor func flushResetsGateState() throws { + try expectGateForgotten { renderer in renderer.flush(removingDisplayedImage: false) } + } + + @Test("A hierarchical-B GOP of 8 learns a lead of exactly 7 frames, and a glitch is bounded at 8") + func hierarchicalBLead() { + let frame = 1.0 / 24 + let maxLead = SampleBufferRenderer.maxReorderPictures * frame + let cap = SampleBufferRenderer.maxLearnedReorderFrames * frame + // Decode order I0 P8 B4 B2 B1 B3 B6 B5 B7, repeated every 8 frames. + var maxSeen: Double? = nil + var lead = 0.0 + for gop in 0..<4 { + for offset in [0, 8, 4, 2, 1, 3, 6, 5, 7] where gop == 0 || offset != 0 { + let d = SampleBufferRenderer.displayOrderPTS(packetPTS: Double(gop * 8 + offset) * frame, maxSeenPTS: maxSeen, + reorderLead: lead, maxLead: maxLead, maxLearnedLead: cap) + maxSeen = d.maxSeen; lead = d.lead + } + } + #expect(abs(lead - 7 * frame) < 1e-9) + // A 12-frame glitch inside the window is bounded at 8. + let g = SampleBufferRenderer.displayOrderPTS(packetPTS: (maxSeen ?? 0) - 12 * frame, maxSeenPTS: maxSeen, + reorderLead: lead, maxLead: maxLead, maxLearnedLead: cap) + #expect(abs(g.lead - 8 * frame) < 1e-9) + } + + @Test("A backward timestamp jump is a discontinuity, not reorder lead") + func discontinuityDoesNotGrowLead() { + // 24 fps, B-frame lead of 3 frames already learned. + let frame = 1.0 / 24 + var maxSeen: Double? = 100 + var lead = 3 * frame + let maxLead = SampleBufferRenderer.maxReorderPictures * frame + // A packet 10 s below the highest PTS seen. + let d = SampleBufferRenderer.displayOrderPTS(packetPTS: 90, maxSeenPTS: maxSeen, reorderLead: lead, maxLead: maxLead) + maxSeen = d.maxSeen; lead = d.lead + #expect(lead == 3 * frame) + #expect(maxSeen == 90) + // The gate still refuses a frame a second ahead of the clock, which a lead of 10 s would not. + let next = SampleBufferRenderer.displayOrderPTS(packetPTS: 90.5, maxSeenPTS: maxSeen, reorderLead: lead, maxLead: maxLead) + #expect(!mayEnqueue(rate: 24, pts: next.pts, clock: 89)) + // Ordinary reordering inside the limit still teaches the lead. + let r = SampleBufferRenderer.displayOrderPTS(packetPTS: 90.5 - 4 * frame, maxSeenPTS: next.maxSeen, reorderLead: lead, maxLead: maxLead) + #expect(abs(r.lead - 4 * frame) < 1e-9) + } + + private func mayEnqueue(rate: Double?, pts: Double, clock: Double, + ready: Bool = true, armed: Bool = true) -> Bool { + SampleBufferRenderer.mayEnqueueAnotherFrame( + rendererReady: ready, felFrameRate: rate, framePTS: pts, + clockSeconds: clock, clockArmed: armed) + } +} From cf93b6a07a4313fa1a1e4186df095392a1fd2200 Mon Sep 17 00:00:00 2001 From: Pavel Kajaba <15980400+pkajaba@users.noreply.github.com> Date: Tue, 13 Oct 2026 20:55:32 +0100 Subject: [PATCH 012/132] fix(io): the VOD packet cache covers audio packets that carry no duration Matroska TrueHD reaches the software read-ahead with no packet duration and millisecond timestamps that repeat (0.83 ms frames on a 1/1000 time base). SoftwarePacketCoverage.insert rejects a zero duration, so audio coverage stayed empty, the combined A/V frontier was nil, and no seek ever hit the packet cache: every seek, even back to a spot just played, refetched from the source. Audio arrives in presentation order, so a duration-less packet now proves coverage up to the next packet's timestamp, as long as the gap is at most 0.5 s (a larger one is a hole). Packets with a duration are unchanged. Apple TV 4K, 4K Matroska with TrueHD 7.1 over HTTP, 5 seeks per run: cache hits went from 0/10 to 5/5 in every run; AC-3 and video-only already hit 10/10. The new test fails without the change. --- CHANGELOG.md | 5 ++ .../Native/SoftwarePacketReadAhead.swift | 28 +++++- ...arePacketReadAheadAudioCoverageTests.swift | 89 +++++++++++++++++++ 3 files changed, 121 insertions(+), 1 deletion(-) create mode 100644 Tests/AetherEngineTests/SoftwarePacketReadAheadAudioCoverageTests.swift diff --git a/CHANGELOG.md b/CHANGELOG.md index c97a6da1d..7a42e8e3f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,11 @@ the public-API contract. ### Fixed +- **Seeks on Matroska TrueHD titles hit the software packet cache again.** TrueHD packets carry no + duration and repeat their millisecond timestamps, so the cache never counted the audio as stored, + and every seek on the software path, even back to a spot just played, refetched from the source. + A duration-less audio packet now covers up to the next one's timestamp (gaps up to 0.5 s). + - **A live GOP longer than the sealed target duration no longer breaks the playlist.** RFC 8216 keeps `TARGETDURATION` fixed for a playlist's life, so a later segment above it made the playlist invalid for AVPlayer (-12642). The engine now retires that playlist URL with its original target and an diff --git a/Sources/AetherEngine/Native/SoftwarePacketReadAhead.swift b/Sources/AetherEngine/Native/SoftwarePacketReadAhead.swift index 3a7ee53d2..4a6f4bd68 100644 --- a/Sources/AetherEngine/Native/SoftwarePacketReadAhead.swift +++ b/Sources/AetherEngine/Native/SoftwarePacketReadAhead.swift @@ -75,6 +75,8 @@ final class SoftwarePacketReadAhead: @unchecked Sendable { private var consumedVideoSeconds: Double? private var videoCoverage: SoftwarePacketCoverage private var audioCoverage: SoftwarePacketCoverage + /// Newest audio timestamp stored this epoch, for duration-less packets. See `insertAudioCoverageLocked`. + private var lastAudioPTS: Int64? private var presentationCoverage: SoftwareVideoPacketCoverage? private struct Keyframe { let seconds: Double @@ -404,6 +406,7 @@ final class SoftwarePacketReadAhead: @unchecked Sendable { ended = false; failure = nil videoCoverage.reset(); audioCoverage.reset(); presentationCoverage?.reset() storedVideoSeconds = nil; consumedVideoSeconds = nil + lastAudioPTS = nil keyframes.removeAll(keepingCapacity: true) } @@ -520,7 +523,7 @@ final class SoftwarePacketReadAhead: @unchecked Sendable { if audioCoverage.isFull, let tick = playheadTick(audio) { audioCoverage.prune(before: tick) } - audioCoverage.insert(pts: packet.pts, duration: packet.duration) + insertAudioCoverageLocked(pts: packet.pts, duration: packet.duration, stream: audio) } } condition.broadcast() @@ -531,6 +534,29 @@ final class SoftwarePacketReadAhead: @unchecked Sendable { } catch { recordFailure(error, token: token) } } + /// Widest gap between two duration-less audio packets that still counts as contiguous. Real + /// frame spacing is 0.8 ms (TrueHD) to ~100 ms (long AAC/Vorbis frames); a larger gap is a hole. + static let maxAudioSuccessorGapSeconds = 0.5 + + /// Audio arrives in presentation order, so a packet without a duration still proves coverage up + /// to the next packet's timestamp. Matroska TrueHD is the case that needs it: no packet carries + /// a duration and the millisecond timestamps repeat (0.83 ms frames), so the duration model + /// rejected every packet, the A/V frontier stayed nil, and no seek ever hit the cache: each one, + /// even back to a spot just played, refetched from the source. + private func insertAudioCoverageLocked(pts: Int64, duration: Int64, stream: Stream) { + guard pts != Int64.min else { return } + let previous = lastAudioPTS + if previous.map({ pts > $0 }) ?? true { lastAudioPTS = pts } + if duration > 0 { + audioCoverage.insert(pts: pts, duration: duration) + return + } + guard let previous, pts > previous, stream.numerator > 0, stream.denominator > 0, + Double(pts - previous) * Double(stream.numerator) / Double(stream.denominator) + <= Self.maxAudioSuccessorGapSeconds else { return } + audioCoverage.insert(pts: previous, duration: pts - previous) + } + private func shouldParkLocked() -> Bool { // One source record can cross the budget. A consumed active chunk cannot be deleted before // cursor rollover, so residency also includes bounded protected chunk slack (not an diff --git a/Tests/AetherEngineTests/SoftwarePacketReadAheadAudioCoverageTests.swift b/Tests/AetherEngineTests/SoftwarePacketReadAheadAudioCoverageTests.swift new file mode 100644 index 000000000..0fb0070c5 --- /dev/null +++ b/Tests/AetherEngineTests/SoftwarePacketReadAheadAudioCoverageTests.swift @@ -0,0 +1,89 @@ +import Foundation +import Testing +@testable import AetherEngine + +/// Matroska TrueHD reaches the read-ahead with no packet duration and millisecond timestamps that +/// repeat (0.83 ms frames on a 1/1000 time base). The duration model rejected every such packet, so +/// audio coverage stayed empty, the combined A/V frontier was nil, and no seek ever hit the packet +/// cache: on the TV every TrueHD seek, even back to a spot just played, refetched from the source. +@Suite("Software VOD read-ahead audio coverage without durations") +struct SoftwarePacketReadAheadAudioCoverageTests { + + private final class Counter: @unchecked Sendable { + private let lock = NSLock() + private var count = 0 + func next() -> Int { lock.lock(); defer { lock.unlock() }; count += 1; return count } + } + + /// 25 fps video (40 ticks) interleaved with 1200 Hz audio frames on a 1/1000 time base, 4 s, + /// then EOF. `audioGapAt` (seconds) drops one second of audio there. + private func runToEnd(audioDuration: Int64, audioGapAt: Double? = nil) throws -> SoftwarePacketReadAhead { + let root = FileManager.default.temporaryDirectory + .appendingPathComponent("readahead-audio-\(UUID().uuidString)", isDirectory: true) + try FileManager.default.createDirectory(at: root, withIntermediateDirectories: true) + + var packets: [SoftwareStoredPacket] = [] + var audioFrame = 0 + for picture in 0..<100 { + let videoPTS = Int64(picture * 40) + packets.append(SoftwareStoredPacket( + pts: videoPTS, dts: videoPTS, duration: 40, position: 0, streamIndex: 0, flags: 0x1, + timeBaseNumerator: 1, timeBaseDenominator: 1000, bytes: Data(repeating: 7, count: 64), + sideData: [])) + // 48 audio frames of 1/1200 s per 40 ms picture. + for _ in 0..<48 { + let seconds = Double(audioFrame) / 1200 + audioFrame += 1 + if let gap = audioGapAt, seconds >= gap, seconds < gap + 1 { continue } + let audioPTS = Int64((seconds * 1000).rounded(.down)) + packets.append(SoftwareStoredPacket( + pts: audioPTS, dts: audioPTS, duration: audioDuration, position: 0, streamIndex: 1, + flags: 0x1, timeBaseNumerator: 1, timeBaseDenominator: 1000, + bytes: Data(repeating: 3, count: 16), sideData: [])) + } + } + let counter = Counter() + let all = packets + let fifo = try SoftwarePacketDiskFIFO(chunkTargetBytes: 64 * 1024, retainConsumed: true, + parentDirectory: root) + let source = SoftwarePacketReadAhead( + video: .init(index: 0, numerator: 1, denominator: 1000), + audio: .init(index: 1, numerator: 1, denominator: 1000), + byteBudget: 64 << 20, forwardSeconds: 100, initialSourceClock: 0, fifo: fifo + ) { _ in + let index = counter.next() - 1 + return index < all.count ? all[index] : nil + } + source.start() + let deadline = Date().addingTimeInterval(10) + while !source.snapshot.sourceEnded, Date() < deadline { Thread.sleep(forTimeInterval: 0.02) } + #expect(source.snapshot.sourceEnded) + return source + } + + private func seekHits(_ source: SoftwarePacketReadAhead, to seconds: Double) throws -> Bool { + let token = source.beginSeek(to: seconds) + return try source.prepareSeek(token, to: seconds) + } + + @Test("duration-less, repeating-millisecond audio still makes a retained seek a cache hit") + func durationlessAudioHits() throws { + let source = try runToEnd(audioDuration: 0) + defer { source.close() } + #expect(try seekHits(source, to: 2.0)) + } + + @Test("audio with durations keeps hitting as before") + func durationAudioHits() throws { + let source = try runToEnd(audioDuration: 1) + defer { source.close() } + #expect(try seekHits(source, to: 2.0)) + } + + @Test("a real hole in duration-less audio is not bridged") + func audioHoleMisses() throws { + let source = try runToEnd(audioDuration: 0, audioGapAt: 1.5) + defer { source.close() } + #expect(try seekHits(source, to: 2.0) == false) + } +} From 936f8b2ff97afe48e4d2620bdd982629b442c7df Mon Sep 17 00:00:00 2001 From: Pavel Kajaba <15980400+pkajaba@users.noreply.github.com> Date: Tue, 13 Oct 2026 21:00:51 +0100 Subject: [PATCH 013/132] fix(tvos): keep the software VOD packet cache in purgeable memory The software path's compressed packet cache was a session file on disk. On an Apple TV 4K (A12) appends to it took 100 to 600 ms while a download ran; the producer holds the store across an append, the demux thread's reads waited behind it (and its own reads stalled up to a second), and video it had already parked reached the decoder late. The layer dropped it. On tvOS the store now lives in memory (SoftwarePacketMemoryFIFO behind a SoftwarePacketStore protocol the disk FIFO also conforms to). Records keep the disk layout in chunks of purgeable VM: unread data stays resident, a chunk the reader has finished is made volatile, so the kernel may reclaim it and it stops counting against the footprint; a seek into reclaimed history is a cache miss and refetches. Bounds: 96 MB forward, 128 MB of played history. AVPlayer keeps its progressive-download cache in memory on tvOS as well. macOS, iOS and visionOS keep the disk store. Apple TV 4K (A12), 4K Dolby Vision title over HTTP, 150 s soaks with 5 seeks: disk store 42/39/22 drops, memory store 0 to 3 in 8 of 9 runs (one run 26, not reproduced), no slow reads left. Peak footprint ~655 MB against ~580 MB with the disk store. --- CHANGELOG.md | 7 + Scripts/test-software-packet-read-ahead.sh | 1 + .../Native/SoftwarePacketDiskFIFO.swift | 9 +- .../Native/SoftwarePacketMemoryFIFO.swift | 357 ++++++++++++++++++ .../Native/SoftwarePacketReadAhead.swift | 4 +- .../Native/SoftwarePacketStore.swift | 17 + .../Native/SoftwarePlaybackHost.swift | 54 ++- .../SoftwarePacketMemoryFIFOTests.swift | 155 ++++++++ docs/architecture.md | 9 + 9 files changed, 594 insertions(+), 19 deletions(-) create mode 100644 Sources/AetherEngine/Native/SoftwarePacketMemoryFIFO.swift create mode 100644 Sources/AetherEngine/Native/SoftwarePacketStore.swift create mode 100644 Tests/AetherEngineTests/SoftwarePacketMemoryFIFOTests.swift diff --git a/CHANGELOG.md b/CHANGELOG.md index c97a6da1d..5a7630bd6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,13 @@ the public-API contract. ### Fixed +- **The software path no longer drops frames on tvOS while it downloads.** Its compressed packet + cache was a temporary disk file; on an Apple TV 4K appends to it took 100 to 600 ms while a download + ran, the demux thread waited behind them, and video it had already parked reached the decoder late. + On tvOS the cache now lives in purgeable memory (96 MB forward, up to 128 MB of reclaimable played + history), as AVPlayer's own cache does there: a 4K title's 150 s soak went from 22 to 42 drops to + 0 to 3. macOS, iOS and visionOS keep the disk cache. + - **A live GOP longer than the sealed target duration no longer breaks the playlist.** RFC 8216 keeps `TARGETDURATION` fixed for a playlist's life, so a later segment above it made the playlist invalid for AVPlayer (-12642). The engine now retires that playlist URL with its original target and an diff --git a/Scripts/test-software-packet-read-ahead.sh b/Scripts/test-software-packet-read-ahead.sh index 7488e7578..68195fdd0 100644 --- a/Scripts/test-software-packet-read-ahead.sh +++ b/Scripts/test-software-packet-read-ahead.sh @@ -13,6 +13,7 @@ xcrun swiftc \ "$TASK_ROOT/Sources/AetherEngine/Native/SoftwarePacketCoverage.swift" \ "$TASK_ROOT/Sources/AetherEngine/Native/SoftwareVideoPacketCoverage.swift" \ "$TASK_ROOT/Sources/AetherEngine/Native/SoftwarePacketDiskFIFO.swift" \ + "$TASK_ROOT/Sources/AetherEngine/Native/SoftwarePacketStore.swift" \ "$TASK_ROOT/Sources/AetherEngine/Native/SoftwarePacketReadAhead.swift" \ "$TASK_ROOT/Sources/AetherEngine/Diagnostics/EngineLog.swift" \ "$TASK_ROOT/Sources/AetherEngine/Diagnostics/LogRedaction.swift" \ diff --git a/Sources/AetherEngine/Native/SoftwarePacketDiskFIFO.swift b/Sources/AetherEngine/Native/SoftwarePacketDiskFIFO.swift index 2126cbf78..3d8f79835 100644 --- a/Sources/AetherEngine/Native/SoftwarePacketDiskFIFO.swift +++ b/Sources/AetherEngine/Native/SoftwarePacketDiskFIFO.swift @@ -12,10 +12,11 @@ final class SoftwarePacketDiskFIFO: @unchecked Sendable { /// Clients can index keyframes by token without retaining packet payloads or forging offsets. struct Cursor: Hashable, Sendable { let chunkID: UInt64 - fileprivate let generation: UUID - fileprivate let offset: UInt64 - fileprivate let recordIndex: Int - fileprivate let payloadPrefix: Int + // Internal, not fileprivate: `SoftwarePacketMemoryFIFO` mints the same tokens. + let generation: UUID + let offset: UInt64 + let recordIndex: Int + let payloadPrefix: Int } struct StaleSweepResult: Sendable { diff --git a/Sources/AetherEngine/Native/SoftwarePacketMemoryFIFO.swift b/Sources/AetherEngine/Native/SoftwarePacketMemoryFIFO.swift new file mode 100644 index 000000000..9ddb45edf --- /dev/null +++ b/Sources/AetherEngine/Native/SoftwarePacketMemoryFIFO.swift @@ -0,0 +1,357 @@ +import Darwin +import Foundation + +/// The software VOD packet store in purgeable memory instead of session files: tvOS's store, the +/// way AVPlayer keeps its progressive-download cache in memory there. +/// +/// On an Apple TV 4K (A12) the disk store's appends took 100 to 600 ms while a download ran, and the +/// consumer's reads stalled for up to a second behind them, on the demux thread: video it already had +/// parked reached the decoder late and the layer dropped it. Here every operation is a copy under a +/// lock. +/// +/// Records use the disk layout (8-byte length, then payload) in chunks of purgeable VM. A chunk the +/// consumer has finished with is history: it is made volatile, so it no longer counts against the +/// footprint and the kernel may take it back under memory pressure. Moving the reader back onto +/// history (a cached seek, a still walk) makes those chunks non-volatile again; one the kernel +/// emptied is evicted together with everything older, so the read-ahead sees a cache miss and +/// refetches. Unread data is never volatile. +/// +/// `snapshot.diskByteCount` reports only non-volatile bytes, which is what the read-ahead's byte +/// budget bounds (the forward window). History is bounded separately by `historyCapBytes`. +final class SoftwarePacketMemoryFIFO: @unchecked Sendable { + private final class Chunk { + let id: UInt64 + let base: UnsafeMutableRawPointer + let capacity: Int + var used = 0 + private(set) var isVolatile = false + + init(id: UInt64, capacity: Int) throws { + var address: vm_address_t = 0 + let size = vm_size_t(capacity) + let kr = vm_allocate(mach_task_self_, &address, size, VM_FLAGS_ANYWHERE | VM_FLAGS_PURGABLE) + guard kr == KERN_SUCCESS, let base = UnsafeMutableRawPointer(bitPattern: UInt(address)) else { + throw SoftwarePacketDiskFIFO.Failure.capacityExceeded + } + self.id = id + self.base = base + self.capacity = capacity + } + + deinit { + vm_deallocate(mach_task_self_, vm_address_t(UInt(bitPattern: base)), vm_size_t(capacity)) + } + + /// `vm_purgable_control` operations; the header macros carry a cast and do not import. + private static let setState: vm_purgable_t = 0 + + /// Returns false when making the chunk non-volatile found it already emptied by the kernel. + /// A chunk that cannot be made volatile simply stays resident. + @discardableResult + func setVolatile(_ volatile: Bool) -> Bool { + guard volatile != isVolatile else { return true } + var state: Int32 = volatile ? VM_PURGABLE_VOLATILE : VM_PURGABLE_NONVOLATILE + let kr = vm_purgable_control(mach_task_self_, vm_address_t(UInt(bitPattern: base)), + Self.setState, &state) + if volatile { + if kr == KERN_SUCCESS { isVolatile = true } + return true + } + isVolatile = false + // SET_STATE hands back the previous state. + return kr == KERN_SUCCESS && state != VM_PURGABLE_EMPTY + } + + /// Empties the chunk now, as the kernel may once it is volatile. Tests only. + func purgeForTesting() { + var state: Int32 = VM_PURGABLE_EMPTY + vm_purgable_control(mach_task_self_, vm_address_t(UInt(bitPattern: base)), Self.setState, &state) + } + } + + private let chunkTargetBytes: Int + private let historyCapBytes: Int + private let lock = NSLock() + /// Contiguous ids, oldest first. `chunks[i].id == oldestChunkID + i`. + private var chunks: [Chunk] = [] + private var generation = UUID() + private var oldestChunkID: UInt64 = 0 + private var headID: UInt64 = 0 + private var headOffset = 0 + private var recordCount = 0 + private var payloadBytes = 0 + private var writtenRecordCount = 0 + private var writtenPayloadBytes = 0 + private var isClosed = false + private var failure: (any Error)? + /// History chunks found emptied by the kernel. Diagnostics and tests. + private(set) var purgedChunks = 0 + + init(chunkTargetBytes: Int = 4 << 20, historyCapBytes: Int) { + let page = Int(getpagesize()) + self.chunkTargetBytes = max(page, (chunkTargetBytes + page - 1) / page * page) + self.historyCapBytes = max(0, historyCapBytes) + } + + var snapshot: SoftwarePacketDiskFIFO.Snapshot { + lock.lock() + defer { lock.unlock() } + return SoftwarePacketDiskFIFO.Snapshot( + count: recordCount, byteCount: payloadBytes, diskByteCount: nonVolatileBytesLocked(), + chunkCount: chunks.count, oldestRetainedChunkID: chunks.isEmpty ? nil : oldestChunkID, + isClosed: isClosed, hasFailure: failure != nil) + } + + @discardableResult + func append(_ data: Data) throws -> SoftwarePacketDiskFIFO.Cursor { + lock.lock() + defer { lock.unlock() } + try requireUsable() + let recordBytes = data.count + 8 + do { + if chunks.isEmpty || chunks[chunks.count - 1].capacity - chunks[chunks.count - 1].used < recordBytes { + let page = Int(getpagesize()) + let capacity = max(chunkTargetBytes, (recordBytes + page - 1) / page * page) + let id = chunks.isEmpty ? oldestChunkID : chunks[chunks.count - 1].id + 1 + if chunks.isEmpty { headID = id; headOffset = 0 } + chunks.append(try Chunk(id: id, capacity: capacity)) + } + } catch { + failure = error + throw error + } + let tail = chunks[chunks.count - 1] + let cursor = SoftwarePacketDiskFIFO.Cursor( + chunkID: tail.id, generation: generation, offset: UInt64(tail.used), + recordIndex: writtenRecordCount, payloadPrefix: writtenPayloadBytes) + var length = UInt64(data.count).littleEndian + withUnsafeBytes(of: &length) { tail.base.advanced(by: tail.used).copyMemory(from: $0.baseAddress!, byteCount: 8) } + data.withUnsafeBytes { raw in + if let src = raw.baseAddress { tail.base.advanced(by: tail.used + 8).copyMemory(from: src, byteCount: data.count) } + } + tail.used += recordBytes + recordCount += 1 + payloadBytes += data.count + writtenRecordCount += 1 + writtenPayloadBytes += data.count + return cursor + } + + func pop() throws -> Data? { + lock.lock() + defer { lock.unlock() } + try requireUsable() + guard recordCount > 0 else { return nil } + var head = try chunkLocked(headID) + if headOffset == head.used, headID < tailIDLocked { + head.setVolatile(true) + headID += 1 + headOffset = 0 + head = try chunkLocked(headID) + } + guard head.used - headOffset >= 8 else { throw fail(.corruptRecord) } + var length: UInt64 = 0 + withUnsafeMutableBytes(of: &length) { $0.copyMemory(from: UnsafeRawBufferPointer(start: head.base.advanced(by: headOffset), count: 8)) } + length = UInt64(littleEndian: length) + guard length <= UInt64(payloadBytes), Int(length) <= head.used - headOffset - 8 else { + throw fail(.corruptRecord) + } + let data = Data(bytes: head.base.advanced(by: headOffset + 8), count: Int(length)) + headOffset += 8 + Int(length) + recordCount -= 1 + payloadBytes -= Int(length) + if headOffset == head.used, headID < tailIDLocked { + head.setVolatile(true) + headID += 1 + headOffset = 0 + } + return data + } + + /// Moves only the consumer cursor. History between the cursor and the old head becomes unread + /// again and is made non-volatile; if the kernel emptied any of it the cursor is refused + /// (`invalidCursor`) and the read-ahead takes its normal uncached path. + func restore(to cursor: SoftwarePacketDiskFIFO.Cursor) throws { + lock.lock() + defer { lock.unlock() } + try requireUsable() + try validateLocked(cursor) + if cursor.chunkID < headID { + try reclaimLocked(from: cursor.chunkID, through: headID - 1) + } else { + // Forward within stored data: what the reader skips becomes history. + var id = headID + while id < cursor.chunkID { try chunkLocked(id).setVolatile(true); id += 1 } + } + let chunk = try chunkLocked(cursor.chunkID) + guard Int(cursor.offset) + 8 <= chunk.used else { throw SoftwarePacketDiskFIFO.Failure.invalidCursor } + headID = cursor.chunkID + headOffset = Int(cursor.offset) + recordCount = writtenRecordCount - cursor.recordIndex + payloadBytes = writtenPayloadBytes - cursor.payloadPrefix + } + + /// AE#605 still walk. Copies one chunk's records at a time under the lock and visits them + /// outside it; a walk that raced a reset or an eviction throws `invalidCursor` after visiting, + /// like the disk store. + func readHistory(from cursor: SoftwarePacketDiskFIFO.Cursor, visit: (Data) throws -> Bool) throws { + lock.lock() + do { + try requireUsable() + try validateLocked(cursor) + } catch { lock.unlock(); throw error } + let walkGeneration = generation + lock.unlock() + + var id = cursor.chunkID + var offset = Int(cursor.offset) + walk: while true { + lock.lock() + guard generation == walkGeneration, !isClosed, id >= oldestChunkID, id <= tailIDLocked else { + lock.unlock() + throw SoftwarePacketDiskFIFO.Failure.invalidCursor + } + let isLast = id == tailIDLocked + var records: [Data] = [] + do { + let chunk = try chunkLocked(id) + let wasVolatile = chunk.isVolatile + if wasVolatile, !chunk.setVolatile(false) { + // The catch below releases the lock. + evictPurgedLocked(through: id) + throw SoftwarePacketDiskFIFO.Failure.invalidCursor + } + while chunk.used - offset >= 8 { + var length: UInt64 = 0 + withUnsafeMutableBytes(of: &length) { $0.copyMemory(from: UnsafeRawBufferPointer(start: chunk.base.advanced(by: offset), count: 8)) } + length = UInt64(littleEndian: length) + guard Int(length) <= chunk.used - offset - 8 else { throw SoftwarePacketDiskFIFO.Failure.corruptRecord } + records.append(Data(bytes: chunk.base.advanced(by: offset + 8), count: Int(length))) + offset += 8 + Int(length) + } + if wasVolatile, id < headID { chunk.setVolatile(true) } + } catch { lock.unlock(); throw error } + lock.unlock() + for record in records where try !visit(record) { break walk } + if isLast { break } + id += 1 + offset = 0 + } + lock.lock() + let stillValid = generation == walkGeneration && !isClosed + lock.unlock() + guard stillValid else { throw SoftwarePacketDiskFIFO.Failure.invalidCursor } + } + + /// History beyond `historyCapBytes` is dropped, oldest first. The argument (the disk store's + /// residency budget) does not apply: in memory the read-ahead's budget bounds the non-volatile + /// forward window through `snapshot`. History the kernel emptied is not looked for here, since + /// this runs on every read and it costs nothing while it waits: `restore` and `readHistory` + /// find it when they reclaim a chunk, and the cap retires it in turn. + func trimConsumed(toByteBudget budget: Int) throws { + lock.lock() + defer { lock.unlock() } + try requireUsable() + var history = chunks.reduce(0) { $0 + ($1.id < headID ? $1.used : 0) } + while history > historyCapBytes, let oldest = chunks.first, oldest.id < headID { + chunks.removeFirst() + oldestChunkID = oldest.id + 1 + history -= oldest.used + } + } + + /// Empties every volatile history chunk now, as the kernel may under memory pressure. Tests only. + func purgeHistoryForTesting() { + lock.lock() + defer { lock.unlock() } + for chunk in chunks where chunk.isVolatile { chunk.purgeForTesting() } + } + + func reset() throws { + lock.lock() + defer { lock.unlock() } + guard !isClosed else { throw SoftwarePacketDiskFIFO.Failure.closed } + clearLocked() + failure = nil + } + + func close() throws { + lock.lock() + defer { lock.unlock() } + isClosed = true + chunks.removeAll() + } + + // MARK: - Private + + private var tailIDLocked: UInt64 { chunks.last?.id ?? oldestChunkID } + + private func chunkLocked(_ id: UInt64) throws -> Chunk { + guard id >= oldestChunkID, Int(id - oldestChunkID) < chunks.count else { + throw fail(.corruptRecord) + } + return chunks[Int(id - oldestChunkID)] + } + + private func validateLocked(_ cursor: SoftwarePacketDiskFIFO.Cursor) throws { + guard cursor.generation == generation, !chunks.isEmpty, + cursor.chunkID >= oldestChunkID, cursor.chunkID <= tailIDLocked, + cursor.recordIndex >= 0, cursor.recordIndex < writtenRecordCount, + cursor.payloadPrefix >= 0, cursor.payloadPrefix <= writtenPayloadBytes else { + throw SoftwarePacketDiskFIFO.Failure.invalidCursor + } + } + + /// Makes history chunks `first...last` non-volatile for re-reading. If the kernel emptied any, + /// everything up to the newest emptied one is evicted and the cursor is refused. + private func reclaimLocked(from first: UInt64, through last: UInt64) throws { + var newestPurged: UInt64? + var id = first + while id <= last { + if !(try chunkLocked(id).setVolatile(false)) { newestPurged = id } + id += 1 + } + guard let newestPurged else { return } + evictPurgedLocked(through: newestPurged) + // What survived stays history until a later restore succeeds. + for chunk in chunks where chunk.id < headID { chunk.setVolatile(true) } + throw SoftwarePacketDiskFIFO.Failure.invalidCursor + } + + /// Never evicts the reader's chunk or anything after it. + private func evictPurgedLocked(through id: UInt64) { + while let oldest = chunks.first, oldest.id <= id, oldest.id < headID { + chunks.removeFirst() + oldestChunkID = oldest.id + 1 + purgedChunks += 1 + } + } + + private func nonVolatileBytesLocked() -> Int { + chunks.reduce(0) { $0 + ($1.isVolatile ? 0 : $1.used) } + } + + private func requireUsable() throws { + guard !isClosed else { throw SoftwarePacketDiskFIFO.Failure.closed } + if let failure { throw failure } + } + + private func fail(_ error: SoftwarePacketDiskFIFO.Failure) -> SoftwarePacketDiskFIFO.Failure { + failure = error + return error + } + + private func clearLocked() { + chunks.removeAll() + generation = UUID() + oldestChunkID = 0 + headID = 0 + headOffset = 0 + recordCount = 0 + payloadBytes = 0 + writtenRecordCount = 0 + writtenPayloadBytes = 0 + } +} + +extension SoftwarePacketMemoryFIFO: SoftwarePacketStore {} diff --git a/Sources/AetherEngine/Native/SoftwarePacketReadAhead.swift b/Sources/AetherEngine/Native/SoftwarePacketReadAhead.swift index 3a7ee53d2..6181e403c 100644 --- a/Sources/AetherEngine/Native/SoftwarePacketReadAhead.swift +++ b/Sources/AetherEngine/Native/SoftwarePacketReadAhead.swift @@ -28,7 +28,7 @@ final class SoftwarePacketReadAhead: @unchecked Sendable { /// Never acquired by a main-thread API. Serializes reset with append/pop, including the /// generation check, so an old consumer cannot pop and discard the first NEW-generation packet. private let operations = NSLock() - private let fifo: SoftwarePacketDiskFIFO + private let fifo: any SoftwarePacketStore private let readSource: @Sendable (@Sendable () -> Bool) throws -> SoftwareStoredPacket? private let beforeConsumerOperation: (@Sendable () -> Void)? private let video: Stream @@ -85,7 +85,7 @@ final class SoftwarePacketReadAhead: @unchecked Sendable { /// Construct only off-main: creating the FIFO touches the temporary volume. init(video: Stream, audio: Stream?, byteBudget: Int, forwardSeconds: Double, - initialSourceClock: Double, fifo: SoftwarePacketDiskFIFO, + initialSourceClock: Double, fifo: any SoftwarePacketStore, videoReorderDepth: Int? = nil, coverageRangeCap: Int = 4096, beforeConsumerOperation: (@Sendable () -> Void)? = nil, diff --git a/Sources/AetherEngine/Native/SoftwarePacketStore.swift b/Sources/AetherEngine/Native/SoftwarePacketStore.swift new file mode 100644 index 000000000..ee1375a8c --- /dev/null +++ b/Sources/AetherEngine/Native/SoftwarePacketStore.swift @@ -0,0 +1,17 @@ +import Foundation + +/// The operations `SoftwarePacketReadAhead` needs from its packet store. `SoftwarePacketDiskFIFO` +/// keeps the records in session files; `SoftwarePacketMemoryFIFO` keeps the same records in +/// purgeable memory, which is what tvOS uses (see there). +protocol SoftwarePacketStore: AnyObject, Sendable { + var snapshot: SoftwarePacketDiskFIFO.Snapshot { get } + @discardableResult func append(_ data: Data) throws -> SoftwarePacketDiskFIFO.Cursor + func pop() throws -> Data? + func restore(to cursor: SoftwarePacketDiskFIFO.Cursor) throws + func readHistory(from cursor: SoftwarePacketDiskFIFO.Cursor, visit: (Data) throws -> Bool) throws + func trimConsumed(toByteBudget budget: Int) throws + func reset() throws + func close() throws +} + +extension SoftwarePacketDiskFIFO: SoftwarePacketStore {} diff --git a/Sources/AetherEngine/Native/SoftwarePlaybackHost.swift b/Sources/AetherEngine/Native/SoftwarePlaybackHost.swift index 9817a8f6e..d69c4488f 100644 --- a/Sources/AetherEngine/Native/SoftwarePlaybackHost.swift +++ b/Sources/AetherEngine/Native/SoftwarePlaybackHost.swift @@ -23,6 +23,23 @@ final class SoftwarePlaybackHost { } } + /// The software VOD packet store's bounds on tvOS, which keeps it in purgeable memory + /// (`SoftwarePacketMemoryFIFO`) instead of session files; nil elsewhere, where the disk store and + /// the session retention budget apply. On an Apple TV 4K (A12) the disk store's appends took + /// 100 to 600 ms while a download ran and the demux thread stalled behind them, so video it had + /// parked was decoded late and dropped: 22 to 42 drops in a 150 s soak of a 4K title, against + /// 0 to 3 in 8 of 9 runs with the store in memory (one run had 26, not seen again). The forward window is 96 MB (about 8 s of a 95 Mbit/s + /// title, on top of the forward-seconds limit); played history up to 128 MB stays volatile, so + /// the kernel may take it back and a seek into what it took refetches. Peak footprint of a 4K + /// FEL session on the A12: ~655 MB against ~580 MB with the disk store. + nonisolated static let memoryPacketStoreBudget: (forwardBytes: Int, historyBytes: Int)? = { + #if os(tvOS) + return (forwardBytes: 96 << 20, historyBytes: 128 << 20) + #else + return nil + #endif + }() + // MARK: - Published state (mirrors NativeAVPlayerHost surface) @@ -1110,19 +1127,30 @@ final class SoftwarePlaybackHost { } : nil let initialSourceClock = initialClockTime.seconds let videoReorderDepth = Self.presentationReorderDepth(codecID: vCodecID) - let cacheResult = await BlockingWork.detached(priority: .utility) { () throws -> (SoftwarePacketReadAhead, RetentionClaims.Claim)? in - let temp = URL(fileURLWithPath: NSTemporaryDirectory(), isDirectory: true) - let available = (try? temp.resourceValues(forKeys: [.volumeAvailableCapacityKey]))? - .volumeAvailableCapacity.map(Int64.init) + let cacheResult = await BlockingWork.detached(priority: .utility) { () throws -> (SoftwarePacketReadAhead, RetentionClaims.Claim?)? in let segments = HLSVideoEngine.clampedForwardWindow(forwardBufferSegments) - let capRelaxed = HLSVideoEngine.retentionCapRelaxed(forwardWindowSegments: segments) - let claim = RetentionClaims.shared.claim(volumeAvailableBytes: available) { - HLSVideoEngine.sessionRetentionBudgetBytes(volumeAvailableBytes: $0, capRelaxed: capRelaxed) + let fifo: any SoftwarePacketStore + let bytes: Int + var claim: RetentionClaims.Claim? + if let memory = Self.memoryPacketStoreBudget { + bytes = memory.forwardBytes + fifo = SoftwarePacketMemoryFIFO(historyCapBytes: memory.historyBytes) + EngineLog.emit("[SWHost] packet cache in memory: forward \(memory.forwardBytes >> 20) MB, " + + "history \(memory.historyBytes >> 20) MB", category: .swPlayback) + } else { + let temp = URL(fileURLWithPath: NSTemporaryDirectory(), isDirectory: true) + let available = (try? temp.resourceValues(forKeys: [.volumeAvailableCapacityKey]))? + .volumeAvailableCapacity.map(Int64.init) + let capRelaxed = HLSVideoEngine.retentionCapRelaxed(forwardWindowSegments: segments) + let diskClaim = RetentionClaims.shared.claim(volumeAvailableBytes: available) { + HLSVideoEngine.sessionRetentionBudgetBytes(volumeAvailableBytes: $0, capRelaxed: capRelaxed) + } + bytes = diskClaim.bytes + guard bytes > 0 else { return nil } + fifo = try SoftwarePacketDiskFIFO( + chunkTargetBytes: min(4 << 20, max(8, bytes)), retainConsumed: true) + claim = diskClaim } - let bytes = claim.bytes - guard bytes > 0 else { return nil } - let fifo = try SoftwarePacketDiskFIFO( - chunkTargetBytes: min(4 << 20, max(8, bytes)), retainConsumed: true) let readAhead = SoftwarePacketReadAhead( video: video, audio: audio, byteBudget: bytes, forwardSeconds: Double(segments) * 4, @@ -1133,14 +1161,14 @@ final class SoftwarePlaybackHost { defer { av_packet_unref(packet); av_packet_free_safe(packet) } return try SoftwareStoredPacket(copying: packet) } - claim.track { [weak readAhead] in readAhead?.snapshot.residentBytes ?? 0 } + claim?.track { [weak readAhead] in readAhead?.snapshot.residentBytes ?? 0 } return (readAhead, claim) }.result let readAhead: SoftwarePacketReadAhead? switch cacheResult { case .success(let cache): readAhead = cache?.0 - if !stopRequested, let claim = cache?.1 { retentionClaims.append(claim) } + if !stopRequested, let claim = cache?.1 ?? nil { retentionClaims.append(claim) } case .failure: // A cache-directory failure must not stop a source that the old direct loop can // still play. Runtime spool corruption is explicit, never silently skipped. diff --git a/Tests/AetherEngineTests/SoftwarePacketMemoryFIFOTests.swift b/Tests/AetherEngineTests/SoftwarePacketMemoryFIFOTests.swift new file mode 100644 index 000000000..d2cf2b47e --- /dev/null +++ b/Tests/AetherEngineTests/SoftwarePacketMemoryFIFOTests.swift @@ -0,0 +1,155 @@ +import Foundation +import Testing +@testable import AetherEngine + +/// The purgeable-memory packet store (tvOS's software VOD packet store) must behave like the +/// disk store for everything the read-ahead relies on: FIFO order across chunks, cursors that +/// restore backwards and forwards, the still walk, and history that is bounded and evictable. +@Suite("Software packet memory FIFO") +struct SoftwarePacketMemoryFIFOTests { + + /// Records of 1000 bytes filled with their index, so a read can be checked by content. + private func record(_ i: Int) -> Data { Data(repeating: UInt8(i & 0xff), count: 1000) } + + /// Page-sized chunks (16 KiB on arm64) hold 16 records each. + private func filled(_ count: Int, historyCap: Int = .max) throws + -> (SoftwarePacketMemoryFIFO, [SoftwarePacketDiskFIFO.Cursor]) { + let fifo = SoftwarePacketMemoryFIFO(chunkTargetBytes: 1, historyCapBytes: historyCap) + var cursors: [SoftwarePacketDiskFIFO.Cursor] = [] + for i in 0.. 1) + for i in 0..<100 { #expect(try fifo.pop() == record(i)) } + #expect(try fifo.pop() == nil) + #expect(fifo.snapshot.count == 0) + } + + @Test("consumed chunks leave the non-volatile byte count, unread ones stay in it") + func consumedHistoryIsNotResident() throws { + let (fifo, _) = try filled(100) + let before = fifo.snapshot.diskByteCount + for _ in 0..<80 { _ = try fifo.pop() } + let after = fifo.snapshot.diskByteCount + #expect(after < before) + #expect(after >= 20 * 1008) + } + + @Test("a cursor behind the reader restores and re-reads the same records") + func restoreBackward() throws { + let (fifo, cursors) = try filled(100) + for _ in 0..<90 { _ = try fifo.pop() } + try fifo.restore(to: cursors[10]) + #expect(fifo.snapshot.count == 90) + for i in 10..<100 { #expect(try fifo.pop() == record(i)) } + } + + @Test("a cursor ahead of the reader skips forward") + func restoreForward() throws { + let (fifo, cursors) = try filled(100) + try fifo.restore(to: cursors[70]) + #expect(fifo.snapshot.count == 30) + #expect(try fifo.pop() == record(70)) + } + + @Test("the still walk visits history without moving the reader") + func readHistory() throws { + let (fifo, cursors) = try filled(100) + for _ in 0..<60 { _ = try fifo.pop() } + var seen: [Data] = [] + try fifo.readHistory(from: cursors[20]) { seen.append($0); return seen.count < 30 } + #expect(seen == (20..<50).map(record)) + #expect(try fifo.pop() == record(60)) + } + + @Test("history over the cap is dropped and its cursors become cache misses") + func historyCap() throws { + let (fifo, cursors) = try filled(200, historyCap: 40_000) + for _ in 0..<190 { _ = try fifo.pop() } + try fifo.trimConsumed(toByteBudget: 0) + let oldest = try #require(fifo.snapshot.oldestRetainedChunkID) + #expect(oldest > cursors[0].chunkID) + #expect(throws: SoftwarePacketDiskFIFO.Failure.invalidCursor) { try fifo.restore(to: cursors[0]) } + // Recent history is still there. + try fifo.restore(to: cursors[185]) + #expect(try fifo.pop() == record(185)) + } + + @Test("history the kernel emptied is a cache miss, and the store stays usable") + func purgedHistoryRestore() throws { + let (fifo, cursors) = try filled(100) + for _ in 0..<90 { _ = try fifo.pop() } + fifo.purgeHistoryForTesting() + #expect(throws: SoftwarePacketDiskFIFO.Failure.invalidCursor) { try fifo.restore(to: cursors[10]) } + #expect(fifo.purgedChunks > 0) + // The reader did not move and unread data was never volatile. + for i in 90..<100 { #expect(try fifo.pop() == record(i)) } + try fifo.append(record(100)) + #expect(try fifo.pop() == record(100)) + } + + @Test("a still walk into emptied history is refused and releases the store") + func purgedHistoryWalk() async throws { + let (fifo, cursors) = try filled(100) + for _ in 0..<90 { _ = try fifo.pop() } + fifo.purgeHistoryForTesting() + #expect(throws: SoftwarePacketDiskFIFO.Failure.invalidCursor) { + try fifo.readHistory(from: cursors[10]) { _ in true } + } + // A lock left held (or released twice) by the refusal would hang or trap here, so the next + // operation runs on its own thread. + let popped = await withCheckedContinuation { (done: CheckedContinuation) in + Thread.detachNewThread { done.resume(returning: try? fifo.pop()) } + } + #expect(popped == record(90)) + } + + @Test("reset invalidates old cursors and starts empty") + func reset() throws { + let (fifo, cursors) = try filled(50) + try fifo.reset() + #expect(fifo.snapshot.count == 0) + #expect(throws: SoftwarePacketDiskFIFO.Failure.invalidCursor) { try fifo.restore(to: cursors[0]) } + try fifo.append(record(7)) + #expect(try fifo.pop() == record(7)) + } + + @Test("the read-ahead plays from the memory store and a backward seek into history hits") + func readAheadSeekBackHits() throws { + // 25 fps video, every picture a keyframe, 4 s, then EOF. No audio. + let packets = (0..<100).map { i in + SoftwareStoredPacket( + pts: Int64(i * 40), dts: Int64(i * 40), duration: 40, position: 0, streamIndex: 0, + flags: 0x1, timeBaseNumerator: 1, timeBaseDenominator: 1000, + bytes: Data(repeating: UInt8(i), count: 2000), sideData: []) + } + final class Counter: @unchecked Sendable { + private let lock = NSLock() + private var count = 0 + func next() -> Int { lock.lock(); defer { lock.unlock() }; count += 1; return count } + } + let counter = Counter() + let source = SoftwarePacketReadAhead( + video: .init(index: 0, numerator: 1, denominator: 1000), audio: nil, + byteBudget: 64 << 20, forwardSeconds: 100, initialSourceClock: 0, + fifo: SoftwarePacketMemoryFIFO(chunkTargetBytes: 16 * 1024, historyCapBytes: 64 << 20) + ) { _ in + let index = counter.next() - 1 + return index < packets.count ? packets[index] : nil + } + defer { source.close() } + source.start() + // Play 3 s: those chunks become volatile history. + for i in 0..<75 { #expect(try source.read()?.pts == Int64(i * 40)) } + let token = source.beginSeek(to: 1.0) + #expect(try source.prepareSeek(token, to: 1.0)) + source.endSeek(token, sourceClock: 1.0) + let first = try #require(try source.read()) + #expect(first.pts <= 1000) + #expect(first.bytes == packets[Int(first.pts / 40)].bytes) + } +} diff --git a/docs/architecture.md b/docs/architecture.md index 51cadd49a..069e541fc 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -238,6 +238,15 @@ cleanup uses session leases without following symlinks or deleting live stores. If creating the store fails, the original direct playback loop remains available. Runtime cache corruption is a reported playback failure, not silently skipped data. +On tvOS the same records live in purgeable memory instead +(`SoftwarePacketMemoryFIFO`, bounds in `SoftwarePlaybackHost.memoryPacketStoreBudget`): +a 96 MB forward window, and up to 128 MB of played history kept volatile, so the +kernel may reclaim it under pressure and a seek into reclaimed history is a normal +cache miss. On an Apple TV 4K the disk store's appends took 100 to 600 ms while a +download ran, the demux thread stalled behind them, and video it had already +parked reached the decoder late; AVPlayer keeps its progressive-download cache in +memory on tvOS too. No session files and no disk retention claim are made there. + The cache frontier is the intersection of selected audio and video presentation coverage containing the playhead. Unknown intervals remain unknown; byte counts are not converted to guessed seconds. H.264 and HEVC use a bounded presentation From e1d328505aecf5ebe3587465da40ec5b3eedc472 Mon Sep 17 00:00:00 2001 From: Pavel Kajaba <15980400+pkajaba@users.noreply.github.com> Date: Wed, 7 Oct 2026 10:34:29 +0100 Subject: [PATCH 014/132] feat(dovi): dual-layer Profile 7 FEL decoder (not routed) DualLayerVideoDecoder: the decoder half of the dual-layer route. It splits each packet into base layer, enhancement layer and RPU (DoviRpuConverter.separateDualLayers), decodes both layers on VideoToolbox through two HardwareVideoDecoder sessions, pairs them by HEVC picture order, composes with DoviFelComposer and delivers the frames in decode order with the Profile 8.1 RPU attached per frame. - Pairing: DualLayerPictureAlignment restamps each enhancement-layer packet with the time of the base picture in the same display slot, so an enhancement layer coded with B-frames over a base layer without them is no longer paired with a neighbouring frame's residual. A layer keeps access-unit pairing until its next sequence-opening IRAP after a seek. - Bounded memory: base-layer skew K = 3 frames, at most 10 waiting enhancement frames, output pool 15 plus a reserve of 3. A late or broken EL falls back to the base layer for that frame. - Seeks: a seek flushes every stage. A stale EL is never paired with a post-seek base layer and a stale BL is never delivered. The composer's output pool is kept across the flush instead of being dropped (it made the frames after a seek allocate a second pool beside the buffers the renderer still held): peak footprint on a 12-seek run of one test clip went from 674 MB to 500 MB. The pool is dropped on close. - HardwareVideoDecoder gains an opt-in no-copy block source and a parameter-set open for the enhancement layer; both are off for every existing caller. FelResidualMeasure (the AETHER_FEL_MEASURE diagnostic, off by default) is here because the decoder calls it. Nothing constructs the decoder yet: no host code selects it until the route PR, so no source plays differently after this one. Public API: none. Not in this PR: the capability gate, the preferDualLayerDolbyVision flag, host wiring, dolby-vision-fel.md. Tests: decoder, GPU slot wait, seek realignment (flush of both sessions and the composer, stale EL not composed, waiting BL dropped at the seek, seek during an in-flight composition), stream classification and packet balance, CPU model, output-pool-across-flush, real-clip tests (gated on DV_FEL_TESTS_DIR, skipped without), and a subtitle compositor test that the Dolby Vision attachments survive compositing. --- CHANGELOG.md | 5 + .../Decoder/DualLayerVideoDecoder.swift | 1706 +++++++++++ .../Decoder/HardwareVideoDecoder.swift | 283 +- .../Decoder/VideoDecoderTypes.swift | 11 + .../Diagnostics/FelResidualMeasure.swift | 105 + .../AetherEngineTests/DoviCpuModelTests.swift | 440 +++ .../DoviFelRealFileTests.swift | 395 +++ .../DoviRealFileExportTests.swift | 96 + .../DualLayerGpuSlotWaitTests.swift | 312 ++ .../DualLayerSeekRealignmentTests.swift | 307 ++ .../DualLayerStreamClassificationTests.swift | 240 ++ .../DualLayerVideoDecoderTests.swift | 2535 +++++++++++++++++ .../SubtitleFrameCompositorTests.swift | 33 + docs/architecture.md | 1 + 14 files changed, 6442 insertions(+), 27 deletions(-) create mode 100644 Sources/AetherEngine/Decoder/DualLayerVideoDecoder.swift create mode 100644 Sources/AetherEngine/Diagnostics/FelResidualMeasure.swift create mode 100644 Tests/AetherEngineTests/DoviCpuModelTests.swift create mode 100644 Tests/AetherEngineTests/DoviRealFileExportTests.swift create mode 100644 Tests/AetherEngineTests/DualLayerGpuSlotWaitTests.swift create mode 100644 Tests/AetherEngineTests/DualLayerSeekRealignmentTests.swift create mode 100644 Tests/AetherEngineTests/DualLayerStreamClassificationTests.swift create mode 100644 Tests/AetherEngineTests/DualLayerVideoDecoderTests.swift diff --git a/CHANGELOG.md b/CHANGELOG.md index 189c9f9fc..37b879c40 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,11 @@ the public-API contract. ## [Unreleased] +### Added + +- **`AETHER_FEL_MEASURE` reports the signed FEL-minus-MEL mean per plane and luma band.** The diagnostic composes every Nth frame a second time with the residual off and compares the two; the signed mean says whether the residual brightens or darkens, which the absolute difference could not. It runs one sample at a time and skips a frame that arrives while one is in flight. Off by default; it is an environment-variable diagnostic, not API. +- **The dual-layer decoder pairs the two layers by picture order.** `DualLayerPictureAlignment` derives each layer's display slot from its POC (`HEVCPictureOrderCount`, no decoding) and stamps each enhancement-layer packet with the time of the base-layer picture in the same slot, so a stream whose enhancement layer is coded with B-frames over a base layer without them pairs the right pictures. A stream whose layers share one GOP keeps every packet's own time. After a seek a layer pairs by access unit until its next sequence-opening IRAP. + ### Fixed - **A live GOP longer than the sealed target duration no longer breaks the playlist.** RFC 8216 keeps diff --git a/Sources/AetherEngine/Decoder/DualLayerVideoDecoder.swift b/Sources/AetherEngine/Decoder/DualLayerVideoDecoder.swift new file mode 100644 index 000000000..1d9f7e94d --- /dev/null +++ b/Sources/AetherEngine/Decoder/DualLayerVideoDecoder.swift @@ -0,0 +1,1706 @@ +import Foundation +import CoreMedia +import CoreVideo +import VideoToolbox +import AetherLibavformat +import AetherLibavcodec +import AetherLibavutil + +/// Side data that travels with a waiting base-layer frame (dolby-vision-fel.md +/// §5.3). The base-layer queue stores only buffer + PTS by itself, so a +/// frame whose enhancement layer arrives last would otherwise reach `onFrame` +/// with `hdr10Data` dropped; carrying it here makes delivery independent of +/// arrival order. `dmNAL` is the Profile 8.1 display-metadata NAL cached for +/// this frame's PTS, resolved once at base-layer arrival for the same reason: +/// `OrderedDelivery.emit` runs on the delivery queue and does not reach +/// into the decoder's frame state (`dmRpuCache`, the queues) itself. `HardwareVideoDecoder` passes +/// `hdr10Data: nil` today (latent), so that field is unused in practice, with +/// room for params if they migrate here too. +private struct FrameSideData: Sendable { + let hdr10Data: Data? + let dmNAL: [UInt8]? +} + +/// What `DualLayerVideoDecoder` needs from its composer. +/// +/// Splitting the GPU-slot wait (`acquireSlot`) from encode-and-commit +/// (`composeAsyncWithAcquiredSlot`) is what lets the decoder decide under its +/// lock and block outside it (dolby-vision-fel.md §5.5, §5.6). `DoviFelComposer` is the +/// production implementation; tests inject a gated fake through +/// `DualLayerVideoDecoder(composer:)`. +protocol FelComposing: Sendable { + func composeAsync( + baseBuffer: CVPixelBuffer, + enhancementBuffer: CVPixelBuffer, + params: DoviMetalParams, + frameIndex: UInt32, + completion: @escaping @Sendable (CVPixelBuffer?) -> Void + ) + func composeAsyncWithAcquiredSlot( + baseBuffer: CVPixelBuffer, + enhancementBuffer: CVPixelBuffer, + params: DoviMetalParams, + frameIndex: UInt32, + completion: @escaping @Sendable (CVPixelBuffer?) -> Void + ) + func acquireSlot() + func releaseSlot() + func flush() + func releaseOutputPool() + func flushTextureCache() + func takePeakSlotsInUse() -> Int + var allocatedPoolBuffers: Int { get } + var meanGPUTimeMs: Double? { get } +} + +extension FelComposing { + func releaseOutputPool() {} + func flushTextureCache() {} + func takePeakSlotsInUse() -> Int { 0 } + var allocatedPoolBuffers: Int { 0 } + var meanGPUTimeMs: Double? { nil } +} + +extension DoviFelComposer: FelComposing {} + +/// What `DualLayerVideoDecoder` needs from each single-layer VideoToolbox +/// decoder (dolby-vision-fel.md §5.8). +/// +/// A small protocol over `HardwareVideoDecoder` so tests can inject a fake +/// that emits a frame from inside `flush()`, proving pre-seek frames drained +/// there never reach delivery. +protocol SingleLayerVideoDecoder: VideoDecodingPipeline { + /// Opens a decoder session from raw HEVC parameter sets (VPS, SPS, PPS), + /// used for both layers when they arrive in-band. + func open( + parameterSets: [[UInt8]], + timeBase: AVRational, + onFrame: @escaping DecodedFrameHandler + ) throws + + /// Moves the decoder's feed epoch to its owner's. Called from inside the owner's `flush()` + /// critical section, so it must not take any lock a VideoToolbox callback can be waiting behind. + func adoptFeedEpoch(_ epoch: UInt64) + /// The flush the owner makes after `adoptFeedEpoch`: drains and discontinues the session without + /// moving the epoch again. + func flushKeepingEpoch() +} + +extension SingleLayerVideoDecoder { + func adoptFeedEpoch(_ epoch: UInt64) {} + func flushKeepingEpoch() { flush() } +} + +extension HardwareVideoDecoder: SingleLayerVideoDecoder {} + +/// Video decoding pipeline that transparently coordinates dual VideoToolbox HEVC decoders +/// (base layer, typically 4K, plus an enhancement layer, typically 1080p) and reconstructs the 12-bit master on Apple GPU +/// using `DoviFelComposer`. Falls back smoothly to single-layer BL if no EL is present (MEL / P8.1). +final class DualLayerVideoDecoder: VideoDecodingPipeline, @unchecked Sendable { + + // The callback itself is owned by `orderedDelivery`, not this + // decoder, so a call already queued for delivery when the host reassigns + // it (or `close()` clears it) is ordered against that write rather than + // racing it under a lock the delivery path no longer takes. + var onFrame: DecodedFrameHandler? { + get { orderedDelivery.currentOnFrame } + set { orderedDelivery.setOnFrame(newValue) } + } + + // Set once by the host before `open()`; guarded by `lock` so a late + // reassignment cannot race a VideoToolbox callback reading it. + var onFirstHDR10PlusDetected: (@Sendable () -> Void)? { + get { lock.lock(); defer { lock.unlock() }; return _onFirstHDR10PlusDetected } + set { lock.lock(); _onFirstHDR10PlusDetected = newValue; lock.unlock() } + } + private var _onFirstHDR10PlusDetected: (@Sendable () -> Void)? + var onA53Captions: (@Sendable ([CCDataParser.CCTriplet], Double) -> Void)? { + get { lock.lock(); defer { lock.unlock() }; return _onA53Captions } + set { lock.lock(); _onA53Captions = newValue; lock.unlock() } + } + private var _onA53Captions: (@Sendable ([CCDataParser.CCTriplet], Double) -> Void)? + /// AE#658: the composed display buffer's format, reported on the first frame and when it changes. + var onDecodedFormat: (@Sendable (DecodedVideoFormat) -> Void)? { + get { lock.lock(); defer { lock.unlock() }; return _onDecodedFormat } + set { lock.lock(); _onDecodedFormat = newValue; lock.unlock() } + } + private var _onDecodedFormat: (@Sendable (DecodedVideoFormat) -> Void)? + private var declaredFormat: (color: ColorDescription, codecID: AVCodecID, profile: Int32)? // guarded by `lock` + private var reportedPixelBufferType: OSType = 0 // guarded by `lock` + + /// What `onFrame` is handed is the composed buffer, so that buffer's type is the decoded format and + /// its colour is what the stream declares. Called on the delivery queue; it takes `lock` for a moment, which is + /// safe because nothing holds `lock` while waiting on that queue (`flush()` waits only after unlocking). + private func reportDecodedFormat(_ buffer: CVPixelBuffer) { + let type = CVPixelBufferGetPixelFormatType(buffer) + lock.lock() + let handler = _onDecodedFormat + let declared = declaredFormat + let isNew = type != reportedPixelBufferType + if handler != nil, isNew { reportedPixelBufferType = type } + lock.unlock() + guard let handler, isNew, let declared else { return } + handler(DecodedVideoFormat( + frame: VideoStreamFormat( + pixelFormat: DecodedVideoFormat.libavPixelFormat(forPixelBufferType: type), + declaredBitDepth: 0, + color: declared.color, + codecID: declared.codecID, + profile: declared.profile), + pixelBufferFormat: DecodedVideoFormat.fourCC(type))) + } + + var skipUntilPTS: CMTime? { + get { lock.lock(); defer { lock.unlock() }; return _skipUntilPTS } + set { + // Lock ordering: `HardwareVideoDecoder.skipUntilPTS` takes that + // decoder's own `skipLock`, while its output callback takes `skipLock` + // and then re-enters this decoder's `lock` via the frame handlers. + // Setting the sub-decoders while holding `lock` would nest the locks + // in the opposite order, so capture the pointers and fan out outside. + lock.lock() + _skipUntilPTS = newValue + let primary = primaryDecoder + let enhancement = enhancementDecoder + lock.unlock() + primary.skipUntilPTS = newValue + enhancement?.skipUntilPTS = newValue + } + } + private var _skipUntilPTS: CMTime? + + /// The primary decoder's own methods are internally synchronized + /// (`HardwareVideoDecoder.lock` / `skipLock`); it is safe to call without holding + /// this decoder's `lock`, and callers must do so to preserve the lock ordering + /// above. `composer` is immutable and `Sendable`. + private let primaryDecoder: any SingleLayerVideoDecoder + /// Guarded by `lock`. Captured into a local before any `decode`/`flush`/`close` + /// call, which runs after the lock is released; swapped out under the lock and + /// closed after releasing it in `flush()`/`close()`. + private var enhancementDecoder: (any SingleLayerVideoDecoder)? + private let composer: (any FelComposing)? + + /// Every outgoing frame — composed or bypassed — reaches the host through + /// here (dolby-vision-fel.md §5.7). Owned by the decoder, not the composer. + /// `DoviFelComposer` still keeps a small relay queue of its own + /// (`deliveryQueue`) that a composed frame's completion hops through + /// before calling `fulfil`. It is a §4.3 firewall against Metal's completion handler ever blocking on + /// something downstream (`DoviComposeAsyncTests. + /// doesNotDeadlockAgainstTheCallersLock` reproduces the deadlock shape + /// removing it would reopen), not a second ordering mechanism — `fulfil` + /// reorders every frame by ticket regardless of which queue called it, + /// so a bypass arriving directly from a VideoToolbox thread and a + /// composition arriving via that relay hop are indistinguishable once + /// they reach `orderedDelivery`. See `OrderedDelivery`'s own type doc + /// and dolby-vision-fel.md §5.7, §5.9 for the full justification. Sized off this + /// decoder's own bounds (`DoviFelComposer.maxFramesInFlight`, + /// `Self.maxLayerSkewFrames`) so the stall guard's threshold tracks them + /// if either changes. + private let orderedDelivery = OrderedDelivery( + maxFramesInFlight: DoviFelComposer.maxFramesInFlight, + maxLayerSkew: DualLayerVideoDecoder.maxLayerSkewFrames + ) + + /// Production use: builds the Metal composer (nil when there is no Metal + /// device, in which case every frame bypasses). Tests inject a fake to + /// control the GPU-slot wait without a GPU. + init(composer: (any FelComposing)? = DoviFelComposer()) { + self.composer = composer + let primary = HardwareVideoDecoder() + primary.useCustomBlockSource = true + self.primaryDecoder = primary + } + + /// Test seam: inject fake single-layer decoders, e.g. ones that + /// emit a pre-seek frame from inside `flush()`. + init( + composer: (any FelComposing)?, + primaryDecoder: any SingleLayerVideoDecoder, + enhancementDecoder: (any SingleLayerVideoDecoder)? + ) { + self.composer = composer + self.primaryDecoder = primaryDecoder + (self.primaryDecoder as? HardwareVideoDecoder)?.useCustomBlockSource = true + self.enhancementDecoder = enhancementDecoder + (self.enhancementDecoder as? HardwareVideoDecoder)?.useCustomBlockSource = true + } + + // Lock discipline: every `private var` below up to `lock` is guarded + // by `lock` unless stated otherwise. `decode(packet:)` runs on the demux + // thread while `handleBaseLayerFrame` / `handleEnhancementLayerFrame` run on + // VideoToolbox output threads, so any state they all touch must be read or + // written under the lock, with at most a pointer captured for use outside it. + // `nextDeliveryTicket` and `resolveDisplayMetadata` require the lock (callers + // hold it). `noteSettledBaseLayer` requires the lock. `orderedDelivery` + // is its own, separately-confined owner, reached only through `fulfil`/`reset`/ + // `setOnFrame`, never through this lock. Sub-decoder calls (`decode`/`flush`/`close`, + // `skipUntilPTS` fan-out, `open`) always run without the lock held — see the + // ordering note on `skipUntilPTS`. + private var timeBase: AVRational = AVRational(num: 1, den: 90000) + // Stream classification (dolby-vision-fel.md §5.2, invariant I9 in §5.12): decided once, from the + // first RPU's `el_type`, before the enhancement decoder opens. A per-frame + // `disable_residual_flag` is a kernel flag (`DoviComposerFlags.disableResidual`, + // honoured by the composer) and never changes these. Guarded by `lock`. + private var isFEL: Bool = false + private var isMEL: Bool = false + private var streamClassified: Bool = false + private var frameIndex: UInt32 = 0 + + private var primaryDecoderInitialized = false + /// Set by the first failed in-band open of the primary decoder. Retrying on every later packet + /// with parameter sets would only repeat the failure and leave the player on a black screen with + /// one log line per keyframe; the failure is reported once, and `decode` refuses packets after it. + /// Guarded by `lock`. + private var primaryOpenFailure: Error? + /// Fired once, on the thread that called `decode`, when the primary decoder cannot be opened from + /// in-band parameter sets: nothing can be shown, so the host should fall back or surface an error + /// (the hvcC path throws the equivalent from `open(stream:)`). + var onFatalDecodeError: (@Sendable (Error) -> Void)? { + get { lock.lock(); defer { lock.unlock() }; return _onFatalDecodeError } + set { lock.lock(); _onFatalDecodeError = newValue; lock.unlock() } + } + private var _onFatalDecodeError: (@Sendable (Error) -> Void)? + /// The error `onFatalDecodeError` reported, if any. + var fatalDecodeError: Error? { lock.lock(); defer { lock.unlock() }; return primaryOpenFailure } + /// Set by the first failed open of the enhancement decoder. Retrying on every later packet would + /// start a new VideoToolbox session attempt and write a log line per frame for the rest of the + /// session; the failure is logged once and the stream stays on the base layer (every frame takes + /// the single-layer bypass). Not fatal, unlike `primaryOpenFailure`: there is still a picture to + /// show. Cleared by `close()`. Guarded by `lock`. + private var enhancementOpenFailure: Error? + /// The error that stopped the enhancement decoder from opening, if any. + var enhancementDecoderOpenError: Error? { lock.lock(); defer { lock.unlock() }; return enhancementOpenFailure } + /// Builds the enhancement-layer session. Replaced by tests only, before the first packet. + var makeEnhancementDecoder: () -> any SingleLayerVideoDecoder = { + let decoder = HardwareVideoDecoder() + decoder.useCustomBlockSource = true + return decoder + } + private var blVPS: [UInt8]? + private var blSPS: [UInt8]? + private var blPPS: [UInt8]? + + private var elVPS: [UInt8]? + private var elSPS: [UInt8]? + private var elPPS: [UInt8]? + /// Fixed-capacity side data ring for Dolby Vision metadata (dolby-vision-fel.md §5.3). + /// Stores `{ params, dmNAL }` keyed by canonical PTS, capacity 32, no sorting. + private var sideDataRing = FrameSideDataRing() + /// Restamps each enhancement-layer packet with the time of the base-layer picture of the same POC, + /// so pairing by time pairs by picture order even when the layers' GOPs differ. Guarded by `lock`. + private var pictureAlignment = DualLayerVideoDecoder.makeAlignment() + private var blFrameQueue: [Int64: (buffer: CVPixelBuffer, pts: CMTime, ticket: UInt64, seq: UInt64, side: FrameSideData)] = [:] + private var elFrameQueue: [Int64: CVPixelBuffer] = [:] + + /// Maximum layer skew in frames (dolby-vision-fel.md §5.5): a waiting base-layer + /// frame is bypassed under its own arrival ticket once this many newer + /// base-layer frames have arrived. Starts at 3; the 1 Hz summary's maximum skew + /// counter (§5.11) is what measures whether it can tighten. + static let maxLayerSkewFrames = 3 + + /// A waiting base frame is bypassed once K newer ones have arrived, so an enhancement picture + /// restamped K or more frames back would find its partner gone: the alignment declines to place those. + static func makeAlignment() -> DualLayerPictureAlignment { + var a = DualLayerPictureAlignment() + a.maxBackwardRestamp = maxLayerSkewFrames - 1 + return a + } + /// The enhancement layer can run ahead of the 4K base layer after a seek or at startup. + /// Retain up to 10 waiting EL frames (~6 MB each) before pruning the oldest. + static let maxEnhancementLayerQueueFrames = 10 + /// Counts base-layer arrivals. A queued entry is due for bypass once + /// `blArrivalSeq - entry.seq >= maxLayerSkewFrames`. Guarded by `lock`. + private var blArrivalSeq: UInt64 = 0 + /// Recently settled (composed or bypassed) base-layer keys. An enhancement + /// frame matching one (±300 ticks) can never pair again and is dropped on + /// arrival (dolby-vision-fel.md §5.5). Not "older than the newest": output is in decode + /// order, so older-PTS base frames settle after newer ones. Guarded by `lock`. + private var settledBlKeys: [Int64] = [] + /// How many settled base-layer keys are remembered (dolby-vision-fel.md §5.5). + static let settledKeyMemory = 64 + /// Last parameters that came from a real RPU, and how many frames in a row have borrowed them. Both + /// move in decode order, in `recordSideData`. + private var lastGoodParams: DoviMetalParams? + private var lastGoodDMNAL: [UInt8]? + private var borrowedParamsRun = 0 + /// A frame whose RPU is missing or failed its CRC may borrow the previous frame's parameters + /// for up to this many frames in a row. Reshaping and NLQ parameters are constant across a scene, + /// so this beats dropping to the base layer; a longer run is not trusted. + static let maxBorrowedParamsRun = 3 + private(set) var summaryBorrowedParams = 0 + + // 1 Hz summary counters (dolby-vision-fel.md §5.11). Guarded by `lock`. + /// Compose outcomes are counted from the GPU completion thread, so they get their own lock rather + /// than `lock` (taking that one inside the delivery path perturbs the pairing races). + private let composeResultLock = NSLock() + private var summaryComposedFrames: Int = 0 + /// Frames the GPU composed since this decoder was created. Unlike `summaryComposedFrames` it is + /// never reset by the 1 Hz summary or by `flush()`, so a probe can ask "did anything compose". + /// Guarded by `composeResultLock`. + private var totalComposedFrames: Int = 0 + private var summaryComposeFailures: Int = 0 + /// Compositions started, and compositions whose frame (composed or fallback) has been handed to + /// `orderedDelivery`. Equal when nothing is on the GPU or in a completion handler. Guarded by + /// `composeResultLock`; read only by `quiesceForTesting`. + private var composeRequestsStarted = 0 + private var composeRequestsHandedOff = 0 + private var summaryBypassNoEL: Int = 0 + private var summaryBypassNoRPU: Int = 0 + private var summaryDropStaleEL: Int = 0 + private var summaryPrunedEL: Int = 0 + private var summaryMaxObservedSkew: Int = 0 + private var lastSummaryTimestamp: TimeInterval = 0 + + /// Held for the whole of `flush()` and `close()`, so two of them (the EOF flush on the demux + /// thread against a seek flush on the main actor, or either against `close()`) run one after + /// the other instead of interleaving their critical sections: without it one call could lower + /// `draining` and clear the queues while the other was still draining the hardware decoders. + /// Lock order: `lifecycleLock` before `lock`, never the other way round. Nothing on the pairing, + /// delivery or VideoToolbox callback paths takes it (`handle*Frame`, `orderedDelivery`, the + /// sub-decoders' output callbacks), so the drains inside `flush()` can wait on those paths with + /// it held; `decode(packet:)` does not take it either. + private let lifecycleLock = NSLock() + private let lock = NSRecursiveLock() + private var _feedEpoch: UInt64 = 0 + /// True while `flush()` is draining the hardware decoders (dolby-vision-fel.md + /// §5.8). `HardwareVideoDecoder.flush()` waits for the frames VideoToolbox + /// still owes, which arrive through the normal callbacks while the host + /// thread is blocked inside it; those are pre-seek frames and are dropped + /// on arrival rather than paired, ticketed or delivered. Guarded by `lock`. + private var draining = false + /// Single use (dolby-vision-fel.md §5.8): set by `close()`, after which `open` + /// refuses with `VideoDecoderError.decoderClosed`. The host builds a new + /// decoder per session, so nothing reopens one; the flag turns a latent + /// stale-state reuse into a loud error. Guarded by `lock`. + private var isClosed = false + + // MARK: - Ordered delivery + // + // Composition is asynchronous, so a frame's pixels are ready on a Metal + // completion thread rather than at the point the decoder handed them over. + // Every outgoing frame — composed, bypassed or pruned — takes a ticket + // here, under `lock`, in decode order, at base-layer arrival. + // Ordering behind that ticket, and `emit` itself, live entirely in + // `orderedDelivery` from here on (dolby-vision-fel.md §5.7): once a ticket is + // issued, this decoder's only remaining job is to hand `orderedDelivery` + // the ticket, the epoch it was issued in, and the finished frame. + private var deliveryNextTicket: UInt64 = 0 + + var feedEpoch: UInt64 { + lock.lock(); defer { lock.unlock() }; return _feedEpoch + } + + static func canonicalKey(pts: CMTime) -> Int64 { + guard pts.isNumeric else { return Int64.min } + return pts.convertScale(90000, method: .roundHalfAwayFromZero).value + } + + static func canonicalKey(rawPTS: Int64, timeBase: AVRational) -> Int64 { + guard rawPTS != Int64.min else { return Int64.min } + // Stream timestamps are untrusted: a product that overflows has no usable key. + let (value, overflow) = rawPTS.multipliedReportingOverflow(by: Int64(timeBase.num)) + guard !overflow else { return Int64.min } + let pts = CMTimeMake(value: value, timescale: max(timeBase.den, 1)) + return canonicalKey(pts: pts) + } + + // MARK: - Lifecycle + + func open(stream: UnsafeMutablePointer, onFrame: @escaping DecodedFrameHandler) throws { + // Single use: refuse before touching `stream` or mutating anything. + lock.lock() + let closed = isClosed + let hasComposer = composer != nil + lock.unlock() + guard !closed else { throw VideoDecoderError.decoderClosed } + // Fail fast if no composer (no Metal device or shader compilation failed). + // The host can then fall back to the native 8.1 route instead of decoding the EL for nothing. + guard hasComposer else { + EngineLog.emit("[DualLayerVideoDecoder] No composer available; FEL composition disabled", category: .swPlayback) + throw VideoDecoderError.composerUnavailable + } + (primaryDecoder as? HardwareVideoDecoder)?.useCustomBlockSource = true + if let codecpar = stream.pointee.codecpar { + lock.lock() + declaredFormat = (ColorDescription(codecpar: codecpar), codecpar.pointee.codec_id, codecpar.pointee.profile) + lock.unlock() + } + self.onFrame = { [weak self] buffer, pts, extra in + self?.reportDecodedFormat(buffer) + onFrame(buffer, pts, extra) + } + lock.lock() + self.timeBase = stream.pointee.time_base + // Only an hvcC record can build the session up front. Extradata of that size that is something + // else (an Annex B parameter-set blob, say) takes the in-band path, where the parameter + // sets are read from the packets, instead of throwing out of `open`. + var hasHVCC = false + if (stream.pointee.codecpar?.pointee.extradata_size ?? 0) > 23, + let par = stream.pointee.codecpar?.pointee, let ed = par.extradata, ed[0] == 1 { + hasHVCC = true + // hvcC: the base layer's SPS/PPS may never repeat in band. + pictureAlignment.seedBaseParameterSets(parseHVCCParameterSets(ed, Int(par.extradata_size))) + } + lock.unlock() + + if hasHVCC { + // `HardwareVideoDecoder.open` builds the VT session; it synchronizes + // internally, so this runs without `lock` held (see the lock ordering note above). + try primaryDecoder.open(stream: stream) { [weak self] baseBuffer, pts, hdr10Data in + self?.handleBaseLayerFrame(baseBuffer: baseBuffer, pts: pts, hdr10Data: hdr10Data) + } + lock.lock() + primaryDecoderInitialized = true + lock.unlock() + } else { + // Parameter sets are carried in-band in the video packets (e.g. hev1 MP4) + lock.lock() + primaryDecoderInitialized = false + lock.unlock() + } + } + + // MARK: - Decode Loop + + func decode(packet: UnsafeMutablePointer, epoch: UInt64? = nil) { + // Single snapshot under the lock: the epoch gate, the primary + // readiness flag and the time base are all shared with the VT threads. + // A closed decoder refuses packets outright: without this, + // `ensurePrimaryDecoder` could reopen the primary session after `close()`. + lock.lock() + if isClosed { + lock.unlock() + return + } + if let epoch, epoch != _feedEpoch { + lock.unlock() + return + } + // The epoch this packet entered under, whether or not the caller passed one: a `flush()` that + // lands while the packet is being parsed below must stop it from touching post-flush state. + let entryEpoch = _feedEpoch + let primed = primaryDecoderInitialized + let tb = timeBase + lock.unlock() + + if !primed { + ensurePrimaryDecoder(from: packet) + } + lock.lock() + guard primaryDecoderInitialized else { lock.unlock(); return } + lock.unlock() + + let ptsRaw = packet.pointee.pts + let ptsKey = Self.canonicalKey(rawPTS: ptsRaw, timeBase: tb) + + // Extract EL NALs, RPU, and rewrite Base Layer packet to Profile 8.1. + // Whether to extract is read from the (immutable once classified) stream state. + // If not composing, skip parameter extraction entirely. + // Per dolby-vision-fel.md §5.3, §5.4: VideoToolbox format description carries no dvcC, + // so VT skips unspec62/63 for both FEL and MEL without rebuilding the base-layer packet. + let (skipEL, canCompose): (Bool, Bool) = { + lock.lock() + defer { lock.unlock() } + let composing = !isClosed && !isMEL && composer != nil + return (isMEL, composing) + }() + // One-shot stream classification: the first definitive RPU decides, later + // RPUs — including per-frame `disable_residual_flag`, which rides to the kernels + // as `DoviComposerFlags.disableResidual` — never change it. In particular this + // never closes the enhancement decoder; decoders are only torn down in + // `flush()`/`close()`, never from `decode(packet:)`. + let layers = DoviRpuConverter.separateDualLayers( + packet, + extractEL: !skipEL, + useCustomBlockSource: true, + extractParams: canCompose + ) + applyStreamClassification(DoviRpuConverter.classifyStream(layers)) + + // One critical section for everything this packet writes into shared state: the side-data + // record (which also moves `lastGoodParams`) and the picture-order restamp. A `flush()` or + // `close()` that landed while `separateDualLayers` ran (outside the lock) means this packet + // belongs to the old feed: recording it would poison the post-flush state, or undo a close. + lock.lock() + guard !isClosed, _feedEpoch == entryEpoch else { + lock.unlock() + var stale = layers.elPacket + trackedPacketFree(&stale) + return + } + if ptsKey != Int64.min { + // In decode order, here, so every frame carries concrete parameters by the time it pairs: + // compose time runs in completion order, which is not stream order. + recordSideData(key: ptsKey, params: layers.rpuParams, dmNAL: layers.convertedRpu, + usesPrevious: layers.usesPreviousMapping, disableResidual: layers.disableResidual) + } + + // The enhancement decoder opens for a classified FEL stream only + // (dolby-vision-fel.md §5.2): MEL never opens it, and the EL keeps decoding across RPUs + // with `disable_residual_flag` set. + // Also require a composer; without one, never engage the EL decoder. + let engageEL = isFEL && !isMEL && composer != nil + if engageEL, let data = packet.pointee.data { + // `separateDualLayers` left the packet intact (custom block source), so both layers are + // still in it. Every unit is read, with or without an EL, to keep the POC state current. + let wasReordered = pictureAlignment.reorderedUnits > 0 + let elPTS = pictureAlignment.enhancementPTS(data, Int(packet.pointee.size), framing: .lengthPrefixed(size: 4), pts: ptsRaw) + let nowReordered = pictureAlignment.reorderedUnits > 0 + lock.unlock() + layers.elPacket?.pointee.pts = elPTS + if nowReordered && !wasReordered { + EngineLog.emit("[DualLayerVideoDecoder] enhancement layer is reordered against the base layer; pairing by picture order", category: .swPlayback) + } + } else { + lock.unlock() + } + if engageEL, let elPacket = layers.elPacket { + ensureEnhancementDecoder(from: elPacket) + // Capture under the lock, decode outside it: `close()`/`flush()` may + // swap the decoder out concurrently, and the sub-decoder call must not + // nest inside this lock. Re-check `isClosed` from + // the same snapshot: a `close()` that landed after the entry gate + // must not feed the (possibly freshly installed) enhancement decoder. + lock.lock() + let el = enhancementDecoder + let closedAfterEnsure = isClosed + lock.unlock() + if !closedAfterEnsure { + el?.decode(packet: elPacket, epoch: epoch) + } + var elOpt: UnsafeMutablePointer? = elPacket + trackedPacketFree(&elOpt) + } else if let elPacket = layers.elPacket { + var elOpt: UnsafeMutablePointer? = elPacket + trackedPacketFree(&elOpt) + } + + // Decode Base Layer packet. + // + // With `useCustomBlockSource` the packet is untouched: `separateDualLayers` only reads it, so + // the base layer's own RPU NAL (unspec62) is still inside it, which + // looks alarming next to a composer that is about to apply the same + // reshaping curve itself. It is inert: the primary decoder's format + // description is built from the base layer's VPS/SPS/PPS alone (see + // `ensurePrimaryDecoder`) with no `dvcC`/`dvvC` record, so VideoToolbox + // decodes plain HEVC and skips the unspec62 NAL as an unknown type. The + // curve is applied exactly once, on the GPU. (`separateDualLayers` also backs + // `convertPacketToProfile81`, the remux route, where it does rewrite the packet to + // Profile 8.1 and the container declares a `dvcC`; that is not this path.) + // + // Re-check `isClosed` before feeding the primary — a `close()` + // that landed after the entry gate must not reopen/feed a torn-down session. + lock.lock() + let closedBeforePrimary = isClosed + lock.unlock() + guard !closedBeforePrimary else { return } + primaryDecoder.decode(packet: packet, epoch: epoch) + } + + /// Applies one RPU's stream classification. The first definitive report wins and every + /// later one is ignored, so one frame can never shut off the enhancement layer for the + /// rest of the session (invariant I9, dolby-vision-fel.md §5.12). Called from `decode(packet:)` — which + /// never closes a decoder — and from the test seam below. + private func applyStreamClassification(_ kind: DoviRpuConverter.StreamKind) { + lock.lock() + defer { lock.unlock() } + guard !streamClassified else { return } + switch kind { + case .undetermined: + return + case .fel: + isFEL = true + isMEL = false + streamClassified = true + EngineLog.emit("[DualLayerVideoDecoder] FEL stream detected; engaging enhancement decoder", category: .swPlayback) + case .mel: + isFEL = false + isMEL = true + streamClassified = true + EngineLog.emit("[DualLayerVideoDecoder] MEL stream detected; secondary decoder bypassed, routing Base Layer directly", category: .swPlayback) + } + } + + // Lazily opens the primary decoder from in-band parameter sets. Only the demux + // thread reaches here (from `decode`), but `primaryDecoderInitialized`, the + // accumulated `blVPS/SPS/PPS` and `timeBase` are shared with the VT threads, so + // every touch is under `lock`; the blocking `open` runs outside it. + // Resurrection guard: `decode()` passes the entry gate before + // `close()` runs, so re-check `isClosed` before opening and again before + // installing — a post-close open is torn down immediately instead of + // leaving a live VT session in a closed decoder. + private func ensurePrimaryDecoder(from packet: UnsafeMutablePointer) { + lock.lock() + guard !primaryDecoderInitialized, !isClosed, primaryOpenFailure == nil else { lock.unlock(); return } + var v = blVPS + var s = blSPS + var p = blPPS + let tb = timeBase + lock.unlock() + guard let data = packet.pointee.data, packet.pointee.size > 4 else { return } + let size = Int(packet.pointee.size) + + A53SEIParser.forEachNAL(data, size, .lengthPrefixed(size: 4)) { nal, len in + let nalType = (nal[0] >> 1) & 0x3F + let bytes = [UInt8](UnsafeBufferPointer(start: nal, count: len)) + switch nalType { + case 32: v = bytes + case 33: s = bytes + case 34: p = bytes + default: break + } + } + + guard let vv = v, let ss = s, let pp = p else { + // Incomplete set: accumulate what this packet carried for the next one. + lock.lock() + if blVPS == nil { blVPS = v } + if blSPS == nil { blSPS = s } + if blPPS == nil { blPPS = p } + lock.unlock() + return + } + do { + try primaryDecoder.open(parameterSets: [vv, ss, pp], timeBase: tb) { [weak self] baseBuffer, pts, hdr10Data in + self?.handleBaseLayerFrame(baseBuffer: baseBuffer, pts: pts, hdr10Data: hdr10Data) + } + lock.lock() + blVPS = vv + blSPS = ss + blPPS = pp + if isClosed || primaryDecoderInitialized { + // Lost the race with `close()` (or another opener): the session + // just built belongs to a closed decoder — tear it down outside + // the lock instead of marking it initialized. + lock.unlock() + primaryDecoder.close() + return + } + primaryDecoderInitialized = true + lock.unlock() + EngineLog.emit("[DualLayerVideoDecoder] primary decoder initialized from in-band parameter sets", category: .swPlayback) + } catch { + // Fail once, visibly: the same sets would fail the same way on every later packet. + lock.lock() + let firstFailure = primaryOpenFailure == nil + if firstFailure { primaryOpenFailure = error } + let handler = _onFatalDecodeError + lock.unlock() + guard firstFailure else { return } + EngineLog.emit("[DualLayerVideoDecoder] failed to open the primary decoder from in-band parameter sets, no video will be produced: \(error)", category: .swPlayback) + handler?(error) + } + } + + // Lazily opens the enhancement decoder. Same discipline as + // `ensurePrimaryDecoder`: shared state under `lock`, VT session build outside + // it, install under the lock only if still absent **and** the decoder is not + // closed (resurrection guard: a thread past the `decode()` entry gate + // that reaches the install after `close()` cleared `enhancementDecoder` to + // nil must close its spare instead of planting a live VT session into the + // closed decoder). + func ensureEnhancementDecoder(from packet: UnsafeMutablePointer) { + lock.lock() + // Never open the EL decoder without a composer. In production this is + // redundant (`engageEL` in `decode()` already gates on `composer != nil`), but + // it makes the invariant explicit and safe against future refactors. + guard enhancementDecoder == nil, !isClosed, composer != nil, enhancementOpenFailure == nil else { lock.unlock(); return } + var v = elVPS + var s = elSPS + var p = elPPS + let skip = _skipUntilPTS + let tb = timeBase + lock.unlock() + guard let data = packet.pointee.data, packet.pointee.size > 4 else { return } + let size = Int(packet.pointee.size) + + A53SEIParser.forEachNAL(data, size, .lengthPrefixed(size: 4)) { nal, len in + let nalType = (nal[0] >> 1) & 0x3F + let bytes = [UInt8](UnsafeBufferPointer(start: nal, count: len)) + switch nalType { + case 32: v = bytes + case 33: s = bytes + case 34: p = bytes + default: break + } + } + + guard let vv = v, let ss = s, let pp = p else { + lock.lock() + if elVPS == nil { elVPS = v } + if elSPS == nil { elSPS = s } + if elPPS == nil { elPPS = p } + lock.unlock() + return + } + let decoder = makeEnhancementDecoder() + decoder.skipUntilPTS = skip + do { + try decoder.open(parameterSets: [vv, ss, pp], timeBase: tb) { [weak self] elBuffer, pts, _ in + self?.handleEnhancementLayerFrame(elBuffer: elBuffer, pts: pts) + } + lock.lock() + elVPS = vv + elSPS = ss + elPPS = pp + if enhancementDecoder == nil, !isClosed { + // Same epoch as this decoder: a `flush()` before the first EL packet has already moved + // `_feedEpoch`, and a decoder left at epoch 0 would drop every packet it is fed. + // Safe under `lock`: the new decoder is not shared yet, so nothing else holds its lock. + decoder.adoptFeedEpoch(_feedEpoch) + enhancementDecoder = decoder + lock.unlock() + EngineLog.emit("[DualLayerVideoDecoder] secondary decoder initialized for the enhancement layer", category: .swPlayback) + } else { + // Lost the race with another opener, or `close()` cleared the + // slot (and set `isClosed`) while the session was building: do + // not plant a live VT session into the closed decoder; tear the + // spare down outside the lock. + lock.unlock() + decoder.close() + } + } catch { + // Fail once: the same sets would fail the same way on every later packet. + lock.lock() + elVPS = vv + elSPS = ss + elPPS = pp + let firstFailure = enhancementOpenFailure == nil + if firstFailure { enhancementOpenFailure = error } + lock.unlock() + guard firstFailure else { return } + EngineLog.emit("[DualLayerVideoDecoder] failed to initialize EL decoder, continuing with the base layer only: \(error)", category: .swPlayback) + } + } + + // MARK: - Frame Emission + + /// Reserves this frame's place in the output order. The caller holds `lock`. + /// + /// Every ticket must reach ``OrderedDelivery/fulfil(ticket:epoch:_:)`` or the + /// queue behind it stalls — `orderedDelivery`'s own stall guard is the backstop + /// if one never does; a flush or close retires outstanding tickets wholesale. + private func nextDeliveryTicket() -> UInt64 { + let t = deliveryNextTicket + deliveryNextTicket &+= 1 + return t + } + + /// Looks up and removes this frame's Profile 8.1 display-metadata NAL, + /// within the same jitter tolerance as the RPU cache. The caller holds + /// `lock`. + /// + /// Resolved once, at base-layer arrival — the same point `hdr10Data` + /// already enters the system as a parameter — so it can travel with the + /// frame instead of living in a cache `OrderedDelivery.emit` would + /// otherwise have to reach back into without the decoder's lock + /// (dolby-vision-fel.md §5.7). + private func resolveDisplayMetadata(key: Int64) -> [UInt8]? { + sideDataRing.removeDisplayMetadata(key: key) + } + + // MARK: - Frame Pairing & Metal Composition + // + // dolby-vision-fel.md §5.5, §5.6: decide under the lock, act outside it. Each + // handler holds `lock` only while it matches queues, consumes caches and + // issues the delivery ticket. The GPU slot wait, encode and commit happen + // after the lock is released (see `runCompose`), so backpressure on the + // calling VideoToolbox thread never stalls `orderedDelivery`, the other + // layer's callback, `flush()` or `close()`. + + /// What one frame handler decided while holding `lock`. Payloads are a + /// baton pass: the handler stops touching them once they are in here, and + /// `runCompose` / `orderedDelivery` is the only reader after that. + /// `dmNAL` and `epoch` are resolved under the same lock hold that issued + /// `ticket` (dolby-vision-fel.md §5.7): `dmNAL` so `OrderedDelivery.emit` never + /// touches `dmRpuCache`, `epoch` so a late completion can tell a stale + /// delivery from a current one without the decoder lock. + private struct ComposeRequest { + let ticket: UInt64 + let epoch: UInt64 + let base: CVPixelBuffer + let enhancement: CVPixelBuffer + let params: DoviMetalParams + let frameIndex: UInt32 + let pts: CMTime + let hdr10Data: Data? + let dmNAL: [UInt8]? + /// Which arrival path produced this, for the compose-failure log. + let arrived: String + } + + private struct BypassRequest { + let ticket: UInt64 + let epoch: UInt64 + let buffer: CVPixelBuffer + let pts: CMTime + let hdr10Data: Data? + let dmNAL: [UInt8]? + let reason: String + } + + private enum PairingOutcome { + case queued + case bypass(BypassRequest) + case compose(ComposeRequest) + } + + /// Hands one bypass request straight to `orderedDelivery`. No lock held. + private func fulfilBypass(_ req: BypassRequest) { + orderedDelivery.fulfil( + ticket: req.ticket, epoch: req.epoch, + DeliveredFrame(buffer: SendableBuffer(value: req.buffer), pts: req.pts, hdr10Data: req.hdr10Data, dmNAL: req.dmNAL) + ) + } + + /// Acquires the GPU slot with no lock held, then encodes and commits. The + /// completed frame (or the base layer when composition fails) reaches + /// `orderedDelivery`, which reorders by ticket. The completion closure + /// captures `orderedDelivery` strongly and the decoder weakly for counters + /// (dolby-vision-fel.md §5.7): a completion that outlives the decoder + /// still calls `fulfil`, which drops it once `orderedDelivery`'s epoch has + /// moved on, instead of silently no-oping on a `weak self`. + private func runCompose(_ req: ComposeRequest) { + guard let composer else { + orderedDelivery.fulfil( + ticket: req.ticket, epoch: req.epoch, + DeliveredFrame(buffer: SendableBuffer(value: req.base), pts: req.pts, hdr10Data: req.hdr10Data, dmNAL: req.dmNAL) + ) + return + } + if let concrete = composer as? DoviFelComposer { + FelResidualMeasure.sampleIfDue(composer: concrete, base: req.base, enhancement: req.enhancement, + params: req.params, frameIndex: req.frameIndex, pts: req.pts) + } + composeResultLock.lock(); composeRequestsStarted += 1; composeResultLock.unlock() + composer.acquireSlot() + let fallback = SendableBuffer(value: req.base) + // Scalars (plus the strongly-captured `delivery`) copied out so the + // `@Sendable` completion captures no CoreVideo types (Swift 6 mode + // rejects the struct itself). + let ticket = req.ticket + let epoch = req.epoch + let pts = req.pts + let hdr10Data = req.hdr10Data + let dmNAL = req.dmNAL + let arrived = req.arrived + let delivery = orderedDelivery + composer.composeAsyncWithAcquiredSlot( + baseBuffer: req.base, + enhancementBuffer: req.enhancement, + params: req.params, + frameIndex: req.frameIndex + ) { [weak self] composed in + self?.recordComposeResult(success: composed != nil) + if composed == nil { + EngineLog.emit("[DualLayer] COMPOSE FAILED (\(arrived)): pts=\(String(format: "%.3f", pts.seconds))", category: .swPlayback, level: .verbose) + } + let buffer = SendableBuffer(value: composed ?? fallback.value) + delivery.fulfil(ticket: ticket, epoch: epoch, DeliveredFrame(buffer: buffer, pts: pts, hdr10Data: hdr10Data, dmNAL: dmNAL)) + self?.composeResultLock.lock(); self?.composeRequestsHandedOff += 1; self?.composeResultLock.unlock() + } + } + + private func recordComposeResult(success: Bool) { + composeResultLock.lock() + if success { summaryComposedFrames += 1; totalComposedFrames += 1 } else { summaryComposeFailures += 1 } + composeResultLock.unlock() + } + + /// Frames composed on the GPU so far (bypassed and failed frames are not counted). + var composedFrameCount: Int { + composeResultLock.lock(); defer { composeResultLock.unlock() } + return totalComposedFrames + } + + /// Internal rather than private so `DualLayerVideoDecoderTests` can drive it directly. + func handleBaseLayerFrame(baseBuffer: CVPixelBuffer, pts: CMTime, hdr10Data: Data?) { + // The ticket is issued when the frame arrives, so a frame that + // waits for a late enhancement layer keeps its place in decode order. + // The arrival may also push older waiting frames past K newer frames, + // which go out here under their own tickets (`due`). + let outcome: PairingOutcome + let due: [BypassRequest] + (outcome, due) = { + lock.lock() + defer { lock.unlock() } + + // Defense in depth: VT callbacks still in flight when + // `close()` runs must not pair, ticket or queue into cleared state. + guard !isClosed else { return (.queued, []) } + // Pre-seek frame drained by `flush()`: drop before it can + // take a ticket or occupy a queue slot meant for post-seek frames. + guard !draining else { return (.queued, []) } + guard isFEL, !isMEL, enhancementDecoder != nil, composer != nil else { + // Not a FEL stream or secondary decoder not engaged (e.g. MEL or P8.1 fallback): emit Base Layer directly + EngineLog.emit("[DualLayer] Bypass: isFEL=\(isFEL), isMEL=\(isMEL), enhDecoder=\(enhancementDecoder != nil)", category: .swPlayback, level: .verbose) + if isFEL && !isMEL && enhancementDecoder == nil { + summaryBypassNoEL += 1 + } + let singleLayerKey = Self.canonicalKey(pts: pts) + let dmNAL = resolveDisplayMetadata(key: singleLayerKey) + return (.bypass(BypassRequest(ticket: nextDeliveryTicket(), epoch: _feedEpoch, buffer: baseBuffer, pts: pts, hdr10Data: hdr10Data, dmNAL: dmNAL, reason: "single-layer")), []) + } + + let key = Self.canonicalKey(pts: pts) + let ticket = nextDeliveryTicket() + let epoch = _feedEpoch + let dmNAL = resolveDisplayMetadata(key: key) + blArrivalSeq &+= 1 + let seq = blArrivalSeq + + var primary: PairingOutcome + // A frame without a timestamp has nothing to pair on: it goes out as its base layer now, under + // the ticket it already holds, rather than waiting for a partner that cannot match it. + guard key != Int64.min else { + EngineLog.emit("[DualLayer] Bypass: base-layer frame without a timestamp", category: .swPlayback, level: .verbose) + return (.bypass(BypassRequest(ticket: ticket, epoch: epoch, buffer: baseBuffer, pts: pts, hdr10Data: hdr10Data, dmNAL: dmNAL, reason: "no-pts")), collectDueBypasses()) + } + let mKey: Int64? + if elFrameQueue[key] != nil { + mKey = key + } else { + // Search within +/- 300 ticks (~3.3ms) for clock jitter tolerance + mKey = elFrameQueue.keys.first(where: { FrameSideDataRing.isWithinJitter($0, key) }) + } + + if let mKey, let elBuffer = elFrameQueue.removeValue(forKey: mKey) { + // Matching EL frame is already ready: compose on the Metal GPU outside the lock + guard let params = takeParams(key: key) else { + // S1: Missing RPU metadata: do NOT synthesize identity NLQ. Emit base layer directly. + EngineLog.emit("[DualLayer] Bypass: missing RPU for key=\(key)", category: .swPlayback, level: .verbose) + summaryBypassNoRPU += 1 + noteSettledBaseLayer(key: key) + primary = .bypass(BypassRequest(ticket: ticket, epoch: epoch, buffer: baseBuffer, pts: pts, hdr10Data: hdr10Data, dmNAL: dmNAL, reason: "missing-rpu")) + return (primary, collectDueBypasses()) + } + frameIndex &+= 1 + noteSettledBaseLayer(key: key) + primary = .compose(ComposeRequest(ticket: ticket, epoch: epoch, base: baseBuffer, enhancement: elBuffer, params: params, frameIndex: frameIndex, pts: pts, hdr10Data: hdr10Data, dmNAL: dmNAL, arrived: "BL arrived")) + } else { + // EL frame has not arrived yet: queue Base Layer frame under its ticket + EngineLog.emit("[DualLayer] QUEUE BL: pts=\(String(format: "%.3f", pts.seconds)) key=\(key), elCount=\(elFrameQueue.count)", category: .swPlayback, level: .verbose) + var displaced: BypassRequest? + if let old = blFrameQueue[key] { + // Same canonical key twice: the old entry's ticket would stall + // delivery forever if dropped, so bypass it under its ticket, + // carrying its stored side data. + displaced = BypassRequest(ticket: old.ticket, epoch: epoch, buffer: old.buffer, pts: old.pts, hdr10Data: old.side.hdr10Data, dmNAL: old.side.dmNAL, reason: "duplicate-key") + noteSettledBaseLayer(key: key) + } + blFrameQueue[key] = (buffer: baseBuffer, pts: pts, ticket: ticket, seq: seq, side: FrameSideData(hdr10Data: hdr10Data, dmNAL: dmNAL)) + let currentSkew = max(blFrameQueue.count, elFrameQueue.count) + if currentSkew > summaryMaxObservedSkew { + summaryMaxObservedSkew = currentSkew + } + primary = .queued + var due = collectDueBypasses() + if let displaced { due.insert(displaced, at: 0) } + return (primary, due) + } + return (primary, collectDueBypasses()) + }() + + // Older tickets first; `orderedDelivery` reorders by ticket anyway. + for req in due { + fulfilBypass(req) + } + switch outcome { + case .queued: + break + case .bypass(let req): + fulfilBypass(req) + case .compose(let req): + runCompose(req) + } + maybeEmitSummary() + } + + /// Internal rather than private so `DualLayerVideoDecoderTests` can drive it directly. + func handleEnhancementLayerFrame(elBuffer: CVPixelBuffer, pts: CMTime) { + let outcome: PairingOutcome = { + lock.lock() + defer { lock.unlock() } + + // Defense in depth: see `handleBaseLayerFrame`. + guard !isClosed else { return .queued } + // Pre-seek frame drained by `flush()`: drop before it can + // pair or occupy a queue slot meant for post-seek frames. + guard !draining else { return .queued } + guard composer != nil else { return .queued } + let key = Self.canonicalKey(pts: pts) + // No timestamp, no partner: it cannot pair with anything. + guard key != Int64.min else { return .queued } + // This enhancement layer's base layer already went out + // (composed or bypassed), so it can never pair; drop it. + if settledBlKeys.contains(where: { FrameSideDataRing.isWithinJitter($0, key) }) { + EngineLog.emit("[DualLayer] DROP stale EL: key=\(key), already settled", category: .swPlayback, level: .verbose) + summaryDropStaleEL += 1 + return .queued + } + var matchingBlKey: Int64? + if blFrameQueue[key] != nil { + matchingBlKey = key + } else { + matchingBlKey = blFrameQueue.keys.first(where: { FrameSideDataRing.isWithinJitter($0, key) }) + } + + if let mKey = matchingBlKey, let waiting = blFrameQueue.removeValue(forKey: mKey) { + // Base Layer frame was waiting: compose on the Metal GPU outside the lock, + // under the ticket its arrival took. + let epoch = _feedEpoch + let dmNAL = waiting.side.dmNAL + guard let params = takeParams(key: mKey) else { + // S1: Missing RPU metadata: do NOT synthesize identity NLQ. Emit base layer directly. + EngineLog.emit("[DualLayer] Bypass: missing RPU for key=\(mKey)", category: .swPlayback, level: .verbose) + summaryBypassNoRPU += 1 + noteSettledBaseLayer(key: mKey) + return .bypass(BypassRequest(ticket: waiting.ticket, epoch: epoch, buffer: waiting.buffer, pts: waiting.pts, hdr10Data: waiting.side.hdr10Data, dmNAL: waiting.side.dmNAL, reason: "missing-rpu")) + } + frameIndex &+= 1 + noteSettledBaseLayer(key: mKey) + return .compose(ComposeRequest(ticket: waiting.ticket, epoch: epoch, base: waiting.buffer, enhancement: elBuffer, params: params, frameIndex: frameIndex, pts: waiting.pts, hdr10Data: waiting.side.hdr10Data, dmNAL: dmNAL, arrived: "EL arrived")) + } else { + // Base Layer frame has not arrived yet: queue EL frame under its own larger cap + EngineLog.emit("[DualLayer] QUEUE EL: pts=\(String(format: "%.3f", pts.seconds)) key=\(key), blCount=\(blFrameQueue.count)", category: .swPlayback, level: .verbose) + elFrameQueue[key] = elBuffer + let currentSkew = max(blFrameQueue.count, elFrameQueue.count) + if currentSkew > summaryMaxObservedSkew { + summaryMaxObservedSkew = currentSkew + } + if elFrameQueue.count > Self.maxEnhancementLayerQueueFrames { + let sortedKeys = elFrameQueue.keys.sorted() + for k in sortedKeys.prefix(elFrameQueue.count - Self.maxEnhancementLayerQueueFrames) { + EngineLog.emit("[DualLayer] PRUNED EL: key=\(k)", category: .swPlayback, level: .verbose) + elFrameQueue.removeValue(forKey: k) + summaryPrunedEL += 1 + } + } + return .queued + } + }() + + switch outcome { + case .queued: + break + case .bypass(let req): + fulfilBypass(req) + case .compose(let req): + runCompose(req) + } + maybeEmitSummary() + } + + /// Resolves what one packet's RPU means for its frame and records it for pairing. Runs in decode order, + /// from `decode(packet:)`, so "the previous mapping" is the previous frame in the stream and not whichever + /// frame happened to finish composing last. The caller holds `lock`. + /// + /// - A real RPU is the frame's parameters, and the new "previous mapping". + /// - `use_prev_vdr_rpu_flag` keeps the previous mapping by definition, for as long as the stream keeps + /// asking. + /// - A missing or corrupt RPU may borrow the previous mapping for up to `maxBorrowedParamsRun` frames in + /// a row; reshaping and NLQ parameters are constant across a scene. Past that the frame carries none + /// and goes out as its base layer. + private func recordSideData(key: Int64, params: DoviMetalParams?, dmNAL: [UInt8]?, usesPrevious: Bool, disableResidual: Bool = false) { + var resolvedParams = params + var resolvedNAL = dmNAL + if let params { + lastGoodParams = params + lastGoodDMNAL = dmNAL + borrowedParamsRun = 0 + } else if let last = lastGoodParams { + if usesPrevious { + // Only `prev_vdr_rpu_id` 0 is followed (the previous RPU in decode order), as on every + // surveyed stream. The curves and NLQ come from that RPU; the residual switch is this + // frame's own header flag, not the previous frame's. + var borrowed = last + if disableResidual { borrowed.flags.insert(.disableResidual) } else { borrowed.flags.remove(.disableResidual) } + resolvedParams = borrowed + resolvedNAL = dmNAL ?? lastGoodDMNAL + } else if borrowedParamsRun < Self.maxBorrowedParamsRun { + borrowedParamsRun += 1 + summaryBorrowedParams += 1 + resolvedParams = last + resolvedNAL = dmNAL ?? lastGoodDMNAL + EngineLog.emit("[DualLayer] RPU missing for key=\(key); reusing the previous frame's parameters (\(borrowedParamsRun)/\(Self.maxBorrowedParamsRun))", category: .swPlayback, level: .verbose) + } + } + guard resolvedParams != nil || resolvedNAL != nil else { return } + sideDataRing.insert(key: key, params: resolvedParams, dmNAL: resolvedNAL) + } + + /// The parameters `recordSideData` resolved for `key`, nil when the frame has none. The caller holds `lock`. + private func takeParams(key: Int64) -> DoviMetalParams? { + sideDataRing.removeParams(key: key) + } + + /// Records a settled base-layer key and drops queued enhancement frames that + /// can never pair with a future base layer (dolby-vision-fel.md §5.5). The caller + /// holds `lock`. + private func noteSettledBaseLayer(key: Int64) { + // Keyed match, not "older than the newest": both layers come out of VideoToolbox in + // decode order, so an older-PTS base frame legitimately settles after a newer one. + settledBlKeys.append(key) + if settledBlKeys.count > Self.settledKeyMemory { settledBlKeys.removeFirst(settledBlKeys.count - Self.settledKeyMemory) } + for k in elFrameQueue.keys where FrameSideDataRing.isWithinJitter(k, key) { + EngineLog.emit("[DualLayer] DROP stale EL: key=\(k), base \(key) settled", category: .swPlayback, level: .verbose) + elFrameQueue.removeValue(forKey: k) + summaryDropStaleEL += 1 + } + } + + /// Bypasses every queued base-layer frame that has seen K newer base-layer + /// arrivals, oldest first, each under the ticket its arrival took. + /// The caller holds `lock`. + private func collectDueBypasses() -> [BypassRequest] { + var due: [BypassRequest] = [] + for k in blFrameQueue.keys.sorted() { + guard let entry = blFrameQueue[k], + blArrivalSeq &- entry.seq >= UInt64(Self.maxLayerSkewFrames) else { continue } + blFrameQueue.removeValue(forKey: k) + summaryBypassNoEL += 1 + EngineLog.emit("[DualLayer] BYPASS no-EL: pts=\(String(format: "%.3f", entry.pts.seconds)) key=\(k)", category: .swPlayback, level: .verbose) + noteSettledBaseLayer(key: k) + due.append(BypassRequest(ticket: entry.ticket, epoch: _feedEpoch, buffer: entry.buffer, pts: entry.pts, hdr10Data: entry.side.hdr10Data, dmNAL: entry.side.dmNAL, reason: "no-el")) + } + // Every arrival takes a new `seq`, so what survives the loop arrived within the last + // `maxLayerSkewFrames` base frames: the queue is bounded by the skew rule alone. + assert(blFrameQueue.count <= Self.maxLayerSkewFrames) + return due + } + + #if DEBUG + var summaryTimestampProviderForTesting: (() -> TimeInterval)? + #endif + + // MARK: - Observability (dolby-vision-fel.md §5.11) + + /// Seconds between `[DualLayer] summary` lines (dolby-vision-fel.md §5.11). + static let summaryIntervalSeconds: TimeInterval = 1.0 + + func maybeEmitSummary() { + #if DEBUG + let now = summaryTimestampProviderForTesting?() ?? ProcessInfo.processInfo.systemUptime + #else + let now = ProcessInfo.processInfo.systemUptime + #endif + lock.lock() + guard isFEL, !isMEL else { + lock.unlock() + return + } + if lastSummaryTimestamp == 0 { + lastSummaryTimestamp = now + lock.unlock() + return + } + guard now - lastSummaryTimestamp >= Self.summaryIntervalSeconds else { + lock.unlock() + return + } + composeResultLock.lock() + let composed = summaryComposedFrames + let composeFailures = summaryComposeFailures + summaryComposedFrames = 0 + summaryComposeFailures = 0 + composeResultLock.unlock() + let noEL = summaryBypassNoEL + let noRPU = summaryBypassNoRPU + let staleEL = summaryDropStaleEL + let prunedEL = summaryPrunedEL + let borrowedParams = summaryBorrowedParams + let skew = summaryMaxObservedSkew + summaryBypassNoEL = 0 + summaryBypassNoRPU = 0 + summaryDropStaleEL = 0 + summaryPrunedEL = 0 + summaryBorrowedParams = 0 + summaryMaxObservedSkew = max(blFrameQueue.count, elFrameQueue.count) + lastSummaryTimestamp = now + let activeComposer = composer + lock.unlock() + + let slots = activeComposer?.takePeakSlotsInUse() ?? 0 + let poolBuffers = activeComposer?.allocatedPoolBuffers ?? 0 + let gpuMs = activeComposer?.meanGPUTimeMs + let pendingPeak = orderedDelivery.takePeakPendingCount() + let gpuStr = gpuMs.map { String(format: "%.1f", $0) } ?? "n/a" + let line = "[DualLayer] summary: composed=\(composed), composeFailures=\(composeFailures), bypassNoEL=\(noEL), bypassNoRPU=\(noRPU), staleELDropped=\(staleEL), prunedEL=\(prunedEL), borrowedParams=\(borrowedParams), maxSkew=\(skew), peakGpuSlots=\(slots), poolBuffers=\(poolBuffers), deliveryPendingPeak=\(pendingPeak), meanGpuMs=\(gpuStr)" + EngineLog.emit(line, category: .swPlayback) + activeComposer?.flushTextureCache() + } + + // MARK: - Flush & Teardown + + func flush() { + #if DEBUG + flushWillTakeLifecycleLockHookForTesting?() + #endif + lifecycleLock.lock() + defer { lifecycleLock.unlock() } + // `orderedDelivery`'s epoch is retired from THIS critical section, the same one that bumps + // `_feedEpoch` and raises `draining`, with `resetAsync` (a plain `queue.async`, safe under + // `lock`) and a `barrier()` once `lock` is released. Retiring it later, after the hardware + // drain, left a gap: a VideoToolbox callback landing between the unlock and the reset (with + // `draining` already false) took a new-epoch ticket that `orderedDelivery` still dropped as + // stale, and the reset then skipped past it, stalling every following frame until the stall + // guard gave up on it. Two things follow from the order used here: + // + // 1. **No ticket can be lost.** `draining` stays raised for the whole window in which + // `orderedDelivery` might still be on the old epoch: it is lowered only in the second + // critical section below, after `barrier()` has confirmed the `resetAsync` ran. No + // thread can issue a new-epoch ticket until `orderedDelivery` is on that epoch. + // 2. **Nothing from before the flush is delivered during the drain.** A composition that + // was in flight before this `flush()` and retires while `primary.flush()` / `el.flush()` + // are still draining finds the epoch already moved and is dropped, so the draining gate + // covers delivery as well as pairing. + // + // Concurrent `flush()`/`flush()` or `flush()`/`close()`: the whole of each call runs under + // `lifecycleLock`, so they never interleave (one cannot lower `draining` or clear the + // queues while the other still drains). Inside a call, `_feedEpoch`/`deliveryNextTicket` + // only ever move forward under `lock`, so `orderedDelivery`'s FIFO queue applies the + // resets in call order and neither can ever move backwards. + // + // `nextTicket` is stable across this whole call: no new ticket can + // be issued between this section and the one below, because + // `handleBaseLayerFrame`/`handleEnhancementLayerFrame` both gate on + // `!draining` under `lock`, and `draining` does not go false until + // after the drain completes. + lock.lock() + guard !isClosed else { lock.unlock(); return } + _feedEpoch &+= 1 + draining = true + // Here, not with the queues below: the demux thread may feed post-seek packets while this + // thread waits on the drain, and those must start from the next IRAP. + pictureAlignment.reset() + // Same reason, and the same section: RPUs the demux thread records for post-seek packets + // during the drain must survive it, so the side data resets here, before any of them can + // exist, and not after the drain. + lastGoodParams = nil + lastGoodDMNAL = nil + borrowedParamsRun = 0 + sideDataRing.removeAll() + let epoch = _feedEpoch + let nextTicket = deliveryNextTicket + let primary = primaryDecoder + let el = enhancementDecoder + // The sub-decoders move to the new epoch in this same critical section. The host feeds the + // demux thread's packets concurrently with a seek's flush: it reads the new epoch the moment + // it is published, and hands it down with the packet. Were the sub-decoders to move only when + // their own `flush()` runs below, a post-seek packet fed in that window would pass the gate + // here and be refused by a sub-decoder still on the old epoch, losing the post-seek IRAP + // (garbage until the next keyframe). `adoptFeedEpoch` takes only the sub-decoder's leaf + // `epochLock`, never its `lock`, so it cannot nest against the VideoToolbox callbacks that + // wait on this lock. DEC-4 still holds: a pre-seek packet is refused from here on, and a + // sub-decoder's `flush()` below serialises against any packet already inside its + // check-and-send. + primary.adoptFeedEpoch(epoch) + el?.adoptFeedEpoch(epoch) + orderedDelivery.resetAsync(epoch: epoch, nextTicket: nextTicket) + lock.unlock() + + #if DEBUG + // Test seam only: lets a test drive a ticket through the window a + // post-flush ticket could be lost in if the reset moved back out of + // the critical section above — right after this + // section's `unlock()`, before the `barrier()` below has confirmed + // the `resetAsync` above has run — without a real GPU or + // VideoToolbox callback. See `raceTicketForTesting`. + flushRaceHookForTesting?() + #endif + + // Waits for the `resetAsync` enqueued above to actually run (and for + // anything already ahead of it in `orderedDelivery`'s queue to + // finish first), with no decoder lock held — this cannot nest + // inside, or be nested by, any lock a concurrent `fulfil`/`emit` + // might touch (I8). + orderedDelivery.barrier() + + // Drain both hardware decoders now that `orderedDelivery` is + // provably on the new epoch: `HardwareVideoDecoder.flush()` waits + // for the frames VideoToolbox still owes, which arrive through the + // normal callbacks while this thread is blocked inside it, but + // `draining` (still up) drops them at the pairing gate before they + // can take a ticket, and any composition that was already + // in flight before this `flush()` is dropped by the epoch check the + // instant it reaches `fulfil` instead. + // + // Post-seek frames that VideoToolbox outputs while this drain is still running (a post-seek + // packet is now accepted from the moment the epoch moved) meet `draining` and are dropped. + // That is deliberate and costs nothing visible: the drain cannot tell them from the pre-seek + // frames it is waiting for, they are decoded from the IRAP onwards, which the host's + // `skipUntilPTS` pre-roll discards anyway until the seek target; and a dropped output frame + // does not affect decoding, VideoToolbox's reference state is intact. What it can cost is + // pairing: a frame dropped on one layer and not on the other leaves its partner waiting, and + // a waiting base frame goes out base-layer only once `maxLayerSkewFrames` newer ones have + // arrived. So the first few frames after a seek can be base-layer only (how many depends on + // how many frames the two sessions output during the drain), and the base frames dropped + // during the drain are never shown. + primary.flushKeepingEpoch() + el?.flushKeepingEpoch() + + // Now that nothing more can arrive, reset pairing and let new tickets flow again. (The + // side-data ring is not touched here: see the first critical section.) + lock.lock() + draining = false + blFrameQueue.removeAll() + elFrameQueue.removeAll() + blArrivalSeq = 0 + settledBlKeys.removeAll() + composeResultLock.lock(); summaryComposedFrames = 0; summaryComposeFailures = 0; composeResultLock.unlock() + summaryBypassNoEL = 0 + summaryBypassNoRPU = 0 + summaryDropStaleEL = 0 + summaryPrunedEL = 0 + summaryBorrowedParams = 0 + summaryMaxObservedSkew = 0 + lastSummaryTimestamp = 0 + lock.unlock() + + composer?.flush() + } + + func close() { + lifecycleLock.lock() + defer { lifecycleLock.unlock() } + // dolby-vision-fel.md §5.8: `close()` resets every field of stream state, so a + // reopened instance could not inherit the MEL latch, a stale + // `primaryDecoderInitialized`, BL parameter sets, `frameIndex` or + // `timeBase` — and then makes reopening impossible anyway: the decoder + // is single use and `open` after this refuses with `decoderClosed`. + // Epoch limitation: the bump below only retires `decode(packet:epoch:)` + // callers that pass an epoch. The DVR-reseed path calls `decode` with + // `epoch: nil`, which never matches and is never retired by the epoch — + // those packets (and any thread already past the entry gate) are stopped + // by the `isClosed` checks in `decode`, `ensure*Decoder` and the frame + // handlers, not by the epoch. Delivery itself IS stopped by the epoch, + // via `orderedDelivery` below. + // + // As in `flush()`, `orderedDelivery.resetAsync` is enqueued from + // inside THIS critical section — the same one that bumps `_feedEpoch`, + // zeroes `deliveryNextTicket` and sets `isClosed`. Nothing can issue a ticket after + // this section unlocks (`isClosed` gates every entry point), so + // there is no gap for one to race `resetAsync` in, but keeping the + // same enqueue-under-lock shape as `flush()` means a `flush()`/ + // `close()` race is decided the same way in both directions: `lock` + // fixes the order the two calls' `resetAsync`s land in + // `orderedDelivery`'s queue, and `_feedEpoch` only ever increases + // under that same lock, so whichever lands later always carries the + // larger epoch. + lock.lock() + _feedEpoch &+= 1 + draining = false + deliveryNextTicket = 0 + pictureAlignment = Self.makeAlignment() + blFrameQueue.removeAll() + elFrameQueue.removeAll() + blArrivalSeq = 0 + settledBlKeys.removeAll() + lastGoodParams = nil + lastGoodDMNAL = nil + borrowedParamsRun = 0 + sideDataRing.removeAll() + composeResultLock.lock(); summaryComposedFrames = 0; summaryComposeFailures = 0; composeResultLock.unlock() + summaryBypassNoEL = 0 + summaryBypassNoRPU = 0 + summaryDropStaleEL = 0 + summaryPrunedEL = 0 + summaryBorrowedParams = 0 + summaryMaxObservedSkew = 0 + lastSummaryTimestamp = 0 + isFEL = false + isMEL = false + streamClassified = false + frameIndex = 0 + primaryDecoderInitialized = false + blVPS = nil + blSPS = nil + blPPS = nil + elVPS = nil + elSPS = nil + elPPS = nil + enhancementOpenFailure = nil + timeBase = AVRational(num: 1, den: 90000) + _skipUntilPTS = nil + isClosed = true + let epoch = _feedEpoch + let nextTicket = deliveryNextTicket + // Swap the enhancement decoder out under the lock and close it after releasing + // it: `close()` runs on the host thread while VideoToolbox callbacks may still + // be delivering into `handleEnhancementLayerFrame`. + let el = enhancementDecoder + enhancementDecoder = nil + orderedDelivery.resetAsync(epoch: epoch, nextTicket: nextTicket) + lock.unlock() + + // Same guarantee as `flush()`: nothing tagged with the + // pre-close epoch reaches `onFrame` after this returns. Clearing the + // callback only after that is what makes "nothing is delivered after + // `close()` returns" hold even for the frame that was legitimately + // mid-delivery when this call started. + orderedDelivery.barrier() + orderedDelivery.setOnFrame(nil) + + // Sessions first: closing them drains the frames VideoToolbox still owes, and those can still be + // holding or about to take pool buffers. Only then flush the composer and release its pool. + primaryDecoder.close() + el?.close() + composer?.flush() + composer?.releaseOutputPool() + } + + // MARK: - Test seams + // + // Both frame handlers are driven by VideoToolbox callbacks, which a unit test cannot produce + // on demand. These put the decoder in the state `decode(packet:)` reaches on a real stream, + // so `DualLayerVideoDecoderTests` can exercise pairing, ordering, metadata and flush with + // synthetic buffers. + + #if DEBUG + /// Regression seam for the post-flush ticket race: if set, `flush()` + /// calls this exactly once, synchronously, in the gap right after the + /// critical section that bumps `_feedEpoch`, captures `nextTicket` and + /// enqueues `orderedDelivery.resetAsync` releases `lock`, but before the + /// `orderedDelivery.barrier()` call that waits for it to actually run. + /// This is the window a post-flush ticket could be lost in — see the + /// comment at the top of `flush()`. Paired with `raceTicketForTesting`, this drives a ticket + /// through that window deterministically, without a real GPU or + /// VideoToolbox callback, so a regression that reopens the window (e.g. + /// moving the `resetAsync` call back outside the lock) fails a test + /// instead of waiting on a sub-microsecond production race. + var flushRaceHookForTesting: (() -> Void)? + + /// Called at the top of every `flush()`, before it queues for `lifecycleLock`. Lets a test know a + /// second flush has reached the lock without sleeping to find out. + var flushWillTakeLifecycleLockHookForTesting: (() -> Void)? + + /// Waits until every composition started so far has finished and handed its frame to + /// `orderedDelivery`, then until that queue has run everything enqueued ahead of this call. After + /// it returns, a frame that was going to be delivered (or dropped as stale) by work already in + /// flight has been, which is what a "nothing more arrives" assertion needs; it is a condition + /// wait, not a delay. Compositions that never finish (a gated fake) would make it wait forever, so a + /// test that holds one releases it first. + func quiesceForTesting() { + while true { + composeResultLock.lock() + let idle = composeRequestsStarted == composeRequestsHandedOff + composeResultLock.unlock() + if idle { break } + usleep(500) + } + orderedDelivery.barrier() + } + + /// Issues a ticket the same way `handleBaseLayerFrame` would — under + /// `lock`, in decode order, tagged with the current epoch — and hands it + /// straight to `orderedDelivery`, bypassing the `draining` gate that + /// would otherwise stop a real VideoToolbox callback from reaching here + /// at this point. Standing in for "a VideoToolbox callback landing in + /// the gap": the gate no + /// longer needs to stop it, now that the epoch is retired before + /// `draining` can go false, but this seam drives the ordering itself, + /// not the gate, so that guarantee is what is actually under test. + func raceTicketForTesting(buffer: CVPixelBuffer, pts: CMTime) { + lock.lock() + let ticket = nextDeliveryTicket() + let epoch = _feedEpoch + lock.unlock() + orderedDelivery.fulfil( + ticket: ticket, epoch: epoch, + DeliveredFrame(buffer: SendableBuffer(value: buffer), pts: pts, hdr10Data: nil, dmNAL: nil)) + } + #endif + + /// Marks the stream as FEL with the enhancement-layer decoder engaged, or as single layer. + /// The engaged decoder is never opened: pairing only checks that one exists. + func primeForTesting(enhancementLayer: Bool) { + lock.lock() + defer { lock.unlock() } + isFEL = enhancementLayer + isMEL = false + streamClassified = true + if enhancementLayer { + // Only engage the EL decoder when a composer exists. Without + // one, the production path (`decode(packet:)`) never reaches + // `ensureEnhancementDecoder`, so neither does the test seam. + if enhancementDecoder == nil, composer != nil { + let decoder = HardwareVideoDecoder() + decoder.useCustomBlockSource = true + enhancementDecoder = decoder + } + } else { + enhancementDecoder = nil + } + } + + /// Puts the decoder in the classified state `decode(packet:)` reaches for `kind` + ///: FEL engages the enhancement-layer stub, MEL never does. Fresh-decoder + /// assumption, like `primeForTesting`: the stub is never opened, pairing only checks + /// that one exists. + func primeStreamForTesting(_ kind: DoviRpuConverter.StreamKind) { + lock.lock() + defer { lock.unlock() } + streamClassified = true + switch kind { + case .fel: + isFEL = true + isMEL = false + // Only engage the EL decoder when a composer exists. + if enhancementDecoder == nil, composer != nil { + let decoder = HardwareVideoDecoder() + decoder.useCustomBlockSource = true + enhancementDecoder = decoder + } + case .mel: + isFEL = false + isMEL = true + enhancementDecoder = nil + case .undetermined: + isFEL = false + isMEL = false + enhancementDecoder = nil + } + } + + /// Applies one `separateDualLayers` result's classification exactly as + /// `decode(packet:)` does, so tests can drive classification sequences without real packets. + func classifyPacketResultForTesting(_ result: DoviRpuConverter.DualLayerPacketResult) { + applyStreamClassification(DoviRpuConverter.classifyStream(result)) + } + + /// The feed epoch of the lazily opened enhancement decoder, nil until it exists. + var enhancementFeedEpochForTesting: UInt64? { + lock.lock() + let el = enhancementDecoder + lock.unlock() + return (el as? HardwareVideoDecoder)?.feedEpoch + } + + /// The stream classification `decode(packet:)` has latched so far. + var streamClassification: DoviRpuConverter.StreamKind { + lock.lock() + defer { lock.unlock() } + if isMEL { return .mel } + if isFEL { return .fel } + return .undetermined + } + + var streamClassificationForTesting: DoviRpuConverter.StreamKind { streamClassification } + + /// The decode-path gate for opening and feeding the enhancement decoder: only a classified FEL stream with a composer engages it; MEL or + /// no composer never does. + func engagesEnhancementLayerForTesting() -> Bool { + lock.lock() + defer { lock.unlock() } + return isFEL && !isMEL && composer != nil + } + + /// Stream state `close()` must reset: classification latch, + /// primary-decoder readiness, composition counter and EL engagement, plus + /// the single-use flag that makes `open` refuse after `close()`. Queue, + /// cache, ticket and epoch counts are included so tests can assert the reset + /// explicitly rather than relying on delivery silence alone. + struct DecoderStateSnapshot: Sendable { + var isClosed: Bool + var isFEL: Bool + var isMEL: Bool + var streamClassified: Bool + var primaryDecoderInitialized: Bool + var frameIndex: UInt32 + var enhancementEngaged: Bool + var blQueueCount: Int + var elQueueCount: Int + var rpuCacheCount: Int + var dmRpuCacheCount: Int + var deliveryPendingCount: Int + var deliveryNextTicket: UInt64 + var deliveryNextToEmit: UInt64 + var feedEpoch: UInt64 + var draining: Bool + var timeBaseNum: Int32 + var timeBaseDen: Int32 + var hasBLVPS: Bool + var hasBLSPS: Bool + var hasBLPPS: Bool + var hasELVPS: Bool + var hasELSPS: Bool + var hasELPPS: Bool + var hasSkipUntilPTS: Bool + var hasOnFrame: Bool + var hasComposer: Bool + } + + var stateSnapshotForTesting: DecoderStateSnapshot { + // Decoder-lock fields first, then release before touching + // `orderedDelivery`: its `queue.sync` reads never need the + // decoder lock, and not holding both at once keeps this snapshot from + // becoming the one place that reintroduces a lock/queue dependency. + lock.lock() + let isClosed = self.isClosed + let isFEL = self.isFEL + let isMEL = self.isMEL + let streamClassified = self.streamClassified + let primaryDecoderInitialized = self.primaryDecoderInitialized + let frameIndex = self.frameIndex + let enhancementEngaged = isFEL && !isMEL && enhancementDecoder != nil && composer != nil + let blQueueCount = blFrameQueue.count + let elQueueCount = elFrameQueue.count + let rpuCacheCount = sideDataRing.paramsCount + let dmRpuCacheCount = sideDataRing.dmNALCount + let deliveryNextTicket = self.deliveryNextTicket + let feedEpoch = _feedEpoch + let draining = self.draining + let timeBaseNum = timeBase.num + let timeBaseDen = timeBase.den + let hasBLVPS = blVPS != nil + let hasBLSPS = blSPS != nil + let hasBLPPS = blPPS != nil + let hasELVPS = elVPS != nil + let hasELSPS = elSPS != nil + let hasELPPS = elPPS != nil + let hasSkipUntilPTS = _skipUntilPTS != nil + let hasComposer = composer != nil + lock.unlock() + + return DecoderStateSnapshot( + isClosed: isClosed, + isFEL: isFEL, + isMEL: isMEL, + streamClassified: streamClassified, + primaryDecoderInitialized: primaryDecoderInitialized, + frameIndex: frameIndex, + enhancementEngaged: enhancementEngaged, + blQueueCount: blQueueCount, + elQueueCount: elQueueCount, + rpuCacheCount: rpuCacheCount, + dmRpuCacheCount: dmRpuCacheCount, + deliveryPendingCount: orderedDelivery.pendingCountForTesting, + deliveryNextTicket: deliveryNextTicket, + deliveryNextToEmit: orderedDelivery.nextToEmitForTesting, + feedEpoch: feedEpoch, + draining: draining, + timeBaseNum: timeBaseNum, + timeBaseDen: timeBaseDen, + hasBLVPS: hasBLVPS, + hasBLSPS: hasBLSPS, + hasBLPPS: hasBLPPS, + hasELVPS: hasELVPS, + hasELSPS: hasELSPS, + hasELPPS: hasELPPS, + hasSkipUntilPTS: hasSkipUntilPTS, + hasOnFrame: orderedDelivery.hasOnFrameForTesting, + hasComposer: hasComposer + ) + } + + /// Caches what `decode(packet:)` would have cached for the packet at `pts`: the composer's + /// reshaping parameters and the Profile 8.1 RPU NAL attached as display metadata. + /// One call is one packet, in decode order: a nil `params` is a packet with a missing RPU (or, with + /// `usesPrevious`, one that says to keep the previous mapping), resolved exactly as in production. + func seedRPUForTesting(pts: CMTime, params: DoviMetalParams?, displayMetadataNAL: [UInt8]?, usesPrevious: Bool = false, disableResidual: Bool = false) { + lock.lock() + defer { lock.unlock() } + let key = Self.canonicalKey(pts: pts) + recordSideData(key: key, params: params, dmNAL: displayMetadataNAL, usesPrevious: usesPrevious, disableResidual: disableResidual) + } +} diff --git a/Sources/AetherEngine/Decoder/HardwareVideoDecoder.swift b/Sources/AetherEngine/Decoder/HardwareVideoDecoder.swift index 125480982..7823a08b8 100644 --- a/Sources/AetherEngine/Decoder/HardwareVideoDecoder.swift +++ b/Sources/AetherEngine/Decoder/HardwareVideoDecoder.swift @@ -79,10 +79,38 @@ final class HardwareVideoDecoder: VideoDecodingPipeline, @unchecked Sendable { /// Protects `session` across the demux thread (decode), main thread (close/flush), and VT callback (delivery). private let lock = NSLock() - /// AE#492: guarded by `lock`, the same one `flush()` and `decode(packet:epoch:)` take. + /// When true, wrap the packet's AVBuffer with a CMBlockBuffer that holds + /// an `av_buffer_ref` and unrefs it on disposal (dolby-vision-fel.md §5.4). + var useCustomBlockSource: Bool = false + + #if DEBUG + /// Called after a packet has been handed to VideoToolbox (whatever the status it returned), and + /// not for one the epoch gate or a missing session refused. + var onSubmittedToVideoToolboxForTesting: (() -> Void)? + var onBlockBufferAllocatedForTesting: ((CMBlockBuffer) -> Void)? + #endif + + /// AE#492: the epoch a fed packet must carry. Guarded by `epochLock`, a leaf lock held only for a + /// read or a write of this integer (nothing is taken, awaited or called while it is held), so + /// `DualLayerVideoDecoder` can move this decoder's epoch from inside its own critical section + /// without nesting into `lock`, which `decode` may hold across a blocking VideoToolbox call whose + /// output callbacks need the owner's lock. `decode` and `flush()` still read and bump it while + /// holding `lock`, so a flush and a feed cannot interleave; `adoptFeedEpoch` alone moves it + /// without `lock`, and only ever to the owner's current epoch. private var _feedEpoch: UInt64 = 0 - var feedEpoch: UInt64 { - lock.lock(); defer { lock.unlock() }; return _feedEpoch + private let epochLock = NSLock() + private var currentEpoch: UInt64 { + epochLock.lock(); defer { epochLock.unlock() }; return _feedEpoch + } + var feedEpoch: UInt64 { currentEpoch } + + /// Moves this decoder onto the feed epoch its owner is already in. `DualLayerVideoDecoder` calls it + /// (a) when it opens the enhancement decoder lazily, after a `flush()` may have moved its own + /// epoch, and (b) from inside its `flush()` critical section, so that a post-seek packet fed with + /// the new epoch while the owner is still draining this decoder is accepted, instead of being + /// dropped as stale by a decoder that has not yet been flushed. Takes only `epochLock`. + func adoptFeedEpoch(_ epoch: UInt64) { + epochLock.lock(); _feedEpoch = epoch; epochLock.unlock() } /// AE#739: the system invalidates the session behind our back (iOS screen lock, media services @@ -102,6 +130,9 @@ final class HardwareVideoDecoder: VideoDecodingPipeline, @unchecked Sendable { private var awaitingKeyframe = false private var droppedAwaitingKeyframe = 0 private var decoderSpecification: NSDictionary? + /// Guarded by `lock`. False for a session opened from parameter sets, which `makeSession` and a + /// rebuild then create without `PropagatePerFrameHDRDisplayMetadata`. + private var propagatesHDRDisplayMetadata = true private var pixelBufferAttributes: NSDictionary? /// Heap-allocated box carrying a weak self reference for the C decompression callback's refCon. @@ -114,6 +145,71 @@ final class HardwareVideoDecoder: VideoDecodingPipeline, @unchecked Sendable { init(_ decoder: HardwareVideoDecoder) { self.decoder = decoder } } + /// Keeps `packet.buf` alive for the lifetime of a CMBlockBuffer that wraps + /// the packet's data without copying it (dolby-vision-fel.md §5.4). + final class CustomBlockSource { + var avBuf: UnsafeMutablePointer? + let dataPtr: UnsafeMutableRawPointer + + /// `data` is the packet's own payload pointer. It is not always `buf->data`: the Matroska + /// demuxer hands out packets whose `data` sits inside a larger buffer (observed at +4 on P7 + /// MKV remuxes), so block memory taken from `buf->data` starts early and VideoToolbox sees + /// a corrupt sample (-12909). Defaults to `buf->data` for callers that own the whole buffer. + init?(buf: UnsafeMutablePointer, data: UnsafeMutablePointer? = nil) { + guard let data = data ?? buf.pointee.data, let ref = av_buffer_ref(buf) else { return nil } + self.avBuf = ref + self.dataPtr = UnsafeMutableRawPointer(data) + } + + func releaseBuffer() { + if avBuf != nil { + av_buffer_unref(&avBuf) + } + } + + deinit { + releaseBuffer() + } + } + + /// Wraps a packet's payload in a CMBlockBuffer without copying it. The block buffer owns + /// `blockSource` (and with it the `av_buffer_ref` of `packet.buf`) until CoreMedia frees the + /// block; if the buffer cannot be created the reference is released here, so the caller has + /// nothing to clean up. `dataLength` is the size of the payload when it differs from the block + /// (only a test needs that, to make the creation fail). + static func makeBlockBuffer(wrapping blockSource: CustomBlockSource, size: Int, dataLength: Int? = nil) -> (buffer: CMBlockBuffer?, status: OSStatus) { + var source = CMBlockBufferCustomBlockSource( + version: kCMBlockBufferCustomBlockSourceVersion, + AllocateBlock: { refCon, _ in + guard let refCon else { return nil } + let holder = Unmanaged.fromOpaque(refCon).takeUnretainedValue() + return holder.dataPtr + }, + FreeBlock: { refCon, _memoryBlock, _size in + guard let refCon else { return } + let holder = Unmanaged.fromOpaque(refCon).takeRetainedValue() + holder.releaseBuffer() + }, + refCon: Unmanaged.passRetained(blockSource).toOpaque() + ) + var blockBuffer: CMBlockBuffer? + let status = CMBlockBufferCreateWithMemoryBlock( + allocator: kCFAllocatorDefault, + memoryBlock: nil, + blockLength: size, + blockAllocator: kCFAllocatorNull, + customBlockSource: &source, + offsetToData: 0, + dataLength: dataLength ?? size, + flags: kCMBlockBufferAssureMemoryNowFlag, + blockBufferOut: &blockBuffer + ) + if status != kCMBlockBufferNoErr, let refCon = source.refCon { + Unmanaged.fromOpaque(refCon).takeRetainedValue().releaseBuffer() + } + return (blockBuffer, status) + } + // MARK: - Lifecycle func open(stream: UnsafeMutablePointer, onFrame: @escaping DecodedFrameHandler) throws { @@ -245,11 +341,14 @@ final class HardwareVideoDecoder: VideoDecodingPipeline, @unchecked Sendable { ) guard status == noErr, let created = sessionOut else { return (nil, status) } // Pass through per-frame HDR metadata for correct tone mapping; unknown-key set returns -12911 on older OSes (swallowed). - VTSessionSetProperty( - created, - key: kVTDecompressionPropertyKey_PropagatePerFrameHDRDisplayMetadata, - value: kCFBooleanTrue - ) + // Not on a session opened from parameter sets: see `open(parameterSets:timeBase:onFrame:)`. + if propagatesHDRDisplayMetadata { + VTSessionSetProperty( + created, + key: kVTDecompressionPropertyKey_PropagatePerFrameHDRDisplayMetadata, + value: kCFBooleanTrue + ) + } return (created, noErr) } @@ -267,7 +366,7 @@ final class HardwareVideoDecoder: VideoDecodingPipeline, @unchecked Sendable { VTDecompressionSessionInvalidate(old) session = nil } - _feedEpoch &+= 1 + epochLock.lock(); _feedEpoch &+= 1; epochLock.unlock() let created = makeSession(formatDescription: formatDescription, refCon: refConBox) guard let fresh = created.session else { sessionLost = true @@ -295,12 +394,111 @@ final class HardwareVideoDecoder: VideoDecodingPipeline, @unchecked Sendable { } #endif + /// Opens a decoder session from raw HEVC parameter sets (VPS, SPS, PPS), used for the Enhancement Layer (EL) + /// in Dolby Vision Profile 7 dual-layer decoding. + func open( + parameterSets: [[UInt8]], + timeBase: AVRational, + onFrame: @escaping DecodedFrameHandler + ) throws { + self.onFrame = onFrame + self.timeBase = timeBase + + guard parameterSets.count >= 3 else { + throw VideoDecoderError.noExtradata + } + + var ptrs: [UnsafePointer] = [] + var sizes: [Int] = [] + var allocations: [UnsafeMutablePointer] = [] + defer { + for a in allocations { a.deallocate() } + } + for ps in parameterSets { + let p = UnsafeMutablePointer.allocate(capacity: ps.count) + ps.withUnsafeBufferPointer { src in + if let base = src.baseAddress { + p.initialize(from: base, count: ps.count) + } + } + allocations.append(p) + ptrs.append(UnsafePointer(p)) + sizes.append(ps.count) + } + + var fd: CMVideoFormatDescription? + let fdStatus = CMVideoFormatDescriptionCreateFromHEVCParameterSets( + allocator: kCFAllocatorDefault, + parameterSetCount: parameterSets.count, + parameterSetPointers: ptrs, + parameterSetSizes: sizes, + nalUnitHeaderLength: 4, + extensions: nil, + formatDescriptionOut: &fd + ) + guard fdStatus == noErr, let formatDesc = fd else { + throw VideoDecoderError.formatDescriptionFailed(status: fdStatus) + } + + let dims = CMVideoFormatDescriptionGetDimensions(formatDesc) + self.width = dims.width + self.height = dims.height + + var decoderSpec: NSDictionary? + if #available(tvOS 17.0, iOS 17.0, *) { + decoderSpec = [ + kVTVideoDecoderSpecification_RequireHardwareAcceleratedVideoDecoder: true, + ] + } + + self.colorPrimaries = kCVImageBufferColorPrimaries_ITU_R_2020 + self.colorTransfer = kCVImageBufferTransferFunction_SMPTE_ST_2084_PQ + self.colorMatrix = kCVImageBufferYCbCrMatrix_ITU_R_2020 + + let pixelBufferAttrs: NSDictionary = [ + kCVPixelBufferPixelFormatTypeKey: kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange, + kCVPixelBufferIOSurfacePropertiesKey: NSDictionary(), + kCVPixelBufferMetalCompatibilityKey: true, + ] + + let box = RefConBox(self) + let unmanaged = Unmanaged.passRetained(box) + + // `session` is read by `decode`/`flush`/`close` on other threads: install it under `lock`. + lock.lock() + propagatesHDRDisplayMetadata = false + decoderSpecification = decoderSpec + pixelBufferAttributes = pixelBufferAttrs + let created = makeSession(formatDescription: formatDesc, refCon: unmanaged) + guard let createdSession = created.session else { + lock.unlock() + unmanaged.release() + throw VideoDecoderError.sessionCreationFailed(status: created.status) + } + formatDescription = formatDesc + session = createdSession + refConBox = unmanaged + lock.unlock() + + // Deliberately no `kVTDecompressionPropertyKey_PropagatePerFrameHDRDisplayMetadata` here, + // unlike `open(stream:)`. This path builds the base layer's session when its parameter sets + // arrive in band, and the enhancement layer's session always. The dual-layer decoder + // attaches its own Dolby Vision RPU to every frame it delivers (composed or bypassed) from + // the RPU it extracted; letting VideoToolbox propagate HDR display metadata as well would put a + // second, possibly different, attachment on the base-layer buffer a bypassed frame goes out as. + + EngineLog.emit( + "[HardwareVideoDecoder] opened secondary HEVC \(width)x\(height) 10-bit from parameter sets", + category: .swPlayback + ) + } + // MARK: - Decode func decode(packet: UnsafeMutablePointer, epoch: UInt64? = nil) { lock.lock() // AE#492: see `SoftwareVideoDecoder.decode`. Same rule, same lock as `flush()`. - if let epoch, epoch != _feedEpoch { lock.unlock(); return } + if let epoch, epoch != currentEpoch { lock.unlock(); return } guard session != nil, !sessionLost, let formatDesc = formatDescription else { lock.unlock() return @@ -312,24 +510,43 @@ final class HardwareVideoDecoder: VideoDecodingPipeline, @unchecked Sendable { // and AVPacket storage is reused for the next packet. guard let data = packet.pointee.data, packet.pointee.size > 0 else { return } let size = Int(packet.pointee.size) - let copied = UnsafeMutableRawPointer.allocate(byteCount: size, alignment: 1) - copied.copyMemory(from: data, byteCount: size) var blockBuffer: CMBlockBuffer? - let blockStatus = CMBlockBufferCreateWithMemoryBlock( - allocator: kCFAllocatorDefault, - memoryBlock: copied, - blockLength: size, - blockAllocator: kCFAllocatorDefault, // ← matching dealloc allocator - customBlockSource: nil, - offsetToData: 0, - dataLength: size, - flags: 0, - blockBufferOut: &blockBuffer - ) + var blockStatus: OSStatus = noErr + var copied: UnsafeMutableRawPointer? + + if useCustomBlockSource, + let buf = packet.pointee.buf, + let blockSource = CustomBlockSource(buf: buf, data: data) { + // Design §5.4: no copy. The CMBlockBuffer holds an `av_buffer_ref` of + // `packet.buf` and unrefs it in FreeBlock, so VT can retain the data past + // the call without a copy. + (blockBuffer, blockStatus) = Self.makeBlockBuffer(wrapping: blockSource, size: size) + #if DEBUG + if let bb = blockBuffer { + onBlockBufferAllocatedForTesting?(bb) + } + #endif + } else { + // Fallback: copy into newly allocated memory when custom block source is disabled, + // packet has no AVBufferRef (e.g. synthetic test packet), or av_buffer_ref failed. + copied = UnsafeMutableRawPointer.allocate(byteCount: size, alignment: 1) + copied?.copyMemory(from: data, byteCount: size) + blockStatus = CMBlockBufferCreateWithMemoryBlock( + allocator: kCFAllocatorDefault, + memoryBlock: copied, + blockLength: size, + blockAllocator: kCFAllocatorDefault, + customBlockSource: nil, + offsetToData: 0, + dataLength: size, + flags: 0, + blockBufferOut: &blockBuffer + ) + } guard blockStatus == kCMBlockBufferNoErr, let bb = blockBuffer else { // CMBlockBuffer takes ownership only on success; we own the allocation on failure. - copied.deallocate() + copied?.deallocate() return } let pts = SourceTimestampBounds.cmTime(ticks: packet.pointee.pts, timeBase: timeBase) @@ -378,7 +595,7 @@ final class HardwareVideoDecoder: VideoDecodingPipeline, @unchecked Sendable { // while the sample buffer is built lets a pre-seek packet into VT after it. Safe to hold: // the output callback never takes `lock`, and `close()` already holds it across the VT wait. lock.lock() - if let epoch, epoch != _feedEpoch { lock.unlock(); return } + if let epoch, epoch != currentEpoch { lock.unlock(); return } guard let session = self.session, !sessionLost else { lock.unlock(); return } var recoveredAfterDropping: Int? if awaitingKeyframe { @@ -404,6 +621,9 @@ final class HardwareVideoDecoder: VideoDecodingPipeline, @unchecked Sendable { let lostNow = decodeStatus == kVTInvalidSessionErr && !sessionLost if lostNow { sessionLost = true } lock.unlock() + #if DEBUG + onSubmittedToVideoToolboxForTesting?() + #endif if let recoveredAfterDropping { EngineLog.emit( "[HardwareVideoDecoder] AE#739 recovered at keyframe pts=\(packet.pointee.pts) " @@ -426,9 +646,18 @@ final class HardwareVideoDecoder: VideoDecodingPipeline, @unchecked Sendable { } } - func flush() { + func flush() { flush(bumpingEpoch: true) } + + /// The flush a `DualLayerVideoDecoder` makes: its owner has already moved this decoder's epoch with + /// `adoptFeedEpoch` (inside its own critical section), so bumping it again here would retire the + /// post-seek packets the owner has been feeding since. + func flushKeepingEpoch() { flush(bumpingEpoch: false) } + + private func flush(bumpingEpoch: Bool) { lock.lock() - _feedEpoch &+= 1 // AE#492 + if bumpingEpoch { // AE#492 + epochLock.lock(); _feedEpoch &+= 1; epochLock.unlock() + } let session = self.session lock.unlock() guard let session else { return } diff --git a/Sources/AetherEngine/Decoder/VideoDecoderTypes.swift b/Sources/AetherEngine/Decoder/VideoDecoderTypes.swift index a1a87601d..800ba93ce 100644 --- a/Sources/AetherEngine/Decoder/VideoDecoderTypes.swift +++ b/Sources/AetherEngine/Decoder/VideoDecoderTypes.swift @@ -46,6 +46,15 @@ enum VideoDecoderError: Error, LocalizedError { case noExtradata case formatDescriptionFailed(status: OSStatus) case sessionCreationFailed(status: OSStatus) + /// `DualLayerVideoDecoder` is single use (dolby-vision-fel.md §5.8): `open` after `close` refuses + /// with this instead of inheriting the previous stream's state. Thrown rather than a + /// trapping precondition so host misuse surfaces as an error and tests can assert it. + case decoderClosed + /// No Metal device or shader compilation failed; FEL composition cannot run. + /// Thrown from `open()` for a decoder built without a composer. The software host does not get + /// there in production: it builds the composer first and decodes the base layer alone when that + /// fails. + case composerUnavailable var errorDescription: String? { switch self { @@ -54,6 +63,8 @@ enum VideoDecoderError: Error, LocalizedError { case .noExtradata: "Missing codec extradata" case .formatDescriptionFailed(let s): "Format description failed (\(s))" case .sessionCreationFailed(let s): "Decoder session failed (\(s))" + case .decoderClosed: "Decoder is closed (single use)" + case .composerUnavailable: "Dolby Vision FEL composer unavailable (no Metal device or shader compilation failed)" } } } diff --git a/Sources/AetherEngine/Diagnostics/FelResidualMeasure.swift b/Sources/AetherEngine/Diagnostics/FelResidualMeasure.swift new file mode 100644 index 000000000..8fc7b442d --- /dev/null +++ b/Sources/AetherEngine/Diagnostics/FelResidualMeasure.swift @@ -0,0 +1,105 @@ +import Foundation +import CoreVideo +import CoreMedia +import os + +/// Diagnostic: how much the enhancement-layer residual changes the picture. With `AETHER_FEL_MEASURE=N` +/// in the environment, every Nth composed frame is composed a second time with the residual off (what a +/// minimal enhancement layer would give) and the two are compared in 10-bit codes per plane: absolute +/// differences for size, and the signed mean (FEL minus MEL) for direction, so a residual that brightens +/// shows as a positive `signed`. Dithering +/// is disabled in both so only the residual differs. Off by default; costs two extra compositions per +/// sampled frame, which can briefly take pool buffers, so it is for measurement runs, not playback. +enum FelResidualMeasure { + static let stride: Int = { + guard let v = ProcessInfo.processInfo.environment["AETHER_FEL_MEASURE"], let n = Int(v), n > 0 else { return 0 } + return n + }() + + private static let queue = DispatchQueue(label: "aether.fel.measure", qos: .utility) + + /// True while a sample is queued or running. A sample retains its two source buffers and takes two + /// pool buffers, so a sample that arrives while one is in flight is skipped rather than queued + /// behind it: a stride smaller than the sampling time would otherwise pile up held buffers. + private static let inFlight = OSAllocatedUnfairLock(initialState: false) + + static func sampleIfDue(composer: DoviFelComposer, base: CVPixelBuffer, enhancement: CVPixelBuffer, + params: DoviMetalParams, frameIndex: UInt32, pts: CMTime) { + guard stride > 0, Int(frameIndex) % stride == 0 else { return } + guard inFlight.withLock({ busy in + if busy { return false } + busy = true + return true + }) else { return } + let b = SendableBuffer(value: base), e = SendableBuffer(value: enhancement) + queue.async { + defer { inFlight.withLock { $0 = false } } + var full = params; full.ditherMode = .none; full.flags.remove(.disableResidual) + var mel = params; mel.ditherMode = .none; mel.flags.insert(.disableResidual) + guard let a = composer.compose(baseBuffer: b.value, enhancementBuffer: e.value, params: full, frameIndex: frameIndex), + let m = composer.compose(baseBuffer: b.value, enhancementBuffer: e.value, params: mel, frameIndex: frameIndex) + else { return } + let planes = ["Y", "Cb", "Cr"] + var parts: [String] = [] + for (k, name) in planes.enumerated() { + let s = stats(a, m, plane: k) + parts.append(String(format: "%@ diff=%.2f%% mean=%.3f signed=%+.3f p99=%d max=%d ge2=%.3f%% ge4=%.3f%%", + name, s.diffPct, s.mean, s.signedMean, s.p99, s.max, s.ge2Pct, s.ge4Pct)) + if k == 0 { + parts.append("Yband[0-255|256-511|512-767|768+] mean=" + s.bandMean.map { String(format: "%.2f", $0) }.joined(separator: "/") + + " signed=" + s.bandSignedMean.map { String(format: "%+.2f", $0) }.joined(separator: "/") + + " share%=" + s.bandFrac.map { String(format: "%.1f", $0) }.joined(separator: "/")) + } + } + EngineLog.emit("[FELMeasure] pts=\(String(format: "%.3f", pts.seconds)) " + parts.joined(separator: " | "), category: .swPlayback) + } + } + + private struct Stats { var diffPct = 0.0, mean = 0.0, signedMean = 0.0, p99 = 0, max = 0, ge2Pct = 0.0, ge4Pct = 0.0, bandMean = [Double](repeating: 0, count: 4), bandSignedMean = [Double](repeating: 0, count: 4), bandFrac = [Double](repeating: 0, count: 4) } + + private static func stats(_ x: CVPixelBuffer, _ y: CVPixelBuffer, plane k: Int) -> Stats { + CVPixelBufferLockBaseAddress(x, .readOnly); CVPixelBufferLockBaseAddress(y, .readOnly) + defer { CVPixelBufferUnlockBaseAddress(x, .readOnly); CVPixelBufferUnlockBaseAddress(y, .readOnly) } + let p = k == 0 ? 0 : 1 + let w = CVPixelBufferGetWidthOfPlane(x, p), h = CVPixelBufferGetHeightOfPlane(x, p) + let stx = CVPixelBufferGetBytesPerRowOfPlane(x, p) / 2, sty = CVPixelBufferGetBytesPerRowOfPlane(y, p) / 2 + let px = CVPixelBufferGetBaseAddressOfPlane(x, p)!.assumingMemoryBound(to: UInt16.self) + let py = CVPixelBufferGetBaseAddressOfPlane(y, p)!.assumingMemoryBound(to: UInt16.self) + var hist = [Int](repeating: 0, count: 1024) + var bandSum = [Int](repeating: 0, count: 4), bandSigned = [Int](repeating: 0, count: 4), bandN = [Int](repeating: 0, count: 4) + var signedSum = 0 + let comps = k == 0 ? 1 : 2, off = k == 2 ? 1 : 0 + for row in 0..> 6), b = Int(py[row * sty + i] >> 6) + let d = abs(a - b) + hist[min(d, 1023)] += 1 + signedSum += a - b + if k == 0 { let band = min(b >> 8, 3); bandSum[band] += d; bandSigned[band] += a - b; bandN[band] += 1 } + } + } + let n = w * h + var s = Stats() + var sum = 0, cum = 0, p99Set = false + for d in 0..<1024 { + sum += d * hist[d] + cum += hist[d] + if hist[d] > 0 { s.max = d } + if !p99Set, cum * 100 >= n * 99 { s.p99 = d; p99Set = true } + } + s.diffPct = Double(n - hist[0]) * 100 / Double(n) + s.mean = Double(sum) / Double(n) + s.signedMean = Double(signedSum) / Double(n) + s.ge2Pct = Double(hist[2...].reduce(0, +)) * 100 / Double(n) + s.ge4Pct = Double(hist[4...].reduce(0, +)) * 100 / Double(n) + for i in 0..<4 { + if bandN[i] > 0 { + s.bandMean[i] = Double(bandSum[i]) / Double(bandN[i]) + s.bandSignedMean[i] = Double(bandSigned[i]) / Double(bandN[i]) + } + s.bandFrac[i] = Double(bandN[i]) * 100 / Double(n) + } + return s + } +} diff --git a/Tests/AetherEngineTests/DoviCpuModelTests.swift b/Tests/AetherEngineTests/DoviCpuModelTests.swift new file mode 100644 index 000000000..9d2786ed5 --- /dev/null +++ b/Tests/AetherEngineTests/DoviCpuModelTests.swift @@ -0,0 +1,440 @@ +import Testing +import Foundation +import CoreVideo +@testable import AetherEngine + +/// M8: an independent model of the composition, written from the RPU syntax and the reshaping +/// equations rather than from the shader, and compared with the GPU on frames captured from real +/// titles. The real titles carry what synthetic test content cannot produce (multi-piece luma curves, +/// MMR chroma, large NLQ scales), so this is the check on those paths. +/// +/// The model works in `Double` and `Int64`, evaluates the curves with explicit `pow`, and samples the +/// enhancement layer with its own bilinear filter. It compares the un-quantised 12-bit VDR code +/// (`DoviDither.debugRawVDR`), so the final rounding cannot hide a disagreement. +/// +/// What it settles: arithmetic, piece selection, MMR term layout, NLQ, clamps. What it cannot: the +/// spec readings the shader and the model share by construction (the chroma-site luma filter, the +/// half-step NLQ reconstruction); those were settled against DoViBaker, an independent implementation. +struct DoviCpuModelTests { + + // MARK: - Planes + + /// One P010 plane as 10-bit codes, `components` interleaved samples per pixel. + struct Plane { + let width: Int, height: Int, components: Int + let raw: [UInt16] + func code(_ x: Int, _ y: Int, _ c: Int = 0) -> Int { + Int(raw[(y * width + x) * components + c] >> 6) + } + /// Raw 16-bit word, for the enhancement-layer filter (the GPU filters the UNORM word). + func word(_ x: Int, _ y: Int, _ c: Int = 0) -> Double { + Double(raw[(y * width + x) * components + c]) + } + } + + static func plane(_ b: CVPixelBuffer, _ index: Int, components: Int) -> Plane { + CVPixelBufferLockBaseAddress(b, .readOnly) + defer { CVPixelBufferUnlockBaseAddress(b, .readOnly) } + let w = CVPixelBufferGetWidthOfPlane(b, index), h = CVPixelBufferGetHeightOfPlane(b, index) + let stride = CVPixelBufferGetBytesPerRowOfPlane(b, index) / 2 + let base = CVPixelBufferGetBaseAddressOfPlane(b, index)!.assumingMemoryBound(to: UInt16.self) + var out = [UInt16](); out.reserveCapacity(w * h * components) + for y in 0.. Int { + let last = Int(c.numPivots) - 2 + var k = 0 + for i in 0...max(last, 0) where v >= Int(c.pivots[i]) { k = i } + return min(max(k, 0), max(last, 0)) + } + + /// Normalised prediction of one component. + func predict(_ c: DoviPolynomialParams, _ mmr: DoviMmrParams, blValue: Int, + y: Double, u: Double, v: Double) -> Double { + if c.numPivots < 2 { return Double(blValue) / blMax } + let k = piece(c, blValue) + if c.method[k] == DoviMapping.mmr.rawValue { + let inv = pow(2.0, -Double(mmr.coefShift)) + var s = Double(mmr.constant[k]) * inv + let terms = [y, u, v, y * u, y * v, u * v, y * u * v] + let order = min(Int(c.order[k]), 3) + for o in 0..= 1 { s += Double(c.coef[k][1]) * inv * x } + if c.order[k] >= 2 { s += Double(c.coef[k][2]) * inv * x * x } + return s + } + + func denormalize(_ n: Double) -> Int { Int((n * blMax * scale).rounded(.toNearestOrEven)) } + + /// Inverse NLQ in `Int64`, reconstructing at the middle of the quantisation step. + func inverseNlq(_ n: DoviNlqParams, _ el: Int) -> Int { + guard n.nlqMethod == .linear, n.nlqScale != 0 else { return 0 } + let r = Int64(el) - Int64(n.nlqOffset) + if r == 0 { return 0 } + let magFp = (2 * abs(r) - 1) * Int64(n.nlqScale) + let thr2 = 2 * Int64(n.nlqThreshold) + var mag = (magFp + thr2) >> Int64(n.nlqShift + 1) + if n.vdrInMax > 0 && mag > Int64(n.vdrInMax) { mag = Int64(n.vdrInMax) } + return Int(r < 0 ? -mag : mag) + } + + func nlqActive(_ n: DoviNlqParams) -> Bool { + !p.flags.contains(.disableResidual) && n.nlqMethod == .linear && n.nlqScale != 0 + } + + /// Bilinear sample of an enhancement-layer plane at the centre of base-layer pixel (x, y) of a + /// plane `baseW` x `baseH`, as a 10-bit code (the word is UNORM x 65535 / 64). + func elCode(_ el: Plane, comp: Int, x: Int, y: Int, baseW: Int, baseH: Int) -> Int { + let u = (Double(x) + 0.5) / Double(baseW) * Double(el.width) - 0.5 + let v = (Double(y) + 0.5) / Double(baseH) * Double(el.height) - 0.5 + let x0 = Int(floor(u)), y0 = Int(floor(v)) + let fx = u - Double(x0), fy = v - Double(y0) + func w(_ xx: Int, _ yy: Int) -> Double { + el.word(min(max(xx, 0), el.width - 1), min(max(yy, 0), el.height - 1), comp) + } + let top = w(x0, y0) * (1 - fx) + w(x0 + 1, y0) * fx + let bot = w(x0, y0 + 1) * (1 - fx) + w(x0 + 1, y0 + 1) * fx + let word = top * (1 - fy) + bot * fy + return Int((word / 64.0).rounded(.toNearestOrEven)) + } + + /// One component's pieces before any clamp: the normalised prediction, the VDR code it + /// denormalises to, the enhancement-layer code and the residual it decodes to. + struct Parts { var predNorm = 0.0; var pred = 0; var elCode = 0; var residual = 0; var active = false + var total: Int { pred + residual } } + + func lumaParts(bl: Plane, blUV: Plane, el: Plane, x: Int, y: Int) -> Parts { + let by = bl.code(x, y) + let cx = min(x >> 1, blUV.width - 1), cy = min(y >> 1, blUV.height - 1) + let yn = Double(by) / blMax + let un = Double(blUV.code(cx, cy, 0)) / blMax + let vn = Double(blUV.code(cx, cy, 1)) / blMax + var r = Parts() + r.predNorm = predict(p.polyY, p.mmrY, blValue: by, y: yn, u: un, v: vn) + r.pred = denormalize(r.predNorm) + if nlqActive(p.nlqY) { + r.active = true + r.elCode = elCode(el, comp: 0, x: x, y: y, baseW: bl.width, baseH: bl.height) + r.residual = inverseNlq(p.nlqY, r.elCode) + } + return r + } + + /// Luma at the chroma site: [1 2 1] around the even column in both rows of the cell, rounded per + /// row, then the two rows averaged; edges clamped (DoViBaker's reading, which Dolby's EL confirms). + static func chromaSiteLuma(_ bl: Plane, x: Int, y: Int) -> Int { + func at(_ c: Int, _ r: Int) -> Int { bl.code(min(max(c, 0), bl.width - 1), min(r, bl.height - 1)) } + let rows = [y * 2, y * 2 + 1].map { r in (at(x * 2 - 1, r) + 2 * at(x * 2, r) + at(x * 2 + 1, r) + 2) >> 2 } + return (rows[0] + rows[1] + 1) >> 1 + } + + func chromaParts(bl: Plane, blUV: Plane, elUV: Plane, x: Int, y: Int) -> (cb: Parts, cr: Parts) { + let by = Self.chromaSiteLuma(bl, x: x, y: y) + let cb = blUV.code(x, y, 0), cr = blUV.code(x, y, 1) + let yn = Double(by) / blMax, un = Double(cb) / blMax, vn = Double(cr) / blMax + var a = Parts(), b = Parts() + a.predNorm = predict(p.polyCb, p.mmrCb, blValue: cb, y: yn, u: un, v: vn) + b.predNorm = predict(p.polyCr, p.mmrCr, blValue: cr, y: yn, u: un, v: vn) + a.pred = denormalize(a.predNorm); b.pred = denormalize(b.predNorm) + if !p.flags.contains(.disableResidual) { + if nlqActive(p.nlqCb) { + a.active = true + a.elCode = elCode(elUV, comp: 0, x: x, y: y, baseW: blUV.width, baseH: blUV.height) + a.residual = inverseNlq(p.nlqCb, a.elCode) + } + if nlqActive(p.nlqCr) { + b.active = true + b.elCode = elCode(elUV, comp: 1, x: x, y: y, baseW: blUV.width, baseH: blUV.height) + b.residual = inverseNlq(p.nlqCr, b.elCode) + } + } + return (a, b) + } + + func luma(bl: Plane, blUV: Plane, el: Plane, x: Int, y: Int) -> Int { + let by = bl.code(x, y) + let cx = min(x >> 1, blUV.width - 1), cy = min(y >> 1, blUV.height - 1) + let yn = Double(by) / blMax + let un = Double(blUV.code(cx, cy, 0)) / blMax + let vn = Double(blUV.code(cx, cy, 1)) / blMax + var v = denormalize(predict(p.polyY, p.mmrY, blValue: by, y: yn, u: un, v: vn)) + if nlqActive(p.nlqY) { + v += inverseNlq(p.nlqY, elCode(el, comp: 0, x: x, y: y, baseW: bl.width, baseH: bl.height)) + } + return min(max(v, 0), Int(p.vdrMax)) + } + + func chroma(bl: Plane, blUV: Plane, elUV: Plane, x: Int, y: Int) -> (Int, Int) { + let by = Self.chromaSiteLuma(bl, x: x, y: y) + let cb = blUV.code(x, y, 0), cr = blUV.code(x, y, 1) + let yn = Double(by) / blMax, un = Double(cb) / blMax, vn = Double(cr) / blMax + var vcb = denormalize(predict(p.polyCb, p.mmrCb, blValue: cb, y: yn, u: un, v: vn)) + var vcr = denormalize(predict(p.polyCr, p.mmrCr, blValue: cr, y: yn, u: un, v: vn)) + if !p.flags.contains(.disableResidual) { + if nlqActive(p.nlqCb) { + vcb += inverseNlq(p.nlqCb, elCode(elUV, comp: 0, x: x, y: y, baseW: blUV.width, baseH: blUV.height)) + } + if nlqActive(p.nlqCr) { + vcr += inverseNlq(p.nlqCr, elCode(elUV, comp: 1, x: x, y: y, baseW: blUV.width, baseH: blUV.height)) + } + } + let m = Int(p.vdrMax) + return (min(max(vcb, 0), m), min(max(vcr, 0), m)) + } + } + + // MARK: - Comparison + + struct Stat { + var n = 0, exact = 0, over1 = 0, maxAbs = 0, sum = 0 + mutating func add(_ d: Int) { + n += 1; let a = abs(d) + if a == 0 { exact += 1 } + if a > 1 { over1 += 1 } + maxAbs = max(maxAbs, a); sum += d + } + var exactPct: Double { n == 0 ? 0 : 100 * Double(exact) / Double(n) } + var over1Pct: Double { n == 0 ? 0 : 100 * Double(over1) / Double(n) } + var description: String { + String(format: "n=%d exact %.2f%% |d|>1 %.3f%% max %d mean %+.3f", n, exactPct, over1Pct, maxAbs, + n == 0 ? 0 : Double(sum) / Double(n)) + } + } + + static func describe(_ c: DoviPolynomialParams, _ mmr: DoviMmrParams) -> String { + guard c.numPivots >= 2 else { return "identity" } + let pieces = Int(c.numPivots) - 1 + let kinds = (0.. Int { Int(p.raw[(y * p.width + x) * p.components + c] >> 4) } + for y in stride(from: 3, to: bl.height, by: 7) { + for x in stride(from: 5, to: bl.width, by: 11) { + yStat.add(model.luma(bl: bl, blUV: blUV, el: el, x: x, y: y) - gpu(gy, x, y, 0)) + } + } + for y in stride(from: 1, to: blUV.height, by: 5) { + for x in stride(from: 2, to: blUV.width, by: 7) { + let (cb, cr) = model.chroma(bl: bl, blUV: blUV, elUV: elUV, x: x, y: y) + cbStat.add(cb - gpu(guv, x, y, 0)) + crStat.add(cr - gpu(guv, x, y, 1)) + } + } + } + print("[m8] \(name): Y \(yStat.description)") + print("[m8] \(name): Cb \(cbStat.description)") + print("[m8] \(name): Cr \(crStat.description)") + // The enhancement-layer filter is the one place the two are not the same arithmetic (the + // hardware bilinear weights are fixed point), so a residual code can land one off; anything + // beyond that is a disagreement about the maths. + for (label, st) in [("Y", yStat), ("Cb", cbStat), ("Cr", crStat)] { + #expect(st.over1Pct < 0.5, "\(label): \(st.description)") + #expect(st.exactPct > 90, "\(label): \(st.description)") + } + } + + // MARK: - Loss census (M1) and error attribution (M3, M4, M6) + + /// Counts, per plane, every pixel at a point where information can be lost. Fractions of the + /// sampled pixels. The output range clamps are video range: 64...940 luma, 64...960 chroma. + struct Census { + var n = 0 + var predBelow0 = 0, predAbove1 = 0 + var nlqAtMax = 0, residualPixels = 0 + var vdrBelow0 = 0, vdrAboveMax = 0 + var outBelowLo = 0, outAboveHi = 0 + var elAt0 = 0, elAt1023 = 0 + var worstOvershoot = 0 + mutating func add(_ q: Model.Parts, _ nlq: DoviNlqParams, vdrMax: Int, scale: Double, lo: Int, hi: Int) { + n += 1 + if q.predNorm < 0 { predBelow0 += 1 } + if q.predNorm > 1 { predAbove1 += 1 } + if q.active { + residualPixels += 1 + if abs(q.residual) >= Int(nlq.vdrInMax) && nlq.vdrInMax > 0 { nlqAtMax += 1 } + if q.elCode <= 0 { elAt0 += 1 } + if q.elCode >= 1023 { elAt1023 += 1 } + } + let t = q.total + if t < 0 { vdrBelow0 += 1 } + if t > vdrMax { vdrAboveMax += 1 } + let c = min(max(t, 0), vdrMax) + let out = Int(floor(Double(c) / scale + 0.5)) + if out < lo { outBelowLo += 1; worstOvershoot = max(worstOvershoot, lo - out) } + if out > hi { outAboveHi += 1; worstOvershoot = max(worstOvershoot, out - hi) } + } + var line: String { + func f(_ v: Int) -> String { String(format: "%.4f%%", n == 0 ? 0 : 100 * Double(v) / Double(n)) } + return "n=\(n) pred<0 \(f(predBelow0)) pred>1 \(f(predAbove1)) | residual px \(f(residualPixels)) at vdr_in_max \(f(nlqAtMax)) EL=0 \(f(elAt0)) EL=1023 \(f(elAt1023)) | vdr<0 \(f(vdrBelow0)) vdr>max \(f(vdrAboveMax)) | outhi \(f(outAboveHi)) worst overshoot \(worstOvershoot) codes" + } + } + + @Test("M1: loss census on real titles, every clamp counted", .enabled(if: !DoviFelRealFileTests.availableFixtures.isEmpty, DoviFelRealFileTests.needsClips), + arguments: DoviFelRealFileTests.availableFixtures) + func lossCensus(name: String) throws { + let rec = try #require(DoviFelRealFileTests.RecordingComposer(limit: 6, stride: 40)) + let s = try DoviFelRealFileTests.open(name, composer: rec) + defer { s.decoder.close(); s.demuxer.close() } + DoviFelRealFileTests.feed(s, count: 300) + s.decoder.flush() + var y = Census(), cb = Census(), cr = Census() + for cap in rec.captures { + let p = cap.params + let m = Model(p) + let bl = Self.plane(cap.bl, 0, components: 1), blUV = Self.plane(cap.bl, 1, components: 2) + let el = Self.plane(cap.el, 0, components: 1), elUV = Self.plane(cap.el, 1, components: 2) + for yy in stride(from: 3, to: bl.height, by: 5) { + for xx in stride(from: 5, to: bl.width, by: 7) { + y.add(m.lumaParts(bl: bl, blUV: blUV, el: el, x: xx, y: yy), p.nlqY, vdrMax: Int(p.vdrMax), scale: m.scale, lo: 64, hi: 940) + } + } + for yy in stride(from: 1, to: blUV.height, by: 3) { + for xx in stride(from: 2, to: blUV.width, by: 5) { + let (a, b) = m.chromaParts(bl: bl, blUV: blUV, elUV: elUV, x: xx, y: yy) + cb.add(a, p.nlqCb, vdrMax: Int(p.vdrMax), scale: m.scale, lo: 64, hi: 960) + cr.add(b, p.nlqCr, vdrMax: Int(p.vdrMax), scale: m.scale, lo: 64, hi: 960) + } + } + } + print("[m1] \(name): Y \(y.line)") + print("[m1] \(name): Cb \(cb.line)") + print("[m1] \(name): Cr \(cr.line)") + } + + @Test("M3/M4/M6: where the GPU and the exact model still differ, split by cause", .enabled(if: !DoviFelRealFileTests.availableFixtures.isEmpty, DoviFelRealFileTests.needsClips), + arguments: DoviFelRealFileTests.availableFixtures) + func errorAttribution(name: String) throws { + let rec = try #require(DoviFelRealFileTests.RecordingComposer(limit: 6, stride: 40)) + let s = try DoviFelRealFileTests.open(name, composer: rec) + defer { s.decoder.close(); s.demuxer.close() } + DoviFelRealFileTests.feed(s, count: 300) + s.decoder.flush() + var lumaNoRes = Stat(), chromaNoRes = Stat(), lumaRes = Stat(), chromaRes = Stat() + for cap in rec.captures { + let bl = Self.plane(cap.bl, 0, components: 1), blUV = Self.plane(cap.bl, 1, components: 2) + let el = Self.plane(cap.el, 0, components: 1), elUV = Self.plane(cap.el, 1, components: 2) + for residual in [false, true] { + var params = cap.params + params.ditherMode = .debugRawVDR + if !residual { params.flags.insert(.disableResidual) } + let out = try #require(rec.inner.compose(baseBuffer: cap.bl, enhancementBuffer: cap.el, params: params, frameIndex: 0)) + let gy = Self.plane(out, 0, components: 1), guv = Self.plane(out, 1, components: 2) + let m = Model(params) + func g(_ p: Plane, _ x: Int, _ y: Int, _ c: Int) -> Int { Int(p.raw[(y * p.width + x) * p.components + c] >> 4) } + for yy in stride(from: 3, to: bl.height, by: 5) { + for xx in stride(from: 5, to: bl.width, by: 7) { + let d = m.luma(bl: bl, blUV: blUV, el: el, x: xx, y: yy) - g(gy, xx, yy, 0) + if residual { lumaRes.add(d) } else { lumaNoRes.add(d) } + } + } + for yy in stride(from: 1, to: blUV.height, by: 3) { + for xx in stride(from: 2, to: blUV.width, by: 5) { + let (a, b) = m.chroma(bl: bl, blUV: blUV, elUV: elUV, x: xx, y: yy) + let d1 = a - g(guv, xx, yy, 0), d2 = b - g(guv, xx, yy, 1) + if residual { chromaRes.add(d1); chromaRes.add(d2) } else { chromaNoRes.add(d1); chromaNoRes.add(d2) } + } + } + } + } + print("[m3] \(name): luma, no residual (table + float): \(lumaNoRes.description)") + print("[m4] \(name): chroma, no residual (MMR float): \(chromaNoRes.description)") + print("[m6] \(name): luma, with residual (+ EL sampling): \(lumaRes.description)") + print("[m6] \(name): chroma, with residual: \(chromaRes.description)") + } + + /// M7: how much the chroma MMR output depends on how the luma is brought down to the chroma site. + /// Each variant replaces only the luma input of the cross terms; the production reading is the baseline. + @Test("M7: chroma MMR sensitivity to the luma downsampling at the chroma site", .enabled(if: !DoviFelRealFileTests.availableFixtures.isEmpty, DoviFelRealFileTests.needsClips), + arguments: DoviFelRealFileTests.availableFixtures) + func lumaSiteSensitivity(name: String) throws { + let rec = try #require(DoviFelRealFileTests.RecordingComposer(limit: 6, stride: 40)) + let s = try DoviFelRealFileTests.open(name, composer: rec) + defer { s.decoder.close(); s.demuxer.close() } + DoviFelRealFileTests.feed(s, count: 300) + s.decoder.flush() + let names = ["top row", "2x2 box", "vertical 2-tap", "bottom row"] + var sq = [Double](repeating: 0, count: names.count), mx = [Int](repeating: 0, count: names.count) + var n = 0 + for cap in rec.captures { + let p = cap.params + let m = Model(p) + let bl = Self.plane(cap.bl, 0, components: 1), blUV = Self.plane(cap.bl, 1, components: 2) + func at(_ x: Int, _ y: Int) -> Int { bl.code(min(max(x, 0), bl.width - 1), min(max(y, 0), bl.height - 1)) } + for yy in stride(from: 1, to: blUV.height, by: 3) { + for xx in stride(from: 2, to: blUV.width, by: 5) { + let cb = blUV.code(xx, yy, 0), cr = blUV.code(xx, yy, 1) + let un = Double(cb) / m.blMax, vn = Double(cr) / m.blMax + func pred(_ luma: Double) -> (Int, Int) { + let yn = luma / m.blMax + return (m.denormalize(m.predict(p.polyCb, p.mmrCb, blValue: cb, y: yn, u: un, v: vn)), + m.denormalize(m.predict(p.polyCr, p.mmrCr, blValue: cr, y: yn, u: un, v: vn))) + } + let x0 = xx * 2, y0 = yy * 2 + let base = pred(Double(Model.chromaSiteLuma(bl, x: xx, y: yy))) + let box = (at(x0, y0) + at(x0 + 1, y0) + at(x0, y0 + 1) + at(x0 + 1, y0 + 1)) + let variants = [pred(Double(at(x0, y0))), pred(Double(box) / 4), + pred(Double((at(x0, y0) + at(x0, y0 + 1) + 1) >> 1)), pred(Double(at(x0, y0 + 1)))] + for (i, v) in variants.enumerated() { + for d in [v.0 - base.0, v.1 - base.1] { sq[i] += Double(d * d); mx[i] = max(mx[i], abs(d)) } + } + n += 2 + } + } + } + for i in 0.. Void) { + record(baseBuffer, enhancementBuffer, params) + inner.composeAsync(baseBuffer: baseBuffer, enhancementBuffer: enhancementBuffer, params: params, + frameIndex: frameIndex, completion: completion) + } + func composeAsyncWithAcquiredSlot(baseBuffer: CVPixelBuffer, enhancementBuffer: CVPixelBuffer, + params: DoviMetalParams, frameIndex: UInt32, + completion: @escaping @Sendable (CVPixelBuffer?) -> Void) { + record(baseBuffer, enhancementBuffer, params) + inner.composeAsyncWithAcquiredSlot(baseBuffer: baseBuffer, enhancementBuffer: enhancementBuffer, + params: params, frameIndex: frameIndex, completion: completion) + } + func acquireSlot() { inner.acquireSlot() } + func releaseSlot() { inner.releaseSlot() } + func flush() { inner.flush() } + func flushTextureCache() { inner.flushTextureCache() } + func takePeakSlotsInUse() -> Int { inner.takePeakSlotsInUse() } + var allocatedPoolBuffers: Int { inner.allocatedPoolBuffers } + var meanGPUTimeMs: Double? { inner.meanGPUTimeMs } + } + + final class Sink: @unchecked Sendable { + private let lock = NSLock() + private(set) var pts: [Double] = [] + private(set) var withRPU = 0 + private(set) var sizes = Set() + private(set) var formats = Set() + + func take(_ buffer: CVPixelBuffer, _ time: CMTime) { + lock.lock(); defer { lock.unlock() } + pts.append(time.seconds) + sizes.insert("\(CVPixelBufferGetWidth(buffer))x\(CVPixelBufferGetHeight(buffer))") + formats.insert(CVPixelBufferGetPixelFormatType(buffer)) + if CVBufferCopyAttachment(buffer, DoviDisplayMetadata.rpuDataKey as CFString, nil) != nil { withRPU += 1 } + } + func reset() { lock.lock(); pts = []; withRPU = 0; lock.unlock() } + } + + struct Session { + let demuxer: Demuxer + let decoder: DualLayerVideoDecoder + let sink: Sink + let videoIndex: Int32 + } + + static func open(_ name: String, composer: (any FelComposing)? = DoviFelComposer()) throws -> Session { + let demuxer = Demuxer() + try demuxer.open(url: url(name)) + let idx = demuxer.videoStreamIndex + let stream = try #require(demuxer.stream(at: idx)) + let sink = Sink() + let decoder = DualLayerVideoDecoder(composer: composer) + try decoder.open(stream: stream) { buf, pts, _ in sink.take(buf, pts) } + return Session(demuxer: demuxer, decoder: decoder, sink: sink, videoIndex: Int32(idx)) + } + + /// Feeds up to `count` video packets; returns how many were fed. + @discardableResult + static func feed(_ s: Session, count: Int, epoch: UInt64? = nil) -> Int { + var fed = 0 + while fed < count { + guard let pkt = try? s.demuxer.readPacket() else { break } + if pkt.pointee.stream_index == s.videoIndex { + fed += 1 + s.decoder.decode(packet: pkt, epoch: epoch) + } + av_packet_unref(pkt) + var opt: UnsafeMutablePointer? = pkt + trackedPacketFree(&opt) + } + return fed + } + + static func lumaPlane(_ b: CVPixelBuffer) -> [UInt16] { + CVPixelBufferLockBaseAddress(b, .readOnly) + defer { CVPixelBufferUnlockBaseAddress(b, .readOnly) } + let w = CVPixelBufferGetWidthOfPlane(b, 0), h = CVPixelBufferGetHeightOfPlane(b, 0) + let stride = CVPixelBufferGetBytesPerRowOfPlane(b, 0) / 2 + let base = CVPixelBufferGetBaseAddressOfPlane(b, 0)!.assumingMemoryBound(to: UInt16.self) + var out = [UInt16](); out.reserveCapacity(w * h) + for y in 0..> 6) } } + return out + } + + // MARK: - Tests + + @Test("every real FEL file reconstructs nearly every packet, in presentation order, each with its RPU", + .enabled(if: !DoviFelRealFileTests.availableFixtures.isEmpty, DoviFelRealFileTests.needsClips), + arguments: DoviFelRealFileTests.availableFixtures) + func reconstructsAndOrders(name: String) throws { + let s = try Self.open(name) + defer { s.decoder.close(); s.demuxer.close() } + let fed = Self.feed(s, count: 240) + s.decoder.flush() + + #expect(s.decoder.streamClassification == .fel) + let got = s.sink.pts.count + #expect(fed == 240) + #expect(got >= Int(Double(fed) * 0.9), "\(got) frames from \(fed) packets") + #expect(s.sink.withRPU == got, "every output frame must carry display metadata") + // Frames that merely pass through as their base layer would satisfy every check above, so the + // composer must also have run on a clear majority of them (a few at the start wait for their + // enhancement layer's first picture, and a few at the end go out as the base layer). + let composed = s.decoder.composedFrameCount + print("[compose] \(name): composed \(composed) of \(got) delivered frames") + #expect(composed > 0, "no frame was composed: the whole file went out as its base layer") + #expect(composed * 10 >= got * 8, "only \(composed) of \(got) frames were composed") + #expect(s.sink.formats == [kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange]) + #expect(s.sink.sizes.count == 1, "\(s.sink.sizes)") + // Delivery is in decode order by design; `SampleBufferRenderer` restores + // presentation order with a sorted buffer that releases once it holds more + // than `reorderDepth` (4) frames. Replay that exact rule: the file's reorder + // depth must fit, or the renderer would show frames out of order. + let pts = s.sink.pts + var buffer: [Double] = [] + var released: [Double] = [] + for t in pts { + buffer.insert(t, at: buffer.firstIndex(where: { $0 > t }) ?? buffer.endIndex) + while buffer.count > 4 { released.append(buffer.removeFirst()) } + } + released.append(contentsOf: buffer) + let bad = zip(released, released.dropFirst()).filter { $1 <= $0 }.count + let inversions = zip(pts, pts.dropFirst()).filter { $1 <= $0 }.count + print("[order] \(name): \(inversions) decode-order inversions of \(pts.count); after the renderer's 4-deep reorder: \(bad) regressions") + #expect(bad == 0, "reorder depth exceeds the renderer buffer: \(bad) regressions") + #expect(Set(pts).count == pts.count, "duplicate presentation timestamps delivered") + } + + @Test("the enhancement layer is aligned with the base layer: the residual acts, and never exceeds vdr_in_max", + .enabled(if: !DoviFelRealFileTests.availableFixtures.isEmpty, DoviFelRealFileTests.needsClips), + arguments: DoviFelRealFileTests.availableFixtures) + func residualIsAlignedAndNonTrivial(name: String) throws { + let rec = try #require(RecordingComposer()) + let s = try Self.open(name, composer: rec) + defer { s.decoder.close(); s.demuxer.close() } + Self.feed(s, count: 300) + s.decoder.flush() + + let caps = rec.captures + try #require(!caps.isEmpty, "no frames reached the composer") + // The frame whose enhancement layer carries the most signal. + func spread(_ c: RecordingComposer.Capture) -> Int { + let y = Self.lumaPlane(c.el); return Int(y.max() ?? 0) - Int(y.min() ?? 0) + } + let cap = caps.max(by: { spread($0) < spread($1) })! + let flatEverywhere = spread(cap) == 0 + #expect(!cap.params.flags.contains(.disableResidual), "fixture is FEL, residual must be live") + + var without = cap.params + without.flags.insert(.disableResidual) + without.ditherMode = .none + var with = cap.params + with.ditherMode = .none + + let a = try #require(rec.inner.compose(baseBuffer: cap.bl, enhancementBuffer: cap.el, params: without, frameIndex: 0)) + let b = try #require(rec.inner.compose(baseBuffer: cap.bl, enhancementBuffer: cap.el, params: with, frameIndex: 0)) + let ya = Self.lumaPlane(a), yb = Self.lumaPlane(b) + #expect(ya.count == yb.count) + + var differing = 0 + var sumAbs = 0 + var maxAbs = 0 + for i in 0.. String { + let m = Double(v.reduce(0) { $0 + Int($1) }) / Double(v.count) + return "mean \(String(format: "%.1f", m)) min \(v.min() ?? 0) max \(v.max() ?? 0)" + } + let elY = Self.lumaPlane(cap.el) + print("[diag] \(name): BL \(CVPixelBufferGetWidth(cap.bl))x\(CVPixelBufferGetHeight(cap.bl)) EL \(CVPixelBufferGetWidth(cap.el))x\(CVPixelBufferGetHeight(cap.el)) flags=\(cap.params.flags.rawValue)") + print("[diag] BL luma \(stat(Self.lumaPlane(cap.bl))) EL luma \(stat(elY)) out(no res) \(stat(ya)) out(res) \(stat(yb))") + let n = cap.params.nlqY + print("[diag] nlqY method=\(n.nlqMethod) offset=\(n.nlqOffset) scale=\(n.nlqScale) thr=\(n.nlqThreshold) vdrInMax=\(n.vdrInMax) shift=\(n.nlqShift) pivots=\(n.nlqPivots)") + print("[diag] polyY pivots=\(cap.params.polyY.pivots.prefix(Int(cap.params.polyY.numPivots))) order=\(cap.params.polyY.order.prefix(3)) coef0=\(cap.params.polyY.coef[0]) shift0=\(cap.params.polyY.coefShift[0]) blMax=\(cap.params.blMax) vdrScale=\(cap.params.vdrScale)") + let fraction = Double(differing) / Double(ya.count) + let meanAbs = differing == 0 ? 0 : Double(sumAbs) / Double(differing) + // A misaligned or mis-scaled EL would either do nothing or look like noise + // of full amplitude; a genuine detail residual is sparse-to-moderate and small. + print("[residual] \(name): \(String(format: "%.1f", fraction * 100))% pixels differ, mean |d|=\(String(format: "%.2f", meanAbs)), max |d|=\(maxAbs)") + // The residual is bounded by construction: |r| <= vdr_in_max VDR codes, i.e. vdr_in_max / vdr_scale output codes. + let bound = Int(cap.params.nlqY.vdrInMax) / Int(cap.params.vdrScale) + 1 + #expect(maxAbs <= bound, "residual \(maxAbs) exceeds vdr_in_max bound \(bound)") + if flatEverywhere { + print("[residual] \(name): EL flat in all \(caps.count) sampled frames; residual correctly does nothing") + #expect(differing == 0) + } else { + #expect(fraction > 0.001, "the residual had no effect on luma") + } + } + + /// N5: what a base-layer fallback frame costs. When the output pool is empty the pipeline presents the + /// base layer as it is; this measures how far that frame is from the composed one on each real title, + /// in 10-bit codes per plane. The comment on the fallback used to claim ~2/4096 codes, which came from + /// the near-identity synthetic stream. + @Test("a base-layer fallback frame differs from the composed frame by a measured amount on real titles", + .enabled(if: !DoviFelRealFileTests.availableFixtures.isEmpty, DoviFelRealFileTests.needsClips), + arguments: DoviFelRealFileTests.availableFixtures) + func fallbackFrameDelta(name: String) throws { + let rec = try #require(RecordingComposer(limit: 6, stride: 40)) + let s = try Self.open(name, composer: rec) + defer { s.decoder.close(); s.demuxer.close() } + Self.feed(s, count: 300) + s.decoder.flush() + let caps = rec.captures + try #require(!caps.isEmpty, "no frames reached the composer") + + func planes(_ b: CVPixelBuffer) -> [[Int]] { + CVPixelBufferLockBaseAddress(b, .readOnly); defer { CVPixelBufferUnlockBaseAddress(b, .readOnly) } + let w = CVPixelBufferGetWidthOfPlane(b, 0), h = CVPixelBufferGetHeightOfPlane(b, 0) + let ys = CVPixelBufferGetBytesPerRowOfPlane(b, 0) / 2 + let y = CVPixelBufferGetBaseAddressOfPlane(b, 0)!.assumingMemoryBound(to: UInt16.self) + let cw = CVPixelBufferGetWidthOfPlane(b, 1), ch = CVPixelBufferGetHeightOfPlane(b, 1) + let cs = CVPixelBufferGetBytesPerRowOfPlane(b, 1) / 2 + let c = CVPixelBufferGetBaseAddressOfPlane(b, 1)!.assumingMemoryBound(to: UInt16.self) + var out = [(0..<(w * h)).map { Int(y[($0 / w) * ys + $0 % w] >> 6) }] + for k in 0..<2 { out.append((0..<(cw * ch)).map { Int(c[($0 / cw) * cs + ($0 % cw) * 2 + k] >> 6) }) } + return out + } + var worstMean = 0.0 + for (i, cap) in caps.enumerated() { + var p = cap.params; p.ditherMode = .none + let composed = planes(try #require(rec.inner.compose(baseBuffer: cap.bl, enhancementBuffer: cap.el, params: p, frameIndex: 0))) + let base = planes(cap.bl) + for (k, plane) in ["Y", "Cb", "Cr"].enumerated() { + let d = zip(composed[k], base[k]).map { abs($0 - $1) }.sorted() + let mean = Double(d.reduce(0, +)) / Double(d.count) + worstMean = max(worstMean, mean) + print(String(format: "[fallback] %@ f%d %@: mean |d| %.2f codes (10-bit), p99 %d, max %d", + name, i, plane, mean, d[Int(Double(d.count) * 0.99)], d.last ?? 0)) + } + } + print(String(format: "[fallback] %@: worst per-plane mean |d| %.2f (10-bit codes)", name, worstMean)) + } + + @Test("a flush before the first enhancement packet leaves the enhancement decoder in step: its frames still arrive", + .enabled(if: DoviFelRealFileTests.available(DoviFelRealFileTests.fixtures[0]), DoviFelRealFileTests.needsClips), + .timeLimit(.minutes(2)), arguments: [DoviFelRealFileTests.fixtures[0]]) + func flushBeforeFirstEnhancementPacket(name: String) async throws { + let rec = try #require(RecordingComposer()) + let s = try Self.open(name, composer: rec) + defer { s.decoder.close(); s.demuxer.close() } + // The host seeks (flush) before the stream has been classified, so the enhancement decoder + // opens afterwards, at an epoch the decoder it belongs to has already moved past. + s.decoder.flush() + #expect(s.decoder.enhancementFeedEpochForTesting == nil) + // The host hands every packet the epoch it read, as it does in playback. + Self.feed(s, count: 60, epoch: s.decoder.feedEpoch) + #expect(s.decoder.enhancementFeedEpochForTesting == s.decoder.feedEpoch) + // A composition needs an enhancement frame, so the composer being called proves they arrive. + try await waitFor { !rec.captures.isEmpty } + } + + @Test("a seek followed by a flush delivers only post-seek frames and recovers immediately", + .enabled(if: !DoviFelRealFileTests.firstTwoClips.isEmpty, DoviFelRealFileTests.needsClips), + arguments: DoviFelRealFileTests.firstTwoClips) + func seekFlushRecovers(name: String) throws { + let s = try Self.open(name) + defer { s.decoder.close(); s.demuxer.close() } + Self.feed(s, count: 60) + s.decoder.flush() + let before = s.sink.pts.count + #expect(before > 0) + + #expect(s.demuxer.seek(to: 60.0)) + s.sink.reset() + Self.feed(s, count: 90) + s.decoder.flush() + + let after = s.sink.pts + #expect(after.count >= 60, "only \(after.count) frames after seek") + // Nothing from before the seek target's keyframe may leak through. + #expect((after.min() ?? 0) > 30, "pre-seek frame leaked: \(after.min() ?? -1)") + #expect(s.sink.withRPU == after.count) + } + + @Test("repeated seeks back and forth never wedge the pipeline", + .enabled(if: DoviFelRealFileTests.available(DoviFelRealFileTests.fixtures[0]), DoviFelRealFileTests.needsClips)) + func seekStorm() throws { + let name = Self.fixtures[0] + let s = try Self.open(name) + defer { s.decoder.close(); s.demuxer.close() } + for target in [10.0, 90.0, 5.0, 100.0, 0.0, 45.0] { + #expect(s.demuxer.seek(to: target)) + s.sink.reset() + Self.feed(s, count: 40) + s.decoder.flush() + #expect(s.sink.pts.count >= 25, "target \(target): \(s.sink.pts.count) frames") + } + } + + @Test("close without a flush, and immediate re-open, are safe and leave no session behind", + .enabled(if: DoviFelRealFileTests.available(DoviFelRealFileTests.fixtures[2]), DoviFelRealFileTests.needsClips), + .timeLimit(.minutes(3))) + func openCloseChurn() throws { + let name = Self.fixtures[2] + for round in 0..<6 { + let s = try Self.open(name) + Self.feed(s, count: round % 2 == 0 ? 5 : 40) + s.decoder.close() // deliberately no flush first + s.demuxer.close() + // Packets fed after close must be refused, not resurrect the session. + } + } + + @Test("feeding packets after close is ignored", + .enabled(if: DoviFelRealFileTests.available(DoviFelRealFileTests.fixtures[2]), DoviFelRealFileTests.needsClips)) + func decodeAfterCloseIgnored() throws { + let name = Self.fixtures[2] + let s = try Self.open(name) + Self.feed(s, count: 20) + s.decoder.flush() + s.decoder.close() + s.sink.reset() + Self.feed(s, count: 20) + s.decoder.flush() + #expect(s.sink.pts.isEmpty, "frames delivered after close") + s.demuxer.close() + } + + @Test("a full pass over a whole file keeps memory flat and never stalls", + .enabled(if: DoviFelRealFileTests.available(DoviFelRealFileTests.fixtures[2]), DoviFelRealFileTests.needsClips), + .timeLimit(.minutes(5))) + func longRunStable() throws { + let name = Self.fixtures[2] + let s = try Self.open(name) + defer { s.decoder.close(); s.demuxer.close() } + + func rss() -> Int { + var info = mach_task_basic_info() + var count = mach_msg_type_number_t(MemoryLayout.size / MemoryLayout.size) + _ = withUnsafeMutablePointer(to: &info) { + $0.withMemoryRebound(to: integer_t.self, capacity: Int(count)) { + task_info(mach_task_self_, task_flavor_t(MACH_TASK_BASIC_INFO), $0, &count) + } + } + return Int(info.resident_size) / (1024 * 1024) + } + + var total = 0 + var samples: [Int] = [] + while true { + let fed = Self.feed(s, count: 200) + total += fed + samples.append(rss()) + if fed < 200 { break } + } + s.decoder.flush() + print("[longRun] packets=\(total) frames=\(s.sink.pts.count) rss MiB by 200-packet step: \(samples)") + #expect(total > 300) + #expect(s.sink.pts.count >= Int(Double(total) * 0.95)) + // Steady state: the last third may not sit more than 400 MiB above the first third. + if samples.count >= 6 { + let early = samples[1..<(samples.count / 3 + 1)].max() ?? 0 + let late = samples[(samples.count * 2 / 3)...].max() ?? 0 + #expect(late - early < 400, "resident memory grew \(early) -> \(late) MiB") + } + } } diff --git a/Tests/AetherEngineTests/DoviRealFileExportTests.swift b/Tests/AetherEngineTests/DoviRealFileExportTests.swift new file mode 100644 index 000000000..a08c97830 --- /dev/null +++ b/Tests/AetherEngineTests/DoviRealFileExportTests.swift @@ -0,0 +1,96 @@ +import Testing +import Foundation +import CoreVideo +import AetherLibavformat +import AetherLibavcodec +import AetherLibavutil +@testable import AetherEngine + +#if DEBUG +/// Cross-check feed: writes four captured frames of each real FEL title (decoded BL and EL as +/// yuv420p10le, an RPU.bin) next to our own raw 12-bit output with and without the residual, in a +/// layout an independent implementation can read. These titles have no reference output, so comparing +/// against another implementation is a second-implementation check of the MMR and 8-piece interpretation, +/// not a ground truth. +/// +/// `AETHER_REAL_EXPORT= swift test --filter exportRealTitle` +struct DoviRealFileExportTests { + + static func planar10(_ b: CVPixelBuffer) -> Data { + var d = Data() + func put(_ v: UInt16) { d.append(UInt8(v & 0xff)); d.append(UInt8(v >> 8)) } + let y = DoviCpuModelTests.plane(b, 0, components: 1) + for v in y.raw { put(v >> 6) } + let uv = DoviCpuModelTests.plane(b, 1, components: 2) + for c in 0..<2 { for i in stride(from: c, to: uv.raw.count, by: 2) { put(uv.raw[i] >> 6) } } + return d + } + + @Test("export a real title for the DoViBaker cross-check", + .enabled(if: ProcessInfo.processInfo.environment["AETHER_REAL_EXPORT"] != nil), + .enabled(if: !DoviFelRealFileTests.firstTwoClips.isEmpty, DoviFelRealFileTests.needsClips), + arguments: [0, 1].filter { DoviFelRealFileTests.available(DoviFelRealFileTests.fixtures[$0]) }) + func exportRealTitle(index: Int) throws { + let name = DoviFelRealFileTests.fixtures[index] + let dir = URL(fileURLWithPath: ProcessInfo.processInfo.environment["AETHER_REAL_EXPORT"]!) + .appendingPathComponent("title\(index)") + try FileManager.default.createDirectory(at: dir, withIntermediateDirectories: true) + + let rec = try #require(DoviFelRealFileTests.RecordingComposer(limit: 4, stride: 40)) + let sess = try DoviFelRealFileTests.open(name, composer: rec) + defer { sess.decoder.close(); sess.demuxer.close() } + DoviFelRealFileTests.feed(sess, count: 300) + sess.decoder.flush() + try #require(rec.captures.count == 4, "only \(rec.captures.count) frames reached the composer") + // Pair every captured frame with its own RPU: the survey's "one parameter set" holds per file section, + // not per frame. The key is the serialised block with the per-frame fields blanked. + func key(_ p: DoviMetalParams) -> Data { + var q = p; q.frameIndex = 0; q.ditherMode = .none; q.blWidth = 0; q.blHeight = 0; q.elWidth = 0; q.elHeight = 0; q.minPq = 0; q.maxPq = 0; q.avgPq = 0 + return q.serialize() + } + var rpuByKey: [Data: [UInt8]] = [:] + let dm = Demuxer() + try dm.open(url: DoviFelRealFileTests.url(name)) + defer { dm.close() } + let vi = Int32(dm.videoStreamIndex) + var seen = 0 + while seen < 600, let pkt = try dm.readPacket() { + defer { av_packet_unref(pkt); var o: UnsafeMutablePointer? = pkt; trackedPacketFree(&o) } + guard pkt.pointee.stream_index == vi else { continue } + seen += 1 + let r = DoviRpuConverter.separateDualLayers(pkt, extractEL: false, useCustomBlockSource: true, extractParams: true) + if let raw = r.rpuData, let params = r.rpuParams { rpuByKey[key(params)] = raw } + } + var rpuBin = Data() + for cap in rec.captures { + let raw = try #require(rpuByKey[key(cap.params)], "no RPU in the first 600 packets matches a captured frame's parameters") + rpuBin.append(contentsOf: [0, 0, 0, 1]) + rpuBin.append(contentsOf: raw.prefix(2) == [0x7c, 0x01] ? Array(raw.dropFirst(2)) : raw) + } + print("[export] \(name): \(rpuByKey.count) distinct parameter sets in \(seen) packets") + + var bl = Data(), el = Data(), ours = Data(), base = Data() + for cap in rec.captures { + bl.append(Self.planar10(cap.bl)); el.append(Self.planar10(cap.el)) + var raw = cap.params; raw.ditherMode = .debugRawVDR + var baseOnly = raw; baseOnly.flags.insert(.disableResidual) + for (p, out) in [(raw, 0), (baseOnly, 1)] { + let buf = try #require(rec.inner.compose(baseBuffer: cap.bl, enhancementBuffer: cap.el, params: p, frameIndex: 0)) + for plane in DoviTestSupport.rawVDR(buf) { for v in plane { + let bytes = [UInt8(v & 0xff), UInt8(v >> 8)] + if out == 0 { ours.append(contentsOf: bytes) } else { base.append(contentsOf: bytes) } + } } + } + } + let first = try #require(rec.captures.first) + try bl.write(to: dir.appendingPathComponent("bl_dec.yuv")) + try el.write(to: dir.appendingPathComponent("el_dec.yuv")) + try rpuBin.write(to: dir.appendingPathComponent("out.rpu")) + try ours.write(to: dir.appendingPathComponent("ours12.yuv")) + try base.write(to: dir.appendingPathComponent("ours12_base.yuv")) + try "\(CVPixelBufferGetWidth(first.bl)) \(CVPixelBufferGetHeight(first.bl)) \(rec.captures.count)\n" + .write(to: dir.appendingPathComponent("geometry.txt"), atomically: true, encoding: .utf8) + print("[export] \(name): \(CVPixelBufferGetWidth(first.bl))x\(CVPixelBufferGetHeight(first.bl)) x\(rec.captures.count) -> \(dir.path)") + } +} +#endif diff --git a/Tests/AetherEngineTests/DualLayerGpuSlotWaitTests.swift b/Tests/AetherEngineTests/DualLayerGpuSlotWaitTests.swift new file mode 100644 index 000000000..2a494c1ed --- /dev/null +++ b/Tests/AetherEngineTests/DualLayerGpuSlotWaitTests.swift @@ -0,0 +1,312 @@ +import Testing +import CoreMedia +import CoreVideo +import Foundation +@testable import AetherEngine + +/// The GPU slot wait must not hold the decoder lock. +/// +/// With a composer whose in-flight limit is 1 and whose first completion is +/// held by the test, a second composition started on another thread blocks in +/// `acquireSlot()` with no lock held, so `flush()` returns promptly. On the +/// pre-fix code the wait sat inside `composeAsync` while the VideoToolbox +/// thread held the decoder lock, and `flush()` hung behind it. +@Suite(.serialized, .timeLimit(.minutes(1))) +struct DualLayerGpuSlotWaitTests { + + /// Fake composer with an in-flight limit of 1. The first composition's + /// completion is held until the test releases it; the slot is held with + /// it, the way the real composer holds the slot until the command buffer + /// retires. + final class GatedFakeComposer: FelComposing, @unchecked Sendable { + private let slots = DispatchSemaphore(value: 1) + private let lock = NSLock() + private var held: (@Sendable (CVPixelBuffer?) -> Void)? + private var heldBase: CVPixelBuffer? + private(set) var acquireCalls = 0 + private(set) var completedCount = 0 + var acquireCallCount: Int { lock.lock(); defer { lock.unlock() }; return acquireCalls } + + func acquireSlot() { + lock.lock(); acquireCalls += 1; lock.unlock() + slots.wait() + } + + func releaseSlot() { slots.signal() } + + func composeAsync( + baseBuffer: CVPixelBuffer, + enhancementBuffer: CVPixelBuffer, + params: DoviMetalParams, + frameIndex: UInt32, + completion: @escaping @Sendable (CVPixelBuffer?) -> Void + ) { + acquireSlot() + composeAsyncWithAcquiredSlot( + baseBuffer: baseBuffer, enhancementBuffer: enhancementBuffer, + params: params, frameIndex: frameIndex, completion: completion) + } + + func composeAsyncWithAcquiredSlot( + baseBuffer: CVPixelBuffer, + enhancementBuffer: CVPixelBuffer, + params: DoviMetalParams, + frameIndex: UInt32, + completion: @escaping @Sendable (CVPixelBuffer?) -> Void + ) { + lock.lock() + let isFirst = held == nil && completedCount == 0 + if isFirst { + held = completion + heldBase = baseBuffer + } + lock.unlock() + if !isFirst { + // Later frames finish synchronously through the base layer once + // the test has released the first slot. Ticket ordering in + // `deliver` is what the ordering test exercises, not the hop. + slots.signal() + lock.lock(); completedCount += 1; lock.unlock() + completion(baseBuffer) + } + } + + /// Releases the held first composition: delivers its base layer as the + /// composed result and frees the slot. + func releaseFirst() { + let completion: (@Sendable (CVPixelBuffer?) -> Void)? + let base: CVPixelBuffer? + lock.lock() + completion = held; base = heldBase + held = nil; heldBase = nil + completedCount += 1 + lock.unlock() + if let completion { + completion(base) + } + slots.signal() + } + + func flush() {} + } + + private static let frameTicks: Int64 = 3754 + private static func pts(_ frame: Int64) -> CMTime { + CMTime(value: frame * frameTicks, timescale: 90000) + } + + private static func makeBuffer(width: Int, height: Int) -> CVPixelBuffer? { + var out: CVPixelBuffer? + let status = CVPixelBufferCreate(kCFAllocatorDefault, width, height, + kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange, + [kCVPixelBufferMetalCompatibilityKey: true, + kCVPixelBufferIOSurfacePropertiesKey: [:] as CFDictionary] as CFDictionary, + &out) + return status == kCVReturnSuccess ? out : nil + } + + private final class Deliveries: @unchecked Sendable { + private let lock = NSLock() + private(set) var points: [CMTime] = [] + private var metadata: [Data?] = [] + func record(_ buffer: CVPixelBuffer, _ pts: CMTime, _ hdr10: Data?) { + let payload = CVBufferCopyAttachment(buffer, DoviDisplayMetadata.rpuDataKey as CFString, nil) as? Data + lock.lock(); points.append(pts); metadata.append(payload); lock.unlock() + } + var count: Int { lock.lock(); defer { lock.unlock() }; return points.count } + var metadataPayloads: [Data?] { lock.lock(); defer { lock.unlock() }; return metadata } + } + + /// Set once by a worker thread when it has finished; read by the test. + private final class Flag: @unchecked Sendable { + private let lock = NSLock() + private var value = false + func set() { lock.lock(); value = true; lock.unlock() } + var isSet: Bool { lock.lock(); defer { lock.unlock() }; return value } + } + + /// Baton-pass box: the test thread stops touching the buffer once it is + /// handed to the background thread, mirroring the decoder's own handoff. + private struct SendablePixelBuffer: @unchecked Sendable { + let value: CVPixelBuffer + } + + @Test("a second composition blocked on the GPU slot does not stall flush()") + func blockedSlotDoesNotStallFlush() async throws { + let bl0 = try #require(Self.makeBuffer(width: 64, height: 64)) + let el0 = try #require(Self.makeBuffer(width: 32, height: 32)) + let bl1 = try #require(Self.makeBuffer(width: 64, height: 64)) + let el1 = try #require(Self.makeBuffer(width: 32, height: 32)) + + let composer = GatedFakeComposer() + let deliveries = Deliveries() + let decoder = DualLayerVideoDecoder(composer: composer) + decoder.onFrame = { buffer, pts, hdr10 in deliveries.record(buffer, pts, hdr10) } + decoder.primeForTesting(enhancementLayer: true) + decoder.seedRPUForTesting(pts: Self.pts(0), params: DoviMetalParams(), displayMetadataNAL: nil) + decoder.seedRPUForTesting(pts: Self.pts(1), params: DoviMetalParams(), displayMetadataNAL: nil) + + // First pair composes and holds the only slot. + decoder.handleEnhancementLayerFrame(elBuffer: el0, pts: Self.pts(0)) + decoder.handleBaseLayerFrame(baseBuffer: bl0, pts: Self.pts(0), hdr10Data: nil) + + // Second pair on another thread: its enhancement layer queues without + // blocking, then its base layer matches and parks in acquireSlot(). + let bl1Box = SendablePixelBuffer(value: bl1) + let el1Box = SendablePixelBuffer(value: el1) + // A dedicated thread, not the global queue: other suites run concurrently and + // can saturate GCD's pool, which starved this closure past the deadline. + let workerDone = Flag() + let worker = Thread { + decoder.handleEnhancementLayerFrame(elBuffer: el1Box.value, pts: Self.pts(1)) + decoder.handleBaseLayerFrame(baseBuffer: bl1Box.value, pts: Self.pts(1), hdr10Data: nil) + workerDone.set() + } + worker.start() + // Wait until the background thread is inside the slot wait (on the + // pre-fix code it instead holds the decoder lock here, which is what + // made flush() hang). Then flush() must still return promptly. + try await waitFor { composer.acquireCallCount == 2 } + + // flush() runs while the second composition is parked in `acquireSlot` and the first is held + // by the gate. Returning at all is the proof (it would wait on the decoder lock otherwise, + // and the suite's time limit would report the hang); no wall-clock bound is asserted. + decoder.flush() + + // Release the held composition and let the second finish; the flush + // retired both tickets, so neither may be delivered after it. + composer.releaseFirst() + try await waitFor { workerDone.isSet } + decoder.quiesceForTesting() + #expect(deliveries.count == 0, "frames composed before flush() must not be delivered after it") + decoder.close() + } + + @Test("a frame with no usable RPU borrows the previous frame's parameters, but only for a short run") + func missingRpuBorrowsPreviousParameters() async throws { + let composer = GatedFakeComposer() + let decoder = DualLayerVideoDecoder(composer: composer) + decoder.onFrame = { _, _, _ in } + decoder.primeForTesting(enhancementLayer: true) + let limit = DualLayerVideoDecoder.maxBorrowedParamsRun + // Packets in decode order: frame 0 has a real RPU, every later one has none. + decoder.seedRPUForTesting(pts: Self.pts(0), params: DoviMetalParams(), displayMetadataNAL: nil) + for i in 1...(limit + 1) { + decoder.seedRPUForTesting(pts: Self.pts(Int64(i)), params: nil, displayMetadataNAL: nil) + } + let box = try (0...(limit + 1)).map { _ -> (SendablePixelBuffer, SendablePixelBuffer) in + let bl = try #require(Self.makeBuffer(width: 64, height: 64)) + let el = try #require(Self.makeBuffer(width: 32, height: 32)) + return (SendablePixelBuffer(value: bl), SendablePixelBuffer(value: el)) + } + let workerDone = Flag() + let worker = Thread { + for (i, pair) in box.enumerated() { + decoder.handleEnhancementLayerFrame(elBuffer: pair.1.value, pts: Self.pts(Int64(i))) + decoder.handleBaseLayerFrame(baseBuffer: pair.0.value, pts: Self.pts(Int64(i)), hdr10Data: nil) + } + workerDone.set() + } + worker.start() + // Frame 0 holds the only slot. Frame 1 borrows and parks in acquireSlot (count 2). Frames after + // the borrow limit must bypass instead of reaching the composer. + try await waitFor { composer.acquireCallCount == 2 } + composer.releaseFirst() + try await waitFor { workerDone.isSet } + #expect(composer.acquireCallCount == 1 + limit) + decoder.close() + } + + @Test("borrowed parameters carry display metadata only within the borrow limit, and flush clears it") + func missingRpuBorrowsPreviousDisplayMetadata() async throws { + let composer = GatedFakeComposer() + let deliveries = Deliveries() + let decoder = DualLayerVideoDecoder(composer: composer) + defer { decoder.close() } + decoder.onFrame = { buffer, pts, hdr10 in deliveries.record(buffer, pts, hdr10) } + decoder.primeForTesting(enhancementLayer: true) + let nal: [UInt8] = [0x7C, 0x01, 0x19, 0x42, 0x03] + let payload = Data(nal.dropFirst(2)) + let limit = DualLayerVideoDecoder.maxBorrowedParamsRun + decoder.seedRPUForTesting(pts: Self.pts(0), params: DoviMetalParams(), displayMetadataNAL: nal) + for i in 1...(limit + 1) { + decoder.seedRPUForTesting(pts: Self.pts(Int64(i)), params: nil, displayMetadataNAL: nil) + } + + for i in 0...(limit + 1) { + let bl = try #require(Self.makeBuffer(width: 64, height: 64)) + let el = try #require(Self.makeBuffer(width: 32, height: 32)) + let pts = Self.pts(Int64(i)) + if i.isMultiple(of: 2) { + decoder.handleEnhancementLayerFrame(elBuffer: el, pts: pts) + decoder.handleBaseLayerFrame(baseBuffer: bl, pts: pts, hdr10Data: nil) + } else { + decoder.handleBaseLayerFrame(baseBuffer: bl, pts: pts, hdr10Data: nil) + decoder.handleEnhancementLayerFrame(elBuffer: el, pts: pts) + } + if i == 0 { composer.releaseFirst() } + } + try await waitFor { deliveries.count >= limit + 2 } + #expect(deliveries.metadataPayloads == Array(repeating: Optional(payload), count: limit + 1) + [nil]) + #expect(composer.acquireCallCount == 1 + limit) + + decoder.seedRPUForTesting(pts: Self.pts(10), params: DoviMetalParams(), displayMetadataNAL: nal) + let bl = try #require(Self.makeBuffer(width: 64, height: 64)) + let el = try #require(Self.makeBuffer(width: 32, height: 32)) + decoder.handleBaseLayerFrame(baseBuffer: bl, pts: Self.pts(10), hdr10Data: nil) + decoder.handleEnhancementLayerFrame(elBuffer: el, pts: Self.pts(10)) + try await waitFor { deliveries.count >= limit + 3 } + #expect(deliveries.count == limit + 3) + decoder.flush() + for i in 11...12 { + decoder.seedRPUForTesting(pts: Self.pts(Int64(i)), params: nil, displayMetadataNAL: nil) + let bl = try #require(Self.makeBuffer(width: 64, height: 64)) + let el = try #require(Self.makeBuffer(width: 32, height: 32)) + decoder.handleBaseLayerFrame(baseBuffer: bl, pts: Self.pts(Int64(i)), hdr10Data: nil) + decoder.handleEnhancementLayerFrame(elBuffer: el, pts: Self.pts(Int64(i))) + } + try await waitFor { deliveries.count >= limit + 5 } + #expect(deliveries.metadataPayloads == Array(repeating: Optional(payload), count: limit + 1) + [nil, payload, nil, nil]) + #expect(composer.acquireCallCount == 2 + limit) + } + + @Test("ordering still holds across two gated compositions") + func gatedCompositionsDeliverInOrder() async throws { + let bl0 = try #require(Self.makeBuffer(width: 64, height: 64)) + let el0 = try #require(Self.makeBuffer(width: 32, height: 32)) + let bl1 = try #require(Self.makeBuffer(width: 64, height: 64)) + let el1 = try #require(Self.makeBuffer(width: 32, height: 32)) + + let composer = GatedFakeComposer() + let deliveries = Deliveries() + let decoder = DualLayerVideoDecoder(composer: composer) + defer { decoder.close() } + decoder.onFrame = { buffer, pts, hdr10 in deliveries.record(buffer, pts, hdr10) } + decoder.primeForTesting(enhancementLayer: true) + decoder.seedRPUForTesting(pts: Self.pts(0), params: DoviMetalParams(), displayMetadataNAL: nil) + decoder.seedRPUForTesting(pts: Self.pts(1), params: DoviMetalParams(), displayMetadataNAL: nil) + + decoder.handleEnhancementLayerFrame(elBuffer: el0, pts: Self.pts(0)) + decoder.handleBaseLayerFrame(baseBuffer: bl0, pts: Self.pts(0), hdr10Data: nil) + // Second composition blocks on the held slot on another thread. + let bl1Box = SendablePixelBuffer(value: bl1) + let el1Box = SendablePixelBuffer(value: el1) + // A dedicated thread, not the global queue: other suites run concurrently and + // can saturate GCD's pool, which starved this closure past the deadline. + let worker = Thread { + decoder.handleEnhancementLayerFrame(elBuffer: el1Box.value, pts: Self.pts(1)) + decoder.handleBaseLayerFrame(baseBuffer: bl1Box.value, pts: Self.pts(1), hdr10Data: nil) + } + worker.start() + try await waitFor { composer.acquireCallCount == 2 } + composer.releaseFirst() + + try await waitFor { deliveries.count >= 2 } + #expect(deliveries.count == 2) + let points = deliveries.points + if points.count == 2 { + #expect(CMTimeCompare(points[0], Self.pts(0)) == 0) + #expect(CMTimeCompare(points[1], Self.pts(1)) == 0) + } + } +} diff --git a/Tests/AetherEngineTests/DualLayerSeekRealignmentTests.swift b/Tests/AetherEngineTests/DualLayerSeekRealignmentTests.swift new file mode 100644 index 000000000..79f5ae60d --- /dev/null +++ b/Tests/AetherEngineTests/DualLayerSeekRealignmentTests.swift @@ -0,0 +1,307 @@ +import Testing +import CoreMedia +import CoreVideo +import Foundation +import AetherLibavformat +@_spi(Diagnostics) @testable import AetherEngine + +/// A seek on the Profile 7 FEL route has to flush and realign both layers on the same picture +/// (the maintainer's parity requirement). The AV1/VP9 software path has one decoder to flush; here +/// there are two VideoToolbox sessions, the composer, the ordered delivery, the two frame queues and +/// the side-data ring, and a seek lands on timestamps the pre-seek frames may already have used (a +/// backward seek replays the same PTS keys). These tests drive the frame handlers with tagged +/// synthetic buffers and check what the composer was asked to pair and what reached `onFrame`. +@Suite(.serialized, .timeLimit(.minutes(2))) +struct DualLayerSeekRealignmentTests { + + // MARK: - Fixtures + + private static let frameTicks: Int64 = 3754 + private static func pts(_ frame: Int64) -> CMTime { CMTime(value: frame * frameTicks, timescale: 90000) } + + private static func buffer(width: Int, height: Int) -> CVPixelBuffer? { + var out: CVPixelBuffer? + let status = CVPixelBufferCreate(kCFAllocatorDefault, width, height, + kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange, + [kCVPixelBufferIOSurfacePropertiesKey: [:] as CFDictionary] as CFDictionary, + &out) + return status == kCVReturnSuccess ? out : nil + } + + private static func params() -> DoviMetalParams { + var p = DoviMetalParams() + p.blWidth = 64; p.blHeight = 64; p.elWidth = 32; p.elHeight = 32 + return p + } + + private struct Pair { let base: CVPixelBuffer; let enhancement: CVPixelBuffer } + + /// Records every pairing the decoder asks for and counts flushes. Completes at once with the + /// base buffer, unless `holdFirst` is set: then the first composition's completion is parked + /// until `releaseHeld()`, the way a command buffer still on the GPU holds it. + private final class FakeComposer: FelComposing, @unchecked Sendable { + private let lock = NSLock() + private var _pairs: [Pair] = [] + private var _flushes = 0 + private var held: (@Sendable (CVPixelBuffer?) -> Void)? + private var heldBase: CVPixelBuffer? + private let holdFirst: Bool + + init(holdFirst: Bool = false) { self.holdFirst = holdFirst } + + var pairs: [Pair] { lock.lock(); defer { lock.unlock() }; return _pairs } + var flushCount: Int { lock.lock(); defer { lock.unlock() }; return _flushes } + + func acquireSlot() {} + func releaseSlot() {} + func flush() { lock.lock(); _flushes += 1; lock.unlock() } + func composeAsync(baseBuffer: CVPixelBuffer, enhancementBuffer: CVPixelBuffer, params: DoviMetalParams, + frameIndex: UInt32, completion: @escaping @Sendable (CVPixelBuffer?) -> Void) { + composeAsyncWithAcquiredSlot(baseBuffer: baseBuffer, enhancementBuffer: enhancementBuffer, + params: params, frameIndex: frameIndex, completion: completion) + } + func composeAsyncWithAcquiredSlot(baseBuffer: CVPixelBuffer, enhancementBuffer: CVPixelBuffer, + params: DoviMetalParams, frameIndex: UInt32, + completion: @escaping @Sendable (CVPixelBuffer?) -> Void) { + lock.lock() + let first = _pairs.isEmpty + _pairs.append(Pair(base: baseBuffer, enhancement: enhancementBuffer)) + if holdFirst, first { + held = completion + heldBase = baseBuffer + lock.unlock() + return + } + lock.unlock() + completion(baseBuffer) + } + + func releaseHeld() { + lock.lock() + let completion = held, base = heldBase + held = nil; heldBase = nil + lock.unlock() + completion?(base) + } + } + + /// A VideoToolbox session stand-in that only counts what the decoder does to it. + private final class FakeSession: SingleLayerVideoDecoder, @unchecked Sendable { + private let lock = NSLock() + private var _flushes = 0 + var flushCount: Int { lock.lock(); defer { lock.unlock() }; return _flushes } + var onFrame: DecodedFrameHandler? + var onFirstHDR10PlusDetected: (@Sendable () -> Void)? + var onA53Captions: (@Sendable ([CCDataParser.CCTriplet], Double) -> Void)? + var onDecodedFormat: (@Sendable (DecodedVideoFormat) -> Void)? + var skipUntilPTS: CMTime? + var feedEpoch: UInt64 = 0 + func open(stream: UnsafeMutablePointer, onFrame: @escaping DecodedFrameHandler) throws {} + func open(parameterSets: [[UInt8]], timeBase: AVRational, + onFrame: @escaping DecodedFrameHandler) throws {} + func decode(packet: UnsafeMutablePointer, epoch: UInt64?) {} + func flush() { lock.lock(); _flushes += 1; lock.unlock() } + func close() {} + } + + private final class Deliveries: @unchecked Sendable { + struct Frame { let buffer: CVPixelBuffer; let pts: CMTime; let afterMarker: Bool } + private let lock = NSLock() + private var frames: [Frame] = [] + private var marker = false + func record(_ buffer: CVPixelBuffer, _ pts: CMTime) { + lock.lock(); frames.append(Frame(buffer: buffer, pts: pts, afterMarker: marker)); lock.unlock() + } + func setMarker() { lock.lock(); marker = true; lock.unlock() } + var afterMarker: [Frame] { lock.lock(); defer { lock.unlock() }; return frames.filter(\.afterMarker) } + var count: Int { lock.lock(); defer { lock.unlock() }; return frames.count } + } + + private struct Rig { + let decoder: DualLayerVideoDecoder + let composer: FakeComposer + let primary: FakeSession + let enhancement: FakeSession + let deliveries: Deliveries + } + + private static func rig(holdFirst: Bool = false) -> Rig { + let composer = FakeComposer(holdFirst: holdFirst) + let primary = FakeSession(), enhancement = FakeSession() + let deliveries = Deliveries() + let decoder = DualLayerVideoDecoder(composer: composer, primaryDecoder: primary, + enhancementDecoder: enhancement) + decoder.onFrame = { buffer, pts, _ in deliveries.record(buffer, pts) } + decoder.primeForTesting(enhancementLayer: true) + return Rig(decoder: decoder, composer: composer, primary: primary, enhancement: enhancement, + deliveries: deliveries) + } + + // MARK: - Tests + + @Test("a seek flushes both VideoToolbox sessions and the composer, and empties the pairing state") + func flushReachesEveryStage() throws { + let r = Self.rig() + defer { r.decoder.close() } + + // Two complete pairs, a lone EL and a lone BL: every queue and the side-data ring hold something. + for i in 0..<4 { + r.decoder.seedRPUForTesting(pts: Self.pts(Int64(i)), params: Self.params(), displayMetadataNAL: nil) + } + for i in 0..<2 { + let bl = try #require(Self.buffer(width: 64, height: 64)) + let el = try #require(Self.buffer(width: 32, height: 32)) + r.decoder.handleEnhancementLayerFrame(elBuffer: el, pts: Self.pts(Int64(i))) + r.decoder.handleBaseLayerFrame(baseBuffer: bl, pts: Self.pts(Int64(i)), hdr10Data: nil) + } + r.decoder.handleEnhancementLayerFrame(elBuffer: try #require(Self.buffer(width: 32, height: 32)), pts: Self.pts(2)) + r.decoder.handleBaseLayerFrame(baseBuffer: try #require(Self.buffer(width: 64, height: 64)), + pts: Self.pts(3), hdr10Data: nil) + let before = r.decoder.stateSnapshotForTesting + #expect(before.rpuCacheCount > 0) + #expect(before.elQueueCount + before.blQueueCount > 0) + + let epoch = r.decoder.feedEpoch + r.decoder.flush() + + #expect(r.primary.flushCount == 1) + #expect(r.enhancement.flushCount == 1) + #expect(r.composer.flushCount == 1) + #expect(r.decoder.feedEpoch == epoch + 1) + let after = r.decoder.stateSnapshotForTesting + #expect(after.blQueueCount == 0 && after.elQueueCount == 0) + #expect(after.rpuCacheCount == 0 && after.dmRpuCacheCount == 0) + #expect(!after.draining) + } + + @Test("an enhancement layer from before a seek is never composed onto a post-seek base layer") + func staleEnhancementLayerIsNotPaired() async throws { + let r = Self.rig() + defer { r.decoder.close() } + + // The pre-seek EL for frame 10 is decoded, its base layer never is: the seek retires it. + let staleEL = try #require(Self.buffer(width: 32, height: 32)) + r.decoder.handleEnhancementLayerFrame(elBuffer: staleEL, pts: Self.pts(10)) + r.decoder.flush() + r.deliveries.setMarker() + + // A backward seek replays the same timestamps. The base layer comes first this time. + let postBL = try #require(Self.buffer(width: 64, height: 64)) + let postEL = try #require(Self.buffer(width: 32, height: 32)) + r.decoder.seedRPUForTesting(pts: Self.pts(10), params: Self.params(), displayMetadataNAL: nil) + r.decoder.handleBaseLayerFrame(baseBuffer: postBL, pts: Self.pts(10), hdr10Data: nil) + r.decoder.quiesceForTesting() + #expect(r.composer.pairs.isEmpty, "the post-seek base layer was composed with a pre-seek enhancement layer") + + r.decoder.handleEnhancementLayerFrame(elBuffer: postEL, pts: Self.pts(10)) + try await waitFor { r.deliveries.afterMarker.count == 1 } + let pairs = r.composer.pairs + #expect(pairs.count == 1) + #expect(pairs.first?.base === postBL) + #expect(pairs.first?.enhancement === postEL) + #expect(pairs.allSatisfy { $0.enhancement !== staleEL }) + } + + @Test("a base layer waiting for its enhancement layer when the seek lands is dropped, not delivered late") + func staleBaseLayerIsNotDelivered() async throws { + let r = Self.rig() + defer { r.decoder.close() } + + let staleBL = try #require(Self.buffer(width: 64, height: 64)) + r.decoder.seedRPUForTesting(pts: Self.pts(5), params: Self.params(), displayMetadataNAL: nil) + r.decoder.handleBaseLayerFrame(baseBuffer: staleBL, pts: Self.pts(5), hdr10Data: nil) + r.decoder.flush() + r.deliveries.setMarker() + + // The post-seek EL for the same key must not find the pre-seek base layer. + let postEL = try #require(Self.buffer(width: 32, height: 32)) + r.decoder.handleEnhancementLayerFrame(elBuffer: postEL, pts: Self.pts(5)) + r.decoder.quiesceForTesting() + #expect(r.composer.pairs.isEmpty) + #expect(r.deliveries.afterMarker.isEmpty) + + let postBL = try #require(Self.buffer(width: 64, height: 64)) + r.decoder.seedRPUForTesting(pts: Self.pts(5), params: Self.params(), displayMetadataNAL: nil) + r.decoder.handleBaseLayerFrame(baseBuffer: postBL, pts: Self.pts(5), hdr10Data: nil) + try await waitFor { r.deliveries.afterMarker.count == 1 } + #expect(r.deliveries.afterMarker.first?.buffer === postBL) + #expect(r.composer.pairs.first?.base === postBL) + #expect(r.composer.pairs.first?.enhancement === postEL) + } + + @Test("after a seek every frame is composed with its own layer's picture, in either arrival order") + func postSeekFramesPairByPicture() async throws { + let r = Self.rig() + defer { r.decoder.close() } + + // Pre-seek leftovers on keys the replay is about to reuse. + for i in 10..<13 { + r.decoder.handleEnhancementLayerFrame(elBuffer: try #require(Self.buffer(width: 32, height: 32)), pts: Self.pts(Int64(i))) + } + r.decoder.handleBaseLayerFrame(baseBuffer: try #require(Self.buffer(width: 64, height: 64)), pts: Self.pts(13), hdr10Data: nil) + r.decoder.flush() + r.deliveries.setMarker() + + var bases: [CVPixelBuffer] = [], enhancements: [CVPixelBuffer] = [] + for i in 0..<6 { + let bl = try #require(Self.buffer(width: 64, height: 64)) + let el = try #require(Self.buffer(width: 32, height: 32)) + bases.append(bl); enhancements.append(el) + let key = Int64(10 + i) + r.decoder.seedRPUForTesting(pts: Self.pts(key), params: Self.params(), displayMetadataNAL: nil) + if i % 2 == 0 { + r.decoder.handleBaseLayerFrame(baseBuffer: bl, pts: Self.pts(key), hdr10Data: nil) + r.decoder.handleEnhancementLayerFrame(elBuffer: el, pts: Self.pts(key)) + } else { + r.decoder.handleEnhancementLayerFrame(elBuffer: el, pts: Self.pts(key)) + r.decoder.handleBaseLayerFrame(baseBuffer: bl, pts: Self.pts(key), hdr10Data: nil) + } + } + try await waitFor { r.deliveries.afterMarker.count == 6 } + let delivered = r.deliveries.afterMarker + #expect(zip(delivered, delivered.dropFirst()).allSatisfy { CMTimeCompare($0.pts, $1.pts) < 0 }) + #expect(zip(delivered, bases).allSatisfy { $0.buffer === $1 }) + let pairs = r.composer.pairs + #expect(pairs.count == 6) + for pair in pairs { + let i = bases.firstIndex { $0 === pair.base } + #expect(i != nil && pair.enhancement === enhancements[i!], "base and enhancement are not the same picture") + } + } + + @Test("a seek during an in-flight composition drops that frame, and the first post-seek frame is the paired one") + func seekDuringInFlightCompose() async throws { + let r = Self.rig(holdFirst: true) + defer { r.decoder.close() } + + let staleBL = try #require(Self.buffer(width: 64, height: 64)) + let staleEL = try #require(Self.buffer(width: 32, height: 32)) + r.decoder.seedRPUForTesting(pts: Self.pts(1), params: Self.params(), displayMetadataNAL: nil) + r.decoder.handleEnhancementLayerFrame(elBuffer: staleEL, pts: Self.pts(1)) + r.decoder.handleBaseLayerFrame(baseBuffer: staleBL, pts: Self.pts(1), hdr10Data: nil) + #expect(r.composer.pairs.count == 1, "the first composition must be on the GPU") + + // The seek lands while it is there. + r.decoder.flush() + r.deliveries.setMarker() + + let postBL = try #require(Self.buffer(width: 64, height: 64)) + let postEL = try #require(Self.buffer(width: 32, height: 32)) + r.decoder.seedRPUForTesting(pts: Self.pts(1), params: Self.params(), displayMetadataNAL: nil) + r.decoder.handleEnhancementLayerFrame(elBuffer: postEL, pts: Self.pts(1)) + r.decoder.handleBaseLayerFrame(baseBuffer: postBL, pts: Self.pts(1), hdr10Data: nil) + try await waitFor { r.deliveries.afterMarker.count == 1 } + + // The pre-seek composition retires last, after the post-seek frame is already out. + r.composer.releaseHeld() + r.decoder.quiesceForTesting() + + let late = r.deliveries.afterMarker + #expect(late.count == 1) + #expect(late.first?.buffer === postBL) + #expect(late.allSatisfy { $0.buffer !== staleBL }) + let pairs = r.composer.pairs + #expect(pairs.count == 2) + #expect(pairs.last?.base === postBL && pairs.last?.enhancement === postEL) + } +} diff --git a/Tests/AetherEngineTests/DualLayerStreamClassificationTests.swift b/Tests/AetherEngineTests/DualLayerStreamClassificationTests.swift new file mode 100644 index 000000000..5fcadd62c --- /dev/null +++ b/Tests/AetherEngineTests/DualLayerStreamClassificationTests.swift @@ -0,0 +1,240 @@ +import Testing +import CoreMedia +import CoreVideo +import Foundation +import Metal +import AetherLibavcodec +@testable import AetherEngine + +/// Stream classification is decided once, from the first RPU's `el_type`, before +/// the enhancement decoder opens. A per-frame `disable_residual_flag` is a kernel flag +/// (`DoviComposerFlags.disableResidual`, honoured by the composer) and must never shut +/// off the enhancement layer for the rest of the session. +@Suite(.serialized, .timeLimit(.minutes(1))) +struct DualLayerStreamClassificationTests { + + // MARK: - Fixtures + + private static let frameTicks: Int64 = 3754 + + private static func pts(_ frame: Int64) -> CMTime { + CMTime(value: frame * frameTicks, timescale: 90000) + } + + private static var metalAvailable: Bool { MTLCreateSystemDefaultDevice() != nil } + + private static func hexToBytes(_ hex: String) -> [UInt8] { + var bytes: [UInt8] = [] + var i = hex.startIndex + while i < hex.endIndex { + let next = hex.index(i, offsetBy: 2) + if let byte = UInt8(hex[i.. [UInt8] { + [UInt8(type << 1), 0x01] + payload + } + + /// Pack complete NALs into an AVCC (4-byte BE length prefix) AVPacket. + private func avccPacket(_ nals: [[UInt8]]) -> UnsafeMutablePointer { + var bytes: [UInt8] = [] + for nal in nals { + let n = nal.count + bytes.append(UInt8((n >> 24) & 0xFF)) + bytes.append(UInt8((n >> 16) & 0xFF)) + bytes.append(UInt8((n >> 8) & 0xFF)) + bytes.append(UInt8(n & 0xFF)) + bytes.append(contentsOf: nal) + } + let pkt = av_packet_alloc()! + _ = av_new_packet(pkt, Int32(bytes.count)) + bytes.withUnsafeBytes { src in + _ = memcpy(pkt.pointee.data, src.baseAddress, bytes.count) + } + return pkt + } + + private func free(_ pkt: UnsafeMutablePointer) { + var p: UnsafeMutablePointer? = pkt + av_packet_free(&p) + } + + private func packetResult( + isFEL: Bool, + isMEL: Bool = false, + params: DoviMetalParams? = nil + ) -> DoviRpuConverter.DualLayerPacketResult { + DoviRpuConverter.DualLayerPacketResult( + elPacket: nil, rpuData: nil, rpuParams: params, + isFEL: isFEL, isMEL: isMEL, success: true + ) + } + + private static func makeBuffer(width: Int, height: Int) -> CVPixelBuffer? { + var out: CVPixelBuffer? + let status = CVPixelBufferCreate(kCFAllocatorDefault, width, height, + kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange, + [kCVPixelBufferMetalCompatibilityKey: true, + kCVPixelBufferIOSurfacePropertiesKey: [:] as CFDictionary] as CFDictionary, + &out) + guard status == kCVReturnSuccess, let buf = out else { return nil } + return buf + } + + private static func baseLayer() -> CVPixelBuffer? { makeBuffer(width: 256, height: 256) } + private static func enhancementLayer() -> CVPixelBuffer? { makeBuffer(width: 128, height: 128) } + + private static func params() -> DoviMetalParams { + var p = DoviMetalParams() + p.blWidth = 256; p.blHeight = 256; p.elWidth = 128; p.elHeight = 128 + return p + } + + private final class Deliveries: @unchecked Sendable { + private let lock = NSLock() + private var buffers: [(CVPixelBuffer, CMTime)] = [] + + func record(_ buffer: CVPixelBuffer, _ pts: CMTime, _ hdr10: Data?) { + lock.lock() + buffers.append((buffer, pts)) + lock.unlock() + } + + var snapshot: [(CVPixelBuffer, CMTime)] { + lock.lock(); defer { lock.unlock() }; return buffers + } + var count: Int { lock.lock(); defer { lock.unlock() }; return buffers.count } + } + + // MARK: - classifyStream unit tests + + @Test("classifyStream maps a FEL packet result to FEL") + func classifyFEL() { + #expect(DoviRpuConverter.classifyStream(packetResult(isFEL: true)) == .fel) + } + + @Test("classifyStream maps a MEL packet result to MEL") + func classifyMEL() { + #expect(DoviRpuConverter.classifyStream(packetResult(isFEL: false, isMEL: true)) == .mel) + } + + @Test("classifyStream maps a packet with no RPU signal to undetermined") + func classifyUndetermined() { + #expect(DoviRpuConverter.classifyStream(packetResult(isFEL: false)) == .undetermined) + } + + @Test("classifyStream ignores the per-frame disable_residual kernel flag") + func classifyIgnoresDisableResidualFlag() { + var p = Self.params() + p.flags.insert(.disableResidual) + // This is what `separateDualLayers` now produces for a FEL RPU with + // `disable_residual_flag` set: FEL reporting plus the kernel flag. + let result = packetResult(isFEL: true, params: p) + #expect(DoviRpuConverter.classifyStream(result) == .fel) + } + + // MARK: - Converter integration over a real RPU + + @Test("a real FEL RPU classifies FEL, not MEL") + func realFELRPUClassifiesFEL() { + // Raw unspec62 NAL (header included) from a FEL title; shared with DoviFelMathTests. + let rpu = Self.hexToBytes(DoviFelMathTests.clipARpuHex) + let pkt = avccPacket([hevcNAL(type: 1, payload: [0xAA, 0xBB]), rpu]) + defer { free(pkt) } + + let result = DoviRpuConverter.separateDualLayers(pkt) + #expect(result.isFEL) + #expect(!result.isMEL) + #expect(DoviRpuConverter.classifyStream(result) == .fel) + #expect(result.rpuParams != nil) + } + + // MARK: - Decoder one-shot latch + + @Test("a FEL stream stays FEL across a disable_residual RPU and later reports") + func felStaysFELAcrossDisableResidualFrame() { + let decoder = DualLayerVideoDecoder() + defer { decoder.close() } + #expect(decoder.streamClassificationForTesting == .undetermined) + + var residualParams = Self.params() + residualParams.flags.insert(.disableResidual) + + decoder.classifyPacketResultForTesting(packetResult(isFEL: true)) + #expect(decoder.streamClassificationForTesting == .fel) + #expect(decoder.engagesEnhancementLayerForTesting()) + + // One RPU with the residual switched off: kernel flag only, the EL keeps decoding. + decoder.classifyPacketResultForTesting(packetResult(isFEL: true, params: residualParams)) + #expect(decoder.streamClassificationForTesting == .fel) + #expect(decoder.engagesEnhancementLayerForTesting()) + + // A later MEL report cannot flip the one-shot latch either. + decoder.classifyPacketResultForTesting(packetResult(isFEL: false, isMEL: true)) + #expect(decoder.streamClassificationForTesting == .fel) + #expect(decoder.engagesEnhancementLayerForTesting()) + } + + @Test("the first RPU decides: MEL never engages the enhancement layer") + func melNeverEngagesEnhancementLayer() { + let decoder = DualLayerVideoDecoder() + defer { decoder.close() } + + decoder.classifyPacketResultForTesting(packetResult(isFEL: false, isMEL: true)) + #expect(decoder.streamClassificationForTesting == .mel) + #expect(!decoder.engagesEnhancementLayerForTesting()) + + // A later FEL report cannot reopen what MEL never opened. + decoder.classifyPacketResultForTesting(packetResult(isFEL: true)) + #expect(decoder.streamClassificationForTesting == .mel) + #expect(!decoder.engagesEnhancementLayerForTesting()) + } + + @Test("a MEL-primed decoder delivers the base layer directly") + func melBypassesToBaseLayer() async throws { + let deliveries = Deliveries() + let decoder = DualLayerVideoDecoder() + decoder.onFrame = { buffer, pts, hdr10 in deliveries.record(buffer, pts, hdr10) } + decoder.primeStreamForTesting(.mel) + defer { decoder.close() } + + let bl = try #require(Self.baseLayer()) + decoder.handleBaseLayerFrame(baseBuffer: bl, pts: Self.pts(0), hdr10Data: nil) + + // A bypass is a `fulfil` onto `OrderedDelivery`'s queue + // (`queue.async`), not a synchronous call to `onFrame` on the calling + // thread, so `handleBaseLayerFrame` returning is no longer proof the + // frame has been delivered. + try await waitFor { deliveries.count == 1 } + let got = deliveries.snapshot + #expect(got.count == 1) + #expect(got.first?.0 === bl) + } + + @Test("a frame whose RPU disables the residual still composes with its enhancement layer") + func disableResidualFrameStillComposes() async throws { + guard Self.metalAvailable else { return } + let bl = try #require(Self.baseLayer()) + let el = try #require(Self.enhancementLayer()) + let deliveries = Deliveries() + let decoder = DualLayerVideoDecoder() + decoder.onFrame = { buffer, pts, hdr10 in deliveries.record(buffer, pts, hdr10) } + decoder.primeStreamForTesting(.fel) + defer { decoder.close() } + + var p = Self.params() + p.flags.insert(.disableResidual) + decoder.seedRPUForTesting(pts: Self.pts(1), params: p, displayMetadataNAL: nil) + + decoder.handleBaseLayerFrame(baseBuffer: bl, pts: Self.pts(1), hdr10Data: nil) + decoder.handleEnhancementLayerFrame(elBuffer: el, pts: Self.pts(1)) + + try await waitFor { deliveries.count == 1 } + let got = try #require(deliveries.snapshot.first) + #expect(got.0 !== bl, "a disable_residual RPU must not bypass composition") + } +} diff --git a/Tests/AetherEngineTests/DualLayerVideoDecoderTests.swift b/Tests/AetherEngineTests/DualLayerVideoDecoderTests.swift new file mode 100644 index 000000000..8cb32b33e --- /dev/null +++ b/Tests/AetherEngineTests/DualLayerVideoDecoderTests.swift @@ -0,0 +1,2535 @@ +import Testing +import CoreMedia +import CoreVideo +import Foundation +import Metal +import AetherLibavformat +import AetherLibavcodec +import AetherLibavutil +import Dovi +@_spi(Diagnostics) @testable import AetherEngine + +/// `DualLayerVideoDecoder` pairs a 4K base-layer frame with its 1080p enhancement-layer frame and +/// the RPU cached for that PTS, composes them on the GPU, and delivers the result in decode order. +/// Both layers arrive on VideoToolbox callback threads, in either order, and compositions retire +/// asynchronously. These tests drive the two frame handlers directly with synthetic P010 buffers, +/// using `primeForTesting` / `seedRPUForTesting` to stand in for what `decode(packet:)` sets up, so +/// the pairing, ordering, metadata and flush behaviour is covered without a real stream. +@Suite(.serialized, .timeLimit(.minutes(2))) +struct DualLayerVideoDecoderTests { + + // MARK: - Fixtures + + /// 23.976 fps at 90 kHz, rounded; only monotonic, well-separated keys matter here. + private static let frameTicks: Int64 = 3754 + + private static func pts(_ frame: Int64, offsetTicks: Int64 = 0) -> CMTime { + CMTime(value: frame * frameTicks + offsetTicks, timescale: 90000) + } + + private static var metalAvailable: Bool { MTLCreateSystemDefaultDevice() != nil } + + private static func makeBuffer(width: Int, height: Int, fill: UInt16) -> CVPixelBuffer? { + var out: CVPixelBuffer? + let status = CVPixelBufferCreate(kCFAllocatorDefault, width, height, + kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange, + [kCVPixelBufferMetalCompatibilityKey: true, + kCVPixelBufferIOSurfacePropertiesKey: [:] as CFDictionary] as CFDictionary, + &out) + guard status == kCVReturnSuccess, let buf = out else { return nil } + CVPixelBufferLockBaseAddress(buf, []) + defer { CVPixelBufferUnlockBaseAddress(buf, []) } + for plane in 0.. CVPixelBuffer? { makeBuffer(width: 256, height: 256, fill: 0x8000) } + private static func enhancementLayer() -> CVPixelBuffer? { makeBuffer(width: 128, height: 128, fill: 0x8000) } + + /// Identity reshaping at the test plane sizes; the plumbing is under test, not the curve. + private static func params() -> DoviMetalParams { + var p = DoviMetalParams() + p.blWidth = 256; p.blHeight = 256; p.elWidth = 128; p.elHeight = 128 + return p + } + + /// A Profile 8.1 `unspec62` NAL shaped like `dovi_write_unspec62_nalu` output: header + /// `0x7C 0x01`, then a payload starting with the RPU prefix `0x19`, tagged so frames differ. + private static func rpuNAL(tag: UInt8) -> [UInt8] { [0x7C, 0x01, 0x19, 0x08, tag, 0xA5, 0x80] } + + /// Records what reached `onFrame`, reading the Dolby Vision attachments at delivery time, + /// which is when the display layer would read them. + private final class Deliveries: @unchecked Sendable { + struct Frame { + let buffer: CVPixelBuffer + let pts: CMTime + let hdr10: Data? + let rpuPayload: Data? + let compatibilityID: CFTypeRef? + let afterMarker: Bool + } + + private let lock = NSLock() + private var frames: [Frame] = [] + private var marker = false + + func record(_ buffer: CVPixelBuffer, _ pts: CMTime, _ hdr10: Data?) { + let payload = CVBufferCopyAttachment(buffer, DoviDisplayMetadata.rpuDataKey as CFString, nil) as? Data + let compat = CVBufferCopyAttachment(buffer, DoviDisplayMetadata.compatibilityIDKey as CFString, nil) + lock.lock() + frames.append(Frame(buffer: buffer, pts: pts, hdr10: hdr10, rpuPayload: payload, + compatibilityID: compat, afterMarker: marker)) + lock.unlock() + } + + /// Marks the point a test's flush/close returned; frames delivered later carry the mark. + func setMarker() { lock.lock(); marker = true; lock.unlock() } + + var snapshot: [Frame] { lock.lock(); defer { lock.unlock() }; return frames } + var count: Int { lock.lock(); defer { lock.unlock() }; return frames.count } + } + + private static func decoder(recordingInto deliveries: Deliveries, enhancementLayer: Bool) -> DualLayerVideoDecoder { + let decoder = DualLayerVideoDecoder() + decoder.onFrame = { buffer, pts, hdr10 in deliveries.record(buffer, pts, hdr10) } + decoder.primeForTesting(enhancementLayer: enhancementLayer) + return decoder + } + + private static func isStrictlyIncreasing(_ frames: [Deliveries.Frame]) -> Bool { + zip(frames, frames.dropFirst()).allSatisfy { CMTimeCompare($0.pts, $1.pts) < 0 } + } + + // MARK: - Single layer + + @Test("without an enhancement layer every base-layer frame is delivered untouched, in order") + func singleLayerBypassesInOrder() async throws { + let deliveries = Deliveries() + let decoder = Self.decoder(recordingInto: deliveries, enhancementLayer: false) + defer { decoder.close() } + var sent: [CVPixelBuffer] = [] + for i in 0..<5 { + let bl = try #require(Self.baseLayer()) + sent.append(bl) + decoder.handleBaseLayerFrame(baseBuffer: bl, pts: Self.pts(Int64(i)), hdr10Data: nil) + } + + // A bypass is now a `fulfil` onto `OrderedDelivery`'s queue + // (`queue.async`) rather than a synchronous call to `onFrame` on + // whichever thread paired the frame, so `handleBaseLayerFrame` + // returning is no longer a guarantee the frame has been delivered. + try await waitFor { deliveries.count == 5 } + let got = deliveries.snapshot + #expect(got.count == 5) + #expect(zip(got, sent).allSatisfy { $0.buffer === $1 }) + #expect(Self.isStrictlyIncreasing(got)) + } + + // MARK: - Pairing + + @Test("base layer first, then enhancement layer: the pair is composed and delivered at the base layer's PTS", .enabled(if: Self.metalAvailable, "needs a Metal device")) + func pairsWhenBaseLayerArrivesFirst() async throws { + let bl = try #require(Self.baseLayer()) + let el = try #require(Self.enhancementLayer()) + let deliveries = Deliveries() + let decoder = Self.decoder(recordingInto: deliveries, enhancementLayer: true) + defer { decoder.close() } + decoder.seedRPUForTesting(pts: Self.pts(1), params: Self.params(), displayMetadataNAL: nil) + + decoder.handleBaseLayerFrame(baseBuffer: bl, pts: Self.pts(1), hdr10Data: nil) + // Not `deliveries.count == 0`: delivery is one more `queue.async` hop + // away, so that would hold whether or not the frame is still waiting. + // `blQueueCount` is synchronous and deterministic: + // `handleBaseLayerFrame` decides queued-vs-ticketed entirely under + // `lock`, so by the time it returns, the frame is provably still + // sitting in `blFrameQueue` (not bypassed, not composed) if and only + // if it is still waiting for its enhancement layer. + #expect(decoder.stateSnapshotForTesting.blQueueCount == 1, "a base-layer frame with a pending enhancement layer must wait for it") + decoder.handleEnhancementLayerFrame(elBuffer: el, pts: Self.pts(1)) + + try await waitFor { deliveries.count == 1 } + let got = try #require(deliveries.snapshot.first) + #expect(got.buffer !== bl, "expected the composer's output, got the bypassed base layer") + #expect(CVPixelBufferGetWidth(got.buffer) == 256 && CVPixelBufferGetHeight(got.buffer) == 256) + #expect(CMTimeCompare(got.pts, Self.pts(1)) == 0) + } + + @Test("enhancement layer first, then base layer: the pair is composed and delivered at the base layer's PTS", .enabled(if: Self.metalAvailable, "needs a Metal device")) + func pairsWhenEnhancementLayerArrivesFirst() async throws { + let bl = try #require(Self.baseLayer()) + let el = try #require(Self.enhancementLayer()) + let deliveries = Deliveries() + let decoder = Self.decoder(recordingInto: deliveries, enhancementLayer: true) + defer { decoder.close() } + decoder.seedRPUForTesting(pts: Self.pts(1), params: Self.params(), displayMetadataNAL: nil) + + decoder.handleEnhancementLayerFrame(elBuffer: el, pts: Self.pts(1)) + decoder.handleBaseLayerFrame(baseBuffer: bl, pts: Self.pts(1), hdr10Data: nil) + + try await waitFor { deliveries.count == 1 } + let got = try #require(deliveries.snapshot.first) + #expect(got.buffer !== bl) + #expect(CMTimeCompare(got.pts, Self.pts(1)) == 0) + } + + @Test("six enhancement-layer frames arriving ahead of the base layer all compose", .enabled(if: Self.metalAvailable, "needs a Metal device")) + func pairsWhenEnhancementLayerRunsSixFramesAhead() async throws { + let deliveries = Deliveries() + let decoder = Self.decoder(recordingInto: deliveries, enhancementLayer: true) + defer { decoder.close() } + + for i in 0..<6 { + let el = try #require(Self.enhancementLayer()) + decoder.seedRPUForTesting(pts: Self.pts(Int64(i)), params: Self.params(), displayMetadataNAL: nil) + decoder.handleEnhancementLayerFrame(elBuffer: el, pts: Self.pts(Int64(i))) + } + + var sent: [CVPixelBuffer] = [] + for i in 0..<6 { + let bl = try #require(Self.baseLayer()) + sent.append(bl) + decoder.handleBaseLayerFrame(baseBuffer: bl, pts: Self.pts(Int64(i)), hdr10Data: nil) + } + + try await waitFor { deliveries.count == 6 } + let got = deliveries.snapshot + #expect(got.count == 6) + #expect(zip(got, sent).allSatisfy { $0.buffer !== $1 }, "every frame must use the composed path, not bypass for a missing EL") + #expect(got.enumerated().allSatisfy { CMTimeCompare($0.element.pts, Self.pts(Int64($0.offset))) == 0 }) + } + + @Test("layers up to 300 ticks apart still pair; further apart they do not", .enabled(if: Self.metalAvailable, "needs a Metal device")) + func jitterTolerance() async throws { + let deliveries = Deliveries() + let decoder = Self.decoder(recordingInto: deliveries, enhancementLayer: true) + defer { decoder.close() } + + let bl1 = try #require(Self.baseLayer()) + let el1 = try #require(Self.enhancementLayer()) + decoder.seedRPUForTesting(pts: Self.pts(1), params: Self.params(), displayMetadataNAL: nil) + decoder.handleEnhancementLayerFrame(elBuffer: el1, pts: Self.pts(1, offsetTicks: 300)) + decoder.handleBaseLayerFrame(baseBuffer: bl1, pts: Self.pts(1), hdr10Data: nil) + try await waitFor { deliveries.count == 1 } + #expect(deliveries.count == 1) + #expect(deliveries.snapshot.first?.buffer !== bl1) + + let bl2 = try #require(Self.baseLayer()) + let el2 = try #require(Self.enhancementLayer()) + decoder.seedRPUForTesting(pts: Self.pts(2), params: Self.params(), displayMetadataNAL: nil) + decoder.handleEnhancementLayerFrame(elBuffer: el2, pts: Self.pts(2, offsetTicks: 301)) + decoder.handleBaseLayerFrame(baseBuffer: bl2, pts: Self.pts(2), hdr10Data: nil) + decoder.quiesceForTesting() + #expect(deliveries.count == 1, "301 ticks apart must not pair") + } + + @Test("a pair with no cached RPU is not composed with invented coefficients: the base layer goes out", .enabled(if: Self.metalAvailable, "needs a Metal device")) + func missingRPUBypassesToBaseLayer() async throws { + let bl = try #require(Self.baseLayer()) + let el = try #require(Self.enhancementLayer()) + let deliveries = Deliveries() + let decoder = Self.decoder(recordingInto: deliveries, enhancementLayer: true) + defer { decoder.close() } + + decoder.handleEnhancementLayerFrame(elBuffer: el, pts: Self.pts(1)) + decoder.handleBaseLayerFrame(baseBuffer: bl, pts: Self.pts(1), hdr10Data: nil) + + try await waitFor { deliveries.count == 1 } + #expect(deliveries.snapshot.first?.buffer === bl) + } + + // MARK: - Ordering + + @Test("a composition still on the GPU is not overtaken by a later frame that bypasses synchronously", .enabled(if: Self.metalAvailable, "needs a Metal device")) + func asyncCompositionKeepsItsPlace() async throws { + let deliveries = Deliveries() + let decoder = Self.decoder(recordingInto: deliveries, enhancementLayer: true) + defer { decoder.close() } + + for round in 0..<8 { + let composed = Int64(round * 2), bypassed = Int64(round * 2 + 1) + let bl0 = try #require(Self.baseLayer()) + let el0 = try #require(Self.enhancementLayer()) + let bl1 = try #require(Self.baseLayer()) + let el1 = try #require(Self.enhancementLayer()) + decoder.seedRPUForTesting(pts: Self.pts(composed), params: Self.params(), displayMetadataNAL: nil) + decoder.handleEnhancementLayerFrame(elBuffer: el0, pts: Self.pts(composed)) + decoder.handleBaseLayerFrame(baseBuffer: bl0, pts: Self.pts(composed), hdr10Data: nil) + // No RPU for this one: it bypasses on the calling thread, ahead of the GPU. + decoder.handleEnhancementLayerFrame(elBuffer: el1, pts: Self.pts(bypassed)) + decoder.handleBaseLayerFrame(baseBuffer: bl1, pts: Self.pts(bypassed), hdr10Data: nil) + } + + try await waitFor { deliveries.count == 16 } + let got = deliveries.snapshot + #expect(got.count == 16) + #expect(Self.isStrictlyIncreasing(got)) + } + + @Test("a base layer whose enhancement layer never arrives is still delivered, in order", .enabled(if: Self.metalAvailable, "needs a Metal device")) + func orphanedBaseLayerIsEventuallyDelivered() async throws { + let deliveries = Deliveries() + let decoder = Self.decoder(recordingInto: deliveries, enhancementLayer: true) + defer { decoder.close() } + + let frames = 40 + for i in 0..= frames - 16 } + decoder.quiesceForTesting() + let got = deliveries.snapshot + // Whatever the queue bound is, a lost enhancement layer must not swallow the picture: + // all but a bounded tail of frames go out, as base layer, oldest first. + #expect(got.count >= frames - 16) + #expect(Self.isStrictlyIncreasing(got)) + } + + @Test("enhancement-layer frames with no base layer are bounded: the oldest can no longer pair", .enabled(if: Self.metalAvailable, "needs a Metal device")) + func orphanedEnhancementLayersAreBounded() async throws { + let deliveries = Deliveries() + let decoder = Self.decoder(recordingInto: deliveries, enhancementLayer: true) + defer { decoder.close() } + + for i in 0..<40 { + let el = try #require(Self.enhancementLayer()) + decoder.handleEnhancementLayerFrame(elBuffer: el, pts: Self.pts(Int64(i))) + } + let oldest = try #require(Self.baseLayer()) + let newest = try #require(Self.baseLayer()) + decoder.seedRPUForTesting(pts: Self.pts(0), params: Self.params(), displayMetadataNAL: nil) + decoder.seedRPUForTesting(pts: Self.pts(39), params: Self.params(), displayMetadataNAL: nil) + + decoder.handleBaseLayerFrame(baseBuffer: oldest, pts: Self.pts(0), hdr10Data: nil) + decoder.handleBaseLayerFrame(baseBuffer: newest, pts: Self.pts(39), hdr10Data: nil) + // Frame 0's enhancement layer is gone, so it waits for `maxLayerSkewFrames` further base-layer + // arrivals and then goes out alone; frame 39 is delivered behind it, in order. Without + // these arrivals nothing is delivered at all, which an earlier version of this test read as a + // pass: its bounded wait ran out and `first?.buffer !== newest` held for `nil`. + for i in 0..= 2 } + let got = deliveries.snapshot + #expect(CMTimeCompare(got[0].pts, Self.pts(0)) == 0) + #expect(got[0].buffer === oldest, "the enhancement layer for frame 0 should have been evicted: it goes out as its base layer") + #expect(CMTimeCompare(got[1].pts, Self.pts(39)) == 0) + #expect(got[1].buffer !== newest, "the newest enhancement layer should still pair") + } + + // MARK: - Dolby Vision display metadata + + @Test("every delivered frame carries its own RPU payload; a frame without one borrows the previous frame's", .enabled(if: Self.metalAvailable, "needs a Metal device")) + func displayMetadataTravelsWithItsFrame() async throws { + let deliveries = Deliveries() + let decoder = Self.decoder(recordingInto: deliveries, enhancementLayer: true) + defer { decoder.close() } + + for i in 0..<4 { + let bl = try #require(Self.baseLayer()) + let el = try #require(Self.enhancementLayer()) + let nal = Self.rpuNAL(tag: UInt8(i)) + decoder.seedRPUForTesting(pts: Self.pts(Int64(i)), params: Self.params(), displayMetadataNAL: nal) + decoder.handleEnhancementLayerFrame(elBuffer: el, pts: Self.pts(Int64(i))) + decoder.handleBaseLayerFrame(baseBuffer: bl, pts: Self.pts(Int64(i)), hdr10Data: nil) + } + // Frame 4 has no RPU at all: it borrows frame 3's parameters and payload (E1), and the stale + // attachment its recycled buffer arrived with must be replaced, not kept. + decoder.seedRPUForTesting(pts: Self.pts(4), params: nil, displayMetadataNAL: nil) + let recycled = try #require(Self.baseLayer()) + DoviDisplayMetadata.attach(rpuNAL: Self.rpuNAL(tag: 0xEE), to: recycled) + decoder.handleBaseLayerFrame(baseBuffer: recycled, pts: Self.pts(4), hdr10Data: nil) + let el4 = try #require(Self.enhancementLayer()) + decoder.handleEnhancementLayerFrame(elBuffer: el4, pts: Self.pts(4)) + + try await waitFor { deliveries.count == 5 } + let got = deliveries.snapshot + #expect(got.count == 5) + for i in 0..<4 where i < got.count { + #expect(got[i].rpuPayload == Data(Self.rpuNAL(tag: UInt8(i)).dropFirst(2)), "frame \(i) carries the wrong RPU") + #expect((got[i].compatibilityID as? String) == "1", "DolbyCompatibilityID must be the CFString \"1\"") + } + if got.count == 5 { + #expect(got[4].rpuPayload == Data(Self.rpuNAL(tag: 3).dropFirst(2)), "a borrowed frame carries the previous payload") + #expect((got[4].compatibilityID as? String) == "1") + } + } + + @Test("a run of use_prev_vdr_rpu_flag frames keeps composing with the previous mapping, past the corrupt-RPU borrow limit", .enabled(if: Self.metalAvailable, "needs a Metal device")) + func usePreviousRunIsNotCapped() async throws { + let deliveries = Deliveries() + let decoder = Self.decoder(recordingInto: deliveries, enhancementLayer: true) + defer { decoder.close() } + + let frames = DualLayerVideoDecoder.maxBorrowedParamsRun + 6 + var inputs: [CVPixelBuffer] = [] + for i in 0.. UInt32? { + lock.lock(); defer { lock.unlock() } + return _blMaxByBase[ObjectIdentifier(base)] + } + func flags(of base: CVPixelBuffer) -> DoviComposerFlags? { + lock.lock(); defer { lock.unlock() } + return _flagsByBase[ObjectIdentifier(base)] + } + func acquireSlot() {} + func releaseSlot() {} + func flush() {} + func composeAsync(baseBuffer: CVPixelBuffer, enhancementBuffer: CVPixelBuffer, params: DoviMetalParams, + frameIndex: UInt32, completion: @escaping @Sendable (CVPixelBuffer?) -> Void) { + composeAsyncWithAcquiredSlot(baseBuffer: baseBuffer, enhancementBuffer: enhancementBuffer, + params: params, frameIndex: frameIndex, completion: completion) + } + func composeAsyncWithAcquiredSlot(baseBuffer: CVPixelBuffer, enhancementBuffer: CVPixelBuffer, + params: DoviMetalParams, frameIndex: UInt32, + completion: @escaping @Sendable (CVPixelBuffer?) -> Void) { + lock.lock() + _blMaxByBase[ObjectIdentifier(baseBuffer)] = params.blMax + _flagsByBase[ObjectIdentifier(baseBuffer)] = params.flags + lock.unlock() + completion(baseBuffer) + } + } + + @Test("the previous mapping is the previous frame in the stream, not the previous frame to finish pairing") + func previousMappingFollowsDecodeOrder() async throws { + let composer = ParamsRecordingComposer() + let deliveries = Deliveries() + let decoder = DualLayerVideoDecoder(composer: composer) + defer { decoder.close() } + decoder.onFrame = { buffer, pts, hdr10 in deliveries.record(buffer, pts, hdr10) } + decoder.primeForTesting(enhancementLayer: true) + + func mapping(_ tag: UInt32) -> DoviMetalParams { var p = Self.params(); p.blMax = tag; return p } + // Packets, in decode order: A, B, "same as before", an RPU that failed, then C. + decoder.seedRPUForTesting(pts: Self.pts(0), params: mapping(100), displayMetadataNAL: nil) + decoder.seedRPUForTesting(pts: Self.pts(1), params: mapping(200), displayMetadataNAL: nil) + decoder.seedRPUForTesting(pts: Self.pts(2), params: nil, displayMetadataNAL: nil, usesPrevious: true) + decoder.seedRPUForTesting(pts: Self.pts(3), params: nil, displayMetadataNAL: nil) + decoder.seedRPUForTesting(pts: Self.pts(4), params: mapping(300), displayMetadataNAL: nil) + + // The layers pair in a different order: 0, then 4 and 2 and 3 before 1. + var bases: [Int: CVPixelBuffer] = [:] + for frame in [0, 4, 2, 3, 1] { + let bl = try #require(Self.baseLayer()) + let el = try #require(Self.enhancementLayer()) + bases[frame] = bl + decoder.handleEnhancementLayerFrame(elBuffer: el, pts: Self.pts(Int64(frame))) + decoder.handleBaseLayerFrame(baseBuffer: bl, pts: Self.pts(Int64(frame)), hdr10Data: nil) + } + try await waitFor { deliveries.count == 5 } + let blMax = (0..<5).map { composer.blMax(of: bases[$0]!) } + #expect(blMax == [100, 200, 200, 200, 300]) + } + + @Test("a use_prev frame keeps the previous mapping but takes its own disable_residual_flag") + func usePreviousTakesOwnResidualFlag() async throws { + let composer = ParamsRecordingComposer() + let deliveries = Deliveries() + let decoder = DualLayerVideoDecoder(composer: composer) + defer { decoder.close() } + decoder.onFrame = { buffer, pts, hdr10 in deliveries.record(buffer, pts, hdr10) } + decoder.primeForTesting(enhancementLayer: true) + + var full = Self.params(); full.blMax = 100 + var noResidual = full; noResidual.flags.insert(.disableResidual) + // A residual-less RPU, then "same mapping" frames that switch the residual on and off again. + decoder.seedRPUForTesting(pts: Self.pts(0), params: noResidual, displayMetadataNAL: nil) + decoder.seedRPUForTesting(pts: Self.pts(1), params: nil, displayMetadataNAL: nil, usesPrevious: true, disableResidual: false) + decoder.seedRPUForTesting(pts: Self.pts(2), params: nil, displayMetadataNAL: nil, usesPrevious: true, disableResidual: true) + + var bases: [CVPixelBuffer] = [] + for frame in 0..<3 { + let bl = try #require(Self.baseLayer()) + let el = try #require(Self.enhancementLayer()) + bases.append(bl) + decoder.handleEnhancementLayerFrame(elBuffer: el, pts: Self.pts(Int64(frame))) + decoder.handleBaseLayerFrame(baseBuffer: bl, pts: Self.pts(Int64(frame)), hdr10Data: nil) + } + try await waitFor { deliveries.count == 3 } + #expect(bases.map { composer.blMax(of: $0) } == [100, 100, 100], "the mapping carries over") + #expect(bases.map { composer.flags(of: $0)?.contains(.disableResidual) } == [true, false, true], + "each frame's own disable_residual_flag wins over the borrowed parameters'") + } + + #if DEBUG + @Test("side data recorded for post-seek packets while flush() drains survives the flush and composes") + func sideDataRecordedDuringTheDrainSurvivesFlush() async throws { + let composer = ParamsRecordingComposer() + let deliveries = Deliveries() + let decoder = DualLayerVideoDecoder(composer: composer) + decoder.onFrame = { buffer, pts, hdr10 in deliveries.record(buffer, pts, hdr10) } + decoder.primeForTesting(enhancementLayer: true) + defer { + decoder.flushRaceHookForTesting = nil + decoder.close() + } + + func mapping(_ tag: UInt32) -> DoviMetalParams { var p = Self.params(); p.blMax = tag; return p } + // A pre-seek RPU that the flush must discard, and a post-seek one the demux thread records + // while the hardware drain is still running (the hook runs between the two critical sections). + decoder.seedRPUForTesting(pts: Self.pts(0), params: mapping(100), displayMetadataNAL: nil) + decoder.flushRaceHookForTesting = { [weak decoder] in + decoder?.seedRPUForTesting(pts: Self.pts(10), params: mapping(777), displayMetadataNAL: nil) + } + decoder.flush() + + let staleBL = try #require(Self.baseLayer()) + let staleEL = try #require(Self.enhancementLayer()) + let bl = try #require(Self.baseLayer()) + let el = try #require(Self.enhancementLayer()) + decoder.handleEnhancementLayerFrame(elBuffer: staleEL, pts: Self.pts(0)) + decoder.handleBaseLayerFrame(baseBuffer: staleBL, pts: Self.pts(0), hdr10Data: nil) + decoder.handleEnhancementLayerFrame(elBuffer: el, pts: Self.pts(10)) + decoder.handleBaseLayerFrame(baseBuffer: bl, pts: Self.pts(10), hdr10Data: nil) + try await waitFor { deliveries.count == 2 } + #expect(composer.blMax(of: bl) == 777, "the RPU recorded during the drain must compose its frame") + #expect(composer.blMax(of: staleBL) == nil, "the pre-seek RPU is gone") + } + + @Test("two flushes gated at the same point run one after the other: no interleaving, no stall, no stale delivery") + func concurrentFlushesAreSerialised() async throws { + let deliveries = Deliveries() + let decoder = DualLayerVideoDecoder(composer: ImmediateFakeComposer()) + decoder.onFrame = { buffer, pts, hdr10 in deliveries.record(buffer, pts, hdr10) } + decoder.primeForTesting(enhancementLayer: false) + defer { + decoder.flushRaceHookForTesting = nil + decoder.close() + } + for i in 0..<3 { + let bl = try #require(Self.baseLayer()) + decoder.handleBaseLayerFrame(baseBuffer: bl, pts: Self.pts(Int64(i)), hdr10Data: nil) + } + try await waitFor { deliveries.count == 3 } + + let firstInHook = BoolFlag() + let hookCalls = LockedCounter() + let release = DispatchSemaphore(value: 0) + decoder.flushRaceHookForTesting = { + // Only the first flush to get here is held; the second must never get this far while it is. + if hookCalls.incrementAndCheckFirst() { + firstInHook.set() + release.wait() + } + } + let epochBefore = decoder.feedEpoch + let firstDone = BoolFlag() + let secondDone = BoolFlag() + BlockingWork.detached { decoder.flush(); firstDone.set() } + try await waitFor { firstInHook.isSet } + #expect(decoder.feedEpoch == epochBefore + 1) + + let secondArrived = BoolFlag() + decoder.flushWillTakeLifecycleLockHookForTesting = { secondArrived.set() } + BlockingWork.detached { decoder.flush(); secondDone.set() } + // The second flush must stay out of its critical section (the epoch must not move, `draining` + // must stay owned by the first) while the first is held. Wait until it has reached the lock, + // by a hook rather than by a delay, then check: if the lock were missing it would be past + // the epoch bump within a few instructions, and `release` is still unsignalled, so the first + // flush cannot be what moved it. + try await waitFor { secondArrived.isSet } + #expect(!secondDone.isSet) + #expect(decoder.feedEpoch == epochBefore + 1, "the second flush interleaved with the first") + + release.signal() + try await waitFor { firstDone.isSet && secondDone.isSet } + #expect(decoder.feedEpoch == epochBefore + 2) + + // Delivery works afterwards: new frames take tickets under the final epoch and come out in order. + for i in 3..<6 { + let bl = try #require(Self.baseLayer()) + decoder.handleBaseLayerFrame(baseBuffer: bl, pts: Self.pts(Int64(i)), hdr10Data: nil) + } + try await waitFor { deliveries.count == 6 } + let got = deliveries.snapshot + #expect(got.count == 6, "a frame was delivered twice or a ticket was lost") + #expect(Self.isStrictlyIncreasing(got)) + } + #endif + + @Test("composedFrameCount counts compositions only, not bypasses, and survives flush()") + func composedFrameCountIgnoresBypassesAndFlush() async throws { + let deliveries = Deliveries() + let decoder = DualLayerVideoDecoder(composer: ParamsRecordingComposer()) + defer { decoder.close() } + decoder.onFrame = { buffer, pts, hdr10 in deliveries.record(buffer, pts, hdr10) } + decoder.primeForTesting(enhancementLayer: true) + #expect(decoder.composedFrameCount == 0) + + // Frame 0 has an RPU and its enhancement layer: composed. Frame 1 has no RPU: bypassed. + decoder.seedRPUForTesting(pts: Self.pts(0), params: Self.params(), displayMetadataNAL: nil) + for frame in 0..<2 { + let bl = try #require(Self.baseLayer()) + let el = try #require(Self.enhancementLayer()) + decoder.handleEnhancementLayerFrame(elBuffer: el, pts: Self.pts(Int64(frame))) + decoder.handleBaseLayerFrame(baseBuffer: bl, pts: Self.pts(Int64(frame)), hdr10Data: nil) + } + try await waitFor { deliveries.count == 2 } + #expect(decoder.composedFrameCount == 1) + decoder.flush() + #expect(decoder.composedFrameCount == 1, "the 1 Hz summary and flush reset their own counters, not this one") + } + + @Test("frames without a timestamp pair with nothing, and never trap the key arithmetic") + func missingTimestampNeverMatchesOrTraps() async throws { + let deliveries = Deliveries() + let decoder = Self.decoder(recordingInto: deliveries, enhancementLayer: true) + defer { decoder.close() } + let waitingBL = try #require(Self.baseLayer()) + let waitingEL = try #require(Self.enhancementLayer()) + let noPtsBL = try #require(Self.baseLayer()) + let noPtsEL = try #require(Self.enhancementLayer()) + + // A base-layer frame and an enhancement-layer frame are each waiting on a partner at a normal time. + decoder.handleBaseLayerFrame(baseBuffer: waitingBL, pts: Self.pts(5), hdr10Data: nil) + decoder.handleEnhancementLayerFrame(elBuffer: waitingEL, pts: Self.pts(9)) + // Then a frame of each layer arrives with no PTS (`AV_NOPTS_VALUE` is `Int64.min`). + decoder.handleEnhancementLayerFrame(elBuffer: noPtsEL, pts: .invalid) + decoder.handleBaseLayerFrame(baseBuffer: noPtsBL, pts: .invalid, hdr10Data: nil) + + // The enhancement layer without a timestamp was dropped, and the two waiting frames are untouched. + let snapshot = decoder.stateSnapshotForTesting + #expect(snapshot.elQueueCount == 1) + #expect(snapshot.blQueueCount == 1) + // The base layer without a timestamp bypassed at once, but under its own ticket: it stays behind + // the waiting frame that arrived first. + #expect(deliveries.count == 0) + // Completing the waiting pair releases both, in arrival order. + decoder.handleEnhancementLayerFrame(elBuffer: Self.enhancementLayer()!, pts: Self.pts(5)) + try await waitFor { deliveries.count == 2 } + #expect(deliveries.snapshot.last?.buffer === noPtsBL) + } + + @Test("the side-data ring does not trap on, or match, a missing key") + func ringIgnoresMissingKeys() { + var ring = FrameSideDataRing() + ring.insert(key: 1_000_000, params: DoviMetalParams(), dmNAL: [1, 2, 3]) + ring.insert(key: -5_000_000_000, params: DoviMetalParams()) + #expect(ring.find(key: Int64.min) == nil) + #expect(ring.removeParams(key: Int64.min) == nil) + #expect(ring.removeDisplayMetadata(key: Int64.min) == nil) + #expect(ring.count == 2) + #expect(FrameSideDataRing.distance(Int64.max, -Int64.max) == nil) + #expect(FrameSideDataRing.distance(10, 4) == 6) + #expect(!FrameSideDataRing.isWithinJitter(Int64.min, Int64.min)) + #expect(FrameSideDataRing.isWithinJitter(1000, 1300)) + #expect(!FrameSideDataRing.isWithinJitter(1000, 1301)) + } + + // MARK: - Flush and close + + @Test("nothing composed before a flush is delivered after the flush returns", .enabled(if: Self.metalAvailable, "needs a Metal device")) + func flushRetiresOutstandingFrames() async throws { + let deliveries = Deliveries() + let decoder = Self.decoder(recordingInto: deliveries, enhancementLayer: true) + defer { decoder.close() } + + for i in 0..<6 { + let bl = try #require(Self.baseLayer()) + let el = try #require(Self.enhancementLayer()) + decoder.seedRPUForTesting(pts: Self.pts(Int64(i)), params: Self.params(), displayMetadataNAL: nil) + decoder.handleEnhancementLayerFrame(elBuffer: el, pts: Self.pts(Int64(i))) + decoder.handleBaseLayerFrame(baseBuffer: bl, pts: Self.pts(Int64(i)), hdr10Data: nil) + } + decoder.flush() + deliveries.setMarker() + + // The decoder is usable after a flush: a fresh single-layer frame goes straight out. + decoder.primeForTesting(enhancementLayer: false) + let after = try #require(Self.baseLayer()) + decoder.handleBaseLayerFrame(baseBuffer: after, pts: Self.pts(100), hdr10Data: nil) + decoder.quiesceForTesting() + + let late = deliveries.snapshot.filter { $0.afterMarker } + #expect(late.count == 1) + #expect(late.first?.buffer === after) + } + + @Test("nothing is delivered after close returns", .enabled(if: Self.metalAvailable, "needs a Metal device")) + func closeStopsDelivery() async throws { + let deliveries = Deliveries() + let decoder = Self.decoder(recordingInto: deliveries, enhancementLayer: true) + + for i in 0..<6 { + let bl = try #require(Self.baseLayer()) + let el = try #require(Self.enhancementLayer()) + decoder.seedRPUForTesting(pts: Self.pts(Int64(i)), params: Self.params(), displayMetadataNAL: nil) + decoder.handleEnhancementLayerFrame(elBuffer: el, pts: Self.pts(Int64(i))) + decoder.handleBaseLayerFrame(baseBuffer: bl, pts: Self.pts(Int64(i)), hdr10Data: nil) + } + decoder.close() + deliveries.setMarker() + decoder.quiesceForTesting() + + #expect(deliveries.snapshot.allSatisfy { !$0.afterMarker }) + } + + @Test("close resets stream state and the decoder refuses to reopen") + func closeResetsStreamStateAndRefusesReopen() async throws { + let deliveries = Deliveries() + let decoder = DualLayerVideoDecoder() + decoder.onFrame = { buffer, pts, hdr10 in deliveries.record(buffer, pts, hdr10) } + + // Latch the MEL state a reopened instance must never inherit. + decoder.primeStreamForTesting(.mel) + #expect(decoder.streamClassificationForTesting == .mel) + + decoder.close() + + let snap = decoder.stateSnapshotForTesting + #expect(snap.isClosed) + #expect(!snap.isFEL && !snap.isMEL && !snap.streamClassified) + #expect(!snap.primaryDecoderInitialized) + #expect(!snap.enhancementEngaged) + #expect(snap.frameIndex == 0) + #expect(snap.blQueueCount == 0) + #expect(snap.elQueueCount == 0) + #expect(snap.rpuCacheCount == 0) + #expect(snap.dmRpuCacheCount == 0) + #expect(snap.deliveryPendingCount == 0) + #expect(snap.deliveryNextTicket == 0) + #expect(snap.deliveryNextToEmit == 0) + // Additional fields made explicit in snapshot (cheap to check): + #expect(!snap.draining) + #expect(snap.timeBaseNum == 1 && snap.timeBaseDen == 90000) + #expect(!snap.hasBLVPS && !snap.hasBLSPS && !snap.hasBLPPS) + #expect(!snap.hasELVPS && !snap.hasELSPS && !snap.hasELPPS) + #expect(!snap.hasSkipUntilPTS) + #expect(!snap.hasOnFrame) + + // Single use: `open` after `close` refuses. `open` checks the flag + // before touching `stream`, so uninitialized storage suffices here. + let raw = UnsafeMutableRawPointer.allocate( + byteCount: MemoryLayout.size, + alignment: MemoryLayout.alignment) + defer { raw.deallocate() } + let stream = raw.assumingMemoryBound(to: AVStream.self) + do { + try decoder.open(stream: stream) { _, _, _ in } + Issue.record("open after close must refuse with VideoDecoderError.decoderClosed") + } catch VideoDecoderError.decoderClosed { + // Expected: single-use decoder. + } + + // A post-close base-layer frame must not reach the cleared `onFrame`. + // `isClosed` gates `handleBaseLayerFrame` before any ticket is issued; + // settling before the negative check makes that hold by construction, + // as in every other post-teardown check in this file. + let bl = try #require(Self.baseLayer()) + decoder.handleBaseLayerFrame(baseBuffer: bl, pts: Self.pts(0), hdr10Data: nil) + decoder.quiesceForTesting() + #expect(deliveries.count == 0) + #expect(decoder.stateSnapshotForTesting.deliveryPendingCount == 0) + + // decode-after-close early-returns before touching the packet payload: + // the entry gate refuses without forwarding to either sub-decoder. + let packet = try #require(av_packet_alloc()) + defer { var p: UnsafeMutablePointer? = packet; av_packet_free(&p) } + decoder.decode(packet: packet) + decoder.decode(packet: packet, epoch: decoder.feedEpoch) + // No sub-decoders to check here (real decoders), but entry gate blocks. + + // flush-after-close is a no-op: still closed, epoch untouched, silent. + decoder.flush() + #expect(decoder.stateSnapshotForTesting.isClosed) + #expect(decoder.feedEpoch == decoder.stateSnapshotForTesting.feedEpoch) + + // Double close is safe and idempotent. + decoder.close() + #expect(decoder.stateSnapshotForTesting.isClosed) + } + + @Test("decode and flush after close are no-ops; double close is safe") + func closeRefusesDecodeAndFlushAndDoubleClose() async throws { + let primary = CountingFakeDecoder() + let enhancement = CountingFakeDecoder() + let decoder = DualLayerVideoDecoder( + composer: ImmediateFakeComposer(), + primaryDecoder: primary, + enhancementDecoder: enhancement + ) + let deliveries = Deliveries() + decoder.onFrame = { buffer, pts, hdr10 in deliveries.record(buffer, pts, hdr10) } + decoder.primeForTesting(enhancementLayer: true) + + let epochBeforeClose = decoder.feedEpoch + decoder.close() + let snap = decoder.stateSnapshotForTesting + #expect(snap.isClosed) + #expect(snap.blQueueCount == 0 && snap.elQueueCount == 0) + #expect(snap.rpuCacheCount == 0 && snap.dmRpuCacheCount == 0) + #expect(snap.deliveryPendingCount == 0) + #expect(decoder.feedEpoch == epochBeforeClose + 1) + + // decode-after-close early-returns before touching the packet payload: + // the entry gate refuses without forwarding to either sub-decoder. + let packet = try #require(av_packet_alloc()) + defer { var p: UnsafeMutablePointer? = packet; av_packet_free(&p) } + decoder.decode(packet: packet) + decoder.decode(packet: packet, epoch: decoder.feedEpoch) + #expect(primary.decodeCount == 0) + #expect(enhancement.decodeCount == 0) + #expect(!decoder.stateSnapshotForTesting.enhancementEngaged) + + // flush-after-close is a no-op: still closed, epoch untouched, silent. + decoder.flush() + #expect(decoder.stateSnapshotForTesting.isClosed) + #expect(decoder.feedEpoch == epochBeforeClose + 1) + #expect(primary.flushCount == 0) + #expect(enhancement.flushCount == 0) + + // Post-close frame handlers and in-flight deliveries stay silent and + // issue no tickets. + let ticketsBefore = decoder.stateSnapshotForTesting.deliveryNextTicket + let bl = try #require(Self.baseLayer()) + let el = try #require(Self.enhancementLayer()) + decoder.handleBaseLayerFrame(baseBuffer: bl, pts: Self.pts(0), hdr10Data: nil) + decoder.handleEnhancementLayerFrame(elBuffer: el, pts: Self.pts(0)) + decoder.quiesceForTesting() + #expect(deliveries.count == 0) + #expect(decoder.stateSnapshotForTesting.deliveryNextTicket == ticketsBefore) + + // Double close is safe and idempotent. + decoder.close() + #expect(decoder.stateSnapshotForTesting.isClosed) + #expect(primary.closeCount >= 1) + decoder.quiesceForTesting() + #expect(deliveries.count == 0) + } + + @Test("frames emitted from inside flush() are dropped and never delivered") + func flushDropsFramesEmittedFromInsideFlush() async throws { + // Synchronous composer, so any composition a drained frame managed to + // start would retire — and deliver — on the calling thread. If nothing + // arrives, the draining gate held on every path. + let primary = FlushEmittingFakeDecoder() + let enhancement = FlushEmittingFakeDecoder() + let decoder = DualLayerVideoDecoder( + composer: ImmediateFakeComposer(), + primaryDecoder: primary, + enhancementDecoder: enhancement + ) + let deliveries = Deliveries() + decoder.onFrame = { buffer, pts, hdr10 in deliveries.record(buffer, pts, hdr10) } + decoder.primeForTesting(enhancementLayer: true) + defer { decoder.close() } + + let drainBL0 = try #require(Self.baseLayer()) + let drainEL0 = try #require(Self.enhancementLayer()) + let drainBL2 = try #require(Self.baseLayer()) + let drainEL2 = try #require(Self.enhancementLayer()) + let pre0 = SendableFrameBuffer(value: drainBL0) + let pre0e = SendableFrameBuffer(value: drainEL0) + let pre2 = SendableFrameBuffer(value: drainBL2) + let pre2e = SendableFrameBuffer(value: drainEL2) + decoder.seedRPUForTesting(pts: Self.pts(0), params: Self.params(), displayMetadataNAL: nil) + decoder.seedRPUForTesting(pts: Self.pts(2), params: Self.params(), displayMetadataNAL: nil) + // What the installed `onFrame` closures would do with owed pre-seek + // frames: re-enter the pairing handlers from inside the flush, in both + // arrival orders (primary drains first, then enhancement). + primary.emitFromInsideFlush = { + decoder.handleBaseLayerFrame(baseBuffer: pre0.value, pts: Self.pts(0), hdr10Data: nil) + decoder.handleEnhancementLayerFrame(elBuffer: pre0e.value, pts: Self.pts(0)) + } + enhancement.emitFromInsideFlush = { + decoder.handleEnhancementLayerFrame(elBuffer: pre2e.value, pts: Self.pts(2)) + decoder.handleBaseLayerFrame(baseBuffer: pre2.value, pts: Self.pts(2), hdr10Data: nil) + } + + decoder.flush() + decoder.quiesceForTesting() + #expect(deliveries.count == 0, "pre-seek frames drained during flush() must never be delivered") + + // The decoder is usable after the flush: a fresh pair composes and delivers. + let bl = try #require(Self.baseLayer()) + let el = try #require(Self.enhancementLayer()) + decoder.seedRPUForTesting(pts: Self.pts(1), params: Self.params(), displayMetadataNAL: nil) + decoder.handleEnhancementLayerFrame(elBuffer: el, pts: Self.pts(1)) + decoder.handleBaseLayerFrame(baseBuffer: bl, pts: Self.pts(1), hdr10Data: nil) + // Even a synchronous composition's delivery is one more + // `queue.async` hop through `OrderedDelivery` away, so + // `handleBaseLayerFrame` returning is not a guarantee `onFrame` has + // run yet for it. + try await waitFor { deliveries.count == 1 } + #expect(deliveries.count == 1) + #expect(CMTimeCompare(deliveries.snapshot.first?.pts ?? .invalid, Self.pts(1)) == 0) + } + + // MARK: - Thread safety + + /// Synchronous composer so the stress test needs no GPU: every composition + /// retires on the calling thread instead of on a Metal completion queue. + /// + /// Delivery is still asynchronous: every frame, composed or bypassed, is + /// one more `queue.async` hop away from `onFrame` regardless of how this + /// fake calls its `completion`. What this fake buys a test: pairing and composition finish deterministically, in program + /// order, with no GPU and no wait for a command buffer to retire, so a + /// test can reason about ticket order and queue depth without also + /// reasoning about the exactly-once assertions being asynchronous; + /// tests below still have to wait for the frame to actually leave + /// `OrderedDelivery`'s own queue (`waitFor`/`settle`). + private final class ImmediateFakeComposer: FelComposing, @unchecked Sendable { + func acquireSlot() {} + func releaseSlot() {} + func flush() {} + func composeAsync( + baseBuffer: CVPixelBuffer, + enhancementBuffer: CVPixelBuffer, + params: DoviMetalParams, + frameIndex: UInt32, + completion: @escaping @Sendable (CVPixelBuffer?) -> Void + ) { + completion(baseBuffer) + } + func composeAsyncWithAcquiredSlot( + baseBuffer: CVPixelBuffer, + enhancementBuffer: CVPixelBuffer, + params: DoviMetalParams, + frameIndex: UInt32, + completion: @escaping @Sendable (CVPixelBuffer?) -> Void + ) { + completion(baseBuffer) + } + } + + private struct SendableFrameBuffer: @unchecked Sendable { + let value: CVPixelBuffer + } + + /// Fake single-layer decoder: runs a hook from inside `flush()`, + /// the way `HardwareVideoDecoder.flush()` re-enters the frame callbacks + /// with the frames VideoToolbox still owes while the host thread is + /// blocked inside the flush. + private final class FlushEmittingFakeDecoder: SingleLayerVideoDecoder, @unchecked Sendable { + var onFrame: DecodedFrameHandler? + var onFirstHDR10PlusDetected: (@Sendable () -> Void)? + var onA53Captions: (@Sendable ([CCDataParser.CCTriplet], Double) -> Void)? + var onDecodedFormat: (@Sendable (DecodedVideoFormat) -> Void)? + var skipUntilPTS: CMTime? + var feedEpoch: UInt64 = 0 + var emitFromInsideFlush: (() -> Void)? + func open(stream: UnsafeMutablePointer, onFrame: @escaping DecodedFrameHandler) throws { + self.onFrame = onFrame + } + func open( + parameterSets: [[UInt8]], + timeBase: AVRational, + onFrame: @escaping DecodedFrameHandler + ) throws { + self.onFrame = onFrame + } + func decode(packet: UnsafeMutablePointer, epoch: UInt64?) {} + func flush() { emitFromInsideFlush?() } + func close() { onFrame = nil } + } + + /// Counting fake: records sub-decoder traffic so tests can prove + /// post-close `decode`/`flush` never reach the torn-down sessions. + private final class CountingFakeDecoder: SingleLayerVideoDecoder, @unchecked Sendable { + private let lock = NSLock() + private var _decodeCount = 0 + private var _flushCount = 0 + private var _closeCount = 0 + var decodeCount: Int { lock.lock(); defer { lock.unlock() }; return _decodeCount } + var flushCount: Int { lock.lock(); defer { lock.unlock() }; return _flushCount } + var closeCount: Int { lock.lock(); defer { lock.unlock() }; return _closeCount } + var onFrame: DecodedFrameHandler? + var onFirstHDR10PlusDetected: (@Sendable () -> Void)? + var onA53Captions: (@Sendable ([CCDataParser.CCTriplet], Double) -> Void)? + var onDecodedFormat: (@Sendable (DecodedVideoFormat) -> Void)? + var skipUntilPTS: CMTime? + var feedEpoch: UInt64 = 0 + func open(stream: UnsafeMutablePointer, onFrame: @escaping DecodedFrameHandler) throws { + self.onFrame = onFrame + } + func open( + parameterSets: [[UInt8]], + timeBase: AVRational, + onFrame: @escaping DecodedFrameHandler + ) throws { + self.onFrame = onFrame + } + func decode(packet: UnsafeMutablePointer, epoch: UInt64?) { + lock.lock(); _decodeCount += 1; lock.unlock() + } + func flush() { lock.lock(); _flushCount += 1; lock.unlock() } + func close() { lock.lock(); _closeCount += 1; lock.unlock(); onFrame = nil } + } + + @Test("concurrent layer arrivals and control-plane churn keep exactly-once delivery; close under load stays silent") + func concurrentArrivalsAndTeardown() async throws { + let frames = 64 + let deliveries = Deliveries() + let decoder = DualLayerVideoDecoder(composer: ImmediateFakeComposer()) + decoder.onFrame = { buffer, pts, hdr10 in deliveries.record(buffer, pts, hdr10) } + decoder.primeForTesting(enhancementLayer: true) + for i in 0.. = [] + do { + // Nudged unconditionally, so the test does not have to decide by a timeout whether + // frames are stranded. `collectDueBypasses` ages a queued base-layer frame out after + // `K` further base-layer *arrivals*, not elapsed time, and only runs from inside + // `handleBaseLayerFrame`. This test's 64-way unthrottled fan-out delivers each layer + // out of decode order essentially every run (measured: 2–3 of 64 base-layer frames + // stranded in `blFrameQueue` in 40 of 40 runs; two threads delivering each layer in + // decode order stranded 0 of 40). That cannot happen mid-stream in production, where + // VideoToolbox outputs each layer in decode order; the only production exposure is + // the last base-layer frames at end of stream whose enhancement layer never arrives, + // which is the K-frame bound itself. A base layer can be scheduled so late that + // "newest settled base layer" has already raced ahead of it; its enhancement layer is + // then correctly dropped as stale, and no further base-layer arrival happens within + // this batch to age it out before the `DispatchGroup` above finishes. It is still + // queued, not lost: a few harmless, never-pairing nudges (fresh buffers, PTS values + // outside this test's range) advance `blArrivalSeq` past it. + for i in 0..<(DualLayerVideoDecoder.maxLayerSkewFrames + 1) { + guard let nudge = Self.baseLayer() else { break } + nudgeIDs.insert(ObjectIdentifier(nudge)) + decoder.handleBaseLayerFrame(baseBuffer: nudge, pts: Self.pts(Int64(900 + i)), hdr10Data: nil) + } + try await waitFor { + deliveries.snapshot.filter { sentIDs.contains(ObjectIdentifier($0.buffer)) }.count >= frames + } + } + // `allSatisfy`, not a `filter`: an enhancement-layer buffer (or + // anything else) delivered on its own is what this catches; sent + // base-layer frames and the nudges above are the only buffers ever + // allowed out. + let allowed = sentIDs.union(nudgeIDs) + #expect(deliveries.snapshot.allSatisfy { allowed.contains(ObjectIdentifier($0.buffer)) }, + "only sent base-layer frames or the nudges that age out stranded ones may ever be delivered") + let got = deliveries.snapshot.filter { sentIDs.contains(ObjectIdentifier($0.buffer)) } + #expect(got.count == frames) + #expect(Set(got.map { ObjectIdentifier($0.buffer) }).count == frames) + + // Teardown under load: arrivals racing close() are either delivered + // before it or dropped after it; nothing lands past the marker. + let teardown = DispatchGroup() + for i in frames..<(frames + 8) { + teardown.enter() + DispatchQueue.global().async { + if let b = Self.baseLayer(), let e = Self.enhancementLayer() { + decoder.handleBaseLayerFrame(baseBuffer: b, pts: Self.pts(Int64(i)), hdr10Data: nil) + decoder.handleEnhancementLayerFrame(elBuffer: e, pts: Self.pts(Int64(i))) + } + teardown.leave() + } + } + teardown.enter() + DispatchQueue.global().async { + decoder.flush() + decoder.close() + teardown.leave() + } + await withCheckedContinuation { cont in teardown.notify(queue: .global()) { cont.resume() } } + deliveries.setMarker() + if let b = Self.baseLayer(), let e = Self.enhancementLayer() { + decoder.handleBaseLayerFrame(baseBuffer: b, pts: Self.pts(10_000), hdr10Data: nil) + decoder.handleEnhancementLayerFrame(elBuffer: e, pts: Self.pts(10_000)) + } + decoder.quiesceForTesting() + #expect(deliveries.snapshot.allSatisfy { !$0.afterMarker }) + } + + // MARK: - Delivery queue + + /// Fake composer whose Nth call (1-indexed) never calls its completion at + /// all — the one failure mode a ticket cannot self-report, unlike a + /// bypass or a "compose failed" fallback. Every other call completes + /// synchronously with the base layer, like `ImmediateFakeComposer`. + private final class DroppingFakeComposer: FelComposing, @unchecked Sendable { + private let dropOnCall: Int + private let lock = NSLock() + private var calls = 0 + init(dropOnCall: Int) { self.dropOnCall = dropOnCall } + func acquireSlot() {} + func releaseSlot() {} + func flush() {} + func composeAsync( + baseBuffer: CVPixelBuffer, + enhancementBuffer: CVPixelBuffer, + params: DoviMetalParams, + frameIndex: UInt32, + completion: @escaping @Sendable (CVPixelBuffer?) -> Void + ) { + composeAsyncWithAcquiredSlot(baseBuffer: baseBuffer, enhancementBuffer: enhancementBuffer, params: params, frameIndex: frameIndex, completion: completion) + } + func composeAsyncWithAcquiredSlot( + baseBuffer: CVPixelBuffer, + enhancementBuffer: CVPixelBuffer, + params: DoviMetalParams, + frameIndex: UInt32, + completion: @escaping @Sendable (CVPixelBuffer?) -> Void + ) { + lock.lock(); calls += 1; let n = calls; lock.unlock() + if n == dropOnCall { + // Simulates a completion that never fires: no call to + // `completion`, so the ticket this composition took is never + // fulfilled at all. + return + } + completion(baseBuffer) + } + } + + @Test("a composition whose completion never fires costs exactly one frame, not the session") + func lostCompletionCostsExactlyOneFrame() async throws { + let threshold = DoviFelComposer.maxFramesInFlight + DualLayerVideoDecoder.maxLayerSkewFrames + 2 + let composer = DroppingFakeComposer(dropOnCall: 1) + let deliveries = Deliveries() + let decoder = DualLayerVideoDecoder(composer: composer) + decoder.onFrame = { buffer, pts, hdr10 in deliveries.record(buffer, pts, hdr10) } + decoder.primeForTesting(enhancementLayer: true) + defer { decoder.close() } + + // Every frame pairs immediately (EL arrives first, so `handleBaseLayerFrame` + // composes on the spot) so ticket `i` and loop index `i` coincide. The + // first composition's ticket (0) is the one `DroppingFakeComposer` drops; + // enough frames follow to push it past `orderedDelivery`'s stall + // threshold (`maxFramesInFlight + K + 2`). + let frames = threshold + 6 + for i in 0.. UnsafeMutablePointer { + let slice = hevcNAL(type: 1, payload: [0x11, 0x22, 0x33, 0x44]) + return try #require(makeAvccPacket([hevcVPS, hevcSPS, hevcPPS, slice], pts: pts)) + } + + #if DEBUG + @Test("a post-seek packet fed with the new epoch between the dual epoch bump and the sub-decoder flush reaches VideoToolbox; a pre-seek one does not") + func postSeekPacketInTheFlushWindowIsNotDropped() async throws { + // The host feeds from the demux thread while a seek's flush() runs. It reads the new epoch the + // moment flush() publishes it. The sub-decoders used to move only in their own flush(), later, + // so a packet fed in this window passed the dual gate and was refused by a decoder still on + // the old epoch: the post-seek IRAP never reached VideoToolbox. + let primary = HardwareVideoDecoder() + let decoder = DualLayerVideoDecoder(composer: ImmediateFakeComposer(), primaryDecoder: primary, enhancementDecoder: nil) + decoder.onFrame = { _, _, _ in } + defer { decoder.flushWillTakeLifecycleLockHookForTesting = nil; decoder.flushRaceHookForTesting = nil; decoder.close() } + let submitted = LockedCounter() + primary.onSubmittedToVideoToolboxForTesting = { submitted.increment() } + + // The first packet opens the primary decoder from its in-band parameter sets. + let pre = try Self.inBandPacket(pts: 0) + defer { var p: UnsafeMutablePointer? = pre; trackedPacketFree(&p) } + decoder.decode(packet: pre, epoch: decoder.feedEpoch) + if decoder.fatalDecodeError != nil { try Test.cancel("hardware HEVC session unavailable: \(decoder.fatalDecodeError!)") } + #expect(submitted.value == 1) + let oldEpoch = decoder.feedEpoch + + let post = try Self.inBandPacket(pts: 90000) + defer { var p: UnsafeMutablePointer? = post; trackedPacketFree(&p) } + let submittedInWindow = LockedCounter() + decoder.flushRaceHookForTesting = { + // flush() has published the new epoch and not yet touched the sub-decoders. + decoder.decode(packet: post, epoch: decoder.feedEpoch) + submittedInWindow.set(submitted.value) + } + decoder.flush() + #expect(decoder.feedEpoch == oldEpoch + 1) + #expect(primary.feedEpoch == oldEpoch + 1, "the sub-decoder's own flush must not move the epoch a second time") + #expect(submittedInWindow.value == 2, "the post-seek packet fed inside the window never reached VideoToolbox") + + // DEC-4: a packet decided under the old epoch must not enter VideoToolbox after the flush. + decoder.decode(packet: pre, epoch: oldEpoch) + #expect(submitted.value == 2) + // And the epoch the host would read next is accepted. + decoder.decode(packet: post, epoch: decoder.feedEpoch) + #expect(submitted.value == 3) + } + #endif + + /// Fake whose in-band open always fails. + private final class FailingOpenFakeDecoder: SingleLayerVideoDecoder, @unchecked Sendable { + private let lock = NSLock() + private var _opens = 0, _decodes = 0 + var openCount: Int { lock.lock(); defer { lock.unlock() }; return _opens } + var decodeCount: Int { lock.lock(); defer { lock.unlock() }; return _decodes } + struct OpenFailed: Error {} + var onFrame: DecodedFrameHandler? + var onFirstHDR10PlusDetected: (@Sendable () -> Void)? + var onA53Captions: (@Sendable ([CCDataParser.CCTriplet], Double) -> Void)? + var onDecodedFormat: (@Sendable (DecodedVideoFormat) -> Void)? + var skipUntilPTS: CMTime? + var feedEpoch: UInt64 = 0 + func open(stream: UnsafeMutablePointer, onFrame: @escaping DecodedFrameHandler) throws {} + func open(parameterSets: [[UInt8]], timeBase: AVRational, onFrame: @escaping DecodedFrameHandler) throws { + lock.lock(); _opens += 1; lock.unlock() + throw OpenFailed() + } + func decode(packet: UnsafeMutablePointer, epoch: UInt64?) { lock.lock(); _decodes += 1; lock.unlock() } + func flush() {} + func close() {} + } + + @Test("an in-band primary open that fails is reported once and not retried on every later packet") + func failedInBandOpenFailsOnce() throws { + let primary = FailingOpenFakeDecoder() + let decoder = DualLayerVideoDecoder(composer: ImmediateFakeComposer(), primaryDecoder: primary, enhancementDecoder: nil) + defer { decoder.close() } + let reported = LockedCounter() + decoder.onFatalDecodeError = { _ in reported.increment() } + let lines = LockedCounter() + let previous = EngineLog.handler + EngineLog.handler = { line in if line.contains("failed to open the primary decoder") { lines.increment() } } + defer { EngineLog.handler = previous } + + for i in 0..<5 { + let pkt = try Self.inBandPacket(pts: Int64(i) * 3000) + decoder.decode(packet: pkt) + var p: UnsafeMutablePointer? = pkt + trackedPacketFree(&p) + } + #expect(primary.openCount == 1, "the open was retried on a later packet") + #expect(reported.value == 1) + #expect(lines.value == 1) + #expect(decoder.fatalDecodeError is FailingOpenFakeDecoder.OpenFailed) + #expect(primary.decodeCount == 0) + } + + @Test("an enhancement decoder that fails to open is logged once and not retried on every later packet") + func failedEnhancementOpenFailsOnce() throws { + let decoder = DualLayerVideoDecoder(composer: ImmediateFakeComposer(), primaryDecoder: FailingOpenFakeDecoder(), enhancementDecoder: nil) + let enhancement = FailingOpenFakeDecoder() + let built = LockedCounter() + decoder.makeEnhancementDecoder = { built.increment(); return enhancement } + defer { decoder.close() } + let lines = LockedCounter() + let previous = EngineLog.handler + EngineLog.handler = { line in if line.contains("failed to initialize EL decoder") { lines.increment() } } + defer { EngineLog.handler = previous } + + for i in 0..<5 { + let pkt = try Self.inBandPacket(pts: Int64(i) * 3000) + decoder.ensureEnhancementDecoder(from: pkt) + var p: UnsafeMutablePointer? = pkt + trackedPacketFree(&p) + } + #expect(enhancement.openCount == 1, "the open was retried on a later packet") + #expect(built.value == 1, "a new session was built for a later packet") + #expect(lines.value == 1) + #expect(decoder.enhancementDecoderOpenError is FailingOpenFakeDecoder.OpenFailed) + #expect(decoder.fatalDecodeError == nil, "a missing enhancement layer leaves the base layer to show") + // close() starts the next session clean. + decoder.close() + #expect(decoder.enhancementDecoderOpenError == nil) + } + + @Test("the alignment's restamp bound is what the pairing skew bound can absorb") + func alignmentBoundTracksSkewBound() { + #expect(DualLayerVideoDecoder.makeAlignment().maxBackwardRestamp == DualLayerVideoDecoder.maxLayerSkewFrames - 1) + } + + @Test("a crafted raw timestamp that overflows the time-base product yields no key instead of trapping") + func canonicalKeyIsOverflowSafe() { + let tb = AVRational(num: 1, den: 90000) + #expect(DualLayerVideoDecoder.canonicalKey(rawPTS: 90000, timeBase: tb) == 90000) + #expect(DualLayerVideoDecoder.canonicalKey(rawPTS: Int64.min, timeBase: tb) == Int64.min) + let wide = AVRational(num: 1_000_000, den: 1) + #expect(DualLayerVideoDecoder.canonicalKey(rawPTS: Int64.max, timeBase: wide) == Int64.min) + #expect(DualLayerVideoDecoder.canonicalKey(rawPTS: Int64.min + 1, timeBase: wide) == Int64.min) + // Large but representable products saturate in CoreMedia rather than trapping. + _ = DualLayerVideoDecoder.canonicalKey(rawPTS: Int64.max / 2, timeBase: AVRational(num: 2, den: 1)) + } + + @Test("a slow onFrame does not block pairing or flush() behind the decoder lock", + .timeLimit(.minutes(1))) + func slowOnFrameDoesNotBlockDecoderOperations() async throws { + // The old `deliver()` took the decoder's lock and called `emit` (hence + // `onFrame`) from inside it, so a slow host callback would have stalled + // every later frame handler and `flush()` behind the same lock. With + // `OrderedDelivery` owning `emit` on its own queue, pairing a new frame + // never waits on `onFrame` at all — but `flush()` is a different claim: + // its `orderedDelivery.barrier()` call legitimately waits for whatever + // `emit` is currently in the middle of before draining can proceed. The + // bound here is "as long as `onFrame` itself takes", not "instantly" + // and not "forever". + // `firstCallStarted` is polled rather than waited on with a semaphore: + // `DispatchSemaphore.wait` is unavailable from this `async` test body + // (it would block the cooperative thread pool), but it is exactly + // right *inside* the plain, non-async `onFrame` closure below, which + // runs on `OrderedDelivery`'s own queue and is what this test needs + // to hold still to prove pairing is not waiting on it. + let firstCallStarted = BoolFlag() + let releaseFirstCall = DispatchSemaphore(value: 0) + let callCount = LockedCounter() + let deliveries = Deliveries() + let decoder = DualLayerVideoDecoder(composer: ImmediateFakeComposer()) + decoder.onFrame = { buffer, pts, hdr10 in + let isFirst = callCount.incrementAndCheckFirst() + if isFirst { + firstCallStarted.set() + releaseFirstCall.wait() + } + deliveries.record(buffer, pts, hdr10) + } + decoder.primeForTesting(enhancementLayer: false) + defer { + releaseFirstCall.signal() // in case an assertion below fails first + decoder.close() + } + + let bl0 = try #require(Self.baseLayer()) + decoder.handleBaseLayerFrame(baseBuffer: bl0, pts: Self.pts(0), hdr10Data: nil) + try await waitFor { firstCallStarted.isSet } + #expect(firstCallStarted.isSet, "onFrame for frame 0 never started") + + // While that first `onFrame` call is still parked, pairing a new frame + // must not wait on it at all: `handleBaseLayerFrame` only ever + // `fulfil`s a ticket onto `OrderedDelivery`'s queue (`queue.async`, + // non-blocking) and never calls into `emit`. The call returning at all, + // with the release semaphore still unsignalled, is the proof: if it + // waited on the parked `onFrame` it would never return and the test's + // time limit would report it. No wall-clock threshold is involved. + let bl1 = try #require(Self.baseLayer()) + decoder.handleBaseLayerFrame(baseBuffer: bl1, pts: Self.pts(1), hdr10Data: nil) + + // `flush()` legitimately waits for the in-flight `emit` (its `barrier()` + // waits on the queue `emit` runs on), so it runs on a thread of its own + // and the release is signalled from here. It must then finish; if the + // decoder's lock were held across the wait it would hang, and the time + // limit would say so. + let flushDone = BoolFlag() + Thread.detachNewThread { + decoder.flush() + flushDone.set() + } + releaseFirstCall.signal() + try await waitFor { flushDone.isSet } + #expect(flushDone.isSet, "flush() never returned after the slow onFrame call was released") + + decoder.quiesceForTesting() + } + + /// Thread-safe increment that also reports whether this call was the + /// first one, without a data race between the check and the increment. + private final class LockedCounter: @unchecked Sendable { + private let lock = NSLock() + private var count = 0 + func incrementAndCheckFirst() -> Bool { + lock.lock(); defer { lock.unlock() } + count += 1 + return count == 1 + } + func increment() { lock.lock(); count += 1; lock.unlock() } + func set(_ v: Int) { lock.lock(); count = v; lock.unlock() } + var value: Int { lock.lock(); defer { lock.unlock() }; return count } + } + + /// A boolean settable from one thread and polled from another, with no + /// torn reads. Used both as a "stop requested" flag and, in + /// `slowOnFrameDoesNotBlockDecoderOperations`, as a "started" flag — the + /// name is generic on purpose. + private final class BoolFlag: @unchecked Sendable { + private let lock = NSLock() + private var value = false + func set() { lock.lock(); value = true; lock.unlock() } + var isSet: Bool { lock.lock(); defer { lock.unlock() }; return value } + } + + /// Lock-protected "flushes started"/"flushes returned" counters for the + /// per-flush I3 detector in `flushFromAnotherThreadInATightLoopKeepsInvariantI3`. + private final class FlushCounters: @unchecked Sendable { + private let lock = NSLock() + private var started = 0 + private var returned = 0 + /// Call immediately before `decoder.flush()`. Returns the new count. + @discardableResult + func beginFlush() -> Int { lock.lock(); started += 1; defer { lock.unlock() }; return started } + /// Call immediately after `decoder.flush()` returns. + func endFlush() { lock.lock(); returned += 1; lock.unlock() } + var startedCount: Int { lock.lock(); defer { lock.unlock() }; return started } + var returnedCount: Int { lock.lock(); defer { lock.unlock() }; return returned } + } + + /// Records each frame's "started" tag (keyed by its unique PTS value) and + /// any I3 violation the detector in + /// `flushFromAnotherThreadInATightLoopKeepsInvariantI3` finds — a class, + /// like `Deliveries`/`FlushCounters` above, so the producer task and the + /// `onFrame` closure can share it as a `let` capture rather than a `var` + /// one, which Swift 6 strict concurrency refuses to let a closure and a + /// detached task mutate directly. + private final class ViolationTracker: @unchecked Sendable { + private let lock = NSLock() + private var startedTagByPTS: [Int64: Int] = [:] + private var violations: [String] = [] + + /// Marks a frame as being inside its pairing call. Delivery is asynchronous and can run before + /// the call returns, so the frame has to be known to the detector before the call starts; the + /// real tag replaces this once the call returns. + func registerInFlight(ptsValue: Int64) { + lock.lock(); startedTagByPTS[ptsValue] = Self.inFlight; lock.unlock() + } + + func recordStartedTag(_ tag: Int, forPTSValue ptsValue: Int64) { + lock.lock(); startedTagByPTS[ptsValue] = tag; lock.unlock() + } + + private static let inFlight = Int.max + + /// Checks one delivery against the rule and records a violation if + /// found. Call from `onFrame`, passing the "returned" count read at + /// that moment. + func checkDelivery(ptsValue: Int64, returnedAtDelivery: Int) { + lock.lock() + defer { lock.unlock() } + guard let startedTag = startedTagByPTS[ptsValue] else { + violations.append("pts=\(ptsValue) delivered with no tag registered") + return + } + // Still inside its pairing call: it cannot have been paired before a flush that has since returned. + guard startedTag != Self.inFlight, startedTag < returnedAtDelivery else { return } + violations.append("pts=\(ptsValue) startedTag=\(startedTag) returnedAtDelivery=\(returnedAtDelivery)") + } + + var snapshot: [String] { lock.lock(); defer { lock.unlock() }; return violations } + } + + @Test("flushing from another thread in a tight loop, while frames keep arriving, never delivers a frame paired before flush k started after flush k returns (I3)") + func flushFromAnotherThreadInATightLoopKeepsInvariantI3() async throws { + // An elapsed-time bound and the epoch bump cannot detect an I3 + // violation, and neither can `isStrictlyIncreasing` alone: a pre-flush + // frame delivered late still carries a lower PTS than every post-flush + // frame the ever-increasing producer goes on to emit. + // + // So this test uses a per-flush detector: two lock-protected counters, "flushes started" and + // "flushes returned". Each frame is tagged, the instant its pairing + // call returns, with the "started" count at that moment — how many + // flushes had already been kicked off by the time this frame took + // its ticket. Each delivery reads the "returned" count at the moment + // `onFrame` fires — how many flushes have already fully completed by + // the time this frame reached the host. The rule: a frame whose + // pairing call returned before flush k started (tag < k) must never + // reach `onFrame` after flush k has returned (returnedAtDelivery >= + // k). Collapsing "there exists such a k" to a single per-frame + // comparison is valid because the tightest possible k for any + // violation is the number of flushes already returned at delivery + // time itself: if `tag < returnedAtDelivery`, then k = + // returnedAtDelivery satisfies `tag < k` and `k <= returnedAtDelivery` + // simultaneously, which is exactly the violation shape. + let counters = FlushCounters() + let tracker = ViolationTracker() + + let deliveries = Deliveries() + let decoder = DualLayerVideoDecoder(composer: ImmediateFakeComposer()) + decoder.onFrame = { buffer, pts, hdr10 in + let returnedAtDelivery = counters.returnedCount + tracker.checkDelivery(ptsValue: pts.value, returnedAtDelivery: returnedAtDelivery) + deliveries.record(buffer, pts, hdr10) + } + decoder.primeForTesting(enhancementLayer: false) + defer { decoder.close() } + + let stop = BoolFlag() + + let producing = BlockingWork.detached { () -> Void in + var i: Int64 = 0 + while !stop.isSet { + if let bl = Self.baseLayer() { + let pts = Self.pts(i) + tracker.registerInFlight(ptsValue: pts.value) + decoder.handleBaseLayerFrame(baseBuffer: bl, pts: pts, hdr10Data: nil) + // The real tag is taken AFTER the handler returns, matching the detector's rule. + let tag = counters.startedCount + tracker.recordStartedTag(tag, forPTSValue: pts.value) + } + i += 1 + } + } + + // Flush back-to-back, on another thread, for as long as the producer + // above keeps feeding frames concurrently. Each call must still + // return promptly (proving no deadlock under contention) and bump the + // feed epoch (proving it actually ran, not a silent no-op). + for _ in 0..<200 { + counters.beginFlush() + let epochBefore = decoder.feedEpoch + // A deadlock under producer load shows as this loop never finishing, which the suite's + // time limit reports; a wall-clock bound per flush would only add scheduler noise. + decoder.flush() + counters.endFlush() + #expect(decoder.feedEpoch == epochBefore + 1) + } + + stop.set() + await producing.value + decoder.quiesceForTesting() + + let finalViolations = tracker.snapshot + #expect(finalViolations.isEmpty, "I3 violated: \(finalViolations.prefix(5))") + + // Order is still a real property worth keeping, just not the one + // that actually proves I3 (see the comment above): any I3 violation + // this detector doesn't already flag would have to slip a stale + // frame in without disturbing PTS order at all, which the pairing + // ticket order already makes exceedingly unlikely — but it costs + // nothing extra to keep checking. + #expect(Self.isStrictlyIncreasing(deliveries.snapshot)) + } + + // MARK: - Known defects + + @Test("a pruned base-layer frame cannot be delivered after a newer composed frame", .enabled(if: Self.metalAvailable, "needs a Metal device")) + func prunedFrameDoesNotOvertakeNewerFrames() async throws { + let deliveries = Deliveries() + let decoder = Self.decoder(recordingInto: deliveries, enhancementLayer: true) + defer { decoder.close() } + + // Frame 0's enhancement layer is lost, so it waits in the base-layer queue. + let orphan = try #require(Self.baseLayer()) + decoder.handleBaseLayerFrame(baseBuffer: orphan, pts: Self.pts(0), hdr10Data: nil) + // Frame 1 pairs immediately and is composed. + let bl1 = try #require(Self.baseLayer()) + let el1 = try #require(Self.enhancementLayer()) + decoder.seedRPUForTesting(pts: Self.pts(1), params: Self.params(), displayMetadataNAL: nil) + decoder.handleEnhancementLayerFrame(elBuffer: el1, pts: Self.pts(1)) + decoder.handleBaseLayerFrame(baseBuffer: bl1, pts: Self.pts(1), hdr10Data: nil) + // More orphans until frame 0 is pushed out of the queue. + var next: Int64 = 2 + while !deliveries.snapshot.contains(where: { $0.buffer === orphan }) && next < 60 { + let bl = try #require(Self.baseLayer()) + decoder.handleBaseLayerFrame(baseBuffer: bl, pts: Self.pts(next), hdr10Data: nil) + next += 1 + } + try await waitFor { deliveries.snapshot.contains { $0.buffer !== orphan && CMTimeCompare($0.pts, Self.pts(1)) == 0 } } + + let got = deliveries.snapshot + #expect(Self.isStrictlyIncreasing(got)) + } + + @Test("a lost enhancement layer delays its base layer by at most K frames, in order") + func lostEnhancementLayerDelaysBaseLayerByAtMostKFrames() async throws { + // Synchronous composer, so delivery is synchronous and the bound is exact: + // frame 0 waits while newer pairs settle, then goes out under its own + // (oldest) ticket once K newer base-layer frames have arrived. + let k = DualLayerVideoDecoder.maxLayerSkewFrames + let deliveries = Deliveries() + let decoder = DualLayerVideoDecoder(composer: ImmediateFakeComposer()) + decoder.onFrame = { buffer, pts, hdr10 in deliveries.record(buffer, pts, hdr10) } + decoder.primeForTesting(enhancementLayer: true) + defer { decoder.close() } + + // Not `deliveries.count == 0`: delivery is always one more + // `queue.async` hop away, so it would read 0 whether or not frame 0 + // was still held back. `blQueueCount` is synchronous and + // deterministic: `collectDueBypasses` runs entirely under `lock` + // inside `handleBaseLayerFrame`, so by the time that call returns, + // frame 0 is provably still queued (not yet bypassed) if and only + // if fewer than K newer base-layer frames have arrived. + let orphan = try #require(Self.baseLayer()) + decoder.handleBaseLayerFrame(baseBuffer: orphan, pts: Self.pts(0), hdr10Data: nil) + #expect(decoder.stateSnapshotForTesting.blQueueCount == 1, "a base-layer frame with a pending enhancement layer must wait for it") + + for i in 1..? = buf + defer { + if bufRef != nil { + av_buffer_unref(&bufRef) + } + } + + #expect(av_buffer_get_ref_count(buf) == 1) + + do { + let source = try #require(HardwareVideoDecoder.CustomBlockSource(buf: buf)) + #expect(av_buffer_get_ref_count(buf) == 2) + #expect(source.dataPtr == UnsafeMutableRawPointer(buf.pointee.data!)) + + // Test idempotent releaseBuffer + source.releaseBuffer() + #expect(av_buffer_get_ref_count(buf) == 1) + #expect(source.avBuf == nil) + + // Second releaseBuffer call is a no-op (idempotent, no double-unref) + source.releaseBuffer() + #expect(av_buffer_get_ref_count(buf) == 1) + } + // After deinit of source, refcount stays 1 (no double-unref) + #expect(av_buffer_get_ref_count(buf) == 1) + } + + @Test("CustomBlockSource addresses the packet payload, not buf->data, when they differ (Matroska packets sit at +4)") + func customBlockSourceHonoursPacketDataOffset() throws { + let buf = try #require(av_buffer_alloc(256)) + var bufRef: UnsafeMutablePointer? = buf + defer { av_buffer_unref(&bufRef) } + + let payload = try #require(buf.pointee.data).advanced(by: 4) + let source = try #require(HardwareVideoDecoder.CustomBlockSource(buf: buf, data: payload)) + #expect(source.dataPtr == UnsafeMutableRawPointer(payload)) + #expect(source.dataPtr != UnsafeMutableRawPointer(buf.pointee.data!)) + source.releaseBuffer() + } + + @Test("CMBlockBuffer with CustomBlockSource holds av_buffer_ref and unrefs on deallocation") + func cmBlockBufferWithCustomBlockSourceLifecycle() throws { + guard let buf = av_buffer_alloc(512) else { + Issue.record("Failed to allocate AVBuffer") + return + } + var bufRef: UnsafeMutablePointer? = buf + defer { + if bufRef != nil { + av_buffer_unref(&bufRef) + } + } + + #expect(av_buffer_get_ref_count(buf) == 1) + + do { + let size = 512 + let blockSource = try #require(HardwareVideoDecoder.CustomBlockSource(buf: buf)) + let (blockBuffer, status) = HardwareVideoDecoder.makeBlockBuffer(wrapping: blockSource, size: size) + #expect(status == kCMBlockBufferNoErr) + #expect(blockBuffer != nil) + // Block buffer holds av_buffer_ref + #expect(av_buffer_get_ref_count(buf) == 2) + + // Verify memory block pointer is accessible and matches buf data + var lengthAtOffset = 0 + var totalLength = 0 + var dataPointer: UnsafeMutablePointer? + let accessStatus = CMBlockBufferGetDataPointer( + blockBuffer!, + atOffset: 0, + lengthAtOffsetOut: &lengthAtOffset, + totalLengthOut: &totalLength, + dataPointerOut: &dataPointer + ) + #expect(accessStatus == kCMBlockBufferNoErr) + #expect(totalLength == size) + #expect(dataPointer == UnsafeMutablePointer(OpaquePointer(buf.pointee.data))) + } + + // CMBlockBuffer went out of scope: FreeBlock ran takeRetainedValue and unref'd avBuf + #expect(av_buffer_get_ref_count(buf) == 1) + } + + @Test("CustomBlockSource cleanup on CMBlockBuffer creation failure balances retain") + func customBlockSourceCleanupOnCreationFailure() throws { + guard let buf = av_buffer_alloc(256) else { + Issue.record("Failed to allocate AVBuffer") + return + } + var bufRef: UnsafeMutablePointer? = buf + defer { + if bufRef != nil { + av_buffer_unref(&bufRef) + } + } + + #expect(av_buffer_get_ref_count(buf) == 1) + + let blockSource = try #require(HardwareVideoDecoder.CustomBlockSource(buf: buf)) + // A payload longer than the block is refused by CoreMedia, which is the failure the decoder's + // own wrapper has to clean up after. + let (blockBuffer, status) = HardwareVideoDecoder.makeBlockBuffer(wrapping: blockSource, size: 256, dataLength: 512) + #expect(status != kCMBlockBufferNoErr) + #expect(blockBuffer == nil) + + // Retain count of AVBuffer drops back to 1 and blockSource is deallocated without leaking + #expect(av_buffer_get_ref_count(buf) == 1) + } + + // MARK: - HardwareVideoDecoder.decode with CustomBlockSource vs Fallback Copy + + private static let hevcVPS: [UInt8] = [ + 0x40, 0x01, 0x0c, 0x01, 0xff, 0xff, 0x02, 0x20, 0x00, 0x00, 0x03, 0x00, + 0x90, 0x00, 0x00, 0x03, 0x00, 0x00, 0x03, 0x00, 0x78, 0x95, 0x94, 0x09 + ] + private static let hevcSPS: [UInt8] = [ + 0x42, 0x01, 0x01, 0x02, 0x20, 0x00, 0x00, 0x03, 0x00, 0x90, 0x00, 0x00, + 0x03, 0x00, 0x00, 0x03, 0x00, 0x78, 0xa0, 0x03, 0xc0, 0x80, 0x11, 0x07, + 0xca, 0xd9, 0x65, 0x65, 0x4a, 0x4c, 0x2f, 0x01, 0x6a, 0x12, 0x20, 0x12, + 0x08, 0x00, 0x00, 0x03, 0x00, 0x08, 0x00, 0x00, 0x03, 0x00, 0xc0, 0x40 + ] + private static let hevcPPS: [UInt8] = [ + 0x44, 0x01, 0xc0, 0x73, 0xc1, 0x89 + ] + + @Test("CustomBlockSource failable initializer returns nil if data is nil") + func customBlockSourceFailableInit() throws { + guard let buf = av_buffer_alloc(64) else { + Issue.record("Failed to allocate AVBuffer") + return + } + var bufRef: UnsafeMutablePointer? = buf + defer { + if bufRef != nil { + av_buffer_unref(&bufRef) + } + } + + #expect(av_buffer_get_ref_count(buf) == 1) + + // Valid buffer succeeds + let validSource = HardwareVideoDecoder.CustomBlockSource(buf: buf) + #expect(validSource != nil) + validSource?.releaseBuffer() + #expect(av_buffer_get_ref_count(buf) == 1) + + // Buffer with nil data returns nil without leaking av_buffer_ref + let originalData = buf.pointee.data + buf.pointee.data = nil + let nilSource = HardwareVideoDecoder.CustomBlockSource(buf: buf) + #expect(nilSource == nil) + #expect(av_buffer_get_ref_count(buf) == 1) + buf.pointee.data = originalData + } + + @Test("HardwareVideoDecoder.decode with useCustomBlockSource holds av_buffer_ref and decrements on release") + func hardwareVideoDecoderHoldsAndDecrementsBufferRef() throws { + let decoder = HardwareVideoDecoder() + do { + try decoder.open( + parameterSets: [Self.hevcVPS, Self.hevcSPS, Self.hevcPPS], + timeBase: AVRational(num: 1, den: 90000), + onFrame: { _, _, _ in } + ) + } catch { + // Hardware HEVC decompression session might not be supported in some environments: + // reported as skipped, not passed. + try Test.cancel("hardware HEVC session creation failed: \(error)") + } + defer { decoder.close() } + + decoder.useCustomBlockSource = true + + guard let avBuf = av_buffer_alloc(64) else { + Issue.record("Failed to allocate AVBuffer") + return + } + var avBufRef: UnsafeMutablePointer? = avBuf + defer { + if avBufRef != nil { + av_buffer_unref(&avBufRef) + } + } + + // Initialize with a 4-byte length-prefixed NAL (SEI prefix) + let dataPtr = avBuf.pointee.data! + dataPtr[0] = 0; dataPtr[1] = 0; dataPtr[2] = 0; dataPtr[3] = 60 + dataPtr[4] = 0x4E; dataPtr[5] = 0x01 + for i in 6..<64 { dataPtr[i] = 0x80 } + + #expect(av_buffer_get_ref_count(avBuf) == 1) + + guard let packet = trackedPacketAlloc() else { + Issue.record("Failed to allocate AVPacket") + return + } + defer { + var p: UnsafeMutablePointer? = packet + trackedPacketFree(&p) + } + packet.pointee.buf = av_buffer_ref(avBuf) + packet.pointee.data = avBuf.pointee.data + packet.pointee.size = 64 + packet.pointee.pts = 0 + packet.pointee.dts = 0 + + #expect(av_buffer_get_ref_count(avBuf) == 2) + + var refCountWhileHeld = 0 + var heldBlockBuffer: CMBlockBuffer? + decoder.onBlockBufferAllocatedForTesting = { bb in + refCountWhileHeld = Int(av_buffer_get_ref_count(avBuf)) + heldBlockBuffer = bb + } + + decoder.decode(packet: packet) + + // CustomBlockSource took an av_buffer_ref when wrapping into CMBlockBuffer + #expect(refCountWhileHeld >= 3) + #expect(heldBlockBuffer != nil) + + // Release caller's packet reference + av_buffer_unref(&packet.pointee.buf) + // Still held by avBufRef and heldBlockBuffer's CustomBlockSource + #expect(av_buffer_get_ref_count(avBuf) == 2) + + // Releasing the block buffer decrements the ref count back to 1 + heldBlockBuffer = nil + #expect(av_buffer_get_ref_count(avBuf) == 1) + + // Flush and close decoder to release asynchronous frames and invalidate session + decoder.flush() + decoder.close() + + #expect(av_buffer_get_ref_count(avBuf) == 1) + } + + @Test("HardwareVideoDecoder.decode falls back cleanly to copy when buf == nil") + func hardwareVideoDecoderFallsBackToCopyWhenBufNil() throws { + let decoder = HardwareVideoDecoder() + do { + try decoder.open( + parameterSets: [Self.hevcVPS, Self.hevcSPS, Self.hevcPPS], + timeBase: AVRational(num: 1, den: 90000), + onFrame: { _, _, _ in } + ) + } catch { + try Test.cancel("hardware HEVC session creation failed: \(error)") + } + defer { decoder.close() } + + decoder.useCustomBlockSource = true + + let rawData = UnsafeMutablePointer.allocate(capacity: 64) + defer { rawData.deallocate() } + rawData.initialize(repeating: 0x80, count: 64) + rawData[0] = 0; rawData[1] = 0; rawData[2] = 0; rawData[3] = 60 + rawData[4] = 0x4E; rawData[5] = 0x01 + + guard let packet = trackedPacketAlloc() else { + Issue.record("Failed to allocate AVPacket") + return + } + defer { + var p: UnsafeMutablePointer? = packet + trackedPacketFree(&p) + } + packet.pointee.buf = nil // Force fallback to copy path + packet.pointee.data = rawData + packet.pointee.size = 64 + packet.pointee.pts = 0 + packet.pointee.dts = 0 + + // Must decode without crashing or throwing + decoder.decode(packet: packet) + + decoder.flush() + decoder.close() + } + + // MARK: - DualLayerVideoDecoder Untouched Packet Delivery + + private static func makeAvccPacket(_ nals: [[UInt8]], pts: Int64 = 0) -> UnsafeMutablePointer? { + var total = 0 + for nal in nals { total += 4 + nal.count } + guard let avBuf = av_buffer_alloc(total + Int(AV_INPUT_BUFFER_PADDING_SIZE)) else { return nil } + guard let dst = avBuf.pointee.data else { + var b: UnsafeMutablePointer? = avBuf + av_buffer_unref(&b) + return nil + } + var w = 0 + for nal in nals { + let n = nal.count + dst[w + 0] = UInt8((n >> 24) & 0xFF) + dst[w + 1] = UInt8((n >> 16) & 0xFF) + dst[w + 2] = UInt8((n >> 8) & 0xFF) + dst[w + 3] = UInt8(n & 0xFF) + w += 4 + nal.withUnsafeBytes { raw in + if let base = raw.baseAddress { + memcpy(dst + w, base, n) + } + } + w += n + } + memset(dst + total, 0, Int(AV_INPUT_BUFFER_PADDING_SIZE)) + guard let pkt = trackedPacketAlloc() else { + var b: UnsafeMutablePointer? = avBuf + av_buffer_unref(&b) + return nil + } + pkt.pointee.buf = avBuf + pkt.pointee.data = dst + pkt.pointee.size = Int32(total) + pkt.pointee.pts = pts + pkt.pointee.dts = pts + return pkt + } + + private static func hevcNAL(type: UInt8, payload: [UInt8]) -> [UInt8] { + [UInt8(type << 1), 0x01] + payload + } + + private static func hexToBytes(_ hex: String) -> [UInt8] { + var bytes: [UInt8] = [] + var i = hex.startIndex + while i < hex.endIndex { + let next = hex.index(i, offsetBy: 2) + if let byte = UInt8(hex[i.. [UInt8] { + let felBytes = Self.hexToBytes(DoviFelMathTests.clipARpuHex) + guard let rpu = dovi_parse_unspec62_nalu(felBytes, felBytes.count) else { return [] } + defer { dovi_rpu_free(rpu) } + // libdovi returns a handle even when parsing fails; an error means there is nothing to convert. + guard dovi_rpu_get_error(rpu) == nil else { return [] } + _ = dovi_convert_rpu_with_mode(rpu, 1) + guard let out = dovi_write_unspec62_nalu(rpu) else { return [] } + defer { dovi_data_free(out) } + return [UInt8](UnsafeBufferPointer(start: out.pointee.data, count: out.pointee.len)) + } + + @Test("DualLayerVideoDecoder leaves FEL packet untouched and extracts EL and RPU") + func dualLayerVideoDecoderLeavesFELPacketUntouched() throws { + let primary = CountingFakeDecoder() + let enhancement = CountingFakeDecoder() + let decoder = DualLayerVideoDecoder( + composer: ImmediateFakeComposer(), + primaryDecoder: primary, + enhancementDecoder: enhancement + ) + defer { decoder.close() } + + let vps = Self.hevcNAL(type: 32, payload: [0x0c, 0x01, 0xff, 0xff]) + let sps = Self.hevcNAL(type: 33, payload: [0x01, 0x02, 0x20, 0x00]) + let pps = Self.hevcNAL(type: 34, payload: [0xc0, 0x73]) + let bl = Self.hevcNAL(type: 1, payload: [0x11, 0x22, 0x33, 0x44]) + let rpu = Self.hexToBytes(DoviFelMathTests.clipARpuHex) + let el = Self.hevcNAL(type: 63, payload: [0x55, 0x66, 0x77, 0x88]) + + guard let pkt = Self.makeAvccPacket([vps, sps, pps, bl, rpu, el], pts: 1000) else { + Issue.record("Failed to build AVCC packet") + return + } + defer { + var p: UnsafeMutablePointer? = pkt + trackedPacketFree(&p) + } + + let originalData = pkt.pointee.data + let originalSize = pkt.pointee.size + let originalBuf = pkt.pointee.buf + + decoder.decode(packet: pkt) + + // Packet pointer, size, and buffer must be COMPLETELY UNTOUCHED + #expect(pkt.pointee.data == originalData) + #expect(pkt.pointee.size == originalSize) + #expect(pkt.pointee.buf == originalBuf) + + // Classified as FEL + #expect(decoder.streamClassificationForTesting == .fel) + + // Primary received the base layer + #expect(primary.decodeCount == 1) + + // Enhancement decoder was engaged and received the extracted EL packet + #expect(enhancement.decodeCount == 1) + + // Display metadata RPU was extracted and cached + #expect(decoder.stateSnapshotForTesting.dmRpuCacheCount == 1) + // RPU params were extracted and cached + #expect(decoder.stateSnapshotForTesting.rpuCacheCount == 1) + } + + @Test("DualLayerVideoDecoder leaves MEL packet untouched and extracts RPU without engaging EL decoder") + func dualLayerVideoDecoderLeavesMELPacketUntouched() throws { + let primary = CountingFakeDecoder() + let enhancement = CountingFakeDecoder() + let decoder = DualLayerVideoDecoder( + composer: ImmediateFakeComposer(), + primaryDecoder: primary, + enhancementDecoder: enhancement + ) + defer { decoder.close() } + + let vps = Self.hevcNAL(type: 32, payload: [0x0c, 0x01, 0xff, 0xff]) + let sps = Self.hevcNAL(type: 33, payload: [0x01, 0x02, 0x20, 0x00]) + let pps = Self.hevcNAL(type: 34, payload: [0xc0, 0x73]) + let bl = Self.hevcNAL(type: 1, payload: [0x11, 0x22, 0x33, 0x44]) + let melRpu = Self.makeMelRpuBytes() + #expect(!melRpu.isEmpty) + let el = Self.hevcNAL(type: 63, payload: [0x55, 0x66]) + + guard let pkt = Self.makeAvccPacket([vps, sps, pps, bl, melRpu, el], pts: 2000) else { + Issue.record("Failed to build AVCC packet") + return + } + defer { + var p: UnsafeMutablePointer? = pkt + trackedPacketFree(&p) + } + + let originalData = pkt.pointee.data + let originalSize = pkt.pointee.size + let originalBuf = pkt.pointee.buf + + decoder.decode(packet: pkt) + + // Base-layer packet pointer, size, and buffer must be COMPLETELY UNTOUCHED + #expect(pkt.pointee.data == originalData) + #expect(pkt.pointee.size == originalSize) + #expect(pkt.pointee.buf == originalBuf) + + // Classified as MEL + #expect(decoder.streamClassificationForTesting == .mel) + + // Primary received the base layer + #expect(primary.decodeCount == 1) + + // Enhancement decoder was NOT engaged for MEL + #expect(enhancement.decodeCount == 0) + + // Display metadata RPU was extracted and cached + #expect(decoder.stateSnapshotForTesting.dmRpuCacheCount == 1) + } + + // MARK: - Per-Frame Logging & 1 Hz Summary + + @Test("per-frame logs (QUEUE, Bypass, DROP) are NOT emitted at default info level") + func fel14PerFrameLogsNotEmittedAtDefaultInfoLevel() throws { + let prevHandler = EngineLog.handler + defer { EngineLog.handler = prevHandler } + + var emittedLines: [String] = [] + let logLock = NSLock() + EngineLog.handler = { line in + logLock.lock() + emittedLines.append(line) + logLock.unlock() + } + + let decoder = DualLayerVideoDecoder(composer: ImmediateFakeComposer()) + defer { decoder.close() } + + // Send a base-layer frame with no EL / no RPU (will bypass) + let buf1 = try #require(Self.baseLayer()) + decoder.handleBaseLayerFrame(baseBuffer: buf1, pts: Self.pts(1), hdr10Data: nil) + + // Simulate FEL stream with EL decoder active: + decoder.primeForTesting(enhancementLayer: true) + + // Send unmatched BL (will queue BL) + let buf2 = try #require(Self.baseLayer()) + decoder.handleBaseLayerFrame(baseBuffer: buf2, pts: Self.pts(2), hdr10Data: nil) + + // Send unmatched EL (will queue EL) + let buf3 = try #require(Self.enhancementLayer()) + decoder.handleEnhancementLayerFrame(elBuffer: buf3, pts: Self.pts(3)) + + logLock.lock() + let captured = emittedLines + logLock.unlock() + + let forbiddenPrefixes = [ + "[DualLayer] QUEUE BL", + "[DualLayer] QUEUE EL", + "[DualLayer] Bypass:", + "[DualLayer] DROP stale EL", + "[DualLayer] PRUNED EL", + "[DualLayer] BYPASS no-EL" + ] + for prefix in forbiddenPrefixes { + let found = captured.filter { $0.contains(prefix) } + #expect(found.isEmpty, "Found forbidden per-frame log at info level: \(found)") + } + } + + @Test("1 Hz summary aggregates and emits expected fields, then resets interval") + func fel14OneHzSummaryAggregationAndFormat() throws { + let prevHandler = EngineLog.handler + defer { EngineLog.handler = prevHandler } + + var summaryLines: [String] = [] + let logLock = NSLock() + EngineLog.handler = { line in + if line.contains("[DualLayer] summary:") { + logLock.lock() + summaryLines.append(line) + logLock.unlock() + } + } + + let composer = ImmediateFakeComposer() + let decoder = DualLayerVideoDecoder(composer: composer) + defer { decoder.close() } + + var currentTime: TimeInterval = 10.0 + decoder.summaryTimestampProviderForTesting = { currentTime } + + decoder.primeForTesting(enhancementLayer: true) + + let baseBuf = try #require(Self.baseLayer()) + let elBuf = try #require(Self.enhancementLayer()) + + // Inject cached RPU and pair a frame at t = 10.0 + decoder.seedRPUForTesting(pts: Self.pts(1), params: Self.params(), displayMetadataNAL: nil) + + // Base layer arrives at t = 10.0: sets lastSummaryTimestamp = 10.0 + decoder.handleBaseLayerFrame(baseBuffer: baseBuf, pts: Self.pts(1), hdr10Data: nil) + + // Enhancement layer arrives at t = 10.0: pairs and composes + decoder.handleEnhancementLayerFrame(elBuffer: elBuf, pts: Self.pts(1)) + + // Advance by 0.5s to 10.5: interval (1.0s) has not elapsed yet + currentTime = 10.5 + decoder.maybeEmitSummary() + logLock.lock() + #expect(summaryLines.isEmpty) + logLock.unlock() + + // Advance by 1.1s to 11.1 (currentTime >= 10.0 + 1.0): summary emits! + currentTime = 11.1 + decoder.maybeEmitSummary() + logLock.lock() + #expect(summaryLines.count == 1) + let summary = summaryLines.first ?? "" + logLock.unlock() + + #expect(summary.contains("[DualLayer] summary:")) + #expect(summary.contains("composed=1")) + #expect(summary.contains("bypassNoEL=0")) + #expect(summary.contains("bypassNoRPU=0")) + #expect(summary.contains("staleELDropped=0")) + #expect(summary.contains("peakGpuSlots=")) + #expect(summary.contains("poolBuffers=")) + #expect(summary.contains("deliveryPendingPeak=")) + #expect(summary.contains("meanGpuMs=")) + + // Next call immediately at 11.2 does not emit again until another second passes + currentTime = 11.2 + decoder.maybeEmitSummary() + logLock.lock() + #expect(summaryLines.count == 1) + logLock.unlock() + + // After another 1.0s at 12.2, emits interval 2 with composed=0 + currentTime = 12.2 + decoder.maybeEmitSummary() + logLock.lock() + #expect(summaryLines.count == 2) + let summary2 = summaryLines.last ?? "" + logLock.unlock() + #expect(summary2.contains("composed=0")) + } + + @Test("flush and close reset 1 Hz summary counters") + func fel14FlushAndCloseResetSummaryCounters() throws { + let prevHandler = EngineLog.handler + defer { EngineLog.handler = prevHandler } + + var summaryLines: [String] = [] + let logLock = NSLock() + EngineLog.handler = { line in + if line.contains("[DualLayer] summary:") { + logLock.lock() + summaryLines.append(line) + logLock.unlock() + } + } + + let decoder = DualLayerVideoDecoder(composer: ImmediateFakeComposer()) + defer { decoder.close() } + + var currentTime: TimeInterval = 20.0 + decoder.summaryTimestampProviderForTesting = { currentTime } + decoder.primeForTesting(enhancementLayer: true) + + let baseBuf = try #require(Self.baseLayer()) + let elBuf = try #require(Self.enhancementLayer()) + decoder.seedRPUForTesting(pts: Self.pts(1), params: Self.params(), displayMetadataNAL: nil) + decoder.handleBaseLayerFrame(baseBuffer: baseBuf, pts: Self.pts(1), hdr10Data: nil) + decoder.handleEnhancementLayerFrame(elBuffer: elBuf, pts: Self.pts(1)) + + // Flush resets counters and timestamp + decoder.flush() + + // Advance time to 21.5 and call maybeEmitSummary: since flush reset lastSummaryTimestamp to 0, + // this call merely records the new baseline and does NOT emit + currentTime = 21.5 + decoder.maybeEmitSummary() + + logLock.lock() + #expect(summaryLines.isEmpty) + logLock.unlock() + + // Advance by 1.1s to 22.6: summary emits with 0 composed frames + currentTime = 22.6 + decoder.maybeEmitSummary() + + logLock.lock() + #expect(summaryLines.count == 1) + let summary = summaryLines.first ?? "" + logLock.unlock() + #expect(summary.contains("composed=0")) + } + + // MARK: - Memory Limits & Queue Bounds + + @Test("output pool enforces its allocation threshold") + func fel18OutputPoolEnforcesAllocationThreshold() throws { + guard let composer = DoviFelComposer() else { + Issue.record("Failed to create DoviFelComposer") + return + } + guard let pool = composer.ensurePoolForTesting(width: 256, height: 256) else { + Issue.record("Failed to create pixel buffer pool") + return + } + + var heldBuffers: [CVPixelBuffer] = [] + // Allocate up to the threshold + for _ in 0..? = melPkt + trackedPacketFree(&optMel) + + // Classification must be MEL, not FEL + #expect(decoder.streamClassification == .mel) + + // PacketBalanceTracker must return to baseline + #expect(PacketBalanceTracker.alive == aliveBefore) + } +} diff --git a/Tests/AetherEngineTests/SubtitleFrameCompositorTests.swift b/Tests/AetherEngineTests/SubtitleFrameCompositorTests.swift index ac6bf6a8e..f2f3ae95f 100644 --- a/Tests/AetherEngineTests/SubtitleFrameCompositorTests.swift +++ b/Tests/AetherEngineTests/SubtitleFrameCompositorTests.swift @@ -203,6 +203,39 @@ struct SubtitleFrameCompositorTests { } } + /// The Profile 7 FEL route hands the renderer composed P010 frames carrying the Dolby Vision RPU + /// and compatibility ID. In PiP with a cue up, the frame the layer receives is the compositor's + /// copy: it has to carry the source frame's attachments (the CFString "1" among them, which the + /// render server parses with `CFStringGetIntValue`) and must not inherit a previous frame's RPU + /// from the recycled pool buffer, which is the stale-trims failure the design doc describes. + @Test("a composited Dolby Vision frame carries its own RPU attachments and a recycled buffer drops stale ones") + func compositedFrameCarriesDolbyVisionAttachments() throws { + let compositor = SubtitleFrameCompositor() + compositor.update(cues: [SubtitleCue(id: 1, startTime: 0, endTime: 10, body: .text("HELLO"))], enabled: true, delaySeconds: 0) + let format = kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange + + let rpuNAL: [UInt8] = [0x7C, 0x01, 0x19, 0x08, 0x42, 0xA5, 0x80] + let dv = try blackFrame(format) + DoviDisplayMetadata.attach(rpuNAL: rpuNAL, to: dv) + let sourcePayload = try #require(attachment(dv, DoviDisplayMetadata.rpuDataKey as CFString) as? Data) + + // Several rounds, so the pool hands out a buffer that already held an RPU. + for _ in 0..<4 { + let out = compositor.composite(dv, ptsSeconds: 5) + #expect(out !== dv) + #expect(attachment(out, DoviDisplayMetadata.rpuDataKey as CFString) as? Data == sourcePayload) + #expect(attachment(out, DoviDisplayMetadata.compatibilityIDKey as CFString) as? String == "1") + } + + // A frame with no RPU (a bypass without display metadata) must not pick one up. + let plain = try blackFrame(format) + for _ in 0..<4 { + let out = compositor.composite(plain, ptsSeconds: 5) + #expect(attachment(out, DoviDisplayMetadata.rpuDataKey as CFString) == nil) + #expect(attachment(out, DoviDisplayMetadata.compatibilityIDKey as CFString) == nil) + } + } + @Test("an untagged source renders through BT.709") func untaggedSourceFallsBackTo709() throws { let space = SubtitleFrameCompositor.renderColorSpace(for: try blackFrame(kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange)) diff --git a/docs/architecture.md b/docs/architecture.md index 3746316d2..297cfe309 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -414,6 +414,7 @@ Sources/AetherEngine/ │ ├── InterlaceProbe.swift Decodes a sample and applies the deinterlacer's own engagement predicate, so a declared field order that no frame backs up (PsF) stops costing a SW decode (#232) │ ├── VideoRoutingPolicy.swift Pure codec-and-field-order dispatch rule: AV1 gated on HW, VP9/VP8/MPEG4/MPEG2/VC1 always SW, interlaced H.264 SW so bwdif can deinterlace (#107, verified against decoded frames on VOD, #232), plus a second-stage gate routing H.264 High 4:2:2/4:4:4/10 + HEVC Rext to SW where VideoToolbox has no HW decoder (#2) │ ├── HardwareVideoDecoder.swift SW path: VideoToolbox HW HEVC / AV1 decoder for sources routed away from AVPlayer +│ ├── DualLayerVideoDecoder.swift SW path: Dolby Vision Profile 7 FEL. Two HardwareVideoDecoders (a base layer, typically 4K, and an enhancement layer, typically 1080p) paired by HEVC picture order with the packet's RPU, composed on Metal by `DoviFelComposer`, delivered in decode order with the Profile 8.1 RPU attached per frame; see [dolby-vision-fel.md](dolby-vision-fel.md) │ ├── SoftwareVideoDecoder.swift SW path: libavcodec/dav1d → CVPixelBuffer (NV12 / P010), HDR10+ side data │ ├── SubtitleDecoder.swift Sidecar URL one-shot decode (text only); decodes several streams of one container in a single pass (#266) │ ├── VideoDecoderTypes.swift DecodedFrameHandler typealias + VideoDecoderError From f31b67c61e7528e76791535e68cc4044cf2f7976 Mon Sep 17 00:00:00 2001 From: Pavel Kajaba <15980400+pkajaba@users.noreply.github.com> Date: Wed, 7 Oct 2026 10:36:42 +0100 Subject: [PATCH 015/132] feat(dovi): route Profile 7 FEL to the dual-layer decoder behind preferDualLayerDolbyVision Host wiring for the composer and decoder. LoadOptions.preferDualLayerDolbyVision (default false) makes a Profile 7 FEL file play through SoftwarePlaybackHost and DualLayerVideoDecoder when the box can afford it; otherwise the existing Profile 8.1 conversion is untouched. With the flag off no source plays differently. - DualLayerRouteDecision: a pure capability gate. The route is taken only for FEL, with a proven hardware HEVC session, a seekable VOD, dolbyVisionHandling .automatic and a stream inside the real-time budget ((3840x2160 + 1920x1080) x 30 = 311.04 Mpx/s; a 24p UHD FEL with a quarter-area EL needs 80%; an unknown frame rate is budgeted as 24). Every Profile 7 load logs exactly one `DV P7 dual-layer route: taken|not taken ()` line. It fails closed: when the base-layer size is unknown the budget cannot be evaluated, so the route is refused rather than assumed. The budget is an assumption until the Apple TV run confirms it. - Wiring in AetherEngine, SoftwarePlaybackHost, VideoRoutingPolicy, CodecRoutePolicy and the reload path (preferDualLayerDolbyVision is a tuning field, correctable through reloadAtCurrentPosition(applying:)). An `hev1` source with in-band parameter sets (an empty hvcC) is completed from the head of the stream so it can be classified and take the route, the same rebuild the native path already does for its init.mp4. - Audio trade-off, documented in LoadOptions, api.md and dolby-vision-fel.md and pinned by DualLayerRouteAudioTests: the route runs on the software host, whose audio FFmpeg decodes and the sample-buffer audio renderer plays, and on tvOS only AVPlayer emits Atmos objects. An E-AC-3 (JOC) track therefore plays as its 5.1 / 7.1 channel-bed core, without the objects, where the native route would hand it to AVPlayer unchanged. Not engineered around. - Docs: dolby-vision-fel.md (the design), api.md, formats.md, README, and the DocumentedConstantsTests pins that tie the document's numbers to the code (route budget, pipeline bounds, picture-order pairing bound). The document arrives here, whole; code comments in the earlier PRs cite its section numbers. Mac soak, measured BEFORE the output-pool fix of the decoder PR (peaks are pre-fix; with it 12 seeks peak at 500 MB instead of 674 MB), M2 Pro, debug aetherctl, --window: | Run | Route | Composed / failures | Seeks landed | Drops | Steady | Peak | |---|---|---|---|---|---|---| | 217 s, 12 seeks | taken, budget 80% | 6143 / 0 | 15 | 6 / 6206 | 328-331 MB | 674 MB | | 120 s, 11 seeks | taken | 4248 / 0 | 12 | 7 / 4275 | 282 MB | 600 MB | | pause + seek while paused | taken | 2005 / 0 | 1 | 1 / 1949 | 330 MB | 352 MB | Apple TV results: pending, nothing measured on device yet. Tests: DualLayerRouteDecisionTests (one per exit), VideoRoutingPolicyTests, SoftwareDecoderSelectionTests, InBandParameterSetRebuildTests, DualLayerEdgeCaseTests, DualLayerRouteAudioTests, LoadOptionsTests, DocumentedConstantsTests. Not in this PR: the pulse probe and the aetherctl tooling. --- CHANGELOG.md | 7 + README.md | 6 +- .../AetherEngine/AetherEngine+Loading.swift | 10 +- .../AetherEngine+ReloadWithOptions.swift | 55 +- Sources/AetherEngine/AetherEngine.swift | 100 ++- .../Decoder/DualLayerRouteDecision.swift | 199 ++++++ .../Decoder/VideoRoutingPolicy.swift | 39 +- .../AetherEngine/Diagnostics/EngineLog.swift | 12 +- .../Native/SoftwarePlaybackHost.swift | 151 +++- .../Native/SoftwareReadAdmission.swift | 5 + Sources/AetherEngine/PlayerState.swift | 40 +- .../AetherEngine/Video/CodecRoutePolicy.swift | 6 +- .../Video/DolbyVisionRecordAudit.swift | 39 ++ .../HLSVideoEngine+SegmentPlanning.swift | 50 +- .../Video/VTCapabilityProbe.swift | 7 + .../Video/VideoConfigRecord+InBand.swift | 140 ++++ .../DocumentedConstantsTests.swift | 64 ++ .../DualLayerEdgeCaseTests.swift | 633 +++++++++++++++++ .../DualLayerRouteAudioTests.swift | 56 ++ .../DualLayerRouteDecisionTests.swift | 213 ++++++ .../InBandParameterSetRebuildTests.swift | 56 ++ .../AetherEngineTests/LoadOptionsTests.swift | 2 + .../SoftwareDecoderSelectionTests.swift | 172 +++++ .../VideoRoutingPolicyTests.swift | 248 +++++++ docs/api.md | 4 +- docs/dolby-vision-fel.md | 658 ++++++++++++++++++ docs/formats.md | 6 +- 27 files changed, 2907 insertions(+), 71 deletions(-) create mode 100644 Sources/AetherEngine/Decoder/DualLayerRouteDecision.swift create mode 100644 Sources/AetherEngine/Video/VideoConfigRecord+InBand.swift create mode 100644 Tests/AetherEngineTests/DualLayerEdgeCaseTests.swift create mode 100644 Tests/AetherEngineTests/DualLayerRouteAudioTests.swift create mode 100644 Tests/AetherEngineTests/DualLayerRouteDecisionTests.swift create mode 100644 Tests/AetherEngineTests/SoftwareDecoderSelectionTests.swift create mode 100644 docs/dolby-vision-fel.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 37b879c40..269c12da4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,9 +12,16 @@ the public-API contract. ### Added +- **Dolby Vision Profile 7 FEL plays as Dolby Vision on the software path.** Opt-in: with `LoadOptions.preferDualLayerDolbyVision` set to `true` (default `false`) and `dolbyVisionHandling` `.automatic`, `SoftwarePlaybackHost` routes a Profile 7 file whose enhancement layer is FEL through `DualLayerVideoDecoder`: the base layer and the enhancement layer (typically 4K and 1080p) decode on VideoToolbox, `DoviFelComposer` rebuilds the 12-bit picture on Metal, and each frame carries its RPU converted to Profile 8.1 (`DolbyVisionRPUData`, `DolbyCompatibilityID`, the latter a `CFString` `"1"`: tvOS parses it with `CFStringGetIntValue` and a `CFNumber` aborts the render server). A MEL enhancement layer keeps the native route. An `hev1` source whose `hvcC` carries no VPS / SPS / PPS (the usual Dolby Vision MP4 recipes) is classified from the parameter sets at the head of the stream, the rebuild the native path already does for its init.mp4 (#19). What the route gives up against the native one (system Picture in Picture, the I-frame playlist's scrub thumbnails, WebVTT renditions, AirPlay picture) is listed in the document. Details in [docs/dolby-vision-fel.md](docs/dolby-vision-fel.md). +- **The Profile 7 dual-layer route is FEL only and capability gated, and says why.** `DualLayerRouteDecision` adds a real-time budget to the route: the base and enhancement layers together may need at most UHD 3840x2160 at 30 fps plus a 1920x1080 enhancement layer at 30 fps (a 24p UHD master uses 80%), otherwise the file keeps the native route, as MEL does and as does a box without hardware HEVC decode (8.1 where the display takes Dolby Vision, the HDR10 base layer where it does not). Every Profile 7 load logs one `DV P7 dual-layer route: taken (...)` or `not taken ()` line; the line is new, and with the preference off it is the only change. - **`AETHER_FEL_MEASURE` reports the signed FEL-minus-MEL mean per plane and luma band.** The diagnostic composes every Nth frame a second time with the residual off and compares the two; the signed mean says whether the residual brightens or darkens, which the absolute difference could not. It runs one sample at a time and skips a frame that arrives while one is in flight. Off by default; it is an environment-variable diagnostic, not API. - **The dual-layer decoder pairs the two layers by picture order.** `DualLayerPictureAlignment` derives each layer's display slot from its POC (`HEVCPictureOrderCount`, no decoding) and stamps each enhancement-layer packet with the time of the base-layer picture in the same slot, so a stream whose enhancement layer is coded with B-frames over a base layer without them pairs the right pictures. A stream whose layers share one GOP keeps every packet's own time. After a seek a layer pairs by access unit until its next sequence-opening IRAP. +### Changed + +- **`.verbose` engine log lines cost nothing unless the category's debug level is enabled.** `EngineLog` redacted and handed every `.verbose` line to OSLog even when nobody was reading debug output; it now checks `OSLog.isEnabled(type: .debug)` for the category first and returns early. `.info` lines and the host `handler` are unchanged (verbose lines never reached the handler). +- **`reloadAtCurrentPosition(applying:)` refuses a `preferDualLayerDolbyVision` change, or a `dolbyVisionHandling` change while the preference is on, on a custom `IOReader` source playing Profile 7**, with the new public case `SessionReloadRefusal.dualLayerRouteNeedsFreshLoad`, before any teardown. A host that switches over `SessionReloadRefusal` exhaustively has to handle the new case. The in-place rebuild of a custom source keeps the route the session was loaded with, so the change would have been logged as applied and ignored; `load()` again with the new option. URL sources re-run `load()` and are unaffected. + ### Fixed - **A live GOP longer than the sealed target duration no longer breaks the playlist.** RFC 8216 keeps diff --git a/README.md b/README.md index 148284e12..89249b306 100644 --- a/README.md +++ b/README.md @@ -65,9 +65,9 @@ A scannable summary; the depth for each row lives in **[docs/formats.md](docs/fo | Disc | DVD-Video and Blu-ray ISO (decrypted): selectable titles and chapters, demuxed through the normal path | | Video (HW) | H.264, HEVC, HEVC Main10 via VideoToolbox; AV1 Main profile where HW AV1 exists | | Video (SW) | AV1 (dav1d) without HW, and AV1 High / Professional (4:4:4, 4:2:2, 12-bit) everywhere, VP9 / VP8, MPEG-4 Part 2 / MPEG-2 / VC-1, QuickTime RLE, the Flash tail (Sorenson Spark, On2 VP6) and anything else the FFmpeg build carries a decoder for (software is the default route; only HEVC, H.264 and HW-decodable AV1 go native), H.264 High 4:2:2 / 4:4:4 / 10 and HEVC Rext where VideoToolbox has no HW decoder (Intel Macs, older chips), interlaced H.264 (AVPlayer does not deinterlace; on VOD the declared field order is verified against decoded frames, so progressive-in-interlaced-carriage keeps hardware decode); GPU deinterlace (yadif_videotoolbox, Metal, field-rate by default) with a CPU bwdif fallback | -| HDR | HDR10, HDR10+ (per-frame ST 2094-40), Dolby Vision (P5, P7 as single-layer 8.1, P8.1, P8.4, AV1 P10.x), HLG, HDR Vivid (played on its HLG / PQ base, CUVA metadata reported by `probe`) | +| HDR | HDR10, HDR10+ (per-frame ST 2094-40), Dolby Vision (P5, P7 on the native route as single-layer 8.1 (HDR10 base layer on a display without Dolby Vision), or composed dual-layer on Metal with per-frame DV metadata for FEL when `preferDualLayerDolbyVision` is set and VideoToolbox decodes HEVC in hardware, P8.1, P8.4, AV1 P10.x), HLG, HDR Vivid (played on its HLG / PQ base, CUVA metadata reported by `probe`) | | Audio | AAC, AC3, EAC3, FLAC, ALAC stream-copy lossless; TrueHD / MLP / DTS / DTS-HD MA / MP3 / MP2 / Opus / Vorbis / LPCM (incl. Blu-ray, G.711) / WMA (Standard, Pro, Lossless, Voice) / Nellymoser / ADPCM-SWF / Speex bridge to EAC3 5.1 (default) or lossless FLAC | -| Dolby Atmos | EAC3+JOC stream-copied on every route (HDMI MAT 2.0, AirPods spatial, BT downmix). No container reliably declares JOC pre-decode, so an honest `TrackInfo.isAtmos` needs a bounded decode: `AetherEngine.probeDetectingAtmos(url:/source:)` answers for a details screen without starting playback, and `LoadOptions.confirmAtmos` has the running session confirm its own tracks in the background and republish `audioTracks`. Both are opt-in and neither sits on the playback-start path. TrueHD Atmos keeps its heights too, opt-in through `LoadOptions.objectAudioRendering`: the engine decodes the Atmos objects, renders them into a speaker bed and delivers Apple Positional Audio, which tvOS sends to an Atmos receiver as Dolby MAT and iOS plays as Spatial Audio ([details](docs/formats.md#truehd-atmos-with-heights)). DTS:X takes the same route, its heights and objects unfolded from the compatible 7.1 ([details](docs/formats.md#dtsx-with-heights)) | +| Dolby Atmos | EAC3+JOC stream-copied on every route (HDMI MAT 2.0, AirPods spatial, BT downmix) except the opt-in Profile 7 FEL dual-layer route (`preferDualLayerDolbyVision`), which runs on the software host and plays an E-AC-3 JOC track as its 5.1 / 7.1 bed without the objects. No container reliably declares JOC pre-decode, so an honest `TrackInfo.isAtmos` needs a bounded decode: `AetherEngine.probeDetectingAtmos(url:/source:)` answers for a details screen without starting playback, and `LoadOptions.confirmAtmos` has the running session confirm its own tracks in the background and republish `audioTracks`. Both are opt-in and neither sits on the playback-start path. TrueHD Atmos keeps its heights too, opt-in through `LoadOptions.objectAudioRendering`: the engine decodes the Atmos objects, renders them into a speaker bed and delivers Apple Positional Audio, which tvOS sends to an Atmos receiver as Dolby MAT and iOS plays as Spatial Audio ([details](docs/formats.md#truehd-atmos-with-heights)). DTS:X takes the same route, its heights and objects unfolded from the compatible 7.1 ([details](docs/formats.md#dtsx-with-heights)) | | Surround | 5.1 / 7.1 with correct `AudioChannelLayout` | | Audio-only | `LoadOptions.audioOnly`: lean pipeline, no video machinery, system Now-Playing on tvOS / iOS | | Background audio | Audio keeps playing when the app backgrounds on iOS: native AVPlayer stays alive, the software path drops video and keeps decoding audio (`backgroundPlaybackEnabled` / `pictureInPictureActive`); a paused session survives quick app switches for a grace window (`backgroundTeardownGraceSeconds`, default 15 s) before the wedge-safe teardown runs. tvOS tears down immediately (wedge-safe) unless a PiP window is active, which keeps the pipeline and loopback server alive; with PiP the software path keeps decoding video too (the window needs frames). Hosts gate corrective actions on the published `isSessionReady` | @@ -93,7 +93,7 @@ On Apple platforms the real choice is between AVPlayer, with deep OS integration | **Approach** | Embeddable engine, Apple-only | Full player with bundled UI, FFmpeg + AVPlayer, Apple-only | Apple's built-in player | libVLC wrapped for Apple | libmpv, cross-platform | | **Container & codec breadth** | Wide, FFmpeg demux | Wide, FFmpeg demux | Narrow, Apple's set | Wide | Wide | | **Hardware decode** | VideoToolbox, dav1d SW fallback | VideoToolbox, FFmpeg SW fallback | VideoToolbox | VideoToolbox plus software | VideoToolbox plus software | -| **Dolby Vision** | P5, P7 as 8.1, P8.1, P8.4, AV1 P10.x, real display switch | P5, P8 via AVPlayer; their matrix lists P5 HDR display without overheating as paid LGPL | P5 and P8.1 only | Tone-maps, no DV display | Tone-maps, no DV display | +| **Dolby Vision** | P5, P7 as 8.1 (or opt-in FEL dual-layer), P8.1, P8.4, AV1 P10.x, real display switch | P5, P8 via AVPlayer; their matrix lists P5 HDR display without overheating as paid LGPL | P5 and P8.1 only | Tone-maps, no DV display | Tone-maps, no DV display | | **Dolby Atmos** | EAC3+JOC stream-copied (HDMI MAT, spatial); TrueHD Atmos and DTS:X rendered to APAC, heights kept | EAC3+JOC via AVPlayer | EAC3+JOC passthrough | Decodes to PCM, no object passthrough | No Atmos passthrough on Apple | | **HDR on tvOS** | Native Match Content switch | Native Match Content on AVPlayer path, else Metal tone-map | Native Match Content | Software tone-mapping | Software tone-mapping | | **Rendering & UI** | OS-native, you ship SwiftUI | Own Metal renderer, bundled controls | OS-native, you ship UI | Own renderer, bundled controls | Own renderer, bundled OSC | diff --git a/Sources/AetherEngine/AetherEngine+Loading.swift b/Sources/AetherEngine/AetherEngine+Loading.swift index 077b2ced6..8ba0e2882 100644 --- a/Sources/AetherEngine/AetherEngine+Loading.swift +++ b/Sources/AetherEngine/AetherEngine+Loading.swift @@ -1935,7 +1935,8 @@ extension AetherEngine { isLive: Bool = false, dvrWindowSeconds: Double? = nil, preopenedDemuxer: Demuxer?, - generation: UInt64 + generation: UInt64, + elType: String? = nil ) async throws { let deinterlaceMode = loadedOptions.deinterlaceMode let waitStarted = DispatchTime.now() @@ -1967,6 +1968,10 @@ extension AetherEngine { // on every build. Without it a software host came up aspect-fit whatever the app had set, // and only a second write mid-session took effect. let host = SoftwarePlaybackHost(videoGravity: _videoGravity) + host.configuredDVProfile = sourceDVProfile + host.dolbyVisionHandling = loadedOptions.dolbyVisionHandling + host.preferDualLayerDolbyVision = loadedOptions.preferDualLayerDolbyVision + host.elType = elType ?? sourceElType host.setAudioDelay(loadedOptions.audioDelaySeconds) // AE#464: before the decoder opens host.deinterlaceConfig = DeinterlaceConfig( mode: loadedOptions.deinterlaceMode, @@ -2556,7 +2561,8 @@ extension AetherEngine { isLive: loadedOptions.isLive, dvrWindowSeconds: loadedOptions.dvrWindowSeconds, preopenedDemuxer: customPreopened, - generation: gen + generation: gen, + elType: sourceElType ) EngineLog.emit("[AetherEngine] reload: loadSoftware done (\(elapsedMs(since: loadStart))ms)", category: .engine) playbackBackend = .software diff --git a/Sources/AetherEngine/AetherEngine+ReloadWithOptions.swift b/Sources/AetherEngine/AetherEngine+ReloadWithOptions.swift index fe22ed3bf..2a6b213c9 100644 --- a/Sources/AetherEngine/AetherEngine+ReloadWithOptions.swift +++ b/Sources/AetherEngine/AetherEngine+ReloadWithOptions.swift @@ -115,6 +115,26 @@ extension AetherEngine { throw AetherEngineError.sessionNotReloadable(refusal) } + // The dual-layer route is chosen by `load`'s probe (el_type, geometry, budget, handling) and a custom + // source's rebuild never re-runs it: it keeps the backend and the el_type the session came + // up with. Decided here for the same reason as the decode-path escape above, before any + // teardown, so the flip is refused instead of being logged as applied and ignored. + if let refusal = SessionOptionCorrection.dualLayerRouteRefusal( + isCustomSource: isCustomSource, + current: loadedOptions, + proposed: proposed, + dvProfile: DolbyVisionRecordAudit.correctedProfile( + record: sourceDVProfile, rpu: sourceDolbyVisionRPUProfile) ?? sourceDVProfile + ) { + EngineLog.emit( + "[AetherEngine] DV P7 dual-layer: preferDualLayerDolbyVision / dolbyVisionHandling correction refused, " + + "\(refusal.rawValue): a custom source keeps the route it was loaded with; load() it " + + "again with the new option", + category: .engine + ) + throw AetherEngineError.sessionNotReloadable(refusal) + } + // Stating the no-op matters for the same reason #364's teletext switch states it: a host // that corrected an option and saw a plain reload cannot otherwise tell "the correction was // already in force" from "the correction did not arrive". @@ -246,6 +266,11 @@ public enum SessionReloadRefusal: String, Sendable, Equatable, CustomStringConve /// side-audio merge lives in the native path's segment producer, so the session would play /// silent. Same reachability as `softwarePathCannotRepresentSource`. case demuxedAudioLiveIsNativeOnly + /// A correction of `preferDualLayerDolbyVision`, or of `dolbyVisionHandling` while the preference is on, on a Profile 7 session that plays from a custom + /// `IOReader` source. The route is decided by `load()`'s probe, which a custom source's in-place + /// rebuild does not re-run, so the change could not take effect. Call `load()` again with the + /// new option. URL sources are not affected: their correction re-runs `load()`. + case dualLayerRouteNeedsFreshLoad public var description: String { switch self { @@ -255,6 +280,8 @@ public enum SessionReloadRefusal: String, Sendable, Equatable, CustomStringConve return "the software path cannot represent this source's Dolby Vision signal" case .demuxedAudioLiveIsNativeOnly: return "this live source's audio is delivered separately and merged on the native path only" + case .dualLayerRouteNeedsFreshLoad: + return "the dual-layer Dolby Vision route of a custom source is decided by load(), not by a rebuild" } } } @@ -360,6 +387,32 @@ enum SessionOptionCorrection { return nil } + /// Why a `preferDualLayerDolbyVision` or `dolbyVisionHandling` correction cannot be honoured, or nil + /// when it can. + /// + /// A URL source re-runs `load()` on a correction, which re-probes and re-decides the route. A + /// custom `IOReader` source rebuilds through `reloadWithAudioOverride`, which keeps the backend + /// and the el_type the session was loaded with, so the change would be logged as applied and move + /// nothing. The route needs the preference on and `dolbyVisionHandling` `.automatic` + /// (`DualLayerRouteDecision`), so a change of the preference always refuses, and a change of the + /// handling refuses when the preference is on, where it can turn the route on or off. With the + /// preference off the handling cannot move the route. Only a Profile 7 source can route + /// differently, so any other source is not refused: the change there moves nothing, which is also + /// what the rebuild does. + static func dualLayerRouteRefusal( + isCustomSource: Bool, + current: LoadOptions, + proposed: LoadOptions, + dvProfile: Int? + ) -> SessionReloadRefusal? { + guard isCustomSource, dvProfile == 7 else { return nil } + let preferenceChanged = proposed.preferDualLayerDolbyVision != current.preferDualLayerDolbyVision + let handlingMovesRoute = proposed.dolbyVisionHandling != current.dolbyVisionHandling + && proposed.preferDualLayerDolbyVision + guard preferenceChanged || handlingMovesRoute else { return nil } + return .dualLayerRouteNeedsFreshLoad + } + /// Every field the proposal changed, for the log line. Reflection is right here and wrong /// above: a diagnostic that names a new field the day it is added is worth more than one that /// cannot be wrong, and being wrong here costs a log line. @@ -390,7 +443,7 @@ enum SessionOptionCorrection { /// deliberately. Update this list and, if the field names the session, `loadIdentityFields`. static let knownFields: [String] = [ "omitCriteriaColorExtensions", "suppressDisplayCriteria", "httpHeaders", "nativeVideoFrameDiagnostics", "nativeVideoFrameRecovery", "hostRetriesExhaustedLiveReplay", - "keepDvh1TagWithoutDV", "forceDolbyVisionOnNonDVDisplay", "dolbyVisionHandling", "matchContentEnabled", + "keepDvh1TagWithoutDV", "forceDolbyVisionOnNonDVDisplay", "dolbyVisionHandling", "preferDualLayerDolbyVision", "matchContentEnabled", "panelIsInHDRMode", "attemptsHDRMasterOnUnprovenPanel", "panelPresentsDolbyVision", "audioBridgeMode", "objectAudioRendering", "isLive", "audioOnly", "dvrWindowSeconds", diff --git a/Sources/AetherEngine/AetherEngine.swift b/Sources/AetherEngine/AetherEngine.swift index 5a5629242..770cd160e 100644 --- a/Sources/AetherEngine/AetherEngine.swift +++ b/Sources/AetherEngine/AetherEngine.swift @@ -614,6 +614,9 @@ public final class AetherEngine: ObservableObject { /// Deriving it keeps the load's answer and the rebuild's the same value rather than two. var sessionDisplayEligibleForHDR: Bool { sessionObservedDisplayCaps?.supportsHDR ?? false } + /// Enhancement layer type for Profile 7 streams ("FEL" or "MEL") extracted from the first RPU. + var sourceElType: String? = nil + /// Whether the loaded source's Dolby Vision has a base layer `LoadOptions.dolbyVisionHandling = /// .baseLayerOnly` can present (`VideoRoutingPolicy.dolbyVisionBaseLayerIsPresentable`), decided /// on the probe stream for the same reason as `sourceDVBLCompatID`: a correction that turns the @@ -4271,6 +4274,7 @@ public final class AetherEngine: ObservableObject { sourceDVProfile = nil dolbyVisionConversion = nil sourceCarriesHDR10PlusMetadata = false + sourceElType = nil sourceDVBLCompatID = nil sourceDolbyVisionBaseLayerPresentable = false sourceDolbyVisionRPUProfile = nil @@ -5019,12 +5023,102 @@ public final class AetherEngine: ObservableObject { category: .engine ) } + // A Profile 7 source with in-band parameter sets (`hev1` with an empty hvcC) is unclassifiable + // to the strict verdict below and would stay native for that reason alone. When the dual-layer + // route is wanted, complete the probe's record from the head of the stream first; the session + // reuses this demuxer, so the software host and its decoders see the completed record too. + if probeOpened, probe.videoStreamIndex >= 0, !options.isLive, probe.isSourceSeekable, + VideoRoutingPolicy.dualLayerRouteIsWanted( + codecID: detectedCodecID, + dvProfile: detectedDVProfileNum, + dolbyVisionHandling: options.dolbyVisionHandling, + preferDualLayerDolbyVision: options.preferDualLayerDolbyVision, + canHardwareDecode: nil) { + let vIdx = probe.videoStreamIndex + await BlockingWork.detached(priority: .userInitiated) { [probe] in + VideoConfigRecord.completeHEVCRecordFromInBandParameterSets(demuxer: probe, streamIndex: vIdx) + }.value + if loadGeneration != gen { + probe.markClosed() + BlockingWork.detached { [probe] in probe.close() } + try checkLoadCurrent(gen) + } + } + // Strict verdict: SoftwarePlaybackHost only opens HardwareVideoDecoder for a proven hardware + // session. An unclassifiable config record stays native, because the host would hand it to + // libavcodec, which cannot compose the dual layers. Shared by the el_type probe's gate and the + // routing decision below, so the two cannot disagree. + let opensHardwareHEVC: () -> Bool = { + if let vStream = probe.stream(at: probe.videoStreamIndex), + let cp = vStream.pointee.codecpar { + return VTCapabilityProbe.opensHardwareDecoder(codecpar: cp) + } + return false + } + // Dual-layer decision for a Profile 7 source: FEL only, behind a capability gate (hardware HEVC + // decode, and a pixel rate the box can decode twice in real time), logged once. The el_type + // probe reads up to 256 packets and seeks back, so it runs only when nothing in + // `blockingReasonBeforeProbe` (preference, handling, live/seekable, hardware) rules the route + // out. The geometry and the budget are NOT in that list: they run after the probe, in + // `decide`, because a MEL or unclassified el_type outranks them in the log line, so a stream + // that its size alone would refuse still pays the probe. + var elType: String? = nil + var dualLayerGeometry = DualLayerRouteDecision.Geometry(width: 0, height: 0, frameRate: nil) + if probeOpened, let vStream = probe.stream(at: probe.videoStreamIndex), + let cp = vStream.pointee.codecpar { + dualLayerGeometry = DualLayerRouteDecision.Geometry( + width: Int(cp.pointee.width), height: Int(cp.pointee.height), + frameRate: Self.detectFrameRate(stream: vStream)) + } + let dualLayerSeekable = probeOpened && probe.videoStreamIndex >= 0 && probe.isSourceSeekable + if DualLayerRouteDecision.blockingReasonBeforeProbe( + codecID: detectedCodecID, + dvProfile: detectedDVProfileNum, + dolbyVisionHandling: options.dolbyVisionHandling, + preferDualLayerDolbyVision: options.preferDualLayerDolbyVision, + isLive: options.isLive, + isSourceSeekable: dualLayerSeekable, + canHardwareDecode: opensHardwareHEVC) == nil { + let vIdx = probe.videoStreamIndex + elType = await BlockingWork.detached(priority: .userInitiated) { [probe] in + let result = DolbyVisionRecordAudit.probeEnhancementLayerType(demuxer: probe, streamIndex: vIdx) + probe.seek(to: 0) + return result + }.value + if loadGeneration != gen { + probe.markClosed() + BlockingWork.detached { [probe] in probe.close() } + try checkLoadCurrent(gen) + } + } + let dualLayerDecision = DualLayerRouteDecision.decide( + codecID: detectedCodecID, + dvProfile: detectedDVProfileNum, + dolbyVisionHandling: options.dolbyVisionHandling, + preferDualLayerDolbyVision: options.preferDualLayerDolbyVision, + isLive: options.isLive, + isSourceSeekable: dualLayerSeekable, + canHardwareDecode: opensHardwareHEVC, + elType: elType, + geometry: dualLayerGeometry) + if let line = dualLayerDecision.logLine { + EngineLog.emit(line, category: .engine) + } + // A route that was not taken leaves no el_type behind, so neither the routing below nor a + // software host the session reaches for another reason composes the dual layers. + if !dualLayerDecision.isTaken { elType = nil } + sourceElType = elType var useSoftwarePath = VideoRoutingPolicy.requiresSoftwarePath( codecID: detectedCodecID, fieldOrder: detectedFieldOrder, av1Available: VTCapabilityProbe.av1Available, spsIndicatesInterlaced: spsIndicatesInterlaced, - stereo3DType: containerStereoType + stereo3DType: containerStereoType, + dvProfile: detectedDVProfileNum, + dolbyVisionHandling: options.dolbyVisionHandling, + preferDualLayerDolbyVision: options.preferDualLayerDolbyVision, + canHardwareDecode: opensHardwareHEVC, + elType: elType ) // #232: a declared interlaced field order is not evidence that any frame IS interlaced. // European 25 fps Blu-ray masters ship as 1080i25 (there is no 1080p25): interlaced carriage, @@ -5247,7 +5341,8 @@ public final class AetherEngine: ObservableObject { isLive: options.isLive, dvrWindowSeconds: options.dvrWindowSeconds, preopenedDemuxer: probeOpened ? probe : nil, - generation: gen + generation: gen, + elType: elType ) playbackBackend = .software // AE#550: the one thing a viewer on a receiver needs said out loud. This backend's @@ -6473,6 +6568,7 @@ public final class AetherEngine: ObservableObject { sourceDVProfile = nil dolbyVisionConversion = nil sourceCarriesHDR10PlusMetadata = false + sourceElType = nil sourceDVBLCompatID = nil sourceDolbyVisionBaseLayerPresentable = false sourceDolbyVisionRPUProfile = nil diff --git a/Sources/AetherEngine/Decoder/DualLayerRouteDecision.swift b/Sources/AetherEngine/Decoder/DualLayerRouteDecision.swift new file mode 100644 index 000000000..406ff8d2a --- /dev/null +++ b/Sources/AetherEngine/Decoder/DualLayerRouteDecision.swift @@ -0,0 +1,199 @@ +import AetherLibavcodec + +/// Whether a Profile 7 load takes the dual-layer route (`SoftwarePlaybackHost` + `DualLayerVideoDecoder`) +/// or keeps the native route (single-layer 8.1 where the display takes Dolby Vision), and if not, why. Pure so every exit is unit-testable; the load asks it +/// once per P7 source and logs `logLine`, so the diagnostic and the routing cannot disagree. +/// +/// The route is FEL only and capability gated: MEL, an unclassifiable enhancement layer, a box without a +/// hardware HEVC decoder, and a stream the box could not decode twice in real time all fall back to the +/// native route, which is the 8.1 conversion where the display takes Dolby Vision and the HDR10 base +/// layer where it does not. (The conversion loses nothing for MEL; for FEL it drops the residual.) +enum DualLayerRouteDecision: Equatable { + + case taken(elType: String, geometry: Geometry, budgetFraction: Double) + case notTaken(Reason) + + /// Every reason the route is not taken, in the order the checks run. + enum Reason: Equatable { + /// Not HEVC Dolby Vision Profile 7: nothing to decide, and nothing is logged. + case notProfile7 + case preferenceOff + case baseLayerOnlyHandling + case liveOrNotSeekable + case noHardwareHEVC + case melEnhancementLayer + case enhancementLayerTypeUnknown + /// The base layer's width or height is not positive (the probe found none): the budget cannot be + /// computed, and a zero requirement must not read as "fits". + case geometryUnknown + case overBudget(requiredPixelsPerSecond: Double, geometry: Geometry) + } + + /// Base-layer picture geometry the budget is computed from. `frameRate` is nil when the container + /// declares none; the budget then assumes `assumedFrameRate`. + struct Geometry: Equatable { + var width: Int + var height: Int + var frameRate: Double? + + var effectiveFrameRate: Double { + guard let frameRate, frameRate.isFinite, frameRate > 0 else { return DualLayerRouteDecision.assumedFrameRate } + return frameRate + } + + var text: String { + let rate = frameRate.map { rateText($0) } ?? "\(rateText(DualLayerRouteDecision.assumedFrameRate)) (assumed)" + return "\(width)x\(height)@\(rate)" + } + + private func rateText(_ value: Double) -> String { + value.rounded() == value ? String(Int(value)) : String(format: "%.3f", value) + } + } + + /// The most pixels per second, base layer plus enhancement layer, the route is willing to decode in + /// real time: UHD 3840x2160 at 30 fps (248.8 Mpx/s) plus a 1920x1080 enhancement layer at 30 fps + /// (62.2 Mpx/s), 311.04 Mpx/s in all. The anchor is a 24p UHD FEL master, the shape UHD Blu-ray + /// remuxes have in practice (248.8 Mpx/s with its enhancement layer, 80% of this). Measured: the Mac + /// soak runs of this stack, and the composer's GPU time on an Apple TV 4K 2nd generation in earlier + /// development builds (12.9-14.3 ms per 23.976 fps frame, docs/dolby-vision-fel.md §4.2). The budget + /// itself, and 30 fps as a ceiling above 24p, are an assumption until the Apple TV run of this + /// stack. Anything heavier (UHD at 60 fps, 8K) keeps the native route instead of + /// risking dropped frames. + static let maximumPixelsPerSecond: Double = + (3840.0 * 2160.0 + 1920.0 * 1080.0) * 30.0 + + /// Frame rate the budget assumes when the container declares none. 24, because FEL masters are 24p. + static let assumedFrameRate: Double = 24 + + /// The enhancement layer's share of the base layer's area when the stream's own enhancement-layer + /// dimensions are not known: a quarter, which is what a 4K FEL disc ships (1080p over UHD). + static let assumedEnhancementLayerAreaFraction: Double = 0.25 + + /// Pixels per second both layers need: the base layer's, plus the enhancement layer's, taken from + /// `enhancementLayerSize` when the stream states it and as a quarter of the base layer's area otherwise. + /// The load passes no `enhancementLayerSize`: it budgets a quarter-area enhancement layer (every retail + /// FEL title's layout), and a 1:1 enhancement layer is not detected at load. + static func requiredPixelsPerSecond( + geometry: Geometry, enhancementLayerSize: (width: Int, height: Int)? = nil + ) -> Double { + let base = Double(geometry.width) * Double(geometry.height) + let enhancement = enhancementLayerSize.map { Double($0.width) * Double($0.height) } + ?? base * assumedEnhancementLayerAreaFraction + return (base + enhancement) * geometry.effectiveFrameRate + } + + /// The reason the route cannot be taken whatever the enhancement layer is, or nil when only the + /// `el_type` (and the budget) remain to decide. The load runs this first so the `el_type` probe, a + /// packet scan and a seek, only happens when its answer can change the route. A nil + /// `canHardwareDecode` assumes hardware decode. + static func blockingReasonBeforeProbe( + codecID: AVCodecID, + dvProfile: Int?, + dolbyVisionHandling: DolbyVisionHandling, + preferDualLayerDolbyVision: Bool, + isLive: Bool, + isSourceSeekable: Bool, + canHardwareDecode: (() -> Bool)? + ) -> Reason? { + guard codecID == AV_CODEC_ID_HEVC, dvProfile == 7 else { return .notProfile7 } + guard preferDualLayerDolbyVision else { return .preferenceOff } + guard dolbyVisionHandling == .automatic else { return .baseLayerOnlyHandling } + guard !isLive, isSourceSeekable else { return .liveOrNotSeekable } + guard canHardwareDecode?() ?? true else { return .noHardwareHEVC } + return nil + } + + /// The full decision. `elType` is the probe's answer ("FEL", "MEL", or nil when it found none). + static func decide( + codecID: AVCodecID, + dvProfile: Int?, + dolbyVisionHandling: DolbyVisionHandling, + preferDualLayerDolbyVision: Bool, + isLive: Bool, + isSourceSeekable: Bool, + canHardwareDecode: (() -> Bool)?, + elType: String?, + geometry: Geometry, + enhancementLayerSize: (width: Int, height: Int)? = nil + ) -> DualLayerRouteDecision { + if let reason = blockingReasonBeforeProbe( + codecID: codecID, dvProfile: dvProfile, dolbyVisionHandling: dolbyVisionHandling, + preferDualLayerDolbyVision: preferDualLayerDolbyVision, isLive: isLive, + isSourceSeekable: isSourceSeekable, canHardwareDecode: canHardwareDecode) { + return .notTaken(reason) + } + switch elType { + case "FEL": + guard geometry.width > 0, geometry.height > 0 else { return .notTaken(.geometryUnknown) } + let required = requiredPixelsPerSecond(geometry: geometry, enhancementLayerSize: enhancementLayerSize) + guard required <= maximumPixelsPerSecond else { + return .notTaken(.overBudget(requiredPixelsPerSecond: required, geometry: geometry)) + } + return .taken(elType: "FEL", geometry: geometry, budgetFraction: required / maximumPixelsPerSecond) + case "MEL": + return .notTaken(.melEnhancementLayer) + default: + return .notTaken(.enhancementLayerTypeUnknown) + } + } + + var isTaken: Bool { + if case .taken = self { return true } + return false + } + + /// The one diagnostic line for a Profile 7 load, or nil for a source this decision does not concern. + var logLine: String? { + switch self { + case .notTaken(.notProfile7): + return nil + case let .taken(elType, geometry, fraction): + return "[AetherEngine] DV P7 dual-layer route: taken (el_type=\(elType), \(geometry.text), " + + "budget \(Int((fraction * 100).rounded()))%)" + case let .notTaken(reason): + return "[AetherEngine] DV P7 dual-layer route: not taken (\(reason.text)), \(reason.fallbackText)" + } + } +} + +extension DualLayerRouteDecision.Reason { + + var text: String { + switch self { + case .notProfile7: + return "not Dolby Vision Profile 7" + case .preferenceOff: + return "preferDualLayerDolbyVision is off" + case .baseLayerOnlyHandling: + return "dolbyVisionHandling is baseLayerOnly" + case .liveOrNotSeekable: + return "live or non-seekable source" + case .noHardwareHEVC: + return "no hardware HEVC decoder for this stream" + case .melEnhancementLayer: + return "el_type=MEL, nothing to compose" + case .enhancementLayerTypeUnknown: + return "el_type unknown, no enhancement layer classified in the first RPU" + case .geometryUnknown: + return "base-layer size unknown" + case let .overBudget(required, geometry): + let percent = Int((required / DualLayerRouteDecision.maximumPixelsPerSecond * 100).rounded()) + return "over the real-time budget, \(geometry.text) needs \(percent)% of it" + } + } + + /// What the not-taken line says plays instead. Deliberately about the ROUTE, not the picture: the + /// native route presents 8.1 only where the display takes Dolby Vision, and the HDR10 base layer + /// where it does not (`CodecRoutePolicy` strips the Dolby Vision) or `preferredDecodePath` is + /// `.software`. Pinned by `DualLayerRouteDecisionTests`. + static let nativeRouteFallback = "keeps the native route (8.1 where the display takes Dolby Vision)" + + /// What plays instead. + var fallbackText: String { + switch self { + case .baseLayerOnlyHandling: return "playing the HDR10 base layer" + default: return Self.nativeRouteFallback + } + } +} diff --git a/Sources/AetherEngine/Decoder/VideoRoutingPolicy.swift b/Sources/AetherEngine/Decoder/VideoRoutingPolicy.swift index e8b9b9b0a..e75673ba3 100644 --- a/Sources/AetherEngine/Decoder/VideoRoutingPolicy.swift +++ b/Sources/AetherEngine/Decoder/VideoRoutingPolicy.swift @@ -12,6 +12,22 @@ enum VideoRoutingPolicy { AV_FIELD_TT, AV_FIELD_BB, AV_FIELD_TB, AV_FIELD_BT ] + /// Whether the Profile 7 dual-layer route is both requested and possible, which is everything that + /// decides it except the stream's `el_type`. `requiresSoftwarePath` adds the FEL condition; the load + /// asks this one first, so the `el_type` probe (a packet scan and a seek) only runs when its answer + /// can change the route. A nil `canHardwareDecode` assumes hardware decode. + static func dualLayerRouteIsWanted( + codecID: AVCodecID, + dvProfile: Int?, + dolbyVisionHandling: DolbyVisionHandling, + preferDualLayerDolbyVision: Bool, + canHardwareDecode: (() -> Bool)? + ) -> Bool { + guard codecID == AV_CODEC_ID_HEVC, dvProfile == 7, + preferDualLayerDolbyVision, dolbyVisionHandling == .automatic else { return false } + return canHardwareDecode?() ?? true + } + /// True when a video codec must use the software decode path (SoftwarePlaybackHost) instead of /// native AVPlayer. `av1Available` is `VTCapabilityProbe.av1Available` (HW AV1 decode support). /// Evaluated only for AV1; registering its decoder must not delay unrelated formats. @@ -33,8 +49,29 @@ enum VideoRoutingPolicy { fieldOrder: AVFieldOrder, av1Available: @autoclosure () -> Bool, spsIndicatesInterlaced: Bool = false, - stereo3DType: AVStereo3DType? = nil + stereo3DType: AVStereo3DType? = nil, + dvProfile: Int? = nil, + dolbyVisionHandling: DolbyVisionHandling = .automatic, + preferDualLayerDolbyVision: Bool = false, + canHardwareDecode: (() -> Bool)? = nil, + elType: String? = nil ) -> Bool { + // Profile 7 FEL dual-layer Metal reconstruction routes to the software host pipeline + // when preferDualLayerDolbyVision is enabled, dolbyVisionHandling is .automatic, and hardware decode is confirmed (or assumed true if closure + // not provided). + // Only elType == "FEL" routes to software; MEL, nil, or unknown stay native + // (the 8.1 conversion where the display takes Dolby Vision, the HDR10 base layer where it does + // not; the conversion is lossless only for MEL). + // If dolbyVisionHandling is .baseLayerOnly, it also stays native (HDR10 base layer). + if codecID == AV_CODEC_ID_HEVC, dvProfile == 7 { + return elType == "FEL" && dualLayerRouteIsWanted( + codecID: codecID, + dvProfile: dvProfile, + dolbyVisionHandling: dolbyVisionHandling, + preferDualLayerDolbyVision: preferDualLayerDolbyVision, + canHardwareDecode: canHardwareDecode + ) + } switch codecID { case AV_CODEC_ID_NONE, AV_CODEC_ID_HEVC: return false diff --git a/Sources/AetherEngine/Diagnostics/EngineLog.swift b/Sources/AetherEngine/Diagnostics/EngineLog.swift index 9f24628af..b0f91355b 100644 --- a/Sources/AetherEngine/Diagnostics/EngineLog.swift +++ b/Sources/AetherEngine/Diagnostics/EngineLog.swift @@ -63,6 +63,14 @@ public enum EngineLog { return map }() + private static let osLogs: [Category: OSLog] = { + var map: [Category: OSLog] = [:] + for cat in Category.allCases { + map[cat] = OSLog(subsystem: subsystem, category: cat.rawValue) + } + return map + }() + public enum Level: Sendable { /// Default: OSLog default level + host handler (in-app overlay, aetherctl stdout). case info @@ -91,12 +99,14 @@ public enum EngineLog { /// `.public` privacy means a Console.app capture or a sysdiagnose would otherwise carry the token /// in clear text even for a host that scrubs its own log. private static func deliver(_ line: String, category: Category, level: Level) { - let line = LogRedaction.redact(line) switch level { case .info: + let line = LogRedaction.redact(line) loggers[category]?.log("\(line, privacy: .public)") handler?(line) case .verbose: + guard let osLog = osLogs[category], osLog.isEnabled(type: .debug) else { return } + let line = LogRedaction.redact(line) loggers[category]?.debug("\(line, privacy: .public)") } } diff --git a/Sources/AetherEngine/Native/SoftwarePlaybackHost.swift b/Sources/AetherEngine/Native/SoftwarePlaybackHost.swift index 9817a8f6e..f8800a22d 100644 --- a/Sources/AetherEngine/Native/SoftwarePlaybackHost.swift +++ b/Sources/AetherEngine/Native/SoftwarePlaybackHost.swift @@ -72,6 +72,18 @@ final class SoftwarePlaybackHost { @Published private(set) var failure: PlaybackErrorInfo? @Published private(set) var didReachEnd: Bool = false + /// Enhancement layer type for Profile 7 streams (FEL or MEL), used for decoder selection + var elType: String? = nil + + /// Which layer of a Dolby Vision source to present (.automatic or .baseLayerOnly). + var dolbyVisionHandling: DolbyVisionHandling = .automatic + + /// Opt-in to full dual-layer Dolby Vision (Profile 7 FEL) decoding and Metal GPU composition over single-layer fallback. + var preferDualLayerDolbyVision: Bool = false + + /// Forwarded Dolby Vision profile number (e.g. 7, 8) if detected from bitstream or container + var configuredDVProfile: Int? + /// #315: `AVSampleBufferDisplayLayer.isReadyForDisplay` for the renderer's layer, this path's /// answer to "there is a picture". Frames enqueued is not that answer: it counts what the /// decoder handed over, and #298 is the report where every one of them went into a layer no @@ -857,6 +869,51 @@ final class SoftwarePlaybackHost { return min(max(untilGateOpens, floor), ceiling) } + /// Pure decoder selection predicate for Profile 7 / HEVC streams on the software host. + /// DualLayerVideoDecoder is selected ONLY when dvProfile == 7 && elType == "FEL" && preferDualLayerDolbyVision + /// && dolbyVisionHandling == .automatic. + /// All other combinations (e.g. MEL, baseLayerOnly, preferDualLayerDolbyVision == false) pick HardwareVideoDecoder. + nonisolated static func shouldUseDualLayerDecoder( + dvProfile: Int?, + elType: String?, + preferDualLayerDolbyVision: Bool, + dolbyVisionHandling: DolbyVisionHandling + ) -> Bool { + dvProfile == 7 + && elType == "FEL" + && preferDualLayerDolbyVision + && dolbyVisionHandling == .automatic + } + + /// Selects and instantiates the appropriate VideoDecodingPipeline based on stream and host options. + func selectVideoDecoder( + codecID: AVCodecID, + canHardwareDecode: Bool, + dvProfile: Int? + ) -> any VideoDecodingPipeline { + if codecID == AV_CODEC_ID_HEVC && canHardwareDecode { + if Self.shouldUseDualLayerDecoder( + dvProfile: dvProfile, + elType: elType, + preferDualLayerDolbyVision: preferDualLayerDolbyVision, + dolbyVisionHandling: dolbyVisionHandling + ) { + // No Metal device: the composer cannot be built, so decode the base layer alone (the + // decoder the software host uses for any other Profile 7 stream) instead of opening a + // decoder whose `open()` can only throw. A shader that fails to compile is caught later, + // once: every frame then goes out as its base layer. + if let composer = DoviFelComposer() { + return DualLayerVideoDecoder(composer: composer) + } + return HardwareVideoDecoder() + } else { + return HardwareVideoDecoder() + } + } else { + return SoftwareVideoDecoder() + } + } + // MARK: - Load func load( @@ -915,6 +972,23 @@ final class SoftwarePlaybackHost { category: .swPlayback ) + // A Profile 7 FEL session on a freshly opened demuxer (the load's probe completes its own) still + // has the empty hvcC of an in-band-parameter-set source: complete it, or the verdict below sends + // the stream to libavcodec and the enhancement layer is never composed. No-op otherwise. + if !isLive, + Self.shouldUseDualLayerDecoder( + dvProfile: configuredDVProfile ?? AetherEngine.dvProfile(stream: vStream), + elType: elType, + preferDualLayerDolbyVision: preferDualLayerDolbyVision, + dolbyVisionHandling: dolbyVisionHandling) { + let vIdx = dem.videoStreamIndex + await BlockingWork.detached(priority: .userInitiated) { + VideoConfigRecord.completeHEVCRecordFromInBandParameterSets(demuxer: dem, streamIndex: vIdx) + }.value + // A stop() can land on this suspension as on the start-position seek below. + guard !stopRequested else { return } + } + // HEVC -> VTDecompressionSession (HW) when VideoToolbox PROVABLY HW-decodes this exact format; // everything else -> libavcodec. Replace wholesale to prevent state bleed. #2: HardwareVideoDecoder // requires a HW decoder and has no software fallback, so an HEVC Rext stream on hardware without a @@ -925,11 +999,40 @@ final class SoftwarePlaybackHost { codecpar.pointee.codec_id == AV_CODEC_ID_HEVC, VTCapabilityProbe.hardwareDecodeVerdict(codecpar: codecpar).opensHardwareDecoder { videoDecoder.close() - videoDecoder = HardwareVideoDecoder() - EngineLog.emit( - "[SWHost] selected HardwareVideoDecoder (VT HEVC) for codec_id=\(codecpar.pointee.codec_id.rawValue)", - category: .swPlayback + let dvProfile = configuredDVProfile ?? AetherEngine.dvProfile(stream: vStream) + let chosen = selectVideoDecoder( + codecID: codecpar.pointee.codec_id, + canHardwareDecode: true, + dvProfile: dvProfile ) + videoDecoder = chosen + if let dual = chosen as? DualLayerVideoDecoder { + // An in-band primary open failure (no hvcC to open from) leaves nothing to show. Surface it + // through the same `failure` the read loops use for a decode error. It belongs to the + // decoder session, not to a seek: the decoder reports it once and a seek does not reopen the + // primary decoder, so gating it by seek generation would lose it for good when a seek lands + // between the report and the main-actor hop (black video, no error). Only a stopped host + // drops it. + dual.onFatalDecodeError = { [weak self] error in + Task { @MainActor [weak self] in + guard let self, SoftwareReadAdmission.admitsSessionFailure(stopRequested: self.stopRequested) else { return } + self.failure = PlaybackErrorInfo( + kind: .softwarePipelineFailed, + message: "Playback error: \(error.localizedDescription)") + } + } + } + if chosen is DualLayerVideoDecoder { + EngineLog.emit( + "[SWHost] selected DualLayerVideoDecoder (Metal P7 FEL) for codec_id=\(codecpar.pointee.codec_id.rawValue)", + category: .swPlayback + ) + } else { + EngineLog.emit( + "[SWHost] selected HardwareVideoDecoder (VT HEVC) for codec_id=\(codecpar.pointee.codec_id.rawValue)", + category: .swPlayback + ) + } } else if !(videoDecoder is SoftwareVideoDecoder) { videoDecoder.close() videoDecoder = SoftwareVideoDecoder() @@ -940,8 +1043,15 @@ final class SoftwarePlaybackHost { ) } - // Audit PERF-104: only the VideoToolbox decoder emits out of presentation order. - renderer.setReorderDepth(SampleBufferRenderer.reorderDepth(forHardwareDecoder: videoDecoder is HardwareVideoDecoder)) + // Audit PERF-104: only the VideoToolbox decoders emit out of presentation order. That includes the + // DualLayerVideoDecoder: OrderedDelivery keeps its frames in the order the base layer came out of + // VideoToolbox, which is decode order (measured on the real fixtures, `DoviFelRealFileTests`), so it + // needs the same reorder depth as the single-layer hardware decoder. + renderer.setReorderDepth(SampleBufferRenderer.reorderDepth( + forHardwareDecoder: videoDecoder is HardwareVideoDecoder || videoDecoder is DualLayerVideoDecoder)) + renderer.setFELLookahead( + enabled: videoDecoder is DualLayerVideoDecoder, + frameRate: AetherEngine.detectFrameRate(stream: vStream)) // Flip display layer into HDR mode before frames arrive; without this preferredDynamicRange stays .standard and PQ/HLG renders desaturated. if let codecpar = vStream.pointee.codecpar { @@ -1081,8 +1191,8 @@ final class SoftwarePlaybackHost { // #254: same off-main, deadline-bounded reposition the transport seek uses. A resume into a // remote source that has to scan for its landing would otherwise block the main thread here. _ = await dem.seekBounded(to: sourceStart, timeout: Self.seekBudgetSeconds, on: seekQueue) - // This is load()'s only suspension point, so it is also the only place a stop() can land - // mid-load. Arming the clock and publishing isReady on a session already torn down would + // Besides the in-band record completion above, this is load()'s only suspension point, so it + // is the other place a stop() can land mid-load. Arming the clock and publishing isReady on a session already torn down would // hand the engine a host it has stopped. guard !stopRequested else { return } // Mirror seek() skip-PTS + clock alignment so demux drops pre-keyframe frames and synchronizer starts at the resume offset. @@ -2548,7 +2658,12 @@ final class SoftwarePlaybackHost { // Back-pressure against renderer queue; also bail on pause without consuming. // The wait can outlast the audio lead, so keep pumping audio while parked. var waitTicks = 0 - while !renderer.isReadyForMoreMediaData && !stopRequested() && isPlaying() + // The FEL look-ahead gate (`mayEnqueueVideoFrame`) holds a packet until its timestamp is + // inside the clock's window; a seek, including a backward DVR seek, moves the cursor and + // releases it here rather than waiting for its old timestamp to come round. + while !renderer.mayEnqueueVideoFrame( + ptsSeconds: pkt.pts, clockSeconds: audioOutput?.currentTimeSeconds ?? .nan, + clockArmed: clockArmed()) && !stopRequested() && isPlaying() && seekGeneration() == generation && readCursor() == cursor && cursor >= ring.seqBounds.first { autoreleasepool { @@ -2767,10 +2882,19 @@ final class SoftwarePlaybackHost { // takes it, holds it, stops reporting `isReadyForMoreMediaData`, and the loop parks on that // signal until its FIFO caps out. Measured here as three to four seconds of `enq=+0` with the // audio lead decaying under it, on a session that reports playing and never rebuffers. + func mayDecodeVideoPacket(_ packet: UnsafeMutablePointer) -> Bool { + let pts = packet.pointee.pts + let seconds = pts != Int64.min && videoTimeBaseSeconds > 0 + ? Double(pts) * videoTimeBaseSeconds : Double.nan + return renderer.mayEnqueueVideoFrame( + ptsSeconds: seconds, clockSeconds: audioOutput?.currentTimeSeconds ?? .nan, + clockArmed: clockArmed()) + } + func drainParkedVideoNonblocking() { if seekGeneration() != parkedSeekGeneration { return } let epoch = videoDecoder.feedEpoch - while !parkedVideo.isEmpty, renderer.isReadyForMoreMediaData, + while let head = parkedVideo.first, mayDecodeVideoPacket(head), !stopRequested(), !backgroundAudioOnly() { // Leave the rest standing: the next iteration of the loop frees the FIFO as a batch // once it has read the new generation. @@ -3165,7 +3289,12 @@ final class SoftwarePlaybackHost { return true } // Back-pressure via SampleBufferRenderer.isReadyForMoreMediaData (not the deprecated layer property). Park on condition while paused to avoid 200 Hz CPU spin. - while !renderer.isReadyForMoreMediaData && !stopRequested() && !backgroundAudioOnly() { + // Queue readiness plus the FEL-only look-ahead cap. Hold the compressed packet here, + // rather than letting an asynchronous decode allocate another composed buffer. + while !mayDecodeVideoPacket(packet) && !stopRequested() && !backgroundAudioOnly() { + // A seek can move the clock backwards while this compressed FEL packet is + // held. Leave the wait so the generation check below discards it immediately. + if videoDecoder is DualLayerVideoDecoder, seekGeneration() != genBeforeRead { break } autoreleasepool { if !isPlaying() { condition.lock() diff --git a/Sources/AetherEngine/Native/SoftwareReadAdmission.swift b/Sources/AetherEngine/Native/SoftwareReadAdmission.swift index 6ed97862f..2215a2efa 100644 --- a/Sources/AetherEngine/Native/SoftwareReadAdmission.swift +++ b/Sources/AetherEngine/Native/SoftwareReadAdmission.swift @@ -9,6 +9,11 @@ enum SoftwareReadAdmission { !stopRequested && readGeneration == requestedGeneration && requestedGeneration == settledGeneration } + + /// A failure that belongs to the session rather than to one seek (the dual-layer decoder's primary + /// open, reported once and not retried by a seek). A seek in flight must not drop it, because + /// nothing reports it a second time; only a stopped host does. + static func admitsSessionFailure(stopRequested: Bool) -> Bool { !stopRequested } } /// The combined VOD loop stays alive after one EOF/error publication. MainActor may reject that diff --git a/Sources/AetherEngine/PlayerState.swift b/Sources/AetherEngine/PlayerState.swift index 68892ce54..26759351e 100644 --- a/Sources/AetherEngine/PlayerState.swift +++ b/Sources/AetherEngine/PlayerState.swift @@ -388,8 +388,14 @@ public enum DecodePath: String, Sendable, Equatable, CaseIterable { } /// Which layer of a Dolby Vision source a session presents. See `LoadOptions.dolbyVisionHandling`. +/// For Profile 7, FEL (Full Enhancement Layer, `el_type == 'FEL'`) is reconstructed from both layers +/// on the software host with `DualLayerVideoDecoder` only when `LoadOptions.preferDualLayerDolbyVision` +/// is true (default false) and the capability gate passes. Otherwise a Profile 7 source goes native: +/// the single-layer 8.1 conversion where the display takes Dolby Vision, the HDR10 base layer where +/// it does not. MEL (Minimal Enhancement Layer, `el_type == 'MEL'`) always goes native, because it has +/// no residual to compose, whatever the preference. public enum DolbyVisionHandling: String, Sendable, Equatable, CaseIterable { - /// Dolby Vision wherever the profile and the display allow it. The default. + /// Dolby Vision wherever the profile and the display allow it. A Profile 7 source takes the native route (converted to single-layer 8.1 where the display takes Dolby Vision, the HDR10 base layer where it does not); its FEL is reconstructed (dual-layer Metal) instead only when `LoadOptions.preferDualLayerDolbyVision` is true, which defaults to false. The default. case automatic /// Present the HDR10 / HLG base layer and leave the Dolby Vision out of the container: `hvc1` / /// `av01` sample entry, `dvcC` stripped, no `SUPPLEMENTAL-CODECS`, HDR10 / HLG display criteria. @@ -465,6 +471,33 @@ public struct LoadOptions: Sendable, Equatable { /// contradicts. public var dolbyVisionHandling: DolbyVisionHandling + /// Opt in to full dual-layer Dolby Vision decoding of Profile 7 FEL streams, composed on the GPU, instead of + /// the native route (the single-layer Profile 8.1 conversion, or the HDR10 base layer). When true, `dolbyVisionHandling` is `.automatic`, the stream's + /// enhancement layer is FEL, VideoToolbox decodes HEVC in hardware and the stream fits the real-time + /// budget (`DualLayerRouteDecision.maximumPixelsPerSecond`: UHD at 30 fps plus a 1080p enhancement + /// layer), the stream routes through the dual-layer pipeline (SoftwarePlaybackHost + + /// DualLayerVideoDecoder). A Profile 7 MEL stream, and anything the device cannot decode twice in real + /// time, keeps the native route either way (8.1 where the display takes Dolby Vision, the HDR10 base + /// layer where it does not); the load logs one line saying which route it took and why. The source + /// must be a seekable VOD: live and non-seekable sources keep the native route. Default false. + /// + /// The route does not consult the display. With this on and a display that does not take Dolby + /// Vision, the route is still taken: the composed full-FEL picture is presented with its attached + /// metadata, and how tvOS presents it is up to the platform. The audio trade-off below applies there + /// too. + /// + /// A tuning field: `reloadAtCurrentPosition(applying:)` carries a change on a URL source (which + /// re-runs `load()`). On a custom `IOReader` source a Profile 7 session refuses it with + /// `SessionReloadRefusal.dualLayerRouteNeedsFreshLoad`, because the in-place rebuild keeps the route + /// the session was loaded with; call `load()` again instead. + /// + /// Known trade-off, audio: the route runs on the software host, whose audio is decoded by FFmpeg and + /// played through the sample-buffer audio renderer, and on tvOS only AVPlayer emits Dolby Atmos objects. + /// An E-AC-3 (JOC) track therefore plays as its channel-bed core (5.1 or 7.1), without the Atmos objects, + /// where the native route would hand it to AVPlayer unchanged. TrueHD behaves as on the native route, + /// which also decodes it and loses the objects. Not engineered around. + public var preferDualLayerDolbyVision: Bool = false + /// Mirror of `AVDisplayManager.isDisplayCriteriaMatchingEnabled`. Default `true`. When `false`, engine routes HDR sources through the media playlist (auto-tonemap path) because AVKit cannot switch the panel. public var matchContentEnabled: Bool @@ -947,6 +980,7 @@ public struct LoadOptions: Sendable, Equatable { keepDvh1TagWithoutDV: Bool = false, forceDolbyVisionOnNonDVDisplay: Bool = false, dolbyVisionHandling: DolbyVisionHandling = .automatic, + preferDualLayerDolbyVision: Bool = false, matchContentEnabled: Bool = true, panelIsInHDRMode: Bool = false, attemptsHDRMasterOnUnprovenPanel: Bool = true, @@ -1000,6 +1034,7 @@ public struct LoadOptions: Sendable, Equatable { self.keepDvh1TagWithoutDV = keepDvh1TagWithoutDV self.forceDolbyVisionOnNonDVDisplay = forceDolbyVisionOnNonDVDisplay self.dolbyVisionHandling = dolbyVisionHandling + self.preferDualLayerDolbyVision = preferDualLayerDolbyVision self.matchContentEnabled = matchContentEnabled self.panelIsInHDRMode = panelIsInHDRMode self.attemptsHDRMasterOnUnprovenPanel = attemptsHDRMasterOnUnprovenPanel @@ -1062,7 +1097,8 @@ public enum VideoFormat: Sendable, Equatable { public enum DolbyVisionConversion: Sendable, Equatable { /// Dual-layer Profile 7 rewritten per packet to single-layer Profile 8.1, on every display (a display /// without Dolby Vision gets the RPU-driven tone map a Profile 8.1 gets). The enhancement layer is - /// discarded: a MEL carries next to nothing, a FEL loses its refinement. + /// discarded: a MEL carries next to nothing, and a FEL played on this route loses its refinement (a FEL + /// on the software dual-layer route keeps it). case profile7ToProfile81 } diff --git a/Sources/AetherEngine/Video/CodecRoutePolicy.swift b/Sources/AetherEngine/Video/CodecRoutePolicy.swift index e7b890bf0..8dc051511 100644 --- a/Sources/AetherEngine/Video/CodecRoutePolicy.swift +++ b/Sources/AetherEngine/Video/CodecRoutePolicy.swift @@ -651,7 +651,11 @@ extension HLSVideoEngine { dvVariant: dvVariant ) case .profile7: - // P7 dual-layer (UHD-BD): convert the RPU P7 -> P8.1 per packet (DoviRpuConverter, libdovi + // P7 dual-layer (UHD-BD). With preferDualLayerDolbyVision on and the box able to afford it, + // VideoRoutingPolicy sends a FEL to the software dual-layer route instead (SoftwarePlaybackHost + + // DualLayerVideoDecoder), which keeps the enhancement layer; every other P7 load lands here. + // + // Native route: convert the RPU P7 -> P8.1 per packet (DoviRpuConverter, libdovi // mode 2), drop the EL, rewrite the container dvcC to P8.1, and from there on serve exactly // what a P8.1 source is served, on every display. The way mpv treats Profile 7 without an EL // decoder: base layer plus RPU, never the bare base layer. diff --git a/Sources/AetherEngine/Video/DolbyVisionRecordAudit.swift b/Sources/AetherEngine/Video/DolbyVisionRecordAudit.swift index 386de2e58..d6dcc6b9f 100644 --- a/Sources/AetherEngine/Video/DolbyVisionRecordAudit.swift +++ b/Sources/AetherEngine/Video/DolbyVisionRecordAudit.swift @@ -232,6 +232,45 @@ public enum DolbyVisionRecordAudit { return nil } + /// Walks the first video packets of an open demuxer for the first RPU of a Profile 7 source and reads + /// its enhancement-layer type ("FEL" or "MEL"). nil when no packet in the budget carries one. + /// + /// Takes the session's own probe demuxer, so the packets read here are packets the caller must not + /// need afterwards: the caller seeks it back or opens another. The same packet and byte ceilings as + /// `rpuProfile(walking:)` apply, for the same reason (audit BIT-2, BIT-103). + static func probeEnhancementLayerType( + demuxer: Demuxer, streamIndex: Int32, packetBudget: Int = auditPacketBudget, + byteBudget: Int64 = Int64(walkByteBudget) + ) -> String? { + guard let stream = demuxer.stream(at: streamIndex) else { return nil } + let codecpar = stream.pointee.codecpar + let framing = A53SEIParser.nalFraming( + codec: .hevc, extradata: codecpar?.pointee.extradata, + size: Int(codecpar?.pointee.extradata_size ?? 0)) + + demuxer.beginInputByteBudget(byteBudget) + defer { demuxer.endInputByteBudget() } + + var walked = 0 + var packetsRead = 0 + var bytesRead = 0 + while walked < packetBudget { + guard !walkExhausted(packetsRead: packetsRead, bytesRead: bytesRead, packetBudget: packetBudget) + else { return nil } + guard let packet = (try? demuxer.readPacket()) ?? nil else { return nil } + defer { + av_packet_unref(packet) + av_packet_free_safe(packet) + } + packetsRead += 1 + bytesRead += Int(max(packet.pointee.size, 0)) + guard packet.pointee.stream_index == streamIndex else { continue } + walked += 1 + if let type = DoviRpuConverter.enhancementLayerType(packet, framing: framing) { return type } + } + return nil + } + /// Audit BIT-2: the walk counted only video packets, so a source that stops delivering video after /// its head made every open read the rest of the file. Every packet counts against these instead. /// No AVDISCARD_ALL on the other streams: the demuxer would then skip them inside one read, where diff --git a/Sources/AetherEngine/Video/HLSVideoEngine+SegmentPlanning.swift b/Sources/AetherEngine/Video/HLSVideoEngine+SegmentPlanning.swift index 1218db350..34d0d0327 100644 --- a/Sources/AetherEngine/Video/HLSVideoEngine+SegmentPlanning.swift +++ b/Sources/AetherEngine/Video/HLSVideoEngine+SegmentPlanning.swift @@ -487,9 +487,7 @@ extension HLSVideoEngine { guard Self.configRecordNeedsInBandRebuild(extradata, size: extradataSize) else { return nil } let naluLengthSize = 4 // the predicate above already required it - var vps: [UInt8]? - var sps: [UInt8]? - var pps: [UInt8]? + var sets = VideoConfigRecord.HEVCParameterSets() // Counted in VIDEO packets: a film interleaves audio and dozens of subtitle tracks, which // would otherwise exhaust the budget before the first video packet. let packetBudget = 16 @@ -501,55 +499,19 @@ extension HLSVideoEngine { if pkt.pointee.stream_index != videoStreamIndex { return false } packetsScanned += 1 guard let pktData = pkt.pointee.data else { return packetsScanned >= packetBudget } - let pktSize = Int(pkt.pointee.size) - - var offset = 0 - while offset + naluLengthSize <= pktSize { - var nalLen = 0 - for i in 0.. pktSize { break } - let nalType = (Int(pktData[offset]) >> 1) & 0x3F // HEVC NAL type: bits 1..6 of byte 0 - let nalBytes = Array(UnsafeBufferPointer(start: pktData + offset, count: nalLen)) - switch nalType { - case 32: if vps == nil { vps = nalBytes } - case 33: if sps == nil { sps = nalBytes } - case 34: if pps == nil { pps = nalBytes } - default: break - } - offset += nalLen - } - - return (vps != nil && sps != nil && pps != nil) || packetsScanned >= packetBudget + sets.collect(from: pktData, size: Int(pkt.pointee.size), naluLengthSize: naluLengthSize) + return sets.isComplete || packetsScanned >= packetBudget } - guard let vps, let sps, let pps else { + guard let vps = sets.vps, let sps = sets.sps, let pps = sets.pps else { EngineLog.emit( "[HLSVideoEngine] in-band parameter-set scan found none: " - + "packets=\(packetsScanned) vps=\(vps != nil) sps=\(sps != nil) pps=\(pps != nil)", + + "packets=\(packetsScanned) vps=\(sets.vps != nil) sps=\(sets.sps != nil) pps=\(sets.pps != nil)", category: .session ) return nil } - - // Assemble hvcC: keep source 22-byte header, set numOfArrays=3, append VPS/SPS/PPS arrays (1-byte type, 2-byte numNalus=1, 2-byte nalUnitLength, NAL bytes). - var hvcC: [UInt8] = [] - hvcC.reserveCapacity(22 + 1 + 5 * 3 + vps.count + sps.count + pps.count) - for i in 0..<22 { hvcC.append(extradata[i]) } - hvcC.append(3) - func appendArray(nalUnitType: UInt8, nal: [UInt8]) { - hvcC.append(0x80 | (nalUnitType & 0x3F)) - hvcC.append(0); hvcC.append(1) - let nl = UInt16(nal.count) - hvcC.append(UInt8(nl >> 8)); hvcC.append(UInt8(nl & 0xFF)) - hvcC.append(contentsOf: nal) - } - appendArray(nalUnitType: 32, nal: vps) - appendArray(nalUnitType: 33, nal: sps) - appendArray(nalUnitType: 34, nal: pps) - return hvcC + return VideoConfigRecord.hevcRecord(header: extradata, vps: vps, sps: sps, pps: pps) } /// Shows the packets at the head of the source to `inspect`, which returns true once it has seen diff --git a/Sources/AetherEngine/Video/VTCapabilityProbe.swift b/Sources/AetherEngine/Video/VTCapabilityProbe.swift index 5187c9c1f..b91f208e6 100644 --- a/Sources/AetherEngine/Video/VTCapabilityProbe.swift +++ b/Sources/AetherEngine/Video/VTCapabilityProbe.swift @@ -68,6 +68,13 @@ enum VTCapabilityProbe { return hardwareDecodeVerdict(codecpar: codecpar).keepsNativeRoute } + /// Strict form of the gate for routes that need a PROVEN hardware session, such as the dual-layer path: + /// `.unclassifiable` is false here, unlike `canHardwareDecode`, because `SoftwarePlaybackHost` would + /// hand an unclassifiable stream to libavcodec. + static func opensHardwareDecoder(codecpar: UnsafePointer) -> Bool { + hardwareDecodeVerdict(codecpar: codecpar).opensHardwareDecoder + } + /// The throwaway session is invalidated immediately; the whole probe costs well under a millisecond and /// runs once per consult. static func hardwareDecodeVerdict(codecpar: UnsafePointer) -> HardwareDecodeVerdict { diff --git a/Sources/AetherEngine/Video/VideoConfigRecord+InBand.swift b/Sources/AetherEngine/Video/VideoConfigRecord+InBand.swift new file mode 100644 index 000000000..c18526e4b --- /dev/null +++ b/Sources/AetherEngine/Video/VideoConfigRecord+InBand.swift @@ -0,0 +1,140 @@ +import Foundation +import AetherLibavcodec +import AetherLibavformat +import AetherLibavutil + +/// HEVC sources whose parameter sets live in-band (`hev1` written with `xps_inband`, the common +/// Dolby Vision MP4 recipes) ship an hvcC with `numOfArrays = 0`. AVPlayer reads the parameter sets +/// out of the samples, but every consumer that builds a format description from the record alone +/// cannot: the hardware-decode probe, `HardwareVideoDecoder`, and the dual-layer decoder's +/// picture-order seed. The native path rebuilds the record for its init.mp4 (#19); these helpers are +/// that rebuild, shared, plus an in-place completion of the demuxer's own record for the +/// software route. +extension VideoConfigRecord { + + /// The VPS / SPS / PPS an in-band scan collects, first of each. + struct HEVCParameterSets { + var vps: [UInt8]? + var sps: [UInt8]? + var pps: [UInt8]? + + var isComplete: Bool { vps != nil && sps != nil && pps != nil } + + /// Walk one length-prefixed packet and keep the first VPS / SPS / PPS seen. Only top-level + /// NAL units count: a Profile 7 enhancement layer's parameter sets travel inside `unspec63` + /// (type 63) wrappers and are never mistaken for the base layer's. + mutating func collect(from data: UnsafePointer, size: Int, naluLengthSize: Int = 4) { + var offset = 0 + while offset + naluLengthSize <= size { + var nalLen = 0 + for i in 0.. size { break } + let nalType = (Int(data[offset]) >> 1) & 0x3F // HEVC NAL type: bits 1..6 of byte 0 + switch nalType { + case 32 where vps == nil: vps = Array(UnsafeBufferPointer(start: data + offset, count: nalLen)) + case 33 where sps == nil: sps = Array(UnsafeBufferPointer(start: data + offset, count: nalLen)) + case 34 where pps == nil: pps = Array(UnsafeBufferPointer(start: data + offset, count: nalLen)) + default: break + } + offset += nalLen + } + } + } + + /// The source record's 22-byte header with `numOfArrays = 3` and one array each for the VPS, + /// SPS and PPS (1-byte type, 2-byte numNalus = 1, 2-byte nalUnitLength, NAL bytes). + static func hevcRecord(header: UnsafePointer, vps: [UInt8], sps: [UInt8], pps: [UInt8]) -> [UInt8] { + var hvcC: [UInt8] = [] + hvcC.reserveCapacity(22 + 1 + 5 * 3 + vps.count + sps.count + pps.count) + for i in 0..<22 { hvcC.append(header[i]) } + hvcC.append(3) + func appendArray(nalUnitType: UInt8, nal: [UInt8]) { + hvcC.append(0x80 | (nalUnitType & 0x3F)) + hvcC.append(0); hvcC.append(1) + let nl = UInt16(nal.count) + hvcC.append(UInt8(nl >> 8)); hvcC.append(UInt8(nl & 0xFF)) + hvcC.append(contentsOf: nal) + } + appendArray(nalUnitType: 32, nal: vps) + appendArray(nalUnitType: 33, nal: sps) + appendArray(nalUnitType: 34, nal: pps) + return hvcC + } + + /// Replace a video stream's empty hvcC with one rebuilt from the parameter sets at the head of the + /// source, so the hardware-decode verdict and `HardwareVideoDecoder` see the record AVPlayer would + /// have reconstructed for itself. Returns true when the record was replaced. + /// + /// Seekable sources only: the scan reads from the head and seeks back to 0, which is where every + /// caller starts playback from anyway (the start position is applied later). A forward-only + /// source is left alone, and keeps today's route. Runs off the main actor; it reads packets. + @discardableResult + static func completeHEVCRecordFromInBandParameterSets(demuxer: Demuxer, streamIndex: Int32) -> Bool { + // Stream access goes through `withStream`, never a pointer held across the packet reads: a + // stop() closes the demuxer from another thread and frees its streams. + let needsRebuild = demuxer.withStream(at: streamIndex) { stream -> Bool in + guard let par = stream.pointee.codecpar, + par.pointee.codec_id == AV_CODEC_ID_HEVC, + let extradata = par.pointee.extradata else { return false } + return HLSVideoEngine.configRecordNeedsInBandRebuild(extradata, size: Int(par.pointee.extradata_size)) + } ?? false + guard needsRebuild, demuxer.isSourceSeekable else { return false } + + // Same budgets as the native rebuild: parameter sets are guaranteed at the first IRAP. + let packetBudget = 16 + let readBudget = 512 + var sets = HEVCParameterSets() + var videoPackets = 0 + var packetsRead = 0 + demuxer.seek(to: 0) + while packetsRead < readBudget, videoPackets < packetBudget, !sets.isComplete { + guard let pkt = try? demuxer.readPacket() else { break } + var toFree: UnsafeMutablePointer? = pkt + defer { trackedPacketFree(&toFree) } + packetsRead += 1 + guard pkt.pointee.stream_index == streamIndex else { continue } + videoPackets += 1 + if let data = pkt.pointee.data { + sets.collect(from: data, size: Int(pkt.pointee.size)) + } + } + demuxer.seek(to: 0) + + guard let vps = sets.vps, let sps = sets.sps, let pps = sets.pps else { + EngineLog.emit( + "[VideoConfigRecord] in-band parameter-set scan found none: packets=\(videoPackets) " + + "vps=\(sets.vps != nil) sps=\(sets.sps != nil) pps=\(sets.pps != nil)", + category: .engine + ) + return false + } + let installed = demuxer.withStream(at: streamIndex) { stream -> (Int32, Int)? in + guard let par = stream.pointee.codecpar, + let extradata = par.pointee.extradata, + HLSVideoEngine.configRecordNeedsInBandRebuild(extradata, size: Int(par.pointee.extradata_size)) + else { return nil } + let record = hevcRecord(header: extradata, vps: vps, sps: sps, pps: pps) + let pad = Int(AV_INPUT_BUFFER_PADDING_SIZE) + guard let buf = av_malloc(record.count + pad) else { return nil } + let typed = buf.assumingMemoryBound(to: UInt8.self) + record.withUnsafeBufferPointer { src in + if let base = src.baseAddress { memcpy(typed, base, record.count) } + } + memset(typed + record.count, 0, pad) + let oldSize = par.pointee.extradata_size + av_freep(&par.pointee.extradata) + par.pointee.extradata = typed + par.pointee.extradata_size = Int32(record.count) + return (oldSize, record.count) + } ?? nil + guard let (oldSize, newSize) = installed else { return false } + EngineLog.emit( + "[VideoConfigRecord] completed hvcC from in-band parameter sets: \(oldSize) B → \(newSize) B", + category: .engine + ) + return true + } +} diff --git a/Tests/AetherEngineTests/DocumentedConstantsTests.swift b/Tests/AetherEngineTests/DocumentedConstantsTests.swift index 2746947a4..866572d99 100644 --- a/Tests/AetherEngineTests/DocumentedConstantsTests.swift +++ b/Tests/AetherEngineTests/DocumentedConstantsTests.swift @@ -45,6 +45,63 @@ final class DocumentedConstantsTests: XCTestCase { assertDocumented("within **64 pictures**", docs) } + func testDualLayerRouteBudgetMatchesDocumentation() throws { + let fel = try felDocument() + XCTAssertEqual(DualLayerRouteDecision.maximumPixelsPerSecond, 311_040_000) + XCTAssertEqual(DualLayerRouteDecision.assumedFrameRate, 24) + XCTAssertEqual(DualLayerRouteDecision.assumedEnhancementLayerAreaFraction, 0.25) + assertDocumented("311.04 Mpx/s", fel) + assertDocumented("UHD\n3840x2160 at 30 fps plus a 1920x1080 enhancement layer at 30 fps", fel) + assertDocumented("an unknown frame rate is budgeted as 24", fel) + assertDocumented("A 24p UHD FEL master needs 80% of the budget", fel) + } + + func testDualLayerPipelineBoundsMatchDocumentation() throws { + let docs = try documentation() + let fel = try felDocument() + XCTAssertEqual(DualLayerVideoDecoder.maxLayerSkewFrames, 3) + XCTAssertEqual(DualLayerVideoDecoder.maxEnhancementLayerQueueFrames, 10) + XCTAssertEqual(FrameSideDataRing.capacity, 32) + XCTAssertEqual(FrameSideDataRing.jitterToleranceTicks, 300) + XCTAssertEqual(DoviFelComposer.maxFramesInFlight, 3) + XCTAssertEqual(SampleBufferRenderer.felVideoLookaheadFrames, 9) + XCTAssertEqual(DoviFelComposer.outputPoolAllocationThreshold, 15) + XCTAssertEqual(DoviFelComposer.outputPoolReserve, 3) + XCTAssertEqual(DualLayerVideoDecoder.maxBorrowedParamsRun, 3) + XCTAssertEqual(SampleBufferRenderer.hardwareDecoderReorderDepth, 4) + XCTAssertEqual(DualLayerVideoDecoder.settledKeyMemory, 64) + XCTAssertEqual(DualLayerVideoDecoder.summaryIntervalSeconds, 1.0) + // About 36 ms: ten attempts with a sleep between each pair. + XCTAssertEqual(Double(DoviFelComposer.outputPoolAttempts - 1) * DoviFelComposer.outputPoolRetryInterval, + 0.036, accuracy: 1e-9) + + assertDocumented("once K = 3 newer base-layer", fel) + assertDocumented("is the maximum layer skew, 3;", fel) + assertDocumented("waiting enhancement frames capped at 10", fel) + assertDocumented("At most 10 enhancement frames wait", fel) + assertDocumented("fixed capacity 32", fel) + assertDocumented("within ±300 ticks", fel) + assertDocumented("wait for one of 3 in-flight slots", fel) + assertDocumented("look-ahead (9)", fel) + assertDocumented("the in-flight slots (3) + 1, so 15", fel) + assertDocumented("a reserve of 3 more", fel) + assertDocumented("allocation threshold 15 + reserve 3", fel) + assertDocumented("`maxBorrowedParamsRun` (3)", fel) + assertDocumented("the renderer's 4-frame reorder buffer", fel) + assertDocumented("last 64\n settled base-layer keys", fel) + assertDocumented("for at most about 36 ms", fel) + assertDocumented("Counters feed a 1 Hz summary", fel) + _ = docs + } + + private func felDocument() throws -> String { + let url = Self.repoRoot.appendingPathComponent("docs/dolby-vision-fel.md") + guard let text = try? String(contentsOf: url, encoding: .utf8) else { + throw XCTSkip("running outside a source checkout; no docs to check against") + } + return text + } + // MARK: - Documentation corpus private static var repoRoot: URL { @@ -345,4 +402,11 @@ final class DocumentedConstantsTests: XCTestCase { else { throw XCTSkip("running outside a source checkout; no sources to check against") } return text } + + // MARK: - Picture-order pairing (docs/dolby-vision-fel.md §5.3) + + func testPictureOrderPairingBoundMatchesDocumentation() throws { + XCTAssertEqual(DualLayerPictureAlignment.maxSlotDistance, 16) + assertDocumented("more than 16 slots off its unit", try felDocument()) + } } diff --git a/Tests/AetherEngineTests/DualLayerEdgeCaseTests.swift b/Tests/AetherEngineTests/DualLayerEdgeCaseTests.swift new file mode 100644 index 000000000..233cbb465 --- /dev/null +++ b/Tests/AetherEngineTests/DualLayerEdgeCaseTests.swift @@ -0,0 +1,633 @@ +import Testing +import CoreMedia +import CoreVideo +import Foundation +import AetherLibavformat +@_spi(Diagnostics) @testable import AetherEngine + +/// Edge cases of the Profile 7 FEL dual-layer route that the other `DualLayer*` suites do not pin: +/// a residual that switches off and on mid-title, unclassifiable first RPUs, damaged or late enhancement +/// layers, the pixel-rate budget at its boundary, 1001-based timebases, and seeks that land on 0, on a +/// replayed range, or in a burst. Everything is synthetic: tagged pixel buffers into the frame handlers, +/// a recording composer, no media and no GPU. +@Suite(.serialized, .timeLimit(.minutes(1))) +struct DualLayerEdgeCaseTests { + + // MARK: - Fixtures + + private static func buffer(width: Int, height: Int) -> CVPixelBuffer? { + var out: CVPixelBuffer? + let status = CVPixelBufferCreate(kCFAllocatorDefault, width, height, + kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange, + [kCVPixelBufferIOSurfacePropertiesKey: [:] as CFDictionary] as CFDictionary, + &out) + return status == kCVReturnSuccess ? out : nil + } + + private static func baseLayer() throws -> CVPixelBuffer { try #require(buffer(width: 64, height: 64)) } + private static func enhancementLayer() throws -> CVPixelBuffer { try #require(buffer(width: 32, height: 32)) } + + private static func params() -> DoviMetalParams { + var p = DoviMetalParams() + p.blWidth = 64; p.blHeight = 64; p.elWidth = 32; p.elHeight = 32 + return p + } + + /// 23.976 fps exactly: frame n at n * 1001 / 24000 s. + private static func pts2397(_ frame: Int64) -> CMTime { CMTime(value: frame * 1001, timescale: 24000) } + /// 24 fps at 90 kHz. + private static func pts(_ frame: Int64) -> CMTime { CMTime(value: frame * 3750, timescale: 90000) } + + private struct Pair { let base: CVPixelBuffer; let enhancement: CVPixelBuffer; let flags: DoviComposerFlags } + + /// Completes every composition at once with the base buffer and records what it was asked to compose. + private final class RecordingComposer: FelComposing, @unchecked Sendable { + private let lock = NSLock() + private var _pairs: [Pair] = [] + private var _flushes = 0 + var pairs: [Pair] { lock.lock(); defer { lock.unlock() }; return _pairs } + var flushCount: Int { lock.lock(); defer { lock.unlock() }; return _flushes } + func acquireSlot() {} + func releaseSlot() {} + func flush() { lock.lock(); _flushes += 1; lock.unlock() } + func composeAsync(baseBuffer: CVPixelBuffer, enhancementBuffer: CVPixelBuffer, params: DoviMetalParams, + frameIndex: UInt32, completion: @escaping @Sendable (CVPixelBuffer?) -> Void) { + composeAsyncWithAcquiredSlot(baseBuffer: baseBuffer, enhancementBuffer: enhancementBuffer, + params: params, frameIndex: frameIndex, completion: completion) + } + func composeAsyncWithAcquiredSlot(baseBuffer: CVPixelBuffer, enhancementBuffer: CVPixelBuffer, + params: DoviMetalParams, frameIndex: UInt32, + completion: @escaping @Sendable (CVPixelBuffer?) -> Void) { + lock.lock() + _pairs.append(Pair(base: baseBuffer, enhancement: enhancementBuffer, flags: params.flags)) + lock.unlock() + completion(baseBuffer) + } + } + + /// A VideoToolbox session stand-in that only counts flushes. + private final class FakeSession: SingleLayerVideoDecoder, @unchecked Sendable { + private let lock = NSLock() + private var _flushes = 0 + var flushCount: Int { lock.lock(); defer { lock.unlock() }; return _flushes } + var onFrame: DecodedFrameHandler? + var onFirstHDR10PlusDetected: (@Sendable () -> Void)? + var onA53Captions: (@Sendable ([CCDataParser.CCTriplet], Double) -> Void)? + var onDecodedFormat: (@Sendable (DecodedVideoFormat) -> Void)? + var skipUntilPTS: CMTime? + var feedEpoch: UInt64 = 0 + func open(stream: UnsafeMutablePointer, onFrame: @escaping DecodedFrameHandler) throws {} + func open(parameterSets: [[UInt8]], timeBase: AVRational, + onFrame: @escaping DecodedFrameHandler) throws {} + func decode(packet: UnsafeMutablePointer, epoch: UInt64?) {} + func flush() { lock.lock(); _flushes += 1; lock.unlock() } + func close() {} + } + + private final class Deliveries: @unchecked Sendable { + struct Frame { let buffer: CVPixelBuffer; let pts: CMTime; let afterMarker: Bool } + private let lock = NSLock() + private var frames: [Frame] = [] + private var marker = false + func record(_ buffer: CVPixelBuffer, _ pts: CMTime) { + lock.lock(); frames.append(Frame(buffer: buffer, pts: pts, afterMarker: marker)); lock.unlock() + } + func setMarker() { lock.lock(); marker = true; lock.unlock() } + var all: [Frame] { lock.lock(); defer { lock.unlock() }; return frames } + var afterMarker: [Frame] { all.filter(\.afterMarker) } + var count: Int { lock.lock(); defer { lock.unlock() }; return frames.count } + } + + private struct Rig { + let decoder: DualLayerVideoDecoder + let composer: RecordingComposer + let primary: FakeSession + let enhancement: FakeSession + let deliveries: Deliveries + } + + private static func rig() -> Rig { + let composer = RecordingComposer() + let primary = FakeSession(), enhancement = FakeSession() + let deliveries = Deliveries() + let decoder = DualLayerVideoDecoder(composer: composer, primaryDecoder: primary, + enhancementDecoder: enhancement) + decoder.onFrame = { buffer, pts, _ in deliveries.record(buffer, pts) } + decoder.primeForTesting(enhancementLayer: true) + return Rig(decoder: decoder, composer: composer, primary: primary, enhancement: enhancement, + deliveries: deliveries) + } + + private static func settle() async throws { + try await waitFor(upTo: .milliseconds(300)) { false } + } + + private static func isStrictlyIncreasing(_ frames: [Deliveries.Frame]) -> Bool { + zip(frames, frames.dropFirst()).allSatisfy { CMTimeCompare($0.pts, $1.pts) < 0 } + } + + // MARK: - 1. Residual switches off and on mid-title + + @Test("an RPU that disables the residual mid-title composes base-only for those frames, FEL resumes after, nothing fails") + func residualDisabledMidTitleThenResumes() async throws { + let r = Self.rig() + defer { r.decoder.close() } + var noResidual = Self.params(); noResidual.flags.insert(.disableResidual) + // Frames 3...5 carry an RPU with no NLQ (what a FEL to P8-shaped switch looks like to the kernel). + let count = 9 + var bases: [CVPixelBuffer] = [] + for i in 0.. DualLayerRouteDecision { + DualLayerRouteDecision.decide( + codecID: AV_CODEC_ID_HEVC, dvProfile: 7, dolbyVisionHandling: .automatic, + preferDualLayerDolbyVision: true, isLive: false, isSourceSeekable: true, + canHardwareDecode: { true }, elType: elType, + geometry: geometry ?? .init(width: 3840, height: 2160, frameRate: 24000.0 / 1001.0)) + } + #expect(decide(elType: "MEL").logLine + == "[AetherEngine] DV P7 dual-layer route: not taken (el_type=MEL, nothing to compose), keeps the native route (8.1 where the display takes Dolby Vision)") + #expect(decide(elType: nil).logLine + == "[AetherEngine] DV P7 dual-layer route: not taken (el_type unknown, no enhancement layer classified in the first RPU), keeps the native route (8.1 where the display takes Dolby Vision)") + // The el_type is judged before the budget, so a MEL or unclassified 4K60 still says MEL / unknown: + // the budget only ever speaks about FEL. + let uhd60 = DualLayerRouteDecision.Geometry(width: 3840, height: 2160, frameRate: 60) + #expect(decide(elType: "MEL", geometry: uhd60) == .notTaken(.melEnhancementLayer)) + #expect(decide(elType: nil, geometry: uhd60) == .notTaken(.enhancementLayerTypeUnknown)) + } + + @Test("first RPUs that cannot be classified leave the stream undetermined and the base layer playing; the first definitive RPU then decides") + func unclassifiableFirstRPUsThenFEL() async throws { + let decoder = DualLayerVideoDecoder(composer: RecordingComposer()) + let deliveries = Deliveries() + decoder.onFrame = { buffer, pts, _ in deliveries.record(buffer, pts) } + defer { decoder.close() } + let undetermined = DoviRpuConverter.DualLayerPacketResult( + elPacket: nil, rpuData: nil, rpuParams: nil, isFEL: false, isMEL: false, success: false) + for _ in 0..<5 { decoder.classifyPacketResultForTesting(undetermined) } + #expect(decoder.streamClassificationForTesting == .undetermined) + #expect(!decoder.engagesEnhancementLayerForTesting()) + + // Undetermined means no enhancement decoder: frames are delivered as their base layer. + let bl = try Self.baseLayer() + decoder.handleBaseLayerFrame(baseBuffer: bl, pts: Self.pts(0), hdr10Data: nil) + try await waitFor { deliveries.count == 1 } + #expect(deliveries.all.first?.buffer === bl) + + let fel = DoviRpuConverter.DualLayerPacketResult( + elPacket: nil, rpuData: nil, rpuParams: nil, isFEL: true, isMEL: false, success: true) + decoder.classifyPacketResultForTesting(fel) + #expect(decoder.streamClassificationForTesting == .fel) + #expect(decoder.engagesEnhancementLayerForTesting()) + } + + // MARK: - 3. Damaged, missing and late enhancement layers + + @Test("an enhancement layer missing for some base-layer frames costs those frames their residual, in order, with bounded waiting; pairing recovers") + func missingEnhancementLayersMidStream() async throws { + let r = Self.rig() + defer { r.decoder.close() } + let missing: Set = [3, 7, 8] + let count = 12 + var bases: [CVPixelBuffer] = [], enhancements: [Int: CVPixelBuffer] = [:] + for i in 0.. DualLayerRouteDecision { + DualLayerRouteDecision.decide( + codecID: AV_CODEC_ID_HEVC, dvProfile: 7, dolbyVisionHandling: .automatic, + preferDualLayerDolbyVision: true, isLive: false, isSourceSeekable: true, + canHardwareDecode: { true }, elType: "FEL", geometry: geometry, enhancementLayerSize: elSize) + } + + @Test("4K at 23.976, 24, 25, 29.97 and 30 is accepted", arguments: [24000.0 / 1001.0, 24.0, 25.0, 30000.0 / 1001.0, 30.0]) + func uhdAcceptedRates(rate: Double) { + let decision = decide(Geometry(width: 3840, height: 2160, frameRate: rate)) + guard case let .taken(_, _, fraction) = decision else { + Issue.record("UHD at \(rate) fps should be inside the budget: \(decision)"); return + } + #expect(fraction <= 1.0) + } + + @Test("4K at 50, 59.94 and 60 is refused cleanly, with the fallback named in the line", + arguments: [50.0, 60000.0 / 1001.0, 60.0]) + func uhdRefusedRates(rate: Double) throws { + let decision = decide(Geometry(width: 3840, height: 2160, frameRate: rate)) + guard case let .notTaken(.overBudget(required, _)) = decision else { + Issue.record("UHD at \(rate) fps should be over budget: \(decision)"); return + } + #expect(required > DualLayerRouteDecision.maximumPixelsPerSecond) + let line = try #require(decision.logLine) + #expect(line.hasPrefix("[AetherEngine] DV P7 dual-layer route: not taken (over the real-time budget")) + #expect(line.hasSuffix("keeps the native route (8.1 where the display takes Dolby Vision)")) + #expect(!decision.isTaken) + } + + @Test("an unknown base-layer size is refused with its own reason, after MEL and unknown el_type") + func unknownGeometryFailsClosed() { + for geometry in [Geometry(width: 0, height: 0, frameRate: nil), Geometry(width: 3840, height: 0, frameRate: 24), + Geometry(width: -1, height: 2160, frameRate: 24)] { + let decision = decide(geometry) + #expect(decision == .notTaken(.geometryUnknown)) + #expect(decision.logLine == "[AetherEngine] DV P7 dual-layer route: not taken (base-layer size unknown), keeps the native route (8.1 where the display takes Dolby Vision)") + for elType in ["MEL", nil] as [String?] { + let other = DualLayerRouteDecision.decide( + codecID: AV_CODEC_ID_HEVC, dvProfile: 7, dolbyVisionHandling: .automatic, + preferDualLayerDolbyVision: true, isLive: false, isSourceSeekable: true, + canHardwareDecode: { true }, elType: elType, geometry: geometry) + #expect(other == .notTaken(elType == nil ? .enhancementLayerTypeUnknown : .melEnhancementLayer)) + } + } + } + + @Test("the budget is a closed bound: exactly 311.04 Mpx/s is in, the next representable rate above it is out") + func exactBudgetBoundary() { + // 1080p with the assumed quarter-area EL needs 2073600 * 1.25 * fps: exactly the budget at 120 fps. + let atLimit = Geometry(width: 1920, height: 1080, frameRate: 120) + #expect(DualLayerRouteDecision.requiredPixelsPerSecond(geometry: atLimit) == DualLayerRouteDecision.maximumPixelsPerSecond) + #expect(decide(atLimit).isTaken) + #expect(decide(Geometry(width: 1920, height: 1080, frameRate: 120.0.nextUp)) == .notTaken(.overBudget( + requiredPixelsPerSecond: DualLayerRouteDecision.requiredPixelsPerSecond( + geometry: Geometry(width: 1920, height: 1080, frameRate: 120.0.nextUp)), + geometry: Geometry(width: 1920, height: 1080, frameRate: 120.0.nextUp)))) + // The same bound reached with the EL stated 1:1 (two full 1080p layers) at 75 fps. + let fullEL = (width: 1920, height: 1080) + let at75 = Geometry(width: 1920, height: 1080, frameRate: 75) + #expect(DualLayerRouteDecision.requiredPixelsPerSecond(geometry: at75, enhancementLayerSize: fullEL) + == DualLayerRouteDecision.maximumPixelsPerSecond) + #expect(decide(at75, elSize: fullEL).isTaken) + #expect(!decide(Geometry(width: 1920, height: 1080, frameRate: 75.0.nextUp), elSize: fullEL).isTaken) + } + + @Test("an enhancement layer at full resolution (1:1) doubles the need: UHD24 is refused, UHD at 15 fps is the 1:1 limit") + func fullResolutionEnhancementLayer() { + let uhd = Geometry(width: 3840, height: 2160, frameRate: 24) + #expect(decide(uhd).isTaken, "the assumed quarter-area EL fits at 24 fps") + #expect(!decide(uhd, elSize: (3840, 2160)).isTaken, "398 Mpx/s needed") + let uhd15 = Geometry(width: 3840, height: 2160, frameRate: 15) + #expect(decide(uhd15, elSize: (3840, 2160)).isTaken) + // An EL smaller than a quarter makes more room, never less. + #expect(DualLayerRouteDecision.requiredPixelsPerSecond(geometry: uhd, enhancementLayerSize: (960, 540)) + < DualLayerRouteDecision.requiredPixelsPerSecond(geometry: uhd)) + } + + @Test("odd and non-UHD sizes are budgeted by their exact pixel count, not rounded to a tier") + func oddSizes() { + // One pixel more than UHD on each axis at the 30 fps ceiling tips it over. + #expect(decide(Geometry(width: 3840, height: 2160, frameRate: 30)).isTaken) + #expect(!decide(Geometry(width: 3841, height: 2161, frameRate: 30)).isTaken) + // DCI 4K (4096x2160): in at 24, out at 30. + #expect(decide(Geometry(width: 4096, height: 2160, frameRate: 24)).isTaken) + #expect(!decide(Geometry(width: 4096, height: 2160, frameRate: 30)).isTaken) + let odd = Geometry(width: 1921, height: 1081, frameRate: 24) + #expect(DualLayerRouteDecision.requiredPixelsPerSecond(geometry: odd) == Double(1921 * 1081) * 1.25 * 24) + // The enhancement layer's own odd size is taken as stated. + #expect(DualLayerRouteDecision.requiredPixelsPerSecond(geometry: odd, enhancementLayerSize: (961, 541)) + == Double(1921 * 1081 + 961 * 541) * 24) + } + + // MARK: - 5. 23.976 against 24: 1001-based timebases + + @Test("canonical keys of 23.976 frames in 1/24000, 1001/24000, 1/1000 and 1/90000 timebases never get within the jitter of a neighbour, and agree across timebases") + func timebaseKeysAt2397() { + let tolerance = FrameSideDataRing.jitterToleranceTicks + let timebases: [(AVRational, (Int64) -> Int64)] = [ + (AVRational(num: 1, den: 24000), { $0 * 1001 }), + (AVRational(num: 1001, den: 24000), { $0 }), + (AVRational(num: 1, den: 1000), { Int64((Double($0) * 1001.0 / 24.0).rounded()) }), + (AVRational(num: 1, den: 90000), { Int64((Double($0) * 3753.75).rounded()) }), + ] + let frames: Int64 = 86_400 // one hour of 23.976 + var previous = [Int64](repeating: Int64.min, count: timebases.count) + for n in 0.. 0 { + let gap = key - previous[i] + guard gap > 2 * tolerance else { + Issue.record("timebase \(i): frames \(n - 1) and \(n) are only \(gap) ticks apart"); return + } + } + previous[i] = key + } + // The same frame stamped in a coarser or finer timebase still pairs. + for k in keys.dropFirst() where !FrameSideDataRing.isWithinJitter(keys[0], k) { + Issue.record("frame \(n): keys \(keys) differ by more than the pairing jitter"); return + } + } + // The exact 1001 grid is 3753.75 ticks per frame: rounding gives only 3753 or 3754 spacing. + #expect(DualLayerVideoDecoder.canonicalKey(rawPTS: 1001, timeBase: AVRational(num: 1, den: 24000)) == 3754) + #expect(DualLayerVideoDecoder.canonicalKey(rawPTS: 2002, timeBase: AVRational(num: 1, den: 24000)) == 7508) + #expect(DualLayerVideoDecoder.canonicalKey(rawPTS: 3003, timeBase: AVRational(num: 1, den: 24000)) == 11261) + } + + @Test("at 23.976 a seek realigns on the right picture: an early neighbour's enhancement layer is never paired with the base layer, even stamped in milliseconds") + func seekRealignmentAt2397() async throws { + let r = Self.rig() + defer { r.decoder.close() } + // Pre-seek: a later stretch, left half-paired. + for i in 100..<103 { + r.decoder.handleEnhancementLayerFrame(elBuffer: try Self.enhancementLayer(), pts: Self.pts2397(Int64(i))) + } + r.decoder.handleBaseLayerFrame(baseBuffer: try Self.baseLayer(), pts: Self.pts2397(103), hdr10Data: nil) + r.decoder.flush() + r.deliveries.setMarker() + + // Seek back to frame 50. The EL decoder ran ahead: frame 51's EL (in ms, as a 1/1000 stream would + // stamp it) comes before frame 50's base layer. + func ms(_ frame: Int64) -> CMTime { CMTime(value: Int64((Double(frame) * 1001.0 / 24.0).rounded()), timescale: 1000) } + var bases: [CVPixelBuffer] = [], enhancements: [CVPixelBuffer] = [] + for i in 50..<56 { + r.decoder.seedRPUForTesting(pts: Self.pts2397(Int64(i)), params: Self.params(), displayMetadataNAL: nil) + bases.append(try Self.baseLayer()); enhancements.append(try Self.enhancementLayer()) + } + r.decoder.handleEnhancementLayerFrame(elBuffer: enhancements[1], pts: ms(51)) + r.decoder.handleBaseLayerFrame(baseBuffer: bases[0], pts: Self.pts2397(50), hdr10Data: nil) + r.decoder.handleEnhancementLayerFrame(elBuffer: enhancements[0], pts: ms(50)) + for i in 1..<6 { + if i != 1 { r.decoder.handleEnhancementLayerFrame(elBuffer: enhancements[i], pts: ms(Int64(50 + i))) } + r.decoder.handleBaseLayerFrame(baseBuffer: bases[i], pts: Self.pts2397(Int64(50 + i)), hdr10Data: nil) + } + try await waitFor { r.deliveries.afterMarker.count == 6 } + let delivered = r.deliveries.afterMarker + #expect(CMTimeCompare(try #require(delivered.first).pts, Self.pts2397(50)) == 0, "the first picture after the seek is the target picture, not its neighbour") + #expect(zip(delivered, bases).allSatisfy { $0.buffer === $1 }) + #expect(delivered.enumerated().allSatisfy { CMTimeCompare($0.element.pts, Self.pts2397(Int64(50 + $0.offset))) == 0 }) + let pairs = r.composer.pairs + #expect(pairs.count == 6) + for pair in pairs { + let i = try #require(bases.firstIndex { $0 === pair.base }) + #expect(pair.enhancement === enhancements[i], "frame \(50 + i) paired with another frame's EL (off by one)") + } + } + + // MARK: - 6. Seek edge cases + + @Test("a seek back to 0 replays timestamps the pre-seek run already settled, and they pair: nothing is dropped as stale") + func seekBackToZero() async throws { + let r = Self.rig() + defer { r.decoder.close() } + // Play the first frames from the start, as a fresh load does. + for i in 0..<4 { + r.decoder.seedRPUForTesting(pts: Self.pts(Int64(i)), params: Self.params(), displayMetadataNAL: nil) + r.decoder.handleEnhancementLayerFrame(elBuffer: try Self.enhancementLayer(), pts: Self.pts(Int64(i))) + r.decoder.handleBaseLayerFrame(baseBuffer: try Self.baseLayer(), pts: Self.pts(Int64(i)), hdr10Data: nil) + } + try await waitFor { r.deliveries.count == 4 } + // Seek away and back to the beginning (two flushes), then replay with the EL first. + r.decoder.flush() + r.decoder.flush() + r.deliveries.setMarker() + var bases: [CVPixelBuffer] = [] + for i in 0..<4 { + r.decoder.seedRPUForTesting(pts: Self.pts(Int64(i)), params: Self.params(), displayMetadataNAL: nil) + let bl = try Self.baseLayer() + bases.append(bl) + r.decoder.handleEnhancementLayerFrame(elBuffer: try Self.enhancementLayer(), pts: Self.pts(Int64(i))) + r.decoder.handleBaseLayerFrame(baseBuffer: bl, pts: Self.pts(Int64(i)), hdr10Data: nil) + } + try await waitFor { r.deliveries.afterMarker.count == 4 } + let after = r.deliveries.afterMarker + #expect(after.first?.pts == .zero, "the picture at time 0 is delivered") + #expect(zip(after, bases).allSatisfy { $0.buffer === $1 }) + #expect(r.composer.pairs.count == 8, "all eight frames, both passes, were composed") + #expect(r.decoder.composedFrameCount == 8) + } + + @Test("a seek next to the end of the file composes the last pair, and the end-of-stream flush leaves a clean, repeatable state") + func seekNearEndOfFile() async throws { + let r = Self.rig() + defer { r.decoder.close() } + let last: Int64 = 172_000 // ~2 h at 24 fps + r.decoder.flush() // the seek + r.deliveries.setMarker() + for i in (last - 1)...last { + r.decoder.seedRPUForTesting(pts: Self.pts(i), params: Self.params(), displayMetadataNAL: nil) + r.decoder.handleBaseLayerFrame(baseBuffer: try Self.baseLayer(), pts: Self.pts(i), hdr10Data: nil) + r.decoder.handleEnhancementLayerFrame(elBuffer: try Self.enhancementLayer(), pts: Self.pts(i)) + } + try await waitFor { r.deliveries.afterMarker.count == 2 } + #expect(CMTimeCompare(try #require(r.deliveries.afterMarker.last).pts, Self.pts(last)) == 0) + r.decoder.flush() + r.decoder.flush() + let s = r.decoder.stateSnapshotForTesting + #expect(s.blQueueCount == 0 && s.elQueueCount == 0 && s.rpuCacheCount == 0 && !s.draining) + #expect(r.primary.flushCount == 3 && r.enhancement.flushCount == 3 && r.composer.flushCount == 3) + } + + @Test("a scrub burst of seeks flushes both sessions and the composer every time, and only the last position's frames come out") + func scrubBurst() async throws { + let r = Self.rig() + defer { r.decoder.close() } + let seeks = 40 + let epoch = r.decoder.feedEpoch + for s in 0.. Bool { + VideoRoutingPolicy.requiresSoftwarePath( + codecID: AV_CODEC_ID_HEVC, fieldOrder: AV_FIELD_PROGRESSIVE, av1Available: true, + dvProfile: 7, dolbyVisionHandling: .automatic, + preferDualLayerDolbyVision: prefer, canHardwareDecode: { true }, elType: "FEL") + } + #expect(requiresSoftware(prefer: true)) + #expect(!requiresSoftware(prefer: false)) + } + + @Test("E-AC-3 is an AVPlayer-native codec, so only the software host's own decode path explains the core-only audio") + func eac3IsNativeToAVPlayer() { + #expect(AetherEngine.avPlayerCanDecodeAudio(AV_CODEC_ID_EAC3)) + } + + @Test("the software host's AudioDecoder opens an E-AC-3 stream and the delivery is the host's own decode") + func softwareHostDecodesEAC3Itself() throws { + let formatContext = avformat_alloc_context() + defer { avformat_free_context(formatContext) } + let stream = try #require(avformat_new_stream(formatContext, nil)) + let codecpar = try #require(stream.pointee.codecpar) + codecpar.pointee.codec_type = AVMEDIA_TYPE_AUDIO + codecpar.pointee.codec_id = AV_CODEC_ID_EAC3 + codecpar.pointee.sample_rate = 48_000 + av_channel_layout_default(&codecpar.pointee.ch_layout, 6) + stream.pointee.time_base = AVRational(num: 1, den: 48_000) + + let decoder = AudioDecoder() + defer { decoder.close() } + try decoder.open(stream: stream) + #expect(decoder.sampleRate == 48_000) + #expect(decoder.channels == 6) + + #expect(SoftwarePlaybackHost.audioDelivery( + resolvedAudioIndex: stream.pointee.index, decoderOpened: true, sourceCarriesAudio: true) == .decoded) + } +} diff --git a/Tests/AetherEngineTests/DualLayerRouteDecisionTests.swift b/Tests/AetherEngineTests/DualLayerRouteDecisionTests.swift new file mode 100644 index 000000000..90d81005f --- /dev/null +++ b/Tests/AetherEngineTests/DualLayerRouteDecisionTests.swift @@ -0,0 +1,213 @@ +import Testing +import AetherLibavcodec +@testable import AetherEngine + +/// The Profile 7 dual-layer route is FEL only and capability gated: MEL, an unclassified enhancement +/// layer, a box without a hardware HEVC decoder and a stream too heavy to decode twice in real time all +/// keep the native route, and the load logs one line saying which. These tests hold every exit of that decision. +@Suite("Dual-layer route decision") +struct DualLayerRouteDecisionTests { + + private typealias Geometry = DualLayerRouteDecision.Geometry + private let uhd24 = Geometry(width: 3840, height: 2160, frameRate: 24000.0 / 1001.0) + + private func decide( + codec: AVCodecID = AV_CODEC_ID_HEVC, profile: Int? = 7, + handling: DolbyVisionHandling = .automatic, prefer: Bool = true, + live: Bool = false, seekable: Bool = true, hardware: (() -> Bool)? = { true }, + elType: String? = "FEL", geometry: Geometry? = nil + ) -> DualLayerRouteDecision { + DualLayerRouteDecision.decide( + codecID: codec, dvProfile: profile, dolbyVisionHandling: handling, + preferDualLayerDolbyVision: prefer, isLive: live, isSourceSeekable: seekable, + canHardwareDecode: hardware, elType: elType, geometry: geometry ?? uhd24) + } + + @Test("a 24p UHD FEL source on a capable box takes the route and logs the budget it uses") + func felTakesTheRoute() { + let decision = decide() + #expect(decision.isTaken) + #expect(decision.logLine == "[AetherEngine] DV P7 dual-layer route: taken (el_type=FEL, 3840x2160@23.976, budget 80%)") + } + + @Test("MEL never takes the route, with the flag on") + func melNeverTakesTheRoute() { + let decision = decide(prefer: true, elType: "MEL") + #expect(decision == .notTaken(.melEnhancementLayer)) + #expect(decision.logLine?.contains("el_type=MEL") == true) + #expect(decision.logLine?.hasSuffix("keeps the native route (8.1 where the display takes Dolby Vision)") == true) + } + + @Test("an enhancement layer the probe could not classify does not take the route") + func unknownElTypeDoesNotTakeTheRoute() { + #expect(decide(elType: nil) == .notTaken(.enhancementLayerTypeUnknown)) + #expect(decide(elType: "other") == .notTaken(.enhancementLayerTypeUnknown)) + } + + @Test("every other exit names its reason") + func everyExitHasItsReason() { + #expect(decide(prefer: false) == .notTaken(.preferenceOff)) + #expect(decide(handling: .baseLayerOnly) == .notTaken(.baseLayerOnlyHandling)) + #expect(decide(live: true) == .notTaken(.liveOrNotSeekable)) + #expect(decide(seekable: false) == .notTaken(.liveOrNotSeekable)) + #expect(decide(hardware: { false }) == .notTaken(.noHardwareHEVC)) + #expect(decide(profile: 8) == .notTaken(.notProfile7)) + #expect(decide(profile: nil) == .notTaken(.notProfile7)) + #expect(decide(codec: AV_CODEC_ID_H264) == .notTaken(.notProfile7)) + } + + @Test("a missing hardware verdict assumes hardware decode, as the routing policy does") + func nilHardwareVerdictAssumesHardware() { + #expect(decide(hardware: nil).isTaken) + } + + @Test("a source that is not Profile 7 logs nothing; every Profile 7 outcome logs exactly one line") + func onlyProfile7LogsALine() { + #expect(decide(profile: 8).logLine == nil) + #expect(decide(codec: AV_CODEC_ID_H264, prefer: false).logLine == nil) + let reasons = [ + decide(prefer: false), decide(handling: .baseLayerOnly), decide(live: true), + decide(hardware: { false }), decide(elType: "MEL"), decide(elType: nil), + decide(geometry: Geometry(width: 3840, height: 2160, frameRate: 60)), + ] + for decision in reasons { + guard let line = decision.logLine else { + Issue.record("a Profile 7 fallback must log its reason: \(decision)") + continue + } + #expect(line.hasPrefix("[AetherEngine] DV P7 dual-layer route: not taken (")) + #expect(!line.contains("\n")) + } + #expect(decide(prefer: false).logLine?.contains("preferDualLayerDolbyVision is off") == true) + #expect(decide(handling: .baseLayerOnly).logLine?.contains("HDR10 base layer") == true) + } + + // MARK: - Real-time budget + + @Test("the budget is UHD at 30 fps plus a 1080p enhancement layer at 30 fps") + func budgetConstant() { + #expect(DualLayerRouteDecision.maximumPixelsPerSecond == 311_040_000) + } + + @Test("the budget boundary: UHD at 30 fps is exactly in, a hair above it is out") + func budgetBoundary() { + let atLimit = decide(geometry: Geometry(width: 3840, height: 2160, frameRate: 30)) + #expect(atLimit.logLine?.contains("budget 100%") == true) + let justOver = decide(geometry: Geometry(width: 3840, height: 2160, frameRate: 30.01)) + guard case .notTaken(.overBudget) = justOver else { + Issue.record("30.01 fps UHD should exceed the budget, got \(justOver)") + return + } + #expect(decide(geometry: Geometry(width: 3840, height: 2160, frameRate: 29.97)).isTaken) + } + + @Test("UHD at 60 fps and 8K at 24 fps are over budget and say how far") + func heavyStreamsFallBack() { + let uhd60 = decide(geometry: Geometry(width: 3840, height: 2160, frameRate: 60)) + #expect(uhd60.logLine == "[AetherEngine] DV P7 dual-layer route: not taken (over the real-time budget, 3840x2160@60 needs 200% of it), keeps the native route (8.1 where the display takes Dolby Vision)") + let uhd8k = decide(geometry: Geometry(width: 7680, height: 4320, frameRate: 24)) + #expect(!uhd8k.isTaken) + } + + @Test("an unknown frame rate is budgeted at 24 and the line says so") + func unknownFrameRateAssumes24() { + let decision = decide(geometry: Geometry(width: 3840, height: 2160, frameRate: nil)) + #expect(decision.logLine == "[AetherEngine] DV P7 dual-layer route: taken (el_type=FEL, 3840x2160@24 (assumed), budget 80%)") + let nonsense = Geometry(width: 3840, height: 2160, frameRate: -1) + #expect(nonsense.effectiveFrameRate == 24) + } + + @Test("the enhancement layer is a quarter of the base layer's area unless the stream says otherwise") + func enhancementLayerSize() { + let geometry = Geometry(width: 3840, height: 2160, frameRate: 30) + let assumed = DualLayerRouteDecision.requiredPixelsPerSecond(geometry: geometry) + #expect(assumed == DualLayerRouteDecision.maximumPixelsPerSecond) + let fullSize = DualLayerRouteDecision.requiredPixelsPerSecond( + geometry: geometry, enhancementLayerSize: (3840, 2160)) + #expect(fullSize > DualLayerRouteDecision.maximumPixelsPerSecond) + } + + @Test("the el_type probe is only wanted when nothing rules the route out first") + func probeRunsOnlyWhenItCanChangeTheRoute() { + func blocked(codec: AVCodecID = AV_CODEC_ID_HEVC, profile: Int? = 7, + handling: DolbyVisionHandling = .automatic, prefer: Bool = true, + live: Bool = false, seekable: Bool = true, + hardware: (() -> Bool)? = { true }) -> DualLayerRouteDecision.Reason? { + DualLayerRouteDecision.blockingReasonBeforeProbe( + codecID: codec, dvProfile: profile, dolbyVisionHandling: handling, + preferDualLayerDolbyVision: prefer, isLive: live, isSourceSeekable: seekable, + canHardwareDecode: hardware) + } + #expect(blocked() == nil) + #expect(blocked(prefer: false) != nil) + #expect(blocked(handling: .baseLayerOnly) != nil) + #expect(blocked(live: true) != nil) + #expect(blocked(seekable: false) != nil) + #expect(blocked(hardware: { false }) != nil) + #expect(blocked(profile: 8) != nil) + } + + // MARK: - Reload of a custom source + + /// A custom source's rebuild keeps the backend and el_type the session was loaded with, so a flip + /// of the flag on a Profile 7 session would be logged as applied and change nothing. It is refused + /// instead, before any teardown; URL sources re-run `load()` and are not. + @Test("flipping preferDualLayerDolbyVision on a custom Profile 7 session is refused, nothing else is") + func customSourceFlagFlipIsRefused() { + var off = LoadOptions() + off.preferDualLayerDolbyVision = false + var on = off + on.preferDualLayerDolbyVision = true + func refusal(custom: Bool, from: LoadOptions, to: LoadOptions, profile: Int?) -> SessionReloadRefusal? { + SessionOptionCorrection.dualLayerRouteRefusal( + isCustomSource: custom, current: from, proposed: to, dvProfile: profile) + } + #expect(refusal(custom: true, from: off, to: on, profile: 7) == .dualLayerRouteNeedsFreshLoad) + #expect(refusal(custom: true, from: on, to: off, profile: 7) == .dualLayerRouteNeedsFreshLoad) + // A URL source re-runs load(), a non-P7 source routes the same either way, an unchanged flag moves nothing. + #expect(refusal(custom: false, from: off, to: on, profile: 7) == nil) + #expect(refusal(custom: true, from: off, to: on, profile: 8) == nil) + #expect(refusal(custom: true, from: off, to: on, profile: nil) == nil) + #expect(refusal(custom: true, from: on, to: on, profile: 7) == nil) + #expect(!SessionReloadRefusal.dualLayerRouteNeedsFreshLoad.description.isEmpty) + } + + /// The route needs `dolbyVisionHandling` `.automatic`, so on a custom Profile 7 session a handling + /// change that can turn the route on or off is refused like a flag flip. With the preference off the + /// handling cannot move the route and is not refused. + @Test("a dolbyVisionHandling change that can move the route on a custom Profile 7 session is refused") + func customSourceHandlingChangeIsRefused() { + var automatic = LoadOptions() + automatic.preferDualLayerDolbyVision = true + automatic.dolbyVisionHandling = .automatic + var baseOnly = automatic + baseOnly.dolbyVisionHandling = .baseLayerOnly + var offAutomatic = automatic + offAutomatic.preferDualLayerDolbyVision = false + var offBaseOnly = offAutomatic + offBaseOnly.dolbyVisionHandling = .baseLayerOnly + func refusal(custom: Bool = true, from: LoadOptions, to: LoadOptions, profile: Int? = 7) -> SessionReloadRefusal? { + SessionOptionCorrection.dualLayerRouteRefusal( + isCustomSource: custom, current: from, proposed: to, dvProfile: profile) + } + #expect(refusal(from: automatic, to: baseOnly) == .dualLayerRouteNeedsFreshLoad) + #expect(refusal(from: baseOnly, to: automatic) == .dualLayerRouteNeedsFreshLoad) + // The preference is off on both sides: the route is not taken either way. + #expect(refusal(from: offAutomatic, to: offBaseOnly) == nil) + // URL sources re-run load(), other profiles route the same, an unchanged handling moves nothing. + #expect(refusal(custom: false, from: automatic, to: baseOnly) == nil) + #expect(refusal(from: automatic, to: baseOnly, profile: 8) == nil) + #expect(refusal(from: automatic, to: automatic) == nil) + } + + /// A fatal decode error is the decoder session's, reported once; only a stopped host drops it, a + /// seek in flight does not. + @Test("a session failure is admitted across a seek and dropped only by a stop") + func sessionFailureIgnoresSeekGeneration() { + #expect(SoftwareReadAdmission.admitsSessionFailure(stopRequested: false)) + #expect(!SoftwareReadAdmission.admitsSessionFailure(stopRequested: true)) + // The per-seek admission it replaces rejects the same event once a seek has opened. + #expect(!SoftwareReadAdmission.admits(readGeneration: 3, requestedGeneration: 4, + settledGeneration: 3, stopRequested: false)) + } +} diff --git a/Tests/AetherEngineTests/InBandParameterSetRebuildTests.swift b/Tests/AetherEngineTests/InBandParameterSetRebuildTests.swift index 14a6c7b1b..713d871bb 100644 --- a/Tests/AetherEngineTests/InBandParameterSetRebuildTests.swift +++ b/Tests/AetherEngineTests/InBandParameterSetRebuildTests.swift @@ -60,4 +60,60 @@ struct InBandParameterSetRebuildTests { #expect(record.count > 23) #expect(record[22] == 3, "numOfArrays must cover VPS, SPS and PPS") } + + /// The Profile 7 FEL route needs a proven hardware session, which the strict verdict cannot give + /// an empty hvcC; the load completes the demuxer's own record so the route and the decoders see + /// what AVPlayer would have rebuilt. + @Test("Completing the record in place makes the strict verdict classifiable and rewinds", + .enabled(if: fixtureExists("hev1-inband-xps.mp4"), + "run Scripts/fetch-fixtures.sh to generate the in-band xPS clip"), + .timeLimit(.minutes(1))) + func completionInPlace() throws { + let demuxer = Demuxer() + try demuxer.open(url: fixtureURL("hev1-inband-xps.mp4"), profile: .playback) + defer { demuxer.close() } + let videoIdx = demuxer.videoStreamIndex + let codecpar = try #require(demuxer.stream(at: videoIdx)?.pointee.codecpar) + #expect(VTCapabilityProbe.hardwareDecodeVerdict(codecpar: codecpar) + == .unclassifiable(reason: "in-band parameter sets")) + + #expect(VideoConfigRecord.completeHEVCRecordFromInBandParameterSets(demuxer: demuxer, streamIndex: videoIdx)) + #expect(codecpar.pointee.extradata_size > 23) + #expect(codecpar.pointee.extradata?[22] == 3) + #expect(VTCapabilityProbe.hardwareDecodeVerdict(codecpar: codecpar) != .unclassifiable(reason: "in-band parameter sets")) + + // Back at the head: the next video packet is the opening keyframe. + var first: (key: Bool, pts: Int64)? + while first == nil, let pkt = try demuxer.readPacket() { + var toFree: UnsafeMutablePointer? = pkt + defer { trackedPacketFree(&toFree) } + if pkt.pointee.stream_index == videoIdx { + first = ((pkt.pointee.flags & AV_PKT_FLAG_KEY) != 0, pkt.pointee.pts) + } + } + #expect(first?.key == true) + #expect(first?.pts == 0) + + // A complete record is left alone. + #expect(!VideoConfigRecord.completeHEVCRecordFromInBandParameterSets(demuxer: demuxer, streamIndex: videoIdx)) + } + + /// A Profile 7 enhancement layer carries its own VPS / SPS / PPS inside `unspec63` NAL units; + /// only the base layer's top-level parameter sets may seed the base layer's record. + @Test("The collector keeps top-level parameter sets and skips enhancement-layer ones") + func collectorSkipsWrappedEnhancementLayerSets() { + func nal(_ bytes: [UInt8]) -> [UInt8] { + let n = bytes.count + return [UInt8(n >> 24), UInt8((n >> 16) & 0xFF), UInt8((n >> 8) & 0xFF), UInt8(n & 0xFF)] + bytes + } + // unspec63 header (0x7E 0x01) wrapping an EL SPS (0x42 0x01), then the BL's VPS, SPS, PPS. + let packet = nal([0x7E, 0x01, 0x42, 0x01, 0xEE]) + + nal([0x40, 0x01, 0xA1]) + nal([0x42, 0x01, 0xB2]) + nal([0x44, 0x01, 0xC3]) + var sets = VideoConfigRecord.HEVCParameterSets() + packet.withUnsafeBufferPointer { sets.collect(from: $0.baseAddress!, size: $0.count) } + #expect(sets.isComplete) + #expect(sets.vps == [0x40, 0x01, 0xA1]) + #expect(sets.sps == [0x42, 0x01, 0xB2]) + #expect(sets.pps == [0x44, 0x01, 0xC3]) + } } diff --git a/Tests/AetherEngineTests/LoadOptionsTests.swift b/Tests/AetherEngineTests/LoadOptionsTests.swift index 6aa3e6590..19c353112 100644 --- a/Tests/AetherEngineTests/LoadOptionsTests.swift +++ b/Tests/AetherEngineTests/LoadOptionsTests.swift @@ -20,6 +20,8 @@ struct LoadOptionsTests { #expect(opts.audioBridgeMode == .surroundCompat) #expect(opts.isLive == false) #expect(opts.audioOnly == false) + // Dual-layer FEL composition is opt-in. + #expect(opts.preferDualLayerDolbyVision == false) // #68: probe-budget overrides default to nil so the engine keeps the // built-in .playback budget (50 MB / 60 s) unless a caller opts in. #expect(opts.probesize == nil) diff --git a/Tests/AetherEngineTests/SoftwareDecoderSelectionTests.swift b/Tests/AetherEngineTests/SoftwareDecoderSelectionTests.swift new file mode 100644 index 000000000..faa314d6e --- /dev/null +++ b/Tests/AetherEngineTests/SoftwareDecoderSelectionTests.swift @@ -0,0 +1,172 @@ +import Testing +import AetherLibavcodec +@testable import AetherEngine + +@Suite("SoftwarePlaybackHost decoder selection") +struct SoftwareDecoderSelectionTests { + + @MainActor + @Test("P7 FEL with default options keeps the base layer on HardwareVideoDecoder: dual-layer is opt-in") + func felDefaultIsOptIn() { + let host = SoftwarePlaybackHost() + host.elType = "FEL" + #expect(host.dolbyVisionHandling == .automatic) + #expect(host.preferDualLayerDolbyVision == false) + + let decoder = host.selectVideoDecoder( + codecID: AV_CODEC_ID_HEVC, + canHardwareDecode: true, + dvProfile: 7 + ) + #expect(decoder is HardwareVideoDecoder) + } + + @MainActor + @Test("P7 FEL opted in selects DualLayerVideoDecoder") + func felOptedInSelectsDualLayer() { + let host = SoftwarePlaybackHost() + host.elType = "FEL" + host.preferDualLayerDolbyVision = true + #expect(host.dolbyVisionHandling == .automatic) + + let decoder = host.selectVideoDecoder( + codecID: AV_CODEC_ID_HEVC, + canHardwareDecode: true, + dvProfile: 7 + ) + #expect(decoder is DualLayerVideoDecoder) + } + + @MainActor + @Test("P7 MEL selects HardwareVideoDecoder across all option combinations") + func melSelectsHardwareDecoder() { + let handlings: [DolbyVisionHandling] = [.automatic, .baseLayerOnly] + for handling in handlings { + for prefer in [true, false] { + let host = SoftwarePlaybackHost() + host.elType = "MEL" + host.dolbyVisionHandling = handling + host.preferDualLayerDolbyVision = prefer + + let decoder = host.selectVideoDecoder( + codecID: AV_CODEC_ID_HEVC, + canHardwareDecode: true, + dvProfile: 7 + ) + #expect( + decoder is HardwareVideoDecoder, + "MEL with handling=\(handling) prefer=\(prefer) must pick HardwareVideoDecoder" + ) + } + } + } + + @MainActor + @Test("P7 FEL with baseLayerOnly selects HardwareVideoDecoder") + func felBaseLayerOnlySelectsHardwareDecoder() { + let host = SoftwarePlaybackHost() + host.elType = "FEL" + host.dolbyVisionHandling = .baseLayerOnly + host.preferDualLayerDolbyVision = true + + let decoder = host.selectVideoDecoder( + codecID: AV_CODEC_ID_HEVC, + canHardwareDecode: true, + dvProfile: 7 + ) + #expect(decoder is HardwareVideoDecoder) + } + + @MainActor + @Test("P7 FEL with preferDualLayerDolbyVision=false selects HardwareVideoDecoder") + func felPreferFalseSelectsHardwareDecoder() { + let host = SoftwarePlaybackHost() + host.elType = "FEL" + host.dolbyVisionHandling = .automatic + host.preferDualLayerDolbyVision = false + + let decoder = host.selectVideoDecoder( + codecID: AV_CODEC_ID_HEVC, + canHardwareDecode: true, + dvProfile: 7 + ) + #expect(decoder is HardwareVideoDecoder) + } + + @MainActor + @Test("P7 without elType (nil) selects HardwareVideoDecoder") + func nilElTypeSelectsHardwareDecoder() { + let host = SoftwarePlaybackHost() + #expect(host.elType == nil) + + let decoder = host.selectVideoDecoder( + codecID: AV_CODEC_ID_HEVC, + canHardwareDecode: true, + dvProfile: 7 + ) + #expect(decoder is HardwareVideoDecoder) + } + + @MainActor + @Test("P8 or other DV profiles select HardwareVideoDecoder regardless of FEL settings") + func p8SelectsHardwareDecoder() { + let host = SoftwarePlaybackHost() + host.elType = "FEL" + host.preferDualLayerDolbyVision = true + host.dolbyVisionHandling = .automatic + + let decoder = host.selectVideoDecoder( + codecID: AV_CODEC_ID_HEVC, + canHardwareDecode: true, + dvProfile: 8 + ) + #expect(decoder is HardwareVideoDecoder) + } + + @MainActor + @Test("Non-HEVC or unsupported hardware decode falls back to SoftwareVideoDecoder") + func nonHardwareFallback() { + let host = SoftwarePlaybackHost() + host.elType = "FEL" + + let av1Decoder = host.selectVideoDecoder( + codecID: AV_CODEC_ID_AV1, + canHardwareDecode: true, + dvProfile: nil + ) + #expect(av1Decoder is SoftwareVideoDecoder) + + let noHwDecoder = host.selectVideoDecoder( + codecID: AV_CODEC_ID_HEVC, + canHardwareDecode: false, + dvProfile: 7 + ) + #expect(noHwDecoder is SoftwareVideoDecoder) + } + + @Test("SoftwarePlaybackHost.shouldUseDualLayerDecoder static predicate matches table") + func shouldUseDualLayerDecoderPredicate() { + // Section 5.2 table: only dvProfile == 7 && elType == "FEL" && preferDualLayerDolbyVision && .automatic + #expect(SoftwarePlaybackHost.shouldUseDualLayerDecoder( + dvProfile: 7, elType: "FEL", preferDualLayerDolbyVision: true, dolbyVisionHandling: .automatic + )) + #expect(!SoftwarePlaybackHost.shouldUseDualLayerDecoder( + dvProfile: 7, elType: "FEL", preferDualLayerDolbyVision: true, dolbyVisionHandling: .baseLayerOnly + )) + #expect(!SoftwarePlaybackHost.shouldUseDualLayerDecoder( + dvProfile: 7, elType: "FEL", preferDualLayerDolbyVision: false, dolbyVisionHandling: .automatic + )) + #expect(!SoftwarePlaybackHost.shouldUseDualLayerDecoder( + dvProfile: 7, elType: "MEL", preferDualLayerDolbyVision: true, dolbyVisionHandling: .automatic + )) + #expect(!SoftwarePlaybackHost.shouldUseDualLayerDecoder( + dvProfile: 7, elType: nil, preferDualLayerDolbyVision: true, dolbyVisionHandling: .automatic + )) + #expect(!SoftwarePlaybackHost.shouldUseDualLayerDecoder( + dvProfile: 8, elType: "FEL", preferDualLayerDolbyVision: true, dolbyVisionHandling: .automatic + )) + #expect(!SoftwarePlaybackHost.shouldUseDualLayerDecoder( + dvProfile: nil, elType: "FEL", preferDualLayerDolbyVision: true, dolbyVisionHandling: .automatic + )) + } +} diff --git a/Tests/AetherEngineTests/VideoRoutingPolicyTests.swift b/Tests/AetherEngineTests/VideoRoutingPolicyTests.swift index 770f28135..53220c78f 100644 --- a/Tests/AetherEngineTests/VideoRoutingPolicyTests.swift +++ b/Tests/AetherEngineTests/VideoRoutingPolicyTests.swift @@ -284,4 +284,252 @@ struct VideoRoutingPolicyTests { codecID: codec, dvProfile: 10, dvBlCompatID: 0)) } } + + // MARK: - Profile 7 dual-layer routing & hardware capability gate + + @Test("HEVC DV P7 routes to software for dual-layer Metal reconstruction when HW decode confirmed") + func dvProfile7RoutesToSoftwareWithHardwareDecode() { + #expect(VideoRoutingPolicy.requiresSoftwarePath( + codecID: AV_CODEC_ID_HEVC, + fieldOrder: AV_FIELD_PROGRESSIVE, + av1Available: true, + dvProfile: 7, + dolbyVisionHandling: .automatic, + preferDualLayerDolbyVision: true, + canHardwareDecode: { true }, + elType: "FEL" + )) + } + + @Test("the el_type probe is only wanted when its answer can change the route") + func enhancementLayerProbeIsGated() { + func wanted(codec: AVCodecID = AV_CODEC_ID_HEVC, profile: Int? = 7, + handling: DolbyVisionHandling = .automatic, prefer: Bool = true, + hardware: (() -> Bool)? = { true }) -> Bool { + VideoRoutingPolicy.dualLayerRouteIsWanted( + codecID: codec, dvProfile: profile, dolbyVisionHandling: handling, + preferDualLayerDolbyVision: prefer, canHardwareDecode: hardware) + } + #expect(wanted()) + #expect(!wanted(prefer: false), "not opted in: no probe") + #expect(!wanted(handling: .baseLayerOnly)) + #expect(!wanted(hardware: { false }), "no proven hardware session: the host would not compose") + #expect(!wanted(profile: 8)) + #expect(!wanted(profile: nil)) + #expect(!wanted(codec: AV_CODEC_ID_H264)) + // The probe's gate and the routing decision share one predicate: FEL routes exactly when it holds. + for prefer in [true, false] { + let routes = VideoRoutingPolicy.requiresSoftwarePath( + codecID: AV_CODEC_ID_HEVC, fieldOrder: AV_FIELD_PROGRESSIVE, av1Available: true, + dvProfile: 7, preferDualLayerDolbyVision: prefer, canHardwareDecode: { true }, elType: "FEL") + #expect(routes == wanted(prefer: prefer)) + } + } + + @Test("HEVC DV P7 FEL stays native when the caller does not opt in") + func dvProfile7FelStaysNativeByDefault() { + #expect(!VideoRoutingPolicy.requiresSoftwarePath( + codecID: AV_CODEC_ID_HEVC, + fieldOrder: AV_FIELD_PROGRESSIVE, + av1Available: true, + dvProfile: 7, + canHardwareDecode: { true }, + elType: "FEL" + )) + } + + @Test("HEVC DV P7 stays native when preferDualLayerDolbyVision is disabled") + func dvProfile7StaysNativeWhenPreferDualLayerDisabled() { + #expect(!VideoRoutingPolicy.requiresSoftwarePath( + codecID: AV_CODEC_ID_HEVC, + fieldOrder: AV_FIELD_PROGRESSIVE, + av1Available: true, + dvProfile: 7, + dolbyVisionHandling: .automatic, + preferDualLayerDolbyVision: false, + canHardwareDecode: { true }, + elType: "FEL" + )) + } + + @Test("HEVC DV P7 stays native when hardware decode is unsupported") + func dvProfile7StaysNativeWhenHardwareUnsupported() { + #expect(!VideoRoutingPolicy.requiresSoftwarePath( + codecID: AV_CODEC_ID_HEVC, + fieldOrder: AV_FIELD_PROGRESSIVE, + av1Available: true, + dvProfile: 7, + dolbyVisionHandling: .automatic, + preferDualLayerDolbyVision: true, + canHardwareDecode: { false }, + elType: "FEL" + )) + } + + @Test("HEVC DV P7 FEL stays native when the hardware probe is unclassifiable") + func dvProfile7StaysNativeWhenProbeUnclassifiable() throws { + // No extradata: the probe has nothing to judge, so the verdict is .unclassifiable. The + // permissive gate keeps native for that; the strict one the FEL route uses must say no, + // because SoftwarePlaybackHost would give such a stream to libavcodec. + let par = try #require(avcodec_parameters_alloc()) + defer { var p: UnsafeMutablePointer? = par; avcodec_parameters_free(&p) } + par.pointee.codec_id = AV_CODEC_ID_HEVC + par.pointee.width = 3840 + par.pointee.height = 2160 + guard case .unclassifiable = VTCapabilityProbe.hardwareDecodeVerdict(codecpar: par) else { + Issue.record("expected an unclassifiable verdict for a config record without extradata") + return + } + #expect(VTCapabilityProbe.canHardwareDecode(codecpar: par)) + #expect(!VTCapabilityProbe.opensHardwareDecoder(codecpar: par)) + #expect(!VideoRoutingPolicy.requiresSoftwarePath( + codecID: AV_CODEC_ID_HEVC, + fieldOrder: AV_FIELD_PROGRESSIVE, + av1Available: true, + dvProfile: 7, + dolbyVisionHandling: .automatic, + preferDualLayerDolbyVision: true, + canHardwareDecode: { VTCapabilityProbe.opensHardwareDecoder(codecpar: par) }, + elType: "FEL" + )) + } + + @Test("HEVC DV P7 stays native when dolbyVisionHandling is baseLayerOnly") + func dvProfile7StaysNativeWhenBaseLayerOnly() { + #expect(!VideoRoutingPolicy.requiresSoftwarePath( + codecID: AV_CODEC_ID_HEVC, + fieldOrder: AV_FIELD_PROGRESSIVE, + av1Available: true, + dvProfile: 7, + dolbyVisionHandling: .baseLayerOnly, + preferDualLayerDolbyVision: true, + canHardwareDecode: { true }, + elType: "FEL" + )) + } + + // MARK: - Profile 7 FEL vs MEL routing + + @Test("Profile 7 FEL / MEL table-driven routing matrix (dolby-vision-fel.md §5.2)") + func profile7RoutingMatrix() { + struct TestCase { + let elType: String? + let handling: DolbyVisionHandling + let preferDualLayer: Bool + let canHardwareDecode: Bool + let expectedSoftware: Bool + } + + let elTypes: [String?] = ["FEL", "MEL", nil] + let handlings: [DolbyVisionHandling] = [.automatic, .baseLayerOnly] + let preferOptions = [true, false] + let hwOptions = [true, false] + + var cases: [TestCase] = [] + for el in elTypes { + for handling in handlings { + for prefer in preferOptions { + for hw in hwOptions { + // Section 5.2 table: + // P7 FEL + .automatic + preferDualLayerDolbyVision (true) + HW decode -> software host + // All other combinations -> native (false) + let expected = (el == "FEL") + && handling == .automatic + && prefer + && hw + cases.append(TestCase( + elType: el, + handling: handling, + preferDualLayer: prefer, + canHardwareDecode: hw, + expectedSoftware: expected + )) + } + } + } + } + + #expect(cases.count == 24) + + for c in cases { + let actual = VideoRoutingPolicy.requiresSoftwarePath( + codecID: AV_CODEC_ID_HEVC, + fieldOrder: AV_FIELD_PROGRESSIVE, + av1Available: true, + dvProfile: 7, + dolbyVisionHandling: c.handling, + preferDualLayerDolbyVision: c.preferDualLayer, + canHardwareDecode: { c.canHardwareDecode }, + elType: c.elType + ) + #expect( + actual == c.expectedSoftware, + "Failed for elType=\(String(describing: c.elType)), handling=\(c.handling), prefer=\(c.preferDualLayer), hw=\(c.canHardwareDecode): expected \(c.expectedSoftware), got \(actual)" + ) + } + } + + @Test("Profile 7 FEL routes to software when HW decode and dual-layer preferred") + func felRoutesToSoftware() { + #expect(VideoRoutingPolicy.requiresSoftwarePath( + codecID: AV_CODEC_ID_HEVC, + fieldOrder: AV_FIELD_PROGRESSIVE, + av1Available: true, + dvProfile: 7, + dolbyVisionHandling: .automatic, + preferDualLayerDolbyVision: true, + canHardwareDecode: { true }, + elType: "FEL" + )) + } + + @Test("Profile 7 MEL stays native regardless of preferences") + func melStaysNativeAlways() { + // All combinations: each dolbyVisionHandling, preferDualLayerDolbyVision, hardware avail + for handling in [DolbyVisionHandling.automatic, .baseLayerOnly] { + for prefer in [true, false] { + for hardware in [true, false] { + #expect(!VideoRoutingPolicy.requiresSoftwarePath( + codecID: AV_CODEC_ID_HEVC, + fieldOrder: AV_FIELD_PROGRESSIVE, + av1Available: true, + dvProfile: 7, + dolbyVisionHandling: handling, + preferDualLayerDolbyVision: prefer, + canHardwareDecode: { hardware }, + elType: "MEL" + ), + "MEL should stay native with handling=\(handling), prefer=\(prefer), hw=\(hardware)") + } + } + } + } + + @Test("Profile 7 MEL ignores dolbyVisionHandling and prefer settings") + func melIgnoresOptions() { + #expect(!VideoRoutingPolicy.requiresSoftwarePath( + codecID: AV_CODEC_ID_HEVC, + fieldOrder: AV_FIELD_PROGRESSIVE, + av1Available: true, + dvProfile: 7, + dolbyVisionHandling: .automatic, + preferDualLayerDolbyVision: true, + canHardwareDecode: { true }, + elType: "MEL" + )) + } + + @Test("Profile 7 no el_type stays native (docs §5.2: only explicit FEL routes software)") + func noElTypeStaysNative() { + #expect(!VideoRoutingPolicy.requiresSoftwarePath( + codecID: AV_CODEC_ID_HEVC, + fieldOrder: AV_FIELD_PROGRESSIVE, + av1Available: true, + dvProfile: 7, + dolbyVisionHandling: .automatic, + preferDualLayerDolbyVision: true, + canHardwareDecode: { true }, + elType: nil + )) + } } diff --git a/docs/api.md b/docs/api.md index ceadecb95..a779d6d5e 100644 --- a/docs/api.md +++ b/docs/api.md @@ -245,8 +245,9 @@ Two refusals, both raised BEFORE any teardown, so a refused correction leaves th | Thrown | When | | --- | --- | | `AetherEngineError.loadIdentityNotCorrectable(fields:)` | the closure changed `isLive`, `audioOnly`, `nativeRemoteHLS` or `sequentialOrigin`. These name the session rather than tune it (each opens the source on a different pipeline, and the engine writes the last two itself), so changing one is a different item, not a correction of this one. Load the source again. | -| `AetherEngineError.sessionNotReloadable(_:)` | there is no session, or the source is a custom `IOReader` that reported itself non-seekable and cannot be reopened at the current position. It carries a `SessionReloadRefusal` (`.noActiveSession` / `.customSourceNotSeekable`) saying which. | +| `AetherEngineError.sessionNotReloadable(_:)` | there is no session, or the source is a custom `IOReader` that reported itself non-seekable and cannot be reopened at the current position. It carries a `SessionReloadRefusal` (`.noActiveSession` / `.customSourceNotSeekable`, and from a correction also the refusals named under `reloadAtCurrentPosition(applying:)`) saying which. | | `AetherEngineError.sessionNotReloadable(.softwarePathCannotRepresentSource)` | the correction moves a native session onto the software path, and this source's only signal is IPT-PQ-c2 (Dolby Vision HEVC Profile 5, AV1 Profile 10.0). The software decoders hand that on as YCbCr, so the picture would render green/purple. | +| `AetherEngineError.sessionNotReloadable(.dualLayerRouteNeedsFreshLoad)` | the correction changes `preferDualLayerDolbyVision` (or `dolbyVisionHandling` while the preference is on) on a custom `IOReader` Profile 7 session. The in-place rebuild keeps the route the session was loaded with, so the change could not take effect; call `load()` again with the new option. `SessionReloadRefusal` is a public enum: this is a new case, so a host that switches over it exhaustively must handle it (a `default:` or a new `case`). | | `AetherEngineError.sessionNotReloadable(.demuxedAudioLiveIsNativeOnly)` | the correction moves a demuxed-audio live session onto the software path, where the side-audio merge does not exist, so it would play silent. | Both are all-or-nothing: a correction refused for one field installs none of it. @@ -1207,6 +1208,7 @@ All flags default to safe values; the table is the full set. Depth for the media | `keepDvh1TagWithoutDV` | false | Diagnostic lever: force dvh1 tags and a master playlist regardless of display capability. | | `forceDolbyVisionOnNonDVDisplay` | false | **Experimental (AE#455).** On a display with no Dolby Vision of its own, serve an HEVC Profile 8.1 source the way a Profile 5 source is served (`dvh1` sample entry, `dvcC` rewritten to profile 5 / compatibility 0, `CODECS="dvh1.05.LL"`), so AVPlayer composes the RPU itself instead of handing the panel the static-metadata HDR10 base layer. The bitstream is untouched; only the container's claim about it changes. Ignored on a display that does Dolby Vision, and Profile 8.1 only. See [formats.md](formats.md#dolby-vision-signaling) for what it buys and what it risks. | | `dolbyVisionHandling` | `.automatic` | A `DolbyVisionHandling`. `.baseLayerOnly` presents the HDR10 / HLG base layer of a Dolby Vision source and leaves the Dolby Vision out of the container on every display: plain `hvc1` / `av01` sample entry, `dvcC` stripped, no `SUPPLEMENTAL-CODECS`, HDR10 / HLG display criteria, `videoFormat` reads the base layer's format while `sourceVideoFormat` and `sourceDVProfile` keep saying what the file carries. The route a host offers as "Dolby Vision: off (HDR10)", for a source whose Dolby Vision is wrong and whose base layer is right: the reported shape is a remux carrying a Profile 7 RPU under a container record claiming Profile 5, where a player that believes the record decodes YCbCr as IPT and the picture comes out green / violet. Applies to HEVC Profile 7 / 8.1 / 8.4, AV1 Profile 10.1 / 10.4, and a Profile 5 (or AV1 10.0) record whose VUI declares a BT.2020 YCbCr PQ or HLG base; a Profile 5 whose VUI is unspecified carries IPT-PQ-c2, has no base layer to present, keeps its route, and the engine says so in the log. Takes precedence over `forceDolbyVisionOnNonDVDisplay`. A tuning field, correctable through `reloadAtCurrentPosition(applying:)`; a Profile 5 record the VUI contradicts also stops refusing the software path under it. See [formats.md](formats.md#dolby-vision-signaling). | +| `preferDualLayerDolbyVision` | false | Opt in to composing Profile 7 FEL sources on the GPU instead of taking the native route (the single-layer 8.1 conversion where the display takes Dolby Vision, the HDR10 base layer where it does not). When true, `dolbyVisionHandling` is `.automatic`, the source is a seekable VOD, the first RPU says `el_type` FEL, VideoToolbox decodes HEVC in hardware and the stream fits the real-time budget (UHD at 30 fps for the base layer plus a 1080p enhancement layer; a 24p UHD master uses 80% of it), Profile 7 routes through `SoftwarePlaybackHost` + `DualLayerVideoDecoder`: both layers decode in VideoToolbox, `DoviFelComposer` reconstructs the picture in Metal and delivers 10-bit P010, and every frame carries its RPU converted to Profile 8.1 as a `DolbyVisionRPUData` attachment, which tvOS applies as Dolby Vision display management. A Profile 7 MEL source, and anything the device cannot decode twice in real time, keeps the native route (8.1 where the display takes Dolby Vision, the HDR10 base layer where it does not; the conversion is lossless only for MEL); the load logs one `DV P7 dual-layer route: taken` / `not taken ()` line either way. What the route gives up against the native one (Picture in Picture, scrub thumbnails, subtitles, AirPlay) is listed in [dolby-vision-fel.md](dolby-vision-fel.md#523-what-the-route-gives-up-against-the-native-route). Known trade-off, audio: the route runs on the software host, whose audio is decoded by FFmpeg and played through the sample-buffer audio renderer, and on tvOS only AVPlayer emits Atmos objects, so an E-AC-3 (JOC) track plays as its channel-bed core (5.1 or 7.1) without the objects; TrueHD behaves as on the native route. The route does not consult the display: with the flag on and a display that does not take Dolby Vision, the route is still taken, the composed full-FEL picture is presented with its attached metadata, and how tvOS presents it is up to the platform (the native route would have played the HDR10 base layer there). The E-AC-3 Atmos trade-off above applies there too. A tuning field, correctable through `reloadAtCurrentPosition(applying:)` on a URL source (which re-runs `load()` and re-decides the route); on a custom `IOReader` source a Profile 7 session refuses a change of this option, or a change of `dolbyVisionHandling` while this option is on, with `SessionReloadRefusal.dualLayerRouteNeedsFreshLoad`, because the in-place rebuild keeps the route the session was loaded with: call `load()` again with the new option. See [dolby-vision-fel.md](dolby-vision-fel.md). | | `preferredDecodePath` | `.automatic` | A `DecodePath`. `.software` serves this source through `SoftwarePlaybackHost` whatever the routing concluded, scoped to this session and costing the source nothing (seeks, the audio switch and the title switch all keep working). The escape for the formats `VTCapabilityProbe` deliberately cannot classify, and for live H.264 / HEVC, which never reaches that gate at all. One-way: there is no `.native`. See [Overriding the decode path](#overriding-the-decode-path). | | `hostRetriesExhaustedLiveReplay` | `false` | The host owns the retry after a verified live replay exhausts the in-engine reopen: the engine publishes `liveSourceReplayExhausted` and holds production until `retryLiveSourceRecovery()`. `false` publishes `liveSourceReset` for the same exhaustion. | | `escalatesToSoftwarePath` | `true` | Whether a native session AVPlayer refuses on its merits (`CoreMediaErrorDomain`), or a live join without an entry point the native route can open (`AetherEngine.LiveJoin`, AE#627), may be rebuilt once on the software path (AE#561). `false` surfaces the failure as `.error` instead, for a host with its own fallback ladder (AE#629). Correctable. | diff --git a/docs/dolby-vision-fel.md b/docs/dolby-vision-fel.md new file mode 100644 index 000000000..c17f6a2a1 --- /dev/null +++ b/docs/dolby-vision-fel.md @@ -0,0 +1,658 @@ +# Dolby Vision Profile 7 FEL on Apple TV + +How AetherEngine plays a Profile 7 dual-layer source (base layer + enhancement layer + RPU) as real +Dolby Vision on an Apple TV 4K, what has been proven on device, and the design that does it with +bounded CPU, GPU and memory while staying correct. + +This is the design document for the feature. User-facing behaviour lives in [formats.md](formats.md#dolby-vision-signaling), options in +[api.md](api.md), tools in [cli.md](cli.md). + +Reference device for every measurement below: Apple TV 4K (2nd generation, A12, 3 GB RAM), tvOS 27.0, +HDMI to an LG OLED, 23.976 fps UHD Blu-ray remuxes. + +- [1. What is being played](#1-what-is-being-played) +- [2. Output contract](#2-output-contract) +- [3. Current implementation](#3-current-implementation) +- [4. Evidence](#4-evidence) +- [5. Design](#5-design) +- [6. Verification](#6-verification) +- [7. Document map](#7-document-map) + +--- + +## 1. What is being played + +A Profile 7 access unit carries three things: + +| Part | Carriage | Content | +|---|---|---| +| Base layer (BL) | ordinary HEVC NAL units | 3840×2160 HEVC Main 10, BT.2020 PQ, a complete HDR10 picture on its own | +| Enhancement layer (EL) | `unspec63` NAL units (`0x7E 0x01` + an embedded HEVC NAL) | 1920×1080 HEVC Main 10, its own VPS / SPS / PPS | +| RPU | one `unspec62` NAL unit (`0x7C 0x01`) | per-component reshaping curves (polynomial or MMR pieces), NLQ parameters for the residual, and display-management metadata (L1, L2, L5, L8, L9, L11) | + +The RPU header says what the EL is for. **MEL** (minimal enhancement layer, `el_type == "MEL"`) +carries no residual: the picture is the reshaped base layer. **FEL** (full enhancement layer) adds a +residual that restores detail and precision the 10-bit base encode lost. `disable_residual_flag` can +switch the residual off for an individual RPU. + +Apple platforms decode Profile 5, 8.1 and 8.4 natively and have no Profile 7 dual-layer decoder. That +leaves two ways to play Profile 7: + +1. **Convert to single-layer 8.1** on the native path: rewrite each RPU with libdovi mode 2, drop the + EL. The conversion is lossless for MEL and loses the residual for FEL. Where the display takes + Dolby Vision that is what plays; where it does not, the native route plays the HDR10 base layer. See `DoviRpuConverter.convertPacketToProfile81`. +2. **Compose BL + EL + RPU ourselves** and present the result with its Dolby Vision metadata. That is + the subject of this document. + +## 2. Output contract + +What a composed frame handed to the display layer must be: + +- **Pixels.** `kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange` (P010), BT.2020 primaries, PQ + transfer, colour attachments copied from the base layer. Reconstruction runs at 12-bit precision + (reshaped base plus inverse-NLQ residual); an ordered 8×8 Bayer dither, inverted on odd frames and + phase-offset between Cb and Cr, takes it to 10 bits. Luma is clamped to 64–940, chroma to 64–960. +- **Metadata, on every frame.** + + | Attachment | Value | + |---|---| + | `DolbyVisionRPUData` | the RPU converted to Profile 8.1 (`dovi_convert_rpu_with_mode(rpu, 2)`), as the `unspec62` NAL **without** its 2-byte header, emulation-prevention bytes kept; a `CFData` | + | `DolbyCompatibilityID` | the **`CFString` `"1"`**, never a number | + + A frame with no RPU has both attachments removed, because output buffers are recycled from pools + and would otherwise carry a previous frame's trims. +- **Why Profile 8.1 metadata.** The composed picture is YCbCr PQ over an HDR10-compatible signal, which + is exactly what a Profile 8.1 RPU describes. Mode 2 keeps L1 / L2 / L5 / L8 / L9 / L11 intact and makes + the reshaping identity, which is correct because the composer has already applied the real curves. + Profile 5 (IPT-PQ-c2) would require re-deriving the reshaping for a different colour space. +- **Who consumes it.** `HDRProcessing.framework` inside **backboardd**, on CoreAnimation's external + display thread, on every vsync (§4.1). Nothing in the app's own process reads it, so no in-process + instrument can observe consumption. + +## 3. Current implementation + +```mermaid +flowchart LR + D["Demux thread
DoviRpuConverter.separateDualLayers"] -->|BL packet| VB["VTDecompressionSession
BL 4K"] + D -->|EL packet| VE["VTDecompressionSession
EL (typically 1080p)"] + D -->|params + 8.1 RPU| C[("rpuCache / dmRpuCache")] + VB --> P{"DualLayerVideoDecoder
pairing"} + VE --> P + C --> P + P -->|compose| G["DoviFelComposer
composeAsync (Metal)"] + P -->|bypass| O + G --> O["OrderedDelivery
fulfil → emit → DoviDisplayMetadata.attach"] + O --> R["SampleBufferRenderer
AVSampleBufferDisplayLayer"] +``` + +| Stage | Code | Thread | What happens per frame | +|---|---|---|---| +| Routing | `VideoRoutingPolicy.requiresSoftwarePath`, `SoftwarePlaybackHost` decoder selection | load | Profile 7 + `preferDualLayerDolbyVision` + `dolbyVisionHandling` `.automatic` + VOD seekable source + hardware HEVC decode + `el_type` FEL + inside the real-time budget (§5.2) → software host, which picks `DualLayerVideoDecoder` (§5.2) | +| Split | `DoviRpuConverter.separateDualLayers` | demux | RPU parsed into composer parameters and converted to an 8.1 NAL; EL NAL units copied into a new `AVPacket`; the BL packet goes to VideoToolbox untouched (it skips the `unspec` NAL units itself) | +| Align | `DualLayerPictureAlignment.enhancementPTS` | demux, under the decoder's lock | each layer's display slot from its POC (`HEVCPictureOrderCount`); the EL packet takes the PTS of the BL picture in the same slot, so pairing by time pairs by picture order when the layers' GOPs differ. A layer whose sequence cannot be trusted (leading pictures of the opener, or more than 16 slots off its unit, or a seek before its next IRAP) keeps its access unit's PTS | +| Decode | two `HardwareVideoDecoder`s | VideoToolbox output threads | asynchronous decode with temporal processing, P010 IOSurface buffers in decode order (the renderer's 4-frame reorder buffer restores display order, as for any VideoToolbox HEVC stream) | +| Pair | `DualLayerVideoDecoder.handleBaseLayerFrame` / `handleEnhancementLayerFrame` | VideoToolbox output threads, under the decoder's recursive lock | match the other layer by 90 kHz canonical PTS within ±300 ticks; a base-layer frame takes its ticket at arrival and is bypassed under it once K = 3 newer base-layer frames have arrived; waiting enhancement frames capped at 10; side data in a 32-entry ring; `hdr10Data` and the display-metadata NAL are resolved once here and travel with the ticket | +| Compose | `DoviFelComposer.composeAsync` | caller, then Metal | wait for one of 3 in-flight slots, rent a P010 buffer (§5.6), copy colour attachments from the base layer, bind 6 texture-cache textures, serialize the parameter block, encode luma and chroma kernels in one concurrent encoder, commit | +| Deliver | Metal completion handler → composer's `deliveryQueue` (composed frames) or the pairing call directly (a bypass) → `OrderedDelivery.fulfil` → `emit` | Metal's completion queue hands off immediately to the composer's own `.userInteractive` queue (§4.3's firewall, not an ordering mechanism), or the pairing thread for a bypass, then `OrderedDelivery`'s own `.userInteractive` queue — never the decoder lock | release the GPU slot inline, nothing else, on Metal's queue; everything else (error logging, GPU-time accounting, `completion`) hops off it first. Either path hands the frame to `OrderedDelivery` (§5.7): one serial queue orders composed and bypassed frames alike regardless of which queue called `fulfil`, drops anything tagged with an epoch a flush/close has since retired, and calls `onFrame` (renderer enqueue) with no decoder lock held | + +A frame bypasses composition, and goes out as its base layer with its own DV metadata, when the +stream is not FEL, when MEL was detected, when no RPU parameters exist for its PTS, or when +composition fails. + +**Kernels** (`DoviShaderSource.swift`). Luma: base code → piece selection → polynomial (evaluated in +normalized float) or MMR when the luma curve uses MMR → plus the inverse-NLQ residual of the +enhancement-layer luma, upsampled with one hardware-bilinear fetch → +12-bit clamp → dither → 10-bit. Chroma: base chroma plus the average of the two vertically adjacent +luma samples (chroma siting) → MMR → plus the residual from the upsampled enhancement chroma → dither. + +**Parameter ABI.** `DoviShaderSource.sharedStructs` on the GPU side, `DoviMetalParams.serialize` on +the Swift side. `DoviLayoutProbeTests` has the GPU read every sub-block back at the offset Swift wrote +it to; `DoviShaderSyncTests` pins the serialized size and that every kernel is present. + +### 3.1 Audio + +Known trade-off of the route, following the maintainer's request to document it rather than work around it. +The dual-layer route runs on `SoftwarePlaybackHost`, so its audio is the host's own `AudioDecoder` (FFmpeg +decode to PCM, played through the sample-buffer audio renderer) and never AVPlayer. On tvOS only AVPlayer +emits Dolby Atmos objects, and the platform does not pass a bitstream through. An E-AC-3 (JOC) track +therefore plays as its channel-bed core (5.1 or 7.1), without the Atmos objects; on the native route the +same track is handed to AVPlayer unchanged. TrueHD behaves as on the native route, which decodes it too and +loses the objects either way. Hosts that want E-AC-3 Atmos on a Profile 7 FEL title leave +`preferDualLayerDolbyVision` off for it. `DualLayerRouteAudioTests` pins the routing half of this. + +## 4. Evidence + +### 4.1 tvOS consumes the attachment, in backboardd + +Two device runs were killed with `SIGKILL` about eleven composed frames in. The app log showed nothing +wrong except `Connection interrupted` and `invalid destination port` from CoreAnimation, which means +the render server died first. The paired Apple TV's own crash reports held two `backboardd-*.ips` +files timed to the runs, and no `JetsamEvent`, which rules out memory pressure. Both had this stack: + +``` +Exception: EXC_CRASH (SIGABRT), -[__NSCFNumber length]: unrecognized selector +Thread: com.apple.coreanimation.display.external-0 + CFStringGetIntValue + -[HDRProcessor applyDoVi81PolicyWithInput:WithRPU:] + -[HDRProcessor processFrameInternalWithLayer0:layer1:outout:metadata:commandbuffer:…] + CA::HDRProcessorInternal::tonemap_surface_internal(__IOSurface*, …, __CFData const*, …) + CA::HDRProcessorInternal::create_surface_with_forward_dm + CA::Render::ImageQueue::forward_dm + CA::WindowServer::IOMFBServer::vsync_callback +``` + +That is the proof of consumption: on the vsync callback, CoreAnimation pulls display management off +our IOSurface and applies Dolby Vision 8.1 policy with our RPU. It is also the bug: the policy parses +`DolbyCompatibilityID` with `CFStringGetIntValue`, and we had attached a `CFNumber`. Decoding a +Profile 8.1 asset through `AVAssetReader` and printing the CF type of VideoToolbox's own attachments +confirmed the fix: + +| Attachment | Class | CF type | Value | +|---|---|---|---| +| `DolbyCompatibilityID` | `NSTaggedPointerString` | `CFString` | `"1"` | +| `DolbyVisionRPUData` | `__NSCFData` | `CFData` | 188 / 189 bytes | + +The payload form was established the same way: VideoToolbox's `DolbyVisionRPUData` matches the +bitstream's `unspec62` NAL byte for byte once the 2-byte header is removed and the emulation-prevention +bytes are left in; the EPB-stripped RBSP is 176 bytes and does not match. + +Two consequences carry forward. A malformed attachment crashes the render server and looks, from +inside the app, like an unexplained kill; suspect the attachment before memory. And the device crash +reporter (`devicectl … --domain-type systemCrashLogs`) is the instrument for anything that happens +out of process. + +### 4.2 Throughput: the composer must not block decode + +Playback first ran with `compose` ending in `waitUntilCompleted()` under the decoder lock. The deficit +was steady, not a hitch: + +| | Blocking compose | `composeAsync` + one encoder | +|---|---|---| +| Delivered rate | 21.4–22.0 fps | real time, ~26.6 fps while catching up after a seek | +| Video lead | slipping 0.09 s per second | flat at about +0.5 s | +| Display drops | +3…+7 every second | flat after seek settling | +| GPU time per frame | 28 ms end to end, serialized | 14.3 → 12.9 ms, off-thread | + +Most of the original 28 ms was the wait and the CPU-side setup, not GPU work: the A12 has about 3× +headroom against the 41.7 ms budget at 23.976 fps. The two serial compute encoders were merged into +one concurrent encoder because the luma and chroma kernels never read each other's output; the +bit-exactness suites passing unchanged is what proves that moved no pixels. + +### 4.3 The completion-handler deadlock + +Metal runs every `addCompletedHandler` block for a command queue on one serial queue. The first async +version delivered from inside that handler, which took the decoder lock, while the caller held the +same lock waiting in `inFlight.wait()` for the signal that only a later handler could send. Playback +stopped after about ten seconds with the session still reporting healthy. The fix, and a rule for any +future change: **release the slot inline in Metal's handler and do nothing there that can block; hop +to another queue for everything else.** `DoviComposeAsyncTests.doesNotDeadlockAgainstTheCallersLock` +reproduces the exact lock shape; against the old handler it hung past a 300 s timeout. + +### 4.4 Earlier fixes worth remembering + +| Symptom | Cause | Rule | +|---|---|---| +| Metal validation abort binding textures | integer kernel access on a Unorm texture | bind BL and output planes as `r16Uint` / `rg16Uint`, EL planes as `r16Unorm` / `rg16Unorm` so they can be filtered | +| Black picture with `COMPOSE SUCCESS` on every frame | 10-bit prediction treated as 12-bit | keep the whole reconstruction in one declared precision | +| Green / purple picture | chroma curves with fewer than 2 pivots returned 0 | an absent curve is identity, not zero | +| GPU reading freed memory | `CVMetalTexture` wrappers released before completion | keep textures *and* their source buffers alive until the command buffer retires (`ComposedFrame`) | +| Layers never pairing | container vs decoder timestamp scales | canonicalize to 90 kHz and match within ±300 ticks | +| EL decoder never opening | EL VPS / SPS / PPS split across packets | accumulate parameter sets across packets | +| Profile 7 plays as plain HDR10 | the container carries no `dvcC` record, so the load never sees a Profile 7 (routing and the `el_type` probe both start from the record) | none: the file plays as its HEVC base layer, the `unspec62` / `unspec63` NAL units ignored. `DolbyVisionRecordAudit.addRecordIfProfile5` repairs a recordless Profile 5 only; for Profile 7 the fix is a remux that writes the record | + +### 4.5 Consumption test assets + +`aetherctl dvsurvey` (in the tooling PR that follows this one) ranks frames by L1 distance from the title's median so that the pin test can +pin a frame whose metadata is a large lie. On the three local test clips: a CMv4.0 4000-nit clip pins +frame 247 (`max_pq` 4095 against a median of 2081, distance 3876, the best asset); a scene clip gives +distance 1391, usable; a credits clip has static L1 (distance 0) and is correctly rejected as `WEAK`. + +The consumption question, whether the panel uses the RPU the engine attaches rather than only +switching into a Dolby Vision mode, is answered by a pulse test that anyone with an Apple TV, a +television and a phone can repeat: clips whose pixels are constant while only the RPU (or the +enhancement-layer residual) switches, played through AVPlayer, VideoToolbox and the FEL route, and +filmed. Clip generator, device host, harness scripts and analysis are in +[`Probes/DolbyVisionPulse`](../Probes/DolbyVisionPulse/README.md), which arrives with the pulse-probe PR +that follows this one; the link resolves once it has landed. + +### 4.6 Caveat on the numbers + +Every device figure above came from DEBUG builds: `-Onone`, with the HDMI probe retrying and the +memory watermark sampling on every emitted frame. They show the direction and the GPU cost; CPU and +memory should be re-measured on a release build. Footprint went from a flat ~390 MB to a plateau of +~705 MB with the async pipeline; §5.10 accounts for the pipeline's own share. + +--- + +## 5. Design + +VideoToolbox decodes both layers, one concurrent Metal encoder composes them, each frame carries the +Profile 8.1 attachment, delivery is ordered by ticket, and the GPU slot is released inline. The +sections below are the rules that keep per-frame work and shared mutable state to a minimum. + +### 5.1 Principles + +1. **Decide once per stream, not per frame.** FEL vs MEL is a property of the encode; routing and + decoder setup decide it, and no callback changes it. +2. **Touch each byte once.** One libdovi parse per RPU, no copy of the base-layer access unit. +3. **Nothing blocks while holding a lock, and every piece of state has one owner.** +4. **Every frame has a bounded lifetime.** Bounded wait for its partner, bounded queues, bounded GPU + work in flight, and nothing from before a flush survives it. +5. **Per-frame work writes no log lines.** Counters feed a 1 Hz summary. + +### 5.2 Routing + +The load probes the first RPU's `el_type` (`DolbyVisionRecordAudit.probeEnhancementLayerType`) and +passes it into `VideoRoutingPolicy`: + +| Source | `dolbyVisionHandling` | `preferDualLayerDolbyVision` | Device | Route | +|---|---|---|---|---| +| P7, `el_type` FEL | `.automatic` | true (opt-in; the default is false) | VideoToolbox HEVC hardware + Metal | software host, `DualLayerVideoDecoder` | +| P7, `el_type` FEL | `.automatic` | false (the default), or no VideoToolbox HEVC hardware decode | — | native (8.1 conversion where the display takes Dolby Vision, otherwise the HDR10 base layer) | +| P7, `el_type` MEL | `.automatic` | any | — | native (as above; the conversion loses nothing for MEL) | +| P7, any | `.baseLayerOnly` | any | — | HDR10 base layer | + +The software host then instantiates `DualLayerVideoDecoder` only for the first row; a Profile 7 forced +onto the software path by any other combination gets a plain `HardwareVideoDecoder` for the base layer. + +### 5.2.1 The capability gate and its diagnostic line + +The route is FEL only and capability gated (`DualLayerRouteDecision`, pure and unit-tested). Everything +else keeps the native route, which plays single-layer 8.1 where the display takes Dolby Vision and the +HDR10 base layer where it does not: the preference is off, `dolbyVisionHandling` is not `.automatic`, a +live or non-seekable source, a box with no hardware HEVC decoder for the stream, a MEL source (nothing +to compose), an enhancement layer the probe could not classify, a base layer of unknown size, and a +stream the box could not decode twice in real time. The checks run in that order (the code's order: +preference, handling, live/seekable, hardware, `el_type`, geometry, budget). The `el_type` probe (a packet +scan and a seek) runs once nothing up to and including the hardware check has ruled the route out, so it +also runs for a stream that the geometry or the budget then refuses: the MEL / unknown answer outranks +the size reasons in the log line, and that precedence needs the probe's answer. + +The real-time budget is a pixel rate, base layer plus enhancement layer, from the stream's width, height +and frame rate: `DualLayerRouteDecision.maximumPixelsPerSecond`, 311.04 Mpx/s, which is UHD +3840x2160 at 30 fps plus a 1920x1080 enhancement layer at 30 fps. The enhancement layer is taken as a +quarter of the base layer's area (what a 4K FEL disc ships), and an unknown frame rate is budgeted as 24 +(the line says "assumed"). The load passes no enhancement-layer size, so it always budgets the quarter-area +layer (every retail FEL title's layout); a 1:1 enhancement layer is not detected at load. A base layer whose +width or height is unknown (not positive) is refused rather than budgeted at zero. A 24p UHD FEL master needs 80% of the budget. What is measured: the Mac soak runs in the PR +description (M2 Pro, debug `aetherctl`), and, from earlier development builds on an Apple TV 4K 2nd +generation, the composer's GPU time of 12.9-14.3 ms per frame against a 41.7 ms frame at 23.976 fps +(§4.2, a debug build and a previous iteration of the pipeline, not this stack). The 311 Mpx/s ceiling, +and 30 fps as a margin above 24p, are an assumption until the Apple TV run of this stack. UHD at 60 fps +(200%) and 8K fall back. + +A Profile 7 load logs exactly one line, whichever way it goes (other sources log nothing here): + +``` +[AetherEngine] DV P7 dual-layer route: taken (el_type=FEL, 3840x2160@23.976, budget 80%) +[AetherEngine] DV P7 dual-layer route: not taken (el_type=MEL, nothing to compose), keeps the native route (8.1 where the display takes Dolby Vision) +[AetherEngine] DV P7 dual-layer route: not taken (preferDualLayerDolbyVision is off), keeps the native route (8.1 where the display takes Dolby Vision) +[AetherEngine] DV P7 dual-layer route: not taken (over the real-time budget, 3840x2160@60 needs 200% of it), keeps the native route (8.1 where the display takes Dolby Vision) +[AetherEngine] DV P7 dual-layer route: not taken (base-layer size unknown), keeps the native route (8.1 where the display takes Dolby Vision) +``` + +A missing Metal device does not fall back to the native route: the routing decision above has +already been made, so the software host keeps the base layer alone on a `HardwareVideoDecoder` (HDR10, +no composition). The same is true after the fact: if the compute shaders fail to compile, the decoder +stays but every frame goes out as its base layer. + +### 5.2.2 The display is not consulted + +The route decision does not look at the display. With `preferDualLayerDolbyVision` on and a display +that does not take Dolby Vision (an SDR or HDR10-only panel or mode), the route is still taken when +the gate above passes: the software host composes the full FEL picture and presents it with the +Profile 8.1 metadata attached (§2), and how tvOS presents that on such a display is up to the platform. +The native route, by contrast, strips the Dolby Vision for such a display (`CodecRoutePolicy`) and plays +the HDR10 base layer. A host that wants the base layer on a non-Dolby-Vision display leaves the flag off, +or sets it from what it knows of the display. The audio trade-off of §3.1 (an E-AC-3 Atmos track plays as +its channel-bed core, without the objects) applies on this route whatever the display. This is the current +behaviour, documented rather than designed; it is not changed here. + +### 5.2.3 What the route gives up against the native route + +The dual-layer route runs on the software host: there is no `AVPlayer` and no `AVPlayerItem`, and the +picture is drawn into an `AVSampleBufferDisplayLayer`. Everything that AVKit or the loopback HLS +server provides to the native route through the item is therefore either replaced by an engine +equivalent or absent. With `preferDualLayerDolbyVision` off none of this changes. The comparison, +checked against the code: + +| Feature | Native route | Dual-layer route | +|---|---|---| +| Picture in Picture | `AVPlayer`-based: the stock `AVPlayerViewController` and a host-built `AVPictureInPictureController` both work. | No `AVPlayer`, so no stock player PiP. The engine publishes `SoftwarePiPSource` (the layer plus the transport answers for a sample-buffer PiP content source) and the host builds the `AVPictureInPictureController` around it; while PiP is active, subtitle cues are drawn into the frames by `SubtitleFrameCompositor`, which carries the pixel-buffer attachments through. Whether the PiP window presents the composed picture as Dolby Vision is not measured. | +| Scrub thumbnails | With `serveIFramePlaylist`, an `EXT-X-I-FRAME-STREAM-INF` rendition next to the master, so a stock `AVPlayerViewController` draws its own thumbnails and scans on I-frames, plus `scrubThumbnail(atSeconds:)`. | No I-frame playlist, so a stock player shows no thumbnails of its own. `scrubThumbnail(atSeconds:)` answers from the software host's retained packet cache (`servesPacketCacheStills`) and only when that cache exists: a local file is read directly and has none, so there the call returns nil. The still is the base layer decoded by libavcodec, not the composed picture. | +| Subtitles | Text tracks can be declared in the init segment and served as WebVTT renditions (`prepareNativeSubtitles`), so `AVMediaSelection`, PiP and external screens render them. | No renditions and no `AVMediaSelection`. The engine decodes the cues and publishes them; the host draws its own overlay in the app. They appear in a PiP window only through the frame compositor above, and the system caption styling does not apply. | +| AirPlay | `AVPlayer` hands the receiver the stream (the engine serves the loopback over the LAN address for it). | The layer draws on the device and only the audio renderer follows the route, so a wireless receiver plays sound only. The engine publishes `airPlayPictureStaysLocal` (iOS) so a host can say so. Not applicable where the device is itself the receiver. | + +A host that needs any of these for a Profile 7 title leaves the flag off for it, as for the Atmos +trade-off of §3.1. + +### 5.3 Packet split + +One walk over the access unit's NAL units, on the demux thread: + +- **RPU:** `dovi_parse_unspec62_nalu` once. From that handle read the header (`disable_residual_flag`), + the data mapping (into `DoviMetalParams`), then convert with mode 2 and write the 8.1 NAL. Strip its + header once into the attachment `Data`. +- **EL:** copy the embedded NAL units into the EL packet (one 1080p copy, unavoidable because + VideoToolbox needs a standalone access unit). +- **BL:** hand the packet to VideoToolbox **untouched**. The format description is built from the base + layer's parameter sets with no `dvcC`, so `unspec62` and `unspec63` are unknown NAL types it skips. + The rebuild stays only in `convertPacketToProfile81`, where the container does declare Dolby Vision. + +Side data goes into a **`FrameSideDataRing`** keyed by canonical PTS: `{ params, dvPayload, flags }`, +fixed capacity 32 (reorder depth plus maximum layer skew with margin). Insert overwrites the oldest, +lookup scans at most 32 entries for the nearest key within ±300 ticks. No sorting, no dictionaries. + +### 5.4 Decode + +Two `HardwareVideoDecoder`s. `HardwareVideoDecoder.decode` wraps the packet's memory with +`CMBlockBufferCreateWithMemoryBlock` and a custom block source that holds an `av_buffer_ref` of +`packet.buf` and unrefs it in `FreeBlock`, so VideoToolbox can retain the data past the call without a +copy. When `packet.buf` is nil it falls back to the copy. This benefits every HEVC source on the +software path, not just FEL. + +### 5.5 Pairing + +Pairing lives in `DualLayerVideoDecoder.handleBaseLayerFrame` / `handleEnhancementLayerFrame`, under +the decoder's lock, and each call is O(K): + +- A **base-layer frame takes its ticket when it arrives.** VideoToolbox outputs in decode order, so + ticket order is decode order, whatever happens to the frame later. Display order is restored + downstream, by the renderer's reorder buffer (4 frames, the same depth as a single-layer hardware + decode; `DoviFelRealFileTests.reconstructsAndOrders` replays that rule on the real files). +- If its enhancement layer is already waiting (within ±300 ticks), the pair is composed. Otherwise it + waits. +- **A waiting base layer is bypassed, under its own ticket, once K newer base-layer frames have + arrived.** K (`maxLayerSkewFrames`) is the maximum layer skew, 3; the summary line reports the + observed skew (§5.11). A lost enhancement frame costs K frames of delay and never reorders output. +- An enhancement frame whose base layer has already gone out (its key matches one of the last 64 + settled base-layer keys) is dropped on arrival. At most 10 enhancement frames wait. +- A frame's side data is looked up once, when its base layer arrives, and travels with the ticket; + delivery never touches a shared cache. +- The `disable_residual_flag` of an individual frame becomes a kernel flag + (`DoviComposerFlags.disableResidual`); it never changes the stream classification. + +### 5.6 Compose + +- **Acquire the in-flight slot outside the decoder lock**, on the VideoToolbox output thread that + produced the `compose` action, then encode and commit. Blocking that thread is the intended + backpressure: VideoToolbox stops emitting, the display layer stops draining, and the demux loop's + existing `mayEnqueueVideoFrame` gate (the renderer's queue readiness plus the look-ahead bound) holds. +- **Parameters** are written straight into one of `maxFramesInFlight` preallocated shared + `MTLBuffer`s (`DoviMetalParams.serialize(into:)`), bound with `setBuffer` for both kernels. No `Data`, + no heap arrays per frame. +- **Output pool** gets `kCVPixelBufferPoolAllocationThresholdKey` = the renderer's FEL look-ahead (9) + + the frame on screen and the one the display layer retains (2) + the in-flight slots (3) + 1, so 15. + A refused buffer would put a base-layer frame on screen, a visible brightness step, so past the + threshold a burst draws on a reserve of 3 more. Only with the reserve spent does the composer wait, + for at most about 36 ms (a 300 ms wait stalled the VideoToolbox callback and starved audio into + rebuffers on device), and then the frame bypasses instead of growing the pool. +- **Texture cache** is flushed from the 1 Hz tick, which releases textures for buffers no longer in use. +- **Colour attachments** are copied from the base layer when encoding, on the calling thread, not on + Metal's completion queue. +- **Pipelines** are compiled once per process, lazily on first composition (a shared `MTLLibrary` and + pipeline states), so a MEL or bypassed stream never compiles and a second decoder instance does not + recompile. +- **Kernels:** a per-frame 1024-entry luma prediction table replaces piece selection and polynomial + evaluation per pixel whenever luma is polynomial (it is a pure function of the 10-bit base code); + residual evaluation is skipped per component when that component's NLQ is inactive; threadgroup sizes + come from `threadExecutionWidth`. Every one of these keeps `DoviGpuCurveSweepTests` bit-exact. + +### 5.7 Delivery + +An `OrderedDelivery` object owned by the decoder, not the composer: + +```swift +final class OrderedDelivery { + private let queue = DispatchQueue(label: "de.superuser404.dovi.delivery", qos: .userInteractive) + // Confined to `queue`: no lock. + private var epoch: UInt64 = 0 + private var nextToEmit: UInt64 = 0 + private var pending: [UInt64: DeliveredFrame] = [:] + + func fulfil(ticket: UInt64, epoch: UInt64, _ frame: DeliveredFrame) // queue.async + func resetAsync(epoch: UInt64, nextTicket: UInt64) // queue.async, callable under a lock + func barrier() // queue.sync, never under a lock + func reset(epoch: UInt64, nextTicket: UInt64) // resetAsync + barrier +} +``` + +- **Every** outgoing frame goes through `fulfil`: composed frames from the composer's relay queue + (after `slot.signal()`, see below), bypassed frames straight from the pairing call. One serial queue orders + both, so no lock and no recursive re-entry are needed. +- **The queue owns its state.** `emit`, which attaches the DV metadata and calls `onFrame`, runs on it + with no decoder lock held, so the renderer's own lock is never nested inside ours. +- **QoS `.userInteractive`.** The work per frame is O(1) (two attachments and an enqueue), it is bound + by the frame rate, and it decides what is on screen next, which is what that class is for. A queue + created with only a label has unspecified QoS and inherits whatever the submitting Metal completion + thread carries, so presentation can wait behind default- or utility-class work. Because the work is + tiny it cannot starve the UI; if a trace ever shows contention on tvOS, `.userInitiated` is the + fallback. +- **Epochs instead of moving the line under a lock.** A ticket carries the epoch it was issued in; + a stale one is dropped when it reaches the head of the queue. `barrier()` runs with `queue.sync`, + which waits for any delivery in progress, so nothing from an old epoch can reach `onFrame` after + `flush()` returns. It asserts `dispatchPrecondition(condition: .notOnQueue(queue))`: a host must + never flush from inside `onFrame`. +- **Stall guard.** If more than `maxFramesInFlight + K + 2` frames are pending behind a missing + ticket, skip the missing ticket and log once. A lost ticket must cost one frame, not the session. +- **No `.global()` fallback.** The completion handler captures the delivery object strongly; a + completion that outlives the decoder delivers into an epoch that no longer exists and is dropped. + +**The composer keeps its own relay queue.** A composed frame's completion is not handed straight to +`OrderedDelivery.fulfil` from Metal's completion handler. `DoviFelComposer` keeps a small serial `deliveryQueue` (`qos: .userInteractive`, +captured as a plain value, no `.global()` fallback) that every composed frame's completion hops through +first; only there does it call `completion`, which is `fulfil` in production. §4.3 requires exactly this +shape — "release the slot inline in Metal's handler and do nothing there that can block; hop to another +queue for everything else" — and `DoviComposeAsyncTests.doesNotDeadlockAgainstTheCallersLock` reproduces +the deadlock removing the hop would reopen: a `completion` that blocks on the caller's lock, run directly +on Metal's shared completion queue, stalls every later handler too, including the `inFlight.signal()` a +different, currently-blocked caller needs. `deliveryQueue` carries no ordering responsibility of its +own — `OrderedDelivery` reorders every frame by ticket regardless of which queue or thread calls +`fulfil`, so a bypass arriving directly from a VideoToolbox thread and a composition arriving via this +relay hop are indistinguishable once they reach it. This is a deliberate queue hop, not a second +ordering mechanism. An `os_signpost` event in the Metal completion handler +(`DoviFelComposer`'s "Compose" signpost) marks the instant Metal invoked it, separately from the +interval's `.end` (which only fires after this hop), so the hop's own latency is visible in Instruments +rather than folded into either "Compose" or "Delivery". + +### 5.8 Flush and close + +Order matters because `HardwareVideoDecoder.flush` waits for frames VideoToolbox still owes, and those +arrive through the normal callbacks: + +1. Under the lock: bump the feed epoch (so `decode(packet:epoch:)` refuses packets from before the + seek), raise `draining`, take `nextTicket`, and enqueue `delivery.resetAsync(epoch:nextTicket:)` — + from inside this same critical section, not after. Then, still outside the lock but before draining, + call `delivery.barrier()` to wait for that reset to actually run. +2. Flush both `HardwareVideoDecoder`s. Their pending frames land in pairing, in the old state, but + `draining` (still up) drops them before they can take a ticket, and any composition already in + flight before this call is dropped by the epoch check the instant it reaches `fulfil`. +3. Under the lock, lower `draining` and reset pairing and the side-data ring. +4. Flush the composer's texture cache. + +The epoch is retired in step 1, before draining, and not after the drain. Retiring it after the +drain, with a plain `delivery.reset` once the lock was released, left a gap: a VideoToolbox callback +landing between that `unlock()` and the `reset` could take a new-epoch ticket and reach `fulfil` before +`reset` had run, so the ticket was dropped as still-old-epoch and `reset` then fast-forwarded past it, +stalling every following frame until the stall guard skipped the hole. Splitting the reset into a +non-blocking enqueue (`resetAsync`, called while still holding the lock) and a wait (`barrier`, called +after releasing it) closes that gap: `draining` cannot go false, so no new ticket can be issued, until +`barrier()` has confirmed the new epoch is installed. It also keeps a composition that was already in +flight before the flush from reaching `onFrame` while `HardwareVideoDecoder.flush()` is still draining. +`OrderedDeliveryTests.newEpochTicketFulfilledBeforeResetIsDeliveredIsNotLost` and +`DualLayerVideoDecoderTests.flushDoesNotLosePostFlushTicketInTheUnlockToBarrierGap` drive that window +directly. + +`close()` follows the same shape (bump epoch, capture `nextTicket`, `resetAsync` — all inside one +critical section that also sets `isClosed` — then `barrier()` after unlocking), then clears `onFrame`, +swaps the enhancement decoder out under the lock and closes it outside, and resets the stream +classification so the object cannot leak one stream's decisions into the next. A closed decoder refuses +`open` (single use) rather than being half reusable. A concurrent `flush()`/`flush()` or +`flush()`/`close()` is serialized by the decoder's lock the same way either order: `_feedEpoch` and +`deliveryNextTicket` only ever move forward under it, and the lock also fixes the order the two calls' +`resetAsync`s land in `delivery`'s own queue, so whichever lands later always carries the larger epoch — +neither the epoch nor `nextToEmit` can move backwards. + +### 5.9 Threading model + +| Thread | Does | Holds | May block on | +|---|---|---|---| +| Demux | NAL split, one libdovi parse, side-data ring insert, VideoToolbox submit | decoder lock, O(1) | nothing | +| VideoToolbox output (BL, EL) | pairing; for `compose`, slot acquisition then encode and commit | decoder lock for the pairing call only | a GPU slot, with no lock held | +| Metal completion | `slot.signal()`, GPU time sample, hop to the composer's relay queue | none | never | +| Composer relay queue (`deliveryQueue`, `.userInteractive`) | `completion` (= `delivery.fulfil` in production) | none | nothing (a slow `completion` queues up behind itself and never blocks Metal; see I7 in §5.12) | +| Delivery queue (`.userInteractive`) | ordering, epoch check, DV attachment, `onFrame` | none of ours | the renderer's enqueue | +| Host (seek, stop) | `flush()`, `close()` | decoder lock briefly; `delivery.barrier()` outside it | VideoToolbox drain, and whatever `onFrame` is doing when `barrier()` catches up to it | + +### 5.10 Budgets + +Memory held by the pipeline (4K P010 = 24.9 MB, 1080p P010 = 6.2 MB; VideoToolbox's own reference +pictures and the display layer's queue are extra): + +| Holder | Bound | +|---|---| +| Base-layer frames waiting for their EL | K = 3 → 75 MB | +| Enhancement-layer frames waiting | 10 → 62 MB | +| Composed output buffers (in flight, pending, idle in pool) | allocation threshold 15 + reserve 3 → 448 MB, the reserve freed after a second unused | + +CPU per composed frame: + +| Work | Cost | +|---|---| +| libdovi RPU parses | 1 | +| Base-layer access-unit copies | 0 (`CustomBlockSource` hands VideoToolbox the packet's own buffer) | +| Enhancement-layer copies | 1 (extraction into a standalone access unit) | +| Log lines at the default level | 0; a 1 Hz summary instead (§5.11) | +| Parameter upload | written into a preallocated `MTLBuffer` | +| Side-data lookup | the 32-entry ring, no sorting | +| Shader compilation | once per process, lazily | + +GPU: 12.9–14.3 ms per frame measured on the A12 (§4.2), against a 41.7 ms budget at 23.976 fps. A +59.94 fps source (16.7 ms budget) has far less headroom. + +### 5.11 Observability + +One `[DualLayer] summary` line per second while composing: frames composed, bypassed without EL, +bypassed without RPU, stale enhancement frames dropped, maximum observed layer skew in frames, peak +GPU slots in use, output pool buffers allocated, delivery pending peak, mean GPU ms. The existing +`[DoviFelComposer] gpu … ms/frame` line stays. `os_signpost` intervals around compose and delivery +make the pipeline readable in Instruments. Nothing else logs per frame; anything genuinely per-frame +for debugging uses `EngineLog` level `.verbose`. + +### 5.12 Invariants and the tests that hold them + +| # | Invariant | Test | +|---|---|---| +| I1 | Every base-layer frame is delivered exactly once, or dropped by a flush or close | `DualLayerVideoDecoderTests.singleLayerBypassesInOrder`, `orphanedBaseLayerIsEventuallyDelivered` | +| I2 | Delivery order is base-layer decode order within an epoch | `asyncCompositionKeepsItsPlace`; `prunedFrameDoesNotOvertakeNewerFrames` | +| I3 | Nothing from before `flush()` is delivered after it returns; nothing after `close()` returns | `flushRetiresOutstandingFrames`, `closeStopsDelivery`, `flushFromAnotherThreadInATightLoopKeepsInvariantI3` (200 concurrent flushes racing delivery, with a per-flush "started tag vs. returned count" detector), `flushDoesNotLosePostFlushTicketInTheUnlockToBarrierGap`, `flushDropsFramesEmittedFromInsideFlush`, and `OrderedDeliveryTests.newEpochTicketFulfilledBeforeResetIsDeliveredIsNotLost` | +| I4 | A frame carries its own RPU; a frame without one carries none; the compatibility ID is a string | `displayMetadataTravelsWithItsFrame` | +| I5 | Composition never invents coefficients | `missingRPUBypassesToBaseLayer` | +| I6 | Layers pair within ±300 ticks and not beyond | `jitterTolerance` | +| I7 | Held frames are bounded: waiting EL, waiting BL, GPU in flight | `orphanedEnhancementLayersAreBounded`, `lostEnhancementLayerDelaysBaseLayerByAtMostKFrames`; `DoviComposeAsyncTests.completionsAreExactlyOnceAndBounded`. **Not bounded on its own: the delivery queue.** A slow `onFrame` does not throttle pairing (a composed frame's GPU slot is released before delivery), so nothing bounds the backlog in `OrderedDelivery` itself. The demux thread's `mayEnqueueVideoFrame` gate limits how fast new packets arrive, which limits the backlog only indirectly; `flush()`'s `barrier()` waits for the queue to empty but does not cap it. | +| I8 | No lock is held across a blocking wait or a host callback | `DualLayerVideoDecoderTests.slowOnFrameDoesNotBlockDecoderOperations` (a blocked `onFrame` never stalls concurrent decoder work); `DualLayerGpuSlotWaitTests` for the GPU slot wait | +| I9 | Stream classification changes only at open, flush or close | `DualLayerStreamClassificationTests` | + +--- + +## 6. Verification + +**Unit tests.** The GPU suites need a Metal device and skip without one. + +```bash +swift test --filter "Dovi|DualLayer|OrderedDelivery|FrameSideDataRing" +``` + +| Suite | Covers | +|---|---| +| `DualLayerVideoDecoderTests` | pairing in both arrival orders, jitter tolerance, missing-RPU bypass, ordered delivery across async composition, DV metadata per frame, flush and close, bounded queues | +| `DoviComposeAsyncTests` | async output identical to blocking compose, exactly-once completion, bounded in-flight work, the lock re-entrancy deadlock | +| `DoviGpuCurveSweepTests` | bit-exact identity round trip and curve sweeps on the GPU | +| `DoviFelMathTests` | NLQ, pivots, MMR, polynomial precision, a real RPU composed on the GPU | +| `DoviDitherTests` | dither pattern, Cb / Cr phase offset, zero mean bias | +| `DoviLayoutProbeTests` | GPU reads every parameter sub-block at the Swift offset | +| `DoviShaderSyncTests` | every kernel present, parameter block size, one pipeline per process | +| `DoviRpuConverterTests` | NAL surgery, framing, degradation to HDR10 | + +**CLI** ([cli.md](cli.md); these subcommands arrive with the tooling PR that follows this one): `aetherctl feltest` (pixel and metadata verdicts, exit codes 0–5), +`dvsurvey`, `dovitest`, and `play --cycle-el` to flip the residual on a timer. + +**Device.** With `DEVELOPER_DIR` pointing at an Xcode that has `devicectl`: + +```bash +# the host app's own log file, if it writes one (a sink on `EngineLog.handler`) +xcrun devicectl device copy from --device --domain-type appDataContainer \ + --domain-identifier --source --destination ./device.log +# crash reports, including backboardd's +xcrun devicectl device info files --device --domain-type systemCrashLogs --no-recurse +# live Dolby Vision diagnostics (text logs; the signposts are categories dovi.compose and dovi.delivery) +log stream --predicate 'subsystem == "de.superuser404.AetherEngine" AND category == "sw.playback"' +``` + +`info files` takes `--subdirectory` and accepts `--username`; `copy from` takes `--source` and rejects +`--username`. The absence of a `JetsamEvent` for the day rules out a memory kill. + +**Consumption test** (the tooling and the pin hook arrive with the tooling PR that follows this one). Pick a pin with `aetherctl dvsurvey`, arm a DEBUG build with +`AetherEngine.armDolbyVisionPinTest("maxpq>=4095,period:2")` (an `@_spi(Diagnostics)` hook; on the +Mac, `aetherctl play --dv-pin`) against that asset, and watch the panel with its own dynamic +picture processing switched off. The substitution happens after composition, so a visible pulse can +only come from the metadata. + +### 6.1 Correctness + +The composer's arithmetic is held by tests that need no media: `DoviCpuModelTests` runs a CPU model of +both kernels against the GPU at 12 bits, before the final rounding (`DoviDither.debugRawVDR`, test only), +and `DoviGpuCurveSweepTests` and `DoviGpuEdgeCaseTests` sweep the curve and NLQ parameter space. On +real titles `DoviFelRealFileTests` and `DoviRealFileExportTests` run when the fixtures are present and +skip otherwise. Bit-exact output cannot be claimed: the EL is lossy in real +streams and Dolby's prediction is not public. + +### 6.2 What is and is not applied + +Applied: base layer (all planes, MMR luma reads base chroma), enhancement layer (bilinear upsampled, inverse +NLQ with offset, scale, threshold, shift, `vdr_in_max`), the per-component polynomial and MMR curves, the +bit depths, and the display metadata (carried into the converted Profile 8.1 RPU). + +- **Missing or CRC-failed RPU:** the frame borrows the previous real RPU's parameters for up to + `maxBorrowedParamsRun` (3) frames in a row, then bypasses to the base layer. Reshaping and NLQ are constant + across a scene. A frame whose RPU sets `use_prev_vdr_rpu_flag` keeps the previous mapping for as long as the + stream asks, with no limit. Both are resolved when the packet is demuxed (`recordSideData`), in decode order, + so "previous" is the previous frame of the stream and each frame reaches pairing with concrete parameters; + resolving at compose time would follow completion order instead. +- **Output precision:** `DoviDither.fullPrecision` keeps the VDR bits below the 10-bit code in P010's unused low + 6 bits. Opt-in, not yet validated on a display path; the default stays rounded or dithered 10-bit. +- **Not implemented, by design:** NLQ methods other than linear (the spec defines only linear; an unknown method + yields no residual), MEL (no residual by definition), NLQ pivots (the linear inverse has one segment). +- **Not implemented, blocked:** polynomial pieces with `poly_order_minus1 == 0` and `linear_interp_flag`. libdovi is + a prebuilt binary that aborts on them, so `DoviRpuGuard` rejects the RPU; supporting them needs a patched + libdovi (or our own full RPU parser, which the 8.1 conversion would still need libdovi for). + +### 6.3 Robustness + +- `DoviRpuGuard` checks the RPU CRC (MPEG-2 CRC32 over the unescaped bytes) and pre-scans for the paths + on which libdovi aborts, before libdovi, which is `panic=abort` and parses before it checks the CRC: + a polynomial piece with order 1 and linear interpolation (`unimplemented!` in the parser and the + writer), a reshaping curve that mixes polynomial and MMR pieces (parses, then the writer indexes the + polynomial list per piece out of bounds; mode 2 keeps the mapping of a MEL or residual-free RPU, only + an FEL RPU's is dropped before the write), pivot and block counts the payload cannot hold, and + display-management block lengths the spec does not define. This is a list of known paths, not a proof. + `DoviRpuGuardTests` holds each of them, including that the same RPU without the mix converts and + writes. `DoviCrcValidFuzzTests` mutates synthetic Profile 7 RPUs built in the test source (an FEL and + two MEL, so it runs without clips) and each clip's RPU when the clips are present, repairs the CRC, and + pushes every mutant the guard lets through the production parse, convert and write; `DoviFelFuzzTests` + does the same for corrupt RPUs and access units without the CRC repair. The random mutations do not + build the mixed curve themselves (it takes an insertion, not a bit flip), so that case rests on the + targeted tests. The clip-based fuzz and real-file tests show as skipped, not passed, without the clips. +- `CustomBlockSource` addresses the packet payload, not `buf->data` (Matroska packets sit 4 bytes in). +- Pairing is key-based: `settledBlKeys` (ring of 64) decides whether an enhancement frame's base layer + has already gone out. +- `DoviFelComposer`'s completion handler keeps the in-flight semaphore alive past the composer. + +## 7. Document map + +| Document | Role | +|---|---| +| this file | design, implementation, evidence, verification | +| [formats.md](formats.md#dolby-vision-signaling) | Profile 7 behaviour as a host sees it | +| [api.md](api.md) | `preferDualLayerDolbyVision`, `dolbyVisionHandling` | +| [cli.md](cli.md) | `feltest`, `dvsurvey`, `dovitest`, `play --cycle-el` (with the tooling PR) | diff --git a/docs/formats.md b/docs/formats.md index e1de54ebf..8862a8c88 100644 --- a/docs/formats.md +++ b/docs/formats.md @@ -198,7 +198,9 @@ For DV streams the demuxer surfaces the source's `AVDOVIDecoderConfigurationReco AV1+DV emits a bare `dav1.10.` primary for Profile 10.0 (DV-only, no base layer to fall back to), and an `av01...` primary plus a supplemental entry for the two cross-compatible profiles: `dav1.10./db1p` with `VIDEO-RANGE=PQ` for Profile 10.1 (HDR10-compat base) and `dav1.10./db4h` with `VIDEO-RANGE=HLG` for Profile 10.4 (HLG-compat base), on hardware-AV1 hosts. Same split as HEVC: the `dav1` sample entry belongs to the profile without a base layer the way `dvh1` belongs to Profile 5, and a cross-compatible profile carries its base layer's own sample entry so a client that cannot read Dolby Vision still plays it. -**Profile 7** (dual-layer, the common UHD-Blu-ray remux profile) has no decoder on any Apple platform, so the engine converts it to single-layer **Profile 8.1** live during muxing: the RPU of each video packet is rewritten with [libdovi](https://github.com/superuser404notfound/LibDovi) (`dovi_convert_rpu_with_mode`, mode 2, the same transform as `dovi_tool -m 2`), the enhancement-layer NALs are dropped, and the container `dvvC` is set to Profile 8.1. The conversion runs on every display and the result is served exactly as a Profile 8.1 is: on a DV-capable display this means real Dolby Vision (`dvh1.08/db1p` supplemental) instead of the plain HDR10 base, and on a display without Dolby Vision the kept `dvvC` lets AVPlayer tone-map from the RPU rather than from the base layer's static metadata, the way mpv plays Profile 7 without an EL decoder. It used to skip the conversion and strip the record there, which cost every Mac (no per-mode table) and every HDR10 / SDR panel the RPU and shipped the enhancement layer into segments nothing decodes. The conversion is loss-free relative to what Apple could show before (the enhancement layer was never decodable on Apple hardware). MEL and FEL sources are both handled; a Full Enhancement Layer (FEL) source is logged, since its enhancement layer, which a native Profile 7 player would fold in, is discarded here while a Minimal (MEL) source loses nothing. Any per-packet conversion failure drops that RPU so the frame degrades to the clean HDR10 base, rather than shipping a Profile 7 RPU inside a container already declared 8.1. A session that converts publishes `dolbyVisionConversion = .profile7ToProfile81`, so a host label can name the profile actually served next to `sourceDVProfile`. +**Profile 7** (dual-layer, the common UHD-Blu-ray remux profile) is handled according to `LoadOptions.preferDualLayerDolbyVision` (default `false`: dual-layer composition is opt-in) and `LoadOptions.dolbyVisionHandling`: +- **Dual-layer composition (`preferDualLayerDolbyVision = true`, `dolbyVisionHandling` `.automatic`, `el_type` FEL, VideoToolbox HEVC hardware decode)**: Profile 7 routes through `SoftwarePlaybackHost` + `DualLayerVideoDecoder`. Each packet is split into the 4K base layer (BL), the enhancement layer (EL, typically 1080p, carried in `unspec63` NAL units) and the RPU (`unspec62`). Both layers decode in their own `VTDecompressionSession`; `DoviFelComposer` applies the RPU's reshaping (polynomial luma, MMR chroma) and the inverse-NLQ residual from the upsampled EL at 12-bit precision, then an ordered 8×8 Bayer dither (inverted on odd frames) takes the result to 10-bit P010 for `AVSampleBufferDisplayLayer`. The composed frame also carries the RPU converted to Profile 8.1 as a `DolbyVisionRPUData` attachment, with `DolbyCompatibilityID` set to the string `"1"`; tvOS 27 reads it on every frame and applies Dolby Vision display management, so L1 / L2 dynamic metadata reaches the panel. A MEL source never reaches the dual-layer decoder: the route is FEL only, and MEL (nothing to compose) keeps the native route below. The route needs a seekable VOD and is logged once per Profile 7 load (`DV P7 dual-layer route: taken` / `not taken ()`). It does not consult the display: on a display that does not take Dolby Vision the route is still taken and the composed picture is presented with its attached metadata, tvOS deciding how; the native route would have played the converted Profile 8.1 there. The software host also plays an E-AC-3 Atmos track as its channel-bed core (no objects). The stream is classified by `el_type` alone; an RPU that sets `disable_residual_flag` only switches the residual off for that frame and never changes the classification. See [dolby-vision-fel.md](dolby-vision-fel.md) for the design. +- **Single-layer conversion (the default: `preferDualLayerDolbyVision = false`, or no VideoToolbox HEVC hardware decode)**: No Apple platform has a Profile 7 decoder, so the engine converts the stream to single-layer **Profile 8.1** live during muxing: the RPU of each video packet is rewritten with [libdovi](https://github.com/superuser404notfound/LibDovi) (`dovi_convert_rpu_with_mode`, mode 2, the same transform as `dovi_tool -m 2`), the enhancement-layer NALs are dropped, and the container `dvvC` is set to Profile 8.1. The conversion runs on every display and the result is served exactly as a Profile 8.1 is: on a DV-capable display this means real Dolby Vision (`dvh1.08/db1p` supplemental) instead of the plain HDR10 base, and on a display without Dolby Vision the kept `dvvC` lets AVPlayer tone-map from the RPU rather than from the base layer's static metadata, the way mpv plays Profile 7 without an EL decoder. It used to skip the conversion and strip the record there, which cost every Mac (no per-mode table) and every HDR10 / SDR panel the RPU and shipped the enhancement layer into segments nothing decodes. The conversion is loss-free relative to what Apple could show before (the enhancement layer was never decodable on Apple hardware). MEL and FEL sources are both handled; a Full Enhancement Layer (FEL) source is logged, since its enhancement layer, which a native Profile 7 player would fold in, is discarded on this route (the dual-layer route above keeps it) while a Minimal (MEL) source loses nothing. Any per-packet conversion failure drops that RPU so the frame degrades to the clean HDR10 base, rather than shipping a Profile 7 RPU inside a container already declared 8.1. A session that converts publishes `dolbyVisionConversion = .profile7ToProfile81`, so a host label can name the profile actually served next to `sourceDVProfile`. **SDR-compatible-base profiles** (HEVC **Profile 8.2**, AV1 **Profile 10.2**) carry a Rec.709 base layer that no Apple platform has a DV decoder for. The engine strips the `dvcC` / DV config and plays the base as plain `hvc1` / `av01` (logging `not DV-routable, playing Rec.709 base`); there is no Dolby Vision on any display for these, on DV panels and SDR panels alike. @@ -540,7 +542,7 @@ A remote `.iso` / `.img` / `.udf` needs an origin that honours HTTP `Range`, bec **`cancel()` unblocks, it does not invalidate.** Its whole job is to wake a `read` that is parked so teardown does not hang; a memory or file reader leaves it at the default no-op. A reader the engine may rebuild in place must be able to serve again immediately afterwards, because the rebuild reuses it. Reading `cancel()` as terminal (the way a socket's in-flight-request cancel is terminal) is the difference between a rebuilt session and a dead one: the reopen fails on its first read and the session lands in `.error` carrying libavformat's `Operation not permitted`. What that death LOOKS like is the reader's own convention, so one host defect has two signatures: a latched reader whose read returns a negative value produces the `Operation not permitted` above, while one that returns `0` is mapped to `AVERROR_EOF` and the session ENDS instead of failing, which on a live source is a stream that was never supposed to have an end (cmcpherson274). The engine cancels a reader it intends to reuse exactly once, before the successor opens. -**In-place rebuilds reuse the reader.** `reloadAtCurrentPosition()` and `reloadAtCurrentPosition(applying:)`, an audio-track switch, a disc-title switch and a background return all tear the pipeline down and reopen on the retained reader instead of reopening a URL, so a custom source gets the same session-preserving rebuild a URL source does. A reader that reports itself non-seekable is refused instead (`SessionReloadRefusal.customSourceNotSeekable`), since the rebuild cannot reposition it. +**In-place rebuilds reuse the reader.** `reloadAtCurrentPosition()` and `reloadAtCurrentPosition(applying:)`, an audio-track switch, a disc-title switch and a background return all tear the pipeline down and reopen on the retained reader instead of reopening a URL, so a custom source gets the same session-preserving rebuild a URL source does (except that the Profile 7 dual-layer route is decided by `load()`: `reloadAtCurrentPosition(applying:)` refuses a `preferDualLayerDolbyVision` change (or a `dolbyVisionHandling` change while the preference is on) on a custom Profile 7 source with `SessionReloadRefusal.dualLayerRouteNeedsFreshLoad`). A reader that reports itself non-seekable is refused instead (`SessionReloadRefusal.customSourceNotSeekable`), since the rebuild cannot reposition it. **Where the reopen starts depends on whether the source is live.** A fresh `AVIOContext` always starts its byte axis at 0, so the reader's cursor and that axis have to agree, and there are two ways to make them: From 73955ad2870070b5f985dc3b191f3f39773182d0 Mon Sep 17 00:00:00 2001 From: Pavel Kajaba <15980400+pkajaba@users.noreply.github.com> Date: Wed, 7 Oct 2026 10:37:51 +0100 Subject: [PATCH 016/132] feat(probes): Dolby Vision pulse test (clip generator, tvOS host, filming harness, analysis) Probes/DolbyVisionPulse: a reproducible measurement of whether a panel follows Dolby Vision metadata. Clips keep the pixels constant and switch only the RPU (or, for the enhancement-layer clip, only the residual) every 72 frames, so a phone filming the television shows whether the display uses the metadata and the composed pixels. It is a measurement, not a unit test, and it commits no media. - Scripts: gen_pulse.py (pattern, RPU configs, EL residual design), build.sh, serve.py, run_tv.sh, cut.py (one phone recording per arm), analyze.py (verdict per arm), report.py (the change-per-patch table and phase-A levels), check_el.py (checks on the Mac that the EL clip carries the designed residual in phase B and none in A), dump_attachments.swift (what VideoToolbox attaches to a decoded frame), sim_recording.py (a fake recording that validates the analysis). - Device/: a minimal tvOS host that plays the clips through AVPlayer, VideoToolbox and the FEL route (preferDualLayerDolbyVision). It uses only public engine API. - CI: compiles the scripts and builds the host for the tvOS simulator, so a moved public API is found in CI instead of at the television. - The pulse test is the measurement behind the consumption evidence in docs/dolby-vision-fel.md section 4.5, which arrived with the route PR. No library code changes and no behaviour change. Public API: none. Not in this PR: the aetherctl tooling. The host builds in CI; the rest needs a television and a phone. Checked here: Python and shell syntax of the scripts, swiftc -typecheck of dump_attachments.swift, xcodebuild of the host for the tvOS simulator (BUILD SUCCEEDED), swift build --build-tests of the package. --- .github/workflows/ci.yml | 24 ++ CHANGELOG.md | 1 + Probes/DolbyVisionPulse/.gitignore | 11 + .../project.pbxproj | 342 ++++++++++++++++++ .../contents.xcworkspacedata | 7 + .../xcschemes/DolbyVisionPulse.xcscheme | 91 +++++ .../Device/Host/DolbyVisionPulseHostApp.swift | 313 ++++++++++++++++ .../DolbyVisionPulse/Device/Host/Info.plist | 29 ++ .../Device/Host/PulseLog.swift | 26 ++ .../Device/Host/PulseVideoToolbox.swift | 220 +++++++++++ Probes/DolbyVisionPulse/Device/project.yml | 62 ++++ Probes/DolbyVisionPulse/README.md | 303 ++++++++++++++++ Probes/DolbyVisionPulse/Scripts/analyze.py | 339 +++++++++++++++++ Probes/DolbyVisionPulse/Scripts/build.sh | 135 +++++++ Probes/DolbyVisionPulse/Scripts/check_el.py | 226 ++++++++++++ Probes/DolbyVisionPulse/Scripts/cut.py | 144 ++++++++ .../Scripts/dump_attachments.swift | 72 ++++ Probes/DolbyVisionPulse/Scripts/fel_count.py | 168 +++++++++ Probes/DolbyVisionPulse/Scripts/gen_pulse.py | 298 +++++++++++++++ Probes/DolbyVisionPulse/Scripts/report.py | 114 ++++++ Probes/DolbyVisionPulse/Scripts/run_tv.sh | 95 +++++ Probes/DolbyVisionPulse/Scripts/serve.py | 72 ++++ .../DolbyVisionPulse/Scripts/sim_recording.py | 77 ++++ docs/dolby-vision-fel.md | 3 +- 24 files changed, 3170 insertions(+), 2 deletions(-) create mode 100644 Probes/DolbyVisionPulse/.gitignore create mode 100644 Probes/DolbyVisionPulse/Device/DolbyVisionPulse.xcodeproj/project.pbxproj create mode 100644 Probes/DolbyVisionPulse/Device/DolbyVisionPulse.xcodeproj/project.xcworkspace/contents.xcworkspacedata create mode 100644 Probes/DolbyVisionPulse/Device/DolbyVisionPulse.xcodeproj/xcshareddata/xcschemes/DolbyVisionPulse.xcscheme create mode 100644 Probes/DolbyVisionPulse/Device/Host/DolbyVisionPulseHostApp.swift create mode 100644 Probes/DolbyVisionPulse/Device/Host/Info.plist create mode 100644 Probes/DolbyVisionPulse/Device/Host/PulseLog.swift create mode 100644 Probes/DolbyVisionPulse/Device/Host/PulseVideoToolbox.swift create mode 100644 Probes/DolbyVisionPulse/Device/project.yml create mode 100644 Probes/DolbyVisionPulse/README.md create mode 100755 Probes/DolbyVisionPulse/Scripts/analyze.py create mode 100755 Probes/DolbyVisionPulse/Scripts/build.sh create mode 100755 Probes/DolbyVisionPulse/Scripts/check_el.py create mode 100755 Probes/DolbyVisionPulse/Scripts/cut.py create mode 100644 Probes/DolbyVisionPulse/Scripts/dump_attachments.swift create mode 100755 Probes/DolbyVisionPulse/Scripts/fel_count.py create mode 100755 Probes/DolbyVisionPulse/Scripts/gen_pulse.py create mode 100755 Probes/DolbyVisionPulse/Scripts/report.py create mode 100755 Probes/DolbyVisionPulse/Scripts/run_tv.sh create mode 100755 Probes/DolbyVisionPulse/Scripts/serve.py create mode 100755 Probes/DolbyVisionPulse/Scripts/sim_recording.py diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 3221315dd..553bf631a 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -74,6 +74,15 @@ jobs: - name: build the transport probe (AE#377) run: swift build --build-tests --package-path Probes/TransportProbe + # The pulse test's scripts are Python, shell and Swift nobody runs in CI; a syntax error in one is found + # at the television. Compiling them is all that can be checked without a phone recording. + - name: Compile the Dolby Vision pulse scripts + run: | + python3 -m py_compile Probes/DolbyVisionPulse/Scripts/*.py + zsh -n Probes/DolbyVisionPulse/Scripts/build.sh + zsh -n Probes/DolbyVisionPulse/Scripts/run_tv.sh + swiftc -typecheck Probes/DolbyVisionPulse/Scripts/dump_attachments.swift + build-tvos: name: xcodebuild (tvOS + visionOS Simulator) runs-on: xcode-27 @@ -115,6 +124,21 @@ jobs: -derivedDataPath .build/xcode-probe \ CODE_SIGNING_ALLOWED=NO + # The Dolby Vision pulse host is the other device harness, and the one that links the engine + # itself, so it is also the one that breaks when a public API it uses moves. It plays clips on an + # Apple TV that someone films, runs nowhere in CI, and would otherwise be found out by the person + # who needed a measurement that evening. EXCLUDED_ARCHS for the reason given above: an app cannot + # link the x86_64 slice. + - name: Build the Dolby Vision pulse host (Probes/DolbyVisionPulse) + run: | + xcodebuild build \ + -project Probes/DolbyVisionPulse/Device/DolbyVisionPulse.xcodeproj \ + -scheme DolbyVisionPulse \ + -destination 'generic/platform=tvOS Simulator' \ + -derivedDataPath .build/xcode-pulse \ + EXCLUDED_ARCHS=x86_64 \ + CODE_SIGNING_ALLOWED=NO + # #344: the package has declared visionOS since 6.0.0 and nothing here compiled for it, so an # availability list that named tvOS/iOS/macOS and then fell through to `*` broke the visionOS # build silently. `*` resolves to the declared visionOS floor (1.0), so any API newer than that diff --git a/CHANGELOG.md b/CHANGELOG.md index 269c12da4..f767bc611 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,7 @@ the public-API contract. - **The Profile 7 dual-layer route is FEL only and capability gated, and says why.** `DualLayerRouteDecision` adds a real-time budget to the route: the base and enhancement layers together may need at most UHD 3840x2160 at 30 fps plus a 1920x1080 enhancement layer at 30 fps (a 24p UHD master uses 80%), otherwise the file keeps the native route, as MEL does and as does a box without hardware HEVC decode (8.1 where the display takes Dolby Vision, the HDR10 base layer where it does not). Every Profile 7 load logs one `DV P7 dual-layer route: taken (...)` or `not taken ()` line; the line is new, and with the preference off it is the only change. - **`AETHER_FEL_MEASURE` reports the signed FEL-minus-MEL mean per plane and luma band.** The diagnostic composes every Nth frame a second time with the residual off and compares the two; the signed mean says whether the residual brightens or darkens, which the absolute difference could not. It runs one sample at a time and skips a frame that arrives while one is in flight. Off by default; it is an environment-variable diagnostic, not API. - **The dual-layer decoder pairs the two layers by picture order.** `DualLayerPictureAlignment` derives each layer's display slot from its POC (`HEVCPictureOrderCount`, no decoding) and stamps each enhancement-layer packet with the time of the base-layer picture in the same slot, so a stream whose enhancement layer is coded with B-frames over a base layer without them pairs the right pictures. A stream whose layers share one GOP keeps every packet's own time. After a seek a layer pairs by access unit until its next sequence-opening IRAP. +- **`Probes/DolbyVisionPulse`, a reproducible Dolby Vision pulse test.** Clips whose pixels are constant while only the RPU (or, for the enhancement-layer clip, only the residual) switches every 72 frames, counter clips whose base and enhancement layers each carry their own frame number (so a dropped or mispaired residual shows on screen), a tvOS host app that plays them through AVPlayer, VideoToolbox, and the FEL route, and scripts that build the clips, cut a phone recording per arm and read back whether the panel followed the metadata. It is the measurement behind [the FEL consumption evidence](docs/dolby-vision-fel.md#45-consumption-test-assets); the host builds in CI, the rest needs a television and a phone. ### Changed diff --git a/Probes/DolbyVisionPulse/.gitignore b/Probes/DolbyVisionPulse/.gitignore new file mode 100644 index 000000000..be9c2cfe9 --- /dev/null +++ b/Probes/DolbyVisionPulse/.gitignore @@ -0,0 +1,11 @@ +# Everything the scripts generate: clips, RPU binaries and configs, logs, recordings. +out/ +*.mp4 +*.mov +*.MOV +*.hevc +*.bin +*.log +recordings/ +.dd/ +__pycache__/ diff --git a/Probes/DolbyVisionPulse/Device/DolbyVisionPulse.xcodeproj/project.pbxproj b/Probes/DolbyVisionPulse/Device/DolbyVisionPulse.xcodeproj/project.pbxproj new file mode 100644 index 000000000..a24a0aded --- /dev/null +++ b/Probes/DolbyVisionPulse/Device/DolbyVisionPulse.xcodeproj/project.pbxproj @@ -0,0 +1,342 @@ +// !$*UTF8*$! +{ + archiveVersion = 1; + classes = { + }; + objectVersion = 77; + objects = { + +/* Begin PBXBuildFile section */ + 1CC1EA98D9C5F77AAC72F7CC /* PulseVideoToolbox.swift in Sources */ = {isa = PBXBuildFile; fileRef = B9672B7938DDC09AAB435BC6 /* PulseVideoToolbox.swift */; }; + 77CB08AE958747D4126970C6 /* DolbyVisionPulseHostApp.swift in Sources */ = {isa = PBXBuildFile; fileRef = 7BCA97A8853B09B4E7B4B99A /* DolbyVisionPulseHostApp.swift */; }; + C3F88D052478C6D03F65C74C /* AetherEngine in Frameworks */ = {isa = PBXBuildFile; productRef = 5DDCF6DD2D95C3E73D4B2B5C /* AetherEngine */; }; + FB3139445F0E0031753BB397 /* PulseLog.swift in Sources */ = {isa = PBXBuildFile; fileRef = DCACA2DF252349E564F73EAC /* PulseLog.swift */; }; +/* End PBXBuildFile section */ + +/* Begin PBXFileReference section */ + 6EC0C4402B0A4CFAB2376CAB /* AetherEngine */ = {isa = PBXFileReference; lastKnownFileType = folder; name = AetherEngine; path = ../../..; sourceTree = SOURCE_ROOT; }; + 7BCA97A8853B09B4E7B4B99A /* DolbyVisionPulseHostApp.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = DolbyVisionPulseHostApp.swift; sourceTree = ""; }; + 81B68F565101F4AD4C23902A /* DolbyVisionPulseHost.app */ = {isa = PBXFileReference; includeInIndex = 0; lastKnownFileType = wrapper.application; path = DolbyVisionPulseHost.app; sourceTree = BUILT_PRODUCTS_DIR; }; + B9672B7938DDC09AAB435BC6 /* PulseVideoToolbox.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = PulseVideoToolbox.swift; sourceTree = ""; }; + DCACA2DF252349E564F73EAC /* PulseLog.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = PulseLog.swift; sourceTree = ""; }; +/* End PBXFileReference section */ + +/* Begin PBXFrameworksBuildPhase section */ + 2BC4135493426C6DF7CC3B11 /* Frameworks */ = { + isa = PBXFrameworksBuildPhase; + buildActionMask = 2147483647; + files = ( + C3F88D052478C6D03F65C74C /* AetherEngine in Frameworks */, + ); + runOnlyForDeploymentPostprocessing = 0; + }; +/* End PBXFrameworksBuildPhase section */ + +/* Begin PBXGroup section */ + 877A8AC0E76ED624DD6E3BF6 /* Packages */ = { + isa = PBXGroup; + children = ( + 6EC0C4402B0A4CFAB2376CAB /* AetherEngine */, + ); + name = Packages; + sourceTree = ""; + }; + 91F1FECFC9FBC63638DFB9F0 = { + isa = PBXGroup; + children = ( + A0F3EC54F7CC8065A3A7DFAC /* Host */, + 877A8AC0E76ED624DD6E3BF6 /* Packages */, + A0369D737D33D357FE77916A /* Products */, + ); + sourceTree = ""; + }; + A0369D737D33D357FE77916A /* Products */ = { + isa = PBXGroup; + children = ( + 81B68F565101F4AD4C23902A /* DolbyVisionPulseHost.app */, + ); + name = Products; + sourceTree = ""; + }; + A0F3EC54F7CC8065A3A7DFAC /* Host */ = { + isa = PBXGroup; + children = ( + 7BCA97A8853B09B4E7B4B99A /* DolbyVisionPulseHostApp.swift */, + DCACA2DF252349E564F73EAC /* PulseLog.swift */, + B9672B7938DDC09AAB435BC6 /* PulseVideoToolbox.swift */, + ); + path = Host; + sourceTree = ""; + }; +/* End PBXGroup section */ + +/* Begin PBXNativeTarget section */ + 1DDB0EFB1D0FFFD6D891F260 /* DolbyVisionPulseHost */ = { + isa = PBXNativeTarget; + buildConfigurationList = AF89BEB4939AE144202F15AF /* Build configuration list for PBXNativeTarget "DolbyVisionPulseHost" */; + buildPhases = ( + 3DBDD39040B8692C0BA1124E /* Sources */, + 2BC4135493426C6DF7CC3B11 /* Frameworks */, + ); + buildRules = ( + ); + dependencies = ( + ); + name = DolbyVisionPulseHost; + packageProductDependencies = ( + 5DDCF6DD2D95C3E73D4B2B5C /* AetherEngine */, + ); + productName = DolbyVisionPulseHost; + productReference = 81B68F565101F4AD4C23902A /* DolbyVisionPulseHost.app */; + productType = "com.apple.product-type.application"; + }; +/* End PBXNativeTarget section */ + +/* Begin PBXProject section */ + 3E6283C4018EFBF976379C00 /* Project object */ = { + isa = PBXProject; + attributes = { + BuildIndependentTargetsInParallel = YES; + LastUpgradeCheck = 1430; + TargetAttributes = { + 1DDB0EFB1D0FFFD6D891F260 = { + ProvisioningStyle = Automatic; + }; + }; + }; + buildConfigurationList = 7345ED8B0955685E060F8EA6 /* Build configuration list for PBXProject "DolbyVisionPulse" */; + developmentRegion = en; + hasScannedForEncodings = 0; + knownRegions = ( + Base, + en, + ); + mainGroup = 91F1FECFC9FBC63638DFB9F0; + minimizedProjectReferenceProxies = 1; + packageReferences = ( + A1D5E5D261EA7A828134329D /* XCLocalSwiftPackageReference "../../.." */, + ); + preferredProjectObjectVersion = 77; + productRefGroup = A0369D737D33D357FE77916A /* Products */; + projectDirPath = ""; + projectRoot = ""; + targets = ( + 1DDB0EFB1D0FFFD6D891F260 /* DolbyVisionPulseHost */, + ); + }; +/* End PBXProject section */ + +/* Begin PBXSourcesBuildPhase section */ + 3DBDD39040B8692C0BA1124E /* Sources */ = { + isa = PBXSourcesBuildPhase; + buildActionMask = 2147483647; + files = ( + 77CB08AE958747D4126970C6 /* DolbyVisionPulseHostApp.swift in Sources */, + FB3139445F0E0031753BB397 /* PulseLog.swift in Sources */, + 1CC1EA98D9C5F77AAC72F7CC /* PulseVideoToolbox.swift in Sources */, + ); + runOnlyForDeploymentPostprocessing = 0; + }; +/* End PBXSourcesBuildPhase section */ + +/* Begin XCBuildConfiguration section */ + 058CE76FD1067ADB93FFA4B8 /* Release */ = { + isa = XCBuildConfiguration; + buildSettings = { + ASSETCATALOG_COMPILER_APPICON_NAME = "App Icon & Top Shelf Image"; + ASSETCATALOG_COMPILER_LAUNCHIMAGE_NAME = LaunchImage; + GENERATE_INFOPLIST_FILE = NO; + INFOPLIST_FILE = Host/Info.plist; + LD_RUNPATH_SEARCH_PATHS = ( + "$(inherited)", + "@executable_path/Frameworks", + ); + PRODUCT_BUNDLE_IDENTIFIER = "$(PULSE_BUNDLE_PREFIX).host"; + SDKROOT = appletvos; + TARGETED_DEVICE_FAMILY = 3; + }; + name = Release; + }; + 42160C4F45DACBA011092C36 /* Debug */ = { + isa = XCBuildConfiguration; + buildSettings = { + ALWAYS_SEARCH_USER_PATHS = NO; + CLANG_ANALYZER_NONNULL = YES; + CLANG_ANALYZER_NUMBER_OBJECT_CONVERSION = YES_AGGRESSIVE; + CLANG_CXX_LANGUAGE_STANDARD = "gnu++14"; + CLANG_CXX_LIBRARY = "libc++"; + CLANG_ENABLE_MODULES = YES; + CLANG_ENABLE_OBJC_ARC = YES; + CLANG_ENABLE_OBJC_WEAK = YES; + CLANG_WARN_BLOCK_CAPTURE_AUTORELEASING = YES; + CLANG_WARN_BOOL_CONVERSION = YES; + CLANG_WARN_COMMA = YES; + CLANG_WARN_CONSTANT_CONVERSION = YES; + CLANG_WARN_DEPRECATED_OBJC_IMPLEMENTATIONS = YES; + CLANG_WARN_DIRECT_OBJC_ISA_USAGE = YES_ERROR; + CLANG_WARN_DOCUMENTATION_COMMENTS = YES; + CLANG_WARN_EMPTY_BODY = YES; + CLANG_WARN_ENUM_CONVERSION = YES; + CLANG_WARN_INFINITE_RECURSION = YES; + CLANG_WARN_INT_CONVERSION = YES; + CLANG_WARN_NON_LITERAL_NULL_CONVERSION = YES; + CLANG_WARN_OBJC_IMPLICIT_RETAIN_SELF = YES; + CLANG_WARN_OBJC_LITERAL_CONVERSION = YES; + CLANG_WARN_OBJC_ROOT_CLASS = YES_ERROR; + CLANG_WARN_QUOTED_INCLUDE_IN_FRAMEWORK_HEADER = YES; + CLANG_WARN_RANGE_LOOP_ANALYSIS = YES; + CLANG_WARN_STRICT_PROTOTYPES = YES; + CLANG_WARN_SUSPICIOUS_MOVE = YES; + CLANG_WARN_UNGUARDED_AVAILABILITY = YES_AGGRESSIVE; + CLANG_WARN_UNREACHABLE_CODE = YES; + CLANG_WARN__DUPLICATE_METHOD_MATCH = YES; + CODE_SIGN_STYLE = Automatic; + COPY_PHASE_STRIP = NO; + DEBUG_INFORMATION_FORMAT = dwarf; + ENABLE_STRICT_OBJC_MSGSEND = YES; + ENABLE_TESTABILITY = YES; + ENABLE_USER_SCRIPT_SANDBOXING = YES; + GCC_C_LANGUAGE_STANDARD = gnu11; + GCC_DYNAMIC_NO_PIC = NO; + GCC_NO_COMMON_BLOCKS = YES; + GCC_OPTIMIZATION_LEVEL = 0; + GCC_PREPROCESSOR_DEFINITIONS = ( + "$(inherited)", + "DEBUG=1", + ); + GCC_WARN_64_TO_32_BIT_CONVERSION = YES; + GCC_WARN_ABOUT_RETURN_TYPE = YES_ERROR; + GCC_WARN_UNDECLARED_SELECTOR = YES; + GCC_WARN_UNINITIALIZED_AUTOS = YES_AGGRESSIVE; + GCC_WARN_UNUSED_FUNCTION = YES; + GCC_WARN_UNUSED_VARIABLE = YES; + MTL_ENABLE_DEBUG_INFO = INCLUDE_SOURCE; + MTL_FAST_MATH = YES; + ONLY_ACTIVE_ARCH = YES; + PRODUCT_NAME = "$(TARGET_NAME)"; + PULSE_BUNDLE_PREFIX = de.superuser404.dolbyvisionpulse; + SDKROOT = appletvos; + SWIFT_ACTIVE_COMPILATION_CONDITIONS = DEBUG; + SWIFT_OPTIMIZATION_LEVEL = "-Onone"; + SWIFT_VERSION = 6.0; + TARGETED_DEVICE_FAMILY = 3; + TVOS_DEPLOYMENT_TARGET = 18.0; + }; + name = Debug; + }; + 58026C6AF77395FFB0ED71C2 /* Debug */ = { + isa = XCBuildConfiguration; + buildSettings = { + ASSETCATALOG_COMPILER_APPICON_NAME = "App Icon & Top Shelf Image"; + ASSETCATALOG_COMPILER_LAUNCHIMAGE_NAME = LaunchImage; + GENERATE_INFOPLIST_FILE = NO; + INFOPLIST_FILE = Host/Info.plist; + LD_RUNPATH_SEARCH_PATHS = ( + "$(inherited)", + "@executable_path/Frameworks", + ); + PRODUCT_BUNDLE_IDENTIFIER = "$(PULSE_BUNDLE_PREFIX).host"; + SDKROOT = appletvos; + TARGETED_DEVICE_FAMILY = 3; + }; + name = Debug; + }; + 6CACE8A95F8A6D578B4F7D53 /* Release */ = { + isa = XCBuildConfiguration; + buildSettings = { + ALWAYS_SEARCH_USER_PATHS = NO; + CLANG_ANALYZER_NONNULL = YES; + CLANG_ANALYZER_NUMBER_OBJECT_CONVERSION = YES_AGGRESSIVE; + CLANG_CXX_LANGUAGE_STANDARD = "gnu++14"; + CLANG_CXX_LIBRARY = "libc++"; + CLANG_ENABLE_MODULES = YES; + CLANG_ENABLE_OBJC_ARC = YES; + CLANG_ENABLE_OBJC_WEAK = YES; + CLANG_WARN_BLOCK_CAPTURE_AUTORELEASING = YES; + CLANG_WARN_BOOL_CONVERSION = YES; + CLANG_WARN_COMMA = YES; + CLANG_WARN_CONSTANT_CONVERSION = YES; + CLANG_WARN_DEPRECATED_OBJC_IMPLEMENTATIONS = YES; + CLANG_WARN_DIRECT_OBJC_ISA_USAGE = YES_ERROR; + CLANG_WARN_DOCUMENTATION_COMMENTS = YES; + CLANG_WARN_EMPTY_BODY = YES; + CLANG_WARN_ENUM_CONVERSION = YES; + CLANG_WARN_INFINITE_RECURSION = YES; + CLANG_WARN_INT_CONVERSION = YES; + CLANG_WARN_NON_LITERAL_NULL_CONVERSION = YES; + CLANG_WARN_OBJC_IMPLICIT_RETAIN_SELF = YES; + CLANG_WARN_OBJC_LITERAL_CONVERSION = YES; + CLANG_WARN_OBJC_ROOT_CLASS = YES_ERROR; + CLANG_WARN_QUOTED_INCLUDE_IN_FRAMEWORK_HEADER = YES; + CLANG_WARN_RANGE_LOOP_ANALYSIS = YES; + CLANG_WARN_STRICT_PROTOTYPES = YES; + CLANG_WARN_SUSPICIOUS_MOVE = YES; + CLANG_WARN_UNGUARDED_AVAILABILITY = YES_AGGRESSIVE; + CLANG_WARN_UNREACHABLE_CODE = YES; + CLANG_WARN__DUPLICATE_METHOD_MATCH = YES; + CODE_SIGN_STYLE = Automatic; + COPY_PHASE_STRIP = NO; + DEBUG_INFORMATION_FORMAT = "dwarf-with-dsym"; + ENABLE_NS_ASSERTIONS = NO; + ENABLE_STRICT_OBJC_MSGSEND = YES; + ENABLE_USER_SCRIPT_SANDBOXING = YES; + GCC_C_LANGUAGE_STANDARD = gnu11; + GCC_NO_COMMON_BLOCKS = YES; + GCC_WARN_64_TO_32_BIT_CONVERSION = YES; + GCC_WARN_ABOUT_RETURN_TYPE = YES_ERROR; + GCC_WARN_UNDECLARED_SELECTOR = YES; + GCC_WARN_UNINITIALIZED_AUTOS = YES_AGGRESSIVE; + GCC_WARN_UNUSED_FUNCTION = YES; + GCC_WARN_UNUSED_VARIABLE = YES; + MTL_ENABLE_DEBUG_INFO = NO; + MTL_FAST_MATH = YES; + PRODUCT_NAME = "$(TARGET_NAME)"; + PULSE_BUNDLE_PREFIX = de.superuser404.dolbyvisionpulse; + SDKROOT = appletvos; + SWIFT_COMPILATION_MODE = wholemodule; + SWIFT_OPTIMIZATION_LEVEL = "-O"; + SWIFT_VERSION = 6.0; + TARGETED_DEVICE_FAMILY = 3; + TVOS_DEPLOYMENT_TARGET = 18.0; + }; + name = Release; + }; +/* End XCBuildConfiguration section */ + +/* Begin XCConfigurationList section */ + 7345ED8B0955685E060F8EA6 /* Build configuration list for PBXProject "DolbyVisionPulse" */ = { + isa = XCConfigurationList; + buildConfigurations = ( + 42160C4F45DACBA011092C36 /* Debug */, + 6CACE8A95F8A6D578B4F7D53 /* Release */, + ); + defaultConfigurationIsVisible = 0; + defaultConfigurationName = Debug; + }; + AF89BEB4939AE144202F15AF /* Build configuration list for PBXNativeTarget "DolbyVisionPulseHost" */ = { + isa = XCConfigurationList; + buildConfigurations = ( + 58026C6AF77395FFB0ED71C2 /* Debug */, + 058CE76FD1067ADB93FFA4B8 /* Release */, + ); + defaultConfigurationIsVisible = 0; + defaultConfigurationName = Debug; + }; +/* End XCConfigurationList section */ + +/* Begin XCLocalSwiftPackageReference section */ + A1D5E5D261EA7A828134329D /* XCLocalSwiftPackageReference "../../.." */ = { + isa = XCLocalSwiftPackageReference; + relativePath = ../../..; + }; +/* End XCLocalSwiftPackageReference section */ + +/* Begin XCSwiftPackageProductDependency section */ + 5DDCF6DD2D95C3E73D4B2B5C /* AetherEngine */ = { + isa = XCSwiftPackageProductDependency; + productName = AetherEngine; + }; +/* End XCSwiftPackageProductDependency section */ + }; + rootObject = 3E6283C4018EFBF976379C00 /* Project object */; +} diff --git a/Probes/DolbyVisionPulse/Device/DolbyVisionPulse.xcodeproj/project.xcworkspace/contents.xcworkspacedata b/Probes/DolbyVisionPulse/Device/DolbyVisionPulse.xcodeproj/project.xcworkspace/contents.xcworkspacedata new file mode 100644 index 000000000..919434a62 --- /dev/null +++ b/Probes/DolbyVisionPulse/Device/DolbyVisionPulse.xcodeproj/project.xcworkspace/contents.xcworkspacedata @@ -0,0 +1,7 @@ + + + + + diff --git a/Probes/DolbyVisionPulse/Device/DolbyVisionPulse.xcodeproj/xcshareddata/xcschemes/DolbyVisionPulse.xcscheme b/Probes/DolbyVisionPulse/Device/DolbyVisionPulse.xcodeproj/xcshareddata/xcschemes/DolbyVisionPulse.xcscheme new file mode 100644 index 000000000..5edd4a080 --- /dev/null +++ b/Probes/DolbyVisionPulse/Device/DolbyVisionPulse.xcodeproj/xcshareddata/xcschemes/DolbyVisionPulse.xcscheme @@ -0,0 +1,91 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/Probes/DolbyVisionPulse/Device/Host/DolbyVisionPulseHostApp.swift b/Probes/DolbyVisionPulse/Device/Host/DolbyVisionPulseHostApp.swift new file mode 100644 index 000000000..4dd62a30a --- /dev/null +++ b/Probes/DolbyVisionPulse/Device/Host/DolbyVisionPulseHostApp.swift @@ -0,0 +1,313 @@ +import AetherEngine +import AVFoundation +import AVKit +import CoreMedia +import SwiftUI +import UIKit + +// Device harness for the Dolby Vision pulse test (see ../../README.md). Every arm plays a clip whose +// pixels are constant while only the RPU (or, for the EL clip, only the residual) switches, so a +// camera pointed at the television can tell whether the metadata reached the panel. +// +// Arms come from the environment variable `PULSE_ARMS`, which `devicectl device process launch +// --environment-variables` does deliver to an app process: +// +// PULSE_ARMS="label|mode|url[|k=v;k=v],label|mode|url,..." +// +// Arms play back to back with a 5 s slate naming the arm in between, so one phone recording covers +// the whole sequence. Modes: +// +// - `avplayer`: plain AVPlayer + AVPlayerLayer on the file (Apple's own path, the positive control). +// - `vt`: AVAssetReader (VideoToolbox) output straight into AVSampleBufferDisplayLayer, with the +// attachments VideoToolbox set. +// - `vtcopy`: as `vt`, but every frame copied into a buffer from our own IOSurface pool, with all of +// VideoToolbox's propagated attachments (Dolby Vision RPU included) copied across byte for byte. +// - `vtstrip`: as `vtcopy`, without `DolbyVisionRPUData` / `DolbyCompatibilityID`. +// - `engine`: AetherEngine with `preferDualLayerDolbyVision: true` (a Profile 7 FEL clip takes the +// dual-layer route). +// - `enginenofel`: as `engine` with `preferDualLayerDolbyVision: false` (native 8.1 route, EL dropped). +// - `soak`: `engine` for `minutes=` with a `seeks=at@to,...` schedule, sampled once a second, ending in +// one `soak summary` line. +// +// The VT modes request Dolby Vision display criteria from the track's own format description, as an +// app on this path would. + +@main +struct DolbyVisionPulseHostApp: App { + var body: some Scene { + WindowGroup { + RootView() + .onAppear { + // About eleven minutes of playback, partly not AVPlayer: keep the screensaver away. + UIApplication.shared.isIdleTimerDisabled = true + } + } + } +} + +private struct RootView: View { + var body: some View { + if let arms = PulseArm.fromEnvironment() { + PulseHarnessView(arms: arms) + } else { + VStack(alignment: .leading, spacing: 24) { + Text("Dolby Vision pulse host").font(.title2) + Text("No PULSE_ARMS in the environment.").font(.title3) + Text(""" + Launch with devicectl device process launch --environment-variables \ + '{"PULSE_ARMS":"label|mode|url,..."}'. Scripts/run_tv.sh does it. + """) + .foregroundStyle(.secondary) + } + .padding(80) + .frame(maxWidth: .infinity, maxHeight: .infinity, alignment: .topLeading) + } + } +} + +struct PulseArm { + let label: String + let mode: String + let url: URL + /// Optional 4th field, `key=value;key=value` (soak: `minutes=5;seeks=60@1800,150@600`). + let params: [String: String] + + static func fromEnvironment() -> [PulseArm]? { + guard let raw = ProcessInfo.processInfo.environment["PULSE_ARMS"] else { return nil } + return parse(raw) + } + + static func parse(_ raw: String) -> [PulseArm]? { + // Arms are comma separated, but a soak's `seeks=` list contains commas too: a new arm starts + // only at a piece shaped `label|mode|...`, anything else belongs to the previous arm's params. + var items: [String] = [] + for piece in raw.split(separator: ",", omittingEmptySubsequences: true).map(String.init) { + if piece.contains("|") || items.isEmpty { items.append(piece) } else { items[items.count - 1] += "," + piece } + } + let arms = items.compactMap { item -> PulseArm? in + let parts = item.split(separator: "|", maxSplits: 3).map(String.init) + guard parts.count >= 3, let url = URL(string: parts[2]) else { return nil } + var params: [String: String] = [:] + if parts.count == 4 { + for kv in parts[3].split(separator: ";") { + let p = kv.split(separator: "=", maxSplits: 1).map(String.init) + if p.count == 2 { params[p[0]] = p[1] } + } + } + return PulseArm(label: parts[0], mode: parts[1], url: url, params: params) + } + return arms.isEmpty ? nil : arms + } +} + +struct PulseHarnessView: View { + let arms: [PulseArm] + @State private var engine: AetherEngine? + @State private var slate: String = "" + @State private var host = PulseHostView() + + var body: some View { + ZStack { + Color.black.ignoresSafeArea() + PulseHostRepresentable(host: host).ignoresSafeArea() + if let engine { + AetherPlayerSurface(engine: engine).ignoresSafeArea() + } + if !slate.isEmpty { + VStack(spacing: 30) { + Text(slate).font(.system(size: 160, weight: .heavy)).foregroundColor(.white) + // Live UTC clock: one readable slate frame pins the video time to wall time. + TimelineView(.periodic(from: .now, by: 0.1)) { ctx in + Text(Self.clock(ctx.date)) + .font(.system(size: 96, weight: .bold).monospacedDigit()) + .foregroundColor(.white) + } + Text("dv-pulse").font(.system(size: 48)).foregroundColor(.gray) + } + .frame(maxWidth: .infinity, maxHeight: .infinity) + .background(Color.black) + } + } + .task { await run() } + } + + private static func clock(_ d: Date) -> String { + let s = d.timeIntervalSince1970 + let day = s.truncatingRemainder(dividingBy: 86400) + return String(format: "%02d:%02d:%04.1f UTC", Int(day / 3600), Int(day / 60) % 60, + day.truncatingRemainder(dividingBy: 60)) + } + + @MainActor + private func run() async { + EngineLog.handler = { line in + PulseLog.shared.note(line) + print(line) + fflush(stdout) + } + let started = Date() + // Wall-clock epoch seconds on every line, so the phone video (whose creation date and the + // slate clock give its own wall-clock start) can be cut per arm afterwards. + let say: @Sendable (String) -> Void = { s in + let now = Date() + let line = String(format: "[Pulse] wall=%.3f t=%.1f %@", now.timeIntervalSince1970, + now.timeIntervalSince(started), s) + print(line) + fflush(stdout) + PulseLog.shared.note(line) + } + say("sequence: " + arms.map { "\($0.label)(\($0.mode))" }.joined(separator: " ")) + for (n, arm) in arms.enumerated() { + slate = "\(n + 1)/\(arms.count) \(arm.label)" + host.window?.avDisplayManager.preferredDisplayCriteria = nil + say("slate \(arm.label)") + try? await Task.sleep(nanoseconds: 5_000_000_000) + slate = "" + say("arm \(arm.label) start mode=\(arm.mode) url=\(arm.url.lastPathComponent)") + switch arm.mode { + case "avplayer": + await host.playAVPlayer(arm.url, log: say) + case "vt", "vtcopy", "vtstrip": + await host.playVT(arm.url, mode: arm.mode, log: say) + case "engine": + await playEngine(arm.url, fel: true, log: say) + case "enginenofel": + await playEngine(arm.url, fel: false, log: say) + case "soak": + await playSoak(arm, log: say) + default: + say("unknown mode \(arm.mode), skipped") + } + say("arm \(arm.label) end") + } + slate = "done" + host.window?.avDisplayManager.preferredDisplayCriteria = nil + try? await Task.sleep(nanoseconds: 3_000_000_000) + say("sequence done") + exit(0) + } + + @MainActor + private func playEngine(_ url: URL, fel: Bool, log: @Sendable (String) -> Void) async { + guard let e = try? AetherEngine() else { log("engine init failed"); return } + engine = e + do { + try await e.load(url: url, startPosition: nil, + options: LoadOptions(dolbyVisionHandling: .automatic, preferDualLayerDolbyVision: fel, + audioBridgeMode: .surroundCompat), + audioSourceStreamIndex: nil) + e.play() + } catch { + log("engine load failed: \(error)") + engine = nil + return + } + let deadline = Date().addingTimeInterval(80) + while Date() < deadline, e.state != .ended { + try? await Task.sleep(nanoseconds: 500_000_000) + } + log("engine state=\(e.state) t=\(e.currentTime)") + e.stop() + engine = nil + } + + /// Long-run FEL playback with a seek schedule. Measures only what the engine exposes publicly: + /// - droppedFrames: `liveTelemetry.droppedFrameCount` (render synchronizer's dropped frames on the + /// software path, session total; nil reads as -1). + /// - composeFailures: sum of `composeFailures=` over the engine's 1 Hz `[DualLayer] summary` log + /// lines (seen through EngineLog.handler; the counter itself is internal). Stays 0 if the FEL + /// route never ran, so `route=` is the thing to check first. + /// - footprint: task_vm_info phys_footprint, sampled each second. + @MainActor + private func playSoak(_ arm: PulseArm, log: @Sendable (String) -> Void) async { + let minutes = arm.params["minutes"].flatMap(Double.init) ?? 5 + var seeks: [(at: Double, to: Double)] = [] + for item in (arm.params["seeks"] ?? "").split(separator: ",") { + let p = item.split(separator: "@") + if p.count == 2, let at = Double(p[0]), let to = Double(p[1]) { seeks.append((at, to)) } + } + seeks.sort { $0.at < $1.at } + let tally = SoakTally() + let previous = EngineLog.handler + EngineLog.handler = { line in + previous?(line) + tally.ingest(line) + } + defer { EngineLog.handler = previous } + guard let e = try? AetherEngine() else { log("engine init failed"); return } + engine = e + do { + try await e.load(url: arm.url, startPosition: nil, + options: LoadOptions(dolbyVisionHandling: .automatic, preferDualLayerDolbyVision: true, + audioBridgeMode: .surroundCompat), + audioSourceStreamIndex: nil) + e.play() + } catch { + log("engine load failed: \(error)") + engine = nil + return + } + log("soak start minutes=\(minutes) seeks=\(seeks.count)") + let started = Date() + let deadline = started.addingTimeInterval(minutes * 60) + var peak = Self.footprintMB() + var played = 0.0 + var nSeeks = 0 + var lastTick = Date() + while Date() < deadline { + try? await Task.sleep(nanoseconds: 1_000_000_000) + let now = Date() + if e.state == .playing { played += now.timeIntervalSince(lastTick) } + lastTick = now + peak = max(peak, Self.footprintMB()) + if let next = seeks.first, now.timeIntervalSince(started) >= next.at { + seeks.removeFirst() + nSeeks += 1 + await e.seek(to: next.to) + log(String(format: "seek to=%.1f landed=%.1f", next.to, e.currentTime)) + lastTick = Date() + } + if e.state == .ended { log("soak ended early"); break } + } + let dropped = e.liveTelemetry?.droppedFrameCount ?? -1 + let route = tally.sawDualLayer ? "FEL" : "\(e.videoRoute.rawValue)-noDualLayerSummary" + log(String(format: "soak summary: played=%.0fs seeks=%d droppedFrames=%d composeFailures=%d peakFootprintMB=%d endFootprintMB=%d route=%@", + played, nSeeks, dropped, tally.composeFailures, peak, Self.footprintMB(), route)) + e.stop() + engine = nil + } + + /// `phys_footprint` (what jetsam judges), MB. + static func footprintMB() -> Int { + var info = task_vm_info_data_t() + var count = mach_msg_type_number_t(MemoryLayout.size / MemoryLayout.size) + let kr = withUnsafeMutablePointer(to: &info) { + $0.withMemoryRebound(to: integer_t.self, capacity: Int(count)) { + task_info(mach_task_self_, task_flavor_t(TASK_VM_INFO), $0, &count) + } + } + return kr == KERN_SUCCESS ? Int(info.phys_footprint / 1024 / 1024) : -1 + } +} + +/// Accumulates the engine's `[DualLayer] summary: ... composeFailures=N ...` lines (1 Hz, per-second deltas). +private final class SoakTally: @unchecked Sendable { + private let lock = NSLock() + private var failures = 0 + private var seen = false + var composeFailures: Int { lock.lock(); defer { lock.unlock() }; return failures } + var sawDualLayer: Bool { lock.lock(); defer { lock.unlock() }; return seen } + func ingest(_ line: String) { + guard line.contains("[DualLayer] summary:") else { return } + lock.lock(); defer { lock.unlock() } + seen = true + if let r = line.range(of: "composeFailures="), + let n = Int(line[r.upperBound...].prefix { $0.isNumber }) { failures += n } + } +} + +private struct PulseHostRepresentable: UIViewRepresentable { + let host: PulseHostView + func makeUIView(context: Context) -> PulseHostView { host } + func updateUIView(_ uiView: PulseHostView, context: Context) {} +} diff --git a/Probes/DolbyVisionPulse/Device/Host/Info.plist b/Probes/DolbyVisionPulse/Device/Host/Info.plist new file mode 100644 index 000000000..00222c321 --- /dev/null +++ b/Probes/DolbyVisionPulse/Device/Host/Info.plist @@ -0,0 +1,29 @@ + + + + + CFBundleDevelopmentRegion + $(DEVELOPMENT_LANGUAGE) + CFBundleDisplayName + Dolby Vision pulse + CFBundleExecutable + $(EXECUTABLE_NAME) + CFBundleIdentifier + $(PRODUCT_BUNDLE_IDENTIFIER) + CFBundleInfoDictionaryVersion + 6.0 + CFBundleName + $(PRODUCT_NAME) + CFBundlePackageType + APPL + CFBundleShortVersionString + 1.0 + CFBundleVersion + 1 + NSAppTransportSecurity + + NSAllowsArbitraryLoads + + + + diff --git a/Probes/DolbyVisionPulse/Device/Host/PulseLog.swift b/Probes/DolbyVisionPulse/Device/Host/PulseLog.swift new file mode 100644 index 000000000..0d1a476d2 --- /dev/null +++ b/Probes/DolbyVisionPulse/Device/Host/PulseLog.swift @@ -0,0 +1,26 @@ +import Foundation + +/// A tiny file logger: every line the harness and the engine print also lands in +/// `Library/Caches/pulse-log.txt`, which `devicectl device copy from` can pull after a run, including +/// after a crash that took the console with it. Truncated at launch. +final class PulseLog: @unchecked Sendable { + static let shared = PulseLog() + + private let queue = DispatchQueue(label: "pulse.log") + private let url: URL + private var handle: FileHandle? + + private init() { + let caches = FileManager.default.urls(for: .cachesDirectory, in: .userDomainMask)[0] + url = caches.appendingPathComponent("pulse-log.txt") + FileManager.default.createFile(atPath: url.path, contents: nil) + handle = try? FileHandle(forWritingTo: url) + } + + func note(_ line: String) { + queue.async { [self] in + guard let data = (line + "\n").data(using: .utf8) else { return } + try? handle?.write(contentsOf: data) + } + } +} diff --git a/Probes/DolbyVisionPulse/Device/Host/PulseVideoToolbox.swift b/Probes/DolbyVisionPulse/Device/Host/PulseVideoToolbox.swift new file mode 100644 index 000000000..04f6a6bd4 --- /dev/null +++ b/Probes/DolbyVisionPulse/Device/Host/PulseVideoToolbox.swift @@ -0,0 +1,220 @@ +import AVFoundation +import AVKit +import CoreMedia +import UIKit + +// The AVPlayer and VideoToolbox arms of the pulse harness: the hosts the engine's own route is +// compared against. None of this touches AetherEngine. + +final class PulseHostView: UIView { + + override func layoutSubviews() { + super.layoutSubviews() + layer.sublayers?.forEach { $0.frame = bounds } + } + + @MainActor + func playAVPlayer(_ url: URL, log: @escaping @Sendable (String) -> Void) async { + let item = AVPlayerItem(url: url) + let player = AVPlayer(playerItem: item) + // AVPlayerLayer has no automatic criteria (that is AVPlayerViewController's); request them from the + // track's own format description exactly as the VT arms do. + if let track = try? await item.asset.loadTracks(withMediaType: .video).first, + let fd = try? await track.load(.formatDescriptions).first, + let fps = try? await track.load(.nominalFrameRate) { + await requestCriteria(fd: fd, fps: fps, log: log) + } + let pl = AVPlayerLayer(player: player) + pl.frame = bounds + pl.videoGravity = .resizeAspect + layer.addSublayer(pl) + player.play() + let deadline = Date().addingTimeInterval(80) + let duration = (try? await item.asset.load(.duration).seconds) ?? 66 + while Date() < deadline, item.status != .failed, player.currentTime().seconds < duration - 0.2 { + try? await Task.sleep(nanoseconds: 500_000_000) + } + log("avplayer status=\(item.status.rawValue) error=\(String(describing: item.error)) t=\(player.currentTime().seconds)") + player.pause() + pl.removeFromSuperlayer() + } + + @MainActor + func playVT(_ url: URL, mode: String, log: @escaping @Sendable (String) -> Void) async { + do { + // AVAssetReader only reads local files: fetch the clip into the app's temp dir first. + var localURL = url + if !url.isFileURL { + let (tmp, _) = try await URLSession.shared.download(from: url) + localURL = FileManager.default.temporaryDirectory.appendingPathComponent(url.lastPathComponent) + try? FileManager.default.removeItem(at: localURL) + try FileManager.default.moveItem(at: tmp, to: localURL) + log("downloaded \(url.lastPathComponent) to temp") + } + let asset = AVURLAsset(url: localURL) + guard let track = try await asset.loadTracks(withMediaType: .video).first, + let fd = try await track.load(.formatDescriptions).first else { log("no video track"); return } + let fps = try await track.load(.nominalFrameRate) + await requestCriteria(fd: fd, fps: fps, log: log) + + let reader = try AVAssetReader(asset: asset) + let out = AVAssetReaderTrackOutput(track: track, outputSettings: [ + kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange, + kCVPixelBufferIOSurfacePropertiesKey as String: [String: Any]()]) + out.alwaysCopiesSampleData = false + reader.add(out) + guard reader.startReading() else { log("reader failed: \(String(describing: reader.error))"); return } + + let dl = AVSampleBufferDisplayLayer() + dl.frame = bounds + dl.videoGravity = .resizeAspect + var tb: CMTimebase? + CMTimebaseCreateWithSourceClock(allocator: kCFAllocatorDefault, sourceClock: CMClockGetHostTimeClock(), timebaseOut: &tb) + guard let tb else { log("timebase failed"); return } + dl.controlTimebase = tb + layer.addSublayer(dl) + + let renderer = dl.sampleBufferRenderer + let queue = DispatchQueue(label: "dv-pulse.feed") + let state = FeedState() + let copier = PulseFrameCopier() + nonisolated(unsafe) let output = out + nonisolated(unsafe) let r = renderer + renderer.requestMediaDataWhenReady(on: queue) { + while r.isReadyForMoreMediaData { + guard let sb = output.copyNextSampleBuffer() else { + r.stopRequestingMediaData() + state.finish() + return + } + let n = state.count + let pts = CMSampleBufferGetPresentationTimeStamp(sb) + if n == 0 { + CMTimebaseSetTime(tb, time: CMTimeSubtract(pts, CMTime(value: 1, timescale: 4))) + CMTimebaseSetRate(tb, rate: 1) + } + guard let enq = copier.transform(sb, mode: mode) else { state.fail(); continue } + if n < 2 || n == 150 { + let pb = CMSampleBufferGetImageBuffer(enq)! + let atts = CVBufferCopyAttachments(pb, .shouldPropagate) as? [String: Any] ?? [:] + let rpu = (atts["DolbyVisionRPUData"] as? Data)?.count ?? 0 + log("frame \(n) pts=\(String(format: "%.3f", pts.seconds)) rpu=\(rpu)B compat=\(atts["DolbyCompatibilityID"] ?? "nil") keys=\(atts.keys.sorted())") + } + r.enqueue(enq) + state.advance(lastPTS: pts) + } + } + let deadline = Date().addingTimeInterval(80) + while Date() < deadline { + try? await Task.sleep(nanoseconds: 500_000_000) + if state.done, CMTimebaseGetTime(tb) > state.lastPTS { break } + if dl.sampleBufferRenderer.status == .failed { + log("renderer failed: \(String(describing: dl.sampleBufferRenderer.error))") + break + } + } + log("vt \(mode): enqueued=\(state.count) copyFailures=\(state.failures) reader=\(reader.status.rawValue)") + renderer.stopRequestingMediaData() + reader.cancelReading() + renderer.flush(removingDisplayedImage: true, completionHandler: nil) + dl.removeFromSuperlayer() + } catch { + log("vt error: \(error)") + } + } + + @MainActor + private func requestCriteria(fd: CMFormatDescription, fps: Float, log: @Sendable (String) -> Void) async { + window?.avDisplayManager.preferredDisplayCriteria = AVDisplayCriteria(refreshRate: fps, formatDescription: fd) + log("criteria requested codec=\(fourCC(CMFormatDescriptionGetMediaSubType(fd))) fps=\(fps)") + let switchDeadline = Date().addingTimeInterval(10) + try? await Task.sleep(nanoseconds: 300_000_000) + while window?.avDisplayManager.isDisplayModeSwitchInProgress == true, Date() < switchDeadline { + try? await Task.sleep(nanoseconds: 100_000_000) + } + try? await Task.sleep(nanoseconds: 1_000_000_000) + log("mode switch settled") + } + + private func fourCC(_ v: FourCharCode) -> String { + String(bytes: [24, 16, 8, 0].map { UInt8((v >> $0) & 0xff) }, encoding: .ascii) ?? "\(v)" + } +} + + +/// Frame copy for the `vtcopy` / `vtstrip` arms, used on the feed queue. +nonisolated private final class PulseFrameCopier: @unchecked Sendable { + private var pool: CVPixelBufferPool? + private var poolKey: (Int, Int, OSType)? + + /// `vt`: unchanged. `vtcopy` / `vtstrip`: pixels copied into our own pool buffer, propagated + /// attachments copied across (minus the two Dolby Vision keys for `vtstrip`), new sample buffer + /// built from the image buffer the way AetherEngine's renderer builds its own. + func transform(_ sb: CMSampleBuffer, mode: String) -> CMSampleBuffer? { + if mode == "vt" { return sb } + guard let src = CMSampleBufferGetImageBuffer(sb), let dst = copyPixels(src) else { return nil } + var atts = CVBufferCopyAttachments(src, .shouldPropagate) as? [String: Any] ?? [:] + CVBufferRemoveAllAttachments(dst) + if mode == "vtstrip" { + atts["DolbyVisionRPUData"] = nil + atts["DolbyCompatibilityID"] = nil + } + CVBufferSetAttachments(dst, atts as CFDictionary, .shouldPropagate) + var fd: CMVideoFormatDescription? + guard CMVideoFormatDescriptionCreateForImageBuffer(allocator: nil, imageBuffer: dst, formatDescriptionOut: &fd) == noErr, + let fd else { return nil } + var timing = CMSampleTimingInfo() + CMSampleBufferGetSampleTimingInfo(sb, at: 0, timingInfoOut: &timing) + var out: CMSampleBuffer? + guard CMSampleBufferCreateReadyWithImageBuffer(allocator: nil, imageBuffer: dst, formatDescription: fd, + sampleTiming: &timing, sampleBufferOut: &out) == noErr else { return nil } + return out + } + + private func copyPixels(_ src: CVPixelBuffer) -> CVPixelBuffer? { + let w = CVPixelBufferGetWidth(src), h = CVPixelBufferGetHeight(src) + let fmt = CVPixelBufferGetPixelFormatType(src) + if pool == nil || poolKey.map({ $0 != (w, h, fmt) }) ?? true { + let attrs: [String: Any] = [ + kCVPixelBufferWidthKey as String: w, kCVPixelBufferHeightKey as String: h, + kCVPixelBufferPixelFormatTypeKey as String: fmt, + kCVPixelBufferIOSurfacePropertiesKey as String: [String: Any](), + kCVPixelBufferMetalCompatibilityKey as String: true] + CVPixelBufferPoolCreate(nil, [kCVPixelBufferPoolMinimumBufferCountKey as String: 12] as CFDictionary, + attrs as CFDictionary, &pool) + poolKey = (w, h, fmt) + } + guard let pool else { return nil } + var dst: CVPixelBuffer? + guard CVPixelBufferPoolCreatePixelBuffer(nil, pool, &dst) == kCVReturnSuccess, let dst else { return nil } + CVPixelBufferLockBaseAddress(src, .readOnly) + CVPixelBufferLockBaseAddress(dst, []) + defer { + CVPixelBufferUnlockBaseAddress(dst, []) + CVPixelBufferUnlockBaseAddress(src, .readOnly) + } + for p in 0..` decodes both code rows from raw 3840x2160 yuv420p +frames of any composer's output and prints `ok`, `MISMATCH` or `no-EL` per frame. + +The soak arm is not a pulse measurement. It plays a FEL title for several minutes through a seek +schedule and reports one line, `soak summary: played= seeks= droppedFrames= composeFailures= +peakFootprintMB= endFootprintMB= route=`. Check `route=` first: it reads `FEL` only if the engine's +dual-layer summary lines were seen, and a run that never took the route reports zero compose failures +for the wrong reason. + +## Run on an Apple TV + +The host app takes its arms from the environment variable `PULSE_ARMS`: + +``` +label|mode|url[|k=v;k=v],label|mode|url,... +``` + +Arms are comma separated; a soak's `seeks=at@to,at@to` list contains commas too, so a piece that does +not look like `label|mode|...` continues the previous arm's parameters. +`xcrun devicectl device process launch --environment-variables` does deliver environment to an app +process. That is not true of an XCTest bundle on tvOS, which is why the transport probe cannot use it. + +```bash +DEVICE="" TEAM= Scripts/run_tv.sh +``` + +`xcrun devicectl list devices` names the devices. Options, all environment variables: + +| variable | default | what it is | +| --- | --- | --- | +| `DEVICE` | required | the Apple TV | +| `TEAM` | none | development team id for signing | +| `PULSE_BUNDLE_PREFIX` | `de.superuser404.dolbyvisionpulse` | the app's id is `.host`; set it when a team cannot register the default | +| `SEQ` | `main` | `main`, `el`, `vt`, `count` or `soak` | +| `SOAK_URL`, `SOAK_MIN`, `SOAK_SEEKS` | none, 5, `60@1800,150@600,240@3000` | the soak arm: a URL or a file under `out/`, minutes, `at@to` seek schedule in seconds | +| `PULSE_OUT` | `out/` | where the clips are | +| `IFACE` | first of `en0`, `en1` with an address | the network interface whose address the Apple TV fetches the clips from | +| `DEVELOPER_DIR` | `xcode-select -p` | the Xcode to build with | + +`run_tv.sh --no-build` plays without rebuilding. The script builds `Device/DolbyVisionPulse.xcodeproj`, +installs it, serves `out/` from this Mac over plain http (`serve.py`; the host allows it), waits ten +seconds so the recording can be started, launches with `--console`, and afterwards pulls the app's own +log (`Library/Caches/pulse-log.txt`) off the device. A launch that ends non-zero, because the app +crashed or was killed, still gets its log pulled. Pull a crash report from the device yourself when +that happens. + +In Xcode: open `Device/DolbyVisionPulse.xcodeproj`, pick the Apple TV, set `PULSE_ARMS` in the scheme's +environment variables and run. The host keeps the screen saver off, because an Apple TV that sleeps +mid-sequence suspends the process. + +Apple TV settings: Format **4K SDR**, Match Content with **Dynamic Range** and **Frame Rate** on, and +leave the HDMI chroma and range settings alone between runs. + +## Film it + +**Television.** Dolby Vision on an LG OLED (the C1 was used): picture mode **Dolby Vision Cinema**, +not *Cinema Home*, which follows the room light. AI Picture Pro, AI Brightness, Energy Saving, OLED +Motion Pro, TruMotion and OLED Care's Logo Luminance Adjustment all off; the last two matter because +black frame insertion flickers on a phone and the logo dimming darkens static areas, which this +pattern is. LG keeps separate Dolby Vision and HDR10 settings and the Dolby Vision ones can only be +changed while Dolby Vision is playing, so set them during any Dolby Vision title. Dark room, ten minutes +of warm-up. Note whether the Dolby Vision badge shows on each arm; it says only that the TV is in +Dolby Vision mode, and the pulse is what says the RPU is used. + +**Phone.** 4K at 30 fps (1080p 30 works), **HDR Video off** so the recording is not itself tone +mapped, Auto Low Light FPS off, Enhanced Stabilization off, Lock Camera on. On a tripod, with the +whole screen and all four corner squares filling most of the frame and nothing else bright in view. +Start recording before the launch; `run_tv.sh` waits ten seconds. Hands off until the sequence ends, +then stop. + +**Exposure must be locked.** With auto exposure the camera compensates for the very brightness change +being measured. The stock Camera app's AE/AF lock (long press) **did not hold** here: exposure moved +on a locked frame. Use a manual camera app with a fixed ISO, a fixed shutter (1/30 or 1/60) and a +fixed white balance (5600 K), set so the brightest patch (top row, right end) is not blown out. +Blackmagic Camera is one. If the stock app is all there is, lock during the first arm's pre-roll and +treat the first run as a shakedown; the closing P arm repeats it. + +## Cut and analyse + +`cut.py` splits one recording into one piece per arm using the `[Pulse] wall=...` lines in the console +log (or the pulled app log), so the labels do not depend on counting runs in order. + +```bash +python3 Scripts/cut.py recording.MOV tv_main_.log --slates +``` + +The video's start comes from its QuickTime creation date, to a second. `--slates` writes one still per +slate; read the UTC clock off one and pass it back for sub-second alignment: + +```bash +python3 Scripts/cut.py recording.MOV tv_main_.log --sync 12.5=17:08:31.2 +``` + +That is "the slate frame at 12.5 s of the video shows 17:08:31.2". The result is a table of arm, mode +and verdict, plus a `.mov` and `.json` per arm (and a `.png` when Pillow is installed) next to the recording. + +To analyse a recording directly, labelling runs in order: + +```bash +python3 Scripts/analyze.py recording.MOV --arms P,V1,V2,T,N,S,P1,T1,P --png result.png +``` + +## Report + +`report.py` reads the arms directory `cut.py` wrote (`summary.json` and every `NN_.json`) and prints +markdown: the change in camera brightness, B against A, per grey patch as a signed percent (FEL-route +rows in bold, a repeated arm labelled "(repeat)", an arm with no decoded run flagged), the number of +patches with |t| above 5 and the strongest t per arm, and the absolute phase-A levels of the arms that +show whether the panel was in Dolby Vision at all. `--png` draws every arm's trace in one figure. + +```bash +python3 Scripts/report.py recording_arms --md report.md --png report.png +python3 Scripts/report.py recording_arms --levels-arms V1,V2,T,S --patches g10,g100,g400,g1000,g4000,g10000 +``` + +## Check the clips before filming + +A steady EL arm only means something if the clip carries the residual. `check_el.py` demuxes the BL and EL +of the muxed `.mp4` (`dovi_tool demux`), or takes an EL-only `.hevc`, decodes the EL to raw +`yuv420p10le` (no range conversion), and checks that the median EL code inside every `EL_PATCHES` box is +512 in phase A and the `el_design()` code in phase B (within one code value). It prints the share of EL +pixels away from 512 per phase (0 in A, about the patch area in B), the composed 10-bit change each patch +should show, and for an `.mp4` that the RPU is the same on every frame. It exits 1 on a failure, so the +flat-EL control clip fails phase B by design. + +```bash +python3 Scripts/check_el.py out/p7fel_el.mp4 +``` + +`dump_attachments.swift` decodes a Profile 8.1 clip through AVAssetReader, that is VideoToolbox's own +decoder, prints every attachment on the chosen frames' pixel buffers, and hashes `DolbyVisionRPUData` on +every frame. It shows the payload the platform attaches itself, to compare with what the engine attaches; +on a pulse clip the hash differs between an A frame and a B frame. + +```bash +swift Scripts/dump_attachments.swift out/pulse_l2.mp4 0 200 +``` + +## Reading the verdicts + +For every patch and every B phase, the effect is `B - (A_before + A_after) / 2`, so slow drift cancels. +Per arm the output is a verdict, the eight patches that moved most (level, B-A change, relative change, +t statistic) and the lag in clip frames between the switch and the panel's response. + +- **PULSE**: at least three patches with |t| above 5. **weak**: one or two. **steady**: none. +- **Real Dolby Vision**: T is about the same as P, and much larger than N. +- **Fake Dolby Vision** (the mode switches, the metadata is ignored): T is about N while P pulses. +- **V1 pulses, V2 does not**: tvOS cares where the buffer came from. +- **V2 pulses, T does not**: the problem is in our RPU bytes or our path, not the platform. +- **TE pulses on the six patches, NE steady**: the residual reaches the panel. The six are the 10, 100, + 400 and 1000 nit greys, red and cyan (about +17 % in 10-bit luma code value after composition); the + other patches are in-frame controls and must not move. + +N and NE are the drift floor. A verdict only means something against them. + +## Proving the analysis first + +`sim_recording.py` writes a fake phone recording of `pulse_l2.mp4`: filmed at 30 fps with perspective, +vignetting and noise, B phases darkened with a three frame latency for a `pulse` run, nothing for a +`steady` one. + +```bash +python3 Scripts/sim_recording.py sim.mp4 pulse,steady # about two minutes +python3 Scripts/analyze.py sim.mp4 --arms pulse,steady # PULSE (28 patches), steady (0 patches) +``` + +`cut.py` can be exercised the same way with a hand-written log of `[Pulse] wall= t= slate +