feat: decode HEVC and AV1 in hardware in the media player (#331)

* feat: decode HEVC and AV1 in hardware in the media player

The player decoded only H.264 and VP9 on the GPU. FFmpeg is now built with the
hardware decoders of HEVC and AV1 as well: Direct3D 11 and 12 on Windows,
NVDEC on Linux, VideoToolbox on macOS.

FFmpeg's own AV1 decoder has no software path, so AV1 gets dav1d as its
software decoder. It is a new submodule (1.5.4), built with Meson before FFmpeg
into a static library that is linked into libavcodec. On Windows that is with
the static C runtime, which is the one FFmpeg is built with. Its version and
options are recorded in components.txt, so changing them rebuilds on its own.
Meson and Ninja are new build prerequisites, installed by the CI preparation
steps. Cross targets get a Meson cross file made from the same variables
FFmpeg is configured with; only windows-x86_64 has been built so far.

FFmpeg prefers libdav1d to its native AV1 decoder, and only the native one has
hardware configurations, so VideoDecoder picks a decoder that has one when
hardware decoding is asked for, and otherwise keeps the preferred one.

The licence of dav1d and a notice go into the platform jar.

Tests compare every frame decoded in hardware with the software decoder for
HEVC and AV1, at 320x240 and 1080p. AV1 has its own property,
webrtc.test.hardwareAv1Decoding, since older GPUs cannot decode it.

* fix: give Meson an SDK when it builds dav1d on macOS

Meson finds the compiler by its path inside the Xcode toolchain, which skips
the /usr/bin/cc shim that points SDKROOT at an SDK, so on the macOS ARM64
runner it reported that the compiler cannot compile programs ("library
'System' not found"). The SDK is named for the Meson run now. FFmpeg's own
configure gets it from the shim, so SDKROOT is unset again before it runs.

* fix: record G.711 into Matroska

Matroska has no codec ID of its own for G.711, so avformat_query_codec
said no and the recorder left PCMU and PCMA tracks out of .mkv files,
although the docs, the error message and a comment all promise that
Matroska holds every codec WebRTC sends. The Matroska muxer writes G.711
all the same, in its compatibility mode for Windows audio codecs
(A_MS/ACM), as "ffmpeg -c copy" does, so the recorder now lets it.

The recorder tests can now send the call's audio in a chosen codec, and
record PCMU and PCMA into Matroska, reading them back as such.

* feat: decode HEVC in hardware only, never in software

The player decoded HEVC in software unless asked for hardware, and fell
back to software wherever the hardware had no decoder, did not take a
stream, or failed. HEVC is now decoded on the hardware only: a player
sends it there whether hardwareDecoding is set or not, and where the
hardware cannot decode it, creating the player fails with "Decoder not
found", or playback stops with an error. No path opens a software
decoder for it.

FFmpeg's hevc decoder stays in the build, because the HEVC hwaccels run
inside it (configure selects it for every one of them); it parses the
headers and hands the slices to the hardware, but VideoDecoder never
lets it decode a picture: no software open, no software pixel format,
no fall back.

The tests compared hardware HEVC with software decoding, which no
longer exists. They now compare with luma checksums the software
decoder gave before this change, which every conforming decoder
matches. A new test fault hides the hardware devices, so that a machine
without a GPU decoder is tested on every machine, CI included: HEVC
does not open there, while H.264 still falls back to software.

* docs: add an HEVC/H.265 disclaimer to the NOTICE files

HEVC is decoded through the hardware decoders of the platform only, and
the project bundles no software HEVC encoder. The disclaimer says so,
and that the FFmpeg libraries of the webrtc-java-media platform jars
contain FFmpeg's HEVC decoder, which the hardware decoding runs through,
and leaves the license for HEVC playback and encoding to end-users and
distributors. It goes into the root NOTICE and into the FFmpeg notice
each webrtc-java-media platform jar carries.

* build: leave FFmpeg's HEVC decoder to the hwaccels that need it

--enable-decoder=hevc put FFmpeg's HEVC decoder into every build, but
each HEVC hwaccel selects that decoder itself, so where there is one
the decoder comes along without it, and the line only made a difference
where there is none: on 32-bit ARM Linux it shipped an HEVC decoder that
VideoDecoder never lets decode. Without the line, that build has no HEVC
decoder at all, and the others are as before (CONFIG_HEVC_DECODER is
still 1 on Windows, through the D3D11 and D3D12 hwaccels).

The HEVC disclaimer in the NOTICE files now says which platform jars
contain the decoder: those that decode HEVC in hardware, all but 32-bit
ARM Linux.

Author: devopvoid

.github/actions/prepare-linux/action.yml

@@ -31,8 +31,9 @@ runs:
     - name: Install required packages
       run: |
         sudo apt-get update -q
-        # nasm is what FFmpeg assembles its x86 code with.
-        sudo apt-get install -y -q binutils cmake git locales lsb-release nasm ninja-build pipewire pipewire-pulse pkg-config python3 python3-setuptools rsync unzip wget xz-utils
+        # nasm is what FFmpeg assembles its x86 code with, and meson builds
+        # dav1d, the software decoder of AV1 in the media module.
+        sudo apt-get install -y -q binutils cmake git locales lsb-release meson nasm ninja-build pipewire pipewire-pulse pkg-config python3 python3-setuptools rsync unzip wget xz-utils
       shell: bash
 
     # Chromium Clang to be used with the clang toolchain file. The JNI code

.github/actions/prepare-macos/action.yml

@@ -16,6 +16,9 @@ runs:
         # nasm is what FFmpeg assembles its x86 code with, which the Intel
         # build needs even though the runner itself is Apple Silicon.
         brew install nasm
+        # Meson and Ninja build dav1d, the software decoder of AV1 in the media
+        # module, and FFmpeg's configure finds it with pkgconf.
+        brew install meson ninja pkgconf
       shell: bash
 
     - name: Select Xcode version

.github/actions/prepare-windows/action.yml

@@ -8,6 +8,8 @@ runs:
     - name: Install required packages
       run: |
         choco install ninja
+        # Meson builds dav1d, the software decoder of AV1 in the media module.
+        python -m pip install meson
       shell: powershell
 
     - name: Install MSYS2 and the FFmpeg build tools

.gitmodules

@@ -2,3 +2,7 @@
 	path = webrtc-java-media/third-party/ffmpeg
 	url = https://github.com/FFmpeg/FFmpeg.git
 	shallow = true
+[submodule "webrtc-java-media/third-party/dav1d"]
+	path = webrtc-java-media/third-party/dav1d
+	url = https://code.videolan.org/videolan/dav1d.git
+	shallow = true

NOTICE

@@ -19,6 +19,20 @@ built from an unmodified FFmpeg release, without --enable-gpl and without
 jar carries the license text and a notice naming the FFmpeg version and where
 its source is, under META-INF/licenses/ffmpeg.
 
+HEVC/H.265
+
+Disclaimer regarding HEVC/H.265:
+This project decodes HEVC/H.265 video through hardware acceleration only, using
+the hardware decoders of the operating system or GPU (VideoToolbox, Direct3D
+11/12, NVDEC). It does not include, bundle, or distribute any software-based
+HEVC encoder (such as x265), and it never decodes HEVC in software. The FFmpeg
+libraries in the webrtc-java-media platform jars that decode HEVC in hardware
+(all but 32-bit ARM Linux) contain FFmpeg's HEVC decoder, because FFmpeg's
+hardware decoding runs through it: it reads the stream's headers and passes the
+coded data to the hardware decoder, which decodes the pictures. End-users and distributors are responsible for ensuring that the
+underlying operating system or hardware platform holds a valid license for HEVC
+playback and encoding.
+
 DOCUMENTATION
 
 WebRTC 1.0: Real-time Communication Between Browsers

docs/guide/media/media-files.md

@@ -15,21 +15,21 @@ Decoding happens entirely in native code. Frames never travel through Java: the
 
 ## Adding the Module
 
-The module is part of the normal build, and it builds FFmpeg from a submodule pinned to a release tag, so the submodule has to be present:
+The module is part of the normal build, and it builds FFmpeg and dav1d (the software decoder of AV1) from submodules pinned to release tags, so the submodules have to be present:
 
 ```shell
-git submodule update --init --depth 1 webrtc-java-media/third-party/ffmpeg
+git submodule update --init --depth 1 webrtc-java-media/third-party/ffmpeg webrtc-java-media/third-party/dav1d
 mvn install
 ```
 
-Building FFmpeg needs `make` and `nasm`. On Windows they come from MSYS2:
+Building FFmpeg needs `make` and `nasm`, and building dav1d needs [Meson](https://mesonbuild.com) (`pip install meson`) and Ninja. On Windows `make` and `nasm` come from MSYS2:
 
 ```shell
 winget install MSYS2.MSYS2
 C:\msys64\usr\bin\bash -lc "pacman -S --needed make nasm diffutils pkgconf"
 ```
 
-Maven still runs from an ordinary shell; the build enters MSYS2 and the Visual Studio environment on its own. The first build compiles FFmpeg, which takes a while; later builds reuse the install directory.
+Maven still runs from an ordinary shell; the build enters MSYS2 and the Visual Studio environment on its own. The first build compiles FFmpeg and dav1d, which takes a while; later builds reuse the install directory.
 
 Once installed, depend on it alongside `webrtc-java`. It takes two entries: one for the Java API, and one for the natives of the platform you are running on.
 
@@ -206,7 +206,7 @@ It also keeps the native side of the media sources it feeds alive until it is cl
 
 ## Decoding in Hardware
 
-Decoding H.264 and VP9 takes a good part of a processor at high resolutions, and for every stream of a camera wall. A player can decode them on the media engine or GPU of the machine instead. It does so when asked to, with a flag on `MediaFileSource` or `MediaPlayer`:
+Decoding H.264, H.265/HEVC, VP9 and AV1 takes a good part of a processor at high resolutions, and for every stream of a camera wall. A player can decode them on the media engine or GPU of the machine instead. It does so when asked to, with a flag on `MediaFileSource` or `MediaPlayer`:
 
 ```java
 // A file whose video is decoded in hardware, where the platform can.
@@ -220,11 +220,13 @@ MediaPlayer player = new MediaPlayer(new MediaReader(path), videoSource, null, t
 boolean hardware = source.getPlayer().isHardwareDecoding();
 ```
 
-The pictures are the same ones software decoding gives; in the tests, every frame of H.264 and VP9 media is compared with its software counterpart. Hardware decoding is off unless asked for.
+The pictures are the same ones software decoding gives; in the tests, every frame of media in each of these codecs is compared with its software counterpart (for HEVC, with checksums of the pictures FFmpeg's software decoder gave once). Hardware decoding is off unless asked for, except for H.265/HEVC.
+
+**HEVC is decoded in hardware only.** A player decodes it on the media engine or GPU whether or not the flag is set, and never in software: `isHardwareDecoding()` is always `true` for it. Where the machine has no hardware decoder for HEVC, creating the player fails with an `IOException`; where the hardware does not take a stream (a profile it cannot decode) or fails while playing, playback stops with an error. The fallback below does not apply to it.
 
 - **Platforms:** macOS through VideoToolbox, Windows through Direct3D 12 (Windows 10 version 2004 or later, with a driver that decodes video through it) and otherwise Direct3D 11, and Linux through NVDEC on NVIDIA GPUs (x86-64 and ARM64; the driver provides `libcuda.so.1` and `libnvcuvid.so.1`, which are loaded when a player asks for hardware). The flag is accepted everywhere; where the platform has no hardware decoder, or the FFmpeg build has none for it (32-bit ARM, VA-API on Intel and AMD GPUs), the video is decoded in software, and `isHardwareDecoding()` says so.
-- **Codecs:** H.264 and VP9. VP8, MPEG-4, MJPEG and H.265/HEVC are decoded in software.
-- **Fallback:** a stream the hardware does not take, such as a profile it cannot decode, is decoded in software without the player noticing more than `isHardwareDecoding()` turning `false`. If the hardware fails before its first picture, the packets it was sent are decoded again in software and none is lost; if it fails after one, decoding goes on in software from the next key frame. A damaged packet is not taken for a hardware failure after the first picture: as with software decoding, playback stops with an error, and the hardware keeps the stream.
+- **Codecs:** H.264, H.265/HEVC, VP9 and AV1, where the GPU decodes them: a GPU that is a few years old may lack AV1, and on a Mac only an M3 or later has it. VP8, MPEG-4 and MJPEG are decoded in software, and so is AV1 where there is no GPU decoder for it (it is decoded by dav1d, which is built into the FFmpeg libraries).
+- **Fallback** (every codec but HEVC): a stream the hardware does not take, such as a profile it cannot decode, is decoded in software without the player noticing more than `isHardwareDecoding()` turning `false`. If the hardware fails before its first picture, the packets it was sent are decoded again in software and none is lost; if it fails after one, decoding goes on in software from the next key frame. A damaged packet is not taken for a hardware failure after the first picture: as with software decoding, playback stops with an error, and the hardware keeps the stream.
 - **What it saves** is processor time, not the copy: a decoded picture is read back from the media engine into system memory, and converted to I420 as WebRTC wants it. On an Apple M2, a 1080p H.264 stream took 11.6 ms of processor time per frame in software and 1.4 ms in hardware, and a 4K stream 35.7 ms against 4.2 ms, about seven to eight times less. Per frame, software on all cores is faster on the clock, which does not matter at playback speed. Windows gains less: on an NVIDIA GeForce RTX 3080 through Direct3D 12, a 1080p H.264 stream took about 2.1 ms of processor time per frame against 3.6 ms in software, and a 4K stream 5.4 ms against 6.6 ms. Direct3D 11 reads the picture back more slowly and took more than software at 4K (12.4 ms), which is why Direct3D 12 comes first. These are figures for one machine each and for synthetic media, not a promise; try it on your own streams before relying on it.
 
 ## Playing a Live Stream
@@ -269,7 +271,7 @@ The FFmpeg build is deliberately small, and carries only what this module plays:
 | | |
 | --- | --- |
 | **Containers** | MP4 and MOV, Matroska and WebM, AVI, MPEG-TS, FLV, WAV, MP3, Ogg, FLAC, AAC |
-| **Video** | H.264, H.265/HEVC, VP8, VP9, MPEG-4 (including Xvid and DivX), Microsoft MPEG-4 v1 to v3, MJPEG |
+| **Video** | H.264, H.265/HEVC (hardware only), VP8, VP9, AV1, MPEG-4 (including Xvid and DivX), Microsoft MPEG-4 v1 to v3, MJPEG |
 | **Audio** | AAC, MP3, MP2, AC-3, Opus, Vorbis, FLAC, PCM, MS and IMA ADPCM |
 
 Audio of any rate or layout is resampled to what WebRTC takes, which is 48 kHz 16-bit PCM in mono or stereo. Video that decodes to I420 — almost all 8-bit H.264, VP8, VP9 and MPEG-4 — reaches the encoder without being copied; anything else is converted first.

docs/guide/media/media-recording.md

@@ -48,6 +48,8 @@ The container follows the file name:
 
 A name FFmpeg does not know gets Matroska. A track whose codec the file cannot hold, VP8 in MP4 for example, is left out and reported as a warning, and the rest is recorded.
 
+Matroska has no codec ID of its own for G.711, so it is stored as Windows audio (`A_MS/ACM`), the way FFmpeg itself stores it. Players built on FFmpeg, such as VLC and mpv, play it; some others do not.
+
 ## Following a Recording
 
 A `MediaRecorderListener` hears when the file begins, what was left out, and whether writing failed:

webrtc-java-media/README.md

@@ -62,7 +62,7 @@ This module is part of the normal build, and it builds FFmpeg from a submodule,
 has to be present:
 
 ```shell
-git submodule update --init --depth 1 webrtc-java-media/third-party/ffmpeg
+git submodule update --init --depth 1 webrtc-java-media/third-party/ffmpeg webrtc-java-media/third-party/dav1d
 mvn install
 ```
 
@@ -77,5 +77,9 @@ winget install MSYS2.MSYS2
 C:\msys64\usr\bin\bash -lc "pacman -S --needed make nasm diffutils pkgconf"
 ```
 
+dav1d, the software decoder of AV1, is built first and linked into FFmpeg. Building it needs Meson
+and Ninja (`pip install meson`, with `meson` on the `PATH`; `ninja` is there for the WebRTC build
+already).
+
 Maven still runs from an ordinary shell. The build enters MSYS2 and the Visual Studio environment
 on its own, because FFmpeg's configure needs a POSIX shell that can also see `cl` and `link`.

webrtc-java-media/src/main/cpp/CMakeLists.txt

@@ -171,6 +171,24 @@ add_custom_command(TARGET ${PROJECT_NAME} POST_BUILD
             "${FFMPEG_NOTICE_FILE}" "${FFMPEG_LICENSES_DIR}/NOTICE.txt"
     COMMENT "Collecting the FFmpeg license and notice for the platform jar")
 
+# dav1d, the software decoder of AV1, is linked into libavcodec, and its license
+# asks for its text to go along with the binaries.
+set(DAV1D_LICENSE_FILE "${CMAKE_CURRENT_SOURCE_DIR}/../../../third-party/dav1d/COPYING")
+
+if(NOT EXISTS "${DAV1D_LICENSE_FILE}")
+    message(FATAL_ERROR
+        "${DAV1D_LICENSE_FILE} is missing, and the platform jar must not ship dav1d "
+        "without it. Check out the dav1d submodule.")
+endif()
+
+set(DAV1D_LICENSES_DIR "${NATIVES_DIR}/META-INF/licenses/dav1d")
+
+add_custom_command(TARGET ${PROJECT_NAME} POST_BUILD
+    COMMAND ${CMAKE_COMMAND} -E make_directory "${DAV1D_LICENSES_DIR}"
+    COMMAND ${CMAKE_COMMAND} -E copy_if_different
+            "${DAV1D_LICENSE_FILE}" "${DAV1D_LICENSES_DIR}/COPYING"
+    COMMENT "Collecting the dav1d license for the platform jar")
+
 # The headers FFmpeg was built against for NVDEC have licenses of their own,
 # which go along as well.
 if(FFMPEG_NVDEC)