feat: decode VP9 with VideoToolbox on macOS (#318)

* feat: decode VP9 with VideoToolbox on macOS

HardwareVideoDecoderFactory now decodes VP9 profile 0 in hardware on
macOS, through the VP9 decoder VideoToolbox offers once it is registered.
H.264 stays with the default decoders, which use VideoToolbox already.

The decoder creates its session on the first key frame, from the header
parsed with WebRTC's VP9 header parser: the format description carries the
vpcC box and the colour properties, and the output pixel format follows the
range of the stream. The whole encoded image goes in as one sample, so a
hidden frame is not decoded on its own. A key frame of another size or
range replaces the session. Frames come out as the CVPixelBuffer
VideoToolbox made, without a copy.

Anything it cannot decode goes to libvpx through FallbackVideoDecoder: other
profiles, sizes outside 64x64 to 4096x4096, frames with spatial layers, a
session that cannot be created or is not in hardware, and a decoder that
keeps failing.

The macOS shortcuts in DefaultVideoCodecFactories are gone; macOS now
defines the platform hooks like Windows and Linux do.

Tests: VP9 through VideoToolbox, VP9 in software by default, and a
resolution change in the middle of a call. A new check makes sure decoded
frames carry a picture.

* fix: keep the VP9 decoder from failing inter frames the parser has no size for

WebRTC's VP9 header parser gives a header only for a frame that states its
size: key frames, and inter frames that do not take the size from a
reference. Most inter frames take it, and the parser returns nothing for
them. The decoder treated that as a failure, so in a stream without temporal
layers every inter frame failed, the receiver asked for a key frame, and
only that key frame decoded: a few frames a second, and a key frame
request for nearly every frame.

A frame without a header is decoded now. A key frame without one is handed
to the software decoder, since there is nothing to create a session from.

The tests did not notice because they waited for ten frames and checked the
picture and the size. New ones read the statistics of the receiver, that a
stream asks for no more than one key frame, and cover temporal layers and
spatial layers, which a real libwebrtc stream now makes: L2T2 at 640x480
ends up in libvpx. The test call can send another size than 320x240 for
that, since WebRTC encodes no spatial layers below a size.

Author: devopvoid

docs/guide/advanced/video-codecs.md

@@ -53,11 +53,13 @@ PeerConnectionFactory factory = PeerConnectionFactory.builder()
 |---|---|
 | Windows | H.264 and AV1, on GPUs that decode them, through the Media Foundation decoders of Windows on Direct3D 11 (DXVA) |
 | Linux | Not yet; decoding is in software |
-| macOS | VideoToolbox, as with `DefaultVideoDecoderFactory` |
+| macOS | H.264 through VideoToolbox, as with `DefaultVideoDecoderFactory`, and VP9 (profile 0) through VideoToolbox's VP9 decoder, on Macs that have one |
 
 AV1 on Windows needs the *AV1 Video Extension*, which Windows 11 includes. Decoded frames are copied from GPU memory back to system memory, where WebRTC's frames are, so hardware decoding pays off mostly at high resolutions and with many streams; at low resolutions WebRTC's software decoders are about as cheap. A hardware decoder that fails, or turns out to decode in software, is replaced by the software decoder, which starts with the next key frame.
 
-Which decoder a stream uses shows in the `decoderImplementation` statistic of its `inbound-rtp` stats, e.g. `MediaFoundation (Microsoft H264 Video Decoder MFT)` or `MediaFoundation (AV1VideoExtension)`.
+On macOS, VP9 decoding in hardware saves CPU: in a measurement on an Apple M2 it took a fraction of the processor time libvpx needs, but multi-threaded libvpx was about as fast on the clock. VideoToolbox decodes VP9 in hardware only where the Mac has a decoder for it, which Apple silicon does; a Mac without one decodes VP9 with libvpx as before. Profile 2 (10 bit), sizes below 64x64 or above 4096x4096, and streams with spatial layers (SVC) are decoded with libvpx too.
+
+Which decoder a stream uses shows in the `decoderImplementation` statistic of its `inbound-rtp` stats, e.g. `MediaFoundation (Microsoft H264 Video Decoder MFT)`, `MediaFoundation (AV1VideoExtension)` or `VideoToolbox (VP9)`.
 
 ### Native Codecs
 

webrtc-jni/src/main/cpp/include/media/video/codec/macos/VTVideoDecoderFactory.h

@@ -0,0 +1,53 @@
+/*
+ * Copyright 2026 Alex Andres
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef JNI_WEBRTC_MEDIA_VIDEO_CODEC_VT_VIDEO_DECODER_FACTORY_H_
+#define JNI_WEBRTC_MEDIA_VIDEO_CODEC_VT_VIDEO_DECODER_FACTORY_H_
+
+#include "api/environment/environment.h"
+#include "api/video_codecs/sdp_video_format.h"
+#include "api/video_codecs/video_decoder.h"
+#include "api/video_codecs/video_decoder_factory.h"
+
+#include <memory>
+#include <vector>
+
+namespace jni
+{
+	// Creates the VideoToolbox decoders that WebRTC's own decoders for macOS
+	// do not offer: VP9, in profile 0, the format the software factory
+	// offers first. H.264 is decoded through VideoToolbox by the default
+	// decoders already.
+	class VTVideoDecoderFactory : public webrtc::VideoDecoderFactory
+	{
+		public:
+			// Returns a factory, or null if this Mac has no hardware decoder
+			// for VP9. The decoder VideoToolbox has for VP9 has to be
+			// registered first; this does it, once for the process.
+			static std::unique_ptr<VTVideoDecoderFactory> Create();
+
+			~VTVideoDecoderFactory() override = default;
+
+			std::vector<webrtc::SdpVideoFormat> GetSupportedFormats() const override;
+			std::unique_ptr<webrtc::VideoDecoder> Create(const webrtc::Environment & env,
+				const webrtc::SdpVideoFormat & format) override;
+
+		private:
+			VTVideoDecoderFactory() = default;
+	};
+}
+
+#endif

webrtc-jni/src/main/cpp/include/media/video/codec/macos/VTVp9Decoder.h

@@ -0,0 +1,117 @@
+/*
+ * Copyright 2026 Alex Andres
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef JNI_WEBRTC_MEDIA_VIDEO_CODEC_VT_VP9_DECODER_H_
+#define JNI_WEBRTC_MEDIA_VIDEO_CODEC_VT_VP9_DECODER_H_
+
+#include "api/video/encoded_image.h"
+#include "api/video_codecs/video_decoder.h"
+#include "modules/video_coding/utility/vp9_uncompressed_header_parser.h"
+
+#include <CoreMedia/CoreMedia.h>
+#include <CoreVideo/CoreVideo.h>
+#include <VideoToolbox/VideoToolbox.h>
+
+#include <cstdint>
+
+namespace jni
+{
+	// Decodes VP9 profile 0 on the media engine or GPU of a Mac, through the
+	// VP9 decoder VideoToolbox offers once it is registered.
+	//
+	// The decompression session needs the properties of the stream, so it is
+	// created on the first key frame, from the header the key frame carries,
+	// and again whenever a key frame changes the size or the range. Decoding
+	// is synchronous: each frame comes out of VideoToolbox before Decode
+	// returns. A frame is handed on as the CVPixelBuffer VideoToolbox made,
+	// without a copy.
+	//
+	// Whatever VideoToolbox cannot decode in place goes to the software
+	// decoder, by returning WEBRTC_VIDEO_CODEC_FALLBACK_SOFTWARE: a profile
+	// other than 0, a size outside what the hardware decodes, frames with
+	// spatial layers, and a decoder that keeps failing.
+	class VTVp9Decoder : public webrtc::VideoDecoder
+	{
+		public:
+			VTVp9Decoder();
+			~VTVp9Decoder() override;
+
+			using webrtc::VideoDecoder::Decode;
+
+			bool Configure(const Settings & settings) override;
+			int32_t Decode(const webrtc::EncodedImage & image, int64_t renderTimeMs) override;
+			int32_t RegisterDecodeCompleteCallback(webrtc::DecodedImageCallback * callback) override;
+			int32_t Release() override;
+			DecoderInfo GetDecoderInfo() const override;
+			const char * ImplementationName() const override;
+
+		private:
+			// What a session is created for. A key frame that differs from it
+			// needs a new format description, and often a new session.
+			struct StreamConfig
+			{
+				int width = 0;
+				int height = 0;
+				bool fullRange = false;
+				webrtc::Vp9ColorSpace colorSpace = webrtc::Vp9ColorSpace::CS_UNKNOWN;
+
+				bool operator==(const StreamConfig & other) const;
+			};
+
+			// The result of decoding one frame, filled in by the callback of
+			// the session.
+			struct Output
+			{
+				OSStatus status = noErr;
+				CVImageBufferRef image = nullptr;
+			};
+
+			static void OnOutput(void * decoder, void * frame, OSStatus status, VTDecodeInfoFlags flags,
+				CVImageBufferRef image, CMTime presentationTime, CMTime duration);
+
+			// Reads the stream properties from a key frame header. Returns
+			// false for a stream the hardware decoder does not take.
+			static bool ReadConfig(const webrtc::Vp9UncompressedHeader & header, StreamConfig & config);
+
+			// Makes sure there is a session for the stream, creating or
+			// replacing it where the stream changed.
+			bool EnsureSession(const StreamConfig & config);
+			bool CreateSession(const StreamConfig & config, CMVideoFormatDescriptionRef format);
+			void DestroySession();
+
+			// Decodes one encoded frame as a single sample.
+			OSStatus DecodeSample(const webrtc::EncodedImage & image, Output & output);
+
+			// Counts a failure and tells WebRTC what to do about it.
+			int32_t Fail(OSStatus status);
+
+			void Deliver(const webrtc::EncodedImage & image, int64_t renderTimeMs, CVImageBufferRef pixels);
+
+		private:
+			webrtc::DecodedImageCallback * callback;
+
+			VTDecompressionSessionRef session;
+			CMVideoFormatDescriptionRef format;
+			StreamConfig active;
+
+			// Set after a failure, until the next key frame.
+			bool requireKeyFrame;
+			int consecutiveErrors;
+			int64_t sampleCount;
+	};
+}
+
+#endif

webrtc-jni/src/main/cpp/src/media/video/codec/DefaultVideoCodecFactories.cpp

@@ -54,17 +54,13 @@ namespace jni
 
 	std::unique_ptr<webrtc::VideoEncoderFactory> CreateHardwareVideoEncoderFactory()
 	{
-#ifdef __APPLE__
-		return CreateDefaultVideoEncoderFactory();
-#else
 		std::vector<std::unique_ptr<webrtc::VideoEncoderFactory>> hardware = CreatePlatformHardwareVideoEncoderFactories();
 
 		if (hardware.empty()) {
 			return CreateDefaultVideoEncoderFactory();
 		}
 
 		return std::make_unique<HardwareVideoEncoderFactory>(std::move(hardware), CreateDefaultVideoEncoderFactory());
-#endif
 	}
 
 	std::unique_ptr<webrtc::VideoDecoderFactory> CreateDefaultVideoDecoderFactory()
@@ -82,16 +78,12 @@ namespace jni
 
 	std::unique_ptr<webrtc::VideoDecoderFactory> CreateHardwareVideoDecoderFactory()
 	{
-#ifdef __APPLE__
-		return CreateDefaultVideoDecoderFactory();
-#else
 		std::vector<std::unique_ptr<webrtc::VideoDecoderFactory>> hardware = CreatePlatformHardwareVideoDecoderFactories();
 
 		if (hardware.empty()) {
 			return CreateDefaultVideoDecoderFactory();
 		}
 
 		return std::make_unique<HardwareVideoDecoderFactory>(std::move(hardware), CreateDefaultVideoDecoderFactory());
-#endif
 	}
 }

webrtc-jni/src/main/cpp/src/media/video/codec/macos/MacHardwareVideoCodecFactories.cpp

@@ -0,0 +1,42 @@
+/*
+ * Copyright 2026 Alex Andres
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "media/video/codec/HardwareVideoDecoderFactory.h"
+#include "media/video/codec/HardwareVideoEncoderFactory.h"
+#include "media/video/codec/macos/VTVideoDecoderFactory.h"
+
+namespace jni
+{
+	std::vector<std::unique_ptr<webrtc::VideoEncoderFactory>> CreatePlatformHardwareVideoEncoderFactories()
+	{
+		// H.264 is encoded through VideoToolbox by the default encoders, and
+		// VideoToolbox has no encoder for the other codecs.
+		return {};
+	}
+
+	std::vector<std::unique_ptr<webrtc::VideoDecoderFactory>> CreatePlatformHardwareVideoDecoderFactories()
+	{
+		std::vector<std::unique_ptr<webrtc::VideoDecoderFactory>> factories;
+
+		// H.264 is decoded through VideoToolbox by the default decoders;
+		// VP9 is the codec they decode in software.
+		if (auto videoToolbox = VTVideoDecoderFactory::Create()) {
+			factories.push_back(std::move(videoToolbox));
+		}
+
+		return factories;
+	}
+}

webrtc-jni/src/main/cpp/src/media/video/codec/macos/VTVideoDecoderFactory.cpp

@@ -0,0 +1,66 @@
+/*
+ * Copyright 2026 Alex Andres
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "media/video/codec/macos/VTVideoDecoderFactory.h"
+#include "media/video/codec/macos/VTVp9Decoder.h"
+
+#include "rtc_base/logging.h"
+
+#include <VideoToolbox/VideoToolbox.h>
+
+namespace jni
+{
+	namespace
+	{
+		// Whether this Mac decodes VP9 in hardware. VideoToolbox has the VP9
+		// decoder only after it is registered, so that comes first. The
+		// answer does not change while the process runs, and is asked once.
+		bool HasHardwareVp9Decoder()
+		{
+			static const bool available = [] {
+				VTRegisterSupplementalVideoDecoderIfAvailable(kCMVideoCodecType_VP9);
+
+				const bool supported = VTIsHardwareDecodeSupported(kCMVideoCodecType_VP9);
+
+				RTC_LOG(LS_INFO) << "VideoToolbox hardware decoder for VP9: " << supported;
+
+				return supported;
+			}();
+
+			return available;
+		}
+	}
+
+	std::unique_ptr<VTVideoDecoderFactory> VTVideoDecoderFactory::Create()
+	{
+		if (!HasHardwareVp9Decoder()) {
+			return nullptr;
+		}
+
+		return std::unique_ptr<VTVideoDecoderFactory>(new VTVideoDecoderFactory());
+	}
+
+	std::vector<webrtc::SdpVideoFormat> VTVideoDecoderFactory::GetSupportedFormats() const
+	{
+		return { webrtc::SdpVideoFormat::VP9Profile0() };
+	}
+
+	std::unique_ptr<webrtc::VideoDecoder> VTVideoDecoderFactory::Create(const webrtc::Environment & env,
+		const webrtc::SdpVideoFormat & format)
+	{
+		return std::make_unique<VTVp9Decoder>();
+	}
+}