feat: add custom video codec factories

README.md

@@ -34,6 +34,7 @@ For more detailed information, check out the documentation:
 - [Media Files](https://jrtc.dev/guide/media/media-files) - Sending video and audio files with the media module
 - [Media Recording](https://jrtc.dev/guide/media/media-recording) - Recording calls into media files with the media module
 - [Encoded Transforms](https://jrtc.dev/guide/advanced/encoded-transforms) - Reading and changing encoded frames, e.g. for end-to-end encryption
+- [Video Codecs](https://jrtc.dev/guide/advanced/video-codecs) - Adding video codecs implemented in Java, or limiting the built-in ones
 - [Build Notes](https://jrtc.dev/guide/build) - Instructions for building the library from source
 
 ## License

docs/.vitepress/sidebar.ts

@@ -74,6 +74,7 @@ function sidebarGuide(): DefaultTheme.SidebarItem[] {
 			items: [
 				{ text: 'Field Trials', link: '/advanced/field-trials' },
 				{ text: 'Encoded Transforms', link: '/advanced/encoded-transforms' },
+				{ text: 'Video Codecs', link: '/advanced/video-codecs' },
 			],
 		},
 	]

docs/guide/advanced/field-trials.md

@@ -24,6 +24,15 @@ PeerConnectionFactory factory = new PeerConnectionFactory(fieldTrials, audioProc
 PeerConnectionFactory factory = new PeerConnectionFactory(fieldTrials, audioModule, audioProcessing);
 ```
 
+`PeerConnectionFactory.builder()` sets them too, next to anything else a factory is created with, such as its [video codecs](/guide/advanced/video-codecs):
+
+```java
+PeerConnectionFactory factory = PeerConnectionFactory.builder()
+        .setFieldTrials(fieldTrials)
+        .setAudioDeviceModule(audioModule)
+        .build();
+```
+
 ## Constraints
 
 Each key and value must be non-null and non-empty. Passing a map that violates this constraint throws an exception when the `PeerConnectionFactory` is created.

docs/guide/advanced/video-codecs.md

@@ -0,0 +1,168 @@
+# Video Codecs
+
+A `PeerConnectionFactory` decides which video codecs it can send and receive through a video encoder factory and a video decoder factory. Unless told otherwise it uses the codecs built into the library. This guide explains how to replace or extend them: to add a codec WebRTC does not have, to use a hardware encoder reached through another library, or to limit the codecs a factory offers.
+
+The API lives in the `dev.onvoid.webrtc.media.video.codec` package.
+
+## Built-in Codecs
+
+`DefaultVideoEncoderFactory` and `DefaultVideoDecoderFactory` provide the codecs built into the library. On Windows and Linux these are VP8, VP9, AV1 and H.264 in software; on macOS they are those of WebRTC's default factories, with H.264 through VideoToolbox. Ask a factory for the exact list:
+
+```java
+for (VideoCodecInfo codec : new DefaultVideoEncoderFactory().getSupportedCodecs()) {
+    System.out.println(codec.getName() + " " + codec.getParameters());
+}
+```
+
+The encoders and decoders these factories create are `NativeVideoEncoder`s and `NativeVideoDecoder`s. They are placeholders that make WebRTC create the built-in codec, which then runs entirely inside WebRTC, so their methods are not to be called from Java.
+
+## Setting the Factories
+
+The codec factories are set with `PeerConnectionFactory.builder()`, which also takes everything the constructors do:
+
+```java
+PeerConnectionFactory factory = PeerConnectionFactory.builder()
+        .setAudioDeviceModule(audioModule)
+        .setVideoEncoderFactory(new MyEncoderFactory())
+        .setVideoDecoderFactory(new MyDecoderFactory())
+        .build();
+```
+
+A factory that is not set is the built-in one. The supported codecs are asked for once, when the `PeerConnectionFactory` is built; an exception thrown there fails `build()`. Encoders and decoders are created later, from WebRTC threads, one per stream.
+
+Both ends have to agree on a codec, so a codec added on the sending side also has to be offered by the decoder factory of the receiving side.
+
+## Adding a Codec
+
+A factory of one's own usually hands out its own codec next to the built-in ones, by delegating to a default factory:
+
+```java
+class MyEncoderFactory implements VideoEncoderFactory {
+
+    private final DefaultVideoEncoderFactory builtIn = new DefaultVideoEncoderFactory();
+
+    @Override
+    public List<VideoCodecInfo> getSupportedCodecs() {
+        List<VideoCodecInfo> codecs = new ArrayList<>();
+        codecs.add(new VideoCodecInfo("X-MYCODEC"));
+        codecs.addAll(builtIn.getSupportedCodecs());
+        return codecs;
+    }
+
+    @Override
+    public VideoEncoder createEncoder(VideoCodecInfo info) {
+        if (info.getName().equalsIgnoreCase("X-MYCODEC")) {
+            return new MyEncoder();
+        }
+        return builtIn.createEncoder(info);
+    }
+}
+```
+
+The list is in order of preference. A codec name WebRTC does not know, such as `X-MYCODEC`, is negotiated like any other and sent with the generic RTP packetization. `RTCRtpTransceiver.setCodecPreferences()` picks among the negotiated codecs per transceiver.
+
+## Implementing an Encoder
+
+WebRTC calls `initEncode()`, then `encode()` for each frame and `setRates()` whenever the target bitrate changes, and finally `release()`. It may initialize a released encoder again, possibly with another frame size. These calls come from one WebRTC thread at a time.
+
+```java
+class MyEncoder implements VideoEncoder {
+
+    private Callback callback;
+
+    @Override
+    public VideoCodecStatus initEncode(Settings settings, Callback callback) {
+        this.callback = callback;
+        // Set up the encoder for settings.width x settings.height at
+        // settings.startBitrateKbps.
+        return VideoCodecStatus.OK;
+    }
+
+    @Override
+    public VideoCodecStatus encode(VideoFrame frame, EncodeInfo info) {
+        I420Buffer pixels = frame.buffer.toI420();
+        ByteBuffer payload = encodeSomehow(pixels, info.isKeyFrameRequested());
+
+        callback.onEncodedFrame(EncodedImage.builder()
+                .setBuffer(payload)
+                .setEncodedWidth(pixels.getWidth())
+                .setEncodedHeight(pixels.getHeight())
+                .setCaptureTimeNs(frame.timestampNs)
+                .setFrameType(info.isKeyFrameRequested()
+                        ? EncodedImage.FrameType.KEY
+                        : EncodedImage.FrameType.DELTA)
+                .build());
+
+        return VideoCodecStatus.OK;
+    }
+
+    @Override
+    public VideoCodecStatus setRates(RateControlParameters parameters) {
+        // Aim for parameters.bitrate.getSum() bps at parameters.framerateFps.
+        return VideoCodecStatus.OK;
+    }
+
+    @Override
+    public VideoCodecStatus release() {
+        return VideoCodecStatus.OK;
+    }
+}
+```
+
+Things to keep in mind:
+
+- The capture time of an `EncodedImage` must be the `timestampNs` of the frame it encodes. That is how WebRTC matches output to input; an image with any other capture time is dropped.
+- The frame passed to `encode()` is valid only until the method returns. An encoder that works asynchronously calls `frame.retain()` and later `frame.release()`, and hands its output to the callback from its own thread, in encoding order. The pixels must not be changed, since other consumers may share them.
+- `onEncodedFrame()` copies the payload, so the buffer may be reused once it returns. Frames handed over after `release()` are ignored.
+- `getScalingSettings()`, `getEncoderInfo()`, `getResolutionBitrateLimits()`, `getImplementationName()` and `isHardwareEncoder()` have defaults and may be overridden. The default scaling settings use WebRTC's quantizer thresholds for VP8, VP9 and H.264; a custom codec without a quantizer should return `ScalingSettings.OFF`.
+
+## Implementing a Decoder
+
+A decoder follows the same pattern with `initDecode()`, `decode()` and `release()`:
+
+```java
+class MyDecoder implements VideoDecoder {
+
+    private Callback callback;
+
+    @Override
+    public VideoCodecStatus initDecode(Settings settings, Callback callback) {
+        this.callback = callback;
+        return VideoCodecStatus.OK;
+    }
+
+    @Override
+    public VideoCodecStatus decode(EncodedImage image) {
+        NativeI420Buffer buffer = decodeSomehow(image.getBuffer());
+        VideoFrame frame = new VideoFrame(buffer, image.getCaptureTimeNs());
+
+        callback.onDecodedFrame(frame, null, null);
+        frame.release();
+
+        return VideoCodecStatus.OK;
+    }
+
+    @Override
+    public VideoCodecStatus release() {
+        return VideoCodecStatus.OK;
+    }
+}
+```
+
+- The image and its buffer are read-only and valid only until `decode()` returns; copy the data to keep it longer.
+- A decoded frame must carry the capture time of the image it was decoded from as its timestamp.
+- `onDecodedFrame()` takes a reference of its own, so the caller still releases the frame it created. Frame buffers other than `NativeI420Buffer` are copied, and have to provide direct byte buffers.
+
+## Errors
+
+Whatever an encoder, decoder or factory method throws is logged and treated as `VideoCodecStatus.ERROR`, and a factory that throws from `createEncoder()` or `createDecoder()` creates no codec for that stream. On an error other than `FALLBACK_SOFTWARE` and `UNINITIALIZED`, WebRTC releases the codec and initializes it again.
+
+## Related API
+
+- `PeerConnectionFactory.Builder` — sets the video encoder and decoder factories.
+- `VideoEncoderFactory`, `VideoDecoderFactory` — create the codecs of a factory.
+- `VideoEncoder`, `VideoDecoder`, `EncodedImage` — codecs implemented in Java.
+- `DefaultVideoEncoderFactory`, `DefaultVideoDecoderFactory` — the built-in codecs.
+- `RTCRtpTransceiver.setCodecPreferences()` — chooses among the negotiated codecs.
+
+For the full API, see the JavaDoc of the `dev.onvoid.webrtc.media.video.codec` package.

docs/guide/index.md

@@ -41,6 +41,7 @@ This section provides detailed guides for various features of the webrtc-java li
 
 - [Field Trials](/guide/advanced/field-trials) - Enabling experimental features and tuning WebRTC internals
 - [Encoded Transforms](/guide/advanced/encoded-transforms) - Reading and changing encoded frames, e.g. for end-to-end encryption
+- [Video Codecs](/guide/advanced/video-codecs) - Adding video codecs implemented in Java, or limiting the built-in ones
 
 ## Additional Resources
 

docs/index.md

@@ -32,6 +32,8 @@ features:
     details: Record what a peer connection sends or receives into MKV, WebM or MP4 files. Encoded frames go into the file as they are, without re-encoding, so recording costs next to no CPU and keeps the exact quality of the call.
   - title: End-to-End Encryption
     details: Encoded transforms let Java code read, change or drop every encoded frame between encoder and network, like insertable streams in the browser; the building block for end-to-end encryption, frame metadata and stream analysis.
+  - title: Custom Video Codecs
+    details: Plug video encoders and decoders written in Java into WebRTC, next to the built-in VP8, VP9, AV1 and H.264, to bring in a codec of your own or a hardware encoder from another library.
   - title: Screen Sharing
     details: Share application windows or the full desktop with minimal setup; integrate screen capture streams like any other media stream.
   - title: Data Channels

webrtc-jni/src/main/cpp/CMakeLists.txt

@@ -55,6 +55,7 @@ file(GLOB SOURCES_MEDIA "src/media/*.cpp")
 file(GLOB SOURCES_MEDIA_AUDIO "src/media/audio/*.cpp")
 file(GLOB SOURCES_MEDIA_AUDIO_OS "src/media/audio/${SOURCE_TARGET}/*.cpp")
 file(GLOB SOURCES_MEDIA_VIDEO "src/media/video/*.cpp")
+file(GLOB SOURCES_MEDIA_VIDEO_CODEC "src/media/video/codec/*.cpp")
 file(GLOB SOURCES_MEDIA_VIDEO_DESKTOP "src/media/video/desktop/*.cpp")
 file(GLOB SOURCES_MEDIA_VIDEO_DESKTOP_OS "src/media/video/desktop/${SOURCE_TARGET}/*.cpp")
 file(GLOB SOURCES_MEDIA_VIDEO_OS "src/media/video/${SOURCE_TARGET}/*.cpp")
@@ -69,6 +70,7 @@ list(APPEND SOURCES
     ${SOURCES_MEDIA_AUDIO}
     ${SOURCES_MEDIA_AUDIO_OS}
     ${SOURCES_MEDIA_VIDEO}
+    ${SOURCES_MEDIA_VIDEO_CODEC}
     ${SOURCES_MEDIA_VIDEO_DESKTOP}
     ${SOURCES_MEDIA_VIDEO_DESKTOP_OS}
     ${SOURCES_MEDIA_VIDEO_OS}

webrtc-jni/src/main/cpp/include/JNI_DefaultVideoDecoderFactory.h

@@ -0,0 +1,36 @@
+/*
+ * Copyright 2026 Alex Andres
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include <jni.h>
+/* Header for class dev_onvoid_webrtc_media_video_codec_DefaultVideoDecoderFactory */
+
+#ifndef _Included_dev_onvoid_webrtc_media_video_codec_DefaultVideoDecoderFactory
+#define _Included_dev_onvoid_webrtc_media_video_codec_DefaultVideoDecoderFactory
+#ifdef __cplusplus
+extern "C" {
+#endif
+	/*
+	 * Class:     dev_onvoid_webrtc_media_video_codec_DefaultVideoDecoderFactory
+	 * Method:    getSupportedCodecsInternal
+	 * Signature: ()[Ldev/onvoid/webrtc/media/video/codec/VideoCodecInfo;
+	 */
+	JNIEXPORT jobjectArray JNICALL Java_dev_onvoid_webrtc_media_video_codec_DefaultVideoDecoderFactory_getSupportedCodecsInternal
+	(JNIEnv *, jclass);
+
+#ifdef __cplusplus
+}
+#endif
+#endif

webrtc-jni/src/main/cpp/include/JNI_DefaultVideoEncoderFactory.h

@@ -0,0 +1,36 @@
+/*
+ * Copyright 2026 Alex Andres
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include <jni.h>
+/* Header for class dev_onvoid_webrtc_media_video_codec_DefaultVideoEncoderFactory */
+
+#ifndef _Included_dev_onvoid_webrtc_media_video_codec_DefaultVideoEncoderFactory
+#define _Included_dev_onvoid_webrtc_media_video_codec_DefaultVideoEncoderFactory
+#ifdef __cplusplus
+extern "C" {
+#endif
+	/*
+	 * Class:     dev_onvoid_webrtc_media_video_codec_DefaultVideoEncoderFactory
+	 * Method:    getSupportedCodecsInternal
+	 * Signature: ()[Ldev/onvoid/webrtc/media/video/codec/VideoCodecInfo;
+	 */
+	JNIEXPORT jobjectArray JNICALL Java_dev_onvoid_webrtc_media_video_codec_DefaultVideoEncoderFactory_getSupportedCodecsInternal
+	(JNIEnv *, jclass);
+
+#ifdef __cplusplus
+}
+#endif
+#endif

webrtc-jni/src/main/cpp/include/JNI_NativeVideoDecoderCallback.h

@@ -0,0 +1,36 @@
+/*
+ * Copyright 2026 Alex Andres
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include <jni.h>
+/* Header for class dev_onvoid_webrtc_media_video_codec_NativeVideoDecoderCallback */
+
+#ifndef _Included_dev_onvoid_webrtc_media_video_codec_NativeVideoDecoderCallback
+#define _Included_dev_onvoid_webrtc_media_video_codec_NativeVideoDecoderCallback
+#ifdef __cplusplus
+extern "C" {
+#endif
+	/*
+	 * Class:     dev_onvoid_webrtc_media_video_codec_NativeVideoDecoderCallback
+	 * Method:    onDecodedFrame
+	 * Signature: (JLdev/onvoid/webrtc/media/video/VideoFrame;II)V
+	 */
+	JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_media_video_codec_NativeVideoDecoderCallback_onDecodedFrame
+	(JNIEnv *, jclass, jlong, jobject, jint, jint);
+
+#ifdef __cplusplus
+}
+#endif
+#endif

webrtc-jni/src/main/cpp/include/JNI_NativeVideoEncoderCallback.h

@@ -0,0 +1,36 @@
+/*
+ * Copyright 2026 Alex Andres
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include <jni.h>
+/* Header for class dev_onvoid_webrtc_media_video_codec_NativeVideoEncoderCallback */
+
+#ifndef _Included_dev_onvoid_webrtc_media_video_codec_NativeVideoEncoderCallback
+#define _Included_dev_onvoid_webrtc_media_video_codec_NativeVideoEncoderCallback
+#ifdef __cplusplus
+extern "C" {
+#endif
+	/*
+	 * Class:     dev_onvoid_webrtc_media_video_codec_NativeVideoEncoderCallback
+	 * Method:    onEncodedFrame
+	 * Signature: (JLdev/onvoid/webrtc/media/video/codec/EncodedImage;)V
+	 */
+	JNIEXPORT void JNICALL Java_dev_onvoid_webrtc_media_video_codec_NativeVideoEncoderCallback_onEncodedFrame
+	(JNIEnv *, jclass, jlong, jobject);
+
+#ifdef __cplusplus
+}
+#endif
+#endif