Repository navigation
feat: list the hardware codecs of the hardware video codec factories - #359
Merged
Merged
Conversation
The example builds its PeerConnectionFactory with the hardware video codec factories, so the RTP capabilities include the codecs only the GPU handles, and compares the codecs of the default and the hardware factories, marking those only available in hardware.
HardwareVideoEncoderFactory and HardwareVideoDecoderFactory get getHardwareCodecs(): the supported codecs, in their order, that a hardware encoder or decoder of the machine takes first, and on macOS H.264, which the default factories handle with VideoToolbox. Both lists come from one native factory, so the hardware is probed once. Natively, the composite hardware factories implement WebRTC's QueryCodecSupport and report a format as power efficient when a hardware sub-factory supports it. CodecListExample marks each video codec as hardware or software with it.
NVDEC and the Media Foundation decoders offered every H.264 format WebRTC decodes, High 4:4:4 Predictive among them, though GPU decoders take 4:2:0 only. A 4:4:4 stream started on the GPU, failed, and lost frames until the next key frame reached the software decoder, and getHardwareCodecs() listed the profile as hardware. Both now offer the 4:2:0 profiles only, so 4:4:4 streams go to the software decoder from the start.
Every hardware codec factory probed the GPU anew, about 40 ms for the decoders on Windows, and a PeerConnectionFactory built with the hardware factories probed twice: once for their Java objects, and once more in the wrappers that took them through Java, where the two probes could also disagree. The platform's hardware factories, which hold nothing but what probing found, are now made once per process and setting of the hardware codec options, and shared. PeerConnectionFactory makes the hardware factories natively, as it does the default ones, without a detour through Java for every codec it creates. A Java factory of one's own makes the built-in factories only when it first hands out a native codec, so one that never does probes no hardware.
SendableMetatype
added a commit
to EduGeyser/webrtc-java
that referenced
this pull request
Oct 15, 2026
Brings in the work upstream added from October 11 to 15: hardware video codecs on every platform, through Media Foundation, VideoToolbox, VA-API and NVDEC, with WebRTC built with H.265 (devopvoid#346 to devopvoid#348, devopvoid#352, devopvoid#353, devopvoid#356, devopvoid#359, devopvoid#371 to devopvoid#377); HEVC and AV1 decoding in the media player, with dav1d and Vulkan Video (devopvoid#331, devopvoid#345); encoded frame injectors (devopvoid#365); receiver source changes and audio packet infos (devopvoid#366); audio codec factories (devopvoid#340); network thread observers called on a callback thread (devopvoid#364); screen capture with ScreenCaptureKit (devopvoid#354); Windows audio devices that mix more than two channels (devopvoid#358); autoninja (devopvoid#357); and the release channels of the docs site (devopvoid#361). Fork-side adaptations: - The factory keeps the CreatePeerConnectionFactory overload without a worker thread, and still switches device capture on the network thread. devopvoid#355 copies WebRTC's factory setup and has the audio module learn a worker thread, which only matters for a worker thread distinct from the network thread, and WebRTC is removing those. The audio codec factories and hardware codec switches of devopvoid#340 and devopvoid#359 go to that overload. - WebRTC stays on a commit on main, so the move to branch 8059 (devopvoid#360) and its fixed clang package are left out. - The data channels variant builds WebRTC without H.265 and without the extra targets, leaves out the audio codec factory bindings and addTransceiverOfType, and skips the new tests that need audio or video. - dav1d is built against the sysroot on native Linux builds as well, as FFmpeg is. CI installs Meson for the full variant only, checks out the dav1d submodule, and adds its commit to the FFmpeg cache key. The fork's Windows SDK step stays, without upstream's second one. - The release workflow records the docs' version in the tagged commit, as upstream's now does. The docs site still builds from main, since this fork publishes none. - The Maven files stay deleted: Jetty is bumped in the version catalog, and the docs lockfile takes upstream's bumps on top of the fork's newer VitePress. - The media module's README builds with Gradle.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Lets applications tell which video codecs the GPU handles, and makes the hardware codec factories cheaper and more consistent to use.
getHardwareCodecs()onHardwareVideoEncoderFactoryandHardwareVideoDecoderFactory: the supported codecs, in their order, that a hardware encoder or decoder of the machine takes first (on macOS, H.264 through VideoToolbox included). Natively, the composite hardware factories implement WebRTC'sQueryCodecSupportand report a format as power efficient when a hardware sub-factory supports it.CodecListExamplebuilds itsPeerConnectionFactorywith the hardware factories and marks every video codec asHARDWAREorSOFTWARE.webrtc.hardware.vp9Encoder,webrtc.hardware.vaapiExperimental) and shared. APeerConnectionFactorybuilt with the hardware factories makes them natively, as it does the default ones, instead of going through Java for every codec; before, it probed twice, and the two probes could disagree. A Java factory of one's own makes the built-in factories only when it first hands out a native codec.On a Windows machine with an AMD GPU, after the first probe of the process, creating both hardware factories takes about 4 ms and the
PeerConnectionFactoryabout 1.3 ms; before, each setup probed twice, about 40 ms per decoder probe.One behaviour change: a GPU, driver or codec extension installed or removed while the process runs is seen by a new process only. The Javadoc and the video codecs guide say so.
Testing
mvn -pl webrtc-jni,webrtc packageon Windows x64 (AMD GPU): 289 tests, 0 failures, 3 skipped (two macOS-only, the opt-in VP9 encoder).HardwareVideoEncoderFactorystill encodes H.264 on the GPU; the H.264 and AV1 hardware tests check that the encoded codec is listed as hardware.CodecListExample: H.264 Constrained Baseline/Baseline (packetization mode 1), AV1 and H.265 encode in hardware; VP9 profile 0, H.264 4:2:0, AV1 profile 0 and H.265 decode in hardware.