Repository navigation
fix: mark encoded frames as the end of their temporal unit - #356
Merged
Merged
Conversation
Since M152 WebRTC expects encoders to set EncodedImage's end-of-temporal-unit flag, and to report every frame they drop through EncodedImageCallback::OnFrameDropped, so that it can release what it keeps of a frame (the captured frame held for corruption detection, for one) as soon as nothing more of it will come out. All encoders here produce a single layer, so every frame ends its temporal unit. EncoderOutputProcessor sets the flag for the hardware encoders, and the Java encoder wrapper sets it for the frames of Java encoders. The wrapper also reports the frames it gives up on: frames a Java encoder produced no output for, the oldest frame when too many are pending, and frames that cannot be converted for the Java encoder. The Media Foundation and VideoToolbox encoders report their drops already; NVENC and VA-API drop none.
SendableMetatype
added a commit
to EduGeyser/webrtc-java
that referenced
this pull request
Oct 15, 2026
Brings in the work upstream added from October 11 to 15: hardware video codecs on every platform, through Media Foundation, VideoToolbox, VA-API and NVDEC, with WebRTC built with H.265 (devopvoid#346 to devopvoid#348, devopvoid#352, devopvoid#353, devopvoid#356, devopvoid#359, devopvoid#371 to devopvoid#377); HEVC and AV1 decoding in the media player, with dav1d and Vulkan Video (devopvoid#331, devopvoid#345); encoded frame injectors (devopvoid#365); receiver source changes and audio packet infos (devopvoid#366); audio codec factories (devopvoid#340); network thread observers called on a callback thread (devopvoid#364); screen capture with ScreenCaptureKit (devopvoid#354); Windows audio devices that mix more than two channels (devopvoid#358); autoninja (devopvoid#357); and the release channels of the docs site (devopvoid#361). Fork-side adaptations: - The factory keeps the CreatePeerConnectionFactory overload without a worker thread, and still switches device capture on the network thread. devopvoid#355 copies WebRTC's factory setup and has the audio module learn a worker thread, which only matters for a worker thread distinct from the network thread, and WebRTC is removing those. The audio codec factories and hardware codec switches of devopvoid#340 and devopvoid#359 go to that overload. - WebRTC stays on a commit on main, so the move to branch 8059 (devopvoid#360) and its fixed clang package are left out. - The data channels variant builds WebRTC without H.265 and without the extra targets, leaves out the audio codec factory bindings and addTransceiverOfType, and skips the new tests that need audio or video. - dav1d is built against the sysroot on native Linux builds as well, as FFmpeg is. CI installs Meson for the full variant only, checks out the dav1d submodule, and adds its commit to the FFmpeg cache key. The fork's Windows SDK step stays, without upstream's second one. - The release workflow records the docs' version in the tagged commit, as upstream's now does. The docs site still builds from main, since this fork publishes none. - The Maven files stay deleted: Jetty is bumped in the version catalog, and the docs lockfile takes upstream's bumps on top of the fork's newer VitePress. - The media module's README builds with Gradle.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follows up on the discuss-webrtc PSA An update to frame drop handling in VideoEncoder and EncodedImageCallback, which applies to M152 (
branch-heads/7977). WebRTC now expects encoders to:EncodedImage::is_end_of_temporal_uniton every output frame (truefor encoders without spatial or simulcast layers), andEncodedImageCallback::OnFrameDropped.The flag is an unset
std::optional<bool>by default. Without it, WebRTC holds what it keeps of a frame longer than needed, for example the captured frame held for corruption detection. The PSA says this may become a hard error later.Changes
EncoderOutputProcessor, which builds every frame they output, sets the flag. They all produce a single layer.VideoEncoderWrapper): the flag is set on each frame, and the frames the wrapper gives up on are now reported throughOnFrameDroppedinstead of disappearing silently:Encode()returns an error code instead).Tests
VideoCodecFactoryTests.javaEncoderDroppingFramesKeepsSending: a Java encoder that drops every other frame keeps a call sending, which exercises the new drop reporting. Encoder drops and the flag don't reachgetStatsin M152, so the test can't check them directly.mvn -pl webrtc-jni,webrtc packageon windows-x86_64: 285 tests, 0 failures, 3 skipped.EncoderOutputProcessor; CI covers those builds.