Repository navigation
feat: add encoded frame transforms and a media recorder - #300
Merged
Merged
Conversation
Encoded frame transforms (insertable streams): RTCRtpSender and RTCRtpReceiver get setTransform(RTCEncodedFrameTransformer), which sees every encoded frame between encoder and packetizer, or depacketizer and decoder, and may read, replace or drop its payload. The transform runs on a thread of its own per sender or receiver, never on a media thread, and frames reach Java without a copy unless the payload is read. Senders gain generateKeyFrame(), receivers requestKeyFrame(). One native transformer is installed per sender or receiver, found through a weak registry, so that a Java transform and native observers share it and a video sender is not restarted on every change. The native extension API gains encoded frame observers, through which the media module's new MediaRecorder writes the frames of senders and receivers into Matroska, WebM or MP4 files without re-encoding. FFmpeg is built with the matroska, webm and mp4 muxers, extract_extradata and the AV1 parser for it.
macOS prefers H.264 through VideoToolbox by default. A transform that changes the whole payload breaks H.264, whose packetizer splits frames at start codes in it, so nothing reached the receiver there; the recorder tests check for VP8 as well. VideoToolbox on CI runners also encodes far fewer frames, so the tests wait for fewer.
SendableMetatype
added a commit
to EduGeyser/webrtc-java
that referenced
this pull request
Oct 10, 2026
Brings in upstream 0.19.0 and the work after it: the FFmpeg-based webrtc-java-media module with its native extension API (devopvoid#291, devopvoid#292, devopvoid#299, devopvoid#300), encoded frame transforms (devopvoid#300), the remaining sender encoding parameters and RTCConfiguration fields (devopvoid#316, devopvoid#317), custom and hardware video codec factories (devopvoid#303, devopvoid#310 to devopvoid#315, devopvoid#318 to devopvoid#330), the log sink and 32-bit ARM stats fixes, and upstream's CI rework (devopvoid#294 to devopvoid#309). Fork-side adaptations: - The media module builds with Gradle, and only in the full variant: it needs audio and video. On Linux it links like the JNI library, since the Chromium sysroots no longer ship the static libgcc_eh.a. The sysroots carry stubs of glibc that no executable links against, so FFmpeg's configure links with lld and allows undefined symbols in shared libraries. - The data channels variant leaves out the new media sources, the codec factory setup and the hardware codec headers, and skips the new tests that need audio or video. - The module tests run on the module path with the test resources patched in, and the core tests open the internal package that NativeApi and NativeLoader are tested in. - The WebRTC checkout keeps this fork's sync, with upstream's checkout_instrumented_libraries setting. - CI builds all platforms from one reusable workflow, as upstream now does. The full variant also builds FFmpeg and tests the media module. The WebRTC and FFmpeg caches are saved as soon as their builds finish, and a release is prepared locally and pushed in one atomic push. - The Maven poms and upstream's Dependabot setup stay out, and the README is the fork's own; the rename script now leaves it alone.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two features that build on each other:
Encoded frame transforms (insertable streams).
RTCRtpSenderandRTCRtpReceivergetsetTransform(RTCEncodedFrameTransformer), which sees every encoded frame between encoder and packetizer, or depacketizer and decoder, and may read, replace or drop its payload. Frames expose their metadata (RTCEncodedVideoFrame: key frame, size, RID, layers;RTCEncodedAudioFrame: sequence number, audio level, CSRCs). Senders gaingenerateKeyFrame(), receiversrequestKeyFrame().MediaRecorder(media module). Records senders and receivers into.mkv,.webmor.mp4without re-encoding: encoded frames go into the file as they are.Design
getData()is called, into a per-thread direct buffer reused across frames. A frame used after its transform returned, or from another thread, throwsIllegalStateException; nothing can reach a native frame that has moved on.webrtc_java_api.h) gains encoded frame observers (appended members, same version). The recorder copies frames on the WebRTC thread into a bounded queue and muxes on its own thread. It starts video at a key frame and requests one (throttled), writes the header once all tracks are described or after 3 s, maps RTP timestamps onto a common timeline, and uses fragmented MP4 so a cut-short file still plays. A codec the container cannot hold (e.g. VP8 in MP4) leaves that track out with a warning instead of failing the file.extract_extradataand the AV1 parser. The CI FFmpeg cache key already hashes the component list, so no workflow change is needed.Also in this PR
EncryptedRecordingExample: AES-GCM end-to-end encryption through transforms (codec header kept in the clear and authenticated), with the decrypted side recorded.ApiCheckhelper used by the player and the recorder.Testing
webrtc: full suite 184/184, including 9 new transform tests (encrypted round trip, metadata, stale-frame access, drop and clear, exceptions, audio, key frame requests, shared transforms). Clean under-Pjni-check(noFATAL ERROR in native method).webrtc-java-media: full suite 47/47, including 7 new recorder tests that read the recording back (codec, size, duration), cover the left-out-codec path, empty-file deletion and stopping from the listener.EncryptedRecordingExampleend to end: 808 frames encrypted/decrypted, 0 rejected, 10 s recording written.