Add VideoIO: cross-platform video encode + frame-accurate decode#5315
Open
shai-almog wants to merge 18 commits into
Open
Add VideoIO: cross-platform video encode + frame-accurate decode#5315shai-almog wants to merge 18 commits into
shai-almog wants to merge 18 commits into
Conversation
New com.codename1.media.VideoIO subsystem (parallel to ImageIO, but deeper) for encoding application-rendered frames + audio and decoding clips to frame-accurate RGBA frames + PCM, using each platform's native codecs. The decode side fills a real gap: Media is a player whose setTime() snaps to key frames, whereas VideoReader.frameAt()/readFrames() decode exact frames and resample VFR->CFR. Gated by VideoIO.isSupported() as usual; not available on TV/Watch/Car. Public API (CodenameOne/src/com/codename1/media): VideoIO, VideoReader, VideoWriter, VideoWriterBuilder, VideoFrame, VideoCodec; wired through Display.getVideoIO() / CodenameOneImplementation.getVideoIO(). Per-platform backends (proper native APIs): - JavaSE simulator: ffmpeg/ffprobe (FFMPEGVideoIO/Reader/Writer + FFMPEGSupport) - iOS / macOS: AVFoundation (AVAssetReader / AVAssetImageGenerator zero-tolerance seek / AVAssetWriter + pixel buffer adaptor) - nativeSources/CN1VideoIO.m - Android: MediaCodec + MediaMuxer / MediaExtractor + MediaMetadataRetriever (pure Java) - Windows: Media Foundation (IMFSourceReader / IMFSinkWriter / MFTEnumEx) - cn1_windows_video.cpp - Linux: GStreamer appsink/appsrc (ACCURATE seek; x264enc/mp4mux) - cn1_linux_video.c - JavaScript: HTML5 video+canvas decode + WebCodecs encode (VideoEncoder/AudioEncoder muxed via mp4-muxer/webm-muxer) - HTML5VideoIO Tests: - maven/core-unittests VideoIOTest: API contract tests against a mock double - maven/javase FFMPEGVideoIORoundTripTest: real H.264/AAC round trip incl. a 6-frame counting animation verifying frame order + audio PCM RMS - hellocodenameone VideoIORoundTripTest: on-device suite test (encode a 6-frame counting clip + audio, decode the frames, verify count order + PCM levels); SKIPs where the platform can't encode so it never false-fails CI Docs: developer-guide Video-IO chapter + compilable snippet demo. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Collaborator
Author
|
Compared 11 screenshots: 11 matched. |
Collaborator
Author
Native Windows port (x64)Compared 138 screenshots: 137 matched, 1 missing reference.
Benchmark ResultsDetailed Performance Metrics
|
Collaborator
Author
Native Windows port (cross-compiled)Compared 138 screenshots: 137 matched, 1 missing reference.
Benchmark ResultsDetailed Performance Metrics
|
Collaborator
Author
Native Windows port (arm64)Compared 138 screenshots: 137 matched, 1 missing reference.
Benchmark ResultsDetailed Performance Metrics
|
Collaborator
Author
|
Compared 139 screenshots: 139 matched. |
Collaborator
Author
|
Compared 139 screenshots: 139 matched. |
… STL-free - HTML5VideoIO.java: use the PolyForm Noncommercial header required for JavaScriptPort sources (JavaScriptPortSmokeIntegrationTest). - developer-guide Video-IO.asciidoc: inline the code samples (like Motion-Sensors) and drop the compiled demos snippet, which referenced the not-yet-released VideoIO API and broke the demos/docs build. - cn1_windows_video.cpp: drop <string>/std::string/std::wstring (plain wchar_t buffers + malloc). The always-compiled native sources are built with clang-cl against the xwin MSVC SDK whose <yvals_core.h> rejects the toolchain Clang version (STL1000) when the C++ STL is pulled in. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
✅ ByteCodeTranslator Quality ReportTest & Coverage
Benchmark Results
Static Analysis
Generated automatically by the PR CI workflow. |
Collaborator
Author
|
Compared 140 screenshots: 140 matched. Benchmark Results
Detailed Performance Metrics
|
Contributor
|
Developer Guide build artifacts are available for download from this workflow run:
Developer Guide quality checks: |
Collaborator
Author
|
Compared 142 screenshots: 142 matched. Native Android coverage
✅ Native Android screenshot tests passed. Native Android coverage
Benchmark ResultsDetailed Performance Metrics
|
Collaborator
Author
|
Compared 133 screenshots: 133 matched. |
…he snapshot pin The Build Hugo Website job compiles the local JavaScriptPort sources (now incl. HTML5VideoIO) via the initializr's cn1-local-workspace profile, which is meant to track the bootstrapped repo HEAD (8.0-SNAPSHOT). Commit 5ad94b0 (a release version bump to 7.0.255) had re-pinned that profile to 7.0.255, so the new HTML5VideoIO failed to compile against the released core (no VideoIO). - scripts/initializr/pom.xml: restore the cn1-local-workspace profile to 8.0-SNAPSHOT so the website demo builds against the repo build. - scripts/initializr/update-cn1-version.sh: after the (release-only) global bump, restore the cn1-local-workspace profile to the repo SNAPSHOT version (read from maven/pom.xml) so a future version bump never re-pins it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The developer-guide-docs workflow treats Vale, asciidoctor and LanguageTool warnings as build-breaking. The new Video-IO chapter tripped 3 Vale alerts and 1 LanguageTool match: - Vale Microsoft.Adverbs: dropped "deliberately". - Vale Microsoft.Contractions: "does not" -> "doesn't", "It is" -> "It's". - LanguageTool TYPOS: "resampler" is a legitimate video term (VFR->CFR converter); added it to languagetool-accept.txt next to "transcoder". Verified locally: vale reports 0 alerts and run_languagetool.py reports 0 matches against the rendered chapter. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
Cloudflare Preview
|
Collaborator
Author
|
Compared 138 screenshots: 138 matched. |
Collaborator
Author
iOS screenshot updatesCompared 137 screenshots: 136 matched, 1 missing reference.
Benchmark Results
Build and Run Timing
Detailed Performance Metrics
|
Collaborator
Author
iOS Metal screenshot updatesCompared 141 screenshots: 140 matched, 1 missing reference.
Benchmark Results
Build and Run Timing
Detailed Performance Metrics
|
The build-test SpotBugs quality gate forbids DE_MIGHT_IGNORE (silently swallowed exceptions). AndroidVideoReader had two empty catch blocks the detector flagged (probeStreams frame-rate fallback and close()), plus the sibling probe catch in the same method. Give each a real body: - probeStreams() frame-rate float fallback: log a one-line diagnostic instead of swallowing (Log.p, no stack spam for a benign fallback). - probeStreams() outer catch: Log.e the real probe failure. - close(): Log.e a retriever.release() failure. The remaining empty catches in the file are inside finally cleanup blocks, which the DroppedException detector excludes (confirmed: readAudio's finally catches were not flagged). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
✅ Continuous Quality ReportTest & Coverage
Static Analysis
Generated automatically by the PR CI workflow. |
…screenshots The VideoIORoundTripTest was registered first in the device runner and a startup VIDEOIO_DIAG block enumerated encoders before any screenshot. Both touch the native media stack before the screenshot suite runs, which broke two unrelated CI jobs: - Mac native: initializing AVFoundation/VideoToolbox shifts the macOS display color space, so MainActivity + AdsScreen (and every later screen) captured with subtly different colors -> "Screenshot differs". Content was identical; only the color profile moved. - Linux musl (Alpine): the round-trip's background GStreamer worker (started first) kept running after the runner's per-test deadline advanced past it, then contended with / deadlocked GTK-Cairo rendering ~38 screenshots in, so the native suite stalled at pngs=38 (need 100). glibc x64/arm64 were fine. Fixes: - Move VideoIORoundTripTest to LAST in DEFAULT_TEST_CLASSES. It takes no screenshot, so every baseline keeps master's exact order/state; its encode/decode now runs only after all 100 captures. - Remove the startup VIDEOIO_DIAG block (native compile coverage still comes from the test referencing VideoIO; the diag added nothing but early init). - Make the round-trip worker a daemon so a blocked native media call can never keep the suite process alive. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…API) Thread.setDaemon(boolean) is not in the Codename One CLDC11 whitelist, so the bytecode-compliance check rejected it, failing hellocodenameone-common and every platform build that depends on it (Android/iOS/Mac/Linux/JS). The daemon flag was only defensive; the real fix -- running the VideoIO round-trip test last so its native media worker can't perturb screenshots -- does not need it. The worker's native calls are already bounded (<=30s waits), so it finishes on its own after the suite has captured every screenshot. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The watch target builds the shared CN1VideoIO.m, but watchOS marks the entire AVFoundation video stack (AVAssetReader / AVAssetWriter / AVAssetImageGenerator and the AVVideoCodecType* constants) API_UNAVAILABLE, so the watch build failed to compile. (tvOS and iOS have these APIs and built fine.) - CN1VideoIO.m: wrap the AVFoundation imports + reader/writer state + helpers + real entry points in #if !TARGET_OS_WATCH, and provide watchOS stub entry points (return unsupported) so the shared translated IOSNative bytecode still links. - IOSImplementation.getVideoIO(): return null on the Watch and TV targets, so VideoIO.isSupported() is correctly false there (matches the documented platform support) and the native video methods are never called at runtime. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Collaborator
Author
|
Compared 216 screenshots: 216 matched. |
The @since-tags gate only scanned CodenameOne/src, so hallucinated/garbage @SInCE markers in the platform ports slipped through: three "@SInCE 8.0" (a release that doesn't exist; the latest is 7.0.x) and three "@SInCE 12130" (a stray build number) across JavaSEPort, AndroidImplementation, BillingSupport, Executor and JavaFXLoader. All pre-existing, all corrected to @SInCE 7.0 (a released tag where those features already shipped). check-since-tags.sh now also scans the CN1-authored implementation packages under Ports/ and maven/ (**/com/codename1/**), in a LENIENT mode that only rejects a non-released @SInCE whose leading version component is >= the latest released major (e.g. 8.0, 7.1, 12130) -- i.e. a hallucinated current/future release. Legacy/upstream refs that legitimately sit next to that code (Java's "@SInCE 1.3" in the adapted Base64, etc.) are left alone, and vendored trees outside com/codename1 are never scanned. The workflow now also triggers on port/maven source changes. Verified: the check flags a stray "@SInCE 8.0" and is clean after the fixes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the animation screenshot test that encodes a six-frame counting clip (digits 1..6 over a ramping grey background), decodes it back with the video decoder, and lays the SIX DECODED frames out as the standard 2x3 animation grid. A decode regression is then visible -- the digits stop appearing, decode out of order, or come back blank. This is the visual companion to VideoIORoundTripTest (which does the strict programmatic checks). The native encode/decode runs off the EDT; only once the decoded frames are in hand does it show the host form and capture the grid (an off-screen image, so it is immune to the macOS AVFoundation colour-space shift). It is registered after the last normal screenshot test for the same reason VideoIORoundTripTest runs last. Where the platform cannot encode (iOS simulator H.264, unsupported targets, a browser without WebCodecs) it reports SKIPPED and emits no screenshot. Lossy per-codec decode is absorbed by per-baseline .tolerance files; baselines are captured per platform from CI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The grey ramp made the grid effectively black and white. Give each of the six frames a distinct saturated hue (red, orange, yellow, green, blue, violet) with a contrast-picked digit, so the decoded frames are told apart by colour as well as number. The solid colour blocks survive 4:2:0 chroma subsampling cleanly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Media Foundation delivers/consumes RGB32 bottom-up by default, so encoded and decoded frames came out upside down on Windows. Request a top-down layout explicitly by setting a positive MF_MT_DEFAULT_STRIDE on both the decoder's RGB32 output media type (pinned to the native frame size) and the encoder's RGB32 input media type, matching Codename One's top-down RGBA convention. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…oid baseline The earlier MF_MT_DEFAULT_STRIDE approach corrupted the Windows decode (forcing w*4 sheared the decoder's padded stride into green striping). Revert that and instead handle Media Foundation's bottom-up RGB32 convention with explicit vertical row-flips on both sides: the encoder writes rows reversed (so frames store right-side-up) and the decoder reads rows reversed (so output is Codename One's top-down RGBA), using MF's natural stride. Round-trip is identity and external clips decode upright. Also seed the Android baseline for VideoIODecodedFramesScreenshotTest from the CI capture (the decoded 1..6 grid renders correctly there) with a tolerance file covering codec edge ringing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e upright Flipping rows on both encode and decode cancelled out, leaving the frames still upside down. Media Foundation flips our top-down RGBA only on the RGB32 *input* (encode); its H.264/HEVC decode hands RGB32 back top-down. So compensate once, on the encoder, and read the decoder straight. Round-trip is now upright and external clips decode the right way up. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
cn1FrameAt drew the decoded CGImage into a raw CGBitmapContext through an extra vertical CTM flip (translate+scale), which mirrored every decoded frame -- the Mac Catalyst capture of the decoded 1..6 grid came back upside down (the iOS simulator can't encode so it was never exercised there, and the round-trip assertion test is flip-invariant). A raw CGBitmapContext already stores row 0 as the top of the image, so draw straight. The encoder writes a top-down CVPixelBuffer, so the round-trip and external clips are now upright. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…frames test Mac Catalyst has full AVFoundation, so it encodes/decodes the 1..6 clip and the screenshot test produces there (unlike the iOS simulator, which skips). Commit the CI capture -- now upright after the AVFoundation orientation fix -- as the baseline with a codec-ringing tolerance, so build-mac-native has a reference to compare against. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A new
com.codename1.media.VideoIOsubsystem — "ImageIO, but for video, and deeper". It uses each platform's native codecs to:getAvailableEncoders()/getAvailableDecoders(), incl. hardware flags).The decode side fills a real gap:
Mediais a player whosesetTime()snaps to key frames with no exact-frame guarantee.VideoReader.frameAt(ms)returns the precise frame;readFrames(fps, cb)walks the whole clip resampling to a constant rate;readAudio()returns interleaved PCM (AudioBuffer).Gated by
VideoIO.isSupported()as usual (mirrors theImageIO/Display.getImageIO()pattern). Supported on JavaScript, iOS, macOS, Android, Windows, Linux and the desktop simulator. Not TV/Watch/Car.API (
CodenameOne/src/com/codename1/media)VideoIO(entry/factory + codec enum),VideoReader,VideoWriter,VideoWriterBuilder,VideoFrame,VideoCodec. Wired viaDisplay.getVideoIO()→CodenameOneImplementation.getVideoIO()(null default).Per-platform backends (proper native APIs)
ffmpeg/ffprobe(FFMPEGVideoIO/Reader/Writer)AVAssetReader/AVAssetImageGenerator(zero-tolerance seek) /AVAssetWriter+ pixel-buffer adaptor (nativeSources/CN1VideoIO.m)MediaCodec+MediaMuxer/MediaExtractor+MediaMetadataRetriever(pure Java)IMFSourceReader/IMFSinkWriter/MFTEnumEx(cn1_windows_video.cpp)filesrc!decodebin!videoconvert!appsink(ACCURATE seek) /appsrc!videoconvert!x264enc!mp4mux!filesink,dlopen'd (cn1_linux_video.c)<video>+<canvas>decode + WebCodecsVideoEncoder/AudioEncodermuxed viamp4-muxer/webm-muxer(HTML5VideoIO)hellocodenameone(HelloCodenameOne.kt+VideoIORoundTripTest) references the API so every port build compiles the native code in CI — the same pattern already used for the Camera/CarPlay natives.Tests
maven/core-unittestsVideoIOTest— API contract/behaviour tests against an in-memory double (always run).maven/javaseFFMPEGVideoIORoundTripTest— real H.264/AAC encode→decode round trip, including a 6-frame counting animation that verifies frame order survives the lossy codec + audio PCM RMS lands in the expected band.hellocodenameoneVideoIORoundTripTest— on-device suite test: encode a 6-frame counting clip (1→6) + a 440 Hz tone, decode the individual frames, and verify the count order + PCM levels. It's an assertion test (no screenshot, so it doesn't touch baselines) and SKIPs where the platform can't encode so it never false-fails CI.docs/developer-guide/Video-IO.asciidoc) + compilable snippet demo.Verification done locally
CN1VideoIO.mcompiles against the real iPhoneOS SDK via the generated ParparVM project; the translator's mangled bindings match (it links). Three real bugs were found & fixed this way.cn1_linux_video.ccompiles clean against real GStreamer 1.24.2 in an arm64 container.Notes
mp4-muxer/webm-muxerfrom a CDN (same mechanism the port already uses for VideoJS). If a fully offline self-contained bundle is wanted, the libs can be vendored intowebapp/assetsas a follow-up.hasAudio()==false); frame decode + full encode work.🤖 Generated with Claude Code