Repository navigation
Optimize EmbeddingGemma 2 CPU inference with AVX-VNNI, faster attention, and multimodal acceleration - #6
Closed
Yazeedx0 wants to merge 32 commits into
Closed
Optimize EmbeddingGemma 2 CPU inference with AVX-VNNI, faster attention, and multimodal acceleration#6Yazeedx0 wants to merge 32 commits into
Yazeedx0 wants to merge 32 commits into
Conversation
…io towers Wall-clock accounting (act_quant, mm_q8, mm_f32, attention, norm, rope, activation, elementwise, conv) printed per tower call on stderr. Off by default; computation is unchanged. Used to size the CPU media bottlenecks on hosts without perf access.
Upstream runs the CPU vision tower in the scalar reference graph. Profile on an i7-14700HX (8 P-core threads, 280-token image, 2520 patches): attention 67.9%, Q8 GEMM 26.3%, serial activation quantization 3.3%; 31.5 s per image. * gemma2_vision_fast.c: same math as the reference (Q8_0 activations for Q8_0 weights, fp32 attention/norms, fp16-rounded patch input, the GGUF's GELU variant), executed with threaded fused row passes (norm+quant, q/k/v norms + axial RoPE from tables, GELU+quant, post-norm+residual), the FFN in 256-token chunks (O(chunk) instead of O(patches) buffers) and a persistent workspace. EI_GEMMA2_CPU_MEDIA_FAST=0 selects the reference. * gemma2_media_x86.c: g2m_attn64, tiled online-softmax attention in a transposed form (S^T/P^T/O^T) so softmax is all vertical vector ops and K/V are read in place; vector exp/QuickGELU/SiLU. Scalar fallbacks. * gemma2_vnni256.c: Q8_0 GEMM for AVX-VNNI CPUs without AVX-512. Weights repacked once into 8-row groups (one 32-byte load = 4 k-bytes of 8 rows), u8 activations with a -128*sum(w) seeded accumulator, per-block scaling of 8 rows with one convert/multiply/FMA, two int32 chains per token, loops unrolled explicitly (GCC keeps them rolled at -O2). Used by every Q8 GEMM of waves over 8 tokens (text, vision, audio); EI_GEMMA2_CPU_VNNI256=0 restores the upstream tile. * test_gemma2_vnni256: route exactness vs a double scalar Q8_0 dot and GMAC/s; test_gemma2_media: fast vs reference tower parity and timing. Results (same process and pixels, 8 P-core threads): GEMM 1.07-1.25 TMAC/s vs 196-213 for the upstream tile (5.2-6.3x), max rel err 4.8e-8; vision tower 125 ms vs 3565 ms (i70, 28x) and 977 ms vs 31459 ms (i280, 32x). Tower cosine vs the HF FP32 tower with the same activation: reference 0.99827/0.99907, fast 0.99863/0.99894 (i70/i280).
Upstream runs the CPU audio tower in the scalar reference graph (15 s clip, 1499 mel frames, i7-14700HX 8 P-core threads: Q8 GEMM 68.6%, subsampling convolutions 11.8%, serial activation quantization 8.2%; 3.3 s). gemma2_audio_fast.c keeps the reference math (Q8_0 activations, the clipped-linear input/output clamps, fp32 convolutions, norms and local relative-position attention, the reference loader's conv_norm/norm_conv mapping) and runs it with channel-vectorized AVX2 subsampling convolutions with fused LayerNorm+ReLU, Q8 GEMMs on the fast-graph routes (VNNI256 here), per-token fused clamp/norm/SiLU/GLU/depthwise-conv/quantize passes, relative-position keys computed once per layer, and a persistent workspace. EI_GEMMA2_CPU_MEDIA_FAST=0 selects the reference. test_gemma2_media gains audio (raw [frames][128] float32 mel) and audio-synthetic modes; its gate is now flattened cosine >= 0.999 and every token >= 0.95 (two Q8-activation graphs differ by rounding noise). Real Clotho clip, HF-extractor mel, tower vs the HF FP32 tower: reference 0.99893 (worst token 0.944), fast 0.99907 (0.961).
…t on x86 g2m_attn generalizes the vision kernel: head widths 64..512 (multiple of 16), separate q/kv/out strides for GQA/MQA, optional per-query key bounds (packed sequences, +-512 sliding window), exact zeros for masked keys (exp8 returns 0 below its range; fully masked lanes cannot produce NaN). The query block adapts to the width so Q^T/O^T stay 32 KB each. The fast text graph's attention (upstream: 8-query blocks, two-pass softmax, 4x4 dot tiles with horizontal sums) routes through it on x86 AVX2; EI_GEMMA2_CPU_FLASH=0 restores it, the NEON path is unchanged. bench_gemma2_cpu on 12 E-cores: 600x2 1298 -> 867 ms, 1100x1 1498 -> 908 ms; min cosine vs the reference graph unchanged (0.99966 / 0.99976). test_gemma2_attn: tiled vs scalar fp32 vs a float64 ground truth for d=64/128/256/512, GQA 4:2 and MQA, windows and packed bounds; the tiled kernel is as accurate as plain fp32 (e.g. d=512: 9.8e-5 vs 1.0e-4 of row RMS, dominated by fp32 score rounding).
Upstream's gemm_f converts four weight rows to fp32 per task on every call and computes 4x4 dot tiles ending in 16 horizontal sums. It serves the BF16 per-layer-input projection (512 -> 12288, every text/media token), the F32 vision patch embedding and the F32 audio input projection. gemma2_sgemm_x86.c expands the weight once (exact for BF16/F16) into 16-column panels and runs a 6-token x 16-output register tile (2 loads + 6 broadcasts per 12 FMAs, no reductions); fp32 throughout. One-time cost ~31 MB resident (fp32 copies). EI_GEMMA2_CPU_SGEMM=0 restores gemm_f. test_gemma2_vnni256 fNxMxK/FNxMxK: 12 E-cores, 2048x12288x512 BF16 250 vs 53 GMAC/s (4.7x), 2520x768x768 F32 269 vs 51; max rel err 1.8e-7. bench_gemma2_cpu (12 E-cores): 600x2 867 -> 735 ms, min cosine vs the reference graph 0.999655 (unchanged).
* EI_GEMMA2_AUDIO_LOGMEL=hf: log(mel + 0.001) as the Transformers Gemma4AudioFeatureExtractor computes it (upstream: log(max(mel, 0.001)), after llama.cpp mtmd-audio). * EI_GEMMA2_VIDEO_TIMESTAMPS=0: no MM:SS text before each frame, the official processor's default (add_timestamps=False). Upstream always inserts it (the add_timestamps=True layout). * (existing) EI_GEMMA2_VISION_GELU=tanh: gelu_pytorch_tanh in the vision MLP as in the official config; the GGUF lacks clip.use_gelu so upstream selects QuickGELU. Defaults are unchanged (upstream semantics) until measured.
…tention bench * gemm_q8: waves of <= 8 tokens also use the repacked VNNI256 GEMM (one task per 8-row group, upstream's short-wave task width). P8: 1-token text forward 4.65 -> 3.37 ms, 8 tokens 11.25 -> 5.37 ms; exact (rel err 2e-8). EI_GEMMA2_CPU_VNNI256_SHORT=0 restores upstream's tile. * gemma2_release.h: once a weight is repacked (VNNI256, SGEMM), its interior pages in the read-only GGUF mapping are dropped with MADV_DONTNEED so the process does not keep two resident copies. gguf.c keeps a registry of open mappings; anonymous memory is never released (it would be zeroed). EI_GEMMA2_CPU_RELEASE_WEIGHTS=0 disables. (Memory effect still to be measured end to end.) * gemma2_vision_fast.c sizes attention tasks with g2m_attn_qblock(); the fixed G2M_ATTN_QB constant is gone. * test_gemma2_attn bench N D HEADS: attention GMAC/s. Rejected (not kept): a 48x48-block attention variant with 6x16 / 4x24 register tiles: +1-3.5% at 2,520 patches, -6.5% at 10,080 (P8).
…upstream Upstream deviates from the official Transformers implementation in three places: the vision MLP uses QuickGELU (the GGUF lacks clip.use_gelu; the official config says gelu_pytorch_tanh), the log-mel floor is log(max(mel, 0.001)) instead of log(mel + 0.001), and every video frame is preceded by MM:SS text (the processor's add_timestamps=True layout; its default is False). Measured against the official Sentence Transformers FP32 embeddings on the 16 benchmark workloads (P8): upstream semantics min cosine 0.8746 (video), mean 0.983; official semantics min 0.99914, mean 0.99970 (images 0.9992-0.9997 -> 0.9996-0.9999, audio 0.9988-0.9993 -> 0.9991-0.9995, video 0.87-0.96 -> 0.99987+). The reference graph honors the same switch, so fast-vs-reference parity tests compare like with like. Adds CPU_MAX.md: what this branch changes relative to upstream d8f9e33, flags, tests.
On Intel hybrid CPUs, a single unpinned pool over every core made short requests slower than using the P-cores alone: over HTTP on an i7-14700HX, 16-token text took 13.6-13.8 ms on 28 threads vs 8.7 ms on 8 P-core threads, while large media gained 12-22% from the E-cores. The calling thread (an HTTP worker or the wave collector) could run on an E-core and sits on the critical path of every parallel step. * parallel.c: ei_cpu_hybrid_order() lists the allowed CPUs (affinity mask) as P-core primary threads, E-cores, P-core SMT siblings from /sys/devices/cpu_core|cpu_atom; a participant cap per pool (ei_thread_pool_set_cap) and the count of leading P-core threads. * engine.c: on hybrid CPUs the gemma2 CPU pool is pinned in that order (EI_CPU_HYBRID=0, non-hybrid CPUs, or more threads than allowed CPUs keep upstream's unpinned pool; thread count rules unchanged). * gemma2_fast.c / vision / audio: the caller is pinned to the pool's first P-core for each wave; text waves under EI_GEMMA2_CPU_HYBRID_TOKENS (512) run on the P-core threads only. HTTP, 28 threads, interleaved twice: t16 13.8/13.6 -> 8.7/8.3 ms, t512 130.8/128.8 -> 130.1/130.5 ms, i280 799/858 -> 810/808 ms, a15 241/251 -> 235/234 ms. In-process (bench_gemma2_cpu) the layouts are within noise, so the gain is the caller's placement.
The goldens are llama.cpp outputs (QuickGELU vision MLP, log(max) mel floor, timestamped video). Under the fork's official default the vision tower golden reads 0.9974 by construction; with EI_GEMMA2_SEMANTICS=upstream the whole suite passes (text 0.999672, vision tower 0.999257 / embedding 0.999772, audio 0.999929 / 0.999635, HTTP stage). Official-semantics accuracy is validated against the HF FP32 model.
Block-size sweep, test_gemma2_attn bench, 8 P-core threads, vision shapes (12 heads x 64): KB 32/64/128 x QB 16/32/48/64/96/128. Smaller query blocks lose (n=10080: QB=16 180-219 GMAC/s vs QB=64 264-284), so K/V re-streaming matters more than L1 fit; from QB=48 up every point is within +-5% of the default KB=64/QB=64 (305/272 GMAC/s at n=2520/10080), the same spread as repeating one configuration. No default changed.
…ling - VNNI256 repacked weights keep the Q8_0 block scales as fp16 (exact; the GGUF stores fp16) and the per-row block sums as int16 (|sum| <= 4064), expanded to fp32 / -128*sum in registers: 36 instead of 40 bytes of metadata+codes per 32 weights of a row (-10% of the repacked weights). - The audio subsampling runs conv0 per task into a thread-local buffer covering only the rows its conv1 rows read (2*grain+1), instead of materializing conv0 for the whole clip (t0 x 64 x 128 floats, 49 MB at 30 s); boundary conv0 rows are recomputed (+6% of conv0, ~11% of the subsampling MACs). The conv1 output buffer is sized for conv1. Outputs are bit-identical to 2cd784b (fast and reference towers, synthetic audio and vision dumps compared with cmp). Fresh-server peak RSS, 12 E-core threads: t512 367->352, i280 582->548, i1120 810->776, a30 845->753, v16 798->764, m_tia 1036->948 MB. Latency: subsample stage 10.2 vs 10.2 ms (1499 mel frames, 3 runs each, interleaved); HTTP p50 within noise. Tests: test_gemma2_vnni256, test_gemma2_attn, test_gemma2_media {audio,vision}-synthetic, make test-gemma2 (incl. HTTP stage) pass.
K and V were read in place from the token-major [n][768] buffers: each key of a head sat 3 KB from the next, on its own 4 KB page, and the per-head K/V stream of a query block (5.2 MB at 10080 patches) spilled L2 with the prefetchers unable to follow it. The QKV post pass now writes normed+roped K as [12][n][64] into h (free until the output projection) and a second pass writes the normed V the same way into the k buffer (free once K has moved), so no memory is added. Q and the output stay token-major (Q is packed once per task). Bit-identical (fast tower dumps for i280 and i1120 compared with cmp). test_gemma2_attn bench, 12 E-core threads, token-major -> head-major: n=2520 143/143 -> 157/153 GMAC/s, n=10080 128/126 -> 159/145 (THP via EI_ATTN_BENCH_THP alone: within noise). Vision tower, 12 E-cores, two interleaved runs: attention i1120 20.3-20.6 -> 17.4-17.8 s, i280 1.18 -> 1.07 s, i70 74 -> 70 ms; rows +45 ms at i1120 for the V pass; tower total i1120 -11%, i280 -6%, i70 -2.5%. test_gemma2_attn bench gains EI_ATTN_BENCH_HEADMAJOR / EI_ATTN_BENCH_THP. Tests: test_gemma2_attn, test_gemma2_media vision-synthetic, make test-gemma2 pass.
Runs a one-token text, a 48x48 image and a 16-frame silent clip through the CPU graphs before the server listens, so the one-time weight repack is not paid by the first request of each modality. Fresh server, first request, 12 E-core threads, two rounds: t512 647/639 -> 400/394 ms, i280 2857/3032 -> 2407/2441 ms, a30 2091/2179 -> 1530/1448 ms (warm: ~370, ~2540, ~1470). Startup +708 ms. Off by default because every tower's repacked weights then stay resident from startup (peak RSS of a t512-only server 352 -> 852 MB).
The token-major V buffer is free once ROW_V_HEADS has moved V into the head-major planes, so attention writes its output there and the separate [n][768] output buffer is gone (31 MB at 10080 patches). Bit-identical (i1120 dump compared with cmp). Fresh-server peak RSS, 12 E-cores: i280 548 -> 541, i1120 776 -> 747, v16 764 -> 761, m_tia 948 -> 941 MB.
The log-mel ran single-threaded in the HTTP worker and evaluated the 320-point Hann window with cosf for every frame. The window is now a table (same expression, computed once with the FFT twiddles), and on the CPU tower path the frames run on the engine pool; the engine lock is taken before the mel instead of before the tower (it covers the pool). GPU paths keep the serial mel outside the lock. Embeddings bit-identical (a5, a15, a30, m_ta over HTTP, compared with numpy array_equal). 12 E-core threads, server stage medians: logmel a5 5.9 -> 0.7 ms, a15 17.4 -> 1.6, a30 35.2 -> 3.0. End to end, 4 rounds with alternating order: a5 274 -> 261 ms, a15 712 -> 698 ms. Also: the streamed subsampling uses 4-row tasks (was 8): a 5 s clip has 125 output rows and 16 tasks for 12 threads. Subsample stage, 499 frames, 12 E-cores, 9 runs each: 2cd784b (unstreamed) median 3.9 ms, 8-row tasks 4.5, 4-row tasks 3.9; 1499 frames: 10-12 ms for all three.
On the CPU backend every tower call holds inference_mutex, so a text wave
queued behind a media request waited for the whole tower (a 280-token image
is ~1 s on 8 P-cores, a 1120-token one ~8-11 s). Media engine sections now
take cpu_media_mutex before inference_mutex (one media request at a time in
the towers), and the vision and audio towers call ei_gemma2_cpu_yield()
between layers, after vision attention and between vision FFN chunks. The
server's hook releases inference_mutex only when text waves are waiting and
blocks on a condition variable until one has acquired it (no barging, no
spinning), then re-locks. Text waves never take cpu_media_mutex or a tower
lock, so the handoff cannot let a second media request in and deadlock on
the tower workspace lock. EI_GEMMA2_CPU_YIELD=0 restores the plain lock;
GPU media slots are unchanged.
Head-of-line probe (parent repo bench/hol_blocking.py), 12 E-cores, one
closed-loop t16 client while one media request runs (t16 alone 22 ms):
i280: t16 p50 1133 -> 24.7 ms, max 2082 -> 183 ms, 2 -> 177 completed;
the image takes 2.29 -> 5.72 s, i.e. by the ~3.5 s of text work done
meanwhile (the engine is shared, total work unchanged)
a30: t16 p50 659 -> 89 ms, max 766 -> 555 ms (the request's own final
text pass, one wave), 2 -> 13 completed; 1.34 -> 1.62 s
Concurrency stress (bench/stress_mixed.py: 6 clients, 90 s, text/image/
audio/video/mixed): no deadlock, all 67 responses bit-identical to the same
request alone. make test-gemma2 passes.
The first use of every Q8_0 tensor (VNNI256 8-row repack) and every float-weight projection (SGEMM 16-column panels) was packed by the calling thread alone while the pool idled: a cold server paid it inside the first request of each modality. Both packers now split the work (8-row groups, 16-column panels) over the caller's pool; the bytes written are unchanged (i280 tower dump compared with cmp). First request on a fresh server, 12 E-cores, two interleaved rounds: t512 581/591 -> 415/479 ms, i280 2552/2602 -> 2318/2292 ms, a30 1868/1860 -> 1484/1509 ms (warm: ~370, ~2300-2500, ~1470). test_gemma2_vnni256 and make test-gemma2 pass.
The tooling and results behind the CPU-Max measurements, next to the engine they measure: - bench/: latency, throughput and memory harness for the native engines (HTTP), Sentence Transformers FP32 and ONNX Runtime int8; quiet-host guard and per-measurement foreign CPU load; head-of-line probe and a concurrency stress that checks responses against solo requests. - scripts/: pinned asset downloads, the upstream baseline as a git worktree of d8f9e33, fixtures, measurement queues, embedding validation, retrieval evaluation, the quantization study, profiles, summaries, tables and plots. - reports/: final results (JSON/CSV), optimization log, bottleneck analysis, plots and raw per-run data; runs not used by any report are kept in reports/raw/excluded/ with the reason. Models, datasets, fixtures, cached embeddings and the virtual environment are not included (cpu-max/.gitignore).
The repository README will describe this fork; upstream's README (engine and HTTP API documentation) moves unchanged so its history stays traceable (git log --follow README.upstream.md).
The repository README now describes this fork: what was optimized, how it differs from upstream, build and run, results against PyTorch FP32, ONNX Runtime int8 and the unmodified upstream engine, correctness and retrieval quality, limitations, reproduction and attribution. Upstream's README (engine and HTTP API documentation) is unchanged under README.upstream.md. CPU_MAX.md: commit IDs of the published history and the new layout.
make_fixtures.py and prepare_retrieval_data.py read the photos from cpu-max/data/flickr1k/images_flickr_1k_test; the archive is part of the pinned dataset snapshot and is now extracted there. Found by building the fixtures from a fresh clone.
An installed binary could not tell CPU-Max from upstream's embeddinggemma. The server now prints its identity with --version and as the first log line, e.g. EmbeddingGemma-2-CPU-Max 0.1.0-dev (commit 1a2b3c4; fork of QuixiAI/embeddinggemma.c v0.3.1-8-gd8f9e33) from CPU_MAX_VERSION (0.1.0-dev until a release is approved), the git commit (-dirty for uncommitted builds) and the upstream base, a literal in the Makefile because upstream's VERSION still reads v0.3.1 eight commits later and shallow clones have no tags. No inference code changes.
Found by building in a clean ubuntu:24.04 container with the toolchains
the new CI uses:
- clang 18 rejects __builtin_cpu_supports("avxvnni") (accepted from
clang 19 and by GCC), in upstream's gemma2_fast_x86.c and in
gemma2_vnni256.c. AVX-VNNI is now read from CPUID.(EAX=7,ECX=1):EAX[4];
the AVX2 check already covers OS support for the YMM state. Same
dispatch on the benchmark machine; g2v_enabled() drops the redundant
builtin (its AVX-VNNI level already implies the feature).
- On non-x86 targets (aarch64 cross build, macOS arm64) the repack caches
of gemma2_vnni256.c and gemma2_sgemm_x86.c were unused statics, an error
under -Werror; they now live inside the x86 sections.
Embeddings unchanged (t16, t128, i70, a5, m_tia over HTTP compared byte
for byte with the 5dd7dbf benchmark run).
…binaries The inherited installer downloaded QuixiAI/embeddinggemma.c release binaries and installed them as ~/.local/bin/embeddinggemma, so a user expecting CPU-Max got upstream. CPU-Max has no release binaries yet, so install.sh now: - builds the checkout it belongs to, or clones this repository at --ref (default main), with the system C compiler; - refuses to install unless the binary's --version names EmbeddingGemma-2-CPU-Max; - installs it as embeddinggemma-cpu-max, never replacing an upstream embeddinggemma; - rejects the GPU variants (unchanged upstream code, not built or tested here) with the command for upstream's installer, and upstream's --version option with an explanation; - warns on untested platforms (Linux/macOS arm64) and refuses others; - --with-models downloads the model files via the new cpu-max/scripts/download_models.sh (curl/wget + SHA-256, no Python; defaults to the server's own model directory). download_assets.sh uses it for the GGUF files.
…, model tests The inherited workflow ran upstream's model-free checks only. Now: - portable: upstream's make check and build on ubuntu-24.04 and macos-14, plus the build identity; - cpu-max-linux: CPU-Max kernel tests at the runner's dispatch level and forced AVX2 and scalar (EI_GEMMA2_CPU_X86), so nothing assumes AVX-VNNI; clang build; aarch64 cross build of the portable fallbacks; - model-tests: the two GGUF files (1.0 GB, pinned revision, SHA-256 checked) cached by checksum, upstream's integration suite (make test-gemma2: tokenizer, towers vs reference graphs, HTTP), and an HTTP smoke test (cpu-max/examples/quickstart.py: text, image, audio, mixed; standard library only, synthetic media). Read-only token, no secrets, actions pinned to commit SHAs, persist-credentials off.
…provenance README (claims checked against cpu-max/reports): - headline scoped to the tested backends on one i7-14700HX; same-budget geomeans (P8 2.20x, min 1.35x) and per-baseline averages alongside the best-budget 2.46x; near-FP32 quality (min 0.99914, mean 0.99970); the precision of each backend (CPU-Max Q8_0, PyTorch FP32, ONNX Runtime INT8); hardware, thread budgets, quiet-host guard, run policy (CPU-Max one run, baselines best of two) and tail latency at 28 threads; - retrieval limits: small subsets, Clotho top-1 agreement 80.2%; - vector-index compatibility between official and upstream semantics; - quick start verified end to end (build, checksum-verified models, localhost server, text/image/audio/mixed smoke test), source installer; - status: pre-release, no binaries, v0.1.0-v0.3.1 tags are upstream's; fork base d8f9e33 is v0.3.1 plus 8 commits. Tables now show p50 (p95) and the backends' precision in their names; plots regenerated with the same names. RELEASE.md, CONTRIBUTING.md: CPU-Max sections above upstream's text. README.upstream.md: note that its installer installs upstream. SECURITY.md: private reporting, default 0.0.0.0 listen, model checksums. cpu-max/reports/README.md: which raw runs feed which report, policies, known anomalies (scaling runs record a later HEAD; 28-thread tails). CPU_MAX.md: the two new engine commits.
|
Important Review skippedToo many files! This PR contains 192 files, which is 92 over the limit of 100. To get a review, reduce the PR to 100 files or fewer by splitting it into smaller PRs or changing its base branch. Upgrade to a paid plan to raise the limit. This review couldn't start because sufficient usage credits or metered capacity aren't available. Add credits or update usage-based reviews in the billing tab, then retry. ⚙️ Run configuration
⛔ Files ignored due to path filters: 21
📒 Files selected for processing: 192
You can disable this status message by setting the
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
|
amazing contribution. |
Contributor
Author
Thanks, really appreciate it! Totally understand about the rebrand. The goal was always to build on your great work and contribute back to the original project. Happy to help bring the optimizations upstream and make the integration easier! |
Contributor
|
Thank you for this work. We wanted the engine but not the rebrand, so rather than merge the fork's main branch (README, installer,
Differences from this PR, on purpose: model semantics stay llama.cpp's by default (your "official" mode is opt-in via The 9950X and EPYC boxes here are AVX-512, so your 256-bit kernel ran under a new |
ehartford
added a commit
that referenced
this pull request
Oct 18, 2026
Port the CPU-Max fork's EmbeddingGemma 2 CPU engine work (PR #6) without the rebrand
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR introduces CPU inference optimizations for EmbeddingGemma 2, building on the existing
embeddinggemma.cimplementation.The goal is to improve inference latency, throughput, and memory efficiency across text, image, audio, video, and mixed inputs without retraining or changing the model weights.
What's changed
Benchmarks
Measured on an Intel Core i7-14700HX:
Results are specific to the tested hardware and configurations.
Validation
Notes
The optimizations preserve the original model weights and include portable fallbacks. Official model semantics are enabled by default, with an option to restore upstream behavior. The optimized engine may use more peak memory than upstream.
Full benchmarks, methodology, and implementation details are available in the repository.
https://github.com/Yazeedx0/embeddinggemma-cpu-max