[BlivionIaG]

What this changes

Add native Metal image preparation, vision encoder and projector for LFM3-VL-2B from the official MLX 4/6/8-bit checkpoints. Projected image rows feed the existing LFM3 decoder and lane engine, including prompt caching, verification and concurrent serving. Stored MLX weights are read directly; Python is limited to development references and clients.

Refs #1491. Depends on admission #1513 and text #1514. Upstream PRs target main until foundations land; review the vision difference.

flowchart LR
    I[Image pixels] --> V[Vision encoder and projector]
    V --> E[Decoder embeddings]
    T[Text tokens] --> E
    E --> D[LFM3 decoder and existing lane engine]

Loading

Verification

Recipe and checkpoint revisions · Reproduction and receipts

• Current K/V-paired production candidate e9a1488abb130af39c21ee08068fe490d5926271cebcd2e9d0f815711e3730b4: 303/303 M4 serving checks across all six text/VL checkpoints, including seeded sampling, drafted/plain, concurrency, cache reuse, image content/order and cancellation/recovery. Actual extracted text candidate separately passed 120/120 and three VL refusals. The earlier greedy/FFN candidate passed 303/303 on M5; that result is not relabeled as qualification of the new binary.
• K/V dispatch reuses the already qualified paired row helper. All six 257-row logit arrays retain 394,752,000 bytes exactly, all 18 direct K/V comparisons match (including odd-row tails), and all 11,520,000 real-image 8-bit logits retain their baseline hash. Chunking/snapshot/concurrent-state exactness: 68/68. Matched M5 4-bit ABBA measurements show about 1% higher solo decode and 1.3–1.4% higher four-stream throughput; all 240 retained full-token comparisons match. This remains below the standard-server gate.
• Greedy/FFN integration: 58 LFM3 kernels pass gates 1–4; all 16 planted controls red; zero failures or limit problems. Text4 exactness: 68/68. All-six projection captures retain 394,752,000 BF16 bytes identical before/after paired FFN dispatch.
• After K/V pairing, native/diagnostic build: 69/69 steps; full test/developer-tool/golden invocation: 724/724 steps; 982/1,000 newly reported tests passed, 18 skipped, with other steps cached. Python reference checks after the explicit rounding replay: 46/46. Formatting/whitespace passed; lean adds zero findings over 428 inherited findings. After the opt-in row twin and trace-only vision snapshots, the canonical default-OFF rebuild passed 726/726 steps; 982/1,000 newly reported tests passed, 18 skipped, with other checks cached. The receipt identifies this rebuilt binary separately; the older device checks are not relabeled as checks of it.
• Earlier structured-output qualification: text4/VL4 on each host, 64/64 full-token configuration groups, plus 80/80 tool/stream/history/image-conditioned checks. This requires the optional grammar library identified in the receipt.

The completed frozen M5 run at 8955faee0 includes the compensated-cancellation correction and reports 310 kernels: 309 pass, one inherited Qwen4 HC failure, two known findings; all 317 controls red; zero limit problems. The Qwen4 failure reproduces exactly on pinned base b30d5b9d. This is not a global pass. The included low-component cancellation correction passes six operator variants and three real controls on both M4/M5, plus 960 nonfinite comparisons per host and unchanged finite checkpoint captures. The complete run also reports zero LFM3 kernel failures; its receipt preserves the executable and log hashes.

All nine GitHub checks pass at combined head 07fa59f3b. These checks do not replace the device, numerical or performance gates below.

Numerical evidence

Independent prompt reports retain all 5,581 teacher-forced rows, top-token differences and margins. Replaying named production rounding sites resolves all disjoint maximum sets; 19 lowest-index differences remain as overlapping maxima. All six genuine one-row decode captures after 63 identical history tokens have zero top differences. No new tie policy or tolerance is asserted; ideal-FP64 comparisons remain in the receipts.

The official model-card cat photograph is frozen with its provenance, canonical PNG, native patches and hashes. Projected-row relative RMS against FP64 is 0.01813 at 4 bits and 0.01606 at 6/8 bits; no acceptance threshold is invented. When both decoders receive identical FP64 projected rows, all 16 text-suffix tops match at 6/8 bits; 4-bit has one overlapping maximum tie and zero disjoint maxima.

The supplemental whole-pipeline comparison uses each implementation's own projected image rows. It matches all 16 suffix tops at 4/6 bits but only 14/16 at 8 bits, with two disjoint maxima against the ideal-FP64 graph. A separate independent CPU arithmetic replay, delegating only explicitly named scalar functions to Metal and following production FP32/BF16 sites, now reproduces all 14 vision layers, all 143,360 projected image values, all 24 decoder layers and all 11,520,000 logits byte-for-byte, including those two suffix rows. The ideal differences and both margins remain in the receipt. This establishes rounding causality; it is not labeled an independent-FP64 fidelity pass. See fidelity replay.

A supplemental independent CPU FP32 comparison retains three finite runs and six reference unit tests. Pure FP32 still gives both disjoint suffix differences. FP32 with explicit model BF16 boundaries gives zero disjoint maxima across the 16 suffix rows: row 81 matches the native contender values, while row 88 has a tied reference maximum containing the native token. Feeding fixed native projected image rows also leaves zero disjoint suffix maxima. The full 90-row scope still contains differences, including image placeholders. This demonstrates sensitivity to arithmetic and rounding; it does not replace FP64, change acceptance rules or establish a fidelity pass.

Shared convolution is now integrated: independent streams use one launch per convolution layer while preserving the same seven-tap helper and per-cache state. The receipt separates private qualification from applied-source checks: 408/408 state checks, 303/303 serving checks, six-width solo logits unchanged, and applied-source kernel gates plus all 353 normal limit records passing on M4/M5. Matched 4-bit M5 four-stream ABBA shows text +3.16–3.32% and VL +2.87–2.93%, with all 192 measured 256-token requests retaining token hashes. Other widths are correctness-tested here, not speed-qualified.

Performance and remaining draft gates

Before the K/V follow-up, byte-preserving greedy/FFN changes improve VL4 solo code/prose from 293/291 to 302/300 tok/s and four-stream aggregate from 656/658 to 813/810 on M5, measured old/new/new/old with equal 256-token outputs. Full token hashes match across 240 retained old/new concurrent comparison requests. The six-checkpoint standard-server baseline separately retains equal-length single-stream/four-stream decode and cold 2k/8k/32k/64k requests. Native rates remain below the standard server; no performance-gate pass is asserted.

Private follow-up experiments remain outside this PR’s production defaults. Shared GPU greedy/state publication passed exact-output and serving probes, but its matched performance screen was flat, so it is not enabled. The private attention candidates retained finite, chunk/serial and capacity-equal outputs across all six checkpoints, but whole-model timing was slower; they are not promoted. Sparse repair scheduling for the exact projection path is still under independent review and device measurement. These experiments do not change the current qualification claims.

A newly isolated finite attention stress case exposes a gap in the registered tests: a large score increase makes an FP32 exponential underflow, losing a numerator contribution and violating the existing numerical bound. The original scalar path and private candidates reproduce the failure. A dedicated regression and range-safe correction are in progress; no bound or acceptance rule has been relaxed. This remains an open correctness gate.

The maintainer-requested prompt-tile/decode-twin and M5 attention paths remain under qualification. Exact tiled projection and private attention experiments are disabled where performance or numerical gates fail. The remaining requirements are qualified kernels, final all-width performance, resolution of the disclosed inherited backend failure, and final fidelity review of the retained ideal-FP64 differences and their explicit rounding reproductions. Measurements on M4 Pro 48 GB and M5 Ultra 256 GB do not qualify smaller-memory machines. Weights use the LFM Open License v1.0; the recipe records revisions and checked Macs.

ATC overlap coverage remains incomplete, and reported integration conflicts with concurrent branches must be reconciled before landing. CUDA is outside this PR. The BlivionIaG ATC reservation remains held until the upstream chain is complete.

[ashhart]

Thanks @BlivionIaG for a careful port and an honest receipt. The same-engine checks are thorough (231 served, with cancellation and recovery), and the rotary fix comes with a test that fails on the old formula.

The merge with main is clean. Here is our answer on the gates.

Fidelity:

• Compare teacher-forced against an FP64 forward of the dequantized checkpoint. MLX is a second opinion, not the reference.
• Compare prompt rows and one-row decode rows separately, on identical input rows, at 4, 6 and 8 bits.
• List every top-1 difference with both tokens and both margins, and report the worst rows, not only an average.
• A difference passes only as a near-tie that a reference rounding where the engine rounds reproduces, as lfm3: serve LFM3-1.7B NVFP4 on CUDA, weight-only (stacked on #1492 and the LFM3 admission) #1508 does for the CUDA side. The 18 rows with disjoint top sets need tracing first.
• For VL: the projected rows of real images against FP64, then teacher-forced text after them.

Performance, on the same Mac and checkpoint against mlx-lm (mlx-vlm for VL), at each width:

• one-stream decode and the 4-stream aggregate, with outputs of equal length;
• cold prefill at 2k, 8k, 32k and 64k. Decode should be at least the standard server's. Cold prefill should take the prompt tiles, as below; a remaining prefill gap is fine to track as its own issue, as we do for the 27B.

Before this leaves draft:

1. Prompt rows go through the row kernels and the decode attention (zig/src/families/lfm3/forward.zig:39-49, :196-216). That is why cold prefill is 7 to 18 times slower than mlx-lm's. LFM2's Metal family takes tile_qmm's prompt tiles with a decode twin of the same bits (zig/src/families/lfm2_moe/forward.zig:30), and the tensor-unit prompt attention on an M5 (:244). Please build on those, and on LFM2: shared-round decode #1377's shared rounds for the 4-stream aggregate.
2. Please split the PR: the admission and the text decoder first, with VL stacked on them. lfm3: serve LFM3-1.7B NVFP4 on CUDA, weight-only (stacked on #1492 and the LFM3 admission) #1508 already builds on the admission.
3. Please state the weights' licence, and the Macs it was checked on, in the recipe.

Once the gates hold, we'll run the full Metal contract and the served checks on our M5 Ultra.