Skip to content

lfm3: serve the MLX text decoder on Metal - #1514

Draft
BlivionIaG wants to merge 44 commits into
ashhart:mainfrom
BlivionIaG:feature/lfm3-text
Draft

BlivionIaG wants to merge 44 commits into
ashhart:mainfrom
BlivionIaG:feature/lfm3-text

Conversation

@BlivionIaG

@BlivionIaG BlivionIaG commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

What changes

Serve LFM3-1.7B from the official MLX 4/6/8-bit checkpoints through the native Metal lane engine. Add decoder operators, per-stream convolution/KV state, verification, snapshots and prompt-cache integration. Paired FFN/KV projections, shared convolution and post-add/next-norm fusion preserve their retained rounding boundaries. The range-safe attention correction repairs underflow/overflow without relaxing numerical bounds.

Refs #1491. Depends on admission #1513; vision is stacked above this in #1502. This text slice refuses VL checkpoints. The PRs target upstream main until their foundations land; review the text-only difference.

flowchart LR
    A[Admission: 1513] --> T[Text: this PR]
    T --> V[VL: 1502]
Loading

Validation and numerical evidence

Recipe and receipts pin source, executable and input hashes. The extracted text range-correction artifact passes its native/contract/host build: 67/67 steps, formatting passed, and zero new lean findings over 428 inherited. Its separate receipt does not claim a separate text-binary GPU run.

The matching combined candidate passed locked local M4 qualification across all six text/VL checkpoints: 408/408 state checks, 303/303 serving checks, including sampled/cancellation/image cases, and 394,752,000 logit bytes identical to the previous public runtime on retained fixtures. The combined range receipt identifies that artifact separately. Registered original-fail/new-pass attention cases retain the same FP64 bounds; skip-repair and clear-risk controls fail as intended. Guarded M5 testing separately passed the current 70-kernel combined LFM3 contract set and normal limits. Live CI is shown by PR checks.

Independent prompt reports retain all 5,581 rows, changed top tokens and both margins. Named production-rounding replay resolves disjoint maximum sets; 19 lowest-index differences remain overlapping maxima. All six genuine one-row decode captures after 63 identical history tokens have zero top differences. These diagnostics do not invent a tie policy or erase ideal-FP64 evidence.

The earlier frozen combined full-backend M5 run at 8955faee0 reports 309/310 kernels passing, one inherited Qwen4 HC failure, two known findings, all 317 controls red and zero limit problems. The failure reproduces on pinned base b30d5b9d. This is not a global pass or a complete-backend check of the extracted text binary.

Performance and remaining draft gates

The final performance gate is not met. The locked M5 comparison retains balanced 4-bit code/prose, equal 256-token solo/four-stream decode, exact cold 2k/8k inputs, memory measurements and fresh environment/model pins. Current text4 code solo/four-stream medians are 315.75/861.85 tok/s, versus standard 344.20/995.0. Standard token hashes are unavailable; cache allocation differs despite a common 70k request budget. Locked M4 evidence is separate. Neither qualifies all widths, long contexts or smaller-memory Macs.

Earlier unlocked timings are historical only. Shared tile/twin and M5 matrix/attention experiments remain private when qualification is incomplete. Before leaving draft: qualify the required shared prompt/decode paths, meet the all-width standard-server gate, finish long-context and final-backend checks, and complete fidelity/integration review. No precision reduction or acceptance-bound relaxation is used to claim a pass.

Weights use LFM Open License v1.0. CUDA is outside this PR. ATC overlap/integration conflicts remain disclosed; the BlivionIaG reservation stays held until the upstream chain is complete.

Prepare text and VL 4/6/8-bit checkpoints with revision-pinned header inspection and native metadata validation. Inference remains unregistered pending operator and model qualification.

Refs ashhart#1491
Serve LFM3-1.7B and LFM3-VL-2B MLX 4/6/8-bit checkpoints through shared lane verification, prompt snapshots and native image rows. Preserve checkpoint quantization and BF16 activation boundaries with FP32 reductions.

Add independent MLX references, real-checkpoint exactness and served cancellation checks, and Metal kernel contracts. Refs ashhart#1491.
Add a direct-rounding CPU FP64 oracle and image-conditioned tracing. Exercise production-shared FP32 reductions with real reduced-precision controls. Bound sliding attention dispatch without changing partial slots or merge order, and reject malformed diagnostic inputs.
jschmied added a commit to jschmied/TensorFold that referenced this pull request Oct 28, 2026
config.servedOn(config, backend) refuses modelopt_nvfp4 on Metal with Lfm3ModeloptNvfp4NotOnMetal; CUDA serves it, and both serve the MLX layouts. Metal's LFM3 open (ashhart#1514) calls it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant