[grantoverton]

Part of #1190. Stacked on #1125, #1189, #1191 and #1192; base for review: the #1192 head 7f82411. This branch carries its parents' commits until they merge. Review the last three commits:

• 21e0ad7: measure()'s timing windows draw varied draft tokens, so MoE expert rereads are priced. This one is flag-independent: measure() runs at serve startup whether or not profiling is on. It is split out so it reviews on its own.
• 6e03930: TF_Q27_PROFILE=1 logs each round's class GPU times, and replies carry gpu_seconds. Opt-in instrumentation, off by default.
• f8c63b6: with the flag set, a draft frame holds its trace, so profile spans no longer reorder the drafter's GPU work. This fixes the acceptance collapse the first cut of this PR listed as a known limitation.

21e0ad7: varied draft tokens in measure()

Constant-token windows drew token 1 on every row. Each window's expert rereads were therefore priced as L2 hits, and the cost tables understated expert diversity. The draws are now a DefaultPrng seeded by the window width, deterministic from run to run, and the pending token varies too. The window cost tables shift for every serve, so the cost key gains a varied-drafts part and stale cached costs are retired rather than compared. native/qwen27_lanes.zig only (+7/−3).

6e03930: TF_Q27_PROFILE class timers

The gpu_profile.Trace per-class timers exist with no production callers. TF_Q27_PROFILE=1 wires them up on the 27B lane backend: each round's class GPU times are logged, and replies carry gpu_seconds.

• core/gpu_profile.zig: the trace gains the classes the 27B engine reports (route, experts, dense/attention path, drafter) and a per-round print.
• core/lanes/backend.zig / stream.zig, core/engine_api.zig, core/lane_host.zig: the timing points and the gpu_seconds field carried on replies.
• families/qwen27/decode_round.zig, dflash/runtime_backend.zig, dflash/runtime_frame.zig, native/qwen27_draft.zig: profile.begin around the round's sections, the drafter pass included (it was uninstrumented).
• native/qwen27_lanes.zig / qwen27_served.zig / qwen27_slots.zig, server/chat.zig, server/cli.zig: the flag and the export.
• README.md / docs/api.md: the env var and the reply field.
• tests/qwen27/dflash_run.zig: the run harness honors it.

With the flag unset, nothing in this commit changes arithmetic or output: it adds stderr logging and a reply field only.

f8c63b6: hold profile spans while a draft frame is open

With TF_Q27_PROFILE set, DFlash2 draft acceptance collapsed while output stayed exact: 8.1% accepted against 62.8–66.2% unset (paired benches, M3 Ultra).

The cause: Trace.begin moved each span's op onto a private command buffer that commits and waits at once. Inside a draft frame, that buffer overtook the frame's pending ops on its long-lived encoder. So borrowed trunk kernels (embed, head quant, gather's unstack) read state that had not been written yet: stale hidden state into the logits, stale own-KV into the merged context.

A draft Frame now holds its trace for the active epoch, so spans keep the frame's encoder. The frame's own buffers still time the drafter class. core/gpu_profile.zig, dflash/runtime_frame.zig (+11/−4). With the flag unset, no code path changes.

Checks

• zig build native -Dcpu=apple_m1 -Dversion=1.0.0 passes clean on 21e0ad7 and 6e03930. f8c63b6 builds; its binary is the one the receipt below ran.
• On 6e03930, the tip tree was byte-identical to the reviewed single-commit cut of this change. On that tree, zig build test test-golden -Dcpu=apple_m1 gave test-golden 0 unexpected differences (306/319 + 26/26 equal). The test step's device-dependent binaries fail on this machine while its GPU is serving a model. The same binaries fail identically on the base head 30e5dcb (run for comparison), so the failure is inherited, not introduced by this diff.
• On f8c63b6: test-lane-api-host 222/222, test-lane-halt-host 13/13.
• tools/zig/lean_check.py: 444 findings, as on the base (zero added).

Receipts

• The fix, profiled, on f8c63b6 itself (M3 Ultra, TF_Q27_PROFILE=1, quiet window quiet-002621): the greedy det cell accepted 88/88 drafts on all 5 reps, the same as unprofiled, and the det hash a420a07487e75082 was unchanged and equal across reps. Before the fix, the same cell ran at 8.1% accepted with the flag set.
• The same changed lines, on a private build, were run profiled and unprofiled in one quiet window: 88/88 on all 5 reps both ways, det hash a420a07487e75082 both ways.
• Decode profile on a 1,073-round drafted serve of this stack's 6e03930 tip: experts 41% (22.3 ms), dense 28% (15.1 ms), DFlash2 drafter 16% (8.7 ms), route 5% (2.9 ms), GDN+attention ~3.1 ms. This is the component table the instrumentation exists to produce, reproducible with TF_Q27_PROFILE=1. It was taken before the fix, so drafting ran at the reduced acceptance, and the class shares describe that profiled round.

Not run

No speed numbers from the fix window are quoted: it gated exactness and acceptance only.

[ashhart]

Thanks @grantoverton. Per-class GPU times will be useful. Rebased onto current main, it needs these before an A/B:

1. gpu_seconds is added to every Qwen3.8-27B reply (qwen27_lanes.zig sets it unconditionally). New reply fields must stay off unless asked for, so please only set it when TF_Q27_PROFILE is on.
2. 21e0ad7 (varied tokens in measure() and a new cost-cache key) changes every drafted server's cost table and depth choice on the dense model. Its rationale is about expert rereads in the MoE stack, which isn't on main. Please move it to its own PR with its own A/B.
3. gpu_seconds adds the whole shared round to each stream (the commit says "the share"), and DFlash drafter time isn't counted. Either is fine if the docs and code agree.
4. f8c63b6 fixes a regression 6e03930 introduced (span splitting fed stale state to the drafter), so please squash them. That bug also means a profiled run needs to equal an unprofiled run token for token; we'll check that on a Mac.
5. Smaller: TF_Q27_PROFILE=0 or an invalid value currently means 64 (better: off, with a warning); "rounds" also counts prefill passes; holds saturates instead of releasing the trace it held; and there are no tests for the reply field, the env parsing or the hold balance. Please also drop tool trailers and card IDs from commit messages.

The branch also sits on 22 commits from the unlanded Ornith/JointHead stack. Rebasing onto main and keeping only this PR's commits makes it reviewable (we did that locally and it applies with small conflicts).

[ashhart]

Review is on #1189, covering the whole stack #1189–#1215.

[ashhart]

Thanks @grantoverton. The round-class GPU timings are useful on their own, so we'll finish this one on our side as a single commit crediting you: gpu_seconds only when TF_Q27_PROFILE is on, the env parsing tightened, and the tool trailers and card ids left out. We've closed the rest of the decision stack (#1189-#1192) as noted there.

[ashhart]

Merged, thank you @grantoverton. We finished it on your branch as one commit with you as co-author.

On an M5 Ultra with Qwen3.8-27B and DFlash2:

• Profiler off (the default): the tokens match main, and prefill and decode after a prompt stay within 0.3% of main across eight alternating runs.
• Profiler on: the tokens still match. Timing every round costs speed: one stream decodes at about a ninth of its usual rate. The changelog therefore describes it as a diagnostic, not something to serve with.

It will be in the next release.

[ashhart]

Thanks @grantoverton: TF_Q27_PROFILE shipped in TensorFold 1.1.9.