Run local LLMs on the RTX 5880 Ada - #216
Draft
samestep wants to merge 91 commits into
Draft
Conversation
Two tiers, because 48 GB of VRAM turns out not to reach a better class of model than the ~30B one it already holds. `ollama` serves what fits entirely in VRAM: qwen3.6 at 27b/q8_0 is the strongest thing that size allows, and the 35b-a3b MoE activates only 3B parameters per token, so it runs roughly 3x faster for six points of Artificial Analysis index. `llama-cpp` serves DeepSeek-V4-Flash, which is 284B parameters but only ~13B active. 97% of it is expert weights, so `fit = "on"` parks those in system RAM and keeps attention and the KV cache on the GPU. That reaches well past anything VRAM-resident, at 5.2 tok/s rather than ~65. llama.cpp is pinned ahead of the b9190 in nixpkgs because DSpark speculative decoding landed in b10231, and it roughly doubles DeepSeek's throughput. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TgwAc8cnQQHo3VAPRjrfHM
DeepSeek-V4-Flash is the wrong tool for a smart speaker: 5.2 tok/s is fine for "go think about this" and useless for "turn off the lights". It also wants ~100 GB of RAM, which fights the sandbox VM. Drop it. What replaces it is Home Assistant plus the Wyoming voice services, all of which nixpkgs already packages. The split that matters is that Home Assistant's own intent matcher handles rote commands without invoking a model at all, and only unmatched utterances reach ollama — routing everything through the LLM would be slower than the Alexa this is meant to replace. qwen3.6:35b-a3b becomes the assistant model. It activates 3B parameters per token, so it answers at conversational latency, and ollama now pins it with OLLAMA_KEEP_ALIVE=-1 rather than evicting it after 30 minutes. Pipelines, wake words and the conversation agent are chosen in the Home Assistant UI; the services here announce themselves over zeroconf. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TgwAc8cnQQHo3VAPRjrfHM
assist_pipeline only handles intents locally when prefer_local_intents is set, and once the agent has control it keeps just GET_STATE and media search. The comment claimed the opposite. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TgwAc8cnQQHo3VAPRjrfHM
The faster-whisper and piper modules default zeroconf on and pass `--zeroconf`; the openwakeword module has no such option, so Home Assistant never discovers it. Upstream has supported the flag since October 2025 and nixpkgs ships v2.1.0, so pass it through extraArgs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TgwAc8cnQQHo3VAPRjrfHM
Passing --zeroconf alone made the service fail to start. The package carries no optional-dependencies, so unlike the faster-whisper module there is nothing to splice, and wyoming.zeroconf was an ImportError; take the extra from the wyoming library instead. Its unit also omits the AF_NETLINK that piper and faster-whisper add for zeroconf, without which interface enumeration cannot work. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TgwAc8cnQQHo3VAPRjrfHM
ollama 0.32.3 faults with "CUDA error: an illegal memory access was encountered" when qwen35moe does constrained decoding for tool calls whose parameters use arrays or enums, which is what Home Assistant sends. Reproduced against the API: simple one-string tools never fail, rich schemas fail intermittently, and the fault poisons the runner's CUDA context so every later request errors until it restarts. The dense qwen3.6:27b does not reproduce it at 8, 10 or 12 rich tools. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TgwAc8cnQQHo3VAPRjrfHM
The model's training data is stale, and Home Assistant exposes any script that is exposed to Assist as a callable tool, returning the script's response to the model. So a script wrapping a local SearXNG gives it lookup without sending queries to Google. SearXNG runs on loopback with its built-in HTTP server; uwsgi is for public instances. Its limiter is off because bot detection would reject Home Assistant's requests, and JSON is added to the output formats. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TgwAc8cnQQHo3VAPRjrfHM
Requiring a manual step before the first rebuild is a bad interface, and the key should not sit in the world-readable Nix store. A oneshot mints one into the state directory if absent. It needs its own unit because systemd reads EnvironmentFile before any ExecStartPre of the service consuming it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TgwAc8cnQQHo3VAPRjrfHM
The module hands searx-init the same EnvironmentFile and searx.service requires it, so searx-init is what reads the file first — and what failed on the missing file during activation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TgwAc8cnQQHo3VAPRjrfHM
It lives in Home Assistant's storage rather than in this config, so a rebuild does not reproduce it. Keeping the wording here because getting the model to reach for its tools rather than answer from memory took several attempts. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TgwAc8cnQQHo3VAPRjrfHM
Piper was archived upstream on 2025-10-06 and is the weakest part of the pipeline to listen to. Kokoro is an 82M StyleTTS2 model under Apache 2.0 with 50-odd voices and much more natural prosody. Nothing in nixpkgs speaks Wyoming for it, so this runs the community server as a container: pinned by digest rather than latest, published on loopback only like the other Wyoming services, and with the model baked into the image so there is no runtime download. Piper stays enabled for now so the two can be compared directly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TgwAc8cnQQHo3VAPRjrfHM
The image is relvacode's fork, whose main.py accepts only --host/--port/--uri/--debug. I took --voice from the nordwestt source I had cloned, which is different code, so the container exited with "unrecognized arguments" on every start until systemd gave up. Voice selection happens per request from Home Assistant in this fork. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TgwAc8cnQQHo3VAPRjrfHM
CPU synthesis costs about 0.4 s of every reply, measured across realistic reply lengths. Only the nordwestt fork publishes a CUDA image, so switch to it; it reads voices from the model the same way the relvacode build does, so the voice list survives. kokoro-onnx picks its execution provider from ONNX_PROVIDER, which is how the upstream Intel compose file selects OpenVINO. The container toolkit turns on Docker's CDI support, so the device reference resolves. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TgwAc8cnQQHo3VAPRjrfHM
The published cuda image reports 'Available providers: AzureExecutionProvider, CPUExecutionProvider' — it ships a CPU-only onnxruntime, so it never had a GPU path. nixpkgs has python3Packages.kokoro, but that is the torch build and a CUDA torch is a multi-hour compile that the cudaCapabilities pin makes uncacheable. Neither is worth the ~0.3 s. Switching to dockerTools.pullImage answers the more useful complaint: the image now downloads during the rebuild rather than afterwards on first start, and the unit drops its network-online.target dependency. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TgwAc8cnQQHo3VAPRjrfHM
26.05 ships 0.32.3, which predates the Muse Glimmer architecture; that landed in 0.32.7, and the unstable input is already pinned at exactly 0.32.7. An overlay is much less work than overriding a buildGoModule by hand, since the package definition moved between the two releases. Instantiating the unstable package set with the stable one's config carries allowUnfree and the cudaCapabilities pin across, so this is still an sm_89-only build; verified realArches is still ["sm_89"]. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TgwAc8cnQQHo3VAPRjrfHM
Muse Glimmer landed in 0.32.7 but took four follow-up fixes through 0.32.11 — parser recovery for mangled tool-invoke names, and a reasoning-template match — and the registry gates on those, so 0.32.7 still gets a 412. Overriding the version by hand does not work either: llamaCppVersion is let-bound rather than an attribute, and 0.32.13 bumps the vendored llama.cpp from b10242 to b10380 with a hard mismatch check. nixos-unstable already carries 0.32.13 while nixpkgs-unstable is two days behind on 0.32.7, and it gates on the NixOS test suite, so it is if anything the more conservative channel. This also unblocks Qwen 3.8, which needs 0.32.12. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TgwAc8cnQQHo3VAPRjrfHM
Grading "turn on the kitchen light" against a refusal message is not grading anything; with real entities it becomes a state assertion. The demo integration supplies lights, switches, fans and covers with plausible names for free. Only a handful get exposed to Assist. Every exposed entity is written into the system prompt, so exposing all of them would inflate prefill and change the thing being measured — "expose new entities" is now off for the conversation assistant, which it was not before. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TgwAc8cnQQHo3VAPRjrfHM
I inverted the filter's polarity. In default_agent.async_handle_intents
a filter returning True means the result is NOT handled locally:
if not isinstance(result, RecognizeResult) or (
intent_filter is not None and intent_filter(result)
):
return None
_async_local_fallback_intent_filter returns True for GET_STATE and
MEDIA_SEARCH_AND_PLAY, so those two are the ones withheld from the
local path, and everything else the sentence matcher recognises is
answered without the model. Confirmed by measurement: 'Turn on the
kitchen lights' returns 'Turned on the light' in 0.00 s.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TgwAc8cnQQHo3VAPRjrfHM
The published `cuda` image never had a GPU path: `kokoro-onnx[gpu]` pulls both `onnxruntime` and `onnxruntime-gpu`, they install over each other under the same import name, and the CPU one wins. nixpkgs' onnxruntime, though, takes `cudaSupport`, and everything above it is small: kokoro-onnx is four Python files, and the Wyoming server is one more. So this drops the container and builds the stack. kokoro-onnx wants espeakng-loader and phonemizer-fork, which between them exist to ship an espeak-ng inside a wheel and to make phonemizer accept it. nixpkgs' phonemizer is already patched to use pkgs.espeak-ng and already carries the upstream `set_data_path` commit, so both are dropped, the same substitution python3Packages.misaki makes. The CUDA provider also gets `cudnn_conv_algo_search=HEURISTIC`. It defaults to EXHAUSTIVE, which benchmarks every convolution algorithm the first time it sees a shape; the model has 88 Conv nodes and a `sequence_length` input, so nearly every request would be a new shape and would pay the search again. Two behaviours the container was hiding: the server logs nothing below WARNING unless LOG_LEVEL says otherwise, so INFO is set and every request now prints its synthesis time; and its SIGTERM handler cancels the asyncio server without catching CancelledError, which exits 1 and would make every `systemctl stop` a failure. Verified on a machine with no GPU: the whole Python stack builds and serves Wyoming synthesis on the CPU (54 voices, correct IPA out of espeak-ng, real audio), the CUDA onnxruntime builds and reports CUDAExecutionProvider, and every op type in the model but the single STFT has a CUDA kernel registered. The speedup itself is not verified — that needs the card. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TgwAc8cnQQHo3VAPRjrfHM
ollama only reuses a cached sequence when the new prompt extends it, never on a shared prefix, so every fresh conversation re-prefills the whole system prompt: 0.65 s against 0.19 s when the prefix happens to be cached, on every single command. OLLAMA_NUM_PARALLEL=1 was my choice, on the assumption that a single warm slot was best; that assumed the wrong caching model. Two slots let the base prefix survive alongside an active conversation. Not more: slots multiply the KV allocation, and at the 32768 context this host requests, four would be ~35 GB of KV beside a 30 GB model. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TgwAc8cnQQHo3VAPRjrfHM
llama-server chooses a slot by LRU unless --slot-prompt-similarity is set. It defaults to 0.0, has no set_env binding, and ollama never passes it, so a fresh conversation takes the least recently used slot regardless of what is cached there. Measured with two slots: a three-turn conversation still evicted the primed prefix and the next command cost 0.63 s again. Since -c is num_ctx x parallel, the second slot was buying extra KV allocation for nothing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TgwAc8cnQQHo3VAPRjrfHM
Every fresh conversation re-ran the whole system prompt: 0.65 s against 0.19 s when the prefix was already cached, on every command. llama-server picks a slot by LRU, because --slot-prompt-similarity defaults to 0 and has no env binding. The LRU path then saves the slot, tries to load a better match, and calls prompt_clear() when that fails — discarding a prefix that ordinary common-prefix reuse would have kept. Enabling the similarity threshold would be the wrong fix anyway: it compares the common prefix against the *incoming prompt length*, so a long enough user message would skip a perfectly good cached prefix. The block is guarded by "update_cache && prompt_cache", and the cache is only constructed when cache_ram_mib != 0, so switching it off avoids the clearing entirely. llama-server inherits ollama's environment (cmd.Env = os.Environ()), so this needs no patch. The level-2 cache exists to restore evicted conversations when serving many at once; this host serves one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TgwAc8cnQQHo3VAPRjrfHM
Predicted that disabling llama-server's level-2 prompt cache would stop it clearing the slot and restore common-prefix reuse. Measured after switching: fresh conversations still cost 0.63 s, unchanged. The prompt_clear theory was wrong or incomplete. Recording what is ruled out so the next attempt does not repeat it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TgwAc8cnQQHo3VAPRjrfHM
llama-server reuses a cached sequence only when the new prompt extends it, so a finished conversation leaves [system][user][assistant] in the slot and the next conversation's [system][user'] diverges: the whole system prompt is recomputed. Measured 0.65 s of prefill on every command against 0.19 s when the slot holds just the prefix. The proxy replays each /api/chat with the conversation stripped, system message and tools only. That is a prefix of what is cached, so it costs about 0.10 s and runs after the response has already gone back to Home Assistant. Replaying the caller's own payload rather than rebuilding it means the bytes match by construction, including the tool schemas, and go on matching when the prompt changes — a hand-built copy would drift silently, with no runtime signal that it had stopped working. Measured through the proxy, streaming as Home Assistant does: 0.185 s, with chunks still arriving incrementally. ollama moves to 11435 on loopback; the primer takes 11434, so nothing that talks to it needs to change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TgwAc8cnQQHo3VAPRjrfHM
llama-server identifies the reusable prefix correctly — the log shows f_sim_best = 0.994 (1836/1848) — and then the context-checkpoint machinery discards it. Checkpoints cannot truncate at an arbitrary position, only roll back to a checkpoint, and the nearest one below the divergence point was at 825. So it re-prefills 1023 tokens. Two candidate environment variables are recorded but not yet tested. Also recorded: everything ruled out by measurement, the claims in earlier commit messages that turn out to be false, and the measurement traps that produced them — chiefly that a synthetic system prompt is not Home Assistant's, and that prompt_eval_count reports the full prompt even on a cache hit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_d93535ae-8d15-4da9-a216-a818cae9694a
llama-server was finding 1836 of 1848 tokens reusable and then throwing 1011 of them away, because a COMMON_CONTEXT_SEQ_RM_TYPE_FULL context cannot truncate at an arbitrary position and has to roll back to the nearest context checkpoint (824). LLAMA_ARG_CTX_CHECKPOINTS=0 removes the rollback path. ollama passes its environment through to llama-server, so no patch is needed. This is the fix the primer was working around, so the primer goes with it, and ollama moves back to 11434. Removing it is also what makes the result measurable: with the primer in the path the slot already holds the prefix, which is the state the fix is supposed to produce. Unverified as of this commit. Checkpoints are what let speculative decoding run on a FULL-only context, so this may cost MTP; the baseline to beat is 77 tok/s generation and the target is ~0.19 s prefill. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The previous commit was wrong. Setting -ctxcp to 0 does not fall back to plain longest-common-prefix reuse; on this model it removes reuse entirely, and every request re-prefills from zero — 1.8 s, even for a byte-exact append. qwen35 has full attention only every 4th layer. The other three quarters carry a recurrent state, which has no per-position structure to truncate, so the context reports COMMON_CONTEXT_SEQ_RM_TYPE_FULL and checkpoints are not an optimization over ordinary prefix reuse — they are the only prefix reuse there is. /api/show reports full_attention_interval = 4 in one call, which would have settled this much earlier. So the defect is the default spacing: checkpoint_min_step is 8192 tokens and the entire prompt is 1848, so no checkpoint is ever created near the end of the system prefix, which is where every new conversation diverges. At 128 the worst-case rollback is 128 tokens and the default 32 checkpoints still cover 4096 tokens of history. They live in host RAM, not VRAM — ~152 MiB each, against 128 GiB. Also disproved en route: --swa-full is irrelevant (no sliding window here), and the env-var channel to llama-server does work, which until now had only been assumed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Also corrects two things in the findings. Checkpoints are placed at user message boundaries, not on a grid; checkpoint_min_step only throttles that placement, so the fix un-suppresses the checkpoint at the end of the system prefix rather than laying down a denser lattice. And the claim that checkpoints cost generation speed is withdrawn: 92.8 tok/s with the fix, against 91 with checkpoints off. The 77 tok/s that comparison rested on was measured on a 28-token prompt. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Checkpoint placement was never really about spacing. llama-server has two placements: is_user_start, at the beginning of each message, and near_prompt_end as a fallback whose window is one n_ubatch. We were only ever getting the fallback, so a fresh conversation rolled back up to 512 tokens before the divergence point and re-prefilled them. is_user_start is directly controllable: message_delimiters is a field on the /completion request, and llama-server tokenizes those strings and scans the prompt for them. Give it the user delimiter and it checkpoints at the end of the system prompt and tool definitions, which is exactly where a new conversation diverges. ollama never sends the field, because it renders qwen's template in Go and posts a flat string. The patch adds an optional MessageDelimiters() method to the renderer interface, implements it for Qwen35Renderer, and threads it through llm.CompletionRequest. It applies cleanly to v0.32.13, which is what nixpkgs ships, and go build plus the renderers and llm tests pass. This should reach the ~185 ms floor and makes the num_batch trade unnecessary -- that only tuned the fallback's granularity, at 13% of bulk prefill throughput. Not taken. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
pymicro-vad uses microWakeWord's architecture -- a classifier over a sliding window. Feeding bursts of real speech: under ~400 ms never fires at all, 500 ms fires and holds 650 ms after the audio stops, 1200 ms holds 540 ms. That is a ~500 ms window, and it explains both edges measured earlier: onset ~340 ms late, release 340-610 ms. So this is not a bug or an oversight in training. A window-length latency is inherent and harmless for detecting a discrete wake word; it only hurts when the model is repurposed to decide when someone stopped speaking. It also means no threshold fixes it, since silence_seconds is added after the model reports silence. Hence reverting the patch that lowered it. The fix is a different VAD; audio_enhancer.py hardcodes MicroVad. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…arget Home Assistant retuned after switching to microVAD. Two weeks after the swap, "Adjust VAD seconds better for microVAD" moved every timing down -- 2.0 to 1.25, 0.5 to 0.25, 1.0 to 0.7 -- and three months later the speech threshold was split into separate before/during values. Both are someone clawing back latency the model added, not choosing how long a pause should feel. The ~1.2 s is the leftover, not the goal. The behaviour it produces is not a coherent tolerance either. Driving HA's own segmenter with real VAD output: pauses up to 850 ms are tolerated, pauses over 1100 ms end before the user resumes, but 900-1050 ms ends the command 30-180 ms AFTER they resumed speaking, because the model needs ~400 ms to notice speech starting as well. Also: do not assume Silero is the fix. HA shipped it in 2025-12 and reverted it a month later for lag, broken end-of-speech detection, crashes and macOS build failures. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Measured on the same clips: microVAD keeps reporting speech for 480-600 ms after it stops and needs 400-500 ms before reporting any, because it is a wake-word architecture classifying over a ~500 ms window. Silero releases in 0-96 ms and fires on 100 ms of speech. That is ~500 ms off every voice command. The patch is HA's own, from 079c6daa633, before 329b2c840d8 reverted it for lag, broken end-of-speech detection, crashes and macOS build failures -- problems for a project shipping to every platform, worth re-testing on one machine that builds its own software. Also documents the knobs. There is one timing, silence_seconds, exposed as VadSensitivity: relaxed 1.25, default 0.7, aggressive 0.25 -- a select entity per satellite, so a dropdown rather than a patch. The earlier patch that lowered it was reinventing that and is gone. For calibration: human turn-taking gaps average ~200 ms and production systems wait 300-800 ms. Our ~1.2 s is well outside that; microVAD's window alone rules the band out. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
nixpkgs runs Home Assistant's test suite during the build, which imports pysilero_vad, so services.home-assistant.extraPackages is too late -- that builds the service's runtime environment, not the package. Adding it to propagatedBuildInputs from home-assistant.passthru.python3Packages, which is the matching interpreter's package set, builds clean. Verified in the output: audio_enhancer.py imports SileroVoiceActivityDetector, pipeline.py constructs SileroVadSpeexEnhancer, and the cache-boundary patch is still applied. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The model now contributes essentially nothing. Onset is detected 50 ms late against 340 before, the end 20 ms late against 640, and end of speech to stt-end drops from 1301 ms to 833 ms. The remaining 704 ms is silence_seconds itself, so the timings finally mean what they say. Also makes silence_seconds settable on a pipeline run. The websocket API already accepted four of the five audio settings; this one was reachable only through the VAD sensitivity select entity that satellite integrations create, so it could not be tuned or tested without buying hardware. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
With Silero the knob is linear: 0.25 gives 376 ms to stt-end and tolerates a 250 ms pause, 0.7 gives 789 ms and 700 ms, 1.25 gives 1401 ms and 1300 ms. Which reframes the Silero win. Latency and tolerance are the same quantity for both VADs; what differs is the constant added to each. microVAD's ~500 ms window added to both, putting its floor near 750 ms of latency even at the aggressive setting. Silero adds ~50 ms, so the fast end of the curve becomes reachable. It is not a better tradeoff, it is access to more of it. Picking a point is therefore preference, not optimisation. Production systems sit at 300-800 ms, spanning 0.25 to 0.7 here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Saying Silero only lowers the reachable floor was wrong. The danger zone -- pauses where the command ends while the user is already speaking again -- is set by how long the VAD takes to notice speech restarting. microVAD needs ~400 ms, giving a band 100-200 ms wide at every setting tried (520-620 ms at 0.25, up to 880-1060 ms at 0.7). Silero notices in 30-100 ms and there is no such band at any setting. So it is two wins, not one: access to the fast end of the curve, and never committing to stop while the user is mid-word. Measurement note: "fired after the user resumed" is the wrong criterion, since firing at the end of the resumed speech is normal. It invented a 260 ms band for Silero that does not exist. The right criterion is fired during the resumed speech. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Home Assistant's conversation agent is set to qwen3.8:27b-mtp-q8_0, but loadModels declared only qwen3.6 models, so the one the assistant depends on was not pulled by a rebuild and a fresh machine would have broken. I probably caused that: runeval.py sets the model through the reconfigure flow, and killing the two-model eval partway would have left it on qwen3.8 without restoring it. Stopping a job that mutates configuration means checking what it changed. Also refreshes the stale comment. qwen3.8 no longer needs a newer ollama than we run, and "generation speed is not the binding constraint" is no longer true now that prefill is ~0.1 s. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The patches were inline in configuration.nix, which meant they could only
be used by the machine that file configures, and only via sudo on stable
nixpkgs. They are now packages/{ollama,home-assistant}.nix behind an
overlay, exposed as flake outputs for x86_64-linux, aarch64-linux and
aarch64-darwin.
That makes the same definitions usable from a Home Manager config, from
the agent VM, or on a Mac -- `nix run .#ollama-patched`. None of the
patches need a GPU to build, and ollama is not marked broken on
aarch64-darwin, so the Mac comparison needs no further packaging work.
The overlay must come last in nixpkgs.overlays: it patches prev.ollama-cuda,
and the overlay above replaces that with the unstable 0.32.13 the patches
were written against. Running first would patch stable's 0.32.3 and fail.
No derivation changed: Home Assistant still evaluates to
1582r9h0l5ajqf9nk1znw4i2ngsw35aq, the path already built, because Nix names
patch files by content and basename rather than by directory.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every voice experiment so far has needed a rebuild of the real machine, which meant a person sitting at a desk watching builds. Home Assistant is what we iterate on and it needs no GPU, so it can run here instead: same package as the host, pointed at the host's ollama over the bridge, with its own state in ~/ha-dev. Three scripts rebuild it from nothing -- run, onboard, setup -- so the dev instance cannot drift into a state nobody can reproduce. It answers "is the bed light on?" in ~450 ms against 433-458 ms on the real machine. Pinned to nixpkgs-stable to match the host, because the Silero patch does not apply to the 2026.8.2 in unstable. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The measurement scripts lived in scratch, so they were not available on any other machine -- including the Mac we want to compare against. This one takes OLLAMA_URL and a model name and reports the three numbers that matter: fresh-conversation prefill, follow-up prefill, and generation. Verified against the NixOS host: 587 ms fresh, 109 ms follow-up, 85.6 tok/s on qwen3.8:27b-mtp-q8_0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two flaws, both of which would have been read as hardware differences. The prompt carried no cache boundary marker, so on a server configured to look for one the delimiter matched nothing and the probe measured the fallback path: 587 ms fresh on the NixOS host against 97 ms once the marker is present. Two machines configured with different markers were being compared to each other. And the generation prompt produced 16-20 tokens, so tokens/sec was mostly startup noise -- 43.6 tok/s on one run against 95.0 tok/s when actually generating 300. It now asks for a longer sequence and says so loudly if the model stops early anyway. Host baseline, qwen3.8:27b-mtp-q8_0: 97 ms fresh, 98 ms follow-up, 95 tok/s. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Same model and machine at temperature 0: 97.3 tok/s counting from 1 to 120, 50.5 tok/s writing prose. Speculative decoding accepts more draft tokens when the continuation is predictable, so counting is near a best case. Which means the 85-95 tok/s quoted throughout the findings came from a counting prompt and overstates a real assistant reply by roughly two times. The probe now uses prose, which is both nearer the workload and comparable across machines. It also avoids a second trap: the counting prompt generated 300 tokens on CUDA and 17 on Metal, because the backends diverge numerically and one gave up early, leaving tokens/sec describing startup overhead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Same model, patched ollama, probe and settings on both. Prefix caching works identically on Metal -- a fresh conversation costs 453 ms where re-reading the prompt would cost 48 s -- so both patches carry over, which was the thing worth checking. The gap is prefill throughput: 131 tok/s against 1957. Every Mac figure follows from it, including the asymmetry where the Mac's follow-up is slower than its fresh conversation while the RTX has them equal, since the cost is proportional to tokens re-read. Generation on prose is 44-50 against 14.1 tok/s. I expected prose to narrow the gap, since MTP flatters the counting prompt; it did not, because both machines lose about half their rate on prose. Also labels the probe's first line honestly -- it is only a cold prefill when the slot is empty, and reports meaningless numbers otherwise. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Smart Turn v3 (pipecat-ai, BSD-2-Clause) is a Whisper Tiny encoder plus a linear head, 8M parameters, 8 MB of int8 ONNX, reading the waveform rather than a transcript so it needs no extra speech-to-text pass. Same shape LiveKit and Pipecat use: cheap VAD for speech and silence, turn model on top. It works on real speech. Sweeping cut points through the JFK sample scores clause and sentence ends at 0.86-0.90 and mid-clause at 0.03. Inference is ~30 ms on this VM's CPU. It cannot be tested with text-to-speech. Piper scored 0.93-0.99 whether the sentence was complete or truncated, because a synthesiser gives a fragment the falling intonation of a finished sentence -- it does not know the text is a fragment, and this model reads prosody. Validating it needs real recordings. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The model ships a labelled test set with synthetic and midfiller flags. Filtering to real non-synthetic English, 607 samples: 93.2% accuracy, against 92.63% published. That also validates the preprocessing port, since a wrong mel pipeline would read as chance. The threshold matters because the errors are not symmetric. At 0.5, 14.6% of unfinished utterances get cut off and 1.9% of finished ones wait; at 0.9, 6.1% and 8.7%. Being cut off mid-thought is the failure worth avoiding, and today every pause longer than silence_seconds cuts you off, so even the default is a large improvement. Appending silence to unfinished utterances barely moves the score -- median 0.033 at zero, 0.060 after two seconds -- so repeated checks will not converge on ending the turn by themselves, and the design needs an explicit cap for someone who simply trails off. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Pure numpy plus onnxruntime, so Home Assistant does not need transformers for this. Whisper's feature extraction is reimplemented here and verified against WhisperFeatureExtractor(chunk_length=8): features match to 0.000000 for audio above, below and exactly at the 8 second window, the mel filterbank matches to 1e-16, and end-to-end probabilities on real recordings are bit-identical to the reference implementation. The docstring carries the warning that cost an experiment to learn: this model must only be asked about audio that ends where a speaker paused. Fed audio cut mid-word, 60.6% of polls score above 0.9, because it judges the prosody up to the cut and cannot know the cut was arbitrary. It answers "does this sound finished", not "has this person stopped talking". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
On reaching silence_seconds the pipeline now asks Smart Turn whether the utterance sounds complete, and grants another silence window if not. Three settings on AudioSettings, all reachable over the websocket API so they can be tuned without satellite hardware: turn_detection, turn_threshold defaulting to 0.9, and turn_max_seconds defaulting to 3. The cap is not optional. The model's verdict does not drift as silence accumulates, so without it someone who trails off holds the turn open forever. Inference runs in the executor. It is ~25 ms of CPU, which would otherwise stall the event loop that is feeding it audio. Any failure is treated as "finished", which is exactly the behaviour without the model, so a broken model cannot lose a command. smart-turn.nix builds against the caller's interpreter, so Home Assistant gets the module built for its own Python, the same reason pysilero-vad is a package input rather than a service extraPackage. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Streaming a real recording through the dev instance, with no artificial pause -- the speaker's own pauses are the test: silence only, 0.25 s ended 2.5s ' And so my fellow Americans' silence only, 0.70 s ended 3.0s ' And so my fellow Americans!' turn detection, 0.25 s ended 5.2s ' And so my fellow Americans, ASK NOT!' The model recognised the pause after "Americans," as unfinished and kept listening, at the shorter threshold. It stops at 5.2 s because turn_max_seconds was 2.0 and this speaker pauses for effect, which is the cap working. Fixes a bug found by that test: process() calls reset() as it reports the command finished, clearing in_command, so granting another silence window must restore the flag as well as the counter. The silence counter only decrements inside a command, so without it the segmenter waits for speech that never comes and the turn never ends -- runs hung to the 60 s timeout while the model logged "keep listening" correctly each time. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sweeping silence_seconds with and without the turn model, measuring both things that matter. Without it every setting cuts a mid-sentence pause short, because tolerance is the threshold. With it the pause is held at every setting tried, down to 0.10. So silence_seconds can go 0.7 -> 0.1, which takes 600 ms off a finished utterance, while pauses keep working. The model costs about 100 ms of that back. Roughly 200 ms from end of speech to decision, against 789 ms for silence alone at 0.7. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Full pipeline through the real conversation agent, timed from the last sample of speech to the answer: 1408 ms with silence 0.7 and no turn model, 824 ms with silence 0.1 and the model. Same answer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
18 tool-call runs across qwen3.6:35b-a3b and ornith with no illegal memory access. That fault, during constrained decoding of tool calls with array or enum parameters, is what disqualified qwen35moe on ollama 0.32.3; we run 0.32.13, and it appears resolved. End to end with the turn model and silence 0.1: on a question answered from the prompt, 817 ms for qwen3.8 against 713-717 ms for the MoE models. On a control command needing a tool call, 1503 ms against 1116 ms for ornith. Not a recommendation yet. qwen3.6:35b-a3b said it was unable to turn on the light where ornith did it, so this needs the scenario eval, not a latency script. The light's state also carried between runs, which makes the replies non-comparable; reset entity state when comparing behaviour. Also factors the shared driving code into dev/halib.py, which hands out websocket ids centrally -- Home Assistant requires them to increase, and callers picking their own collided. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
dev/eval.py runs the 15 scenarios against the development instance, resetting entity state before every run so one scenario cannot flatter the next. Three reps each: qwen3.8:27b-mtp-q8_0 38/45 median 441 ms qwen3.6:35b-a3b-q4_K_M 39/45 median 466 ms ornith:35b-q4_K_M 45/45 median 678 ms ornith's higher median is partly an artefact of being correct: declining to act returns faster than acting. On real audio it is the fastest of the three, 713 ms against 817 for a question and 1116 against 1503 for a command. <|fim_pad|> is one special token in all three vocabularies, so the cache boundary carries over. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The settings were opt-in per API call, so a real satellite would have kept the old behaviour. turn_detection now defaults to on, silence_seconds to 0.25 against upstream's 0.7, and VadSensitivity to 0.7 / 0.25 / 0.1 against upstream's 1.25 / 0.25 / 0.7. Safe only because the model supplies the pause tolerance. Without it silence_seconds has to be both how fast the assistant answers and how long a pause it forgives, which is why upstream's numbers are long. With no per-run settings at all: 940 ms from end of speech to answer, against 1408 ms on the old defaults. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Per stage from the last sample of speech: 243 ms to decide the turn ended, 211 ms to transcribe, 383 ms to answer. The first two are both dead time and were consecutive for no reason. speculative_stt snapshots the audio the moment speech stops and transcribes that during the wait, so the text is ready when the turn is confirmed over. Speech resuming cancels the snapshot and the normal path is used. Costs one extra transcription of audio that is discarded when the speaker had not finished. 118 ms, bounded by how much wait there is to hide behind. Holding a pause still works with it on: the JFK clip still reaches "ASK NOT". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
From the last sample of speech to the answer, through the real conversation agent. The turn model and speculative transcription take it to 717 ms, and ornith to 600 ms. It also holds a mid-sentence pause now, which no setting could do before. The control-command column of that table is not trustworthy, because entity state carries between runs and a model that says "already on" looks faster than one that switches the light. Behaviour belongs in dev/eval.py, which resets state before every run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Ending the snapshot on the last phoneme, without the silence the full stream would carry, changed what whisper heard: of six clips two differed, one a real mishearing -- "set the bad light" for "set the bed light" -- and one only capitalisation. Appending 250 ms of silence makes all six identical. Found by comparing the speculative transcript against the normal one clip by clip rather than trusting that the same audio gives the same text. This is a property of the engine, so check it again if the engine changes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Silence with no speech, a 0.4 s blip, and an 11 s recording all behave the same with our settings as with stock ones, so none of this work regressed them. Silence and the blip run to the pipeline's own 60 s timeout in both configurations, which is upstream behaviour: no voice command ever starts, so the segmenter never reports one finishing. The long recording was reported as "no stt-end" when the transcript was plainly there. That was the harness: it measures from the moment the last sample is streamed, and a turn that ends early -- holding pauses, then hitting the cap -- never reaches that point. It now says so instead of implying the pipeline failed, and error events carry their reason through. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Puts the result and the one idea that matters at the top of the findings, ahead of the chronological detail, and records what is still open: real recordings of the person who will use it, speculating past the point where tool calls fire, streaming speech synthesis, and whether to switch the live assistant to ornith. Also lists the experiment scripts in the development README, along with the transcriber they need, and four things that each cost an hour: websocket ids must increase, truncating a log Home Assistant holds open fills it with nul bytes, subentry ids are not in the REST listing, and turn-detection verdicts are invisible without the logger block. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Nothing was spoken until the last token of a reply was generated, because Kokoro did not advertise supports_synthesize_streaming and Home Assistant will not stream text into an engine that does not. The hard part was already done: kokoro-wyoming already splits into sentences and emits audio for each. Only the input side was missing, so this adds SynthesizeStart / SynthesizeChunk / SynthesizeStop, buffers the incoming text, and synthesises each sentence as soon as it is complete. take_complete_sentences hands over only text up to the last sentence-ending punctuation, so nothing is spoken from a fragment more text would change. The model itself does not need to stream. Kokoro synthesises a whole utterance at once and quickly; it just has to be fed sooner. Tested with a stub synthesiser, because building the real one needs onnxruntime with CUDA: fed a reply the way a model produces it, the first sentence is synthesised after the second chunk while the third sentence is still arriving, and the event order is audio-start, chunks, audio-stop, synthesize-stopped. Unverified against the real synthesiser and against Home Assistant end to end. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.