Conversation
vLLM (>= the DeepSeek-V4.1 day-0 image) hashes prefixes every prefix_match_unit (hash_block_size) tokens, which can be finer than the cache block size, and publishes the prompt tail that ends inside a cache block as a BlockStored whose block_size is the sub-block length (32/64/96 for a 128-token block). The standalone indexer treated any block_size mismatch on a main-attention event as a fatal --block-size misconfig and exited, so every V4.1 indexer crash-looped. Treat a proper divisor of the configured block size as a partial-prefix entry: drop it (rate-limited warn). Non-divisor mismatches stay fatal.
…ered pod Pod discovery registered each engine pod once, as dp_rank 0 on tcp://<ip>:<zmq_port>. vLLM offsets the KV-event ZMQ port and the /kv_recover port by data_parallel_rank, so on a --data-parallel-size N engine ranks 1..N-1 publish to ports nobody subscribes to and everything they cache is invisible to /query while the engine still hits it. Measured on deepseek-ai/DeepSeek-V4.1-Flash (DP=2): every prefill the scheduler placed on rank 1 (about half) showed 0 in the standalone indexer and in rank 0's /kv_recover dump, and all its blocks in rank 1's dump; the KV ladder read actual 0.78 > h24 0.65 > perfect 0.63 > reality 0.59. Add --watch-dp-size (default 1, so single-rank engines are unchanged). The watcher registers one listener per rank under the pod's instance: dp_rank r, tcp://<ip>:<zmq_port + r>, http://<ip>:<recover_port + r>. register() rejects a rank that is already present, so a pod whose registration fails part-way is deregistered before the retry instead of being left half-subscribed forever. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
standalone-indexer: subscribe to every data-parallel rank of a discovered pod
Shang-Pin
had a problem deploying
to
external_collaborator
September 17, 2026 21:15 — with 
GitHub Actions
Failure
GitHub Actions
Failure
github-actions
Bot
added
the
documentation
A listener that detects a gap fetches the worker's dump with a reqwest client whose total timeout was 10 s, and a failed fetch was final: the live batch that revealed the gap then advanced the watermark past it, so the missed range was lost for good. On DeepSeek-V4.1-Flash engines (DP=2, 28M-token KV pools) a TreeDump is ~70 MB serialized inside EngineCore. When three indexer flavors started against 39 such pods, each engine got six concurrent dump requests, a third of them ran past 10 s, the indexer logged "error decoding response body", the engine logged BrokenPipeError in kv_events.py do_GET, and the affected listeners carried orphaned chains (ParentBlockNotFound floods) from then on. - --recover-timeout-secs (default 120) replaces the hard-coded 10 s; connect timeout stays short at 5 s. - --recover-concurrency (default 8): a process-wide semaphore around the download, so a fleet-wide (re)subscription queues instead of stampeding every engine. - A failed download is retried up to 3 times with 2 s / 4 s backoff before the gap is given up; the final failure keeps the "kv_recover request failed" substring ops already grep for. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A listener blocked in /kv_recover (now up to 120 s with the gate) can let its SUB pipe hit the default RCVHWM of 1000. With heartbeats enabled libzmq 4.3.4 then aborts on `Assertion failed: _input_stopped` (zeromq/libzmq#3596, ai-dynamo#3937). Hit in prod on frank/DeepSeek-V4.1-Flash kv-indexer:h24: 6 restarts in 10 min while 66 startup TreeDumps were applied under the single H24Indexer mutex. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Two concurrent build-indexer-image.sh runs on different branches share the /cargo-target cache mount. git archive gives every file the commit mtime, so the second build saw a kv-router rlib the first build had just written, judged it fresh, and failed compiling the bindings against a struct from the other branch. Touch the tree after extraction. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
vLLM attaches each image's identifier to a stored block in extra_keys, and the shared ZMQ normalizer mixed it into the block's tokens hash. The standalone indexer's queriers hash plain token ids: deepapi's probe does, and so do the engine local-indexer TreeDumps served by /kv_recover. Every query therefore stopped matching at a conversation's first image block. Measured on frank/DeepSeek-V4.1-Flash with a live ZMQ capture: for chains whose first image fell in the window, a plain probe matched exactly up to the image block (217, 1098, 1569, 358 blocks) while an image-aware probe matched the whole chain (589, 2432, 3054, 6450). The indexer predicted 0.77 vs actual 0.96 on prompts over 500k tokens. Add ZmqEventNormalizer::with_plain_mm_hashing() and use it in the standalone listener. The library default is unchanged for Dynamo's own router, which queries with image info. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
standalone-indexer: bound, gate and retry /kv_recover downloads
Shang-Pin
had a problem deploying
to
external_collaborator
September 22, 2026 23:03 — with
GitHub Actions
Failure
GitHub Actions
Failure
standalone-indexer: hash image blocks from tokens only
This branch had an error being deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
vLLM (from the DeepSeek-V4.1-Flash day-0 image,
vllm/vllm-openai:deepseekv41-flash-0909) hashes prefixes everyprefix_match_unit(hash_block_size, 32 for V4.1) tokens, which is finer than the cache block size (128). It publishes the prompt tail that ends inside a cache block as aBlockStoredon the main MLA group whoseblock_sizeis the sub-block length (32/64/96).The standalone indexer treats any main-attention block-size mismatch as a fatal
--block-sizemisconfiguration and callsstd::process::exit(1), so all threekv-indexer:*flavors fordeepseek-ai/DeepSeek-V4.1-Flashcrash-looped (engine block size 32 != configured --block-size 128). No indexer--block-sizeworks: 128 dies on the partial events, 32 dies on the whole-block events.Observed event mix on a V4.1 pod (150 s tap): groups 0-3
sliding_window_mla@32 (already filtered as non-main), group 4mla_attention@128 (whole blocks) plus @32 partial entries.Fix
convert_event: whenblock_size < kv_block_sizetreat the event as a partial-prefix entry and drop it (rate-limited warn) instead of erroring. A larger event block size stays fatal (that can only be a misconfig). Whole-block routing information is unaffected; only sub-block tail entries are ignored.Unit test added;
cargo test --lib zmq_wire::tests= 19 passed.Same commit cherry-picked onto
24h-indexerasv41-partial-skip-h24for the h24 image.Refs DEE-640.