standalone-indexer (h24 line): subscribe to every data-parallel rank of a discovered pod - #40
Merged
Merged
Conversation
…ered pod Pod discovery registered each engine pod once, as dp_rank 0 on tcp://<ip>:<zmq_port>. vLLM offsets the KV-event ZMQ port and the /kv_recover port by data_parallel_rank, so on a --data-parallel-size N engine ranks 1..N-1 publish to ports nobody subscribes to and everything they cache is invisible to /query while the engine still hits it. Measured on deepseek-ai/DeepSeek-V4.1-Flash (DP=2): every prefill the scheduler placed on rank 1 (about half) showed 0 in the standalone indexer and in rank 0's /kv_recover dump, and all its blocks in rank 1's dump; the KV ladder read actual 0.78 > h24 0.65 > perfect 0.63 > reality 0.59. Add --watch-dp-size (default 1, so single-rank engines are unchanged). The watcher registers one listener per rank under the pod's instance: dp_rank r, tcp://<ip>:<zmq_port + r>, http://<ip>:<recover_port + r>. register() rejects a rank that is already present, so a pod whose registration fails part-way is deregistered before the retry instead of being left half-subscribed forever. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit bb6dcfd)
github-actions
Bot
added
the
documentation
Author
|
Built |
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-pick of #39 (
bb6dcfd5c5) ontov41-partial-skip-h24, the branch the h24 indexer image is built from (it carries--h24, which thev41-partial-skipimage lacks, so the #39 image exits withunexpected argument '--h24'on the h24 service). Same change, same tests; see #39 for the analysis.🤖 Generated with Claude Code