Skip to content

feat(config, llm): Add a vllm provider - #1174

Open
rxgrant wants to merge 2 commits into
mainfrom
rgrant-dev-vllm
Open

rxgrant wants to merge 2 commits into
mainfrom
rgrant-dev-vllm

Conversation

@rxgrant

@rxgrant rxgrant commented Sep 16, 2026

Copy link
Copy Markdown
Collaborator

For vLLM compatibility, the openai provider has the Bearer token auth, but it speaks a Responses dialect that vLLM does not serve. The llamacpp provider speaks the Chat Completions dialect that vLLM serves, but it sends no auth and it strips the vendor prefix from the model id. We need a new provider for vLLM.

Users can now point jp at a self-hosted vLLM server by setting a model's provider to vllm. vLLM speaks the same OpenAI-compatible /v1/chat/completions dialect as llama.cpp, so it supports streaming chat, tool calls, structured output, and thinking control the same way. Model listings come from GET /v1/models, and the reported max_model_len becomes the model's context window; the model name keeps its full id (e.g. Qwen/Qwen3-8B) since vLLM only accepts that form in requests.

providers.llm.vllm configures the new provider, with api_key_env defaulting to VLLM_API_KEY and base_url defaulting to http://127.0.0.1:8000:

[providers.llm.vllm]
api_key_env = "VLLM_API_KEY"
base_url = "http://127.0.0.1:8000"

To support this without duplicating llama.cpp's SSE parsing and event conversion, the shared OpenAI-compatible message/tool conversion and stream assembly logic moves from provider/llamacpp.rs into provider/openai_compat.rs, where both providers now call it. This is behavior-preserving for llama.cpp.

Closes: #0khs4b1

For vLLM compatibility, the openai provider has the Bearer token
auth, but it speaks a Responses dialect that vLLM does not serve.
The llamacpp provider speaks the Chat Completions dialect that vLLM
serves, but it sends no auth and it strips the vendor prefix from
the model id.  We need a new provider for vLLM.

Users can now point `jp` at a self-hosted vLLM server by setting a
model's provider to `vllm`. vLLM speaks the same OpenAI-compatible
`/v1/chat/completions` dialect as llama.cpp, so it supports streaming
chat, tool calls, structured output, and thinking control the same
way. Model listings come from `GET /v1/models`, and the reported
`max_model_len` becomes the model's context window; the model name
keeps its full id (e.g. `Qwen/Qwen3-8B`) since vLLM only accepts that
form in requests.

`providers.llm.vllm` configures the new provider, with `api_key_env`
defaulting to `VLLM_API_KEY` and `base_url` defaulting to
`http://127.0.0.1:8000`:

```toml
[providers.llm.vllm]
api_key_env = "VLLM_API_KEY"
base_url = "http://127.0.0.1:8000"
```

To support this without duplicating llama.cpp's SSE parsing and event
conversion, the shared OpenAI-compatible message/tool conversion and
stream assembly logic moves from `provider/llamacpp.rs` into
`provider/openai_compat.rs`, where both providers now call it. This is
behavior-preserving for llama.cpp.

Closes: #0khs4b1
Signed-off-by: rgrant <rgrant@contract.design>
vLLM renders chat requests through the served model's own chat
template, and several templates reject a system message that isn't
the first message in the list. `create_request` now joins the system
prompt, prompt sections, and attachment XML into a single leading
`system` message instead of emitting them as separate messages via
`to_system_messages`, keeping every served model reachable regardless
of its template's constraints.

Adds a regression test,
`create_request_joins_system_parts_into_one_message`, covering the
merged-message behavior, and updates the recorded provider snapshot
fixtures to match the new request shape.

Signed-off-by: rgrant <rgrant@contract.design>
@rxgrant

rxgrant commented Sep 16, 2026

Copy link
Copy Markdown
Collaborator Author

rebased

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant