Sonar

Sonar is an inference engine for Hugging Face-compatible language and multimodal models. It provides continuous batching, paged KV-cache management, optimized kernels, quantization, speculative decoding, and distributed serving.

Sonar is based on vLLM. It includes additional model and quantization formats, sampling methods, kernels, platforms, and deployment features. It serves production workloads for the Dolphin Inference Network and PygmalionAI.

Documentation

Read the documentation at sonar.dphn.ai.

• Installation
• Supported models
• Quantization support
• Server arguments
• Optimization
• Parallelism
• Production deployment

The model, quantization, and server-argument references are generated from the current source tree.

Install

The published NVIDIA wheel supports Linux x86-64, Python 3.10-3.13, and NVIDIA GPUs with compute capability 8.0 or newer.

Install uv, then create an environment:

uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install aphrodite-engine --torch-backend=cu130

Use the complete installation guide for AMD ROCm, Intel XPU, CPU, Apple silicon, Google TPU, Docker, WSL 2, source builds, and nightly wheels.

Serve a model

aphrodite serve Qwen/Qwen3-0.6B \
  --served-model-name qwen3

The server listens on http://127.0.0.1:2242 by default. It provides OpenAI-compatible APIs, health checks, metrics, and an OpenAPI schema.

See the OpenAI-compatible API guide for streaming, embeddings, tool calls, reasoning output, and Sonar request parameters.

Key features

• Continuous batching and paged KV-cache management
• Prefix caching enabled by default
• Tensor, pipeline, data, and expert parallelism
• Multi-node multiprocessing without a Ray cluster
• Prefill/decode disaggregation through NIXL and other KV connectors
• Quantized weights and FP8 KV cache
• Speculative decoding with MTP, EAGLE, DSpark, DFlash, n-gram, and other methods
• Structured output, reasoning parsers, and automatic tool calling
• Image, audio, and video model support
• LoRA adapter serving
• Prometheus metrics and health endpoints
• OpenAI, Anthropic, pooling, scoring, reranking, transcription, and Kobold APIs

Support depends on the model, device, data type, and quantization method. Check the generated model matrix and quantization matrix before deployment.

Read the optimization guide before you tune scheduler, cache, compilation, quantization, or speculative decoding settings.

Development

git clone https://github.com/dphnAI/sonar.git
cd sonar
uv venv --python 3.13 --seed --prompt sonar
source .venv/bin/activate
APHRODITE_USE_PRECOMPILED=1 \
  uv pip install --editable . --torch-backend=cu130