Skip to content
KeyLiaoHPC
sglang
Public
forked from sgl-project/sglang

About

SGLang is a high-performance serving framework for large language models and multimodal models.

sglang.io

Resources

Readme
Apache-2.0 license
Activity

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository
BranchesTags
 
 

Latest commit

 

History

13,492 Commits

Folders and files

Repository files navigation

Notes on 2026-09-01 by KeyLiaoHPC

Qwen3.8-Flash-Next FP8 on 8xV100-SXM3 (one NVLink island, TP8+EP8, EAGLE 3/1/3, CUDA graphs 1/2/4). Max concurrency at 4; do not add graph sizes above 4.

Live v2rc4 on GPU 0–7 / port 31245 (TurboMind grouped FP8 MoE, QSA MTP index-share off) is the working bar for optimization. Measured performance:

  • Prefill, uncached 2048-token chunks: median 3126 tok/s (about 3050–3250).
  • Decode conc=1 at about 30k context: median 60 tok/s, accept_len 1.70.
  • Decode conc=1 at about 63k–69k context: median 50 tok/s, accept_len 1.56.

Due to the amateur model architecture, unevenly distributed MoE experts and insufficient-trained MTP layers, the decoding speed could change around 30-60 depending on the actual payloads. The accept rates will often higher in coding tasks and lower in writing tasks.

Specifications:

  • Longest passing chat context: 208k tokens (207944 measured; KV pool 213760).
  • Tool-call parser on; 20-argument calls and 8k/32k tool turns passed. --enable-multimodal is in the command; no separate image-accuracy suite was run.
  • Well functional in my daily coding and analysis work with OpenCode, OpenWebUI, pi, zoo and the Hermes agent.

Let your agent install TurboMind and this repo. Startup is the flat script scripts/sglang_serve_v100_qwen38_flash_next_fp8_dgx2_v2rc4.sh. Change the paths for your machine. Stop with fuser -k 8000/tcp only (never pkill -f). There is also a script for BF16 weight ONLY FOR EXPERIMENT. The variability and tailed latency in allreduce leads to heavy performance degradation when using 16 V100s for serving a single model.

On in this recipe: TurboMind grouped FP8 MoE (SGLANG_SM70_FP8_MOE=1, backend auto), QSA FA-V100, SPEC_V2, Custom AR 1stage + NCCL_P2P_LEVEL=NVL, attn-TP n-gram, MTP QSA index-share off, compiled HC RMSNorm (SGLANG_SM70_COMPILE_HC_NORM=1), --enable-multimodal, EAGLE 3/1/3 (draft tokens 3), CUDA graphs 1/2/4, PLE host offload, --context-length 208000. Left off (measured, do not flip expecting a decode win): PLE prefetch, Triton grouped RMSNorm, HC GEMM autotune.

export CUDA_HOME=/path/to/cuda
export HPCX_ENV_SCRIPT=/path/to/hpcx/init/script
export SGLANG_VENV=/path/to/sglang-venv
export MODEL_PATH=/path/to/Qwen3.8-Flash-Next-FP8
export CHAT_TEMPLATE=/path/to/chat_template.jinja # I suggest, and have embbeded froggeric's V22.4 customize template for Qwen series. Feel free to use any other templates.
export CUDA_VISIBLE_DEVICES="0,1,2,3,4,5,6,7"

export NCCL_P2P_LEVEL=NVL
export SGLANG_SM70_FP8_MOE=1
export SGLANG_ENABLE_SPEC_V2=1
export SGLANG_QSA_SM70_FA_V100=1
export SGLANG_DISABLE_QSA_MTP_INDEX_SHARE=1
export SGLANG_USE_ATTN_TP_NGRAM=1

bash scripts/sglang_serve_v100_qwen38_flash_next_fp8_dgx2_v2rc4.sh

Notes on 2026-08-19 by KeyLiaoHPC

Forked from haohervchb/sglang-V100. I added DFlash2 support for Qwen3.8-27B, taking Z-Lab's Qwen3.8-27B-DFlash as the draft model. More optimization and verifications are on progress.

With BF16 weights and 4xV100 SXM3 on DGX-2 and HPCX-2.25.1/CUDA-12.8, est. 3000t/s prefill and 50 to 100t/s (avg. 70t/s) generation for a single request at around 50K tokens. I suggest starting from 2 parallel requests at 262144 context.

Please change the content according to your own environment before testing, and take it AS-IS. See original README for installation and more info.

export CUDA_HOME=/path/to/cuda
source /path/to/hpcx/init/script
source /path/to/sglang-V100/bin/activate
export CUDA_VISIBLE_DEVICES="0,1,2,3"

export NCCL_P2P_LEVEL=NVL
export SGLANG_CUSTOM_ALLREDUCE_ALGO=1stage
export SGLANG_MAMBA_CONV_DTYPE=float16
export SGLANG_MAMBA_SSM_DTYPE=float16
export SGLANG_ENABLE_SPEC_V2=1
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export FLASHINFER_DISABLE_VERSION_CHECK=1

set -Eeuo pipefail

exec sglang serve \
  --model /path/to/Qwen3.8-27B \
  --speculative-draft-model-path /path/to/Qwen3.8-27B-DFlash2 \
  --trust-remote-code \
  --mamba-scheduler-strategy extra_buffer \
  --mamba-full-memory-ratio 0.1 \
  --dtype float16 \
  --attention-backend flash_attn_v100 \
  --tensor-parallel-size 4 \
  --host 0.0.0.0 \
  --port 8000 \
  --mem-fraction-static 0.75 \
  --context-length 262144 \
  --max-running-requests 4 \
  --max-queued-requests 32 \
  --chunked-prefill-size 4096 \
  --cuda-graph-max-bs 4 \
  --cuda-graph-bs 1 2 4 \
  --enable-multimodal \
  --enable-nccl-nvls \
  --served-model-name "Qwen3.8-27B" \
  --chat-template /i/suggest/using/froggeric/chat_template_froggeric_v22.jinja \
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder \
  --speculative-algorithm DFLASH \
  --speculative-dflash-block-size 8 \
  --speculative-draft-model-quantization unquant

===========================================

Original README.md

SGLang for 4x NVIDIA V100 32 GB NVLink

Current V100 performance

All currently documented model checkpoints are listed below. LLM results use TP4, one cold request, and 256 greedy output tokens. Prefill is the exact input length divided by client time to first token; decode excludes that first-token time. The measurements came from separate tuning runs, so treat this as a practical reference rather than a perfectly controlled cross-model leaderboard. H3 reports wall-clock video generation time in the Results column rather than LLM token throughput. — means the metric does not apply or the supported configuration has no comparable retained end-to-end benchmark.

Model checkpoint Measured configuration 1K prefill 1K decode 25K prefill 25K decode Results
MiniMaxAI/MiniMax-H3 TP4 W4A16, 960×544, 15 s clip, 10 steps — — — — ~500 s/video
Qwen/Qwen3.8-27B-FP8 DSpark-7, FP16 KV 2,749 tok/s 107.8 tok/s 3,140 tok/s 78.3 tok/s 13-point TP2/TP4 sweep
Qwen/Qwen3.6-27B-FP8 DFlash-16, FP16 KV 2,774 tok/s 154.0 tok/s 3,128 tok/s 126.2 tok/s 13-point TP2/TP4 sweep
Qwen/Qwen3.6-27B FP16, DFlash-16 3,261 tok/s 101.2 tok/s 3,631 tok/s 86.6 tok/s Audited context sweep
Qwen/Qwen3.6-35B-A3B FP16, DFlash-16 4,240 tok/s 150.1 tok/s 12,258 tok/s 136.4 tok/s 35B optimization results
QuantTrio/Qwen3.6-35B-A3B-AWQ AWQ target/DFlash — — — — Supported; comparable run not retained
Qwen/Qwen3.5-122B-A10B-GPTQ-Int4 GPTQ-Marlin, DFlash-16 3,426 tok/s 109.2 tok/s 4,718 tok/s 81.9 tok/s Audited context sweep
QuantTrio/Qwen3.5-122B-A10B-AWQ AWQ-Marlin target only — — — — Supported; comparable run not retained
poolside/Laguna-S-2.1-INT4 Marlin, DFlash-8 3,334 tok/s† 77.3 tok/s 4,327 tok/s† 67.0 tok/s Context sweep and Laguna tuning

Target-only and MTP modes are also supported where commands are provided below. †Laguna prefill comes from the retained DFlash sweep; its later Marlin selector changed low-token-count decode and left effective prefill unchanged within normal cold-run variation. The Laguna decode columns are the later tuned block-8 results.

Install on the host

if [[ -d "$HOME/sglang-V100/.git" ]]; then
  git -C "$HOME/sglang-V100" pull --ff-only
else
  git clone https://github.com/haohervchb/sglang-V100.git "$HOME/sglang-V100"
fi

bash "$HOME/sglang-V100/scripts/install_v100.sh"
conda activate sglang-v100

Validate an existing installation:

conda activate sglang-v100
bash "$HOME/sglang-V100/scripts/smoke_v100.sh"

Docker

Pull the published image:

docker pull geesegeesegeese/sglang-v100:latest

Build the current checkout:

cd "$HOME/sglang-V100"
DOCKER_BUILDKIT=1 docker build --network=host \
  -f docker/v100.Dockerfile \
  -t sglang-v100:latest .

Create the shared model and JIT caches used by the Docker examples:

mkdir -p "$HOME/.cache/huggingface"
docker volume create sglang-v100-jit

MiniMax-H3 video and audio

Serve H3 on the host

Recommended W4A16 command:

conda activate sglang-v100

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
NCCL_P2P_LEVEL=NVL \
sglang serve \
  --model-path MiniMaxAI/MiniMax-H3 \
  --model-variant fl2va \
  --num-gpus 4 \
  --tp-size 4 \
  --sp-degree 1 \
  --ulysses-degree 1 \
  --ring-degree 1 \
  --performance-mode speed \
  --quantization v100_w4a16_awq \
  --attention-backend tilelang_fa_v100 \
  --dit-cpu-offload false \
  --text-encoder-cpu-offload \
  --vae-cpu-offload \
  --enable-torch-compile false \
  --warmup false \
  --server-warmup false \
  --host 0.0.0.0 \
  --port 30010

W8A16 with DiT offload:

conda activate sglang-v100

NCCL_P2P_LEVEL=NVL \
sglang serve \
  --model-path MiniMaxAI/MiniMax-H3 \
  --model-variant fl2va \
  --num-gpus 4 \
  --tp-size 4 \
  --sp-degree 1 \
  --ulysses-degree 1 \
  --ring-degree 1 \
  --performance-mode speed \
  --quantization v100_w8a16 \
  --attention-backend tilelang_fa_v100 \
  --dit-cpu-offload \
  --text-encoder-cpu-offload \
  --vae-cpu-offload \
  --enable-torch-compile false \
  --host 0.0.0.0 \
  --port 30010

Serve H3 from Docker

mkdir -p "$HOME/.cache/huggingface"

docker run --rm --gpus all --network host --ipc host \
  --ulimit memlock=-1 --ulimit stack=67108864 \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  -v sglang-v100-jit:/root/sglang-v100-jit \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  geesegeesegeese/sglang-v100:latest \
  --model-path MiniMaxAI/MiniMax-H3 \
  --model-variant fl2va \
  --num-gpus 4 \
  --tp-size 4 \
  --sp-degree 1 \
  --ulysses-degree 1 \
  --ring-degree 1 \
  --performance-mode speed \
  --quantization v100_w4a16_awq \
  --attention-backend tilelang_fa_v100 \
  --dit-cpu-offload false \
  --text-encoder-cpu-offload \
  --vae-cpu-offload \
  --enable-torch-compile false \
  --warmup false \
  --server-warmup false \
  --host 0.0.0.0 \
  --port 30010

Generate a text-to-video-and-audio clip

This produces 960x544 output:

video_id=$(
  curl -sS -X POST http://127.0.0.1:30010/v1/videos \
    -H 'Content-Type: application/json' \
    -d '{
      "model": "MiniMaxAI/MiniMax-H3",
      "prompt": "A tiger walks slowly through morning fog while birds and leaves are heard around it.",
      "task": "t2va",
      "conditions": [],
      "target": {
        "short_edge": 544,
        "aspect_ratio": "16:9",
        "duration_seconds": 5.0
      },
      "num_inference_steps": 50,
      "flow_shift": 12.0,
      "audio_flow_shift": 3.0,
      "seed": 1101
    }' | jq -r '.id'
)

while true; do
  status=$(curl -sS "http://127.0.0.1:30010/v1/videos/$video_id" | jq -r '.status')
  [ "$status" = completed ] && break
  [ "$status" = failed ] && exit 1
  sleep 2
done

curl -sS -L "http://127.0.0.1:30010/v1/videos/$video_id/content" \
  -o minimax-h3-t2va.mp4

Generate from a first frame

Replace /absolute/path/first-frame.png with a file visible to every server rank:

video_id=$(
  curl -sS -X POST http://127.0.0.1:30010/v1/videos \
    -H 'Content-Type: application/json' \
    -d '{
      "model": "MiniMaxAI/MiniMax-H3",
      "prompt": "Continue this scene with calm natural motion and synchronized ambient sound.",
      "task": "fl2va",
      "conditions": [{
        "type": "image",
        "uri": "file:///absolute/path/first-frame.png",
        "role": "keyframe",
        "frame_index": 0
      }],
      "target": {
        "short_edge": 544,
        "aspect_ratio": "auto",
        "duration_seconds": 5.0
      },
      "num_inference_steps": 50,
      "flow_shift": 12.0,
      "audio_flow_shift": 3.0,
      "seed": 2101
    }' | jq -r '.id'
)

while true; do
  status=$(curl -sS "http://127.0.0.1:30010/v1/videos/$video_id" | jq -r '.status')
  [ "$status" = completed ] && break
  [ "$status" = failed ] && exit 1
  sleep 2
done

curl -sS -L "http://127.0.0.1:30010/v1/videos/$video_id/content" \
  -o minimax-h3-fl2va.mp4

Use "frame_index": -1 for a last frame. Launch with --model-variant ref2va for reference-image, reference-audio, reference-video, or video-to-video jobs.

LLM serving examples

Run conda activate sglang-v100 first. These commands use all four V100s and listen on port 8082.

Qwen3.8-27B-FP8 with DSpark

FLASHINFER_DISABLE_VERSION_CHECK=1 \
NCCL_P2P_LEVEL=NVL \
SGLANG_CUSTOM_ALLREDUCE_ALGO=1stage \
SGLANG_MAMBA_CONV_DTYPE=float16 \
SGLANG_MAMBA_SSM_DTYPE=float16 \
SGLANG_ENABLE_SPEC_V2=1 \
SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 \
sglang serve \
  --trust-remote-code \
  --model-path Qwen/Qwen3.8-27B-FP8 \
  --dtype float16 \
  --kv-cache-dtype fp8_e4m3 \
  --attention-backend flash_attn_v100 \
  --tensor-parallel-size 4 \
  --host 0.0.0.0 \
  --port 8082 \
  --mem-fraction-static 0.75 \
  --context-length 262144 \
  --max-running-requests 1 \
  --chunked-prefill-size 4096 \
  --mamba-full-memory-ratio 0.1 \
  --mamba-scheduler-strategy extra_buffer \
  --cuda-graph-max-bs 1 \
  --cuda-graph-bs 1 \
  --enable-nccl-nvls \
  --speculative-algorithm DSPARK \
  --speculative-draft-model-path RadixArk/Qwen3.8-27B-DSpark \
  --speculative-dspark-block-size 7 \
  --speculative-draft-model-quantization unquant \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder

Use --kv-cache-dtype auto for the faster FP16 KV cache when its lower capacity is acceptable.

Measured on V100-SXM2-32GB with FP8 weights, FP16 KV, DSpark block 7, one cold-cache request, and 256 greedy output tokens:

Input TP2 prefill TP4 prefill TP2 decode TP4 decode
1K 1,761 tok/s 2,749 tok/s 76.5 tok/s 107.8 tok/s
9K 1,888 tok/s 3,355 tok/s 60.7 tok/s 84.7 tok/s
17K 1,778 tok/s 3,278 tok/s 56.9 tok/s 86.7 tok/s
25K 1,686 tok/s 3,140 tok/s 51.9 tok/s 78.3 tok/s

Across the complete 1K-to-25K sweep, TP4 is 1.81x faster for prefill and 1.44x faster for decode by geometric mean. See the full 13-point benchmark.

Qwen3.8-27B BF16 with DFlash2

Same --speculative-algorithm DFLASH flag as Qwen3.6. The z-lab/Qwen3.8-27B-DFlash2 checkpoint sets architectures=["DFlash2DraftModel"] plus grouped-conv and selector fields, so the worker switches itself. Keep Qwen3.6 DFlash1 on GPUs 0-3 if it is already serving.

The deploy helper is scripts/sglang-serve-Qwen3.8-27B-DFlash2.sh (a second GPU island). A condensed command:

FLASHINFER_DISABLE_VERSION_CHECK=1 \
NCCL_P2P_LEVEL=NVL \
SGLANG_CUSTOM_ALLREDUCE_ALGO=1stage \
SGLANG_MAMBA_CONV_DTYPE=float16 \
SGLANG_MAMBA_SSM_DTYPE=float16 \
SGLANG_ENABLE_SPEC_V2=1 \
SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 \
CUDA_VISIBLE_DEVICES=8,9,10,11 \
sglang serve \
  --trust-remote-code \
  --model-path /path/to/Qwen3.8-27B \
  --dtype float16 \
  --attention-backend flash_attn_v100 \
  --tensor-parallel-size 4 \
  --host 0.0.0.0 \
  --port 8000 \
  --mem-fraction-static 0.70 \
  --context-length 8192 \
  --max-running-requests 1 \
  --chunked-prefill-size 4096 \
  --mamba-full-memory-ratio 0.1 \
  --mamba-scheduler-strategy extra_buffer \
  --cuda-graph-max-bs 1 \
  --cuda-graph-bs 1 \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path /path/to/Qwen3.8-27B-DFlash2 \
  --speculative-dflash-block-size 8 \
  --speculative-draft-model-quantization unquant \
  --reasoning-parser qwen3

Qwen3.6-27B-FP8 with DFlash

NCCL_P2P_LEVEL=NVL \
SGLANG_CUSTOM_ALLREDUCE_ALGO=1stage \
SGLANG_MAMBA_CONV_DTYPE=float16 \
SGLANG_MAMBA_SSM_DTYPE=float16 \
SGLANG_ENABLE_SPEC_V2=1 \
SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 \
sglang serve \
  --model Qwen/Qwen3.6-27B-FP8 \
  --dtype float16 \
  --kv-cache-dtype fp8_e4m3 \
  --attention-backend flash_attn_v100 \
  --tensor-parallel-size 4 \
  --host 0.0.0.0 \
  --port 8082 \
  --mem-fraction-static 0.75 \
  --context-length 262144 \
  --max-running-requests 1 \
  --chunked-prefill-size 4096 \
  --mamba-full-memory-ratio 0.1 \
  --mamba-scheduler-strategy extra_buffer \
  --cuda-graph-max-bs 1 \
  --cuda-graph-bs 1 \
  --enable-nccl-nvls \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path z-lab/Qwen3.6-27B-DFlash \
  --speculative-dflash-block-size 16 \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder

Use --kv-cache-dtype auto for FP16 KV cache.

Qwen3.6-35B-A3B FP16 with DFlash

NCCL_P2P_LEVEL=NVL \
SGLANG_MAMBA_CONV_DTYPE=float16 \
SGLANG_MAMBA_SSM_DTYPE=float16 \
SGLANG_ENABLE_SPEC_V2=1 \
SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 \
sglang serve \
  --model Qwen/Qwen3.6-35B-A3B \
  --dtype float16 \
  --attention-backend flash_attn_v100 \
  --tensor-parallel-size 4 \
  --host 0.0.0.0 \
  --port 8082 \
  --mem-fraction-static 0.70 \
  --context-length 262144 \
  --max-running-requests 4 \
  --chunked-prefill-size 8192 \
  --mamba-scheduler-strategy extra_buffer \
  --cuda-graph-max-bs 4 \
  --cuda-graph-bs 1 2 4 \
  --enable-nccl-nvls \
  --enable-multimodal \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path z-lab/Qwen3.6-35B-A3B-DFlash \
  --speculative-dflash-block-size 16 \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder

Qwen3.5-122B-A10B GPTQ-Int4 with DFlash

NCCL_P2P_LEVEL=NVL \
SGLANG_CUSTOM_ALLREDUCE_ALGO=1stage \
SGLANG_MAMBA_CONV_DTYPE=float16 \
SGLANG_MAMBA_SSM_DTYPE=float16 \
SGLANG_ENABLE_SPEC_V2=1 \
SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 \
sglang serve \
  --model Qwen/Qwen3.5-122B-A10B-GPTQ-Int4 \
  --dtype float16 \
  --quantization gptq_marlin \
  --attention-backend flash_attn_v100 \
  --tensor-parallel-size 4 \
  --host 0.0.0.0 \
  --port 8082 \
  --mem-fraction-static 0.78 \
  --context-length 262144 \
  --max-running-requests 4 \
  --chunked-prefill-size 4096 \
  --mamba-scheduler-strategy extra_buffer \
  --cuda-graph-max-bs 4 \
  --cuda-graph-bs 1 2 4 \
  --enable-nccl-nvls \
  --enable-multimodal \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path z-lab/Qwen3.5-122B-A10B-DFlash \
  --speculative-dflash-block-size 16 \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder

Poolside Laguna-S-2.1 INT4 with DFlash

NCCL_P2P_LEVEL=NVL \
SGLANG_CUSTOM_ALLREDUCE_ALGO=1stage \
SGLANG_ENABLE_SPEC_V2=1 \
SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 \
sglang serve \
  --model poolside/Laguna-S-2.1-INT4 \
  --trust-remote-code \
  --dtype float16 \
  --kv-cache-dtype auto \
  --attention-backend flash_attn_v100 \
  --moe-runner-backend marlin \
  --tensor-parallel-size 4 \
  --host 0.0.0.0 \
  --port 8082 \
  --mem-fraction-static 0.76 \
  --swa-full-tokens-ratio 0.08 \
  --context-length 262144 \
  --page-size 16 \
  --max-running-requests 4 \
  --chunked-prefill-size 4096 \
  --triton-attention-num-kv-splits 128 \
  --cuda-graph-max-bs 4 \
  --cuda-graph-bs 1 2 4 \
  --enable-nccl-nvls \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path poolside/Laguna-S-2.1-DFlash-INT4 \
  --speculative-dflash-block-size 8 \
  --reasoning-parser poolside_v1 \
  --tool-call-parser poolside_v1

For target-only serving, remove the three --speculative-* arguments. SGLANG_ENABLE_SPEC_V2 can also be omitted.

Docker: Qwen3.6-27B-FP8 with DFlash

mkdir -p "$HOME/.cache/huggingface"

docker run --rm --gpus all --network host --ipc host \
  --ulimit memlock=-1 --ulimit stack=67108864 \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  -v sglang-v100-jit:/root/sglang-v100-jit \
  -e SGLANG_CUSTOM_ALLREDUCE_ALGO=1stage \
  -e SGLANG_ENABLE_SPEC_V2=1 \
  -e SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 \
  geesegeesegeese/sglang-v100:latest \
  --model Qwen/Qwen3.6-27B-FP8 \
  --dtype float16 \
  --kv-cache-dtype fp8_e4m3 \
  --attention-backend flash_attn_v100 \
  --tensor-parallel-size 4 \
  --host 0.0.0.0 \
  --port 8082 \
  --mem-fraction-static 0.75 \
  --context-length 262144 \
  --max-running-requests 1 \
  --chunked-prefill-size 4096 \
  --mamba-full-memory-ratio 0.1 \
  --mamba-scheduler-strategy extra_buffer \
  --cuda-graph-max-bs 1 \
  --cuda-graph-bs 1 \
  --enable-nccl-nvls \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path z-lab/Qwen3.6-27B-DFlash \
  --speculative-dflash-block-size 16 \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder

OpenAI-compatible chat request

curl -sS http://127.0.0.1:8082/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "default",
    "messages": [{"role": "user", "content": "Write a short hello-world program."}],
    "temperature": 0,
    "max_tokens": 256
  }' | jq

Additional example serve commands

Qwen3.5-122B-A10B GPTQ-Int4 target only

NCCL_P2P_LEVEL=NVL \
SGLANG_CUSTOM_ALLREDUCE_ALGO=1stage \
SGLANG_MAMBA_CONV_DTYPE=float16 \
SGLANG_MAMBA_SSM_DTYPE=float16 \
sglang serve \
  --model Qwen/Qwen3.5-122B-A10B-GPTQ-Int4 \
  --dtype float16 \
  --quantization gptq_marlin \
  --attention-backend flash_attn_v100 \
  --tensor-parallel-size 4 \
  --host 0.0.0.0 \
  --port 8082 \
  --mem-fraction-static 0.78 \
  --context-length 262144 \
  --max-running-requests 4 \
  --chunked-prefill-size 16384 \
  --cuda-graph-max-bs 4 \
  --cuda-graph-bs 1 2 4 \
  --enable-nccl-nvls

Qwen3.5-122B-A10B AWQ target only

NCCL_P2P_LEVEL=NVL \
SGLANG_CUSTOM_ALLREDUCE_ALGO=1stage \
SGLANG_MAMBA_CONV_DTYPE=float16 \
SGLANG_MAMBA_SSM_DTYPE=float16 \
sglang serve \
  --model QuantTrio/Qwen3.5-122B-A10B-AWQ \
  --dtype float16 \
  --quantization awq_marlin \
  --attention-backend flash_attn_v100 \
  --tensor-parallel-size 4 \
  --host 0.0.0.0 \
  --port 8082 \
  --mem-fraction-static 0.78 \
  --context-length 262144 \
  --max-running-requests 4 \
  --chunked-prefill-size 16384 \
  --cuda-graph-max-bs 4 \
  --cuda-graph-bs 1 2 4 \
  --enable-nccl-nvls

Qwen3.6-35B-A3B FP16 target only

NCCL_P2P_LEVEL=NVL \
SGLANG_MAMBA_CONV_DTYPE=float16 \
SGLANG_MAMBA_SSM_DTYPE=float16 \
SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 \
sglang serve \
  --model Qwen/Qwen3.6-35B-A3B \
  --dtype float16 \
  --attention-backend flash_attn_v100 \
  --tensor-parallel-size 4 \
  --host 0.0.0.0 \
  --port 8082 \
  --mem-fraction-static 0.70 \
  --context-length 262144 \
  --max-running-requests 4 \
  --chunked-prefill-size 8192 \
  --mamba-scheduler-strategy extra_buffer \
  --cuda-graph-max-bs 4 \
  --cuda-graph-bs 1 2 4 \
  --enable-nccl-nvls \
  --enable-multimodal \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder

Qwen3.6-35B-A3B AWQ target only

NCCL_P2P_LEVEL=NVL \
SGLANG_CUSTOM_ALLREDUCE_ALGO=1stage \
SGLANG_MAMBA_CONV_DTYPE=float16 \
SGLANG_MAMBA_SSM_DTYPE=float16 \
sglang serve \
  --model QuantTrio/Qwen3.6-35B-A3B-AWQ \
  --dtype float16 \
  --quantization awq_marlin \
  --attention-backend flash_attn_v100 \
  --tensor-parallel-size 4 \
  --host 0.0.0.0 \
  --port 8082 \
  --mem-fraction-static 0.78 \
  --context-length 262144 \
  --max-running-requests 4 \
  --chunked-prefill-size 16384 \
  --cuda-graph-max-bs 4 \
  --cuda-graph-bs 1 2 4 \
  --enable-nccl-nvls

Qwen3.6-27B FP16 target only

NCCL_P2P_LEVEL=NVL \
SGLANG_CUSTOM_ALLREDUCE_ALGO=1stage \
SGLANG_MAMBA_CONV_DTYPE=float16 \
SGLANG_MAMBA_SSM_DTYPE=float16 \
sglang serve \
  --model Qwen/Qwen3.6-27B \
  --dtype float16 \
  --attention-backend flash_attn_v100 \
  --tensor-parallel-size 4 \
  --host 0.0.0.0 \
  --port 8082 \
  --mem-fraction-static 0.80 \
  --context-length 262144 \
  --max-running-requests 4 \
  --chunked-prefill-size 16384 \
  --cuda-graph-max-bs 4 \
  --cuda-graph-bs 1 2 4 \
  --enable-nccl-nvls \
  --enable-multimodal \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder

Qwen3.6-27B-FP8 target only

NCCL_P2P_LEVEL=NVL \
SGLANG_CUSTOM_ALLREDUCE_ALGO=1stage \
SGLANG_MAMBA_CONV_DTYPE=float16 \
SGLANG_MAMBA_SSM_DTYPE=float16 \
sglang serve \
  --model Qwen/Qwen3.6-27B-FP8 \
  --dtype float16 \
  --kv-cache-dtype fp8_e4m3 \
  --attention-backend flash_attn_v100 \
  --tensor-parallel-size 4 \
  --host 0.0.0.0 \
  --port 8082 \
  --mem-fraction-static 0.80 \
  --context-length 262144 \
  --max-running-requests 1 \
  --chunked-prefill-size 4096 \
  --mamba-full-memory-ratio 0.1 \
  --mamba-scheduler-strategy extra_buffer \
  --cuda-graph-max-bs 1 \
  --cuda-graph-bs 1 \
  --enable-nccl-nvls \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder

Poolside Laguna-S-2.1 INT4 target only

NCCL_P2P_LEVEL=NVL \
SGLANG_CUSTOM_ALLREDUCE_ALGO=1stage \
sglang serve \
  --model poolside/Laguna-S-2.1-INT4 \
  --trust-remote-code \
  --dtype float16 \
  --kv-cache-dtype auto \
  --attention-backend flash_attn_v100 \
  --moe-runner-backend marlin \
  --tensor-parallel-size 4 \
  --host 0.0.0.0 \
  --port 8082 \
  --mem-fraction-static 0.76 \
  --swa-full-tokens-ratio 0.08 \
  --context-length 262144 \
  --page-size 16 \
  --max-running-requests 2 \
  --chunked-prefill-size 4096 \
  --triton-attention-num-kv-splits 128 \
  --cuda-graph-max-bs 2 \
  --cuda-graph-bs 1 2 \
  --enable-nccl-nvls \
  --reasoning-parser poolside_v1 \
  --tool-call-parser poolside_v1

Qwen3.6-27B FP16 with DFlash

NCCL_P2P_LEVEL=NVL \
SGLANG_CUSTOM_ALLREDUCE_ALGO=1stage \
SGLANG_MAMBA_CONV_DTYPE=float16 \
SGLANG_MAMBA_SSM_DTYPE=float16 \
SGLANG_ENABLE_SPEC_V2=1 \
SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 \
sglang serve \
  --model Qwen/Qwen3.6-27B \
  --dtype float16 \
  --attention-backend flash_attn_v100 \
  --tensor-parallel-size 4 \
  --host 0.0.0.0 \
  --port 8082 \
  --mem-fraction-static 0.70 \
  --context-length 262144 \
  --max-running-requests 4 \
  --chunked-prefill-size 4096 \
  --mamba-scheduler-strategy extra_buffer \
  --cuda-graph-max-bs 4 \
  --cuda-graph-bs 1 2 4 \
  --enable-nccl-nvls \
  --enable-multimodal \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path z-lab/Qwen3.6-27B-DFlash \
  --speculative-dflash-block-size 16 \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder

Qwen3.6-35B-A3B AWQ with DFlash

NCCL_P2P_LEVEL=NVL \
SGLANG_CUSTOM_ALLREDUCE_ALGO=1stage \
SGLANG_MAMBA_CONV_DTYPE=float16 \
SGLANG_MAMBA_SSM_DTYPE=float16 \
SGLANG_ENABLE_SPEC_V2=1 \
SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 \
sglang serve \
  --model QuantTrio/Qwen3.6-35B-A3B-AWQ \
  --dtype float16 \
  --quantization awq_marlin \
  --attention-backend flash_attn_v100 \
  --tensor-parallel-size 4 \
  --host 0.0.0.0 \
  --port 8082 \
  --mem-fraction-static 0.78 \
  --context-length 262144 \
  --max-running-requests 4 \
  --chunked-prefill-size 4096 \
  --mamba-scheduler-strategy extra_buffer \
  --cuda-graph-max-bs 4 \
  --cuda-graph-bs 1 2 4 \
  --enable-nccl-nvls \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path z-lab/Qwen3.6-35B-A3B-DFlash \
  --speculative-dflash-block-size 8 \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder

Qwen3.5-122B-A10B GPTQ-Int4 with MTP

NCCL_P2P_LEVEL=NVL \
SGLANG_CUSTOM_ALLREDUCE_ALGO=1stage \
SGLANG_MAMBA_CONV_DTYPE=float16 \
SGLANG_MAMBA_SSM_DTYPE=float16 \
SGLANG_ENABLE_SPEC_V2=1 \
sglang serve \
  --model Qwen/Qwen3.5-122B-A10B-GPTQ-Int4 \
  --dtype float16 \
  --quantization gptq_marlin \
  --attention-backend flash_attn_v100 \
  --tensor-parallel-size 4 \
  --host 0.0.0.0 \
  --port 8082 \
  --mem-fraction-static 0.70 \
  --context-length 8192 \
  --max-running-requests 4 \
  --chunked-prefill-size 4096 \
  --mamba-scheduler-strategy extra_buffer \
  --cuda-graph-max-bs 4 \
  --cuda-graph-bs 1 2 4 \
  --enable-nccl-nvls \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4

Qwen3.6-27B FP16 with MTP

NCCL_P2P_LEVEL=NVL \
SGLANG_CUSTOM_ALLREDUCE_ALGO=1stage \
SGLANG_MAMBA_CONV_DTYPE=float16 \
SGLANG_MAMBA_SSM_DTYPE=float16 \
SGLANG_ENABLE_SPEC_V2=1 \
sglang serve \
  --model Qwen/Qwen3.6-27B \
  --dtype float16 \
  --attention-backend flash_attn_v100 \
  --tensor-parallel-size 4 \
  --host 0.0.0.0 \
  --port 8082 \
  --mem-fraction-static 0.70 \
  --context-length 8192 \
  --max-running-requests 4 \
  --chunked-prefill-size 4096 \
  --mamba-scheduler-strategy extra_buffer \
  --cuda-graph-max-bs 4 \
  --cuda-graph-bs 1 2 4 \
  --enable-nccl-nvls \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4

About

SGLang is a high-performance serving framework for large language models and multimodal models.

sglang.io

Resources

Readme
Apache-2.0 license
Activity

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages