👓 specspecs

Observability for speculative decoding.

Sample sequences and inspect draft models.

View the accept/reject statistics for your draft model, and actually see the drafts. In my opinion this will give you really strong intuition for how good the model is.

Installation

pip install specspecs --all-extras

Demo

Full-quality video

Usage

As of v0.2, specspecs ships with a vllm plugin!

vLLM

For vLLM, usage is very simple:

1. set a SPECSPECS_DIR and SPECSPECS_RUN_ID env var
2. run vLLM with --no-async-scheduling

For example, to serve Qwen3-4B with a dFlash head:

export SPECSPECS_DIR="./logs" SPECSPECS_RUN_ID="dflash-qwen-4b"
vllm serve Qwen/Qwen3-4B \
    --speculative-config '{"method": "dflash", "model": "z-lab/Qwen3-4B-DFlash-b16", "num_speculative_tokens": 8}' \
    --no-async-scheduling

Custom Generation Loop

If instead you want to instrument a custom generate loop, start by checking out examples/qwen3_dflash.py.

You can run it with:

python examples/qwen3_dflash.py "What is speculative decoding?"

Which will populate ./logs with an observed sequence and all its drafts.

Server

To start the server:

specspecs-serve

This will default to monitoring ./logs as the parquet directory.

Underlying Philosophy

There are two calls you instrument your generation loop with: register_sequence (once at the beginning of each sequence) and update_sequence (once for each draft).

import specspecs

spec_logger = specspecs.SpecLogger("logs/", "Run Name", tokenizer)

tokenizer may be a tokenizer or a model id; its name is stored on every sequence so the viewer can decode the tokens. To log sequences from more than one model, override it per sequence:

spec_logger.register_sequence(prompt_token_ids, sequence_id, tokenizer)

Register Sequence:

spec_logger.register_sequence(prompt_token_ids, sequence_id)

If you don't pass a sequence_id it creates one and returns it.

Update Sequence:

Pass all of the relevant per-draft info here.

spec_logger.update_sequence(sequence_id, draft_tokens, acceptance_length, emitted_token, k, step)

How it Works and Future Plans

It writes sequences and drafts to parquet files, then analyzes them with DuckDB.

I'm working on turning it into a vLLM plugin, and moving the parquet writer to a separate thread. This should be enough to run it on production serving infra (sampling traffic, worst case).