LoRA-Spec

LoRA-Spec is an MLSys 2027 research artifact for deciding how speculative decoding should be deployed for each new LoRA adapter.

The project is no longer a paper about discovering LoRA-induced speculative-decoding degradation or first-order logit-shift theory. Those are prior-art boundaries after the novelty audit: EDA (2026) covers the degradation phenomenon and parameter-efficient draft adaptation; the AFA-LoRA cluster covers trained drafter alignment; "Formalising the Logit Shift Induced by LoRA" (2026) covers first-order logit-shift theory.

LoRA-Spec instead asks an operator-facing question:

Given a new LoRA adapter, should serving speculate normally, shorten speculation or apply a cheap correction, or disable speculation / adapt the drafter?

Core Design

The central artifact is a two-stage decision framework.

1. Stage 1 predicts acceptance-level mismatch from adapter checkpoint features.
2. Stage 2 maps predicted mismatch through a runtime cost model to a deployment action.

Checkpoint features predict mismatch, not throughput. Serving benefit depends on load, adapter cache behavior, prompt lengths, draft/verify costs, and traffic skew, so acceptance and throughput can have different rank orderings.

Predictor Modes

• checkpoint_only: adapter tensors and module-pattern features only.
• checkpoint_meta: checkpoint features plus adapter metadata such as rank, domain, license, target family, and training recipe.
• checkpoint_calib: checkpoint+metadata plus small calibration features: KL/JS divergence, top-1 agreement, and entropy shift.

Calibration features are mode-isolated and cannot leak into checkpoint-only results.

Gate Actions

• green: use unmodified speculation.
• amber: use shortened speculation depth and optionally a sub-millisecond correction.
• red: disable speculation, or train/use a micro-LoRA drafter only if the adapter is both harmful and traffic-hot.
• abstain: return manual review when uncertainty is too wide or the adapter is out of distribution.

Required baselines:

• Predictor-gated disablement.
• Predictor-gated shortened speculation depth.

Recovery methods must beat these near-zero-cost baselines to be considered useful.

Experiment Flow

The design is screen-confirm-stress.

1. scripts/phase0_vllm_runner.py: plan/execute pinned-vLLM composability cells and merge measured observations.
2. scripts/phase0_composability.py: overlay measured cells onto the full conceptual matrix and emit the fallback policy.
3. scripts/phase1_collect.py: collect paired prompt-level baseline/adapted measurements with immutable model and adapter revisions.
4. scripts/phase1_effect_gate.py: evaluate the measured aggressive-adapter effect with explicit GO / INVESTIGATE / STOP.
5. scripts/phase2_screen.py: screen 40-80 adapters on the primary Llama 8B/1B pair.
6. scripts/phase2_confirm.py: confirm predicted extremes on Qwen 2.5 7B/0.5B.
7. scripts/phase3_predictor.py: fit ElasticNet, GAM, and gradient-boosted-tree predictors by feature mode.
8. scripts/phase4_recovery.py: compare micro-LoRA and two cheap corrections against gating baselines.
9. scripts/phase5_serving.py: multi-tenant traffic stress under uniform, Zipf, and bursty traffic with warm and limited cache modes.
10. scripts/diagnostic.py: standalone adapter diagnostic from checkpoint-derived features and an exported predictor JSON.

Setup

python -m pip install -e ".[dev,analysis]"
pytest -q
ruff check .

GPU serving experiments additionally require the pinned serving stack in the Colab/runtime extra:

python -m pip install -e ".[colab]"

CPU tests do not require vLLM.

CPU smoke commands must be explicitly marked as smoke tests so synthetic values cannot be mistaken for evidence. Phase 0 can run on CPU, but without measured observations it writes a contract-only matrix:

python scripts/phase0_composability.py --output-dir results/phase0
python scripts/phase3_predictor.py --smoke-test --output-dir results/phase3
python scripts/phase4_recovery.py --smoke-test --output-dir results/phase4
python scripts/phase5_serving.py --smoke-test --output-dir results/phase5

Measured Phase 0 evidence should be passed back as an observations JSON:

python scripts/phase0_composability.py \
  --observations-json results/phase0/vllm_observations.json \
  --output-dir results/phase0

The collectors are plan-only unless --execute is explicit. On a CPU machine, these commands validate every frozen input without importing vLLM or downloading weights:

python scripts/phase0_vllm_runner.py --config configs/phase0_runtime.yaml
python scripts/phase1_collect.py --config configs/phase1_measurement.yaml

Configs

• configs/models.yaml: primary and confirmatory model pairs; 70B is appendix-only.
• configs/adapter_pool.yaml: screening pool with license fields and controlled-adapter slots.
• configs/gate.yaml: green/amber/red thresholds and cost-model defaults.
• configs/traffic.yaml: traffic patterns, cache modes, prompt-length mix, and seeds.
• configs/prompts.yaml: frozen prompt split hashes.
• configs/baselines.yaml: baseline method registry.
• configs/phase0_runtime.yaml: pinned vLLM composability cells and cache semantics.
• configs/phase1_measurement.yaml: frozen paired Phase 1 measurement design.

Every experiment writes JSON with config hash, git hash, runtime metadata, timestamp, seed, and full config.

Double-Blind Rule

Until review is complete, do not create public issues, public PRs, public discussion posts, or identity-linked artifacts about this project. Any vLLM bugs should be recorded privately in the research log.