Vidur / spec-decode

An independent extension of Microsoft Vidur

Speculative decoding,
under the microscope.

A faster decode step starts with a proposal. Whether it pays off depends on the cost of checking it.

Model proposal generation, accepted-token progress, and KV memory. Then explore how two attention-cost assumptions change the same simulation.

Explore the resultsRead the modeling notes
Inside one verify step Example: k = 4, two accepted proposals
aacceptedbaccepted+1targetcdiscardedddiscarded
5tokens computed
3tokens of progress

Verification costs k + 1 tokens.
Progress is accepted + 1 tokens.

Recorded simulator predictionsLlama-2-7B · A100 profile · 300 requests per runView source CSV

One workload. Two different answers.

Compare the predicted median time per output token (TPOT).
A speedup above 1× is faster than the no-speculation baseline.

GPU validation pending

Controls select from the bundled 39-row sweep. No simulation runs in your browser.

Predicted median TPOT speedup

KV read oncePer-token cost

Loading recorded results…

Each line connects two attention-cost assumptions. It is not a confidence interval.

Inspect the selected results
Recorded results for the selected load and acceptance rate
Proposals Attention assumption TPOT p50 TTFT p50 E2E p50 TPOT speedup

What the model accounts for.

The work of computing tokens and the progress of accepting them are different quantities.

Follow the implementation
Propose
N-gram or fixed overhead, or a separately trained draft-model predictor. Draft weights and KV cache consume memory.
Verify
Charge for k + 1 computed tokens. Compare a shared-KV lower bound with a per-token upper bound; an interpolated mode is also available in the simulator.
Advance
Keep the accepted prefix plus one target token. Respect Sarathi’s token budget, request boundaries, and lookahead KV allocation.

The next question belongs on a GPU.

The repository includes a vLLM validation harness: replay matched workloads, calibrate acceptance from server counters, then compare measured and simulated latencies request by request.

No real GPU validation results are published here. Unit tests and a fake-server harness check behavior; they do not establish timing accuracy.

Open the validation workflow
  1. Generate a workloadSeeded arrivals and exact lengths
  2. Replay against vLLMMeasured latency and server counters
  3. Calibrate and simulateMatched acceptance and KV budget
  4. Compare the resultsTTFT, TPOT, E2E, and acceptance

Read the bounds with the assumptions.

Scheduler scopeSpeculative decoding supports Sarathi with pipeline parallelism 1. Acceptance is an input, not a prediction.

Approximate costsVerify attention uses existing decode profiles. Short drafts and draft pass 1 are approximate; rejection-sampler overhead is omitted.

Profile and load limitsThe bundled profiles predate current vLLM kernels. The 6 requests/s baseline is overloaded, so ratios depend on run length.