Speculative Decoding¶

Model implementation support

This feature runs on the vllm model implementation only. See the Feature Matrix for feature support by model implementation.

Overview¶

Decode produces one token per forward pass, so a decode step costs a full pass regardless of how predictable the next token is. Speculative decoding proposes several tokens ahead with a cheaper proposer, then verifies them in a single pass of the target model, accepting the longest prefix the target agrees with. Speculative decoding can reduce inter-token latency when enough proposed tokens are accepted, which makes the feature most useful at medium-to-low request rates where the NPU is not already saturated.

RBLN vLLM configures this exactly as upstream vLLM does, through speculative_config.

Supported methods¶

`method`	Proposer	Notes
`ngram`	Draft-free	N-gram. Matches repeats in the prompt and the output so far
`suffix`	Draft-free	Suffix decoding. Matches against a suffix tree built from the request history
`eagle`	Draft model	EAGLE
`eagle3`	Draft model	EAGLE3
`medusa`	Draft model	Medusa
`mtp`	Draft model	Multi-token prediction, using the target checkpoint's MTP layers as the draft

The draft-free methods need no second checkpoint, so they are the cheapest to try: they speculate by finding a repeat, which makes them effective on workloads with repetitive or quoted input and weak on freely generated text. The draft-model methods need a matching draft checkpoint for the target.

Upstream vLLM methods not listed here are not validated on RBLN NPUs.

Configuration¶

Pass speculative_config to LLM(...), or --speculative-config with the same keys as JSON to vllm serve.

Common keys:

method: The proposer to use, one of the method values in the table above.
model: The draft checkpoint. Required by the draft-model methods, and omitted by the draft-free ones.
num_speculative_tokens: How many tokens the proposer proposes per step. A larger value raises the possible saving per step and the wasted work when the proposal is rejected.

Method-specific keys, for the draft-free methods:

prompt_lookup_min / prompt_lookup_max: N-gram only. The shortest and longest n-gram the proposer matches.
suffix_decoding_max_spec_factor: Suffix decoding only. Scales how many tokens the proposer speculates relative to the match it found.

An n-gram example, which needs no draft checkpoint:

vllm serve <MODEL_ID> \
  --model-impl vllm \
  --speculative-config '{"method":"ngram","prompt_lookup_min":3,"prompt_lookup_max":5,"num_speculative_tokens":3}'

An EAGLE example, which pairs a draft checkpoint with the target:

vllm serve <MODEL_ID> \
  --model-impl vllm \
  --speculative-config '{"method":"eagle","model":"<DRAFT_MODEL_ID>","num_speculative_tokens":3}'

The same configuration through the offline API:

from vllm import LLM

llm = LLM(
    model="<MODEL_ID>",
    speculative_config={
        "method": "eagle",
        "model": "<DRAFT_MODEL_ID>",
        "num_speculative_tokens": 3,
    },
)

Known incompatibilities¶

The transformers (optimum) model implementation. Speculative decoding requires the vllm model implementation. The engine rejects a speculative_config there.
Asynchronous scheduling. The two cannot run together, because the asynchronous path feeds one sampled token per step into the next. When both are requested the engine disables asynchronous scheduling, logs a warning, and runs synchronously.