Skip to content

Attention Modes

Overview

RBLN vLLM supports Full Attention, Sliding-Window Attention (SWA), and Multi-head Latent Attention (MLA).

  • Full Attention attends to all context tokens allowed by the attention mask.
  • Sliding-Window Attention (SWA) limits attention to tokens within the model's configured window.
  • Multi-head Latent Attention (MLA) stores a compressed latent representation of keys and values in the KV cache and uses an MLA backend.

For Full Attention, both the vllm and transformers (optimum) model implementations support two computation methods.

  • Naive Attention uses one KV cache block per request. It computes attention scores for valid tokens, applies softmax, and multiplies the resulting weights by the value matrix (V).
  • Flash Attention processes each request's KV cache in partitions and combines the results using online softmax.

vllm model implementation

Full Attention

Full attention layers use causal masking to determine which tokens each query can attend to.

Kernel selection

  • Naive Attention is used when block_size == max_model_len and the layer has no attention sinks.
  • Flash Attention is used when block_size < max_model_len or the layer has attention sinks.

block_size must be a power of two within the supported range shown below.

Device Naive / Flash Attention
Rebel100™ 512–16,384 tokens

Examples

The following examples show how to select Naive Attention or Flash Attention by adjusting block_size. They apply to full attention layers without attention sinks in models that do not use MLA. Replace <MODEL_ID> with your model ID.

1
2
3
4
5
6
7
# Naive Attention
vllm serve <MODEL_ID> --model-impl vllm \
  --max-model-len 8192 --block-size 8192

# Flash Attention
vllm serve <MODEL_ID> --model-impl vllm \
  --max-model-len 8192 --block-size 4096

FP8 KV cache

Naive Attention does not support FP8 KV cache.

Sliding-Window Attention (SWA)

Sliding-Window Attention layers limit attention to tokens within the model's configured window.

See the GPT-OSS-120B recipe.

Multi-head Latent Attention (MLA)

MLA stores a compressed latent representation of keys and values in the KV cache, in either a dense or a sparse form. DeepSeek-V3.2 uses Sparse MLA.

FP8 KV cache

FP8 KV cache is supported only for Sparse MLA.

transformers (optimum) model implementation

The transformers (optimum) implementation uses models compiled by RBLN Optimum, which determines their attention configuration during compilation.

For precompiled models, the engine uses the attention configuration recorded in rbln_config.json.

Full Attention

Kernel selection

When the engine compiles a model at startup, the following Naive/Flash selection conditions apply to full attention layers.

  • Naive Attention is used when block_size == max_model_len.
  • Flash Attention is used when block_size < max_model_len and max_model_len % block_size == 0.

block_size must be a power of two within the supported ranges shown below, which depend on the device and attention method.

Device Naive Attention Flash Attention
ATOM™ Up to 32,768 tokens 1,024–32,768 tokens
Rebel100™ Up to 16,384 tokens 1,024–16,384 tokens

Pooling and encoder-decoder models use Naive Attention and require block_size == max_model_len. If block_size is omitted, it is automatically set to max_model_len. Qwen3-backbone pooling models are an exception. To use Flash Attention, explicitly specify a block_size that satisfies the conditions above.

Examples

The following examples show how to select Naive Attention or Flash Attention by adjusting block_size when the engine compiles a model at startup. The Flash Attention example requires a model that supports Flash Attention. Replace <MODEL_ID> with your model ID.

1
2
3
4
5
6
7
# Naive Attention
vllm serve <MODEL_ID> --model-impl transformers \
  --max-model-len 8192 --block-size 8192

# Flash Attention
vllm serve <MODEL_ID> --model-impl transformers \
  --max-model-len 8192 --block-size 4096

Sliding-Window Attention (SWA)

Sliding-Window Attention layers limit attention to tokens within the model's configured window. See the following recipes.