Disaggregated Prefill / Decode¶

Overview¶

Prefill and decode place different demands on an NPU. Prefill processes a whole prompt at once and is compute-bound. Decode produces one token per step and is bound by KV cache reads. When both run on the same instance, a long prefill delays the decode steps queued behind it.

Disaggregated prefill and decode (PD disaggregation) splits the two phases across separate vLLM instances. A prefill instance computes the prompt's KV cache and a decode instance pulls it across the network and generates tokens from it. Prefill and decode instances can be configured and scaled independently.

RBLN vLLM implements this with RblnNixlConnector, a KV connector that transfers the KV cache over RDMA using nixl-rbln.

NIXL (NVIDIA Inference Xfer Library) is the transfer library vLLM's KV connectors are built on. NIXL provides memory registration and transfer interfaces and accesses each transport through a backend plugin. RBLN NPU memory becomes one of the memories NIXL can register through nixl-rbln, as described in the next section.

Architecture¶

RblnNixlConnector extends vLLM's pull-based NIXL connector, so the decode side drives the transfer: it reads the prefill instance's blocks into its own KV cache. A connector for the opposite direction is also available.

A request flows through the two instances as follows:

1. A proxy in front of the two instances forwards the request to the prefill instance first, then to the decode instance, passing the KV transfer parameters along.
2. The prefill instance processes the prompt and registers the blocks holding its KV cache.
3. The decode instance completes a handshake with the prefill instance, then pulls those blocks directly into its own KV cache and starts decoding.

Under the push direction the transfer itself is reversed: the decode instance registers the blocks that will receive the KV cache, and the prefill instance writes into them once the prompt is done.

The two instances must agree on the model, dtype, and attention backend. The connector checks these with a compatibility hash during the handshake and rejects incompatible configurations. The KV cache layout has to match as well; it is not included in the compatibility hash, so the handshake checks it separately.

The connector's own version and the transfer direction also enter the hash, as does the draft model under speculative decoding. A pull instance therefore cannot pair with a push instance.

Prerequisites¶

• The vllm model implementation: --model-impl vllm.
• Device tensors, which the connector requires in both transfer modes. The vllm model implementation keeps the KV cache in device memory by default, so an unmodified configuration already meets this.
• RDMA-capable NICs (RoCE) on every host that runs an instance, reachable from the others. Each NIC that can communicate with the peer provides one transfer path (rail), so a host contributes as many rails as it has such NICs, and an instance pair can transfer over the rails both sides share.
• nixl-rbln, for device-to-device mode.

The connector requires a RoCE NIC that sits on the NPU's NUMA node, has an ACTIVE port, carries an IPv4 address, and is not the interface holding the host's default route. These commands show whether a host has one:

1
2
3
4
5

rbln-smi -j                                      # each NPU's `pci.numa_node`
rdma link show                                   # port state, and the netdev each RDMA device uses
ip route show default                            # the management interface, which is excluded
ip -4 -brief addr show                           # which netdevs carry an IPv4 address
grep . /sys/class/infiniband/*/device/numa_node  # the NUMA node each RDMA device sits on

rbln-smi ships with the RBLN driver, and the rest come from iproute2 and from sysfs, so a host able to run RDMA already has them. Place the engine on an NPU whose NUMA node has such a NIC.

Configuration¶

Both instances are launched with --kv-transfer-config. The connector reads the following keys from it:

kv_connector

RblnNixlPullConnector for the pull direction, or RblnNixlPushConnector for the experimental push direction. RblnNixlConnector remains registered as an alias of the pull connector, since that is the name existing deployments carry.

kv_role

kv_producer, kv_consumer, or kv_both. vLLM requires it whenever a connector is set, and kv_both counts as either role. The example uses kv_both on both instances.

kv_buffer_device

Either "rbln" for device-to-device or "cpu" for host-bounce. Any other value is rejected.

engine_id

Identifies the instance. The decode side checks the identifier it receives against the one it expects.

kv_connector_extra_config

The connector's own settings, listed below.

kv_connector_extra_config is shared with the upstream NIXL connector, which reads keys of its own from it. The ones this connector reads:

swa_window_mode

Default: false. Cuts a sliding-window group's block into the granules its window moves through, and sends only the ones the window sits in. Device-to-device only.

chunk_mode

Default: false. Cuts a block into chunks. A request's last block then goes out as the chunks that hold its tokens, and the empty ones are left out. The chunk the last token falls in still goes in full. Device-to-device only.

chunk_tokens

Default: unset, which takes one prefill step (--max-num-batched-tokens). This value sets the minimum chunk size. The actual chunk is the smallest size at or above this minimum that evenly divides the token range covered by one descriptor. It must be at least one prefill step because each step closes at most one chunk.

push_stream

Default: false. Hands a block over as the prefill closes it. With the setting off, the request's KV is held until the prompt is done. Push direction and device-to-device only.

stripe_width

Default: unset, which the RBLN NIXL backend reads as 1, one rail per transfer. Set it to 0 and the backend opens a connection on every rail the two instances share. Any other value is rejected.

link_down_exit_s

Default: 0, which disables the exit. An instance that has lost every local link exits after this many seconds, so the deployment can start a fresh one. kv_role: kv_producer only: a decode instance keeps its running requests and is replaced from outside.

enforce_handshake_compat belongs to the upstream connector and defaults to true. Set it to false to skip the compatibility-hash check the handshake runs.

chunk_mode and swa_window_mode both narrow what crosses the fabric, and they are independent of each other: each cuts a different part of the cache. chunk_mode keeps the empty part of a request's last block off the fabric, so it saves more as block_size grows relative to the prompts being served. swa_window_mode applies only to a model with sliding-window attention. Its window group occupies a whole block however long the prompt is, and only a window's worth of that block is live, so it saves more the larger block_size is against the window. In the push direction, push_stream shortens the tail: the longer the prompt, the more of its KV has crossed by the time the prompt is done.

Example¶

A pull-direction pair on one host: a prefill instance on the first NPU, a decode instance on the second, and a proxy in front of them.

The prefill instance:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

RBLN_VISIBLE_DEVICES=0 \
VLLM_NIXL_SIDE_CHANNEL_PORT=5559 \
vllm serve Qwen/Qwen3-0.6B \
  --model-impl vllm \
  --port 8100 \
  --block-size 1024 \
  --max-model-len 4096 \
  --max-num-batched-tokens 512 \
  --kv-transfer-config \
  '{"kv_connector":"RblnNixlPullConnector","kv_role":"kv_both","kv_buffer_device":"rbln"}'

The decode instance takes the same configuration, on the other NPU and its own ports:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

RBLN_VISIBLE_DEVICES=1 \
VLLM_NIXL_SIDE_CHANNEL_PORT=5659 \
vllm serve Qwen/Qwen3-0.6B \
  --model-impl vllm \
  --port 8200 \
  --block-size 1024 \
  --max-model-len 4096 \
  --max-num-batched-tokens 512 \
  --kv-transfer-config \
  '{"kv_connector":"RblnNixlPullConnector","kv_role":"kv_both","kv_buffer_device":"rbln"}'

The proxy is upstream vLLM's toy_proxy_server.py:

1
2
3
4

python3 toy_proxy_server.py \
  --port 8192 \
  --prefiller-port 8100 \
  --decoder-port 8200

Send the request to the proxy. Do not send it to either instance directly:

1
2
3

curl http://localhost:8192/v1/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "Qwen/Qwen3-0.6B", "prompt": "The capital of France is", "max_tokens": 16}'

toy_proxy_server.py is an upstream example without routing policies, health checks, or horizontal scaling. A production deployment puts a router in front of the pair.

Both instances above sit on one host, which is why they take different side-channel ports. Across two hosts the ports no longer collide, and VLLM_NIXL_SIDE_CHANNEL_HOST becomes the setting to change. Each instance advertises the address it is given there, and the peer connects to the address it receives. The default is localhost, so an instance that does not set it advertises an address that sends the peer back to the peer's own host, and the handshake fails. That side channel is ZMQ over TCP and carries the handshake only. The KV cache itself travels over RoCE.

Verified configurations are in the Recipes, where prefill and decode run on separate nodes behind the llm-d router.

Verification¶

The logs identify the transfer mode and peer connection status. Search for the following log prefixes.

RBLN NIXL (D2D): registered

The instance registered its KV cache with NIXL in device-to-device mode. Registration follows warm-up; until it runs, the connector logs RBLN NIXL (D2D): deferring registration.

RBLN NIXL: nixl-rbln not available

This instance is running on upstream NIXL's UCX transport. The RBLN backend is not in use.

remote descriptor(s) for engine

The handshake with the peer completed and this instance built the descriptors it transfers with. It appears on the first request that needs that peer, and on the side that issues the transfers: the decode instance under the pull direction, the prefill instance under push. KV cache transfer starts after the handshake completes.

Troubleshooting¶

Start-up fails with no usable data-plane RoCE NIC on numa=N

The instance found nothing on that NUMA node meeting the conditions in the prerequisites, and the message names the node it searched. Run the check there.

Both instances register and neither reaches the handshake

Each prints its registration line and no descriptor line follows. The side-channel address is the usual cause, and the example says what it has to be across two hosts.

Transfer granularity¶

A descriptor's length is fixed when the memory is registered. A transfer chooses which descriptors to send. It cannot send part of one. A partially filled final block is therefore sent in full, unless chunk_mode is on, which cuts the last block into chunks and leaves out the ones holding no tokens. On device-to-device one block spans several descriptors, one per shard.

push_stream changes when those descriptors go out. The set remains the same: a block leaves as the prefill closes it, and the remainder goes at the end of the request.

block_size consequently reaches beyond PD disaggregation. It also sets the compiler's KV cache partition length, constrains the sub-block prefix cache granularity, and must be a multiple of the sliding window where one applies. Change it with the transfer and those three in mind.

Limitations¶

• The prefill and decode instances must use the same block_size. A differing pair is refused.
• Sub-block prefix caching and a KV connector both extend a full-block prefix cache hit, and the scheduler runs only one of them per request, taking whichever offers more tokens, with sub-block winning a tie. See Prefix Caching.