Skip to content

Disaggregated Encoder

Disaggregated Encoder beta status

This feature is in beta. Backward compatibility is not guaranteed, and production use is not recommended.

Model implementation support

The configuration described on this page applies to the transformers (optimum) model implementation.

Overview

Disaggregated Encoder runs a multimodal model's visual encoder and language model in separate vLLM processes. The encoders extract visual features from images, and the language model uses these features to generate text.

This lets you allocate NPUs to the visual encoder and language model independently and scale image encoding by adding encoder instances.

Supported models

Disaggregated Encoder supports the Qwen3-VL family only.

Architecture

The deployment consists of a proxy, N encoder instances, and one prefill/decode (PD) instance. The PD instance runs both language-model prefill and decode in the same process.

Assign a separate set of NPUs to each encoder instance and to the PD instance.

Component Responsibility
Proxy Accepts client requests, distributes image items across encoders, and forwards the original request to the PD
Encoder Runs the visual encoder and produces image features
PD Retrieves the encoder features and runs both language-model prefill and decode in the same instance

The following diagram and table show the stages of a chat request containing images.

---
config:
  theme: base
  themeVariables:
    actorLineColor: "#0d9488"
---
sequenceDiagram
    participant Client
    participant Proxy
    participant Encoder as Encoder × N
    participant PD as PD (Prefill + Decode)
    Client->>Proxy: Chat request
    Proxy->>Encoder: 1. Image-only requests
    Note over Encoder: 2. Encode images
    Encoder->>PD: 3. Output tensor metadata (ZMQ)
    Encoder-->>Proxy: Encoder requests succeed
    Proxy->>PD: 4. Original request
    Encoder->>PD: 5. Encoder output tensors (NIXL)
    Note over PD: 6. Run prefill and decode
    PD-->>Proxy: Completion
    Proxy-->>Client: Completion
Step Action What happens
1 Route images The proxy distributes images across encoder instances in round-robin order, then sends one image-only request per image concurrently.
2 Encode images Each encoder computes visual features and registers the resulting tensors with NIXL.
3 Send output tensor metadata Each encoder sends metadata about its output tensors (location, shape, and data type) to the PD over ZMQ, then reports success to the proxy.
4 Forward original request Once all encoder requests succeed, the proxy forwards the original request, including text and images, to the PD.
5 Retrieve output tensors The PD uses the metadata to pull encoder output tensors over NIXL and stores them in its encoder cache.
6 Run prefill and decode The PD runs prefill and decode with the cached encoder output tensors. The proxy returns the completion to the client.

The proxy forwards text-only requests directly to the PD without sending them to the encoders.

Configuration

Installing nixl

Disaggregated Encoder requires nixl in the Python environments that run the encoder and PD instances. See Optional dependencies for installation instructions.

Encoder and PD

HTTP serving

Each encoder and PD instance runs an HTTP server that receives requests from the proxy. Configure its bind address and port with the following vllm serve CLI options:

CLI option Set on Description
--host Encoder, PD Bind address for HTTP requests from the proxy
--port Encoder, PD Listening port for HTTP requests from the proxy

Set --host to 127.0.0.1 to accept requests only from a proxy on the same host. For a proxy on another host, bind to a local interface IP or to 0.0.0.0 to listen on all local interfaces.

Visual feature transfer

To make the encoders' visual features available to the PD, configure an encoder-cache (EC) connector.

On each encoder and PD instance, pass a JSON object to the vllm serve CLI option --ec-transfer-config. The object has the following structure:

1
2
3
4
5
6
--ec-transfer-config (JSON)
├── ec_connector              # Connector selection
├── ec_role                   # Producer or consumer role
└── ec_connector_extra_config # Metadata endpoint settings
    ├── pull_host, pull_port          # PD only
    └── llm_host, llm_pull_port       # Encoder only
ec_connector

Set to RblnECNixlConnector on both encoder and PD instances to transfer visual features through NIXL.

ec_role

Set the role for each instance:

Set on JSON value Description
Encoder ec_producer Provides visual features
PD ec_consumer Retrieves visual features
ec_connector_extra_config

Configure how encoders send metadata to the PD:

JSON key Set on Description
pull_host PD Bind address for metadata sent by encoders
pull_port PD Listening port for metadata sent by encoders
llm_host Encoder Destination address of the PD's metadata socket
llm_pull_port Encoder Destination port. Must match the PD's pull_port.

On a single host, use 127.0.0.1 for both pull_host and llm_host.

Across hosts, set the PD's pull_host to a local interface IP or 0.0.0.0, and each encoder's llm_host to a reachable PD address.

Proxy

The proxy receives client requests on its own HTTP port, sends image requests to the encoders, and forwards the original request to the PD. Configure the proxy script with the following CLI options:

CLI option Set on Description
--host Proxy Bind address for HTTP requests from clients
--port Proxy Listening port for HTTP requests from clients
--encode-servers-urls Proxy Comma-separated HTTP URLs of the encoder instances. Update this list when adding encoders.
--decode-servers-urls Proxy HTTP URL of the single PD instance

Set the proxy's --host to 127.0.0.1 to accept requests only from clients on the same host. For clients on other hosts, bind to a local interface IP or to 0.0.0.0 to listen on all local interfaces.

Example

This example deploys Qwen3-VL-8B-Instruct on a single host with ATOM™ NPUs.

Precompiling the model is recommended for this setup because encoder instances require a compiled model.

Before starting, install RBLN vLLM and nixl.

Ports and devices

This example runs eight encoder instances and one prefill/decode (PD) instance on 16 NPUs. The commands use the following ports and devices:

Component HTTP URL PD metadata port NPU devices
Proxy http://127.0.0.1:19000 — —
PD http://127.0.0.1:19100 19500 0–7
Encoders 0–7 http://127.0.0.1:19101 to http://127.0.0.1:19108 19500 8–15 (one per encoder)

Choose available ports and device IDs for your environment. This example uses RBLN_VISIBLE_DEVICES to assign NPUs to each encoder and PD process. See Configuration for the connector and proxy settings.

Compile the model

Compile the model once with the optimum-rbln Python API and save it to Qwen3-VL-8B-Instruct for use by both the encoder and PD instances:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
from optimum.rbln import RBLNQwen3VLForConditionalGeneration

model_id = "Qwen/Qwen3-VL-8B-Instruct"
model = RBLNQwen3VLForConditionalGeneration.from_pretrained(
    model_id,
    export=True,
    rbln_config={
        "visual": {
            "max_seq_len": 5440,
            "num_devices": 1,
        },
        "num_devices": 8,
        "max_seq_len": 32_768,
        "batch_size": 8,
        "decoder_batch_sizes": [8],
    },
)
model.save_pretrained("Qwen3-VL-8B-Instruct")

Run the serving commands from the directory containing Qwen3-VL-8B-Instruct, or replace the model path in each command.

Launch the PD instance

Start the PD as the EC consumer:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
PD_EC_CONFIG='{
    "ec_connector": "RblnECNixlConnector",
    "ec_role": "ec_consumer",
    "ec_connector_extra_config": {
        "pull_host": "127.0.0.1",
        "pull_port": 19500
    }
}'

RBLN_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
    vllm serve Qwen3-VL-8B-Instruct \
        --model-impl transformers \
        --host 127.0.0.1 \
        --port 19100 \
        --mm-processor-kwargs '{"max_pixels": 802816}' \
        --ec-transfer-config "$PD_EC_CONFIG"

Optionally, run the following command in another terminal to wait for the PD to start accepting requests. You can skip this step if the server logs show that the PD is ready.

1
2
3
until curl -sf http://127.0.0.1:19100/health > /dev/null; do
    sleep 2
done

Launch encoder instances

Once the PD is ready, run the following block in a separate terminal for each encoder, using a unique ENCODER_ID from 0 to 7.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
ENCODER_ID=0

ENCODER_EC_CONFIG='{
    "ec_connector": "RblnECNixlConnector",
    "ec_role": "ec_producer",
    "ec_connector_extra_config": {
        "llm_host": "127.0.0.1",
        "llm_pull_port": 19500
    }
}'

RBLN_VISIBLE_DEVICES=$((8 + ENCODER_ID)) \
    vllm serve Qwen3-VL-8B-Instruct \
        --model-impl transformers \
        --host 127.0.0.1 \
        --port $((19101 + ENCODER_ID)) \
        --mm-processor-kwargs '{"max_pixels": 802816}' \
        --ec-transfer-config "$ENCODER_EC_CONFIG"

Optionally, run the following command in another terminal to wait for all encoder instances to start accepting requests before launching the proxy. You can skip this step if their logs show that they are all ready.

1
2
3
4
5
for port in $(seq 19101 19108); do
    until curl -sf "http://127.0.0.1:$port/health" > /dev/null; do
        sleep 2
    done
done

Launch the proxy

Run the example proxy in a separate terminal. Replace /path/to/vllm-rbln with the path to your local vllm-rbln checkout:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
ENCODER_URLS="http://127.0.0.1:19101,\
http://127.0.0.1:19102,\
http://127.0.0.1:19103,\
http://127.0.0.1:19104,\
http://127.0.0.1:19105,\
http://127.0.0.1:19106,\
http://127.0.0.1:19107,\
http://127.0.0.1:19108"

python /path/to/vllm-rbln/examples/optimum/ec_disagg/client_ec_disaggregated.py \
    --host 127.0.0.1 \
    --port 19000 \
    --encode-servers-urls "$ENCODER_URLS" \
    --decode-servers-urls "http://127.0.0.1:19100"

Send a request

Once the proxy is ready, send an image and a text prompt to its /v1/chat/completions endpoint from another terminal:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
REQUEST_BODY='{
    "model": "Qwen3-VL-8B-Instruct",
    "messages": [
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "http://images.cocodataset.org/val2017/000000039769.jpg"
                    }
                },
                {
                    "type": "text",
                    "text": "Briefly describe this image."
                }
            ]
        }
    ],
    "max_tokens": 256
}'

curl -sS http://127.0.0.1:19000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d "$REQUEST_BODY"

The response contains the model's description of the image in choices[0].message.content.