Qwen3-VL-2B (VLM)¶

Compiling and loading both submodules on one NPU¶

On Rebel100™, visual and the LM each run on a single NPU. The Model Zoo example sets create_runtimes to False at both levels, so compilation only produces and saves the compiled artifacts without creating runtimes.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

import os

from optimum.rbln import RBLNAutoModelForImageTextToText

model_id = "Qwen/Qwen3-VL-2B-Instruct"
model = RBLNAutoModelForImageTextToText.from_pretrained(
    model_id,
    export=True,
    rbln_config={
        "visual": {
            "max_seq_len": 16384,
            "num_devices": 1,
            "create_runtimes": False,  # do not create the visual runtime at compile time
        },
        "num_devices": 1,
        "kvcache_partition_len": 8_192,
        "max_seq_len": 262_144,
        "create_runtimes": False,  # do not create the LM runtime at compile time
    },
)
model.save_pretrained(os.path.basename(model_id))

At load time, the device field in rbln_config places each runtime. The Model Zoo inference.py places both visual and the LM on device 0.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

import os

from optimum.rbln import RBLNAutoModelForImageTextToText

model_id = "Qwen/Qwen3-VL-2B-Instruct"
model = RBLNAutoModelForImageTextToText.from_pretrained(
    os.path.basename(model_id),
    export=False,
    rbln_config={
        "visual": {
            "device": 0,  # visual on device 0
        },
        "device": 0,  # LM on device 0
    },
)

With this layout, the ViT graphs, the LM graphs, and the KV cache must all fit in the memory of one NPU. If runtime creation fails as the batch size or context length grows, reduce the workload as described in Other scenarios below.

Bucketing the ViT input: visual.max_seq_len¶

Qwen3-VL's ViT runs on one image (or video frame) at a time, and its graph shape is fixed at compile time. A ViT compiled with max_seq_len=16384 always performs work proportional to 16,384 patches, regardless of whether the actual input contains 1,024 patches or 16,384 patches. In real serving workloads, a single request often contains images of different sizes or videos with different numbers of frames. If a single max_seq_len is sized for the largest expected input, smaller inputs still consume the full capacity, which increases latency and wastes memory.

optimum-rbln addresses this with a multi-size bucketing strategy. visual.max_seq_len accepts not only a single int but also a List[int], and the behavior is as follows.

• When a list is provided, optimum-rbln compiles a separate ViT graph for each length in a single compilation.
• At inference time, the smallest bucket that can accommodate the actual patch count is selected automatically.
• If an input exceeds every bucket in the list, an error is raised. Choose the largest value so that it covers the upper bound of your workload.

The example below is a modified version of the Model Zoo compile.py, where the single value (max_seq_len: 16384) is replaced with a list of three buckets compiled together.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

import os

from optimum.rbln import RBLNAutoModelForImageTextToText

model_id = "Qwen/Qwen3-VL-2B-Instruct"
model = RBLNAutoModelForImageTextToText.from_pretrained(
    model_id,
    export=True,
    rbln_config={
        "visual": {
            "max_seq_len": [1024, 3136, 16384],  # ← compile three buckets together
            "num_devices": 1,
            "create_runtimes": False,
        },
        "num_devices": 1,
        "kvcache_partition_len": 8_192,
        "max_seq_len": 262_144,
        "create_runtimes": False,
    },
)
model.save_pretrained(os.path.basename(model_id))

At runtime, the compiled model behaves as follows.

1

patches = (H / 16) × (W / 16)

Bucketing the decoder batch size: decoder_batch_sizes¶

Qwen3-VL's language-model decoder is also a graph whose shape is fixed at compile time. A decoder compiled with batch_size=4 always performs work proportional to 4 slots, whether the actual batch is 3 or 4. In real serving (continuous batching, in-flight batching), requests finish at different times and the active batch frequently shrinks below the compiled maximum, so the compute spent on unused slots is wasted.

decoder_batch_sizes addresses this with bucketing at the decoder graph level. It is a top-level (LM-side) field defined on RBLNDecoderOnlyModelConfig and accepts a List[int]. The behavior is as follows.

• When a list is provided, optimum-rbln compiles a separate decoder graph (decoder_batch_{N}) for each batch size.
• Every value must be ≤ batch_size, and the maximum value should equal batch_size. If it is smaller, optimum-rbln automatically appends batch_size and emits a warning.
• At inference time, the engine selects the most appropriate decoder graph for the active batch size during the decoding phase.

The configuration below adds batch_size=4 to the Model Zoo compile.py settings and compiles three decoder graphs (1, 2, 4). It also lowers max_seq_len to 32,768: at batch_size=4 the Model Zoo's 262,144-token context does not fit on one Rebel100™ together with the ViT, and loading fails with Failed to create RBLN runtime ... Lack of device memory. With 32,768, both submodules load on one NPU. To keep the full context at a larger batch, place visual on a second NPU (see Troubleshooting Multi-Module Models).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

rbln_config = {
    "visual": {
        "max_seq_len": 16384,
        "num_devices": 1,
        "create_runtimes": False,
    },
    "num_devices": 1,
    "kvcache_partition_len": 8_192,
    "max_seq_len": 32_768,  # ← 262_144 at batch_size=4 does not fit one Rebel100™
    "batch_size": 4,
    "decoder_batch_sizes": [1, 2, 4],  # ← compile three decoder graphs
    "create_runtimes": False,
}

At runtime, the compiled model selects a decoder graph based on the active batch size.

Other scenarios¶

Specific memory and device configurations, as well as compile/inference workflows, are covered step-by-step in Troubleshooting Multi-Module Models. Frequently encountered scenarios and their corresponding guides: