Configuration Guide¶

Overview¶

This guide covers RBLN-specific options and constraints on standard vLLM engine arguments.

Configuration Methods¶

Standard vLLM arguments are configured through LLM(...), vllm serve, or EngineArgs. RBLN-specific options can be passed in three ways:

CLI flags: --rbln-<field>, with underscores replaced by dashes. Boolean options accept --no-rbln-<field> to disable them. Run vllm serve --help=rblnconfig to list the flags.
Python dictionary: Pass additional_config={"<field>": value} to LLM(...).
Python configuration object: Pass additional_config=RBLNConfig(...) or additional_config=OptimumRBLNConfig(...) to LLM(...).

Each entry names the field and its --rbln-* flag. The additional_config key is the field name.

Model Implementation¶

The model implementation determines which RBLN-specific options are available. The options for each implementation are listed in separate sections below.

Use --model-impl to select the implementation. When you pass a Python configuration object, the object's class determines the model implementation:

RBLNConfig selects vllm.
OptimumRBLNConfig selects transformers (optimum).

If both --model-impl and a configuration object are provided, they must select the same implementation. Passing an option that the selected implementation does not support raises an error.

See the Design Overview for a comparison of the model implementations.

Engine Arguments¶

Engine arguments are configured the same way as in upstream vLLM. The following arguments have RBLN-specific constraints.

`model_impl`¶

Default: auto.

Selects which implementation serves the model:

transformers (alias optimum): Runs the model through optimum-rbln.
vllm: Runs vLLM's own implementation. ATOM™ hosts (RBLN-CAxx) reject this setting.
auto: Selects transformers if the model path contains a compiled optimum-rbln artifact or optimum-rbln implements the model architecture. Otherwise, selects vllm.

Upstream vLLM's auto setting prefers vllm instead.

`block_size`¶

Together with max_model_len, determines the attention mode:

block_size == max_model_len: Naive Attention.
max_model_len is a multiple of block_size: Flash Attention.

Configuring block_size when compiling a model

Most models: Specify block_size explicitly.
Pooling and encoder-decoder models: block_size must equal max_model_len, except for the Qwen3 case described below. If omitted, it is automatically set to max_model_len.
Qwen3-backbone pooling models: To enable Flash Attention, specify a block_size smaller than max_model_len.

For precompiled models in the transformers (optimum) implementation, the block size is taken from the compiled model configuration.

See Attention Modes for details.

`max_model_len`¶

Default: The model's maximum sequence length.

Combined with block_size, determines the attention mode. See block_size.

`max_num_seqs`¶

Default: 1.

Uses 1 as the default, compared with 128 in upstream vLLM.

`max_num_batched_tokens`¶

Default: Depends on the model implementation and NPU.

The meaning and default value depend on the model implementation:

vllm: The token budget for one scheduler step. Uses the upstream vLLM default.
transformers (optimum): The prefill chunk size for decoder-only and multimodal models. Defaults to 512 on Rebel100™ and 128 on ATOM™ NPUs.

For transformers, the engine processes prompts in chunks of max_num_batched_tokens tokens, and the chunk size is compiled into the model. When serving a precompiled model, omit this argument to use the recorded chunk size. A different value raises an error.

`tensor_parallel_size`¶

Default: 1.

Sets vLLM-level tensor parallelism. Support depends on the model implementation:

vllm: Splits the model across worker processes that communicate through Rebellions Collective Communications Library (RCCL).
transformers (optimum): Accepts only 1 because the model is a precompiled artifact. Other values raise an error.

For transformers, use num_devices_per_local_rank to run a model across multiple NPUs in one process. The compiler partitions the model using Rebellions Scalable Design (RSD). See Model Parallelism for the differences between RSD and vLLM-level tensor parallelism.

Example of engine argument configuration:

from vllm import LLM

llm = LLM(
    model="<huggingface-model-id>",
    block_size=4096,
    max_model_len=32768,
    max_num_seqs=1,
    tensor_parallel_size=1,
)

RBLN Options: `vllm`¶

Options available under the vllm model implementation (--model-impl vllm).

`sub_block_size`¶

CLI option: --rbln-sub-block-size

Default: The prefill chunk size, that is max_num_batched_tokens.

Sets the sub-block size in tokens, which determines the granularity of prefix-cache hits.

Must be greater than 0.
Every KV cache group's block_size must be larger than this value and a multiple of it.
Setting this option while enable_sub_block_cache is disabled raises an error.

See Prefix Caching for details.

`use_custom_sampler`¶

CLI option: --rbln-use-custom-sampler

Default: True.

Runs sampling on the RBLN NPU with RBLNSampler.

Use --no-rbln-use-custom-sampler to select the upstream vLLM sampler.

`enforce_model_fp32`¶

CLI option: --rbln-enforce-model-fp32

Default: False.

Forces the model dtype to fp32 instead of the dtype from the model config.

`use_dynamic_kv_cache`¶

CLI option: --rbln-use-dynamic-kv-cache

Default: unset (automatically enabled when supported).

Sizes the KV cache using the compiled model placement and device capacity. After warm-up, the engine reads the compiled artifact's memory profile and adjusts the cache to fit the device.

The cache's block dimension is dynamic. Compilation uses a small block-count hint, leaving the final block count to be determined at runtime.

The setting has three modes:

Unset: Enables dynamic sizing, but disables it automatically if the configuration is unsupported.
--rbln-use-dynamic-kv-cache: Requires dynamic sizing. Startup fails if the cache cannot be sized for the configuration.
--no-rbln-use-dynamic-kv-cache: Disables dynamic sizing and keeps the precompile estimate.

`enable_sub_block_cache`¶

CLI option: --rbln-enable-sub-block-cache

Default: True.

Enables sub-block prefix caching at sub_block_size granularity when prefix caching is enabled. See Prefix Caching.

`specialize_moe_decode`¶

CLI option: --rbln-specialize-moe-decode

Default: True.

Compiles a specialized graph for a mixture-of-experts (MoE) model when every request in the batch is at the decode stage.

`use_moe_tokens_mask`¶

CLI option: --rbln-use-moe-tokens-mask

Default: True.

Applies the token mask to the MoE expert kernel.

`use_all2all_dispatch`¶

CLI option: --rbln-use-all2all-dispatch

Default: False.

Uses all-to-all dispatch instead of all-gather for MoE data-parallel dispatch.

`use_all2all_combine`¶

CLI option: --rbln-use-all2all-combine

Default: False.

Uses all-to-all combine instead of reduce-scatter for MoE data-parallel combine.

`decode_batch_bucket_strategy`¶

CLI option: --rbln-decode-batch-bucket-strategy

Default: exponential.

Selects how decode batch buckets are generated: exponential, linear, or manual.

Each decode batch is padded to the smallest bucket that can hold it. This lets the engine handle new batch sizes without compiling a new graph.

`decode_batch_bucket_min`¶

CLI option: --rbln-decode-batch-bucket-min

Default: 1.

Sets the smallest decode batch bucket size.

`decode_batch_bucket_step`¶

CLI option: --rbln-decode-batch-bucket-step

Default: 2.

Sets the step between decode batch buckets:

exponential: The division factor.
linear: The subtraction step.

`decode_batch_bucket_limit`¶

CLI option: --rbln-decode-batch-bucket-limit

Default: 1.

Sets the maximum number of decode batch buckets.

`decode_batch_bucket_manual_buckets`¶

CLI option: --rbln-decode-batch-bucket-manual-buckets

Default: [].

Specifies decode batch sizes when decode_batch_bucket_strategy is manual.

Values must be unique and greater than 0.

`use_w8a8`¶

CLI option: --rbln-use-w8a8

Default: False.

Enables W8A8 quantized execution.

W8A8 requires an NPU whose kernels support fp8 activations. W8A16 is supported on all RBLN NPUs.

`compile_dtype`¶

CLI option: --rbln-compile-dtype

Default: "".

Choices: "", float16, bfloat16.

Sets the execution dtype of compiled graphs on the NPU. When set to "", uses the compiler's default dtype for the target NPU.

Requires RBLN Compiler v0.12.0 or later.

Info

compile_dtype is separate from vLLM's dtype option. It controls the dtype of compiled NPU graphs.

RBLN Options: `transformers` (`optimum`)¶

Options available under the transformers (optimum) model implementation (--model-impl transformers, or its alias optimum).

`num_devices_per_local_rank`¶

CLI option: --rbln-num-devices-per-local-rank

Default: 1.

Sets the number of RBLN NPU devices assigned to one local rank (one engine process).

The compiler partitions the model across these devices using Rebellions Scalable Design (RSD). Running in a single process reduces runtime overhead.

Rebel100™: Only 1 is supported. The compiler handles partitioning across the card's four chiplets.

See Model Parallelism for details.

`sub_block_size`¶

CLI option: --rbln-sub-block-size

Default: The prefill chunk size, that is max_num_batched_tokens.

Sets the sub-block size in tokens, which determines the granularity of prefix-cache hits.

Must be a multiple of max_num_batched_tokens.
The attention block size must be a multiple of this value.

See Prefix Caching for details.

`use_custom_sampler`¶

CLI option: --rbln-use-custom-sampler

Default: True.

Runs sampling on the RBLN NPU with RBLNSampler.

Use --no-rbln-use-custom-sampler to select the upstream vLLM sampler.

`optimum_overrides`¶

CLI option: --rbln-optimum-overrides

Default: {}.

Overrides the optimum-rbln model configuration that RBLN vLLM derives from the engine settings at compile time.

For precompiled models, only device entries apply.

See RblnModelConfig for the available entries. The entries below interact with the engine settings described on this page.

`device`¶

Default: None.

If omitted, the device IDs are [0, 1, ..., N - 1], where N is the configured device count.

Specifies the device IDs to use. Set device at the top level of optimum_overrides for the default assignment, or under a submodule's configuration key to assign devices to that submodule. The number of IDs must match the configured device count.

To place submodules such as a vision encoder and a language model on different devices, see Fine-Grained Device Assignment.

`decoder_batch_sizes`¶

Default: None.

If omitted, only one decoder is compiled with a batch size of max_num_seqs. Otherwise, one decoder is compiled for each specified batch size. At each decode step, the engine selects the smallest compiled batch size that can process all requests in that step.

Batch sizes can be listed in any order, but each must be at most max_num_seqs. If the list does not include max_num_seqs, it is added automatically. See Dynamic Batch Sizes for configuration examples.

The following example sets sub_block_size and assigns devices to the visual and language submodules through optimum_overrides:

from vllm import LLM

llm = LLM(
    ...,
    additional_config={
        "sub_block_size": 256,
        "optimum_overrides": {
            "visual": {
                "device": [0],
            },
            "language_model": {
                "device": [1, 2, 3, 4],
            },
        },
    },
)

Environment Variables¶

Set the following RBLN-specific environment variables before launching the engine. For upstream vLLM environment variables, see the vLLM Environment Variables Guide.

`VLLM_RBLN_ENABLE_WARM_UP`¶

Default: True.

Runs a model warm-up pass at engine startup so the first request is not delayed by one-time initialization. Set to False to skip warm-up.

`VLLM_RBLN_COMPILE_STRICT_MODE`¶

Default: True.

Raises an error if graph compilation fails. Set to False to allow CPU fallback where supported.

`VLLM_RBLN_COMPILE_ONLY`¶

Default: False.

Compiles model graphs into the compile cache without executing them. Use this mode to prepare cached artifacts for later execution, including on a host without an NPU.

Info

On a host without an NPU, set RBLN_FORCE_NPU_NAME to the target NPU name (for example, RBLN-CA25). The compile cache must be enabled (VLLM_DISABLE_COMPILE_CACHE=0).

`VLLM_RBLN_DISABLE_OFFLOAD`¶

Default: True.

Applies to: vllm only.

When True, keeps host-side weight data in memory during model loading and warm-up. Set to False to allow this data to be stored on disk.

`VLLM_RBLN_METRICS`¶

Default: False.

Collects per-step latency metrics in each worker and prints a performance report when the engine shuts down.

`VLLM_RBLN_METRICS_FILE`¶

Default: "".

Applies to: transformers (optimum) only.

Requires VLLM_RBLN_METRICS=True.

Saves the performance report to a text file. When set to "", the report is output only to the log.

Each worker writes to a separate file: report.log becomes report.<PID>.log.

`VLLM_RBLN_METRICS_DIR`¶

Default: "".

Applies to: vllm only.

Requires VLLM_RBLN_METRICS=True.

Saves performance reports as JSON files in this directory. When set to "", reports are output only to the log.

File names reflect worker ranks, such as metrics_tp0.json. Without multiple ranks, the file name is metrics.json.

Deprecated Configuration¶

Removal in RBLN vLLM v0.14.0

The environment variables and additional_config keys listed below are deprecated and will be removed in RBLN vLLM v0.14.0. Until then, they remain supported and emit a deprecation warning when used.

Environment Variables¶

Use the corresponding RBLN parameter instead of a deprecated environment variable. The table below maps deprecated environment variables to their replacement parameters.

If both a deprecated environment variable and the parameter listed under Use instead are set, the parameter value takes precedence.

Variable	Use instead
`VLLM_RBLN_USE_VLLM_MODEL`	`model_impl`
`VLLM_RBLN_NUM_DEVICES_PER_LOCAL_RANK`, `VLLM_RBLN_TP_SIZE`	`num_devices_per_local_rank`
`VLLM_RBLN_SAMPLER`	`use_custom_sampler`
`VLLM_RBLN_SUB_BLOCK_CACHE`	`enable_sub_block_cache`
`VLLM_RBLN_DISPATCH_ALL2ALL`	`use_all2all_dispatch`
`VLLM_RBLN_COMBINE_ALL2ALL`	`use_all2all_combine`

`additional_config` Keys¶

For the transformers (optimum) model implementation, replace the following keys in additional_config with their current names.

Deprecated key	Use instead
`rbln_config`	`optimum_overrides`
`prefix_block_size`	`sub_block_size`

Configuration Guide¶

Overview¶

Configuration Methods¶

Model Implementation¶

Engine Arguments¶

model_impl¶

block_size¶

max_model_len¶

max_num_seqs¶

max_num_batched_tokens¶

tensor_parallel_size¶

RBLN Options: vllm¶

sub_block_size¶

use_custom_sampler¶

enforce_model_fp32¶

use_dynamic_kv_cache¶

enable_sub_block_cache¶

specialize_moe_decode¶

use_moe_tokens_mask¶

use_all2all_dispatch¶

use_all2all_combine¶

decode_batch_bucket_strategy¶

decode_batch_bucket_min¶

decode_batch_bucket_step¶

decode_batch_bucket_limit¶

decode_batch_bucket_manual_buckets¶

use_w8a8¶

compile_dtype¶

RBLN Options: transformers (optimum)¶

num_devices_per_local_rank¶

sub_block_size¶

use_custom_sampler¶

optimum_overrides¶

device¶

decoder_batch_sizes¶

Environment Variables¶

VLLM_RBLN_ENABLE_WARM_UP¶

VLLM_RBLN_COMPILE_STRICT_MODE¶

VLLM_RBLN_COMPILE_ONLY¶

VLLM_RBLN_DISABLE_OFFLOAD¶

VLLM_RBLN_METRICS¶

VLLM_RBLN_METRICS_FILE¶

VLLM_RBLN_METRICS_DIR¶

Deprecated Configuration¶

Environment Variables¶

additional_config Keys¶

`model_impl`¶

`block_size`¶

`max_model_len`¶

`max_num_seqs`¶

`max_num_batched_tokens`¶

`tensor_parallel_size`¶

RBLN Options: `vllm`¶

`sub_block_size`¶

`use_custom_sampler`¶

`enforce_model_fp32`¶

`use_dynamic_kv_cache`¶

`enable_sub_block_cache`¶

`specialize_moe_decode`¶

`use_moe_tokens_mask`¶

`use_all2all_dispatch`¶

`use_all2all_combine`¶

`decode_batch_bucket_strategy`¶

`decode_batch_bucket_min`¶

`decode_batch_bucket_step`¶

`decode_batch_bucket_limit`¶

`decode_batch_bucket_manual_buckets`¶

`use_w8a8`¶

`compile_dtype`¶

RBLN Options: `transformers` (`optimum`)¶

`num_devices_per_local_rank`¶

`sub_block_size`¶

`use_custom_sampler`¶

`optimum_overrides`¶

`device`¶

`decoder_batch_sizes`¶

`VLLM_RBLN_ENABLE_WARM_UP`¶

`VLLM_RBLN_COMPILE_STRICT_MODE`¶

`VLLM_RBLN_COMPILE_ONLY`¶

`VLLM_RBLN_DISABLE_OFFLOAD`¶

`VLLM_RBLN_METRICS`¶

`VLLM_RBLN_METRICS_FILE`¶

`VLLM_RBLN_METRICS_DIR`¶

`additional_config` Keys¶