Skip to content

Qwen3.5-4B on RBLN NPU

Install the RBLN Driver on the host. An RBLN Portal account is required to install the RBLN Compiler (rebel-compiler). See the installation matrix for compatible package versions.

1
2
3
$ pip install \
  --extra-index-url https://pypi.rbln.ai/simple \
  rebel-compiler==0.12.0

1
2
3
4
$ pip install \
  --extra-index-url https://wheels.vllm.ai/0.30.0/cpu \
  --extra-index-url https://download.pytorch.org/whl/cpu \
  vllm-rbln==0.12.0
Guide — overview and configuration

Overview

Qwen/Qwen3.5-4B is a natively multimodal (vision-language) model with a hybrid decoder — linear_attention layers (GatedDeltaNet: conv_state + recurrent_state) interleaved with gated full_attention (paged KV) layers. On RBLN NPUs, the RBLN vLLM plugin compiles the language model and vision encoder on first run, then serves them over vLLM's OpenAI-compatible API. Include images using image_url content items in chat completion requests.

Configuration

--rbln-num-devices-per-local-rank sets how many NPU devices one engine process uses, and the RBLN Compiler partitions the model for those devices with Rebellions Scalable Design (RSD). See Model Parallelism for how RSD applies on each NPU. The vision encoder is configured through --additional-config (optimum_overrides.visual). Changing these settings recompiles the model on the next start. Compile Cache lists every setting that does.

See the Configuration Guide for all engine arguments, and Fine-Grained Device Assignment for placing the vision encoder and the language model on different devices.

Notes and limitations

  • The sequence length, NPU count, and vision settings are fixed at compilation; changing any of these settings triggers recompilation at the next startup.
  • --additional-config carries the vision encoder settings (optimum_overrides.visual). For multi-image or video workloads, adjust the encoder's max_seq_len, and see Fine-Grained Device Assignment to place the encoder on its own devices.
  • Attention is fixed to flash_attn — the hybrid backbone's multi-window prefill is mis-lowered by the eager op (partial-last-window attention across prior KV blocks can flip the argmax); flash_attn is correct (see vllm-rbln#795). Do not override.
  • Unverified configuration. Parameters mirror the reference compile configuration for this model line; optimum-rbln#631 introduces RBLNQwen3_5ForConditionalGeneration and vllm-rbln#795 wires qwen3_5 into the transformers model implementation.