Skip to content

Qwen2.5-VL-7B-Instruct on RBLN NPU

Install the RBLN Driver on the host. An RBLN Portal account is required to install the RBLN Compiler (rebel-compiler). See the installation matrix for compatible package versions.

1
2
3
$ pip install \
  --extra-index-url https://pypi.rbln.ai/simple \
  rebel-compiler==0.12.0

1
2
3
4
$ pip install \
  --extra-index-url https://wheels.vllm.ai/0.30.0/cpu \
  --extra-index-url https://download.pytorch.org/whl/cpu \
  vllm-rbln==0.12.0
Guide — overview and configuration

Overview

Qwen/Qwen2.5-VL-7B-Instruct is a vision-language model. On RBLN NPUs, the RBLN vLLM plugin compiles the language model and vision encoder on first run, then serves them over vLLM's OpenAI-compatible API. Include images using image_url content items in chat completion requests.

Configuration

--rbln-num-devices-per-local-rank sets how many NPU devices one engine process uses, and the RBLN Compiler partitions the model for those devices with Rebellions Scalable Design (RSD). See Model Parallelism for how RSD applies on each NPU. The vision encoder is configured through --additional-config (optimum_overrides.visual). Changing these settings recompiles the model on the next start. Compile Cache lists every setting that does.

See the Configuration Guide for all engine arguments, and Fine-Grained Device Assignment for placing the vision encoder and the language model on different devices.

Notes and limitations

  • The sequence length, NPU count, and vision settings are fixed at compilation; changing any of these settings triggers recompilation at the next startup.
  • --additional-config carries the vision encoder settings (optimum_overrides.visual). For multi-image or video workloads, adjust the encoder's max_seq_len, and see Fine-Grained Device Assignment to place the encoder on its own devices.