RBLN vLLM: A vLLM Plugin for RBLN NPU¶

RBLN vLLM (vllm-rbln) is a hardware plugin for vLLM. It runs large language model inference and serving on RBLN NPUs.

Installation¶

Install the latest rebel-compiler before installing RBLN vLLM.

$ pip install \
  --extra-index-url https://pypi.rbln.ai/simple \
  rebel-compiler==0.12.0

Install RBLN vLLM from PyPI, or build it from source.

See the Release Notes for version information and changes.

PyPI install¶

Install the latest release with pip:

$ pip install \
  --extra-index-url https://wheels.vllm.ai/0.30.0/cpu \
  --extra-index-url https://download.pytorch.org/whl/cpu \
  vllm-rbln==0.12.0

Source-code install¶

1. Clone the `vllm` and `vllm-rbln` repositories¶

The required vllm version may differ from the vllm-rbln version.

$ git clone -b vv0.30.0 --single-branch https://github.com/vllm-project/vllm.git
$ git clone https://github.com/RBLN-SW/vllm-rbln.git

2. Install `vllm`¶

Set VLLM_TARGET_DEVICE=empty when building vLLM for RBLN NPUs.

$ cd vllm/
$ VLLM_TARGET_DEVICE=empty pip install -e .
$ cd ..

3. Install `vllm-rbln`¶

$ cd vllm-rbln
$ pip install -e .
$ cd ..

Optional dependencies¶

Install the additional dependencies for the features you use.

KV Cache Offloading and Disaggregated Prefill / Decode¶

The kv_connectors extra installs nixl-rbln and lmcache-rbln, which KV Cache Offloading and Disaggregated Prefill / Decode require. These dependencies are not included in the default installation.

$ pip install \
  --extra-index-url https://pypi.rbln.ai/simple \
  --extra-index-url https://wheels.vllm.ai/0.30.0/cpu \
  --extra-index-url https://download.pytorch.org/whl/cpu \
  "vllm-rbln[kv_connectors]==0.12.0"

For a source-code install, run pip install -e '.[kv_connectors]' in the vllm-rbln directory.

Disaggregated Encoder¶

Disaggregated Encoder requires nixl in the Python environments that run the encoder and PD instances. Install it in addition to vllm-rbln:

$ pip install nixl

Speech recognition¶

The audio extra installs the audio decoders required by Whisper and other models served through /v1/audio/transcriptions. Without these dependencies, the server starts, but the first transcription request fails.

$ pip install \
  --extra-index-url https://pypi.rbln.ai/simple \
  --extra-index-url https://wheels.vllm.ai/0.30.0/cpu \
  --extra-index-url https://download.pytorch.org/whl/cpu \
  "vllm-rbln[audio]==0.12.0"

For a source-code install, run pip install -e '.[audio]' in the vllm-rbln directory.

Design Overview¶

RBLN vLLM runs a model through one of two model implementations, vllm and transformers. These names are values of upstream vLLM's --model-impl option. The transformers implementation is also called optimum because the optimum-rbln package provides it. The implementations differ in model code, compilation workflow, workers, scheduling, and supported features. The Feature Matrix lists the feature support of each.

Both implementations compile the model with the RBLN Compiler (rebel-compiler) and cache the result, so a later run with the same configuration reuses it instead of compiling again. See Compile Cache for details.

Aspect	`vllm`	`transformers` (`optimum`)
Model implementation source	vLLM's own model definitions	`optimum-rbln`
Compilation	At engine startup	At engine startup, or ahead of time with `optimum-rbln`
Multi-device	TP, PP, EP and DP across processes	RSD, inside one process
Devices	Rebel100™	Rebel100™ and ATOM™

`vllm`¶

The model code comes from vLLM's own model definitions. The engine compiles it through torch.compile with the rbln backend at engine startup.

Multi-device execution uses vLLM's parallelism mechanisms: tensor parallelism (TP), pipeline parallelism (PP), expert parallelism (EP), and data parallelism (DP). See Model Parallelism for details.

`transformers` (`optimum`)¶

The model code comes from optimum-rbln, which provides an RBLN model class for each supported architecture, such as RBLNLlamaForCausalLM.

Set model to either a HuggingFace model ID or the path to a directory containing a model compiled separately with optimum-rbln. RBLN vLLM compiles the model at engine startup when given a model ID, or loads the compiled model when given a directory path.

Multi-device execution uses Rebellions Scalable Design (RSD), which lets the RBLN Compiler partition the model across devices rather than relying on vLLM-level model partitioning.

Model Implementation Selection¶

Choose the implementation with vLLM's --model-impl option, or with the model_impl argument of LLM().

Value	Behavior
`auto` (default)	Uses the `transformers` implementation if `optimum-rbln` supports the model's architecture, or the `vllm` implementation otherwise.
`vllm`	Uses the `vllm` implementation.
`transformers` (alias: `optimum`)	Uses the `transformers` (`optimum`) implementation.

vllm serve --help shows upstream vLLM's description of --model-impl, which does not fully match its behavior in RBLN vLLM. In particular, auto in RBLN vLLM prefers transformers, whereas upstream's auto prefers vllm.

RBLN-specific options use the --rbln-* prefix. Each option applies to only one model implementation. Options that do not apply to the selected implementation are rejected when the engine configuration is built.

The environment variables replaced by --rbln-* options are deprecated. See the Configuration Guide for the affected VLLM_RBLN_* variables and which options to use instead.

Device support

The vllm model implementation runs on Rebel100™ only. On an ATOM™ host the engine raises an error naming the device, whether --model-impl vllm selected the implementation or auto resolved to it. Serving on ATOM™ therefore requires an architecture the transformers (optimum) implementation supports.

Features¶

RBLN vLLM feature guides:

Supported Architectures¶

The tables below list the architectures RBLN vLLM supports, grouped by model implementation. The model architecture is specified in the checkpoint's config.json file.

`vllm` Implementation Architectures¶

These architectures run on the vllm model implementation, --model-impl vllm.

Architecture	Recipe
GptOssForCausalLM	gpt-oss-120b
MiniMaxM2ForCausalLM	MiniMax-M2.7
DeepseekV32ForCausalLM	DeepSeek-V3.2
AXK2ForCausalLM	A.X-K2

`transformers` (`optimum`) Implementation Architectures¶

The tables are organized by device family. The Rebel100™ tab lists the architectures verified on Rebel100™. The ATOM™ tab lists every architecture the implementation supports.

Decoder-only Models¶

Rebel100™ATOM™

Architecture	RBLN Model Class	Recipe
LlamaForCausalLM	RBLNLlamaForCausalLM	Llama-3.1-8B-Instruct
GemmaForCausalLM	RBLNGemmaForCausalLM	gemma-7b-it
Gemma2ForCausalLM	RBLNGemma2ForCausalLM	gemma-2-9b-it
PhiForCausalLM	RBLNPhiForCausalLM	phi-2
OPTForCausalLM	RBLNOPTForCausalLM	opt-6.7b
GPT2LMHeadModel	RBLNGPT2LMHeadModel	gpt2
MistralForCausalLM	RBLNMistralForCausalLM	Mistral-7B-Instruct-v0.3
ExaoneForCausalLM	RBLNExaoneForCausalLM	EXAONE-3.5-7.8B-Instruct
Qwen2ForCausalLM	RBLNQwen2ForCausalLM	Qwen2.5-7B-Instruct
Qwen3ForCausalLM	RBLNQwen3ForCausalLM	Qwen3-8B
GptOssForCausalLM	RBLNGptOssForCausalLM	gpt-oss-20b

Architecture	RBLN Model Class	Recipe
LlamaForCausalLM	RBLNLlamaForCausalLM	Llama-3.1-8B-Instruct
GemmaForCausalLM	RBLNGemmaForCausalLM	gemma-7b-it
Gemma2ForCausalLM	RBLNGemma2ForCausalLM	gemma-2-9b-it
PhiForCausalLM	RBLNPhiForCausalLM	phi-2
OPTForCausalLM	RBLNOPTForCausalLM	opt-6.7b
GPT2LMHeadModel	RBLNGPT2LMHeadModel	gpt2
MistralForCausalLM	RBLNMistralForCausalLM	Mistral-7B-Instruct-v0.3
ExaoneForCausalLM	RBLNExaoneForCausalLM	EXAONE-3.5-7.8B-Instruct
Qwen2ForCausalLM	RBLNQwen2ForCausalLM	Qwen2.5-7B-Instruct
Qwen3ForCausalLM	RBLNQwen3ForCausalLM	Qwen3-8B
Qwen3MoeForCausalLM	RBLNQwen3MoeForCausalLM	Qwen3-30B-A3B
GptOssForCausalLM	RBLNGptOssForCausalLM	gpt-oss-20b

Encoder-Decoder Models¶

Architecture	RBLN Model Class	Recipe
WhisperForConditionalGeneration	RBLNWhisperForConditionalGeneration	whisper-large-v3

Change

As of RBLN vLLM v0.10.1, V0 has been deprecated. Consequently, Whisper is now the only supported encoder–decoder model, and support for all other encoder–decoder models has been removed.
For more information, see the vLLM V1 User Guide.

Multimodal Language Models¶

Rebel100™ATOM™

Architecture	RBLN Model Class	Recipe
LlavaNextForConditionalGeneration	RBLNLlavaNextForConditionalGeneration	llava-v1.6-mistral-7b-hf
Qwen2VLForConditionalGeneration	RBLNQwen2VLForConditionalGeneration	Qwen2-VL-7B-Instruct
Qwen2_5_VLForConditionalGeneration	RBLNQwen2_5_VLForConditionalGeneration	Qwen2.5-VL-7B-Instruct
Qwen3VLForConditionalGeneration	RBLNQwen3VLForConditionalGeneration	Qwen3-VL-8B-Instruct
Qwen3_5ForConditionalGeneration	RBLNQwen3_5ForConditionalGeneration	Qwen3.5-9B, Qwen3.6-27B
Idefics3ForConditionalGeneration	RBLNIdefics3ForConditionalGeneration	Idefics3-8B-Llama3
Gemma3ForConditionalGeneration	RBLNGemma3ForConditionalGeneration	gemma-3-12b-it
Gemma4ForConditionalGeneration	RBLNGemma4ForConditionalGeneration	gemma-4-31B-it
Blip2ForConditionalGeneration	RBLNBlip2ForConditionalGeneration	blip2-opt-2.7b
PaliGemmaForConditionalGeneration	RBLNPaliGemmaForConditionalGeneration	paligemma-3b-mix-224

Architecture	RBLN Model Class	Recipe
LlavaNextForConditionalGeneration	RBLNLlavaNextForConditionalGeneration	llava-v1.6-mistral-7b-hf
Qwen2VLForConditionalGeneration	RBLNQwen2VLForConditionalGeneration	Qwen2-VL-7B-Instruct
Qwen2_5_VLForConditionalGeneration	RBLNQwen2_5_VLForConditionalGeneration	Qwen2.5-VL-7B-Instruct
Qwen3VLForConditionalGeneration	RBLNQwen3VLForConditionalGeneration	Qwen3-VL-8B-Instruct
Qwen3_5ForConditionalGeneration	RBLNQwen3_5ForConditionalGeneration	Qwen3.5-9B, Qwen3.6-27B
Idefics3ForConditionalGeneration	RBLNIdefics3ForConditionalGeneration	Idefics3-8B-Llama3
Gemma3ForConditionalGeneration	RBLNGemma3ForConditionalGeneration	gemma-3-12b-it
Gemma4ForConditionalGeneration	RBLNGemma4ForConditionalGeneration	gemma-4-31B-it
LlavaForConditionalGeneration	RBLNLlavaForConditionalGeneration	llava-1.5-7b-hf
Blip2ForConditionalGeneration	RBLNBlip2ForConditionalGeneration	blip2-opt-2.7b
PaliGemmaForConditionalGeneration	RBLNPaliGemmaForConditionalGeneration	paligemma-3b-mix-224
PaliGemmaForConditionalGeneration	RBLNPaliGemmaForConditionalGeneration	paligemma2-3b-mix-224
Exaone4_5_ForConditionalGeneration	RBLNExaone4_5_ForConditionalGeneration	EXAONE-4.5-33B

Pooling Models¶

Rebel100™ATOM™

Not supported.

Architecture	RBLN Model Class	Recipe
BertModel	RBLNBertModel	bge-base-en-v1.5
XLMRobertaModel	RBLNXLMRobertaModel	bge-m3
XLMRobertaForSequenceClassification	RBLNXLMRobertaForSequenceClassification	bge-reranker-v2-m3
RobertaForSequenceClassification	RBLNRobertaForSequenceClassification	twitter-roberta-base-emotion
Qwen3ForSequenceClassification	RBLNQwen3ForCausalLM	Qwen3-Reranker-4B
Qwen3Model	RBLNQwen3Model	Qwen3-Embedding-4B

RBLN vLLM: A vLLM Plugin for RBLN NPU¶

Installation¶

PyPI install¶

Source-code install¶

1. Clone the vllm and vllm-rbln repositories¶

2. Install vllm¶

3. Install vllm-rbln¶

Optional dependencies¶

KV Cache Offloading and Disaggregated Prefill / Decode¶

Disaggregated Encoder¶

Speech recognition¶

Design Overview¶

vllm¶

transformers (optimum)¶

Model Implementation Selection¶

Features¶

Supported Architectures¶

vllm Implementation Architectures¶

transformers (optimum) Implementation Architectures¶

Decoder-only Models¶

Encoder-Decoder Models¶

Multimodal Language Models¶

Pooling Models¶

1. Clone the `vllm` and `vllm-rbln` repositories¶

2. Install `vllm`¶

3. Install `vllm-rbln`¶

`vllm`¶

`transformers` (`optimum`)¶

`vllm` Implementation Architectures¶

`transformers` (`optimum`) Implementation Architectures¶