Skip to content

Quick Start

The RBLN SDK enables developers to run deep learning models efficiently on the RBLN Neural Processing Unit (NPU). This guide walks you through the complete workflow—from setup to inference—using practical examples with PyTorch and HuggingFace models.

The RBLN SDK is a single SDK for every RBLN NPU product line: Rebel100™ and ATOM™ install the same packages at the same versions, so setup and installation are common to both. Only compilation and inference differ by product line. Follow these steps:

  1. Setup & Installation
  2. Compilation and Inference

1. Setup & Installation

System Requirements

  • Ubuntu 22.04 LTS (Debian bullseye) or higher
  • Python: 3.10–3.14
  • A system equipped with an RBLN NPU
  • RBLN Driver

This tutorial assumes that the above system requirements have been met. To check the RBLN Driver installation and the presence of the RBLN NPU, use the rbln-smi CLI as follows:

$ rbln-smi
+-------------------------------------------------------------------------------------------------+
|                                 Device Information KMD ver: X.X.XX                              |
+-----+-----------+---------+---------------+------+---------+------+---------------------+-------+
| NPU |    Name   | Device  |   PCI BUS ID  | Temp |  Power  | Perf |  Memory(used/total) |  Util |
+-----+-----------+---------+---------------+------+---------+------+---------------------+-------+
| 0   | RBLN-CA25 | rbln0   |  0000:05:00.0 |  37C |  54.7W  |  P14 |    0.0B / 15.7GiB   |   0.0 |
| 1   |           | rbln1   |  0000:06:00.0 |  34C |         |  P14 |    0.0B / 15.7GiB   |   0.0 |
| 2   |           | rbln2   |  0000:07:00.0 |  28C |         |  P14 |    0.0B / 15.7GiB   |   0.0 |
| 3   |           | rbln3   |  0000:08:00.0 |  26C |         |  P14 |    0.0B / 15.7GiB   |   0.0 |
+-----+-----------+---------+---------------+------+---------+------+---------------------+-------+
+-------------------------------------------------------------------------------------------------+
|                                       Context Information                                       |
+-----+---------------------+--------------+-----+----------+------+---------------------+--------+
| NPU | Process             |     PID      | CTX | Priority | PTID |           Memalloc  | Status |
+-----+---------------------+--------------+-----+----------+------+---------------------+--------+
| N/A | N/A                 |     N/A      | N/A |   N/A    | N/A  |                N/A  |  N/A   |
+-----+---------------------+--------------+-----+----------+------+---------------------+--------+

Install the RBLN SDK

The RBLN SDK is distributed as .whl packages. Installing rebel-compiler requires an RBLN Portal account.

1
2
3
4
5
6
7
8
$ pip install \
  --extra-index-url https://pypi.rbln.ai/simple \
  rebel-compiler==0.12.0
$ pip install \
  --extra-index-url https://wheels.vllm.ai/0.30.0/cpu \
  --extra-index-url https://pypi.rbln.ai/simple \
  --extra-index-url https://download.pytorch.org/whl/cpu \
  "vllm-rbln[kv-connectors]==0.12.0"

Note

The command above installs vllm-rbln and also installs optimum-rbln, so RBLN Optimum is ready to use in the same step.

2. Compilation and Inference

The RBLN SDK runs models built with frameworks such as tensorflow, torch, transformers, and diffusers. The available options differ by product line. Choose the tab for your product line, then choose the option that matches your model and follow only its steps. Each option runs from start to finish on its own.

Option Model SDK component
Option 1: vLLM Model Large language model served through an OpenAI-compatible API • RBLN Compiler (rebel-compiler)
• RBLN vLLM (vllm-rbln)
Option 2: HuggingFace Model transformers model • RBLN Compiler (rebel-compiler)
• RBLN Optimum (optimum-rbln)
Option 3: Custom PyTorch / TensorFlow Model Custom PyTorch or TensorFlow model • RBLN Compiler (rebel-compiler)

Option 1: vLLM Model

Large language models can be deployed with RBLN vLLM (vllm-rbln), installed in Install the RBLN SDK. To run a model, pass its HuggingFace model ID to vllm serve. RBLN vLLM compiles the model for the RBLN NPU the first time the server starts, then serves it through the vLLM OpenAI-compatible API.

1. Start the Server

The example below serves the Mixture-of-Experts (MoE) model openai/gpt-oss-120b on a single Rebel100™ device (RBLN-CR13). --data-parallel-size 1 runs the model on a single RBLN NPU. --model-impl vllm loads the model through the vLLM-native model path:

1
2
3
4
5
6
7
8
vllm serve openai/gpt-oss-120b \
    --served-model-name gpt-oss-120b \
    --model-impl vllm \
    --data-parallel-size 1 \
    --quantization mxfp4 \
    --max-model-len 131072 \
    --block-size 1024 \
    --max-num-seqs 1

The values of --data-parallel-size, --quantization, --max-model-len, --block-size, and --max-num-seqs above are fixed in the compiled model and cannot be changed afterward; changing any of them recompiles the model on the next start. RBLN vLLM caches the compiled model, so a later start with the same values reuses it. For more details, refer to Compile Cache. The gpt-oss-120b recipe covers this model's configuration in detail. vLLM Recipes lists recipes for more models.

2. Send a Request

Once the server is ready, send requests to the vLLM OpenAI-compatible API. The model field matches the --served-model-name value:

1
2
3
4
5
6
curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
      "model": "gpt-oss-120b",
      "messages": [{"role": "user", "content": "Hello"}]
    }'

Option 2: HuggingFace Model

Models built with HuggingFace libraries like transformers are compiled and run with optimum-rbln, an RBLN-optimized extension of HuggingFace APIs.

1. Compile the Model

The example below compiles the meta-llama/Llama-3.2-3B-Instruct model for a single Rebel100™ device using RBLNAutoModelForCausalLM, which adapts HuggingFace causal language models for the RBLN NPU. You do not need to construct or import the model separately. Pass the HuggingFace model ID and export=True to the RBLN Optimum from_pretrained() method, which loads and compiles the model in one step:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
import os

from optimum.rbln import RBLNAutoModelForCausalLM

model_id = "meta-llama/Llama-3.2-3B-Instruct"

# Compile and export
model = RBLNAutoModelForCausalLM.from_pretrained(
    model_id=model_id,
    export=True,  # export a PyTorch model to an RBLN model with optimum
    rbln_batch_size=1,
    rbln_max_seq_len=131_072,  # default "max_position_embeddings"
    rbln_num_devices=1,
    rbln_attn_impl="flash_attn",
    rbln_kvcache_partition_len=8_192,  # Length of KV cache partitions for flash attention
)

# Save compiled results to disk
model.save_pretrained(os.path.basename(model_id))

2. Run Inference

Inference uses the same RBLNAutoModelForCausalLM class. Unlike compilation in the previous step, set export=False and use the local path of the compiled model:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
import os

from optimum.rbln import RBLNAutoModelForCausalLM
from transformers import AutoTokenizer

model_id = "meta-llama/Llama-3.2-3B-Instruct"
text = "Hey, are you conscious? Can you talk to me?"

# Load compiled model
model = RBLNAutoModelForCausalLM.from_pretrained(
    model_id=os.path.basename(model_id),
    export=False,
    rbln_device=0,
)

# Prepare inputs
tokenizer = AutoTokenizer.from_pretrained(model_id)
tokenizer.pad_token = tokenizer.eos_token
conversation = [{"role": "user", "content": text}]
text = tokenizer.apply_chat_template(
    conversation, add_generation_prompt=True, tokenize=False
)
inputs = tokenizer(text, return_tensors="pt", padding=True)

# Generate tokens
output_sequence = model.generate(
    inputs.input_ids,
    attention_mask=inputs.attention_mask,
    max_length=131_072,
)

input_len = inputs.input_ids.shape[-1]
generated_texts = tokenizer.decode(
    output_sequence[0][input_len:],
    skip_special_tokens=True,
    clean_up_tokenization_spaces=True,
)

# Show result
print(f"Result: {generated_texts}")

Beyond this example, RBLN Model Zoo provides examples for more models.

Option 3: Custom PyTorch / TensorFlow Model

Custom models, such as PyTorch or TensorFlow models, are compiled with rebel-compiler and run with its rebel.Runtime() API.

1. Construct or Import a Model

The example below defines a simple PyTorch model:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
import torch
import torch.nn as nn


class SimpleConvBNRelu(nn.Module):
    def __init__(
        self, in_channels=3, out_channels=16, kernel_size=3, stride=1, padding=1
    ):
        super(SimpleConvBNRelu, self).__init__()
        self.conv = nn.Conv2d(in_channels, out_channels, kernel_size, stride, padding)
        self.bn = nn.BatchNorm2d(out_channels)
        self.relu = nn.ReLU(inplace=True)

    def forward(self, x):
        x = self.conv(x)
        x = self.bn(x)
        x = self.relu(x)
        return x


model = SimpleConvBNRelu()
print(model)

2. Compile the Model

rebel-compiler converts the model into an RBLN NPU-compatible format. For PyTorch models, like the SimpleConvBNRelu in the example below, use the rebel.compile_from_torch() function:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
import rebel
import torch
import torch.nn as nn


class SimpleConvBNRelu(nn.Module):
    def __init__(
        self, in_channels=3, out_channels=16, kernel_size=3, stride=1, padding=1
    ):
        super(SimpleConvBNRelu, self).__init__()
        self.conv = nn.Conv2d(in_channels, out_channels, kernel_size, stride, padding)
        self.bn = nn.BatchNorm2d(out_channels)
        self.relu = nn.ReLU(inplace=True)

    def forward(self, x):
        x = self.conv(x)
        x = self.bn(x)
        x = self.relu(x)
        return x


model = SimpleConvBNRelu().eval()

x = torch.rand(1, 3, 224, 224)

input_info = [
    ("x", list(x.shape), torch.float32),
]

compiled_model = rebel.compile_from_torch(model, input_info)
compiled_model.save("simple_conv_bn_relu.rbln")

PyTorch models can also be compiled with torch.compile() and the rbln backend. This path needs no .rbln file and no rebel.Runtime() step below. Calling the compiled model compiles it for the RBLN NPU and runs it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
import rebel  # registers the "rbln" backend for torch.compile
import torch
import torch.nn as nn


class SimpleConvBNRelu(nn.Module):
    def __init__(
        self, in_channels=3, out_channels=16, kernel_size=3, stride=1, padding=1
    ):
        super(SimpleConvBNRelu, self).__init__()
        self.conv = nn.Conv2d(in_channels, out_channels, kernel_size, stride, padding)
        self.bn = nn.BatchNorm2d(out_channels)
        self.relu = nn.ReLU(inplace=True)

    def forward(self, x):
        x = self.conv(x)
        x = self.bn(x)
        x = self.relu(x)
        return x


torch.manual_seed(0)
model = SimpleConvBNRelu().eval()

compiled_model = torch.compile(
    model,
    backend="rbln",
    options={"cache_dir": "./.cache", "mode": "strict"},
    dynamic=False,
)

x = torch.rand(1, 3, 224, 224)

with torch.no_grad():
    result = compiled_model(x)  # first call: compiles, or loads from cache_dir
    result = compiled_model(x)  # warm path: runs the compiled model

print("--- Result ---")
print(result)
  • First call: compiles the model and saves it to cache_dir. If cache_dir already holds a compiled model with the same weights and input shape, the first call loads it instead of compiling. torch.manual_seed(0) keeps the example's random weights the same across runs, so the second run of the script hits the cache.
  • Later calls: take the warm path. A call with the same input shape runs the compiled model on the RBLN NPU without compiling again.

"mode": "strict" raises an error when the rbln backend fails to compile the graph. Without it, torch.compile() falls back to eager execution on the CPU and only logs a warning. For all options, refer to the torch.compile API.

3. Run Inference

Load the compiled model with the rebel.Runtime() API and run it on the RBLN NPU:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
import rebel
import torch

x = torch.rand(1, 3, 224, 224)

module = rebel.Runtime("simple_conv_bn_relu.rbln")
inputs = x.numpy()
result = module.run(inputs)

print("--- Input ---")
print(inputs)
print("--- Result ---")
print(result)
Option Model SDK component
Option 1: vLLM Model Large language model served through an OpenAI-compatible API • RBLN Compiler (rebel-compiler)
• RBLN vLLM (vllm-rbln)
Option 2: HuggingFace Model transformers or diffusers model • RBLN Compiler (rebel-compiler)
• RBLN Optimum (optimum-rbln)
Option 3: Custom PyTorch / TensorFlow Model Custom PyTorch or TensorFlow model • RBLN Compiler (rebel-compiler)

Option 1: vLLM Model

Large language models can be deployed with RBLN vLLM (vllm-rbln), installed in Install the RBLN SDK. To run a model, pass its HuggingFace model ID to vllm serve. RBLN vLLM compiles the model for the RBLN NPU the first time the server starts, then serves it through the vLLM OpenAI-compatible API.

1. Start the Server

The example below serves the dense model Qwen/Qwen3-0.6B on a single ATOM™ device (RBLN-CA22 or RBLN-CA25). --rbln-num-devices-per-local-rank sets how many NPUs the model is partitioned across with tensor parallelism:

1
2
3
4
5
6
7
vllm serve Qwen/Qwen3-0.6B \
    --model-impl transformers \
    --served-model-name Qwen3-0.6B \
    --rbln-num-devices-per-local-rank 1 \
    --max-model-len 40960 \
    --block-size 4096 \
    --max-num-seqs 1

On ATOM™, vllm serve runs the model through the transformers (optimum) model implementation, not the vllm implementation. For more details, refer to the transformers (optimum) section of the RBLN vLLM documentation.

The values of --rbln-num-devices-per-local-rank, --max-model-len, --block-size, and --max-num-seqs above are fixed in the compiled model and cannot be changed afterward; changing any of them recompiles the model on the next start. RBLN vLLM caches the compiled model, so a later start with the same values reuses it. For more details, refer to Compile Cache. The Qwen3-0.6B recipe covers this model's configuration in detail. vLLM Recipes lists recipes for more models.

2. Send a Request

Once the server is ready, send requests to the vLLM OpenAI-compatible API. The model field matches the --served-model-name value:

1
2
3
4
5
6
curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
      "model": "Qwen3-0.6B",
      "messages": [{"role": "user", "content": "Hello"}]
    }'

Option 2: HuggingFace Model

Models built with HuggingFace libraries like transformers or diffusers are compiled and run with optimum-rbln, an RBLN-optimized extension of HuggingFace APIs.

1. Compile the Model

The example below compiles the stabilityai/sdxl-turbo model using RBLNAutoPipelineForText2Image, which adapts HuggingFace text-to-image pipelines for the RBLN NPU. You do not need to construct or import the model separately. Pass the HuggingFace model ID and export=True to the RBLN Optimum from_pretrained() method, which loads and compiles the model in one step:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
import os

import torch
from optimum.rbln import RBLNAutoPipelineForText2Image

model_id = "stabilityai/sdxl-turbo"

# Compile and export
# As SDXL-turbo does not use guidance_scale, we disable it with rbln_guidance_scale=0.0
pipe = RBLNAutoPipelineForText2Image.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    export=True,  # export a PyTorch model to an RBLN model with optimum
    rbln_guidance_scale=0.0,
)

# Save compiled results to disk
pipe.save_pretrained(os.path.basename(model_id))

2. Run Inference

Inference uses the same RBLNAutoPipelineForText2Image class. Unlike compilation in the previous step, set export=False and use the local path of the compiled model:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
import os

from optimum.rbln import RBLNAutoPipelineForText2Image

model_id = "stabilityai/sdxl-turbo"
prompt = "A cinematic shot of a baby racoon wearing an intricate italian priest robe."

# Load compiled model
pipe = RBLNAutoPipelineForText2Image.from_pretrained(
    model_id=os.path.basename(model_id),
    export=False,
)

# Generate image
image = pipe(prompt, num_inference_steps=1, guidance_scale=0.0).images[0]

# Save image result
image.save(f"{prompt}.png")

Beyond this example, RBLN Model Zoo provides examples for more models.

Option 3: Custom PyTorch / TensorFlow Model

Custom models, such as PyTorch or TensorFlow models, are compiled with rebel-compiler and run with its rebel.Runtime() API.

1. Construct or Import a Model

The example below defines a simple PyTorch model:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
import torch
import torch.nn as nn


class SimpleConvBNRelu(nn.Module):
    def __init__(
        self, in_channels=3, out_channels=16, kernel_size=3, stride=1, padding=1
    ):
        super(SimpleConvBNRelu, self).__init__()
        self.conv = nn.Conv2d(in_channels, out_channels, kernel_size, stride, padding)
        self.bn = nn.BatchNorm2d(out_channels)
        self.relu = nn.ReLU(inplace=True)

    def forward(self, x):
        x = self.conv(x)
        x = self.bn(x)
        x = self.relu(x)
        return x


model = SimpleConvBNRelu()
print(model)

2. Compile the Model

rebel-compiler converts the model into an RBLN NPU-compatible format. For PyTorch models, like the SimpleConvBNRelu in the example below, use the rebel.compile_from_torch() function:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
import rebel
import torch
import torch.nn as nn


class SimpleConvBNRelu(nn.Module):
    def __init__(
        self, in_channels=3, out_channels=16, kernel_size=3, stride=1, padding=1
    ):
        super(SimpleConvBNRelu, self).__init__()
        self.conv = nn.Conv2d(in_channels, out_channels, kernel_size, stride, padding)
        self.bn = nn.BatchNorm2d(out_channels)
        self.relu = nn.ReLU(inplace=True)

    def forward(self, x):
        x = self.conv(x)
        x = self.bn(x)
        x = self.relu(x)
        return x


model = SimpleConvBNRelu().eval()

x = torch.rand(1, 3, 224, 224)

input_info = [
    ("x", list(x.shape), torch.float32),
]

compiled_model = rebel.compile_from_torch(model, input_info)
compiled_model.save("simple_conv_bn_relu.rbln")

PyTorch models can also be compiled with torch.compile() and the rbln backend. This path needs no .rbln file and no rebel.Runtime() step below. Calling the compiled model compiles it for the RBLN NPU and runs it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
import rebel  # registers the "rbln" backend for torch.compile
import torch
import torch.nn as nn


class SimpleConvBNRelu(nn.Module):
    def __init__(
        self, in_channels=3, out_channels=16, kernel_size=3, stride=1, padding=1
    ):
        super(SimpleConvBNRelu, self).__init__()
        self.conv = nn.Conv2d(in_channels, out_channels, kernel_size, stride, padding)
        self.bn = nn.BatchNorm2d(out_channels)
        self.relu = nn.ReLU(inplace=True)

    def forward(self, x):
        x = self.conv(x)
        x = self.bn(x)
        x = self.relu(x)
        return x


torch.manual_seed(0)
model = SimpleConvBNRelu().eval()

compiled_model = torch.compile(
    model,
    backend="rbln",
    options={"cache_dir": "./.cache", "mode": "strict"},
    dynamic=False,
)

x = torch.rand(1, 3, 224, 224)

with torch.no_grad():
    result = compiled_model(x)  # first call: compiles, or loads from cache_dir
    result = compiled_model(x)  # warm path: runs the compiled model

print("--- Result ---")
print(result)
  • First call: compiles the model and saves it to cache_dir. If cache_dir already holds a compiled model with the same weights and input shape, the first call loads it instead of compiling. torch.manual_seed(0) keeps the example's random weights the same across runs, so the second run of the script hits the cache.
  • Later calls: take the warm path. A call with the same input shape runs the compiled model on the RBLN NPU without compiling again.

"mode": "strict" raises an error when the rbln backend fails to compile the graph. Without it, torch.compile() falls back to eager execution on the CPU and only logs a warning. For all options, refer to the torch.compile API.

3. Run Inference

Load the compiled model with the rebel.Runtime() API and run it on the RBLN NPU:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
import rebel
import torch

x = torch.rand(1, 3, 224, 224)

module = rebel.Runtime("simple_conv_bn_relu.rbln")
inputs = x.numpy()
result = module.run(inputs)

print("--- Input ---")
print(inputs)
print("--- Result ---")
print(result)

Model Serving Frameworks

The model in Option 1 is already served with RBLN vLLM. To deploy the models compiled in Option 2 and Option 3 in production environments, use one of the following model serving frameworks with the RBLN NPU. Consult the linked documentation for detailed setup and configuration: