Skip to content

Installation Guide

Quick Installation

Use the installation matrix to choose your environment and copy the recommended command.

RBLN SDK Setup Deep Dive

This section describes how to install each component of the RBLN SDK. RBLN Driver and RBLN Compiler are required to use the other components, so install them first. Then install only the remaining components your workload needs, in any order.

RBLN Driver

Check driver status using rbln-smi

Run rbln-smi to confirm the driver is loaded and each NPU is reachable. A healthy system produces output like this:

$ rbln-smi
+-------------------------------------------------------------------------------------------------+
|                                Device Information KMD ver: 3.4.1                                |
+-----+-----------+---------+---------------+------+---------+------+---------------------+-------+
| NPU |    Name   |  Device |   PCI BUS ID  | Temp |  Power  | Perf |  Memory(used/total) |  Util |
+=====+===========+=========+===============+======+=========+======+=====================+=======+
| 0   | RBLN-CA22 | rbln0   |  0000:1b:00.0 |  35C |  18.4W  | P14  |    0.0B / 15.7GiB   |   0.0 |
| 1   | RBLN-CA22 | rbln1   |  0000:1c:00.0 |  34C |  18.2W  | P14  |    0.0B / 15.7GiB   |   0.0 |
+-----+-----------+---------+---------------+------+---------+------+---------------------+-------+
+-------------------------------------------------------------------------------------------------+
|                                       Context Information                                       |
+-----+---------------------+--------------+-----+----------+------+---------------------+--------+
| NPU |       Process       |      PID     | CTX | Priority | PTID |       Memalloc      | Status |
+-----+---------------------+--------------+-----+----------+------+---------------------+--------+
| N/A | N/A                 |   N/A        | N/A |   N/A    |  N/A |                 N/A |  N/A   |
+-----+---------------------+--------------+-----+----------+------+---------------------+--------+

If the tables look like the sample above, the kernel driver is healthy. Skip the remaining driver steps and continue to RBLN Compiler.

If rbln-smi is missing or the tables are empty, stay in this section and finish the driver installation before you move on to the RBLN Compiler section.

The RBLN Driver contains the Linux kernel driver and firmware, enabling the OS to recognize RBLN NPU devices. It is pre-installed on most cloud servers.

For driver installation instructions, see:

RBLN Compiler (rebel-compiler)

RBLN Compiler (rebel-compiler) is the core component of the RBLN SDK, used to convert pre-trained models into an NPU-executable format. It also provides runtime environments (Python and C/C++) and profiling tools. Installing the RBLN Compiler requires an RBLN Portal account.

Key Features

  • Compile API: Converts pre-trained models into RBLN NPU-executable formats.
  • Runtime API:
    1. Python runtime: Installed via a .whl package.
    2. C/C++ runtime: Requires GPG key registration and apt-based installation.
      See C/C++ runtime installation for details.
  • Profiler Support: Offers performance analysis and optimization with the RBLN Profiler.

Installation

  • Distributed as a .whl package:
1
2
3
$ pip install \
  --extra-index-url https://pypi.rbln.ai/simple \
  rebel-compiler==0.12.0

RBLN vLLM (vllm-rbln)

RBLN vLLM (vllm-rbln) is a hardware plugin for vLLM that delivers high-performance large language model (LLM) inference and serving on RBLN NPUs.

Key Features

  • vLLM Integration: Serves LLMs on RBLN NPUs through the vLLM library.
  • Usage: RBLN vLLM Recipes provide deployment examples for a wide range of models.

Installation

  • Distributed as a .whl package:
1
2
3
4
$ pip install \
  --extra-index-url https://wheels.vllm.ai/0.30.0/cpu \
  --extra-index-url https://download.pytorch.org/whl/cpu \
  vllm-rbln==0.12.0

Note

Installing vllm-rbln also installs compatible versions of torch-rbln and optimum-rbln as dependencies, so you can skip the RBLN PyTorch and RBLN Optimum installation steps.

  • For source-code installation, installation of additional compatible packages, and usage, see the RBLN vLLM overview.

RBLN Optimum (optimum-rbln)

RBLN Optimum (optimum-rbln) integrates with HuggingFace APIs, making it easy to compile and run models provided through transformers and diffusers on RBLN NPUs.

Key Features

  • HuggingFace Integration: Enables RBLN SDK-based inference for transformers and diffusers models with minimal effort.
  • Easy Deployment: Supports model optimization, compilation, and loading on RBLN NPUs.
  • Usage: RBLN Model Zoo provides examples for a wide range of models.

Installation

Note

If you installed RBLN vLLM, optimum-rbln is already installed. Install it separately only to use RBLN Optimum without RBLN vLLM.

  • Distributed as a .whl package:
1
2
3
$ pip install \
  --extra-index-url https://download.pytorch.org/whl/cpu \
  optimum-rbln==0.12.0

RBLN PyTorch (torch-rbln)

RBLN PyTorch (torch-rbln) is a PyTorch out-of-tree extension that lets you run PyTorch workloads on RBLN NPUs. It autoloads when you import torch. For more information, see the RBLN PyTorch overview.

Key Features

Installation

Note

If you installed RBLN vLLM, torch-rbln is already installed. Install it separately only to use RBLN PyTorch without RBLN vLLM.

  • Prerequisite: Install rebel-compiler as described in RBLN Compiler.
  • Distributed as a .whl package. With RBLN Compiler installed, install it with:
1
2
3
4
5
$ pip install \
  --index-url https://download.pytorch.org/whl/cpu \
  torch==2.13.0+cpu
$ pip install \
  torch-rbln==0.12.0

RBLN vLLM Recipes

RBLN vLLM Recipes provide ready-to-run deployment commands and recommended configurations for serving a wide range of models on RBLN NPUs with RBLN vLLM. Each recipe covers one model and pairs a vllm serve command with a sample request to the OpenAI-compatible API. Find the recipe for each model in vLLM Recipes.

Key Features

  • Hardware-specific Commands: Provides the serve command and parallelism settings for each RBLN NPU.
  • Execution: Each recipe provides the install commands, the serve command, and a client request.

Installation

  • No separate installation is required. Each recipe lists the packages it needs, typically rebel-compiler and vllm-rbln.

RBLN Model Zoo

RBLN Model Zoo provides ready-to-use examples for compiling and running pre-trained models on RBLN NPUs. It serves as a reference for adapting custom models.

Key Features

  • Diverse Models: Includes run scripts for a wide range of tasks and models, such as speech, text, and vision.
  • Implementation Guides: Offers step-by-step instructions for compiling and running models on RBLN NPUs.

Installation

  • Hosted on GitHub. Clone the repository with:
1
$ git clone --recursive https://github.com/RBLN-SW/rbln-model-zoo.git

Serving Frameworks

RBLN NPUs integrate with popular serving solutions, including NVIDIA Triton Inference Server and TorchServe. For LLM serving with vLLM, see RBLN vLLM.

Key Features

Installation

Next Steps

The RBLN SDK installation is now complete. For detailed instructions on running PyTorch and TensorFlow models on RBLN NPUs, see the tutorials for each component.