Skip to content

NPU Allocation

RBLN NPUs can be utilized inside Docker containers, enabling controlled deployment and resource management. On servers with multiple NPUs, containers can be configured to use specific NPUs, enabling efficient resource allocation.

Rebellions Scalable Design (RSD)

Rebellions Scalable Design (RSD) is a technology that enables efficient management and utilization of multiple RBLN NPUs. RSD provides:

  • Device Grouping: NPUs are organized into groups for better resource management
  • Tensor Parallelism: Large language models can be distributed across multiple NPUs within the same group
  • Fault Isolation: Separate groups prevent issues in one group from impacting others
  • Independent Scheduling: Each group can be managed independently for optimal performance

All RBLN NPUs operate through RSD, whether used individually or in multi-NPU configurations. This means that even a single NPU container benefits from RSD's management capabilities.

How many devices one card presents depends on the card model, and that decides how the devices group. See the note under Container Allocation Strategy.

Note

The rbln-smi tool for managing and monitoring RBLN NPUs is included in the RBLN driver package. When using the RBLN Container Toolkit, rbln-smi is automatically available inside containers without manual mounting.

RBLN Container Toolkit

RBLN Container Toolkit simplifies container access to RBLN NPUs using the Container Device Interface (CDI) specification.

For installation, CLI reference, Kubernetes deployment, and troubleshooting, see the RBLN Container Toolkit Guide.

Quick Setup

After installing the toolkit, configure and run:

1
2
3
4
5
6
7
8
9
# Configure
$ sudo rbln-ctk cdi generate
$ sudo rbln-ctk runtime configure
$ sudo systemctl restart docker

# Run container with NPU access
$ docker run \
  --device rebellions.ai/npu=all \
  -it IMAGE_NAME:TAG

With the toolkit configured, all containers using --device rebellions.ai/npu=all have RBLN libraries and tools (including rbln-smi) automatically available.

Container Creation Guidelines

This section summarizes common requirements and options for using RBLN NPUs with Docker containers.

RSD Group Management

The steps below use rbln-smi, which writes to the driver directly and needs only the driver installed.

The daemon can perform the same operations

On a host already running rbln-smd, the same group operations are available through rbln-smdi group. That path suits a host administered remotely, or one where group changes should pass through the daemon's access control rather than a root shell — see the RBLN System Management Daemon page.

Check RSD Group Configuration

$ rbln-smi -g
+-------------------------------------------------------------------------------------------------------+
|                                  RSD Management Group KMD ver: 3.5.1                                  |
+-----+-----+-----------+---------+---------------+------+---------+------+---------------------+-------+
| Grp | NPU |    Name   |  Device |   PCI BUS ID  | Temp |  Power  | Perf |  Memory(used/total) |  Util |
+=====+=====+===========+=========+===============+======+=========+======+=====================+=======+
| 0   | 0   | RBLN-CR13 | rbln0   |  0000:1b:00.0 |  37C |  100.0W |  P14 |    0.0B / 140.0GiB  |   0.0 |
|     | 1   | RBLN-CR13 | rbln1   |  0000:1c:00.0 |  36C |  100.0W |  P14 |    0.0B / 140.0GiB  |   0.0 |
+-----+-----+-----------+---------+---------------+------+---------+------+---------------------+-------+
+-------------------------------------------------------------------------------------------------------+
|                                        RSD Context Information                                        |
+-----+-----+---------------------+--------------+-----------+----------+------+---------------+--------+
| Grp | NPU |       Process       |      PID     |    CTX    | Priority | PTID |    Memalloc   | Status |
+=====+=====+=====================+==============+===========+==========+======+===============+========+
| N/A | N/A | N/A                 |      N/A     |    N/A    |    N/A   |  N/A |           N/A |   N/A  |
+-----+-----+---------------------+--------------+-----------+----------+------+---------------+--------+

$ rbln-smi -g
+-------------------------------------------------------------------------------------------------------+
|                                  RSD Management Group KMD ver: 3.4.1                                  |
+-----+-----+-----------+---------+---------------+------+---------+------+---------------------+-------+
| Grp | NPU |    Name   |  Device |   PCI BUS ID  | Temp |  Power  | Perf |  Memory(used/total) |  Util |
+=====+=====+===========+=========+===============+======+=========+======+=====================+=======+
| 0   | 0   | RBLN-CA22 | rbln0   |  0000:1b:00.0 |  37C |  19.5W  |  P14 |    0.0B / 15.7GiB   |   0.0 |
|     | 1   | RBLN-CA22 | rbln1   |  0000:1c:00.0 |  36C |  19.2W  |  P14 |    0.0B / 15.7GiB   |   0.0 |
+-----+-----+-----------+---------+---------------+------+---------+------+---------------------+-------+
+-------------------------------------------------------------------------------------------------------+
|                                        RSD Context Information                                        |
+-----+-----+---------------------+--------------+-----------+----------+------+---------------+--------+
| Grp | NPU |       Process       |      PID     |    CTX    | Priority | PTID |    Memalloc   | Status |
+=====+=====+=====================+==============+===========+==========+======+===============+========+
| N/A | N/A | N/A                 |      N/A     |    N/A    |    N/A   |  N/A |           N/A |   N/A  |
+-----+-----+---------------------+--------------+-----------+----------+------+---------------+--------+

By default, all NPUs are shown under group 0 (as indicated in the Grp column). For optimal container isolation and performance, create separate RSD groups for each container.

Create an RSD Group

Create dedicated RSD groups for your containers:

1
$ sudo rbln-smi group -c <group_id> -a <device_id_list>

Delete an RSD Group

You can destroy a group you created with the following command:

1
$ sudo rbln-smi group -d <group_id>

Regenerate the CDI spec after group changes

Both rbln-smi group -c and rbln-smi group -d change the host's /dev/rsd* topology, which the CDI spec captured at the last rbln-ctk cdi generate run. Rerun sudo rbln-ctk cdi generate after the change if any container uses the rebellions.ai/npu=N handle; the toolkit's automatic refresh paths react only to driver and library changes, not to rbln-smi group operations.

Accessing NPUs in Containers

Containers access NPUs through the --device rebellions.ai/npu=<handle> flag. For the available handles (=all, =N), single- and multi-NPU patterns, and runtime-specific behavior, see Device Selection.

Allocate a Single RBLN NPU

This section demonstrates how to configure a Docker container to use a single RBLN NPU with proper RSD group isolation.

Important

Before proceeding, read the Container Creation Guidelines above to understand the common requirements and options referenced in this step-by-step guide.

Step 1: Create an RSD Group

Create a separate RSD group for your single NPU container:

1
$ sudo rbln-smi group -c 1 -a 1

This command creates group 1 and assigns rbln1 to it, removing it from the default group (group 0).

Step 2: Verify RSD Group Configuration

Verify that the NPU is now in its dedicated group:

1
$ rbln-smi -g

Your output should show the NPU in its own group:

+-------------------------------------------------------------------------------------------------------+
|                                  RSD Management Group KMD ver: 3.5.1                                  |
+-----+-----+-----------+---------+---------------+------+---------+------+---------------------+-------+
| Grp | NPU |    Name   |  Device |   PCI BUS ID  | Temp |  Power  | Perf |  Memory(used/total) |  Util |
+=====+=====+===========+=========+===============+======+=========+======+=====================+=======+
| 0   | 0   | RBLN-CR13 | rbln0   |  0000:1b:00.0 |  35C |  100.0W |  P14 |    0.0B / 140.0GiB  |   0.0 |
+-----+-----+-----------+---------+---------------+------+---------+------+---------------------+-------+
| 1   | 0   | RBLN-CR13 | rbln1   |  0000:1c:00.0 |  34C |  100.0W |  P14 |    0.0B / 140.0GiB  |   0.0 |
+-----+-----+-----------+---------+---------------+------+---------+------+---------------------+-------+
+-------------------------------------------------------------------------------------------------------+
|                                        RSD Context Information                                        |
+-----+-----+---------------------+--------------+-----------+----------+------+---------------+--------+
| Grp | NPU |       Process       |      PID     |    CTX    | Priority | PTID |    Memalloc   | Status |
+=====+=====+=====================+==============+===========+==========+======+===============+========+
| N/A | N/A | N/A                 |      N/A     |    N/A    |    N/A   |  N/A |           N/A |   N/A  |
+-----+-----+---------------------+--------------+-----------+----------+------+---------------+--------+

+-------------------------------------------------------------------------------------------------------+
|                                  RSD Management Group KMD ver: 3.4.1                                  |
+-----+-----+-----------+---------+---------------+------+---------+------+---------------------+-------+
| Grp | NPU |    Name   |  Device |   PCI BUS ID  | Temp |  Power  | Perf |  Memory(used/total) |  Util |
+=====+=====+===========+=========+===============+======+=========+======+=====================+=======+
| 0   | 0   | RBLN-CA22 | rbln0   |  0000:1b:00.0 |  35C |  18.4W  |  P14 |    0.0B / 15.7GiB   |   0.0 |
+-----+-----+-----------+---------+---------------+------+---------+------+---------------------+-------+
| 1   | 0   | RBLN-CA22 | rbln1   |  0000:1c:00.0 |  34C |  18.2W  |  P14 |    0.0B / 15.7GiB   |   0.0 |
+-----+-----+-----------+---------+---------------+------+---------+------+---------------------+-------+
+-------------------------------------------------------------------------------------------------------+
|                                        RSD Context Information                                        |
+-----+-----+---------------------+--------------+-----------+----------+------+---------------+--------+
| Grp | NPU |       Process       |      PID     |    CTX    | Priority | PTID |    Memalloc   | Status |
+=====+=====+=====================+==============+===========+==========+======+===============+========+
| N/A | N/A | N/A                 |      N/A     |    N/A    |    N/A   |  N/A |           N/A |   N/A  |
+-----+-----+---------------------+--------------+-----------+----------+------+---------------+--------+

Notice that rbln1 is now in group 1, separate from rbln0 in group 0.

Step 3: Run Docker Container with Single NPU

Run a Docker container with access to the dedicated NPU:

1
2
3
$ docker run \
  --device rebellions.ai/npu=1 \
  -it IMAGE_NAME:TAG

The container now has exclusive access to rbln1 through RSD group 1, ensuring optimal performance and isolation.

Allocate Multiple RBLN NPUs

This section demonstrates how to configure Docker containers to use multiple RBLN NPUs. Multiple NPUs can be used for large language models that leverage tensor parallelism.

A list of models that support RSD can be found in RBLN Optimum Multi-NPU Supported Models.

Container Allocation Strategy

You can create multiple containers, each with a dedicated NPU group. For example, with 8 RBLN NPUs, you can create 2 containers, with 4 NPUs assigned to separate RSD groups.

Important

Before proceeding, read the Container Creation Guidelines above to understand the common requirements and options referenced in this step-by-step guide.

One card is one device

RBLN-CR13 presents one device per card, so each entry in rbln-smi is one card.

How many devices a card presents depends on the card model

RBLN-CA25 presents four devices per card, which appear as four entries in rbln-smi. Every other ATOM™ card presents one device per card, so one card is one device.

Check the layout with rbln-smi before you plan the split. When a card presents several devices, keep all of them in the same RSD group and the same container: they share the card, and separating them splits a card across two workloads.

Step 1: Create an RSD Group

Ensure all NPUs that work together in a single container are in the same RSD group.

For example, if you have 8 devices and want to create a container with 4 NPUs:

1
$ sudo rbln-smi group -c 1 -a 4,5,6,7

Step 2: Verify Group Configuration

Verify that the NPUs are properly grouped:

1
$ rbln-smi -g

Your output should show the NPUs organized in separate groups:

+-------------------------------------------------------------------------------------------------------+
|                                  RSD Management Group KMD ver: 3.5.1                                  |
+-----+-----+-----------+---------+---------------+------+---------+------+---------------------+-------+
| Grp | NPU |    Name   |  Device |   PCI BUS ID  | Temp |  Power  | Perf |  Memory(used/total) |  Util |
+=====+=====+===========+=========+===============+======+=========+======+=====================+=======+
| 0   | 0   | RBLN-CR13 | rbln0   |  0000:1b:00.0 |  38C |  100.0W |  P14 |    0.0B / 140.0GiB  |   0.0 |
|     | 1   | RBLN-CR13 | rbln1   |  0000:1c:00.0 |  37C |  100.0W |  P14 |    0.0B / 140.0GiB  |   0.0 |
|     | 2   | RBLN-CR13 | rbln2   |  0000:1f:00.0 |  36C |  100.0W |  P14 |    0.0B / 140.0GiB  |   0.0 |
|     | 3   | RBLN-CR13 | rbln3   |  0000:22:00.0 |  39C |  100.0W |  P14 |    0.0B / 140.0GiB  |   0.0 |
+-----+-----+-----------+---------+---------------+------+---------+------+---------------------+-------+
| 1   | 0   | RBLN-CR13 | rbln4   |  0000:41:00.0 |  36C |  100.0W |  P14 |    0.0B / 140.0GiB  |   0.0 |
|     | 1   | RBLN-CR13 | rbln5   |  0000:42:00.0 |  36C |  100.0W |  P14 |    0.0B / 140.0GiB  |   0.0 |
|     | 2   | RBLN-CR13 | rbln6   |  0000:45:00.0 |  40C |  100.0W |  P14 |    0.0B / 140.0GiB  |   0.0 |
|     | 3   | RBLN-CR13 | rbln7   |  0000:48:00.0 |  36C |  100.0W |  P14 |    0.0B / 140.0GiB  |   0.0 |
+-----+-----+-----------+---------+---------------+------+---------+------+---------------------+-------+
+-------------------------------------------------------------------------------------------------------+
|                                        RSD Context Information                                        |
+-----+-----+---------------------+--------------+-----------+----------+------+---------------+--------+
| Grp | NPU |       Process       |      PID     |    CTX    | Priority | PTID |    Memalloc   | Status |
+=====+=====+=====================+==============+===========+==========+======+===============+========+
| N/A | N/A | N/A                 |      N/A     |    N/A    |    N/A   |  N/A |           N/A |   N/A  |
+-----+-----+---------------------+--------------+-----------+----------+------+---------------+--------+

+-------------------------------------------------------------------------------------------------------+
|                                  RSD Management Group KMD ver: 3.4.1                                  |
+-----+-----+-----------+---------+---------------+------+---------+------+---------------------+-------+
| Grp | NPU |    Name   |  Device |   PCI BUS ID  | Temp |  Power  | Perf |  Memory(used/total) |  Util |
+=====+=====+===========+=========+===============+======+=========+======+=====================+=======+
| 0   | 0   | RBLN-CA22 | rbln0   |  0000:1b:00.0 |  38C |  19.5W  |  P14 |    0.0B / 15.7GiB   |   0.0 |
|     | 1   | RBLN-CA22 | rbln1   |  0000:1c:00.0 |  37C |  19.3W  |  P14 |    0.0B / 15.7GiB   |   0.0 |
|     | 2   | RBLN-CA22 | rbln2   |  0000:1f:00.0 |  36C |  18.5W  |  P14 |    0.0B / 15.7GiB   |   0.0 |
|     | 3   | RBLN-CA22 | rbln3   |  0000:22:00.0 |  39C |  19.6W  |  P14 |    0.0B / 15.7GiB   |   0.0 |
+-----+-----+-----------+---------+---------------+------+---------+------+---------------------+-------+
| 1   | 0   | RBLN-CA22 | rbln4   |  0000:41:00.0 |  36C |  17.9W  |  P14 |    0.0B / 15.7GiB   |   0.0 |
|     | 1   | RBLN-CA22 | rbln5   |  0000:42:00.0 |  36C |  21.5W  |  P14 |    0.0B / 15.7GiB   |   0.0 |
|     | 2   | RBLN-CA22 | rbln6   |  0000:45:00.0 |  40C |  20.6W  |  P14 |    0.0B / 15.7GiB   |   0.0 |
|     | 3   | RBLN-CA22 | rbln7   |  0000:48:00.0 |  36C |  17.9W  |  P14 |    0.0B / 15.7GiB   |   0.0 |
+-----+-----+-----------+---------+---------------+------+---------+------+---------------------+-------+
+-------------------------------------------------------------------------------------------------------+
|                                        RSD Context Information                                        |
+-----+-----+---------------------+--------------+-----------+----------+------+---------------+--------+
| Grp | NPU |       Process       |      PID     |    CTX    | Priority | PTID |    Memalloc   | Status |
+=====+=====+=====================+==============+===========+==========+======+===============+========+
| N/A | N/A | N/A                 |      N/A     |    N/A    |    N/A   |  N/A |           N/A |   N/A  |
+-----+-----+---------------------+--------------+-----------+----------+------+---------------+--------+

Notice that rbln4~rbln7 are now in group 1, separate from rbln0~rbln3 in group 0.

Step 3: Run Docker Containers with Multiple NPUs

Container 1: Assign rbln0~rbln3 (Group 0)

1
2
3
4
5
6
$ docker run \
  --device rebellions.ai/npu=0 \
  --device rebellions.ai/npu=1 \
  --device rebellions.ai/npu=2 \
  --device rebellions.ai/npu=3 \
  -it IMAGE_NAME:TAG

Container 2: Assign rbln4~rbln7 (Group 1)

1
2
3
4
5
6
$ docker run \
  --device rebellions.ai/npu=4 \
  --device rebellions.ai/npu=5 \
  --device rebellions.ai/npu=6 \
  --device rebellions.ai/npu=7 \
  -it IMAGE_NAME:TAG

In this setup, each Docker container can utilize 4 RBLN NPUs:

  • Container 1: Uses rbln0~rbln3 through RSD group 0 (/dev/rsd0)
  • Container 2: Uses rbln4~rbln7 through RSD group 1 (/dev/rsd1)

The /dev/rsd* paths above are the host's node names. Inside each container the attached group device always appears as /dev/rsd0, so Container 2 also sees /dev/rsd0 even though the host node is /dev/rsd1. See Device Selection for the handle behavior behind this.