Skip to content

Docker Support

The RBLN NPUs can be utilized within a Docker container for streamlined deployment and management. If a server has multiple RBLN NPUs, each container can be configured to use the specific NPU, enabling efficient resource allocation in the multi-NPU environment.

Allocate a single RBLN NPU

When creating a Docker container, you can mount the device file for each RBLN NPU to enable its use. 

1
$ docker run --device /dev/rbln0 -ti IMAGE_NAME:TAG

Allocate multiple RBLN NPUs

When using multiple RBLN NPUs, you can mount as many as needed while creating the Docker container. For example, if you have 8 RBLN NPUs, you can create 2 containers, each mounting 4 RBLN NPUs, as belows:

1
2
3
4
5
# Assign RBLN NPUs 0 to 3
$ docker run \
    --device /dev/rbln0 --device /dev/rbln1 \
    --device /dev/rbln2 --device /dev/rbln3 \
    -ti IMAGE_NAME:TAG

1
2
3
4
5
# Assign RBLN NPUs 4 to 7
$ docker run \
    --device /dev/rbln4:/dev/rbln0 --device /dev/rbln5:/dev/rbln1 \
    --device /dev/rbln6:/dev/rbln2 --device /dev/rbln7:/dev/rbln3 \
    -ti IMAGE_NAME:TAG

In this setup, each Docker container can utilize 4 RBLN NPUs (/dev/rbln0, /dev/rbln1, /dev/rbln2, /dev/rbln3).

Allocate multiple RBLN NPUs for RSD support

Through Rebellions Scalable Design (RSD), large language models can be effectively tensor-parallelized and run across multiple NPUs. A list of models that support RSD can be found in Optimum RBLN Multi-NPU Supported Models.

When using RSD, the container execution command must include the additional mounting of the /dev/rsd0 file as shown below:

1
2
3
4
5
$ docker run \
    --device /dev/rsd0 \
    --device /dev/rbln0 --device /dev/rbln1 \
    --device /dev/rbln2 --device /dev/rbln3 \
    -ti IMAGE_NAME:TAG

Multi-Container Configuration Utilizing RSD

The /dev/rsd0 must be mounted in all Docker containers utilizing RSD.

For instance, on a server with 8 RBLN NPUs, if you create two Docker containers, each utilizing the RSD with 4 RBLN NPUs, both the first and the second Docker containers should be created with mounting /dev/rsd0 as demonstrated below:

1
2
3
4
5
6
# Assign RBLN NPUs 0 to 3 with RSD
$ docker run \
    --device /dev/rsd0 \
    --device /dev/rbln0 --device /dev/rbln1 \
    --device /dev/rbln2 --device /dev/rbln3 \
    -ti IMAGE_NAME:TAG

1
2
3
4
5
6
# Assign RBLN NPUs 4 to 7 with RSD
$ docker run \
    --device /dev/rsd0 \
    --device /dev/rbln4:/dev/rbln0 --device /dev/rbln5:/dev/rbln1 \
    --device /dev/rbln6:/dev/rbln2 --device /dev/rbln7:/dev/rbln3 \
    -ti IMAGE_NAME:TAG