Skip to content

RBLN Kubernetes Device Plugin

RBLN SDK provides Kubernetes Device Plugin to support RBLN NPUs on Kubernetes cluster environment.

Step 1. Prepare NPU nodes

First, you need to prepare Kubernetes nodes equipped with RBLN NPUs and install the RBLN Driver. Typically, the stable version of the RBLN Driver is already installed on the cloud server you are currently using. If you can see the RBLN NPUs by executing the command rbln-smi, a command-line interface (CLI) utility which is already included in the RBLN Driver package, then you can skip the installation of the RBLN Driver. For more information about the installation, please refer to the Installation Guide.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
$ rbln-smi
Mon May 27 03:28:44 2024
+-------------------------------------------------------------------------------------------------+
|                        Device Infomation KMD ver: 0.12.37-6eed3e9-release                       |
+-----+-----------+-----------+---------------+------+---------+--------------------------+-------+
| NPU |    Name   | Device    |   PCI BUS ID  | Temp |  Power  |    Memory(used/total)    |  Util |
+-----+-----------+-----------+---------------+------+---------+--------------------------+-------+
| 0   | RBLN-CA02 | rbln0     |  0000:b6:00.0 |  25C |   6.2W  |      0.0B / 15.7GiB      |   0.0 |
+-----+-----------+-----------+---------------+------+---------+--------------------------+-------+
+-------------------------------------------------------------------------------------------------+
|                                        Context Infomation                                       |
+-----+---------------------+--------------+-----+----------+------+---------------------+--------+
| NPU | Process             |     PID      | CTX | Priority | PTID |            Memalloc | Status |
+-----+---------------------+--------------+-----+----------+------+---------------------+--------+
| N/A | N/A                 |     N/A      | N/A |   N/A    | N/A  |                 N/A |  N/A   |
+-----+---------------------+--------------+-----+----------+------+---------------------+--------+

CDI Configuration (Required)

The device plugin requires CDI to be enabled in the container runtime. You can configure containerd manually (and restart it), or let the Container Toolkit handle CDI setup.

containerd v1.7.x (/etc/containerd/config.toml)

1
2
3
4
[plugins]
  [plugins."io.containerd.grpc.v1.cri"]
    enable_cdi = true
    cdi_spec_dirs = ["/etc/cdi", "/var/run/cdi"]

containerd v2.x.x (/etc/containerd/config.toml)

1
2
3
4
[plugins]
  [plugins."io.containerd.cri.v1.runtime"]
    enable_cdi = true
    cdi_spec_dirs = ["/etc/cdi", "/var/run/cdi"]

When CDI is enabled, the device plugin adds a CDI annotation that references rebellions.ai/npu=all. This allows required libraries and tools (such as rbln-smi) to be automatically mounted when a workload requests the NPU resource (for example, rebellions.ai/npu).

Step 2. Install Device Plugin

The device plugin is recommended to be installed using the NPU Operator. However, if a standalone installation is required, it can be installed using the following command.

1
2
3
4
$ git clone https://github.com/RBLN-SW/k8s-device-plugin
$ cd k8s-device-plugin/deployments/helm/rbln-device-plugin

$ helm install rbln-device-plugin . -n kube-system

You can see the DaemonSet rbln-device-plugin under kube-system namespace, and Pods created from the Daemonset as below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
$ kubectl get ds -n kube-system rbln-device-plugin
NAME                  DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR              AGE
rbln-device-plugin   4         4         4       4            4           kubernetes.io/arch=amd64   26s

$ kubectl get pod -n kube-system -l name=rbln-device-plugin -o wide
NAME                        READY   STATUS    RESTARTS   AGE
rbln-device-plugin-5tj4z   1/1     Running   0          41s
rbln-device-plugin-7zqgh   1/1     Running   0          41s
rbln-device-plugin-ddtts   1/1     Running   0          41s
rbln-device-plugin-zrw8s   1/1     Running   0          41s

You can also check the RBLN NPU resources (rebellions.ai/npu) with the kubectl describe command as below:

1
2
3
4
5
6
7
8
9
$ kubectl describe node <your-node-name>
...
Capacity:
  ...
  rebellions.ai/npu:  1
Allocatable:
  ...
  rebellions.ai/npu:  1
...

Step 3. Create a Pod with NPUs

To create a Pod with NPU resources, you should add spec.containers[].resources.limits in your Pod spec as below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
apiVersion: v1
kind: Pod
metadata:
  name: rbln-device-plugin-testpod
spec:
  containers:
  - name: ubuntu
    image: ubuntu:latest
    imagePullPolicy: IfNotPresent
    command: ["/bin/bash", "-c", "--"]
    args: ["while true; do sleep 300000; done;"]
    resources:
      requests:
        rebellions.ai/npu: 1
      limits:
        rebellions.ai/npu: 1

You can create a Pod with the Pod spec using kubectl create command:

1
$ kubectl create -f https://raw.githubusercontent.com/rebellions-sw/rbln-k8s-device-plugin/master/deployments/rbln/pod-tc.yaml

A single rebellions.ai/npu resource has been assigned to the Pod:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
$ kubectl describe pod rbln-device-plugin-testpod
Name:             rbln-device-plugin-testpod
Namespace:        default
...
Containers:
  ubuntu:
    ...
    Limits:
      rebellions.ai/npu:  1
    Requests:
      rebellions.ai/npu:  1
    ...

The device plugin automatically mounts the rbln-smi from the host machine to the Pod container. You can check it on the container as below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
$ kubectl exec -it rbln-device-plugin-testpod -- bash
root@rbln-device-plugin-testpod:/# rbln-smi
Mon May 27 08:44:05 2024
+-------------------------------------------------------------------------------------------------+
|                        Device Infomation KMD ver: 0.12.37-6eed3e9-release                       |
+-----+-----------+-----------+---------------+------+---------+--------------------------+-------+
| NPU |    Name   | Device    |   PCI BUS ID  | Temp |  Power  |    Memory(used/total)    |  Util |
+-----+-----------+-----------+---------------+------+---------+--------------------------+-------+
| 0   | RBLN-CA02 | rbln0     |  0000:b6:00.0 |  25C |   6.2W  |      0.0B / 15.7GiB      |   0.0 |
+-----+-----------+-----------+---------------+------+---------+--------------------------+-------+
+-------------------------------------------------------------------------------------------------+
|                                        Context Infomation                                       |
+-----+---------------------+--------------+-----+----------+------+---------------------+--------+
| NPU | Process             |     PID      | CTX | Priority | PTID |            Memalloc | Status |
+-----+---------------------+--------------+-----+----------+------+---------------------+--------+
| N/A | N/A                 |     N/A      | N/A |   N/A    | N/A  |                 N/A |  N/A   |
+-----+---------------------+--------------+-----+----------+------+---------------------+--------+

Device Health Isolation

The device plugin checks the state of each NPU through the kernel driver on every device-discovery cycle and reports Healthy/Unhealthy to the kubelet. A device on which workloads cannot run is automatically excluded from scheduling, and it automatically returns to a schedulable state once the driver reports READY again.

NPU Allocation Tracing

The device plugin instruments the NPU allocation path with OpenTelemetry traces. For each allocation request, the span records which devices and Rebellions Scalable Design (RSD) group were handed to the container, so an OpenTelemetry Protocol (OTLP) backend can trace the per-node NPU allocation history.

Tracing is disabled by default and can be enabled by specifying an OTLP gRPC endpoint through the following Helm value.

1
2
devicePlugin:
  otlpEndpoint: <OTLP_ENDPOINT>

Tracing is a supplementary observability feature, so the plugin continues to run without tracing even when the endpoint format is invalid or the OTLP exporter fails to initialize, and NPU scheduling on the node is unaffected.

An Allocate span is created per allocation request and an allocateContainer span per container, with the following attributes. When an allocation fails, the error is recorded on the span.

Attribute Description
rbln.resource_name Resource name the allocation was requested for (e.g., rebellions.ai/npu)
rbln.device.ids List of NPU device IDs allocated to the container
rbln.device.count Number of allocated devices
rbln.device.bus_ids List of PCI bus IDs of the allocated devices
rbln.rsd.host_path Host path of the RSD group used for the allocation

Every span carries the resource attributes service.name=rbln-device-plugin and service.version. In addition, when the NODE_NAME environment variable is set (the Helm DaemonSet always injects spec.nodeName), a k8s.node.name attribute is added so spans can be distinguished per node.

Logging

The plugin writes structured logs to stdout. Configure the log level and output format with the following environment variables:

Variable Description Default
RBLN_DEVICE_PLUGIN_LOG_LEVEL Verbosity: error, warning (or warn), info, debug. info
RBLN_DEVICE_PLUGIN_LOG_FORMAT Output format: json or text. json

An invalid value falls back to the default and logs a warning rather than failing startup. Records for unhealthy devices are logged at warn.

The plugin emits no info logs while the node state is unchanged. To verify that the scan loop is running, set the level to debug and look for the Reconciled device inventory record emitted on each scan.

When deployed by the RBLN NPU Operator, configure these settings with the chart's devicePlugin.logging values. See Component Logging.