RBLN Kubernetes Device Plugin¶

The RBLN SDK provides the Kubernetes Device Plugin for using RBLN NPUs in Kubernetes clusters.

Follow Installing the NPU Operator and Device Plugin configuration to deploy the plugin. For the basic request and verification workflow, see Run an NPU Pod. To select a family or allocate multiple NPUs, see Schedule a Pod by NPU family and Allocate two NPUs to one Pod.

CDI Configuration (Required)¶

The Device Plugin requires CDI to be enabled in the container runtime. Configure containerd manually and restart it, or use the Container Toolkit to configure CDI.

containerd v1.7.x (/etc/containerd/config.toml)

1
2
3
4

[plugins]
  [plugins."io.containerd.grpc.v1.cri"]
    enable_cdi = true
    cdi_spec_dirs = ["/etc/cdi", "/var/run/cdi"]

containerd v2.x.x (/etc/containerd/config.toml)

1
2
3
4

[plugins]
  [plugins."io.containerd.cri.v1.runtime"]
    enable_cdi = true
    cdi_spec_dirs = ["/etc/cdi", "/var/run/cdi"]

On every allocation, the Device Plugin adds the CDI annotation rebellions.ai/npu=runtime to the container. The plugin has no setting to disable this annotation. For the runtime to act on it, both the enable_cdi setting and the CDI spec generated on the host by the Container Toolkit must be in place. When both are configured, the runtime mounts the required libraries and tools (such as rbln-smi) into workloads that request an NPU resource (for example, rebellions.ai/npu).

If the node has an RDS (Rebellions Direct Storage) device node (/dev/rblnfs*), the plugin also adds rebellions.ai/rds=all, which mounts the RDS device through the same CDI path. The plugin checks for the RDS device node at each NPU allocation. If none is present, it does not add the RDS annotation. See RDS Support for node and NVMe selection.

Scheduling and Allocation¶

If two containers each request one NPU, each receives a separate device. If one container requests two NPUs, it receives both devices.

To check how many NPUs remain available, compare Allocatable with existing requests in kubectl describe node <NODE_NAME>. The scheduler also checks constraints such as selectors, taints, and CPU and memory requirements. Hardware labels alone do not show whether the driver or Device Plugin is ready.

To allow any product within a family, keep the NPU resource limit and add this under the Pod's spec:

1
2
3
4
5
6
7
8

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: rebellions.ai/npu.family
          operator: In
          values: [atom]

Check the node's family label with kubectl get nodes -L rebellions.ai/npu.family before selecting a family.

RSD Groups for Multiple NPUs¶

During allocation, the Device Plugin creates an RSD group from the selected PCI bus IDs. It maps the host group's device to /dev/rsd0 inside the container. In the output of rbln-smi -j -g, all devices allocated to the container must have the same non-empty group_id. Do not manually create or delete host RSD groups while Kubernetes workloads use the devices.

Device Health Isolation¶

The Device Plugin checks the state of each NPU through the kernel driver on every device-discovery cycle and reports Healthy/Unhealthy to the kubelet. A device on which workloads cannot run is excluded from scheduling and becomes schedulable again once the driver reports READY.

NPU Allocation Tracing¶

The Device Plugin records NPU allocations with OpenTelemetry traces. For each allocation request, the span records the devices and Rebellions Scalable Design (RSD) group allocated to the container. An OpenTelemetry Protocol (OTLP) backend can use these traces to track NPU allocations on each node.

Tracing is disabled by default. To enable it, specify an OTLP gRPC endpoint with the following Helm value.

1
2

devicePlugin:
  otlpEndpoint: <OTLP_ENDPOINT>

An endpoint given as host:port uses an insecure gRPC connection; pass a full URL such as https://collector:4317 to use TLS.

If the endpoint format is invalid or the OTLP exporter fails to initialize, the plugin continues to run without tracing. These tracing errors do not affect NPU scheduling on the node.

An Allocate span is created per allocation request and an allocateContainer span per container, with the following attributes. When an allocation fails, the error is recorded on the span.

Every span carries the resource attributes service.name=rbln-device-plugin and service.version. In addition, when the NODE_NAME environment variable is set (the Helm DaemonSet always injects spec.nodeName), a k8s.node.name attribute is added so spans can be distinguished per node.