Allocate two NPUs to one Pod¶

Allocate two NPUs to one container and verify that they belong to the same RSD group.

Use the Device Plugin configuration. If the Operator is not installed, follow Installing the NPU Operator.

Available NPU count

To check how many NPUs remain available on a node, run kubectl describe node <NODE_NAME> and compare Allocatable with Allocated resources. Allocatable includes NPUs that are already allocated.

1. Request two NPUs¶

Create the npu-tutorial namespace and a Pod. Set the rebellions.ai/npu: 2 limit to request two NPUs for the container.

kubectl create namespace npu-tutorial --dry-run=client -o yaml | kubectl apply -f -
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: two-npus
  namespace: npu-tutorial
spec:
  containers:
  - name: ubuntu
    image: ubuntu:22.04
    command: ["sleep", "infinity"]
    resources:
      limits:
        rebellions.ai/npu: 2
EOF

The first line of the output depends on whether the namespace already exists.

Creation result

namespace/npu-tutorial configured
pod/two-npus created

$ kubectl wait -n npu-tutorial --for=condition=Ready pod/two-npus --timeout=180s
pod/two-npus condition met

Pod stays Pending

Run kubectl describe pod -n npu-tutorial two-npus and check Events. Insufficient rebellions.ai/npu means no eligible node has two available NPUs.

2. Verify the requested NPU count¶

Check the assigned node and the requested NPU count.

Rebel100™ATOM™

$ kubectl get pod -n npu-tutorial two-npus \
  -o 'custom-columns=NAME:.metadata.name,NODE:.spec.nodeName,NPUS:.spec.containers[0].resources.limits.rebellions\.ai/npu'
NAME       NODE            NPUS
two-npus   node-rebel100   2

$ kubectl exec -n npu-tutorial two-npus -- rbln-smi -L
NPU 0 : RBLN-CR13 (UUID: NPU-34428712-d887-840b-a7b9-1c9e05807fc3)
NPU 1 : RBLN-CR13 (UUID: NPU-9f4b0832-3c21-4d79-a62c-7816b0a27f45)

$ kubectl get pod -n npu-tutorial two-npus \
  -o 'custom-columns=NAME:.metadata.name,NODE:.spec.nodeName,NPUS:.spec.containers[0].resources.limits.rebellions\.ai/npu'
NAME       NODE        NPUS
two-npus   node-atom   2

$ kubectl exec -n npu-tutorial two-npus -- rbln-smi -L
NPU 0 : RBLN-CA25 (UUID: NPU-c5ba8327-5b18-847e-953e-4ca4b4daeb44)
NPU 1 : RBLN-CA25 (UUID: NPU-69bd0b6c-3c88-858f-bfc1-264c5c2f17b5)

The NPUS value must be 2. The device list must show two NPUs with different UUIDs.

Check their RSD group and device interface. jq runs on the machine where you run kubectl.

kubectl exec -n npu-tutorial two-npus -- rbln-smi -j -g | \
  jq '[.devices[] | {npu, name, group_id, status}]'

Rebel100™ATOM™

Example output

[
  {
    "npu": 0,
    "name": "RBLN-CR13",
    "group_id": "1",
    "status": "normal"
  },
  {
    "npu": 1,
    "name": "RBLN-CR13",
    "group_id": "1",
    "status": "normal"
  }
]

Example output

[
  {
    "npu": 0,
    "name": "RBLN-CA25",
    "group_id": "1",
    "status": "normal"
  },
  {
    "npu": 1,
    "name": "RBLN-CA25",
    "group_id": "1",
    "status": "normal"
  }
]

$ kubectl exec -n npu-tutorial two-npus -- sh -c \
  'test -c /dev/rsd0 && echo "/dev/rsd0: character device present"'
/dev/rsd0: character device present

The check passes when all three conditions hold:

Both devices have the same group_id.
Both devices report normal.
/dev/rsd0 is a character device.

If these checks fail, inspect the Device Plugin logs on the assigned node.

Clean up¶

$ kubectl delete pod -n npu-tutorial two-npus
pod "two-npus" deleted