Workload Configuration by Node¶

This guide explains how to use node labels to select which nodes the NPU Operator manages and set each node's workload mode.

• Exclude a node from component deployment: Remove Operator-managed components from a node and exclude it from deployment.
• Select a workload mode: Configure a node for containers (container) or VM passthrough (vm-passthrough).

For a tutorial, see Separate NPU nodes for containers and VMs.

Excluding a node from component deployment¶

Use this setting for maintenance or separately managed nodes. Stop or move the node's NPU workloads first, because the Operator removes the components.

1

kubectl label node <NODE_NAME> rebellions.ai/npu.deploy.skip=true --overwrite

The Operator removes component deployment labels and excludes the node from automatic driver upgrades. Hardware discovery labels remain. This setting affects Operator-managed DaemonSets; it does not block separately installed components or scheduling of ordinary Pods.

Remove the label to resume deployment.

1

kubectl label node <NODE_NAME> rebellions.ai/npu.deploy.skip-

To skip only driver upgrades, use the upgrade skip label.

Selecting a workload mode¶

Choose the allocation method in the Helm settings for mixed container and VM workloads and merge its settings into the existing values.

Select a node's mode with this label.

1

kubectl label node <NODE_NAME> rebellions.ai/npu.workload.config=vm-passthrough --overwrite

Set container to select container mode regardless of the cluster default. Remove the label to follow the default again.

1
2

kubectl label node <NODE_NAME> rebellions.ai/npu.workload.config=container --overwrite
kubectl label node <NODE_NAME> rebellions.ai/npu.workload.config-

Switching from container mode to VM passthrough mode¶

To switch an NPU node from containers to VM passthrough, drain the node first. Resume scheduling after the components and resources are ready. Follow Separate NPU nodes for containers and VMs to configure and verify VM nodes.

Before switching a driver-bound NPU to vfio-pci, Driver Manager evicts remaining pods using rebellions.ai/npu or a container-mode DRA claim. It follows the NPU pod eviction policy. Passthrough claims are excluded; nodes already bound to vfio-pci skip eviction.

If eviction is blocked, the node remains cordoned while eviction is retried. Check rebellions.ai/npu-pod-eviction-blocked on the node for the cause.

Select the node and drain it. First apply the Helm settings for the chosen allocation method and complete the VM node prerequisites.

1
2
3

export VM_NODE='<VM_NODE_NAME>'
kubectl cordon "$VM_NODE"
kubectl drain "$VM_NODE" --ignore-daemonsets --timeout=300s

After drain succeeds, select VM passthrough mode.

1

kubectl label node "$VM_NODE" rebellions.ai/npu.workload.config=vm-passthrough --overwrite

Check the components and NPU resources for your allocation method.

1
2
3
4

kubectl get pods -n rbln-system --field-selector "spec.nodeName=$VM_NODE" \
  -l 'app.kubernetes.io/component in (vfio-manager,sandbox-device-plugin)'
kubectl get node "$VM_NODE" -o json | \
  jq '.status.allocatable | with_entries(select(.key | startswith("rebellions.ai/")))'

1
2
3
4
5
6
7
8

kubectl get pods -n rbln-system --field-selector "spec.nodeName=$VM_NODE" \
  -l 'app.kubernetes.io/component in (vfio-manager,dra-kubelet-plugin)'
kubectl get resourceslices -o json | \
  jq -r --arg node "$VM_NODE" '.items[] |
    select(.spec.driver == "npu.rebellions.ai" and .spec.pool.name == $node) |
    [.spec.pool.name,
     ([.spec.devices[]? | select(.attributes.type.string == "npu")] | length),
     ([.spec.devices[]? | select(.attributes.type.string == "vfio")] | length)] | @tsv'

After these checks pass, wait for the Operator to become ready.

1
2

kubectl wait --for=jsonpath='{.status.state}'=ready \
  rblnclusterpolicy/rbln-cluster-policy --timeout=600s

Once ready, resume scheduling.

1

kubectl uncordon "$VM_NODE"

Switching from VM passthrough mode to container mode¶

To switch an NPU node from VM passthrough to containers, stop its VMs and release their NPUs first. During the switch, Driver Manager in the driver Pod unbinds the devices from vfio-pci and installs the native driver.

Select the node and drain it.

1
2
3

export VM_NODE='<VM_NODE_NAME>'
kubectl cordon "$VM_NODE"
kubectl drain "$VM_NODE" --ignore-daemonsets --timeout=300s

After drain succeeds, follow the tab for your allocation method.

1

export NPU_RESOURCE='rebellions.ai/RBLN-CR13_PF'

1

export NPU_RESOURCE='rebellions.ai/RBLN-CA25_PF'

1
2

NPU_COUNT=$(kubectl get node "$VM_NODE" -o json | \
  jq -r --arg resource "$NPU_RESOURCE" '.status.allocatable[$resource]')

$ kubectl label node "$VM_NODE" rebellions.ai/npu.workload.config=container --overwrite
node/node-rebel100 labeled

$ kubectl wait --for=jsonpath='{.status.allocatable.rebellions\.ai/npu}'="$NPU_COUNT" \
  "node/$VM_NODE" --timeout=600s
node/node-rebel100 condition met

1

kubectl get pods -n rbln-system --field-selector "spec.nodeName=$VM_NODE"

NAME                                                                READY   STATUS    RESTARTS   AGE
rbln-container-toolkit-29xff                                        1/1     Running   0          58s
rbln-device-plugin-cr4l2                                            1/1     Running   0          58s
rbln-driver-rebel100-rebel100-ubuntu22.04-6.8.0-138-generic-f8fzt   1/1     Running   0          94s
rbln-driver-rebel100-smd-lgp4v                                      1/1     Running   0          58s
rbln-metrics-exporter-cfb56                                         1/1     Running   0          58s
rbln-npu-feature-discovery-qh7sp                                    1/1     Running   0          58s
rbln-operator-validator-qklt8                                       1/1     Running   0          58s

$ kubectl get node "$VM_NODE" \
  -o 'custom-columns=NAME:.metadata.name,NPUS:.status.allocatable.rebellions\.ai/npu'
NAME            NPUS
node-rebel100   1

$ kubectl label node "$VM_NODE" rebellions.ai/npu.workload.config=container --overwrite
node/node-atom labeled

$ kubectl wait --for=jsonpath='{.status.allocatable.rebellions\.ai/npu}'="$NPU_COUNT" \
  "node/$VM_NODE" --timeout=600s
node/node-atom condition met

1

kubectl get pods -n rbln-system --field-selector "spec.nodeName=$VM_NODE"

NAME                                                        READY   STATUS    RESTARTS   AGE
rbln-container-toolkit-29xff                                1/1     Running   0          58s
rbln-device-plugin-cr4l2                                    1/1     Running   0          58s
rbln-driver-atom-atom-ubuntu22.04-6.8.0-138-generic-f8fzt   1/1     Running   0          94s
rbln-driver-atom-smd-lgp4v                                  1/1     Running   0          58s
rbln-metrics-exporter-cfb56                                 1/1     Running   0          58s
rbln-npu-feature-discovery-qh7sp                            1/1     Running   0          58s
rbln-operator-validator-qklt8                               1/1     Running   0          58s

$ kubectl get node "$VM_NODE" \
  -o 'custom-columns=NAME:.metadata.name,NPUS:.status.allocatable.rebellions\.ai/npu'
NAME        NPUS
node-atom   16

1
2
3
4
5
6
7
8

kubectl label node "$VM_NODE" rebellions.ai/npu.workload.config=container --overwrite
kubectl get pods -n rbln-system --field-selector "spec.nodeName=$VM_NODE"
kubectl get resourceslices -o json | \
  jq -r --arg node "$VM_NODE" '.items[] |
    select(.spec.driver == "npu.rebellions.ai" and .spec.pool.name == $node) |
    [.spec.pool.name,
     ([.spec.devices[]? | select(.attributes.type.string == "npu")] | length),
     ([.spec.devices[]? | select(.attributes.type.string == "vfio")] | length)] | @tsv'

After checking the components and resources, wait for the Operator to become ready. If the switch stalls, inspect the Driver Manager initContainer logs and keep the node cordoned.

1
2

kubectl wait --for=jsonpath='{.status.state}'=ready \
  rblnclusterpolicy/rbln-cluster-policy --timeout=600s

Once ready, resume scheduling.

1

kubectl uncordon "$VM_NODE"