Separate NPU nodes for containers and VMs¶

Separate nodes for container workloads and VM passthrough, then verify the NPU resources on each node.

Use the Helm settings for mixed container and VM workloads and complete the VM node prerequisites. If the Operator is not installed, follow Installing the NPU Operator.

Start with no NPU workloads on the selected nodes.

1. Apply the Helm values¶

Export the current release's user settings to npu-values.yaml.

helm get values rbln-npu-operator -n rbln-system -o yaml > npu-values.yaml

Choose the allocation method in the Helm settings for mixed container and VM workloads and merge its settings into npu-values.yaml.

Apply the updated values file using the currently installed chart version.

CHART_VERSION=$(helm get metadata rbln-npu-operator -n rbln-system -o json | jq -r .version)
helm upgrade rbln-npu-operator oci://docker.io/rebellions/rbln-npu-operator-chart \
  --version "$CHART_VERSION" -n rbln-system \
  --reset-values -f npu-values.yaml --wait --timeout 10m

2. Select the VM node¶

Replace <VM_NODE_NAME> with the node to use for VM passthrough. The label applies to all NPUs on that node. Other nodes use the default container mode.

export VM_NODE='<VM_NODE_NAME>'
kubectl label node "$VM_NODE" rebellions.ai/npu.workload.config=vm-passthrough --overwrite

3. Verify both node types¶

Check the components on the VM node and the NPU resources on each node.

Sandbox Device PluginDRA

Check the VM node's VFIO Manager and Sandbox Device Plugin Pods, and its product-specific NPU resources.

$ kubectl get pods -n rbln-system --field-selector "spec.nodeName=$VM_NODE" \
  -l 'app.kubernetes.io/component in (vfio-manager,sandbox-device-plugin)'
NAME                               READY   STATUS    RESTARTS   AGE
rbln-sandbox-device-plugin-87lj8   1/1     Running   0          2m42s
rbln-vfio-manager-w9c97            1/1     Running   0          2m48s

$ kubectl get node "$VM_NODE" -o json | \
  jq '.status.allocatable | with_entries(select(.key | startswith("rebellions.ai/")))'
{
  "rebellions.ai/RBLN-CA25_PF": "16",
  "rebellions.ai/npu": "0"
}

Compare the resource counts. The following example uses RBLN-CA25_PF on the VM node.

$ kubectl get nodes -l rebellions.ai/npu.present=true \
  -o 'custom-columns=NAME:.metadata.name,CONTAINER_NPUS:.status.allocatable.rebellions\.ai/npu,VM_NPUS:.status.allocatable.rebellions\.ai/RBLN-CA25_PF'
NAME            CONTAINER_NPUS   VM_NPUS
node-rebel100   1                <none>
node-atom       0                16

Both VM components must be Ready. The container node's CONTAINER_NPUS value and the VM node's VM_NPUS value must each be at least 1. A resource unused on a node can appear as 0 or <none>.

Check the VFIO Manager and DRA Kubelet Plugin Pods on the VM node.

$ kubectl get pods -n rbln-system --field-selector "spec.nodeName=$VM_NODE" \
  -l 'app.kubernetes.io/component in (vfio-manager,dra-kubelet-plugin)'
NAME                            READY   STATUS    RESTARTS   AGE
rbln-dra-kubelet-plugin-z72sz   1/1     Running   0          2m42s
rbln-vfio-manager-w9c97         1/1     Running   0          2m48s

Use DRA ResourceSlices to distinguish container devices (type: npu) from VM devices (type: vfio). The output columns show the node name, container NPU count and VM NPU count.

kubectl get resourceslices -o json | \
  jq -r '.items[] | select(.spec.driver == "npu.rebellions.ai") |
    [.spec.pool.name,
     ([.spec.devices[]? | select(.attributes.type.string == "npu")] | length),
     ([.spec.devices[]? | select(.attributes.type.string == "vfio")] | length)] | @tsv'

Example output

node-rebel100   1   0
node-atom   0   16

Both VM components must be Ready. Container nodes must list at least one npu device, and VM nodes must list at least one vfio device.

Check the status of both workload types.

$ kubectl get rblnclusterpolicy rbln-cluster-policy \
  -o jsonpath='{range .status.workloads[*]}{.type}{"\t"}{.state}{"\n"}{end}'
container   ready
vm-passthrough  ready

Both workload states must be ready.

The cluster is ready for container and VM workloads. To change the mode of a node that is already active, follow Switching workload by label.