Skip to content

Sandboxed Workloads (KubeVirt)

The RBLN NPU Operator uses VFIO Manager to bind NPUs to vfio-pci and attach them to KubeVirt VMs through PCI passthrough. Choose either the Sandbox Device Plugin or DRA for NPU allocation.

Prerequisites

The table lists recommended minimum versions and prerequisites for each allocation method. Configure the cluster and worker nodes for your chosen method, and install KubeVirt so it is ready to run VMs.

Requirement Sandbox Device Plugin DRA Driver (VM Passthrough)
Kubernetes v1.19+ v1.34+
KubeVirt Kubernetes-compatible release v1.9.0+
NPU Operator v0.4.4+ v0.6.0+
NPU DRA Driver Not used v0.2.1+
Helm v3.9+ v3.9+
IOMMU Enabled in BIOS and the host kernel (intel_iommu=on or amd_iommu=on) Enabled in BIOS and the host kernel (intel_iommu=on or amd_iommu=on)
vIOMMU Required for VM-based worker nodes Required for VM-based worker nodes
VFIO kernel modules vfio, vfio_pci, vfio_iommu_type1 vfio, vfio_pci, vfio_iommu_type1
Node Feature Discovery (NFD) Installed in the cluster Installed in the cluster

For IOMMU configuration, see PCI Pass-Through.

VFIO Configuration

For the Helm settings of a VM-only cluster, see KubeVirt VM passthrough.

Hybrid Clusters

A hybrid cluster uses separate worker nodes for containers and VM passthrough. The default mode is container; use the workload label to select VM nodes. Mode changes apply to all NPUs on a node.

For Helm settings, see the Helm settings for mixed container and VM workloads. For mode behavior and label settings, see Workload Settings by Node.

Before changing an active node's mode, move or stop its workloads and follow Switching the Mode of a Node Currently in Use.

For examples of selecting nodes and checking resources, see Separate NPU nodes for containers and VMs.

Device Lifecycle

The following steps make an NPU available to a VM:

  1. Node selection: NFD discovers the NPU hardware, and the operator deploys components according to each node's workload mode.
  2. Preparation: VFIO Manager pauses related components and checks that no host workloads are using the NPU device files.
  3. VFIO binding: VFIO Manager detaches the NPU PCI devices from the host's existing driver and binds them to vfio-pci. The allocation plugin checks VFIO readiness before making the devices available for allocation.
  4. Allocation: When a VM requests an NPU, an available NPU is allocated through the selected method (Sandbox Device Plugin or DRA).
  5. VM attachment: KubeVirt attaches the allocated PCI device to the guest VM. NPU applications in the guest use the device through the RBLN Driver installed in the guest.

Shut down VMs with attached NPUs before switching allocation methods.

KubeVirt Configuration

List the KubeVirt resources and edit the one in use.

1
2
kubectl get kubevirt -A
kubectl edit kubevirt <KUBEVIRT_NAME> -n <KUBEVIRT_NAMESPACE>

Add HostDevices to the existing feature gate list. Keep any gates already enabled.

1
2
3
4
5
spec:
  configuration:
    developerConfiguration:
      featureGates:
      - HostDevices

For DRA, also add HostDevicesWithDRA to the same list. KubeVirt v1.9.0 requires both HostDevices and HostDevicesWithDRA for DRA host devices.

Follow Sandbox Device Plugin setup or DRA Driver setup for your chosen allocation method, then attach an NPU to a VM. For the steps from creating the VM to checking the NPU inside the guest, see Attach an NPU to a KubeVirt VM.