Skip to content

Using PCI Pass-Through

PCI pass-through assigns a physical NPU directly to a single VM. In this mode, only the driver running in the assigned VM has exclusive access to the NPU, and the NPU is not shared between VMs.

Enable IOMMU and Configure Recommended Kernel Parameters

First, enable the following setting in the server platform BIOS.

  • IOMMU (Intel VT-d / AMD-Vi)

The BIOS menu names and configuration steps may vary depending on the system vendor.

PCI pass-through uses IOMMU to isolate DMA access from the NPU device assigned to the VM, protecting the host and other VMs' memory. PCI pass-through does not work when IOMMU is disabled.

Note

IOMMU is already enabled in the default kernels of the currently supported distributions.

The recommended kernel parameters for an NPU PCI pass-through environment are as follows. Add them to GRUB_CMDLINE_LINUX in /etc/default/grub.

Note

The example below assumes a server with an Intel processor. For AMD processors, omit intel_iommu=on; the IOMMU is active once AMD-Vi is enabled in the BIOS. Some kernel parameters may differ depending on the server and the installed cards.

1
transparent_hugepage=madvise pcie_aspm=force pci=pcie_bus_perf pci=bfsort pci=noats iommu=pt intel_iommu=on iommu.strict=1

Update the GRUB configuration and reboot.

1
2
$ sudo update-grub
$ sudo reboot

After the reboot, confirm that the parameters are applied:

$ cat /proc/cmdline

Using PCI Pass-Through on Linux KVM

You can configure NPU pass-through on Ubuntu Linux KVM using the virsh command.

The following prerequisites must be met before configuring NPU pass-through.

  • Ubuntu Linux KVM must be installed (sudo apt install qemu-kvm libvirt-daemon-system libvirt-clients bridge-utils).
  • The VM machine type must be q35. q35 is required for NPU pass-through because it supports the PCIe native topology.
  • The guest's 64-bit MMIO address space must be large enough for the cards assigned to the VM, as listed below. When you assign several cards to the same VM, scale the size with the number of cards. Also enable Above 4G Decoding in the host BIOS.

Provide 512 GB per RBLN-CR13 card.

Provide 128 GB per RBLN-CA25 card.

Configuring NPU Pass-Through to a VM with virsh

This procedure covers how to assign an NPU to an existing VM configured with the q35 machine type via pass-through. VM creation and basic configuration are out of scope for this document — see Ubuntu Server - Virtual Machine Manager and libvirt - PCI passthrough of host devices for those.

  1. Identify the PCI device BDF (bus/device/function) of the NPU to assign to the VM in pass-through mode.

    In the following example, the NPU's PCI device BDF is 1b:00.0.

    1
2
    $ lspci -nn | grep accelerators
1b:00.0 Processing accelerators [1200]: Device [1eff:2130] (rev 02)

    1
2
    $ lspci -nn | grep accelerators
1b:00.0 Processing accelerators [1200]: Device [1eff:1220] (rev 03)

  2. Convert the BDF identified in step 1 into the domain, bus, slot, and function format, specify it in the XML, and assign the NPU device to the VM using that XML file.

    An RBLN-CR13 card is a single PCI device, so one <hostdev> is enough.

    An RBLN-CA25 card is four PCI devices; attach all four to the same VM. An RBLN-CA22 card is a single PCI device, so one <hostdev> is enough.

    1
2
3
4
5
6
7
8
9
    $ cat > npu-device.xml << EOF
<hostdev mode='subsystem' type='pci' managed='yes'>
  <source>
    <address domain='0x0000' bus='0x1b' slot='0x00' function='0x0'/>
  </source>
</hostdev>
EOF

$ virsh attach-device vm-name npu-device.xml --config

    Note

    You can also configure this by editing the VM XML directly with virsh edit and adding the <hostdev> entry.

  3. Start the VM with the assigned NPU.

    1
    $ virsh start vm-name

  4. Connect to the VM and verify that the NPU has been assigned correctly.

    1
2
3
    $ virsh console vm-name
$ lspci -nn | grep accelerators
05:00.0 Processing accelerators [1200]: Device [1eff:2130] (rev 02)

    Run the following commands in the guest. Verify that Region 0 has an assigned memory address and shows [size=256G]. Check the kernel log for BAR allocation errors for the device, such as no space for.

    1
2
    $ sudo lspci -vv -d 1eff:2130
$ sudo dmesg | grep -i 'no space for'

    1
2
3
    $ virsh console vm-name
$ lspci -nn | grep accelerators
07:00.0 Processing accelerators [1200]: Device [1eff:1220] (rev 03)

Next Steps

After assigning the NPU to the VM, install the RBLN driver in the guest OS as explained in the RBLN Driver Installation Guide. The host does not require the RBLN driver for the NPU assigned to the VM. The guest OS and kernel must be in the support matrix.

Once the driver is installed, rbln-smi in the guest lists the NPU assigned to the VM.

$ rbln-smi