Skip to content

OpenStack

In OpenStack environments, you can assign NPUs to instances through the Compute (Nova) service.

Assigning NPUs with PCI Pass-Through

Identify the PCI address of the physical NPU device, register it in the Nova configuration, and assign it to instances through a flavor.

Prerequisites

  • OpenStack must be installed and configured.
  • IOMMU must be enabled on the compute nodes that have NPUs installed.
  • The compute node kernel must include vfio-pci support, and if it is provided as a module, vfio-pci must be loaded.

Note

This document is written against OpenStack Bobcat. Some setting names or behavior may differ in other OpenStack releases.

1. Identify NPU PCI Devices on the Compute Node

Identify the PCI address of the NPU on each compute node.

1
2
$ lspci -nn | grep accelerators
1b:00.0 Processing accelerators [1200]: Device [1eff:1220] (rev 03)

In the output above, the vendor_id is 1eff and the product_id is 1220. These values are used in the configuration that follows.

For IOMMU setup and the recommended kernel parameters for PCI pass-through, see Enable IOMMU and Configure Recommended Kernel Parameters.

3. Compute Node Nova Configuration (nova.conf)

Configure the PCI pass-through device specification in /etc/nova/nova.conf on each compute node.

Note

The configuration below is for RBLN-CA22. When using RBLN-CA25, refer to the RBLN-CA25 configuration in Product-specific Configuration.

1
2
[pci]
device_spec = { "vendor_id":"1eff", "product_id": "1220", "resource_class": "ATOM_PF" }

After applying the changes, restart the Nova Compute service.

4. Controller Node Nova Configuration (nova.conf)

Configure the PCI device alias in /etc/nova/nova.conf on the controller node. This alias is used when requesting an NPU from a flavor.

1
2
[pci]
alias =  { "vendor_id":"1eff", "resource_class":"ATOM_PF", "device_type":"type-PF", "name":"atom-pci" }

Note

The same [pci] alias configuration is also required on the node where nova-scheduler runs.

Enable PciPassthroughFilter in nova-scheduler.

1
2
3
[filter_scheduler]
enabled_filters = ...,PciPassthroughFilter
available_filters = nova.scheduler.filters.all_filters
The following is the recommended filter configuration for combining NPU PCI pass-through with the NUMA topology-based optimization covered in Performance Optimization Configuration. 
1
2
3
[filter_scheduler]
available_filters = nova.scheduler.filters.all_filters
enabled_filters = ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,PciPassthroughFilter,AggregateInstanceExtraSpecsFilter,NUMATopologyFilter

After applying the changes, restart the Nova API and Scheduler services.

5. Create the Image and NPU Flavor

Register the image with the q35 machine type and UEFI firmware specified for NPU PCI pass-through.

Note

NPU pass-through recommends the combination of the q35 machine type and UEFI firmware that supports the PCIe native topology. PCIe device enumeration and 64-bit BAR mapping may not work correctly in the legacy i440fx/BIOS environment.

1
2
3
4
5
6
7
$ openstack image create "Ubuntu-22.04-Cloud" \
    --file jammy-server-cloudimg-amd64.img \
    --disk-format qcow2 \
    --container-format bare \
    --public \
    --property hw_machine_type=q35 \
    --property hw_firmware_type=uefi

Create a flavor for instances that include an NPU.

1
$ openstack flavor create --ram 131072 --disk 50 --vcpus 8 ca22.n1.c8.m128

Add the NPU PCI device request to the flavor.

Note

hw:cpu_mode=host-passthrough exposes the host CPU model and all of its CPU features (such as virtualization extensions and IOMMU-related instructions) to the guest unchanged. PCI pass-through devices expect the same CPU behavior as on the host, so this mode keeps the NPU device stable.

1
$ openstack flavor set ca22.n1.c8.m128 --property "pci_passthrough:alias"="atom-pci:1" --property hw:cpu_mode=host-passthrough

Note

In atom-pci:1, the 1 is the number of NPU devices to assign to the instance. To assign multiple NPUs, increase the number (for example, atom-pci:2).

6. Create an NPU Instance

Launch an instance using the flavor you created.

1
2
3
4
$ openstack server create --flavor ca22.n1.c8.m128 \
    --image <image-name> \
    --network <network-name> \
    npu-instance

7. Verify NPU Assignment

After the instance is created, connect to it and verify that the NPU has been assigned correctly.

1
2
$ openstack server ssh npu-instance -- lspci -nn | grep accelerators
05:00.0 Processing accelerators [1200]: Device [1eff:1220] (rev 03)

Product-specific Configuration

This section describes product-specific configuration differences for OpenStack PCI pass-through.

RBLN-CA25 nova.conf Configuration

On RBLN-CA25, four PCI devices per slot make up a single NPU, so devices must be assigned in slot units (multiples of four). Specify each PCI address explicitly in device_spec and group them with a per-slot resource_class so that devices in the same slot are assigned to the same instance.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
[pci]
alias = { "vendor_id":"1eff", "resource_class":"ATOM_PCI", "device_type":"type-PCI", "name":"atom-pci" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:05:00.0", "resource_class": "ATOM_PCI_CA25_SLOT1" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:06:00.0", "resource_class": "ATOM_PCI_CA25_SLOT1" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:07:00.0", "resource_class": "ATOM_PCI_CA25_SLOT1" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:08:00.0", "resource_class": "ATOM_PCI_CA25_SLOT1" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:0c:00.0", "resource_class": "ATOM_PCI_CA25_SLOT2" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:0d:00.0", "resource_class": "ATOM_PCI_CA25_SLOT2" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:0e:00.0", "resource_class": "ATOM_PCI_CA25_SLOT2" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:0f:00.0", "resource_class": "ATOM_PCI_CA25_SLOT2" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:46:00.0", "resource_class": "ATOM_PCI_CA25_SLOT3" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:47:00.0", "resource_class": "ATOM_PCI_CA25_SLOT3" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:48:00.0", "resource_class": "ATOM_PCI_CA25_SLOT3" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:49:00.0", "resource_class": "ATOM_PCI_CA25_SLOT3" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:4d:00.0", "resource_class": "ATOM_PCI_CA25_SLOT4" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:4e:00.0", "resource_class": "ATOM_PCI_CA25_SLOT4" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:4f:00.0", "resource_class": "ATOM_PCI_CA25_SLOT4" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:50:00.0", "resource_class": "ATOM_PCI_CA25_SLOT4" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:85:00.0", "resource_class": "ATOM_PCI_CA25_SLOT5" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:86:00.0", "resource_class": "ATOM_PCI_CA25_SLOT5" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:87:00.0", "resource_class": "ATOM_PCI_CA25_SLOT5" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:88:00.0", "resource_class": "ATOM_PCI_CA25_SLOT5" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:8c:00.0", "resource_class": "ATOM_PCI_CA25_SLOT6" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:8d:00.0", "resource_class": "ATOM_PCI_CA25_SLOT6" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:8e:00.0", "resource_class": "ATOM_PCI_CA25_SLOT6" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:8f:00.0", "resource_class": "ATOM_PCI_CA25_SLOT6" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:c5:00.0", "resource_class": "ATOM_PCI_CA25_SLOT7" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:c6:00.0", "resource_class": "ATOM_PCI_CA25_SLOT7" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:c7:00.0", "resource_class": "ATOM_PCI_CA25_SLOT7" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:c8:00.0", "resource_class": "ATOM_PCI_CA25_SLOT7" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:cc:00.0", "resource_class": "ATOM_PCI_CA25_SLOT8" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:cd:00.0", "resource_class": "ATOM_PCI_CA25_SLOT8" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:ce:00.0", "resource_class": "ATOM_PCI_CA25_SLOT8" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:cf:00.0", "resource_class": "ATOM_PCI_CA25_SLOT8" }

Performance Optimization Configuration

To achieve optimal performance in an NPU virtualization environment, configure NUMA topology, CPU pinning, and Hugepages.

NUMA Topology Optimization

Placing the instance's vCPUs and memory on the same NUMA node as the NPU (hw:numa_nodes=1) provides the following performance benefits.

  • Reduced memory access latency: vCPUs and memory reside on the same NUMA node as the NPU, eliminating cross-node memory access (Remote Memory Access) through a remote NUMA node and significantly reducing memory latency.
  • PCIe transaction optimization: DMA transfers with the NPU use local memory on the same NUMA node, allowing maximum PCIe bandwidth utilization and removing the indirect path through the QPI/UPI interconnect.

Configure the NUMA policy on the flavor.

1
2
$ openstack flavor set ca22.n1.c8.m128 \
    --property hw:numa_nodes=1

Note

When the requested resources cannot fit on a single NUMA node — for example, when assigning all NPUs installed on the host to a single instance — and the NPU slots are distributed across two NUMA nodes, set hw:numa_nodes=2. For an even split, Nova divides vCPUs and memory automatically; specify hw:numa_cpus.N and hw:numa_mem.N only when an uneven split is required.

1
2
$ openstack flavor set ca22.n8.c64.m1024 \
    --property hw:numa_nodes=2

CPU Pinning

Pinning vCPUs to physical CPU cores eliminates context-switching overhead and increases the L1/L2 cache hit rate, ensuring stable latency for data transfers with the NPU. It also prevents core contention with other instances (noisy neighbor), providing consistent p99 latency.

Specify the CPUs to pin in /etc/nova/nova.conf on the compute node.

1
2
3
[compute]
cpu_dedicated_set = 0-43,48-91
cpu_shared_set = 44-47,92-95

Note

cpu_dedicated_set and cpu_shared_set must be configured so that SMT sibling pairs are not split. The lscpu -e command shows the per-NUMA-node CPU and sibling mapping. hw:cpu_thread_policy=require places vCPUs on sibling pairs to use hyper-threading; use isolate when exclusive core ownership is required (consumes 2 logical CPUs per vCPU).

Configure CPU pinning on the flavor.

1
2
3
$ openstack flavor set ca22.n1.c8.m128 \
    --property hw:cpu_policy=dedicated \
    --property hw:cpu_thread_policy=require

Hugepages

In a PCI pass-through environment, VFIO must lock (mlock) the entire guest memory in the host. Using 1GB Hugepages eliminates the memory pinning overhead and minimizes EPT walks, improving instance allocation stability and runtime performance.

Add the Hugepage kernel parameters to /etc/default/grub on the compute node.

1
GRUB_CMDLINE_LINUX="... default_hugepagesz=1G hugepagesz=1G hugepages=960"

Update GRUB and reboot.

1
$ update-grub && reboot

Configure Hugepages on the flavor.

1
2
$ openstack flavor set ca22.n1.c8.m128 \
    --property hw:mem_page_size=1GB

Combined Flavor Example

1
2
3
4
5
6
7
$ openstack flavor set ca22.n1.c8.m128 \
    --property "pci_passthrough:alias"="atom-pci:1" \
    --property hw:cpu_mode=host-passthrough \
    --property hw:cpu_policy=dedicated \
    --property hw:cpu_thread_policy=require \
    --property hw:numa_nodes=1 \
    --property hw:mem_page_size=1GB