OpenStack¶

In OpenStack environments, you can assign NPUs to instances through the Compute (Nova) service.

Assigning NPUs with PCI Pass-Through¶

Identify the PCI address of the physical NPU device, register it in the Nova configuration, and assign it to instances through a flavor.

Prerequisites¶

OpenStack must be installed and configured.
IOMMU must be enabled on the compute nodes that have NPUs installed.
The compute node kernel must include vfio-pci support, and if it is provided as a module, vfio-pci must be loaded.

Note

This document is written against OpenStack Bobcat. Some setting names or behavior may differ in other OpenStack releases.

Rebel100™ATOM™

1. Identify NPU PCI Devices on the Compute Node¶

Identify the PCI address of the NPU on each compute node.

$ lspci -nn | grep accelerators
03:00.0 Processing accelerators [1200]: Rebellions Inc. RBLN-CR13 (PF) [1eff:2130] (rev 02)

In the output, the vendor_id is 1eff and the product_id of RBLN-CR13 is 2130. These values are used in the configuration that follows.

2. Enable IOMMU and Configure Recommended Kernel Parameters¶

For IOMMU setup and the recommended kernel parameters for PCI pass-through, see Enable IOMMU and Configure Recommended Kernel Parameters.

3. Compute Node Nova Configuration (nova.conf)¶

Configure the PCI pass-through device specification in /etc/nova/nova.conf on each compute node.

Note

The following is the nova.conf configuration for Rebel100™ (RBLN-CR13). The product_id values are listed in the PCI identifier table.

[pci]
device_spec = { "vendor_id":"1eff", "product_id": "2130", "resource_class": "RBLN_CR13" }

After applying the changes, restart the Nova Compute service.

4. Controller Node Nova Configuration (nova.conf)¶

Configure the PCI device alias in /etc/nova/nova.conf on the controller node. This alias is used when requesting an NPU from a flavor.

[pci]
alias = { "vendor_id":"1eff", "resource_class":"RBLN_CR13", "name":"RBLN-CR13" }

Note

The same [pci] alias configuration is also required on the node where nova-scheduler runs.

Enable PciPassthroughFilter in nova-scheduler.

[filter_scheduler]
enabled_filters = ...,PciPassthroughFilter
available_filters = nova.scheduler.filters.all_filters

The following is the recommended filter configuration for combining NPU PCI pass-through with the NUMA topology-based optimization covered in Performance Optimization Configuration.

[filter_scheduler]
available_filters = nova.scheduler.filters.all_filters
enabled_filters = ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,PciPassthroughFilter,AggregateInstanceExtraSpecsFilter,NUMATopologyFilter

After applying the changes, restart the Nova API and Scheduler services.

5. Create the Image and NPU Flavor¶

Register the image with the q35 machine type and UEFI firmware specified for NPU PCI pass-through.

Note

The combination of the q35 machine type and UEFI firmware, which supports the PCIe native topology, is recommended for NPU pass-through. PCIe device enumeration and 64-bit BAR mapping may not work correctly in the legacy i440fx/BIOS environment.

$ openstack image create "Ubuntu-22.04-Cloud" \
    --file jammy-server-cloudimg-amd64.img \
    --disk-format qcow2 \
    --container-format bare \
    --public \
    --property hw_machine_type=q35 \
    --property hw_firmware_type=uefi

Create a flavor for instances that include an NPU and add the NPU PCI device request.

Note

hw:cpu_mode=host-passthrough exposes the host CPU model and all of its CPU features (such as virtualization extensions and IOMMU-related instructions) to the guest unchanged. PCI pass-through devices expect the same CPU behavior as on the host, so this mode keeps the NPU device stable.

$ openstack flavor create --ram 131072 --disk 50 --vcpus 8 cr13.n1.c8.m128
$ openstack flavor set cr13.n1.c8.m128 --property "pci_passthrough:alias"="RBLN-CR13:1" --property hw:cpu_mode=host-passthrough

Note

In RBLN-CR13:1, the 1 is the number of NPU devices to assign to the instance. To assign multiple NPUs, increase the number (for example, RBLN-CR13:2).

6. Create an NPU Instance¶

Launch an instance using the flavor you created.

$ openstack server create --flavor cr13.n1.c8.m128 \
    --image <IMAGE_NAME> \
    --network <NETWORK_NAME> \
    npu-instance

7. Verify NPU Assignment¶

After the instance is created, connect to it and verify that the NPU has been assigned correctly.

$ openstack server ssh npu-instance -- lspci -nn | grep accelerators
05:00.0 Processing accelerators [1200]: Rebellions Inc. RBLN-CR13 (PF) [1eff:2130] (rev 02)

1. Identify NPU PCI Devices on the Compute Node¶

Identify the PCI address of the NPU on each compute node.

ATOM™ (RBLN-CA22) appears as follows.

$ lspci -nn | grep accelerators
1b:00.0 Processing accelerators [1200]: Device [1eff:1220] (rev 03)

ATOM™-Max (RBLN-CA25) appears as follows.

$ lspci -nn | grep accelerators
05:00.0 Processing accelerators [1200]: Rebellions Inc. RBLN-CA25 (PF) [1eff:1250] (rev 03)

In the output, the vendor_id is 1eff and the product_id is 1220 for RBLN-CA22 and 1250 for RBLN-CA25. These values are used in the configuration that follows.

2. Enable IOMMU and Configure Recommended Kernel Parameters¶

For IOMMU setup and the recommended kernel parameters for PCI pass-through, see Enable IOMMU and Configure Recommended Kernel Parameters.

3. Compute Node Nova Configuration (nova.conf)¶

Configure the PCI pass-through device specification in /etc/nova/nova.conf on each compute node.

Note

The following is the nova.conf configuration for ATOM™ (RBLN-CA22). The product_id values are listed in the PCI identifier table.

Note

In the ATOM™ family, ATOM™-Max (RBLN-CA25) must be assigned per slot (four PCI devices) and therefore requires a separate configuration. For the device configuration and the flavor, refer to ATOM™-Max (RBLN-CA25) Configuration.

[pci]
device_spec = { "vendor_id":"1eff", "product_id": "1220", "resource_class": "RBLN_CA22" }

After applying the changes, restart the Nova Compute service.

4. Controller Node Nova Configuration (nova.conf)¶

Configure the PCI device alias in /etc/nova/nova.conf on the controller node. This alias is used when requesting an NPU from a flavor.

[pci]
alias = { "vendor_id":"1eff", "resource_class":"RBLN_CA22", "name":"RBLN-CA22" }

Note

The same [pci] alias configuration is also required on the node where nova-scheduler runs.

Enable PciPassthroughFilter in nova-scheduler.

[filter_scheduler]
enabled_filters = ...,PciPassthroughFilter
available_filters = nova.scheduler.filters.all_filters

The following is the recommended filter configuration for combining NPU PCI pass-through with the NUMA topology-based optimization covered in Performance Optimization Configuration.

[filter_scheduler]
available_filters = nova.scheduler.filters.all_filters
enabled_filters = ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,PciPassthroughFilter,AggregateInstanceExtraSpecsFilter,NUMATopologyFilter

After applying the changes, restart the Nova API and Scheduler services.

5. Create the Image and NPU Flavor¶

Register the image with the q35 machine type and UEFI firmware specified for NPU PCI pass-through.

Note

The combination of the q35 machine type and UEFI firmware, which supports the PCIe native topology, is recommended for NPU pass-through. PCIe device enumeration and 64-bit BAR mapping may not work correctly in the legacy i440fx/BIOS environment.

$ openstack image create "Ubuntu-22.04-Cloud" \
    --file jammy-server-cloudimg-amd64.img \
    --disk-format qcow2 \
    --container-format bare \
    --public \
    --property hw_machine_type=q35 \
    --property hw_firmware_type=uefi

Create a flavor for instances that include an NPU and add the NPU PCI device request.

Note

hw:cpu_mode=host-passthrough exposes the host CPU model and all of its CPU features (such as virtualization extensions and IOMMU-related instructions) to the guest unchanged. PCI pass-through devices expect the same CPU behavior as on the host, so this mode keeps the NPU device stable.

$ openstack flavor create --ram 131072 --disk 50 --vcpus 8 ca22.n1.c8.m128
$ openstack flavor set ca22.n1.c8.m128 --property "pci_passthrough:alias"="RBLN-CA22:1" --property hw:cpu_mode=host-passthrough

Note

In RBLN-CA22:1, the 1 is the number of NPU devices to assign to the instance. To assign multiple NPUs, increase the number (for example, RBLN-CA22:2).

6. Create an NPU Instance¶

Launch an instance using the flavor you created.

$ openstack server create --flavor ca22.n1.c8.m128 \
    --image <IMAGE_NAME> \
    --network <NETWORK_NAME> \
    npu-instance

7. Verify NPU Assignment¶

After the instance is created, connect to it and verify that the NPU has been assigned correctly.

$ openstack server ssh npu-instance -- lspci -nn | grep accelerators
05:00.0 Processing accelerators [1200]: Device [1eff:1220] (rev 03)

ATOM™-Max (RBLN-CA25) Configuration¶

ATOM™-Max (RBLN-CA25) uses the configuration below instead of the device_spec in step 3, the alias in step 4, and the flavor in steps 5 and 6. The scheduler filters and the image registration are the same as in steps 4 and 5.

On RBLN-CA25, four PCI devices per slot make up a single NPU, so devices must be assigned in slot units (multiples of four). Specify each PCI address explicitly in device_spec and group them with a per-slot resource_class so that devices in the same slot are assigned to the same instance.

The PCI addresses (BDF) of the devices depend on the server model, not on the NPU. Identify the addresses on each compute node before writing the configuration:

$ lspci -d 1eff:1250
00.0 Processing accelerators: Device 1eff:1250 (rev 03)
00.0 Processing accelerators: Device 1eff:1250 (rev 03)
00.0 Processing accelerators: Device 1eff:1250 (rev 03)
00.0 Processing accelerators: Device 1eff:1250 (rev 03)
00.0 Processing accelerators: Device 1eff:1250 (rev 03)
00.0 Processing accelerators: Device 1eff:1250 (rev 03)
00.0 Processing accelerators: Device 1eff:1250 (rev 03)
00.0 Processing accelerators: Device 1eff:1250 (rev 03)
...

The four devices that belong to one slot appear as four consecutive bus numbers (23–26 in the output above form one slot). Assign one resource_class per group of four, numbering the slots in ascending address order.

Fill in the addresses identified on the node and repeat the four-line group for every slot on the compute node:

[pci]
alias = { "vendor_id":"1eff", "resource_class":"RBLN_CA25", "device_type":"type-PCI", "name":"RBLN-CA25" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:<SLOT1_BUS_1>:00.0", "resource_class": "RBLN_CA25_SLOT1" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:<SLOT1_BUS_2>:00.0", "resource_class": "RBLN_CA25_SLOT1" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:<SLOT1_BUS_3>:00.0", "resource_class": "RBLN_CA25_SLOT1" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:<SLOT1_BUS_4>:00.0", "resource_class": "RBLN_CA25_SLOT1" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:<SLOT2_BUS_1>:00.0", "resource_class": "RBLN_CA25_SLOT2" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:<SLOT2_BUS_2>:00.0", "resource_class": "RBLN_CA25_SLOT2" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:<SLOT2_BUS_3>:00.0", "resource_class": "RBLN_CA25_SLOT2" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:<SLOT2_BUS_4>:00.0", "resource_class": "RBLN_CA25_SLOT2" }

The following configurations are verified on specific server models.

Warning

The PCI addresses below are reference values captured on a specific system. Even on the same server model, the actual addresses can differ because the detailed hardware configuration may vary between systems. Before applying a configuration, run the lspci command above on the compute node and confirm that every address matches; if any address differs, write the device_spec entries from the addresses identified on the node instead.

Supermicro AS-4125GS-TNRT2 (8 slots)

[pci]
alias = { "vendor_id":"1eff", "resource_class":"RBLN_CA25", "device_type":"type-PCI", "name":"RBLN-CA25" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:05:00.0", "resource_class": "RBLN_CA25_SLOT1" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:06:00.0", "resource_class": "RBLN_CA25_SLOT1" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:07:00.0", "resource_class": "RBLN_CA25_SLOT1" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:08:00.0", "resource_class": "RBLN_CA25_SLOT1" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:0c:00.0", "resource_class": "RBLN_CA25_SLOT2" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:0d:00.0", "resource_class": "RBLN_CA25_SLOT2" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:0e:00.0", "resource_class": "RBLN_CA25_SLOT2" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:0f:00.0", "resource_class": "RBLN_CA25_SLOT2" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:46:00.0", "resource_class": "RBLN_CA25_SLOT3" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:47:00.0", "resource_class": "RBLN_CA25_SLOT3" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:48:00.0", "resource_class": "RBLN_CA25_SLOT3" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:49:00.0", "resource_class": "RBLN_CA25_SLOT3" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:4d:00.0", "resource_class": "RBLN_CA25_SLOT4" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:4e:00.0", "resource_class": "RBLN_CA25_SLOT4" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:4f:00.0", "resource_class": "RBLN_CA25_SLOT4" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:50:00.0", "resource_class": "RBLN_CA25_SLOT4" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:85:00.0", "resource_class": "RBLN_CA25_SLOT5" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:86:00.0", "resource_class": "RBLN_CA25_SLOT5" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:87:00.0", "resource_class": "RBLN_CA25_SLOT5" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:88:00.0", "resource_class": "RBLN_CA25_SLOT5" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:8c:00.0", "resource_class": "RBLN_CA25_SLOT6" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:8d:00.0", "resource_class": "RBLN_CA25_SLOT6" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:8e:00.0", "resource_class": "RBLN_CA25_SLOT6" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:8f:00.0", "resource_class": "RBLN_CA25_SLOT6" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:c5:00.0", "resource_class": "RBLN_CA25_SLOT7" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:c6:00.0", "resource_class": "RBLN_CA25_SLOT7" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:c7:00.0", "resource_class": "RBLN_CA25_SLOT7" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:c8:00.0", "resource_class": "RBLN_CA25_SLOT7" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:cc:00.0", "resource_class": "RBLN_CA25_SLOT8" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:cd:00.0", "resource_class": "RBLN_CA25_SLOT8" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:ce:00.0", "resource_class": "RBLN_CA25_SLOT8" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:cf:00.0", "resource_class": "RBLN_CA25_SLOT8" }

GIGABYTE G294-Z21-AAP2 (4 slots)

[pci]
alias = { "vendor_id":"1eff", "resource_class":"RBLN_CA25", "device_type":"type-PCI", "name":"RBLN-CA25" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:23:00.0", "resource_class": "RBLN_CA25_SLOT1" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:24:00.0", "resource_class": "RBLN_CA25_SLOT1" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:25:00.0", "resource_class": "RBLN_CA25_SLOT1" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:26:00.0", "resource_class": "RBLN_CA25_SLOT1" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:43:00.0", "resource_class": "RBLN_CA25_SLOT2" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:44:00.0", "resource_class": "RBLN_CA25_SLOT2" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:45:00.0", "resource_class": "RBLN_CA25_SLOT2" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:46:00.0", "resource_class": "RBLN_CA25_SLOT2" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:a3:00.0", "resource_class": "RBLN_CA25_SLOT3" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:a4:00.0", "resource_class": "RBLN_CA25_SLOT3" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:a5:00.0", "resource_class": "RBLN_CA25_SLOT3" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:a6:00.0", "resource_class": "RBLN_CA25_SLOT3" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:c3:00.0", "resource_class": "RBLN_CA25_SLOT4" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:c4:00.0", "resource_class": "RBLN_CA25_SLOT4" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:c5:00.0", "resource_class": "RBLN_CA25_SLOT4" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:c6:00.0", "resource_class": "RBLN_CA25_SLOT4" }

GIGABYTE G494-ZB4-AAP2 (8 slots)

[pci]
alias = { "vendor_id":"1eff", "resource_class":"RBLN_CA25", "device_type":"type-PCI", "name":"RBLN-CA25" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:05:00.0", "resource_class": "RBLN_CA25_SLOT1" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:06:00.0", "resource_class": "RBLN_CA25_SLOT1" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:07:00.0", "resource_class": "RBLN_CA25_SLOT1" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:08:00.0", "resource_class": "RBLN_CA25_SLOT1" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:0c:00.0", "resource_class": "RBLN_CA25_SLOT2" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:0d:00.0", "resource_class": "RBLN_CA25_SLOT2" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:0e:00.0", "resource_class": "RBLN_CA25_SLOT2" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:0f:00.0", "resource_class": "RBLN_CA25_SLOT2" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:5c:00.0", "resource_class": "RBLN_CA25_SLOT3" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:5d:00.0", "resource_class": "RBLN_CA25_SLOT3" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:5e:00.0", "resource_class": "RBLN_CA25_SLOT3" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:5f:00.0", "resource_class": "RBLN_CA25_SLOT3" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:63:00.0", "resource_class": "RBLN_CA25_SLOT4" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:64:00.0", "resource_class": "RBLN_CA25_SLOT4" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:65:00.0", "resource_class": "RBLN_CA25_SLOT4" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:66:00.0", "resource_class": "RBLN_CA25_SLOT4" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:8a:00.0", "resource_class": "RBLN_CA25_SLOT5" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:8b:00.0", "resource_class": "RBLN_CA25_SLOT5" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:8c:00.0", "resource_class": "RBLN_CA25_SLOT5" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:8d:00.0", "resource_class": "RBLN_CA25_SLOT5" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:91:00.0", "resource_class": "RBLN_CA25_SLOT6" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:92:00.0", "resource_class": "RBLN_CA25_SLOT6" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:93:00.0", "resource_class": "RBLN_CA25_SLOT6" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:94:00.0", "resource_class": "RBLN_CA25_SLOT6" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:de:00.0", "resource_class": "RBLN_CA25_SLOT7" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:df:00.0", "resource_class": "RBLN_CA25_SLOT7" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:e0:00.0", "resource_class": "RBLN_CA25_SLOT7" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:e1:00.0", "resource_class": "RBLN_CA25_SLOT7" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:e5:00.0", "resource_class": "RBLN_CA25_SLOT8" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:e6:00.0", "resource_class": "RBLN_CA25_SLOT8" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:e7:00.0", "resource_class": "RBLN_CA25_SLOT8" }
device_spec = { "vendor_id":"1eff", "product_id": "1250", "address": "0000:e8:00.0", "resource_class": "RBLN_CA25_SLOT8" }

Create a flavor for RBLN-CA25 instances and add the NPU PCI device request. One slot consists of four PCI devices, so the device request count in the flavor must be a multiple of four. Request 4 per NPU.

$ openstack flavor create --ram 131072 --disk 50 --vcpus 8 ca25.n1.c8.m128
$ openstack flavor set ca25.n1.c8.m128 --property "pci_passthrough:alias"="RBLN-CA25:4" --property hw:cpu_mode=host-passthrough

Note

In RBLN-CA25:4, the 4 is the number of PCI devices in one slot (one NPU). To assign multiple NPUs, increase the number in multiples of four (for example, RBLN-CA25:8 for two NPUs). Do not use a value that is not a multiple of four, because the devices of one slot could then be split across instances.

Launch an instance using the flavor you created.

$ openstack server create --flavor ca25.n1.c8.m128 \
    --image <IMAGE_NAME> \
    --network <NETWORK_NAME> \
    npu-instance

Performance Optimization Configuration¶

To achieve optimal performance in an NPU virtualization environment, configure NUMA topology, CPU pinning, and Hugepages.

NUMA Topology Optimization¶

Placing the instance's vCPUs and memory on the same NUMA node as the NPU (hw:numa_nodes=1) provides the following performance benefits.

Reduced memory access latency: vCPUs and memory reside on the same NUMA node as the NPU, eliminating cross-node memory access (Remote Memory Access) through a remote NUMA node and significantly reducing memory latency.
PCIe transaction optimization: DMA transfers with the NPU use local memory on the same NUMA node, allowing maximum PCIe bandwidth utilization and removing the indirect path through the QPI/UPI interconnect.

Configure the NUMA policy on the flavor.

$ openstack flavor set <FLAVOR_NAME> \
    --property hw:numa_nodes=1

Note

When the requested resources cannot fit on a single NUMA node — for example, when assigning all NPUs installed on the host to a single instance — and the NPU slots are distributed across two NUMA nodes, set hw:numa_nodes=2. For an even split, Nova divides vCPUs and memory automatically; specify hw:numa_cpus.N and hw:numa_mem.N only when an uneven split is required.

$ openstack flavor set <FLAVOR_NAME> \
    --property hw:numa_nodes=2

CPU Pinning¶

Pinning vCPUs to physical CPU cores eliminates context-switching overhead and increases the L1/L2 cache hit rate, ensuring stable latency for data transfers with the NPU. It also prevents core contention with other instances (noisy neighbor), providing consistent p99 latency.

Specify the CPUs to pin in /etc/nova/nova.conf on the compute node.

[compute]
cpu_dedicated_set = 0-43,48-91
cpu_shared_set = 44-47,92-95

Note

cpu_dedicated_set and cpu_shared_set must be configured so that SMT sibling pairs are not split. The lscpu -e command shows the per-NUMA-node CPU and sibling mapping. hw:cpu_thread_policy=require places vCPUs on sibling pairs to use hyper-threading; use isolate when exclusive core ownership is required (consumes 2 logical CPUs per vCPU).

Configure CPU pinning on the flavor.

$ openstack flavor set <FLAVOR_NAME> \
    --property hw:cpu_policy=dedicated \
    --property hw:cpu_thread_policy=require

Hugepages¶

In a PCI pass-through environment, VFIO must lock (mlock) the entire guest memory in the host. Using 1GB Hugepages eliminates the memory pinning overhead and minimizes EPT walks, improving instance allocation stability and runtime performance.

Add the Hugepage kernel parameters to /etc/default/grub on the compute node.

1	`GRUB_CMDLINE_LINUX="... default_hugepagesz=1G hugepagesz=1G hugepages=960"`

Update GRUB and reboot.

$ update-grub && reboot

Configure Hugepages on the flavor.

$ openstack flavor set <FLAVOR_NAME> \
    --property hw:mem_page_size=1GB

Combined Flavor Example¶

The following example applies every optimization property to a flavor. Set <FLAVOR_NAME> and <ALIAS> to the flavor name and alias created in step 5 (for example, RBLN-CR13:1 or RBLN-CA22:1).

$ openstack flavor set <FLAVOR_NAME> \
    --property "pci_passthrough:alias"="<ALIAS>" \
    --property hw:cpu_mode=host-passthrough \
    --property hw:cpu_policy=dedicated \
    --property hw:cpu_thread_policy=require \
    --property hw:numa_nodes=1 \
    --property hw:mem_page_size=1GB