PCIe Hotplug Unbinding on NPU Ports¶
Important
Apply this only to RHCOS nodes based on RHEL 10.2. RHCOS nodes based on RHEL 9.x do not need it.
Important
ATOM™ nodes do not need the configuration on this page.
When several NPUs behind the same PCIe switch are reset at once, the link can stay down long enough for the hotplug driver pciehp to remove the card from the kernel while it is in use. Unbinding pciehp from the downstream ports that host NPUs prevents this.
Deploy the pciehp-unbinding hook to RHCOS nodes with a MachineConfig.
Prerequisites¶
- OpenShift Container Platform 4.x cluster (validated on 4.21 and 4.22)
cluster-adminprivileges- OpenShift CLI (
oc) installed and logged in
Confirm the Ignition spec version the cluster runs. OCP 4.21 and 4.22 use 3.5.0, which is what the manifest below declares:
Which Pool to Apply It To¶
A MachineConfig applies to a MachineConfigPool, not to a single node. Apply it to the pool that holds the Rebel100™ nodes running RHCOS based on RHEL 10.2. Create Custom MachineConfigPool covers how a custom pool is built.
- First check which RHEL release the node's RHCOS is based on. If
VERSION_IDis not10.2, this page does not apply.
- Apply it to the whole pool. On a node that does not have it, the card can disappear while it is in use.
- A pool whose nodes are all VM guests can skip it. The command below prints nothing on a guest:
- A Rebel100™ pool mixing bare metal and VM nodes still gets it pool-wide. On VM nodes, the hook exits without making changes.
The manifest and the commands below use __MCP_NAME__ for the pool. Replace it with the pool that holds the Rebel100™ nodes: worker when every worker in the cluster is a Rebel100™ node, otherwise the custom pool. Use the same name in every command on this page. Only a three-node compact cluster, where the masters also carry NPUs, uses master; in that case the control plane reboots one node at a time, so watch API availability as well.
Manifest¶
The manifest deploys the following files to each node in the pool:
| Item | Path | Purpose |
|---|---|---|
| Hook script | /usr/local/sbin/rbln-pciehp-hook.sh | Unbinds pciehp from the downstream ports that host NPUs |
| udev rule | /etc/udev/rules.d/99-rbln-pciehp.rules | Starts the systemd service for the port when an RBLN device is added |
| systemd unit | rbln-pciehp-unbind@.service | Runs the hook for one port |
The hook script and the udev rule are embedded gzip-compressed. Apply them as they are; do not edit the contents.source values. Replace only __MCP_NAME__.
Applying¶
Warning
Applying this triggers a rolling reboot. For each node in the pool, MCO drains the node, writes the files, and reboots the node. A reboot is required for the udev rule to take effect.
The rollout is complete when the pool reports UPDATED=True, UPDATING=False and DEGRADED=False.
Verify¶
The hook script is expected to carry the bin_t SELinux type and ausearch is expected to report no denials.
On a VM guest status finds the ports and exits with rc=0, but no BOUND -> unbound transition is observed. That transition only appears on a bare metal node.
Rollback¶
Do not run rbln-pciehp-hook.sh uninstall on these nodes. MCO manages the deployed files. To roll back, delete the MachineConfig. This triggers a rolling reboot of the target pool.
Notes¶
Warning
Declare the unit under systemd.units. A unit written as a file under /etc/systemd/system through storage.files is removed by MCO as an undeclared unit, and the node does not converge.
The unit spells out RequiresMountsFor=/var/usrlocal/sbin rather than /usr/local/sbin. RequiresMountsFor= does not resolve symbolic links, and on an ostree host /usr/local is a symlink to /var/usrlocal with /var on a separate mount. udev coldplug for PCI runs before local-fs.target, so without the resolved path the unit would not start and pciehp would not be unbound at boot. Confirm the symlink on a node with: