Skip to content

Node preparation

Goal: every node is ready for the stack:

  • The RoCE NIC driver and the RBLN Driver are loaded, so each node recognizes its NICs and NPUs.
  • The RBLN NPU Operator and dranet publish their DeviceClasses (npu.rebellions.ai and dranet).
  • Rebellions Direct Storage (RDS) is enabled in the RBLN NPU Operator, and the nodes that use RDS carry the label rebellions.ai/rds.present=true.

RoCE NIC driver on the host

Install the NIC vendor's RoCE driver on the host OS of each node. The reference nodes run the Broadcom bnxt_en and bnxt_re drivers on Ubuntu 22.04.

RBLN Driver and RBLN NPU Operator in DRA mode

Install the RBLN NPU Operator in DRA mode, as described in RBLN NPU DRA Driver. By default (devices.npu.mode: dra), the chart requests NPUs through Dynamic Resource Allocation (DRA). If the RBLN NPU Operator is not in DRA mode, the Operator does not publish the npu.rebellions.ai DeviceClass. The prefill and decode Pods then stay Pending with the event device class npu.rebellions.ai does not exist.

This page describes DRA mode. For a cluster without DRA, see Device-plugin mode.

The RBLN Driver creates the NPU device nodes (/dev/rbln*). The RBLN Driver runs in one of two driver modes:

  • Container driver mode (driver.enabled=true in the RBLN NPU Operator Helm values): the RBLN NPU Operator deploys and manages the RBLN Driver as a container. The reference nodes use this mode.
  • Host driver mode (driver.enabled=false, the default of the RBLN NPU Operator chart): you install the RBLN Driver on the host OS of each node before installing the RBLN NPU Operator.

Both container driver mode and host driver mode support RDS. You enable RDS separately with the RBLN NPU Operator Helm value rds.enabled (see RDS setup). To choose a driver mode, see NPU Driver Installation. For the procedure to install the RBLN Driver on the host, see the Installation Guide, "RBLN Driver" section.

For the installation and upgrade procedures of the RBLN NPU Operator, see Installing the RBLN NPU Operator and Upgrading the RBLN NPU Operator. For the node labels that describe the NPUs, see RBLN NPU Feature Discovery.

dranet for RoCE NICs

Install dranet after the RoCE NIC driver. The validated dranet version is listed in Software Stack and Versions. dranet publishes the RoCE NICs as DRA devices. dranet reports a NIC as able to use RDMA only when the NIC vendor's RDMA driver is loaded.

The ResourceClaimTemplate (RCT), not a chart value, defines which NICs a Pod receives. The RCT names the dranet DeviceClass and defines NIC selectors on the attributes that dranet publishes. To use a different RoCE DRA driver, change the DeviceClass name and the selectors in the RCT (Unverified). For devices.rdma.count, see Device-plugin mode.

RDS setup

In this stack, LMCache runs inside the prefill Pods and stores the tier 2 KV cache on RDS. In AGG recipes, LMCache runs inside the decode Pods. On each node that runs a Pod with LMCache, enable RDS in the RBLN NPU Operator and label the node rebellions.ai/rds.present=true. For the procedure, including how to select only the NVMe devices dedicated to RDS, see RDS Support.

The serving stack does not request RDS devices itself. On a node with RDS enabled, the NPU allocation plugin of the RBLN NPU Operator adds the RDS devices (/dev/rblnfs*) to every container that receives NPUs. After the stack is installed, list the RDS devices inside a prefill or decode Pod with sh -c 'ls /dev/rblnfs*'.

containerd LimitMEMLOCK drop-in

Raise the memlock limit of the containers (ulimit -l) to unlimited when both of the following hold:

  • devices.rdma.enabled is true, the default.
  • The Pods run in the default non-root run mode with uid 2000.

In P/D disaggregation, NIXL registers the KV-transfer buffers for RDMA. The registered memory counts toward the memlock limit.

At startup, the modelserver entrypoint tries to raise the memlock limit to unlimited. If the entrypoint cannot raise the memlock limit, the entrypoint acts on devices.rdma.enabled as follows:

  • With devices.rdma.enabled: true, the entrypoint logs an error and exits.
  • With devices.rdma.enabled: false, the entrypoint logs a warning and continues the startup.

If you cannot raise the containerd memlock limit on the nodes, use root mode (see Permissions and run mode).

# /etc/systemd/system/containerd.service.d/20-memlock.conf

[Service]
LimitMEMLOCK=infinity

containerd applies to each container the RLIMIT_MEMLOCK value of the Container Runtime Interface (CRI) base runtime spec (base_runtime_spec, for example /etc/containerd/cri-base.json). If the base runtime spec does not set RLIMIT_MEMLOCK, containerd applies the limit of the containerd process. Set the memlock limit of the containers on each node as follows.

  1. Check the base runtime spec. If the base runtime spec sets RLIMIT_MEMLOCK, change the RLIMIT_MEMLOCK entry so that the container limit becomes unlimited. Otherwise, add the 20-memlock.conf drop-in.

  2. Before restarting, clear leftover Pod sandboxes. containerd restores the CRI state at startup. If two Pod sandboxes have the same name, containerd exits abnormally. The node then reports Ready=False (container runtime is down), and you cannot recover the node from inside the cluster.

    1
    sudo crictl pods --state NotReady    # if a NotReady Pod sandbox remains, remove it first with sudo crictl rmp <pod-id>

  3. Restart containerd with sudo systemctl daemon-reload && sudo systemctl restart containerd.

On a node whose CRI is CRI-O, set the same memlock limit with default_ulimits = ["memlock=-1:-1"] in the [crio.runtime] table of a drop-in file under /etc/crio/crio.conf.d/, then restart CRI-O. On OpenShift, deploy the CRI-O drop-in file with a MachineConfig. The CRI-O setting is Unverified: the reference validation used containerd.

After the stack is installed, check the memlock limit of the modelserver container in a decode Pod with the following command. When the base runtime spec sets RLIMIT_MEMLOCK, the output of systemctl show containerd -p LimitMEMLOCK on the node differs from the actual limit of the containers.

1
kubectl -n <ns> exec deploy/<release-name>-decode -c modelserver -- sh -c 'ulimit -H -l'  # expected: unlimited

Node preparation checks

Run the following commands on each node.

1
2
ls /sys/class/infiniband/    # checks that the RoCE NIC driver is loaded
lspci -d 1eff:              # checks that the NPUs are detected (vendor ID 1eff; listed even if pci.ids has no name for 1eff)

To verify the RBLN Driver and the RBLN NPU Operator, follow Installing the RBLN NPU Operator › Verifying the installation. In host driver mode, ls /dev/rbln* /dev/rblnfs* on the node lists the NPU and RDS device nodes. In container driver mode, check the NPU and RDS device nodes inside a Pod, as described in Verify RDS devices in an NPU Pod.

Confirm with items (1) and (2) of Device allocation checks that the DeviceClasses and ResourceSlices are published. ResourceSlices may not be published immediately after the RBLN NPU Operator and dranet are installed.