Run different driver versions on separate nodes¶

Assign driver versions to nodes with one of the following configurations:

• ATOM™ + Rebel100™: Install a different driver version for each family.
• Two Rebel100™ nodes: Use nodeSelector to assign a different driver version to each node.

If the Operator is not installed, follow Installing the NPU Operator.

1. Update the Helm values¶

Export the current release's user settings to npu-values.yaml.

1

helm get values rbln-npu-operator -n rbln-system -o yaml > npu-values.yaml

Choose one of the two examples and merge its driver settings into npu-values.yaml. Replace any existing driver.instances entries with those from the selected example, and retain the remaining settings.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

driver:
  enabled: true
  deployDefault: false
  instances:
    atom:
      image:
        tag: "3.4.1"
      nodeSelector:
        rebellions.ai/npu.family: atom
    rebel100:
      image:
        tag: "3.5.1"
      nodeSelector:
        rebellions.ai/npu.family: rebel100

1
2

kubectl label node <REBEL100_NODE_1> driver-group=stable --overwrite
kubectl label node <REBEL100_NODE_2> driver-group=canary --overwrite

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

driver:
  enabled: true
  deployDefault: false
  image:
    tag: "3.5.0"
  instances:
    rebel100-stable:
      nodeSelector:
        rebellions.ai/npu.family: rebel100
        driver-group: stable
    rebel100-canary:
      image:
        tag: "3.5.1"
      nodeSelector:
        rebellions.ai/npu.family: rebel100
        driver-group: canary

With driver.deployDefault: false, drivers are installed only on nodes selected by an instance. Include every NPU node that needs a driver in one of the instances. Values omitted from an instance, such as the registry and image pull secrets, are inherited from the shared driver settings.

2. Apply the Helm values¶

Apply the updated values file using the currently installed chart version.

1
2
3
4

CHART_VERSION=$(helm get metadata rbln-npu-operator -n rbln-system -o json | jq -r .version)
helm upgrade rbln-npu-operator oci://docker.io/rebellions/rbln-npu-operator-chart \
  --version "$CHART_VERSION" -n rbln-system \
  --reset-values -f npu-values.yaml --wait --timeout 10m

Wait for the RBLNDriver instances and the RBLNClusterPolicy instance to become ready.

1
2
3
4

kubectl wait --for=jsonpath='{.status.state}'=ready \
  rblndriver --all --timeout=600s
kubectl wait --for=jsonpath='{.status.state}'=ready \
  rblnclusterpolicy/rbln-cluster-policy --timeout=600s

3. Check each node's RBLNDriver instance¶

Each entry in driver.instances creates a resource named rbln-driver-<instance-name>. Check the driver versions and each node's rebellions.ai/npu.driver.owner label. The example output uses one node per instance; node names and counts depend on your environment.

$ kubectl get rblndriver \
  -o 'custom-columns=NAME:.metadata.name,VERSION:.spec.version,STATE:.status.state,READY:.status.readyNodes,DESIRED:.status.desiredNodes'
NAME                   VERSION   STATE   READY   DESIRED
rbln-driver-atom       3.4.1     ready   1       1
rbln-driver-rebel100   3.5.1     ready   1       1

$ kubectl get nodes -l rebellions.ai/npu.present=true \
  -o 'custom-columns=NAME:.metadata.name,DRIVER:.metadata.labels.rebellions\.ai/npu\.driver\.owner'
NAME            DRIVER
node-atom       rbln-driver-atom
node-rebel100   rbln-driver-rebel100

$ kubectl get rblndriver \
  -o 'custom-columns=NAME:.metadata.name,VERSION:.spec.version,STATE:.status.state,READY:.status.readyNodes,DESIRED:.status.desiredNodes'
NAME                          VERSION   STATE   READY   DESIRED
rbln-driver-rebel100-canary   3.5.1     ready   1       1
rbln-driver-rebel100-stable   3.5.0     ready   1       1

$ kubectl get nodes -l rebellions.ai/npu.present=true \
  -o 'custom-columns=NAME:.metadata.name,DRIVER:.metadata.labels.rebellions\.ai/npu\.driver\.owner'
NAME              DRIVER
node-rebel100-a   rbln-driver-rebel100-stable
node-rebel100-b   rbln-driver-rebel100-canary

Confirm that each node's DRIVER matches the selected instance and both instances are ready.