Upgrade NPU drivers automatically¶

Configure the NPU Operator to upgrade drivers automatically.

Use the per-family driver configuration. If the Operator is not installed, follow Installing the NPU Operator.

Start with an Operator installed through Helm and managing RBLNDriver instances.

1. Check the current versions¶

This tutorial uses the atom and rebel100 RBLNDriver instances. Use the instance names from your installation. Check the number of Ready nodes.

$ kubectl get rblndriver \
  -o 'custom-columns=NAME:.metadata.name,VERSION:.spec.version,READY:.status.readyNodes,DESIRED:.status.desiredNodes'
NAME                   VERSION   READY   DESIRED
rbln-driver-atom       3.4.0     1       1
rbln-driver-rebel100   3.5.0     1       1

2. Set the target versions¶

In this tutorial, upgrade ATOM™ from 3.4.0 to 3.4.1 and Rebel100™ from 3.5.0 to 3.5.1. Keep the currently installed chart version and enable automatic upgrades.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

CHART_VERSION=$(helm get metadata rbln-npu-operator -n rbln-system -o json | jq -r .version)
export ATOM_TARGET_VERSION=3.4.1
export REBEL100_TARGET_VERSION=3.5.1
helm upgrade rbln-npu-operator oci://docker.io/rebellions/rbln-npu-operator-chart \
  --version "$CHART_VERSION" -n rbln-system --reuse-values \
  --set driver.upgradePolicy.autoUpgrade=true \
  --set driver.upgradePolicy.maxParallelUpgrades=1 \
  --set-string driver.instances.atom.image.tag="$ATOM_TARGET_VERSION" \
  --set-string driver.instances.rebel100.image.tag="$REBEL100_TARGET_VERSION" \
  --wait --timeout 10m

3. Verify the upgrade¶

Wait for the replacement driver Pod to become Ready, then check the instance versions and readiness.

$ kubectl get rblndriver \
  -o 'custom-columns=NAME:.metadata.name,VERSION:.spec.version,READY:.status.readyNodes,DESIRED:.status.desiredNodes'
NAME                   VERSION   READY   DESIRED
rbln-driver-atom       3.4.1     1       1
rbln-driver-rebel100   3.5.1     1       1

Read the kernel module version loaded on each node. If a driver Pod is still being replaced, repeat this command once it is Ready.

1
2
3
4
5
6
7

kubectl get pods -n rbln-system -l app.kubernetes.io/component=rbln-driver -o json | \
  jq -r '.items[] | [.metadata.name, .spec.nodeName] | @tsv' | \
  while IFS=$'\t' read -r pod node; do
    version=$(kubectl exec -n rbln-system "$pod" -c rbln-driver-container \
      -- cat /sys/module/rebellions/version) || exit 1
    printf '%s\t%s\n' "$node" "$version"
  done

node-atom   3.4.1
node-rebel100   3.5.1

ATOM™ must report 3.4.1 and Rebel100™ must report 3.5.1, with both instances ready.

Check the upgrade result.

$ kubectl get rblnclusterpolicy rbln-cluster-policy -o json | \
  jq '.status.driverUpgrade | {state, progress, failed}'
{
  "state": "Complete",
  "progress": "2/2",
  "failed": 0
}

If a node remains cordoned or the upgrade fails, follow automatic upgrade recovery before resuming workloads.