NPU Driver Manual Upgrade¶

The administrator manually deletes driver pods to control the timing and order of node upgrades.

Upgrade Configuration¶

Set driver.upgradePolicy.autoUpgrade: false in your existing Helm values.

driver:
  enabled: true
  upgradePolicy:
    autoUpgrade: false
    npuPodDeletion:
      force: false
      deleteEmptyDirData: false

Use npuPodDeletion to control which NPU pods may be evicted.

Setting	Effect	Default
`npuPodDeletion.force`	Allow eviction of NPU pods without a controller	`false`
`npuPodDeletion.deleteEmptyDirData`	Allow eviction of NPU pods using `emptyDir`	`false`

Upgrading Individual Nodes¶

Update and apply driver.image.tag or the target instance's driver.instances.<key>.image.tag in your existing Helm values. For a directly managed RBLNDriver, update spec.version.

Change the driver tag while preserving the existing Helm values and chart version. Replace <DRIVER_VERSION> with the desired driver version.

CHART_VERSION=$(helm get metadata rbln-npu-operator -n rbln-system -o json | jq -r .version)
export DRIVER_VERSION='<DRIVER_VERSION>'

Default driverIndividual driver instance

helm upgrade rbln-npu-operator oci://docker.io/rebellions/rbln-npu-operator-chart \
  --version "$CHART_VERSION" -n rbln-system --reuse-values \
  --set driver.upgradePolicy.autoUpgrade=false \
  --set-string driver.image.tag="$DRIVER_VERSION" \
  --wait --timeout 10m

Replace <INSTANCE_KEY> with an existing key under driver.instances, such as atom or rebel100.

export DRIVER_INSTANCE='<INSTANCE_KEY>'
helm upgrade rbln-npu-operator oci://docker.io/rebellions/rbln-npu-operator-chart \
  --version "$CHART_VERSION" -n rbln-system --reuse-values \
  --set driver.upgradePolicy.autoUpgrade=false \
  --set-string "driver.instances.${DRIVER_INSTANCE}.image.tag=${DRIVER_VERSION}" \
  --wait --timeout 10m

Find and delete the driver pod on the node you want to upgrade. The new pod uses the updated version.

kubectl get pods -n rbln-system -o wide -l app.kubernetes.io/component=rbln-driver
kubectl delete pod <DRIVER_POD> -n rbln-system

When the loaded driver is replaced, the node is cordoned and its NPU pods are evicted.

Upgrade Verification¶

Check that the new driver pod on the target node is Ready, then run rbln-smi to check the upgraded driver version.

kubectl get pods -n rbln-system -o wide -l app.kubernetes.io/component=rbln-driver
kubectl exec -n rbln-system <DRIVER_POD> -c rbln-driver-container -- rbln-smi

For sample output of these checks, see the loaded-version verification step in Upgrade NPU drivers automatically.

Eviction Issues and Retries¶

If eviction is blocked, the node remains cordoned. Eviction is retried when the initContainer restarts. Check the node's rebellions.ai/npu-pod-eviction-blocked annotation for the cause.

Set the Driver Manager eviction timeout with NPU_POD_EVICTION_TIMEOUT_SECONDS; the default is 300 seconds. After a timeout, the node remains cordoned while eviction is retried.