NPU Driver Auto Upgrade¶

Upgrade Policy¶

Node States¶

Read the current state from the rebellions.ai/npu-driver-upgrade-state node label.

maxParallelUpgrades limits concurrent upgrades. Failed nodes also hold a slot; skipped and completed nodes do not.

Rollout Status¶

Query the rollout status with the following commands:

1
2

kubectl get rblnclusterpolicy
kubectl get rblnclusterpolicy <POLICY_NAME> -o yaml

status.driverUpgrade contains the following fields.

Upgrade Conditions¶

Check the status and reason of the following conditions under status.conditions.

When UpgradeIncomplete=False, the reason is RolloutInProgress if skipped nodes exist, or NoSkippedNodes otherwise. If failed nodes occupy every upgrade slot and no nodes are in progress, the conditions report UpgradeDegraded=True and UpgradeStalled=False.

Verifying the Running Driver¶

Check that the new driver pod on the target node is Ready, then run rbln-smi to check the upgraded driver version.

1
2

kubectl get pods -n rbln-system -o wide -l app.kubernetes.io/component=rbln-driver
kubectl exec -n rbln-system <DRIVER_POD> -c rbln-driver-container -- rbln-smi

Driver Installation Status¶

The RBLNClusterPolicy instance may report notReady while a driver is being replaced. If it does not return to ready, check the nodes named in the Ready condition message and inspect their driver initContainer logs.

1

kubectl get rblnclusterpolicy <POLICY_NAME> -o jsonpath='{.status.conditions[?(@.type=="Ready")].message}'

Node State and Annotations¶

1
2

kubectl get nodes -L rebellions.ai/npu-driver-upgrade-state
kubectl describe node <NODE_NAME>

All of the following annotation names have the rebellions.ai/ prefix.

If an annotation message is truncated, check the operator logs for the full details.

Events¶

1
2
3

kubectl get events -A \
  --field-selector involvedObject.kind=Node,involvedObject.name=<NODE_NAME> \
  --sort-by=.lastTimestamp

Inspecting state¶

Check which nodes are excluded and the progress of each node:

1
2
3
4

kubectl get nodes \
  -L rebellions.ai/npu.present \
  -L rebellions.ai/npu-driver-upgrade-state \
  -L rebellions.ai/npu-driver-upgrade.skip

Skip Reasons¶

Eviction targets pods requesting rebellions.ai/* resources or holding a container-mode NPU DRA claim. The default container DeviceClass is npu.rebellions.ai; custom classes bridged to rebellions.ai/* also qualify.

Check the node's rebellions.ai/npu-driver-upgrade-skip-reason annotation for the cause. A PodDisruptionBudget blocking eviction may be reported as an eviction timeout.

Driver Pod Timeout¶

Image pull or container startup failures raise a DriverUpgradePodStuck warning. The node becomes upgrade-failed when podRestartTimeoutSeconds expires (default: 1800 seconds) or an unready container has restarted more than ten times. See timeout settings.

Retry Paths¶

After fixing the cause, request a retry and check the state:

1
2

kubectl annotate node <NODE_NAME> rebellions.ai/npu-driver-upgrade-requested=true --overwrite
kubectl get node <NODE_NAME> -L rebellions.ai/npu-driver-upgrade-state

If automatic upgrades are disabled, re-enable them before requesting a retry. Re-enabling automatic upgrades alone does not guarantee a retry for failed nodes.

In addition to an explicit retry request, upgrades resume under the following conditions.

A node that failed validation does not recover on its own when its pod becomes Ready. Request a retry explicitly or change the driver version or driver container configuration.