NPU Driver Upgrades¶
Upgrade NPU drivers on individual nodes in environments using container drivers (driver.enabled: true). Manual and automatic upgrades involve the same tasks but start differently and uncordon the node at different points.
For a step-by-step walkthrough from changing the version to confirming completion, see Upgrade NPU drivers automatically.
Upgrade Lifecycle¶
An upgrade proceeds in the following order on each target node.
- Cordon: Block new pod scheduling on the target node. Pods already running are not terminated at this stage.
- NPU pod eviction: Terminate NPU pods using the driver. If eviction is blocked, driver replacement does not start.
- Driver replacement: Remove the previous driver, then install and load the specified version. NPUs on the node are unavailable during replacement.
- Completion check: Confirm that the new driver pod is Ready and that the loaded driver version matches the specified version.
- Uncordon: Allow new pods to be scheduled on the node again. Manual mode uncordons the node before installing the new driver.
Manual and Automatic Modes¶
Choose the mode with driver.upgradePolicy.autoUpgrade. The default is false.
| Aspect | Manual Upgrade | Automatic Upgrade |
|---|---|---|
| Start | After applying the target version, the administrator deletes the driver pod on the target node. | The operator starts the upgrade when the target version is applied. |
| Cordon and eviction | Run after the driver pod is recreated. | Run before the driver pod is replaced. |
| Job completion wait | The administrator checks running jobs and decides when to start. | Wait for pods selected by waitForCompletion before eviction. |
| Driver replacement and node return | Remove the previous driver → uncordon → install the new driver → check completion | Replace the driver → validate → uncordon |
| Multiple nodes | The administrator chooses the nodes and their upgrade order. | maxParallelUpgrades limits the number of nodes upgraded at once. |
In manual mode, check the new driver pod and driver version after the upgrade completes, then resume workloads.