Skip to content

NPU Driver Upgrade Workflow

This document covers the container driver mode. If you install the kernel driver directly on hosts in host driver mode, see NPU Driver Installation.

This document covers:

  • Defining the RBLNDriver: the CRD structure, what the Driver Manager installs, and how the driver image is selected per node
  • Architecture and flow: the two components that coordinate upgrades (operator and driver-manager) and what happens on each node when a driver pod starts
  • Upgrade modes and policy: rollout managed by the operator and manual rollout, and how to configure upgradePolicy (cordon, drain, reboot, etc.)
  • Running multiple driver versions: pinning a group of nodes to their own driver version with driver.instances
  • Operational options: how to exclude specific nodes from upgrades

Defining the RBLNDriver

The operator defines the RBLNDriver CRD to manage NPU driver installation. When you create an RBLNDriver custom resource with the desired driver version, the Driver Manager installs and maintains that version across the cluster.

RBLNDriver Sample

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
apiVersion: rebellions.ai/v1alpha1
kind: RBLNDriver
metadata:
  labels:
    app.kubernetes.io/name: rbln-driver
  name: rbln-driver
spec:
  registry: repo.rebellions.ai
  image: rebellions/rbln-driver
  version: "3.2.2"
  imagePullPolicy: IfNotPresent
  imagePullSecrets:
    - drivercred
  resources:
    requests:
      cpu: 250m
      memory: 64Mi
    limits:
      cpu: 500m
      memory: 128Mi
  manager:
    registry: docker.io
    image: rebellions/rbln-k8s-driver-manager
    version: v0.2.2
    imagePullPolicy: IfNotPresent
  smd:
    registry: repo.rebellions.ai
    image: rebellions/rbln-smd

Leave the NPU family out of spec.image. The operator adds it per node pool, as described in Driver Image Selection.

What the Driver Manager Installs

When an RBLNDriver resource is applied, the Driver Manager installs:

  • Kernel driver
  • UMD libraries
  • Tools such as rbln-smi

Each RBLNDriver also gets one RSMD DaemonSet, named <CR_NAME>-smd. It carries no version field of its own: the rbln-smd image tag follows spec.version, keeping the node daemon aligned with the driver during upgrades. spec.smd only sets the registry and repository. Its pods wait for the driver to finish installing on the node, and they are replaced when the node's driver pod is replaced. Readiness is reported separately under .status.smd.

Driver Manager v0.2.2 or later required

Driver Manager must be v0.2.2 or later to update the rebellions.ai/npu.deploy.rbln-smd node label that controls the RSMD DaemonSet. The chart pins a compatible version in driver.manager.image.tag; keep it unless you have a specific reason to override it.

Driver Image Selection

A kernel driver build supports one NPU product family and one OS/kernel combination. To run several NPU products in one cluster, the operator groups NPU nodes into node pools by product family, OS, and kernel and renders one driver DaemonSet per pool. The operator uses these labels to determine each node's pool:

Label Applied by Example value
rebellions.ai/npu.family The operator's own NodeFeatureRule named rbln-npu-family atom
feature.node.kubernetes.io/system-os_release.ID Node Feature Discovery ubuntu
feature.node.kubernetes.io/system-os_release.VERSION_ID Node Feature Discovery 22.04
feature.node.kubernetes.io/kernel-version.full Node Feature Discovery 6.8.0-90-generic

The pool name is <family>-<os><version>-<kernel>, and the DaemonSet is named <CR_NAME>-<POOL_NAME>. In the image path, the family is inserted in front of the last segment of spec.image:

1
<registry>/<org>/<family>/<name>:<version>-<kernel>-<os>

An ATOM node running Ubuntu 22.04 with kernel 6.8.0-90-generic therefore joins the pool atom-ubuntu22.04-6.8.0-90-generic and pulls:

1
repo.rebellions.ai/rebellions/atom/rbln-driver:3.2.2-6.8.0-90-generic-ubuntu22.04

A node with no valid rebellions.ai/npu.family label is not assigned to a pool. The operator reports it as DriverFamilyLabelMissing and keeps the existing DaemonSets in place, so a missing label does not uninstall a running driver.

Registry check before rollout

Before rendering a pool's DaemonSet, the operator confirms that the composed image exists in its registry. Only a 404 fails the pool; the custom resource then reports Ready=False with reason DriverImageNotFound and names the exact path that is missing, instead of leaving pods in ImagePullBackOff. The check repeats about every five minutes, so publishing the image resolves the condition without further action.

For every other outcome — an authentication failure, a timeout, or a TLS or DNS error — the operator logs a warning and lets the rollout proceed. Kubelet then determines whether the image pull succeeds.

Set operator.driverImageCheck=false to skip the check entirely. Do this when the operator cannot reach the registry that nodes pull from, such as a disconnected or mirror-only cluster, or when the pull is authenticated by node-level credentials the operator cannot see, such as a ServiceAccount pull secret or a cloud instance identity.

Architecture and Flow

Architecture Overview

NPU driver upgrades are coordinated by two components:

Component Role
rbln-npu-operator Cluster level orchestration and upgrade policy enforcement
rbln-k8s-driver-manager Driver lifecycle reconciliation on each node

Depending on configuration, upgrades operate in one of the following modes:

Mode Setting Description
Rollout managed by the operator autoUpgrade: true Operator orchestrates the rollout across nodes
Manual rollout autoUpgrade: false Administrator triggers upgrades explicitly

Driver upgrades are handled by two layers with distinct responsibilities.

1. rbln-npu-operator (Cluster Orchestration)

The operator manages upgrade orchestration across the cluster.

Responsibilities include:

  • Detecting nodes that require driver upgrades
  • Enforcing upgrade policy (upgradePolicy)
  • Controlling rollout parallelism (maxParallelUpgrades)
  • Coordinating node maintenance actions such as:
    • cordon
    • drain
    • reboot
  • Managing rollout progression across nodes

The operator does not directly manage driver state on nodes. Instead, it triggers driver pod restarts, which start local reconciliation on the node.

2. rbln-k8s-driver-manager (Node Driver Reconciliation)

rbln-k8s-driver-manager runs within the driver DaemonSet and reconciles the driver state on each node.

Responsibilities include:

  • Detecting the current driver state on the node
  • Temporarily pausing related components during driver upgrades
  • Performing driver uninstall/install when necessary
  • Restoring node labels so workloads can resume

Reconciliation runs whenever a driver pod starts on a node.

Driver Reconciliation Flow

When a driver pod starts on a node, the initContainer runs the reconcile-driver-state logic implemented by rbln-k8s-driver-manager.

Step Action
1. Read node labels Read rebellions.ai/npu.deploy.* labels that determine which Rebellions components run on the node
2. Pause related Rebellions components Replace labels with paused-for-driver-upgrade so DaemonSets stop and existing pods terminate
3. Wait for pods to terminate Wait until relevant Rebellions component pods have exited
4. Reconcile driver state If the driver image digest matches the desired state, skip uninstalling the existing driver. Otherwise, unload kernel modules, remove old artifacts, and install the new driver.
5. Restore node labels Restore original labels so Rebellions components can be scheduled again

As a result, all node components restart with the upgraded driver.

Cleanup of stale driver DaemonSets

A pool is identified by the NPU family, OS, and kernel of its nodes, so a node kernel change — for example after apt upgrade and a reboot — moves the node into a different pool and creates a new driver DaemonSet. The operator automatically deletes driver DaemonSets whose node selector no longer matches any node, preventing stale driver pods from accumulating after kernel upgrades.

If any pool fails to render, the operator pauses stale-DaemonSet cleanup and the creation of new pool DaemonSets. A retained DaemonSet could otherwise schedule pods on another pool's nodes, resulting in two driver installers on one host.

Driver readiness signaling

The operator determines driver readiness by checking a marker file that the driver container writes after the driver installation completes. Readiness probes do not mark the pod ready while a firmware update or module reload is in progress.

1
2
3
4
5
6
$ kubectl get rblndriver rbln-driver -o jsonpath='{.status.state}'
ready

$ kubectl get rblndriver rbln-driver \
    -o jsonpath='{range .status.nodePools[*]}{.name}{"\t"}{.state}{"\t"}{.ready}/{.desired}{"\n"}{end}'
rbln-driver-atom-ubuntu22.04-6.8.0-90-generic        ready   1/1

.status.nodePools counts driver pods only. The node daemon is reported separately, and the custom resource stays Ready=False with reason SmdNotReady until .status.smd reports RSMD as ready:

1
2
$ kubectl get rblndriver rbln-driver -o jsonpath='{.status.smd}'
{"desired":1,"ready":1,"state":"ready"}

Upgrade Modes and Policy

AutoUpgrade Mode (autoUpgrade: true)

When AutoUpgrade is enabled, the operator performs a rollout across nodes according to policy.

Upgrade behavior is controlled through upgradePolicy.

Upgrade Flow

For each node selected by the operator, the following steps run:

Step Action
1 The node is cordoned to prevent new workloads from being scheduled.
2 Existing NPU workloads are handled according to policy: waitForCompletion, npuPodDeletion, drain
3 Driver pod restart triggers local reconciliation on the node
4 Node reboot is performed if configured
5 The node is validated
6 Node is uncordoned

The operator then proceeds to the next batch of nodes according to maxParallelUpgrades.

The operator records each stage as Kubernetes events on the node (DriverUpgradeStarted → NodeDrained → DriverUpgradeCompleted). See Operator Observability for details.

Node Upgrade Selection

Nodes that require an upgrade are detected when:

  • driver DaemonSet revision changes
  • an explicit upgrade request is issued

The operator selects nodes for upgrade based on maxParallelUpgrades:

Value Behavior
1 Upgrade one node at a time (sequential)
0 Unlimited parallel upgrades

Reboot Workflow

When a GRUB setting has changed, an optional reboot restarts the node so the change takes effect. If you also set drain.enable: true and npuPodDeletion, the operator cordons and drains the worker node before the reboot. Enable these settings together:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
driver:
  upgradePolicy:
    npuPodDeletion:
      force: true
      timeoutSeconds: 300
    drain:
      enable: true
      force: true
      deleteEmptyDirData: false
      podSelector: ""
      timeoutSeconds: 300
    reboot:
      enable: true
      rebootTimeoutSeconds: 0
      image:
        registry: docker.io
        image: rebellions/rbln-node-reboot
        version: v1.0.1

drain.enable and reboot.enable are both false by default in the chart.

When enabled, the workflow is:

Step Action
1 Operator triggers a reboot through the reboot helper pod
2 Node temporarily becomes NotReady
3 Node returns to Ready after reboot validation

Manual Mode (autoUpgrade: false)

When AutoUpgrade is disabled, the driver DaemonSet uses the OnDelete strategy.

Driver pods are not automatically restarted when the DaemonSet template changes.

Instead, upgrades occur only when an administrator performs explicit actions.

Manual Upgrade Procedure

  1. Administrator selects a node
  2. Node maintenance actions are performed (typically cordon and drain)
  3. Administrator deletes the driver pod: 
    1
    $ kubectl delete pod <driver-pod>
  4. Kubernetes DaemonSet controller creates a new driver pod
  5. The initContainer triggers the driver reconciliation flow

During reconciliation, rbln-k8s-driver-manager:

  • pauses related Rebellions components
  • updates the node driver state
  • restores node labels once complete

After reconciliation, Rebellions components are automatically rescheduled.

Helm Configuration

If driver.enabled: false at the chart level, the RBLNDriver resource is never created and the upgrade behavior described on this page does not apply. The remainder of this section assumes driver.enabled: true.

All upgrade actions must be enabled explicitly. The chart ships with autoUpgrade, drain.enable, and reboot.enable set to false by default.

To turn on rollout managed by the operator, set autoUpgrade: true and enable the subblocks (drain.enable, reboot.enable, etc.) for any maintenance actions you want the operator to perform.

Example configuration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
driver:
  upgradePolicy:
    autoUpgrade: true
    maxParallelUpgrades: 1
    waitForCompletion:
      timeoutSeconds: 0
      podSelector: ""
    npuPodDeletion:
      force: false
      timeoutSeconds: 300
    drain:
      enable: true
      force: false
      deleteEmptyDirData: false
      podSelector: ""
      timeoutSeconds: 300
    reboot:
      enable: true
      rebootTimeoutSeconds: 0
      image:
        registry: docker.io
        image: rebellions/rbln-node-reboot
        version: v1.0.1

Upgrade Policy Reference

Setting Description
autoUpgrade Driver upgrades managed by the operator. false (default) = operator does not orchestrate rollout; true = operator performs rollout
maxParallelUpgrades Maximum number of nodes upgraded concurrently. 1 = sequential (default), 0 = unlimited
waitForCompletion.timeoutSeconds Maximum seconds to wait for selected pods to complete before removal. 0 (default) = wait indefinitely
waitForCompletion.podSelector Label selector for pods to wait on. Empty string (default) = skip the wait step entirely
npuPodDeletion.force false (default) = conservative removal (blocks on pods lacking a controller); true = forced removal
npuPodDeletion.timeoutSeconds Max seconds before forcibly deleting remaining pods. 0 = wait indefinitely (default 300)
drain.enable false (default) = drain skipped; true = operator drains the node before pod restart
drain.force false (default) = drain fails on blocking pods; true = drain proceeds even when blocking pods exist
drain.deleteEmptyDirData false (default) = pods using emptyDir storage block the drain; true = those pods are removed (data is lost)
drain.podSelector Label selector restricting drain to matching pods. Empty string (default) = drain all pods on the node
drain.timeoutSeconds Max seconds for drain to complete. 0 = wait indefinitely (default 300)
reboot.enable false (default) = no reboot; true = node is rebooted as part of the upgrade
reboot.rebootTimeoutSeconds Max seconds to wait for the rebooted node to return Ready. Only relevant when reboot.enable: true. 0 (default) = no timeout

Multiple Driver Versions

One cluster can run more than one driver version at a time: a canary version on a labeled subset of nodes, or a different version per NPU family. Each entry under driver.instances renders its own RBLNDriver custom resource named rbln-driver-<key>.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
driver:
  enabled: true
  image:
    tag: "3.2.2"            # fallback for every node not claimed below
  instances:
    atom-canary:
      image:
        tag: "3.3.0-rc1"
      nodeSelector:
        rebellions.ai/npu.family: atom
        env: canary

An instance key must be a lowercase DNS-1123 label of at most 40 characters, because the resulting resource name is written onto nodes as a label value.

Node Routing

The driver DaemonSet does not use your nodeSelector directly. On every reconcile, the operator recomputes which resource owns each NPU node and writes the selected resource's name to the node label rebellions.ai/npu.driver.owner. DaemonSets then select on that label. This guarantees one driver per node at the scheduling level, regardless of selector overlap.

The most specific matching selector owns the node. Its key-value set is a strict superset of every other matching selector's key-value set. Because the top-level driver.nodeSelector is empty by default, the default resource matches every node and acts as the fallback for anything a more specific instance does not claim:

1
{} ⊂ {npu.family: atom} ⊂ {npu.family: atom, env: canary}

Two selectors that are identical, or where neither contains the other, tie on any node that both match. A tied node keeps whichever driver it already has, every other node routes normally, and both resources report Ready=False with reason ConflictingNodeSelector listing a sample of the affected nodes. Editing either selector clears the tie on the next reconcile.

rebellions.ai/npu.driver.owner and rebellions.ai/npu.deploy.driver are reserved. A resource whose nodeSelector sets either one is rejected with InvalidSpec and dropped from routing, without affecting any other resource.

Value Inheritance

Fields you leave unset on an instance are inherited from the top-level driver block:

Field kind Examples Behavior
Maps image, resources, annotations, manager, smd Merged key by key. To drop an inherited key, override the whole map on the instance.
Lists tolerations, env, imagePullSecrets Replaced as a whole, never merged. [] clears the inherited value.
Scalars priorityClassName Overridden. "" clears the inherited value.
nodeSelector — Never inherited. Required and non-empty on every instance; only the default resource may have an empty selector.

driver.upgradePolicy is rendered once onto the RBLNClusterPolicy and applies to every instance alike. Per-instance upgrade cadence is not supported.

deployDefault: false removes the fallback driver

Setting driver.deployDefault=false removes the default resource, so every node not claimed by a more specific instance loses its driver. List those nodes before you change it: 

1
$ kubectl get nodes -l 'rebellions.ai/npu.driver.owner=rbln-driver'

Verifying Instance Routing

1
2
3
4
5
6
7
$ kubectl get rblndrivers

# Which resource owns each node
$ kubectl get nodes -L rebellions.ai/npu.driver.owner

# Nodes eligible for a driver but currently unowned, which means a selector is wrong
$ kubectl get nodes -l 'rebellions.ai/npu.deploy.driver=true,!rebellions.ai/npu.driver.owner'

Mistyped instance keys inherit silently

A mistyped instance key inherits every field from the default, including the version, and renders without complaint.

Confirm what each instance will actually run before applying:

1
2
3
$ helm template rbln-npu-operator oci://docker.io/rebellions/rbln-npu-operator-chart \
    --version ${CHART_VERSION} -f my-values.yaml \
  | yq 'select(.kind == "RBLNDriver") | .metadata.name + " " + .spec.version'

helm uninstall removes the custom resources but leaves the rebellions.ai/npu.driver.owner labels on nodes. They are harmless on reinstall, since the operator overwrites them on the first reconcile, but clean them up when decommissioning:

1
$ kubectl label nodes --all rebellions.ai/npu.driver.owner-

Operational Options

Skipping Driver Upgrades

To exclude a node from driver upgrades, set the following label:

1
$ kubectl label node <node-name> rebellions.ai/npu-driver-upgrade.skip=true

To re-enable upgrades, remove the label:

1
$ kubectl label node <node-name> rebellions.ai/npu-driver-upgrade.skip-

The operator re-checks this label on every upgrade attempt, so it applies to both autoUpgrade: true and manual rollouts.

Don't confuse with rebellions.ai/npu.deploy.skip

Label Effect
rebellions.ai/npu-driver-upgrade.skip=true Pauses driver upgrades only. The current driver keeps running, and other NPU components are not affected.
rebellions.ai/npu.deploy.skip=true Removes all RBLN NPU components from the node, including the driver itself. NPU workloads on the node will stop working.

Use npu-driver-upgrade.skip to hold back upgrades. Use npu.deploy.skip only when you want to exclude the node from NPU workloads entirely. See Workload Labeling by Node for the full npu.deploy.skip workflow.

Inspecting state

To see which nodes are excluded and where the remaining nodes are in the upgrade state machine:

1
2
3
4
$ kubectl get nodes \
  -L rebellions.ai/npu.present \
  -L rebellions.ai/npu-driver-upgrade-state \
  -L rebellions.ai/npu-driver-upgrade.skip

Verifying the Running Driver

To confirm the driver pod has loaded the kernel module and the expected KMD version is in use, run rbln-smi inside the driver container on the target node.

First, find the driver pod for the node. Driver pods follow the rbln-driver-<family>-<os>-<kernel>-<hash> name pattern (see Driver Image Selection):

1
$ kubectl get pods -n rbln-system -o wide | grep rbln-driver

Then exec into the rbln-driver-container and invoke rbln-smi:

1
$ kubectl exec -n rbln-system <driver-pod> -c rbln-driver-container -- rbln-smi

The header reports the running KMD version, and the device table lists the NPUs the driver has bound on this node:

1
2
3
4
5
6
7
8
+-------------------------------------------------------------------------------------------------+
|                                Device Information KMD ver: 3.0.0                                |
+-----+-----------+---------+---------------+------+---------+------+---------------------+-------+
| NPU |    Name   | Device  |   PCI BUS ID  | Temp |  Power  | Perf |  Memory(used/total) |  Util |
+=====+===========+=========+===============+======+=========+======+=====================+=======+
| 0   | RBLN-CA25 | rbln0   |  0000:05:00.0 |  37C |  58.7W  | P14  |    0.0B / 15.7GiB   |   0.0 |
| 1   |           | rbln1   |  0000:06:00.0 |  39C |         | P14  |    0.0B / 15.7GiB   |   0.0 |
+-----+-----------+---------+---------------+------+---------+------+---------------------+-------+

After an upgrade, the KMD ver line should match spec.version in your RBLNDriver.