Running Multiple Driver Versions¶

One cluster can run different driver versions for different NPU families or node groups. This guide covers the container driver mode described in NPU Driver Upgrades.

Driver Settings and Node Selection¶

In Helm values, use the top-level driver block for shared image and resource settings, and driver.instances for settings specific to each node group. Each instance overrides the values it specifies and uses the shared settings for omitted values. driver.upgradePolicy applies to all instances.

For multiple driver versions, set driver.deployDefault: false and specify the target nodes and version in each instance. When deployDefault is true, nodes not selected by any instance may also receive a driver with the shared settings. Instances still inherit the shared settings when this option is false.

Each instance selects its target nodes with nodeSelector. This field is not inherited, so specify it on every instance. If several instances match a node, the most specific instance whose selector includes all the other matching conditions takes precedence. For example, an instance that selects an NPU family and env: canary takes precedence over one that selects only the same family.

---
config:
  htmlLabels: false
  flowchart:
    curve: linear
---
flowchart TD
    A["Find RBLNDriver selectors<br/>matching the node"] --> B{"Unique most specific<br/>selector?"}
    B -->|"Yes"| C["Assign the owner<br/>Set the owner label"]
    C --> D["Schedule the driver Pod<br/>using the owner label"]
    B -->|"No"| E["Report selector conflict<br/>Keep the current driver"]

If several instances match but no single instance takes precedence, the operator reports the reason ConflictingNodeSelector. Adjust the conflicting selectors. The labels rebellions.ai/npu.driver.owner and rebellions.ai/npu.deploy.driver are managed by the operator and cannot be used in nodeSelector.

Check that the instances' nodeSelector settings cover every node that needs a driver. When changing deployDefault from true to false on an existing installation, include the nodes previously handled by the default driver.

Configuration Examples¶

The following examples show versions assigned directly by NPU family and a shared version running alongside a canary version within one family. Adjust the versions and node selectors for your environment.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

driver:
  enabled: true
  # Deploy drivers only to nodes selected by the instances below.
  deployDefault: false
  instances:
    # Specify the version and target nodes directly for ATOM™.
    atom:
      image:
        tag: "3.4.1"
      nodeSelector:
        rebellions.ai/npu.family: atom
    # Assign a separate driver version to Rebel100™ nodes.
    rebel100:
      image:
        tag: "3.5.1"
      nodeSelector:
        rebellions.ai/npu.family: rebel100

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

driver:
  enabled: true
  # Use shared settings, with target nodes specified by each instance.
  deployDefault: false
  image:
    tag: "3.5.1"
  instances:
    # Omitting image.tag uses the shared version 3.5.1.
    rebel100:
      nodeSelector:
        rebellions.ai/npu.family: rebel100
    rebel100-canary:
      image:
        # Enter the driver version to use for this instance.
        tag: "<DRIVER_VERSION>"
      nodeSelector:
        # Takes precedence over rebel100 on nodes matching both conditions.
        rebellions.ai/npu.family: rebel100
        env: canary

Verifying Instance Routing¶

The node label rebellions.ai/npu.driver.owner records the selected instance's resource name.

1
2
3
4
5
6
7

kubectl get rblndrivers

# Which resource owns each node
kubectl get nodes -L rebellions.ai/npu.driver.owner

# Nodes eligible for a driver but currently unowned, which means a selector is wrong
kubectl get nodes -l 'rebellions.ai/npu.deploy.driver=true,!rebellions.ai/npu.driver.owner'