Configuring Kernel Parameters on OpenShift¶

Manage NPU node boot parameters through MachineConfig. The settings below were validated on the listed hardware.

Target hardware¶

Kernel parameters¶

Before applying¶

You need cluster-admin access and oc. Identify the hardware, inspect the current configuration, and save each target node's /proc/cmdline.

1
2
3
4
5
6
7

oc get nodes
oc get mcp
oc debug node/<NODE_NAME> -- chroot /host dmidecode -s system-manufacturer
oc debug node/<NODE_NAME> -- chroot /host dmidecode -s system-product-name
oc debug node/<NODE_NAME> -- chroot /host lscpu
oc debug node/<NODE_NAME> -- chroot /host lspci -d 1eff: -n
oc debug node/<NODE_NAME> -- chroot /host cat /proc/cmdline

The PCI device ID identifies the card. Select the profile by NPU model even when the server model is the same.

Selecting the target nodes¶

flowchart LR
    MC["MachineConfig"] -->|"machineConfigSelector"| MCP["MachineConfigPool"]
    MCP -->|"nodeSelector"| N["Target nodes"]

Use a dedicated MCP for a subset of workers. If the cluster has nodes of both product lines, keep them in separate pools because their parameters differ.

Create custom MachineConfigPool¶

Save this as npu-mcp.yaml. Replace every __MCP_NAME__ with the selected profile name. Retain the worker configuration inheritance and ensure that another custom MCP does not also select these nodes.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
  name: __MCP_NAME__
spec:
  machineConfigSelector:
    matchExpressions:
      - key: machineconfiguration.openshift.io/role
        operator: In
        values:
          - worker
          - __MCP_NAME__
  nodeSelector:
    matchLabels:
      node-role.kubernetes.io/__MCP_NAME__: ""

1
2
3

oc apply -f npu-mcp.yaml
oc label node <NODE_NAME> node-role.kubernetes.io/__MCP_NAME__=""
oc get mcp __MCP_NAME__

Confirm that MACHINECOUNT matches the target node count and that the pool is healthy. If using automatic NFD identification, use the profile selector in the next section instead of the role label above.

(Optional) Automatic Identification via NFD NodeFeatureRule¶

With NFD v0.16+, the system.dmiid.sys_vendor, cpu.model.vendor_id and pci.device features can be combined in a NodeFeatureRule that assigns the profile label automatically, so a node joining the cluster later lands in the right MCP without manual labeling.

Create a NodeFeatureRule¶

Each rule lists several features and matchFeatures requires all of them to match. The server vendor and the CPU vendor cannot tell the two product lines apart on the same server, so the rules also match the device id under pci.device. If the cluster has nodes of both product lines, put the rules from both tabs in one NodeFeatureRule.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

apiVersion: nfd.k8s-sigs.io/v1alpha1
kind: NodeFeatureRule
metadata:
  name: rbln-hardware-detection
spec:
  rules:
    - name: "amd-smci-cr13"
      labels:
        rebellions.io/profile: "amd-smci-cr13"
      matchFeatures:
        - feature: system.dmiid
          matchExpressions:
            sys_vendor: {op: In, value: ["Supermicro"]}
        - feature: cpu.model
          matchExpressions:
            vendor_id: {op: In, value: ["AMD"]}
        - feature: pci.device
          matchExpressions:
            vendor: {op: In, value: ["1eff"]}
            device: {op: In, value: ["2130"]}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

apiVersion: nfd.k8s-sigs.io/v1alpha1
kind: NodeFeatureRule
metadata:
  name: rbln-hardware-detection
spec:
  rules:
    - name: "amd-smci-ca25"
      labels:
        rebellions.io/profile: "amd-smci-ca25"
      matchFeatures:
        - feature: system.dmiid
          matchExpressions:
            sys_vendor: {op: In, value: ["Supermicro"]}
        - feature: cpu.model
          matchExpressions:
            vendor_id: {op: In, value: ["AMD"]}
        - feature: pci.device
          matchExpressions:
            vendor: {op: In, value: ["1eff"]}
            device: {op: NotIn, value: ["2130"]}
    - name: "amd-gigabyte-ca25"
      labels:
        rebellions.io/profile: "amd-gigabyte-ca25"
      matchFeatures:
        - feature: system.dmiid
          matchExpressions:
            sys_vendor: {op: In, value: ["Giga Computing"]}
        - feature: cpu.model
          matchExpressions:
            vendor_id: {op: In, value: ["AMD"]}
        - feature: pci.device
          matchExpressions:
            vendor: {op: In, value: ["1eff"]}
            device: {op: NotIn, value: ["2130"]}

1

oc apply -f <NFR_YAML_FILE>

Verify Labels¶

After applying, confirm that the profile label is assigned to the target node:

1

oc get node <NODE_NAME> -o jsonpath='{.metadata.labels}' | python3 -m json.tool | grep rebellions.io/profile

1

"rebellions.io/profile": "amd-smci-cr13",

1

"rebellions.io/profile": "amd-smci-ca25",

Before creating the rule, verify the CPU vendor using the built-in NFD label feature.node.kubernetes.io/cpu-model.vendor_id=AMD, which NFD assigns without a NodeFeatureRule.

Use NFD Labels as MCP nodeSelector¶

Once NFD has assigned the profile label, it can be used directly in the MCP nodeSelector without manual labeling. Update the nodeSelector in the MCP YAML in Create Custom MachineConfigPool as follows.

1
2
3

  nodeSelector:
    matchLabels:
      rebellions.io/profile: __MCP_NAME__

Selecting a MachineConfig¶

Save one manifest matching the hardware profile as npu-kargs.yaml. Replace __MCP_NAME__ with the target MCP name. Defining the same parameter in multiple configurations for a pool duplicates it on the kernel command line.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  name: 99-grub-kargs-amd-smci-cr13
  labels:
    machineconfiguration.openshift.io/role: __MCP_NAME__
    rebellions.io/component: grub-kargs
    rebellions.io/profile: amd-smci-cr13
    rebellions.io/cpu-vendor: amd
    rebellions.io/server-vendor: supermicro
    rebellions.io/accelerator: cr13
spec:
  kernelArguments:
    - transparent_hugepage=madvise
    - pcie_aspm=force
    - pci=pcie_bus_perf
    - pci=bfsort

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  name: 99-grub-kargs-amd-smci-ca25
  labels:
    machineconfiguration.openshift.io/role: __MCP_NAME__
    rebellions.io/component: grub-kargs
    rebellions.io/profile: amd-smci-ca25
    rebellions.io/cpu-vendor: amd
    rebellions.io/server-vendor: supermicro
    rebellions.io/accelerator: ca25
spec:
  kernelArguments:
    - transparent_hugepage=madvise
    - pcie_aspm=force
    - pci=pcie_bus_perf
    - pci=bfsort
    - iommu.strict=1

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  name: 99-grub-kargs-amd-gigabyte-ca25
  labels:
    machineconfiguration.openshift.io/role: __MCP_NAME__
    rebellions.io/component: grub-kargs
    rebellions.io/profile: amd-gigabyte-ca25
    rebellions.io/cpu-vendor: amd
    rebellions.io/server-vendor: gigabyte
    rebellions.io/accelerator: ca25
spec:
  kernelArguments:
    - transparent_hugepage=madvise
    - pcie_aspm=force
    - pci=pcie_bus_perf
    - pci=bfsort
    - iommu.strict=1

Applying and verifying¶

1
2

oc apply -f npu-kargs.yaml
oc get mcp __MCP_NAME__ -w

$ oc get mcp __MCP_NAME__ \
>   -o 'custom-columns=NAME:.metadata.name,UPDATED:.status.conditions[?(@.type=="Updated")].status,UPDATING:.status.conditions[?(@.type=="Updating")].status,DEGRADED:.status.conditions[?(@.type=="Degraded")].status'
NAME            UPDATED   UPDATING   DEGRADED
amd-smci-ca25   True      False      False

When the pool reports UPDATED=True, UPDATING=False, and DEGRADED=False, inspect the target nodes.

1

oc debug node/<NODE_NAME> -- chroot /host cat /proc/cmdline

$ oc debug node/<NODE_NAME> -- chroot /host cat /sys/kernel/mm/transparent_hugepage/enabled
...
always [madvise] never

All parameters for the selected profile must be present, and the THP output must select [madvise].

Rolling back¶

Delete the MachineConfig added by this page and wait for the MCP rollout to finish.

1
2

oc delete -f npu-kargs.yaml
oc get mcp __MCP_NAME__ -w

Once the pool is healthy, compare /proc/cmdline with the saved baseline. A parameter remains if another MachineConfig still defines it.

Remove the MCP only when no other MachineConfig depends on it. Order: remove node labels → wait for worker-pool return and configuration completion → delete the MCP. For NFD selection, also check the profile-label rules.

1
2

oc label node <NODE_NAME> node-role.kubernetes.io/__MCP_NAME__-
oc get mcp worker __MCP_NAME__ -w

After the nodes have moved and finished applying their configuration, run:

1

oc delete mcp __MCP_NAME__

Troubleshooting an incomplete rollout¶

1
2
3

oc describe mcp __MCP_NAME__
oc describe node <NODE_NAME>
oc get pods -n openshift-machine-config-operator -o wide

Read the MCP status messages and the affected node's Machine Config Daemon logs. Check node selectors, overlapping custom pools, workloads blocking drain, and missing MachineConfigs. Manually changing a node's currentConfig or desiredConfig annotation can make it disagree with the applied configuration.

See Red Hat's MachineConfig documentation for MCO configuration and recovery.