Operator Observability¶

This page covers the three ways to monitor the RBLN NPU Operator: operator metrics, Kubernetes events paired with resource conditions, and the operator's log stream. For telemetry of the NPU devices themselves (utilization, temperature, power, and so on), see the Metrics Exporter.

Operator Metrics¶

The operator exposes its own state metrics in Prometheus format: policy and driver reconcile outcomes, NPU node counts, and driver upgrade progress. The Helm value operator.metrics.enabled controls the endpoint (default true).

Metrics Information¶

Metric	Type	Labels	Description
`rbln_operator_clusterpolicy_reconcile_status`	Gauge	—	Last `RBLNClusterPolicy` reconcile outcome (0=success, 1=notReady, 2=unavailable)
`rbln_operator_driver_reconcile_status`	Gauge	`name`	Last reconcile outcome per `RBLNDriver` CR
`rbln_operator_reconcile_total`	Counter	`controller`	Reconciliations executed
`rbln_operator_reconcile_failed_total`	Counter	`controller`	Reconciliations that did not reach ready
`rbln_operator_npu_nodes`	Gauge	`workload`	NPU-eligible nodes per workload type
`rbln_operator_workload_coverage_state`	Gauge	`workload`	Aggregate workload coverage state (0=empty ~ 3=uncovered)
`rbln_operator_driver_pool_ready_ratio`	Gauge	`driver`, `pool`	Ready/desired ratio per driver pool
`rbln_operator_driver_owned_nodes`	Gauge	`driver`	Nodes routed to each `RBLNDriver`. The series is initialized to zero, so a selector that matches no nodes reports `0` instead of causing the series to disappear
`rbln_operator_driver_uncovered_nodes`	Gauge	—	Driver-eligible NPU nodes with no owning `RBLNDriver`
`rbln_operator_driver_selector_conflict_nodes`	Gauge	`driver`	Nodes where this driver's selector ties with another `RBLNDriver`
`rbln_operator_driver_upgrade_nodes`	Gauge	`state`	Nodes per driver upgrade state

The operator also exposes the standard controller-runtime metrics (controller_runtime_*, workqueue_*, rest_client_*, go_*) on the same endpoint.

Scraping with Prometheus Operator¶

If you use the Prometheus Operator, enable the ServiceMonitor provided by the chart in your Helm values.

operator:
  metrics:
    serviceMonitor:
      enabled: true

Because the endpoint requires authorization, bind the metrics-reader ClusterRole created by the chart to the Prometheus ServiceAccount to bring the scrape target UP.

$ kubectl create clusterrolebinding rbln-operator-metrics-scraper \
  --clusterrole=<release>-rbln-npu-operator-metrics-reader \
  --serviceaccount=<monitoring-namespace>:<prometheus-serviceaccount>

403 responses

If the scrape target stays at 403 Forbidden even with the ServiceMonitor enabled, check whether the metrics-reader binding above is missing.

Operator Events¶

The operator records state transitions and failures as Kubernetes events, so kubectl describe alone shows what happened and when. Failures on custom resources are also reflected in each resource's Ready condition according to the mapping below. The condition remains the authoritative current state after an event expires.

Reason	Type	Object	When	Paired `Ready` condition
`DriverUpgradeStarted`	Normal	Node	Node selected for upgrade and entering the cordon stage	—
`NodeDrained`	Normal	Node	Node drain succeeded	—
`NodeDrainFailed`	Warning	Node	Cordon or drain failed	—
`DriverUpgradeCompleted`	Normal	Node	Node upgrade completed	—
`DriverUpgradeFailed`	Warning	Node	Node upgrade failed	—
`DriverOwnerChanged`	Normal	Node	The node's owning `RBLNDriver` was set or changed	—
`DriverNodeUncovered`	Warning	Node	The node lost its owning `RBLNDriver`, because no selector matches it or a selector tie is unresolved	—
`ComponentApplyFailed`	Warning	RBLNClusterPolicy	Failed to apply a manifest of a policy-managed component (Device Plugin, NPU Feature Discovery, and so on)	`False / ComponentApplyFailed`
`DriverInstallFailed`	Warning	RBLNDriver	Failed to apply a manifest of a driver-install component (per-node-pool driver DaemonSet, and so on)	`False / Error`
`InvalidSpec`	Warning	RBLNDriver	`nodeSelector` uses a reserved key, or the resource name exceeds 63 characters	`False / InvalidSpec`
`ConflictingNodeSelector`	Warning	RBLNDriver	`nodeSelector` ties with another `RBLNDriver` on specific nodes	`False / ConflictingNodeSelector`
`DriverImageNotFound`	Warning	RBLNDriver	A pool's composed driver image is missing from its registry. The operator rechecks the image about every five minutes, and the condition clears once the image is published	`False / DriverImageNotFound`
`DriverFamilyLabelMissing`	Warning	RBLNDriver	An owned node has no usable `rebellions.ai/npu.family` label	`False / DriverFamilyLabelMissing`
`AllActiveWorkloadsReady`	Normal	RBLNClusterPolicy	Every policy-managed component became ready, so the policy state changed to `ready`	`True / AllActiveWorkloadsReady`
`DriverReady`	Normal	RBLNDriver	The driver became ready on every node pool, so the driver state changed to `ready`	`True / AllDriverPoolsReady`
`PolicyIgnored`	Normal	RBLNClusterPolicy	Ignored a later-created policy because an active `RBLNClusterPolicy` already exists (only one policy per cluster takes effect)	`False / PolicyIgnored`

The operator emits each event once, when the state actually changes; steady-state reconcile loops do not re-emit it. The four driver-validation warnings (InvalidSpec, ConflictingNodeSelector, DriverImageNotFound, and DriverFamilyLabelMissing) fire once per event reason and resource generation, so a problem that clears and later recurs under an unchanged spec updates the condition without producing a second event. The two node-routing events fire only on the reconcile pass where a node's owner actually changes.

Because both CRDs are cluster-scoped, their events are recorded in the default namespace rather than the operator's namespace.

During a driver auto upgrade, the operator records events on each node in the order DriverUpgradeStarted → NodeDrained → DriverUpgradeCompleted.

Inspecting Events¶

List the upgrade events of a specific node in chronological order.

$ kubectl get events -A \
  --field-selector involvedObject.kind=Node,involvedObject.name=<node> \
  --sort-by=.lastTimestamp

Events for the custom resources appear in the Events section at the bottom of the describe output.

$ kubectl describe rblnclusterpolicy <name>
$ kubectl describe rblndriver <name>

Event retention

Kubernetes keeps events for one hour by default. Read .status.conditions for the current authoritative state, and use the rbln_operator_* metrics for long-term trends and alerting.

$ kubectl logs -n <namespace> deployment/<release>-rbln-npu-operator-controller-manager
$ kubectl get rblnclusterpolicy <name> -o jsonpath='{.status.conditions}'

Missing events

When many distinct failures occur on the same object in a short period, the Kubernetes client's spam filter can drop some of their events; the conditions and metrics still reflect the current state.

Logging¶

The operator writes structured logs to stdout, including records produced by the Kubernetes client libraries. Configure the logs with the following chart values:

operator:
  logging:
    level: info              # error | info | debug | panic, or an integer N for logr V(N)
    encoder: json            # json | console
    timeEncoding: rfc3339nano
    develMode: false

Invalid logging values stop the operator

An invalid level or encoder stops the operator at startup rather than falling back. Verify the value before rolling it out.

To increase the verbosity of the Kubernetes client library records, set a higher level and add a klog -v container argument as well.

The CRD synchronization hook runs before the operator exists, so it cannot inherit these values. Configure it separately with crds.upgrade.logging.level and crds.upgrade.logging.format; unlike the operator, an invalid value there falls back to the default.

Components managed by the operator have their own log settings. See Component Logging.