Operator Observability¶

This page covers the two ways to monitor the RBLN NPU Operator: operator metrics and Kubernetes events. For telemetry of the NPU devices themselves (utilization, temperature, power, and so on), see the Metrics Exporter.

Operator Metrics¶

The operator exposes its own state metrics in Prometheus format: policy and driver reconcile outcomes, NPU node counts, and driver upgrade progress. The Helm value operator.metrics.enabled controls the endpoint (default true).

Metrics Information¶

Metric	Type	Labels	Description
`rbln_operator_clusterpolicy_reconcile_status`	Gauge	—	Last `RBLNClusterPolicy` reconcile outcome (0=success, 1=notReady, 2=unavailable)
`rbln_operator_driver_reconcile_status`	Gauge	`name`	Last reconcile outcome per `RBLNDriver` CR
`rbln_operator_reconcile_total`	Counter	`controller`	Reconciliations executed
`rbln_operator_reconcile_failed_total`	Counter	`controller`	Reconciliations that did not reach ready
`rbln_operator_npu_nodes`	Gauge	`workload`	NPU-eligible nodes per workload type
`rbln_operator_workload_coverage_state`	Gauge	`workload`	Aggregate workload coverage state (0=empty ~ 3=uncovered)
`rbln_operator_driver_pool_ready_ratio`	Gauge	`driver`, `pool`	Ready/desired ratio per driver pool
`rbln_operator_driver_upgrade_nodes`	Gauge	`state`	Nodes per driver upgrade state

The operator also exposes the standard controller-runtime metrics (controller_runtime_*, workqueue_*, rest_client_*, go_*) on the same endpoint.

Scraping with Prometheus Operator¶

If you use the Prometheus Operator, enable the ServiceMonitor provided by the chart in your Helm values.

operator:
  metrics:
    serviceMonitor:
      enabled: true

Because the endpoint requires authorization, bind the metrics-reader ClusterRole created by the chart to the Prometheus ServiceAccount to bring the scrape target UP.

$ kubectl create clusterrolebinding rbln-operator-metrics-scraper \
  --clusterrole=<release>-rbln-npu-operator-metrics-reader \
  --serviceaccount=<monitoring-namespace>:<prometheus-serviceaccount>

403 responses

If the scrape target stays at 403 Forbidden even with the ServiceMonitor enabled, check whether the metrics-reader binding above is missing.

Operator Events¶

The operator records state transitions and failures as Kubernetes events, so kubectl describe alone shows what happened and when.

Reason	Type	Object	When
`DriverUpgradeStarted`	Normal	Node	Node selected for upgrade and entering the cordon stage
`NodeDrained`	Normal	Node	Node drain succeeded
`NodeDrainFailed`	Warning	Node	Cordon or drain failed
`DriverUpgradeCompleted`	Normal	Node	Node upgrade completed
`DriverUpgradeFailed`	Warning	Node	Node upgrade failed
`ComponentApplyFailed`	Warning	RBLNClusterPolicy	Failed to apply a manifest of a policy-managed component (Device Plugin, NPU Feature Discovery, and so on)
`DriverInstallFailed`	Warning	RBLNDriver	Failed to apply a manifest of a driver-install component (per-node-pool driver DaemonSet, and so on)
`AllComponentsReady`	Normal	RBLNClusterPolicy	Every policy-managed component became ready, so the policy state changed to `ready`
`DriverReady`	Normal	RBLNDriver	The driver became ready on every node pool, so the driver state changed to `ready`
`PolicyIgnored`	Normal	RBLNClusterPolicy	Ignored a later-created policy because an active `RBLNClusterPolicy` already exists (only one policy per cluster takes effect)

The operator emits each event once, when the state actually changes; steady-state reconcile loops do not re-emit it.

During a driver auto upgrade, the operator records events on each node in the order DriverUpgradeStarted → NodeDrained → DriverUpgradeCompleted.

Inspecting Events¶

List the upgrade events of a specific node in chronological order.

$ kubectl get events -A \
  --field-selector involvedObject.kind=Node,involvedObject.name=<node> \
  --sort-by=.lastTimestamp

Events for the custom resources appear in the Events section at the bottom of the describe output.

$ kubectl describe rblnclusterpolicy <name>
$ kubectl describe rblndriver <name>

Event retention

Kubernetes keeps events for one hour by default. After events expire, the operator logs and the conditions on the custom resources hold the same history.

$ kubectl logs -n <namespace> deployment/<release>-rbln-npu-operator-controller-manager
$ kubectl get rblnclusterpolicy <name> -o jsonpath='{.status.conditions}'