Operator Observability¶

Monitor the RBLN NPU Operator using metrics, Kubernetes events, resource Ready conditions, and logs. For NPU utilization, temperature, and power, see the Metrics Exporter.

Operator Metrics¶

The operator exposes its own state metrics in Prometheus format: reconcile outcomes for RBLNClusterPolicy and RBLNDriver instances, NPU node counts, and driver upgrade progress. The Helm value operator.metrics.enabled controls the endpoint (default true).

1
2
3
4

operator:
  metrics:
    serviceMonitor:
      enabled: true

1
2
3

kubectl create clusterrolebinding rbln-operator-metrics-scraper \
  --clusterrole=rbln-npu-operator-metrics-reader \
  --serviceaccount=<MONITORING_NAMESPACE>:<PROMETHEUS_SERVICE_ACCOUNT>

1

kubectl get clusterrolebinding rbln-operator-metrics-scraper

Operator Events¶

The operator records state changes and errors as Kubernetes events. Use kubectl describe to see what happened and when. Errors on custom resources also appear in their Ready conditions, as shown in the following table. After an event expires, check the Ready condition for the current state.

The operator records an event once when the state changes. Repeated reconcile loops do not record it again while the state stays the same. DriverUpgradePodStuck is the exception: it repeats while the driver pod is waiting in the replacement step. The four driver-validation warnings (InvalidSpec, ConflictingNodeSelector, DriverImageNotFound, and DriverFamilyLabelMissing) are recorded once per event reason and resource generation (metadata.generation). If a problem clears and then recurs without a spec change, the operator updates the Ready condition without recording another event. The two node-routing events are recorded only when a node's owner changes during reconciliation.

Because both CRDs are cluster-scoped, their events are recorded in the default namespace rather than the operator's namespace.

During a driver auto upgrade, the operator records DriverUpgradeStarted and then DriverUpgradeCompleted on each node, or DriverUpgradeSkipped or DriverUpgradeFailed when a step fails. See Events for the state each event marks.

1
2
3

kubectl get events -A \
  --field-selector involvedObject.kind=Node,involvedObject.name=<NODE_NAME> \
  --sort-by=.lastTimestamp

1

kubectl describe rblnclusterpolicy <name>

1

kubectl describe rblndriver <name>

1
2

kubectl logs -n <NAMESPACE> deployment/rbln-npu-operator-controller-manager
kubectl get rblnclusterpolicy <name> -o jsonpath='{.status.conditions}'

Driver and RSMD Readiness¶

An RBLNDriver reports driver pod readiness under .status.nodePools and RSMD readiness under .status.smd. Even when the driver pods are ready, the resource reports Ready=False with reason SmdNotReady until RSMD is ready.

1
2
3

kubectl get rblndriver
kubectl get rblndriver <DRIVER_NAME> -o jsonpath='{.status.nodePools}{"\n"}'
kubectl get rblndriver <DRIVER_NAME> -o jsonpath='{.status.smd}{"\n"}'

Logging¶

The operator writes structured logs to stdout, including records produced by the Kubernetes client libraries. Configure the logs with the following chart values:

1
2
3
4
5
6

operator:
  logging:
    level: info              # error | info | debug | panic, or an integer N for logr V(N)
    encoder: json            # json | console
    timeEncoding: rfc3339nano
    develMode: false