Collect NPU metrics with Prometheus¶

Connect the Operator's Metrics Exporter to Prometheus and query NPU temperatures by node.

Use the Metrics Exporter configuration. If the Operator is not installed, follow Installing the NPU Operator.

Start with a cluster that has Prometheus Operator installed. If it is not installed, follow the kube-prometheus-stack installation guide.

1. Connect the scrape target¶

Check the labels Prometheus uses to select ServiceMonitors and the port name of the Exporter Service.

$ kubectl get prometheus -n monitoring kube-prometheus-stack-prometheus \
  -o jsonpath='{.spec.serviceMonitorSelector.matchLabels}{"\n"}'
{"release":"kube-prometheus-stack"}

$ kubectl get service -n rbln-system rbln-metrics-exporter-service \
  -o jsonpath='{.spec.ports[0].name}{"\t"}{.spec.ports[0].port}{"\n"}'
http    9090

The ServiceMonitor's metadata.labels must match the Prometheus selector. Set spec.selector to match the Exporter Service labels. Set endpoints.port to the Service port name http rather than the port number.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

kubectl apply -f - <<'EOF'
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: npu-tutorial-metrics
  namespace: monitoring
  labels:
    release: kube-prometheus-stack
spec:
  namespaceSelector:
    matchNames: [rbln-system]
  selector:
    matchLabels:
      app.kubernetes.io/component: metrics-exporter
      app.kubernetes.io/name: rbln-npu-operator
  endpoints:
  - port: http
    path: /metrics
    interval: 15s
EOF

2. Query metrics¶

Start port forwarding to Prometheus in one terminal.

1

kubectl port-forward -n monitoring svc/kube-prometheus-stack-prometheus 19090:9090

Keep port forwarding running and query the metrics from another terminal. If the results are empty, wait for Prometheus to discover the new scrape target and try again.

$ curl -sS --get http://127.0.0.1:19090/api/v1/query \
  --data-urlencode 'query=up{service="rbln-metrics-exporter-service"}' | \
  jq -r '.data.result[] | [.metric.pod, .value[1]] | @tsv'
rbln-metrics-exporter-k6rs8 1
rbln-metrics-exporter-frs5x 1

$ curl -sS --get http://127.0.0.1:19090/api/v1/query \
  --data-urlencode 'query=count by (hostname) (rbln_npu_temperature)' | \
  jq -r '.data.result[] | [.metric.hostname, .value[1]] | @tsv'
node-rebel100   1

$ curl -sS --get http://127.0.0.1:19090/api/v1/query \
  --data-urlencode 'query=avg by (hostname) (rbln_npu_temperature)' | \
  jq -r '.data.result[] | [.metric.hostname, .value[1]] | @tsv'
node-rebel100   42

$ curl -sS --get http://127.0.0.1:19090/api/v1/query \
  --data-urlencode 'query=count by (hostname) (rbln_npu_temperature)' | \
  jq -r '.data.result[] | [.metric.hostname, .value[1]] | @tsv'
node-atom   16

$ curl -sS --get http://127.0.0.1:19090/api/v1/query \
  --data-urlencode 'query=avg by (hostname) (rbln_npu_temperature)' | \
  jq -r '.data.result[] | [.metric.hostname, .value[1]] | @tsv'
node-atom   28.375

The check passes when each Exporter reports up=1 and NPU temperatures are collected from each node. Device counts and temperatures vary by environment.

Clean up¶

Press Ctrl+C in the terminal running port forwarding, then delete the ServiceMonitor.

1

kubectl delete servicemonitor -n monitoring npu-tutorial-metrics