Skip to content

RBLN Metrics Exporter

The RBLN SDK provides a Metrics Exporter that exposes detailed metrics related to Rebellions' NPU devices in Prometheus format. These metrics are designed to be easily scraped by Prometheus and visualized with Grafana, helping you comprehensively monitor your Rebellions NPU device.

Deployment

Step 1: Prepare NPU Nodes

Follow the same steps as outlined in the device plugin documentation to prepare Kubernetes nodes equipped with RBLN NPUs and ensure the RBLN Driver is installed.

Step 2: Deploy Prometheus

Install Prometheus in your Kubernetes cluster using either Helm or the Prometheus Operator.

Note that deploying the RBLN Metrics Exporter does not require Prometheus to be set up beforehand.

Step 3: Deploy RBLN Metrics Exporter

Deploy the RBLN Metrics Exporter as a DaemonSet pod on each node with the following command:

1
$ kubectl apply -f https://raw.githubusercontent.com/rebellions-sw/rbln-metrics-exporter/refs/heads/main/deployments/kubernetes/daemonset.yaml

The provided manifest includes affinity rules to ensure the Metrics Exporter is deployed only on nodes equipped with RBLN NPUs. Specifically, it uses nodeAffinity to target nodes where the rebellions.ai/npu.present label is set to "true", which is typically set by rbln-npu-feature-discovery. 

1
2
3
4
5
6
7
8
9
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - key: rebellions.ai/npu.present
              operator: In
              values:
                - "true"

Kubernetes Mode

The exporter supports a Kubernetes mode switch to disable pod-resource lookups and label dependencies for non-Kubernetes environments.

  • Manifest env: set RBLN_METRICS_EXPORTER_KUBERNETES_MODE=off
  • Binary flag: run ./rbln-metrics-exporter --kubernetes-mode=off

Prometheus-convention Metric Names

The exporter emits metric names that follow standard Prometheus naming conventions (rbln_npu_temperature, rbln_npu_power, etc.). This change applies starting with Metrics Exporter v0.2.4; environments installed with an earlier version must migrate their dashboards and alerts to the new names. The legacy RBLN_DEVICE_STATUS:* names are deprecated and will be removed in a future release.

The NPU Operator (chart 0.4.3 and later) deploys the exporter with PROMETHEUS_METRIC_NAMES=true by default, so dashboards and alerts query the rbln_npu_* names listed in Exported Metrics.

Migration from legacy metric names

Existing dashboards and alerts that match RBLN_DEVICE_STATUS:* must be updated to the rbln_npu_* names before upgrading the operator chart to 0.4.3. To keep the legacy names during the transition, set PROMETHEUS_METRIC_NAMES=false on the exporter pod through metricsExporter.env in your Helm values. Complete the migration before the release that removes the legacy names.

Step 4: (Optional) Configure Prometheus to Scrape Metrics

To allow Prometheus to automatically discover and scrape metrics from the RBLN Metrics Exporter, you can create a ServiceMonitor resource. This is especially useful if you're using the Prometheus Operator. Here's an example ServiceMonitor configuration: 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: rbln-metrics-exporter
  namespace: monitoring
  labels:
    release: prometheus  # Must match the serviceMonitorSelector you configured when installing Prometheus so that this ServiceMonitor is discovered
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: rbln-metrics-exporter
  namespaceSelector:
    matchNames:
      - kube-system
  endpoints:
  - port: metrics
    path: /metrics
    interval: 30s
    scrapeTimeout: 10s
Make sure the selector labels in the ServiceMonitor match the labels on your RBLN Metrics Exporter Pods, and the release label (if used) matches your Prometheus deployment. You can apply this ServiceMonitor with kubectl apply -f servicemonitor.yaml.

Step 5: (Optional) Deploy Grafana

If you wish to visualize the Prometheus metrics through Grafana dashboards, deploy Grafana in your Kubernetes cluster using either Helm or the Grafana Operator.

Exported Metrics

The following metrics are exported for each NPU device, tagged with the device UUID, card name, and character device node (rblnN).

Name Legacy alias (deprecated) Description Unit
rbln_npu_temperature RBLN_DEVICE_STATUS:TEMPERATURE Temperature °C
rbln_npu_power RBLN_DEVICE_STATUS:CARD_POWER Power usage W
rbln_npu_memory_used RBLN_DEVICE_STATUS:DRAM_USED DRAM in use Bytes
rbln_npu_memory_total RBLN_DEVICE_STATUS:DRAM_TOTAL Total DRAM Bytes
rbln_npu_utilization RBLN_DEVICE_STATUS:UTILIZATION Utilization %
rbln_npu_health RBLN_DEVICE_STATUS:HEALTH NPU health status 0/1

Note: rbln_npu_health (legacy RBLN_DEVICE_STATUS:HEALTH) is a binary state metric. 0 means the NPU is active, while 1 means it is inactive.

Common NPU Metrics Label Attributes

Label Description
name Character device node exposed by the kernel driver (Device.name, e.g., rbln0).
uuid Globally unique identifier for the NPU device (Device.uuid).
card Hardware card string surfaced via DeviceInfo.name (e.g., RBLN-CA25).
deviceID PCIe device ID reported in the proto (Device.dev_id, e.g., 1250).
hostname Name of the Kubernetes node where the Pod using the device is scheduled.
driver_version Kernel driver build returned by VersionInfo.drv_version.
firmware_version Firmware revision returned by VersionInfo.fw_version.

Kubernetes NPU Metrics Label Attributes

Label Description
namespace Namespace for the workload using the device. Taken from Pod.metadata.namespace.
container Name of the container consuming the NPU. Taken from Pod.spec.containers[].name.
pod Name of the Pod holding the NPU allocation. Taken from Pod.metadata.name.

Metrics Example

Here's a sample of the metrics text format the exporter produces (Prometheus-convention names; legacy names follow the same shape):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
# TYPE rbln_npu_memory_total gauge
rbln_npu_memory_total{card="RBLN-CA25",container="ubuntu",deviceID="1250",driver_version="2.0.1",firmware_version="2.0.1",hostname="sw-mpc-clsdk-bm-worker-01",name="rbln0",namespace="default",pod="rebel-device-plugin-testpod-1",uuid="55668c63-d739-4193-8212-ad7ba933520c"} 16877882368
rbln_npu_memory_total{card="RBLN-CA25",container="ubuntu",deviceID="1250",driver_version="2.0.1",firmware_version="2.0.1",hostname="sw-mpc-clsdk-bm-worker-01",name="rbln1",namespace="default",pod="rebel-device-plugin-testpod-1",uuid="84389d45-ebf3-4b74-9d80-6ec8a09d8be4"} 16877882368

# TYPE rbln_npu_memory_used gauge
rbln_npu_memory_used{card="RBLN-CA25",container="ubuntu",deviceID="1250",driver_version="2.0.1",firmware_version="2.0.1",hostname="sw-mpc-clsdk-bm-worker-01",name="rbln0",namespace="default",pod="rebel-device-plugin-testpod-1",uuid="55668c63-d739-4193-8212-ad7ba933520c"} 0
rbln_npu_memory_used{card="RBLN-CA25",container="ubuntu",deviceID="1250",driver_version="2.0.1",firmware_version="2.0.1",hostname="sw-mpc-clsdk-bm-worker-01",name="rbln1",namespace="default",pod="rebel-device-plugin-testpod-1",uuid="84389d45-ebf3-4b74-9d80-6ec8a09d8be4"} 150994944

# TYPE rbln_npu_temperature gauge
rbln_npu_temperature{card="RBLN-CA25",container="ubuntu",deviceID="1250",driver_version="2.0.1",firmware_version="2.0.1",hostname="sw-mpc-clsdk-bm-worker-01",name="rbln0",namespace="default",pod="rebel-device-plugin-testpod-1",uuid="55668c63-d739-4193-8212-ad7ba933520c"} 51
rbln_npu_temperature{card="RBLN-CA25",container="ubuntu",deviceID="1250",driver_version="2.0.1",firmware_version="2.0.1",hostname="sw-mpc-clsdk-bm-worker-01",name="rbln1",namespace="default",pod="rebel-device-plugin-testpod-1",uuid="84389d45-ebf3-4b74-9d80-6ec8a09d8be4"} 54

# TYPE rbln_npu_utilization gauge
rbln_npu_utilization{card="RBLN-CA25",container="ubuntu",deviceID="1250",driver_version="2.0.1",firmware_version="2.0.1",hostname="sw-mpc-clsdk-bm-worker-01",name="rbln0",namespace="default",pod="rebel-device-plugin-testpod-1",uuid="55668c63-d739-4193-8212-ad7ba933520c"} 0
rbln_npu_utilization{card="RBLN-CA25",container="ubuntu",deviceID="1250",driver_version="2.0.1",firmware_version="2.0.1",hostname="sw-mpc-clsdk-bm-worker-01",name="rbln1",namespace="default",pod="rebel-device-plugin-testpod-1",uuid="84389d45-ebf3-4b74-9d80-6ec8a09d8be4"} 0

# TYPE rbln_npu_power gauge
rbln_npu_power{card="RBLN-CA25",container="ubuntu",deviceID="1250",driver_version="2.0.1",firmware_version="2.0.1",hostname="sw-mpc-clsdk-bm-worker-01",name="rbln0",namespace="default",pod="rebel-device-plugin-testpod-1",uuid="55668c63-d739-4193-8212-ad7ba933520c"} 59.33700180053711
rbln_npu_power{card="RBLN-CA25",container="ubuntu",deviceID="1250",driver_version="2.0.1",firmware_version="2.0.1",hostname="sw-mpc-clsdk-bm-worker-01",name="rbln1",namespace="default",pod="rebel-device-plugin-testpod-1",uuid="84389d45-ebf3-4b74-9d80-6ec8a09d8be4"} 59.33700180053711

# HELP rbln_npu_health NPU health status
# TYPE rbln_npu_health gauge
rbln_npu_health{card="RBLN-CA25",container="ubuntu",deviceID="1250",driver_version="2.0.1",firmware_version="2.0.1",hostname="sw-mpc-clsdk-bm-worker-01",name="rbln0",namespace="default",pod="rebel-device-plugin-testpod-1",uuid="55668c63-d739-4193-8212-ad7ba933520c"} 0
rbln_npu_health{card="RBLN-CA25",container="ubuntu",deviceID="1250",driver_version="2.0.1",firmware_version="2.0.1",hostname="sw-mpc-clsdk-bm-worker-01",name="rbln1",namespace="default",pod="rebel-device-plugin-testpod-1",uuid="84389d45-ebf3-4b74-9d80-6ec8a09d8be4"} 0