RBLN Metrics Exporter¶
The RBLN SDK provides a Metrics Exporter that exposes detailed metrics related to Rebellions' NPU devices in Prometheus format. These metrics are designed to be easily scraped by Prometheus and visualized with Grafana, helping you comprehensively monitor your Rebellions NPU device.
The exporter runs in one of two execution modes: the default local mode collects metrics from its own node, and gateway mode collects metrics on behalf of multiple remote hosts.
Deployment¶
Step 1: Prepare NPU Nodes¶
Follow the same steps as outlined in the device plugin documentation to prepare Kubernetes nodes equipped with RBLN NPUs and ensure the RBLN Driver is installed.
Step 2: Deploy Prometheus¶
Install Prometheus in your Kubernetes cluster using either Helm or the Prometheus Operator.
Note that deploying the RBLN Metrics Exporter does not require Prometheus to be set up beforehand.
Step 3: Deploy RBLN Metrics Exporter¶
Deploy the RBLN Metrics Exporter as a DaemonSet pod on each node with the following command:
The provided manifest includes affinity rules to ensure the Metrics Exporter is deployed only on nodes equipped with RBLN NPUs. Specifically, it uses nodeAffinity to target nodes where the rebellions.ai/npu.present label is set to "true", which is typically set by rbln-npu-feature-discovery.
Kubernetes Mode¶
The exporter supports a Kubernetes mode switch to disable pod-resource lookups and label dependencies for non-Kubernetes environments.
- Manifest env: set
RBLN_METRICS_EXPORTER_KUBERNETES_MODE=off - Binary flag: run
./rbln-metrics-exporter --kubernetes-mode=off
Prometheus-convention Metric Names¶
The exporter emits metric names that follow standard Prometheus naming conventions (rbln_npu_temperature, rbln_npu_power, etc.) when PROMETHEUS_METRIC_NAMES=true. The RBLN_DEVICE_STATUS:* names remain available as deprecated aliases.
The NPU Operator enables PROMETHEUS_METRIC_NAMES by default, so dashboards and alerts should query the rbln_npu_* names listed in Metrics Information.
Step 4: (Optional) Configure Prometheus to Scrape Metrics¶
To allow Prometheus to automatically discover and scrape metrics from the RBLN Metrics Exporter, you can create a ServiceMonitor resource. This is especially useful if you're using the Prometheus Operator. Here's an example ServiceMonitor configuration:
selector labels in the ServiceMonitor match the labels on your RBLN Metrics Exporter Pods, and the release label (if used) matches your Prometheus deployment. You can apply this ServiceMonitor with kubectl apply -f servicemonitor.yaml.
Step 5: (Optional) Deploy Grafana¶
If you wish to visualize the Prometheus metrics through Grafana dashboards, deploy Grafana in your Kubernetes cluster using either Helm or the Grafana Operator.
Multi-target Gateway Mode¶
Starting with Metrics Exporter v0.3.1, a single exporter instance can collect metrics on behalf of multiple remote hosts in gateway mode. Gateway mode follows the Prometheus multi-target exporter pattern and is intended for bare-metal/non-Kubernetes environments where deploying an exporter per host is impractical.
| Aspect | local (default) |
gateway |
|---|---|---|
| Deployment shape | One per node (DaemonSet) | One central instance |
| Collection target | rbln-smd on the local node |
Remote rbln-smd specified per scrape |
| Target selection | None (own node) | ?target=<host:port> query parameter |
| Kubernetes pod labels | Provided | Not provided (hardware metrics only) |
Operation¶
When Prometheus scrapes the gateway as /metrics?target=<host:port>, the gateway connects to the target's rbln-smd over gRPC, collects the full metrics on demand, and responds in Prometheus format. The gateway keeps no metric state between requests and holds no target list; the target list is managed entirely by the Prometheus scrape configuration. Per-target gRPC connections are cached for reuse and reconnected automatically when dropped.
Gateway Startup¶
The exporter runs in one of two execution modes, selected at startup with the --mode flag or the RBLN_METRICS_EXPORTER_MODE environment variable.
| Variable | Description | Default |
|---|---|---|
RBLN_METRICS_EXPORTER_MODE |
Execution mode of the exporter. local collects metrics from the rbln-smd on the exporter's own node; gateway collects metrics from the remote rbln-smd specified by each scrape request. |
local |
Prometheus Scrape Configuration¶
List the NPU hosts as targets, convert each target into the ?target= parameter with relabeling, and send the actual HTTP request to the gateway.
With this configuration, Prometheus scrapes http://rbln-gateway:9200/metrics?target=npu-host-1:50051 for each host and stores every series with the instance="npu-host-1:50051" label.
Limitations¶
- Collection happens inside the scrape request, so slow targets consume scrape time. The gateway honors the
X-Prometheus-Scrape-Timeout-Secondsheader sent by Prometheus; increasescrape_timeoutfor slow links. - Pod/namespace/container labels are not provided. The kubelet pod-resources API is node-local only, so allocation information on remote hosts cannot be looked up. Use
localmode (DaemonSet) in Kubernetes clusters. - The
hostnamelabel is filled with the host part of the target address. - The gRPC connection to the target is plaintext. Keep the daemon port reachable from the gateway but restricted to a trusted network.
Metrics Information¶
The following metrics are exported for each NPU device, tagged with the device UUID, card name, and character device node (rblnN).
| Name | Deprecated alias | Description | Unit | Minimum version |
|---|---|---|---|---|
rbln_npu_temperature |
RBLN_DEVICE_STATUS:TEMPERATURE |
Temperature | °C | v0.2.0 |
rbln_npu_power |
RBLN_DEVICE_STATUS:CARD_POWER |
Power usage | W | v0.2.0 |
rbln_npu_memory_used |
RBLN_DEVICE_STATUS:DRAM_USED |
DRAM in use | Bytes | v0.2.0 |
rbln_npu_memory_total |
RBLN_DEVICE_STATUS:DRAM_TOTAL |
Total DRAM | Bytes | v0.2.0 |
rbln_npu_utilization |
RBLN_DEVICE_STATUS:UTILIZATION |
Utilization | % | v0.2.0 |
rbln_npu_health |
RBLN_DEVICE_STATUS:HEALTH |
NPU health status | 0/1 | v0.2.0 |
rbln_npu_device_status |
N/A | Device state machine status (one series per state label; only the current state is 1) |
0/1 | v0.3.0 |
rbln_npu_power_state |
N/A | DVFS performance state | Level | v0.3.0 |
rbln_npu_pcie_link_speed_gts |
N/A | Current PCIe link speed | GT/s | v0.3.0 |
rbln_npu_pcie_link_width |
N/A | Current PCIe link width | Lanes | v0.3.0 |
rbln_npu_device_info |
N/A | Device identity and static attributes exposed as labels | Always 1 | v0.3.0 |
rbln_up |
N/A | Whether the last metrics collection from RSMD succeeded | 0/1 | v0.3.1 |
rbln_npu_device_shared |
N/A | Whether the device's ResourceClaim is shared by more than one Pod, so its Kubernetes labels name only one of them |
0/1 | v0.3.3 |
Note
rbln_npu_health (deprecated alias: RBLN_DEVICE_STATUS:HEALTH) is a binary state metric. 0 means the NPU is active, while 1 means it is inactive.
Note
rbln_up is exposed as 0 between exporter start and the first collection cycle.
The exporter does not expose the following metrics when it cannot determine their values.
rbln_npu_power_state: devices whose daemon does not report a performance state valuerbln_npu_pcie_link_speed_gts,rbln_npu_pcie_link_width: devices whose topology information could not be queried
Common NPU Metrics Label Attributes¶
| Label | Description |
|---|---|
name |
Character device node exposed by the kernel driver (Device.name, e.g., rbln0) |
uuid |
Globally unique identifier for the NPU device (Device.uuid) |
card |
Card product name surfaced via Device.card_name (e.g., RBLN-CA25) |
deviceID |
PCIe device ID reported in the proto (Device.dev_id, e.g., 1250) |
hostname |
Name of the Kubernetes node where the Pod using the device is scheduled |
driver_version |
Kernel driver build returned by VersionInfo.drv_version |
firmware_version |
Firmware revision returned by VersionInfo.fw_version |
Kubernetes NPU Metrics Label Attributes¶
| Label | Description |
|---|---|
namespace |
Namespace for the workload using the device. Taken from Pod.metadata.namespace |
container |
Name of the container consuming the NPU. Taken from Pod.spec.containers[].name |
pod |
Name of the Pod holding the NPU allocation. Taken from Pod.metadata.name |
Dynamic Resource Allocation¶
The exporter reads both the device-plugin and DRA fields of the kubelet pod-resources API, so devices receive pod labels whether a cluster uses the Device Plugin or the NPU DRA Driver.
DRA attribution requirements
DRA attribution requires Metrics Exporter v0.3.3 or later and Kubernetes 1.34 or later. Kubernetes 1.34 enables the required kubelet feature gates by default, so no kubelet flags are needed. The DRA driver uses the resource.k8s.io/v1 API.
A metric series carries one set of pod labels, but a DRA device can have more than one claimant. The exporter always picks the same claimant across scrapes so a device's series does not split:
- Containers of one Pod sharing a claim:
namespaceandpodstay correct, and one of the container names is used. - Several Pods sharing one
ResourceClaim: the other Pods are not included in the labels, andrbln_npu_device_sharedis set to1for those devices so the partial attribution is visible on a dashboard rather than only in the logs.
Metric-specific Label Attributes¶
In addition to the common labels, rbln_npu_device_status uses a state label.
| Label | Metric | Values |
|---|---|---|
state |
rbln_npu_device_status |
ready · busy · init · fault · finish · not_found |
Device Info Metric Label Attributes¶
In addition to the common labels, rbln_npu_device_info exposes the device's static attributes with the following labels.
| Label | Description |
|---|---|
smc_version |
SMC firmware version |
pci_bus_id |
PCI bus address |
numa_node |
NUMA node number the device is attached to (-1 when NUMA information is unavailable) |
rsd_group |
RSD group the device belongs to |
cpu_list |
Local CPU list of the device |
is_vf |
Whether the device is an SR-IOV virtual function (VF) (true/false) |
parent_name |
Parent physical device name when the device is a VF |
num_vfs |
Number of VFs configured on the physical device |
Note
The exporter exposes the numa_node, rsd_group, and cpu_list labels as empty values when topology information cannot be queried.
Metrics Example¶
Here's a sample of the Prometheus-convention metrics text format that the exporter produces:
Logging¶
The exporter writes structured logs to stdout. Configure the log level and output format with the following environment variables:
| Variable | Description | Default |
|---|---|---|
RBLN_METRICS_EXPORTER_LOG_LEVEL |
Verbosity: error, warning (or warn), info, debug, trace. |
info |
RBLN_METRICS_EXPORTER_LOG_FORMAT |
Output format: json or text. |
json |
warning is emitted in records as "level":"warn", so write dashboard and alert filters against warn. trace dumps request payloads and is not intended for production. An invalid value falls back to the default and logs a warning rather than failing startup.
When deployed by the RBLN NPU Operator, configure these settings with the chart's metricsExporter.logging values. See Component Logging.