Monitoring¶

Goal: collect the metrics and traces that the deployed serving stack emits, and read the metrics and traces correctly.

Metric collection targets¶

This section lists what the chart's monitoring CRs collect and how each target authenticates.

• The monitoring CRs collect modelserver metrics per role. The selector is the llm-d.ai/role: decode|prefill label. The chart creates one VMPodScrape per role (PodMonitor with Prometheus Operator), so the job label (<namespace>/<name>) differs by role.
• EPP's /metrics (port 9090) requires a bearer token. The modelserver's /metrics has no authentication.
• The monitoring CRs collect gateway metrics as well. On the default Istio data plane, the monitoring CR collects the Envoy metrics (port 15090, /stats/prometheus). With agentgateway, the same CR collects port 15020.
• LMCache metrics are not a separate collection target. The modelserver exposes LMCache metrics on its own /metrics.

With the default VictoriaMetrics Operator, enable monitoring with the following values. monitoring.scraper.serviceAccount.{name,namespace} names the ServiceAccount of VMAgent, the VictoriaMetrics agent that collects the metrics. If monitoring.scraper.serviceAccount is not set, the render fails.

1
2
3
4
5

helm upgrade <release-name> ./coding-assistant-stack -n <ns> \
  --set monitoring.enabled=true \
  --set monitoring.scraper.serviceAccount.name=<vmagent-sa> \
  --set monitoring.scraper.serviceAccount.namespace=<vmagent-sa-namespace> \
  <values given at install>

The chart binds the permission to read EPP's /metrics to the VMAgent ServiceAccount. The chart does not create a bearer-token Secret. VMAgent authenticates with the projected token mounted in its Pod, and kubelet renews the projected token.

The chart's bundled examples/observability.yaml enables both metrics and tracing. Installing with examples/observability.yaml therefore also requires an OTel collector in the cluster. Fill in the <...> slots of examples/observability.yaml and pass the file with -f. To collect metrics only, remove the tracing and llm-d-router-gateway blocks from the file.

On a cluster that uses Prometheus Operator, set monitoring.backend=prometheus. The chart then creates PodMonitor and ServiceMonitor. The chart also creates the ServiceAccount and token Secret for EPP's /metrics, so monitoring.scraper.serviceAccount is not needed. monitoring.labels adds the label that your Prometheus selector requires (e.g. release: <prometheus-release>). This option is Unverified on the Target Configuration.

Prefix cache hit metrics¶

This stack has two cache tiers: tier 1 in high-bandwidth memory (HBM) and tier 2 in Rebellions Direct Storage (RDS). This section covers only the metrics that tell the two tiers apart. For the full metric list, see vLLM Production Metrics.

Read the metric values as follows.

• Tier 1 hit rate: divide vllm:prefix_cache_hits_total by vllm:prefix_cache_queries_total. vllm:prefix_cache_queries_total is the number of prompt tokens looked up.
• Summing DP ranks: a Pod that runs with DP exposes the vLLM counters of each rank on a different port. Decode uses one port per rank, starting at 8200. Add up the values of all ranks to get the Pod's value.
• LMCache values per worker: LMCache records its metrics separately for each PP worker (worker_id). A sum over the workers is inflated by the PP degree, so read the value of one worker.
• lmcache:num_hit_tokens_total: exposed as 0 even when LMCache is not running. Use it for diagnosis only, not to judge reuse.

To see sub-block reuse alone, send a prompt shorter than --block-size twice. Then read the increase of the vllm:prompt_tokens_by_source_total value whose source is local_cache_hit.

NPU device metrics¶

The RBLN Metrics Exporter exposes NPU device-layer metrics (temperature and utilization). The RBLN Metrics Exporter is an SDK component, not part of the serving stack chart.

Per-stage engine instrumentation¶

With VLLM_RBLN_METRICS=True, each RBLN vLLM worker collects per-step latency split into the device, host, collective, and prepare stages. The engine prints a performance report when it shuts down (Configuration Guide).

The chart has no dedicated values key for VLLM_RBLN_METRICS. Set it with the following keys.

• The same value for prefill and decode: modelserver.serveEnv
• Different values for prefill and decode: modelserver.extraServeEnv

Traces¶

The EPP, the routing sidecar, and vLLM send their spans to the same OTel collector. The processing of one request by the EPP, the routing sidecar, and vLLM, in that order, therefore appears as one trace. The chart default leaves tracing disabled (tracing.enabled: false).

To enable tracing, set tracing.enabled and, on the EPP subchart side, llm-d-router-gateway.router.tracing.enabled to the same value. The render fails if the two keys differ.

1
2
3
4

helm upgrade <release-name> ./coding-assistant-stack -n <ns> \
  --set tracing.enabled=true \
  --set llm-d-router-gateway.router.tracing.enabled=true \
  <values given at install>

Set the OTel collector address in both of the following keys. With tracing enabled, the render fails if the two addresses differ. The chart compares the two addresses as strings, so write the same address in the same form in both keys.