Operations¶
Overview¶
This section covers how to operate the installed serving stack.
Prerequisites
The serving stack is installed with the procedure in Deployment on Kubernetes and is serving requests. Monitoring is optional. monitoring.enabled (metrics) and tracing.enabled (traces) both default to false. To collect metrics, the cluster needs the VictoriaMetrics Operator (default) or Prometheus Operator. To collect traces, the cluster needs an OpenTelemetry (OTel) collector.
The following table lists the pages in this section and what each page covers.
| Page | What it covers |
|---|---|
| Monitoring | Metrics to scrape, prefix-cache hit metrics, NPU device metrics, traces |
| Scaling and Upgrades | Replica count and P:D ratio, rules and impact of each change method, rolling updates, version compatibility, removal |
| Failure response | Diagnosis by symptom, failure impact by P/D role, P/D transfer failure, RoCE NIC faults, NPU and RDS device faults |
Related documents¶
- Deployment on Kubernetes - installation procedure, device allocation, compile cache, startup and readiness probes
- Serving Configuration - serve arguments, environment variables, and the reason for each value
- Software Stack and Versions - validated SDK and Kubernetes combinations
- Validation and Limits - validated Target Configuration and serving configurations, and the reproduction method
- llm-d Cluster Observability - telemetry of the routing layer
- RBLN Metrics Exporter - NPU device telemetry
- Installing the RBLN NPU Operator and Upgrading the RBLN NPU Operator - installing and upgrading the RBLN Driver and RBLN NPU Operator
- RBLN Cloud Native Stack (RCNS) - Kubernetes and platform component versions validated together