Skip to content

Deployment on Kubernetes

This section describes how to deploy the Coding Assistant serving stack on Kubernetes. You prepare the RBLN-CR13 nodes, configure NPU and RoCE NIC allocation for Pods, and install the stack with the Helm chart. The last deployment step checks the first inference response. Chart configuration reference lists the chart values that the deployment steps refer to.

Prerequisites

This section does not cover building the Kubernetes cluster itself.

Stack layers

The serving stack consists of four layers, divided by who builds each one.

Layer Built by Components
Serving stack The chart Gateway, HTTPRoute, InferencePool, llm-d Endpoint Picker (EPP), prefill and decode Deployments with the routing sidecar, serve ConfigMap, ServiceAccount, compile-cache PersistentVolumeClaim (PVC), EPP DestinationRule when the GatewayClass is istio
User resources You Namespace, Image Pull Secret, HF token Secret, model PVC with the weights preloaded, StorageClass, ResourceClaimTemplate (RCT) npu-rdma
Platform software You RBLN NPU Operator, dranet, Kubernetes Gateway API CustomResourceDefinitions (CRDs), Kubernetes Gateway API Inference Extension (GAIE) CRDs, GatewayClass provider
Node You NPU, RoCE NIC, raw NVMe for Rebellions Direct Storage (RDS), RoCE NIC driver, RBLN Driver

Observability is optional. When you use observability, the chart and you add the following components.

Layer Built by Components
Observability The chart and you VMPodScrape and VMServiceScrape resources (default; PodMonitor and ServiceMonitor with Prometheus Operator), OpenTelemetry collector, metrics scraper

For the observability configuration, see Operations.

Deployment steps

Build the stack layer by layer, starting from the Node layer.

Step Page What you create Completion check
1 Node preparation RoCE NIC driver on the host, RBLN NPU Operator, dranet, RDS enabled in RBLN NPU Operator, RDS node label rebellions.ai/rds.present=true Every command in Node preparation checks prints the expected result, and kubectl get deviceclass lists npu.rebellions.ai and dranet
2 Device allocation Namespace, RCT npu-rdma kubectl get resourceslices shows a npu.rebellions.ai ResourceSlice and a dra.net ResourceSlice for each node, and kubectl -n <ns> get resourceclaimtemplate npu-rdma returns the RCT
3 Cluster foundation Gateway API CRDs, GAIE CRDs, GatewayClass provider, Secrets and model store in the stack's Namespace Every command in Cluster checks and key settings succeeds and DRA ok is printed
4 Pod spec requirements None The Pod Security Standards level of the stack Namespace is not restricted (non-root mode meets baseline). The allocatable memory of each node is at least the formula in Node memory budget check
5 Compile cache The storage.compileCache settings in the chart values. The chart creates the compile-cache PVCs during the install in step 7 After the install, kubectl -n <ns> get pvc shows the compile-cache PVCs as Bound
6 Startup and readiness probes None After the install, the Pod events show no Startup probe failed
7 Installing the serving stack (Helm chart) The Helm release and the serving-stack layer objects in the Stack layers table (Gateway, HTTPRoute, InferencePool, EPP, the prefill and decode Deployments, compile-cache PVCs, and so on) kubectl -n <ns> get pods shows the prefill, decode, and EPP Pods as Ready, and the Chat Completions request in First inference returns a response