Scaling and Upgrades¶

Scaling¶

Goal: adjust the replica count, the P:D ratio, and the llm-d Endpoint Picker (EPP) resources.

Replica adjustment¶

Set the prefill replica count with modelserver.replicas.prefill and the decode replica count with modelserver.replicas.decode. The chart default is 4 for both. The following values block sets 4 prefill replicas and 5 decode replicas.

modelserver:
  replicas:
    prefill: 4
    decode: 5

All prefill replicas together use devices.npu.count × the prefill replica count NPUs. The same applies to decode. Apply the new replica counts with helm upgrade as described in Upgrade, rollback, and removal.

Scaling up requires spare NPU and RoCE NIC capacity for each added Pod. For the per-node hardware counts, see Hardware. For how the devices are allocated, see Device allocation.

When you reduce a replica count, the Deployment deletes the Pods beyond the new count. Requests that a deleted Pod was handling fail with 5xx.

A mode switch (replicas ↔ nodePlacement) and a node reassignment cannot be applied with helm upgrade. Apply them with uninstall and then install (Change rules).

P:D ratio selection¶

Choose the P:D ratio with Hardware › P:D ratio, which lists the measured layouts. The default layout is the reference 4P4D, which places one prefill Pod and one decode Pod on each node.

EPP resources¶

Adjust the EPP resources with llm-d-router-gateway.router.epp.resources. The chart default request is 4 CPU cores and 8Gi of memory, with a memory limit of 16Gi.

Include the EPP memory request (8Gi) in the node budget. For the formula, see Pod spec requirements › Resources and the node budget.

Upgrade, rollback, and removal¶

Goal: change, roll back, and remove a serving stack safely.

Every time you run helm upgrade, pass the values file you used at install (-f values-env.yaml) again.

The reference 4P4D has no spare devices. Before a change that recreates Pods, see No spare devices.

Change rules¶

Apply each kind of change as follows.

Apply a value change with helm upgrade. For the Pods that the change recreates, see Impact by change type.
Apply a topology mode switch (replicas ↔ nodePlacement) and a node reassignment with uninstall and then install. Reordering the entries of the nodePlacement list also counts as a node reassignment. If you apply these changes with helm upgrade, a new Pod can request a device before the old Pod returns that device.
Roll back with uninstall and then install, using the previous image tag and values. helm rollback is not supported.

Impact by change type¶

The following table lists which Pods each value change recreates.

Change	Pod recreation	Method
`modelserver.replicas.{prefill,decode}` (same mode)	None for the Pods that stay. Setting a role to 0 removes the Deployment of the role set to 0, while its compile-cache PersistentVolumeClaim (PVC) stays	`helm upgrade`
Everything under `gateway` / `monitoring.enabled`	None. Only objects other than modelserver change: the Gateway and the monitoring custom resources (CRs)	`helm upgrade`
Under `modelserver`: `image` · `serveArgs` · `serveEnv` · `extraServeArgs` · `extraServeEnv` · `removeServeArgs` · `probes` · `resources` · `podOverrides`, plus every `storage.` volume key except the PVC spec keys in the rows below, `routing.proxy.`, `tracing.*`, `imagePullSecrets`, `eppServiceName`	All Pods, with these exceptions: `routing.proxy.` recreates decode Pods only; `extraServeEnv.<role>` and `podOverrides.<role>` recreate the `<role>` Pods only; `tracing.collector.` has no effect while tracing is disabled	`helm upgrade`. The values in the Change column change the Pod spec or the serve ConfigMap, so the affected modelservers restart
`devices.claimTemplateName`, `devices.{npu,rdma}.mode`, `devices.rdma.enabled`, `devices.rdma.discoveryTimeoutSeconds`, `devices.rdma.sideChannel.host`, and, in device-plugin mode, `count` and `resourceName`	All Pods	`helm upgrade`. The chart renders the keys in the Change column into the Pod spec (`resourceClaims`, resource requests, container env), so the Deployment recreates the Pods
`devices.rdma.sideChannel.basePort.<role>`	The `<role>` Pods	`helm upgrade`
`storage.compileCache.{storageClass,accessModes}` and the same keys of `storage.model` (`kind: pvc`)	None. Only the PVC spec would change	uninstall, delete the PVC (or switch to `kind: existingClaim`), then install. `storageClass` and `accessModes` cannot change after the PVC is created, so `helm install` and `helm upgrade` refuse an in-place upgrade at render time while the old PVC exists. The check reads the PVC from the cluster, so `helm template` and GitOps renders skip it (see Removal › What remains)
`storage.compileCache.size` and `storage.model.size` (`kind: pvc`)	None. Only the PVC capacity changes	`helm upgrade` grows the PVC on a StorageClass that allows volume expansion. `helm install` and `helm upgrade` refuse a value smaller than the remaining PVC at render time; `helm template` and GitOps renders skip this check
`replicas` ↔ `nodePlacement` mode switch, node reassignment	All Pods	uninstall → install (Change rules)
Switching between AGG and P/D disaggregation, or changing the model	All Pods	uninstall → install. Upgrading a P/D install with AGG values deletes the prefill Deployment while its compile-cache PVC remains
`llm-d-router-gateway.router.epp.pluginsCustomConfig` (EPP plugin configuration)	None. The subchart emits no ConfigMap checksum annotation, so the EPP does not restart after `pluginsCustomConfig` changes	`helm upgrade`, then restart the EPP yourself: `kubectl -n <ns> rollout restart deploy/<release-name>-epp`
`llm-d-router-gateway.router.inferencePool.create` toggle	All Pods	uninstall → install. Changing `inferencePool.create` changes the EPP Deployment selector, and a selector is immutable, so an in-place upgrade fails at apply

The reference configuration allocates devices with Dynamic Resource Allocation (DRA). The devices a Pod receives are defined in the ResourceClaimTemplate (RCT). Changing devices.rdma.count therefore does not change the number of NICs a Pod receives.

To change the devices a Pod receives, you need a new RCT, because the spec of an existing RCT cannot be modified. Apply the new RCT under a name different from the existing RCT. Then set devices.claimTemplateName to the new RCT name and run helm upgrade.

After applying a change, check that every Pod is Ready. Then send an inference request as described in Installing the serving stack (Helm chart) › First inference.

Rolling updates¶

For the general behavior of rolling updates, see the Kubernetes documentation on Deployment strategy. This section covers only what is specific to this chart.

With the chart's default update strategy (maxSurge: 1, maxUnavailable: 0), the Deployment adds a new Pod first and then terminates the old Pod. A change that recreates the prefill Pods therefore needs one prefill Pod's worth of spare devices. The same applies to the decode Pods. A change that recreates both the prefill and the decode Pods, such as an image swap, needs two Pods' worth of spare devices.

Check the spare devices by comparing the per-node device list with the resources currently allocated.

kubectl get resourceslices -o wide      # Device inventory per node
kubectl -n <ns> get resourceclaims      # What is currently assigned

No spare devices¶

The reference 4P4D uses every device, so it has no spare devices. In this case, set modelserver.strategy as follows. With the default strategy, the new Pod cannot receive devices and the rollout does not progress.

modelserver:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1

With this strategy, the Deployment terminates the old Pod first. The role being replaced therefore has one replica fewer during the replacement. Confirm that the remaining Pods can handle the load.

In nodePlacement mode, the chart ignores modelserver.strategy and always replaces Pods with Recreate.

Other situations to watch¶

The following cases slow down or stop a rolling update.

Situation	What happens	What to do
Changed to `maxSurge: 0`, but the old Pod returns its NPU claim, RDMA queue pair, and port late	The new Pod waits (Pending, or retrying its startup) until the old Pod returns the NPU claim, RDMA queue pair, and port	Check the old Pod's claim return with `kubectl -n <ns> get resourceclaims`. Once the claim is returned, the new Pod continues its startup
The compile cache is empty	The new Pod compiles first (Compile cache › First startup time). The old Pod keeps serving meanwhile, so the rollout only takes longer	Proceed with the compile-cache PVC preserved (Compile cache)
The topology mode is changed at the same time	The Deployment name changes, so a rolling update does not apply	Do uninstall → install as in Change rules
The compile-cache `accessModes` is `ReadWriteOncePod`	Only one Pod can mount the compile-cache PVC, so the default combination that starts the new Pod first does not work. The chart rejects `ReadWriteOncePod` with the default rolling update at render time	Change to `maxSurge: 0`, `maxUnavailable: 1` or to `type: Recreate`
The compile cache is `ReadWriteOnce`	The render rejects `ReadWriteOnce` when a role's Pods span more than one node (`nodePlacement` mode) or when a role has more than one replica (`replicas` mode). Even when the render accepts the layout, a new Pod scheduled onto another node cannot attach the volume	Keep the Pods on one node, or replace with `maxSurge: 0`, `maxUnavailable: 1`
The model volume is not `ReadWriteMany`	A new Pod scheduled onto another node cannot attach the volume. For a PVC the chart creates (`storage.model.kind: pvc`), the render rejects `ReadWriteOnce` with two or more Pods. For a claim from outside the chart (`existingClaim`), the render does not catch a volume that is not `ReadWriteMany`, because the chart cannot see the accessModes of a volume the chart did not create	Keep the Pods on one node, or replace with `maxSurge: 0`, `maxUnavailable: 1`

The chart derives each Deployment's progressDeadlineSeconds from the startup budget, which is 2 hours by default (Startup and readiness probes › Deployment progress deadline). Kubernetes therefore does not mark a slow rollout as failed within those 2 hours. List the role Deployments with kubectl -n <ns> get deploy, then check progress with the following command.

kubectl -n <ns> rollout status deploy/<name>

Change operation examples¶

The following commands check the current Helm release, then show an image swap and an uninstall → install.

# Current Helm release and the values applied
helm list -n <ns>
helm get values <release-name> -n <ns>

# An image swap recreates every Pod
# (without spare devices, change the strategy first; see Rolling updates)
helm upgrade <release-name> ./coding-assistant-stack -n <ns> \
  --set modelserver.image.tag=<new-tag> <values given at install>
# For an install pinned by digest, clear the digest in the same upgrade with --set modelserver.image.digest=.
# A tag and a digest given together make the render fail.
# A tag that is all digits needs --set-string. --set reads the tag as a number, and the schema rejects the tag.

# Rollback and topology mode switch: uninstall, confirm the Pods are gone, then install
# (helm rollback is not a supported path)
helm uninstall <release-name> -n <ns>
kubectl -n <ns> get pods -l 'llm-d.ai/role'      # Install after no Pod remains
helm install <release-name> ./coding-assistant-stack -n <ns> <previous or target values>

Version compatibility¶

The MAJOR.MINOR of the chart version matches the RBLN SDK release (0.12 for this release), and a chart-only fix is published as a PATCH release (for example, 0.12.1). The vllm-rbln version inside the default modelserver image is recorded separately as the chart's appVersion. An image swap changes every SDK component inside the image at once. For the component list, see Software Stack and Versions › Components.

The validated combinations of the SDK inside the image with Kubernetes and RBLN NPU Operator follow the RBLN Cloud Native Stack (RCNS) table in the Support Matrix. The RCNS table pins the Kubernetes, container runtime, RBLN Driver, RBLN NPU Operator, DRA driver, and RBLN SDK versions together per RCNS version, and each RCNS version corresponds to one Kubernetes minor version. The combination this stack validated is on Software Stack and Versions.

When the SDK is changed, decide at the same time whether the compile cache can be reused. Cache reuse requires the same RBLN Compiler major.minor version, the same serving configuration, and the same NPU type. The conditions and the cleanup method are in Compile cache › Reuse condition.

RBLN Driver and RBLN NPU Operator upgrade¶

The RBLN Driver and RBLN NPU Operator are outside the serving stack chart. For the procedure, see Upgrading the RBLN NPU Operator and Installing the RBLN NPU Operator.

After an RBLN NPU Operator upgrade, check that DeviceClass and ResourceSlice are published again, with the same commands as Device allocation › Device allocation checks. Publication can pause briefly during the upgrade. Until they are published again, newly scheduled Pods stay Pending. Pods already serving keep the claims they were assigned and are not affected.

Removal¶

Remove the serving stack with the following command.

helm uninstall <release-name> -n <ns>

What is deleted¶

helm uninstall deletes the objects the chart created. The main objects are the following.

The prefill and decode modelserver Deployments
The EPP Deployment and the InferencePool
The Gateway and the HTTPRoute
The ConfigMaps, ServiceAccounts, and RBAC objects the chart created

What remains¶

The following items remain after helm uninstall.

What remains	Reason	To delete it
The compile-cache PVC, which exists only for the role that had Pods	`helm.sh/resource-policy: keep` preserves the filled cache	`kubectl -n <ns> delete pvc <release-name>-compile-cache-<role>` for each role that had Pods
What you created: namespace, Image Pull Secret, `llm-d-hf-token`, model store	The chart did not create them. However, with `storage.model.kind: pvc` the chart creates the model PVC (`<release-name>-model-cache`) with `resource-policy: keep`, and the model PVC also remains after uninstall	Delete directly
The ResourceClaimTemplate `npu-rdma`	You applied the RCT, so the chart does not manage the RCT	Delete directly
The node label `rebellions.ai/rds.present`	You attached the label to the node (RDS Support), so the chart does not manage the label	`kubectl label node <node> rebellions.ai/rds.present-`
The Helm release history of a failed install	While the failed release's record remains, Helm refuses the next install under the same name	`helm uninstall <release-name> -n <ns>`

After helm uninstall, check what remains with the following commands.

kubectl -n <ns> get pvc,resourceclaimtemplate
kubectl -n <ns> get pods -l 'llm-d.ai/role'   # nothing should remain

If you enabled the offload volume (storage.modelOffload), the offload volume is deleted together with the Pod. However, if the StorageClass reclaimPolicy is Retain, the PersistentVolume (PV) remains, so delete the PV directly.

A remaining compile-cache PVC can be reused by the next install. If you reinstall with the same Helm release name, the chart uses the remaining PVC, so the new Pods do not compile again. If you change the Helm release name, the chart creates a new PVC, so the new Pods compile from the beginning.