Installing the serving stack (Helm chart)¶

Goal: install the entire stack (gateway, the Endpoint Picker (EPP) of the llm-d router, and modelserver) in the reference configuration with Helm, and get the first inference response.

Chart preparation and the default configuration¶

Pull the coding-assistant-stack chart and unpack the chart with --untar. The install commands on this page use the unpacked directory. The unpacked directory also holds the chart's values.yaml, the bundled examples (examples/), and the scripts (scripts/).

1
2
3
4

helm registry login repo.rebellions.ai    # pulling the chart requires credentials
helm pull oci://repo.rebellions.ai/rebellions/coding-assistant-stack --version <CHART_VERSION> --untar

# -> ./coding-assistant-stack/ (values.yaml, examples/, scripts/)

The chart and the modelserver image are in repo.rebellions.ai, and both require credentials. Log in with helm registry login to pull the chart. Create the Image Pull Secret that the nodes use to pull the modelserver image (see Secrets and the model store).

The routing sidecar image and the EPP image are on the public registry ghcr.io/llm-d and need no credentials. To pull the modelserver image from your own registry, set modelserver.image.repository to that registry's repository.

The chart defaults are the reference configuration. With the chart defaults, the stack serves MiniMax-M2.7 as 4P4D: four prefill Pods and four decode Pods, and each Pod receives the 4 NPUs that the RCT npu-rdma requests. To serve another model, use the recipe files the chart lists in examples/models/README.md (see Common configuration changes).

Save the following values as values-env.yaml. The chart defaults do not include the values in values-env.yaml, because each of those values names an object in your cluster.

1
2
3
4
5
6
7
8

storage:
  model:
    kind: existingClaim
    existingClaim: <pvc-name>          # the PVC that holds the model weights
  compileCache:
    storageClass: <storage-class>      # a StorageClass that supports ReadWriteMany; empty uses the cluster's default StorageClass
imagePullSecrets:
  - <pull-secret-name>                 # the Image Pull Secret for the modelserver image

Check the following chart defaults before the first install.

• The chart creates the compile-cache PVC with accessModes: ["ReadWriteMany"], because the four Pods of a role mount one PVC. Choose a StorageClass that supports ReadWriteMany (see PVC shape, capacity, and accessModes).
• storage.compileCache.size defaults to 200Gi. A PVC cannot shrink, so decide storage.compileCache.size before the first install.
• modelserver.resources requests 40 CPUs and 630Gi of memory per Pod. Check the node capacity against modelserver.resources in Resources and the node budget.

If the EPP is installed outside the chart, set eppServiceName to the name of that EPP's Service.

Preflight check¶

Before installing, check the following items.

Install procedure¶

You choose the Helm release name (<release-name>). The Helm release name can be up to 39 characters when the chart creates an Istio Gateway (the default), and up to 40 characters otherwise.

1
2
3
4
5
6
7
8
9

# 1) first install: 1P1D, fills the compile cache
helm install <release-name> ./coding-assistant-stack -n <ns> \
  --set modelserver.replicas.prefill=1 --set modelserver.replicas.decode=1 \
  -f values-env.yaml

# 2) after the Pods are Ready: scale to 4P4D, passing values-env.yaml again
helm upgrade <release-name> ./coding-assistant-stack -n <ns> \
  --set modelserver.replicas.prefill=4 --set modelserver.replicas.decode=4 \
  -f values-env.yaml

On the first install, compilation delays Ready (see First startup time).

On every later helm upgrade, pass the values given at install time again. For the rule on passing values again, see Scaling and Upgrades › Upgrade, rollback, and removal.

To pin Pods to specific nodes, use modelserver.nodePlacement instead of modelserver.replicas (see Key chart settings).

First inference¶

Check that the modelserver Pods are running and that Istio created the gateway Service. Then send a first inference request through the gateway Service.

1
2

kubectl -n <ns> get pods -l 'llm-d.ai/role'             # a Pod Pending for more than 5 minutes: see Failure response › Diagnosis by symptom
kubectl -n <ns> get svc <release-name>-inference-gateway-istio   # Istio creates this Service

1
2
3
4

# <gateway-address>: the EXTERNAL-IP of the gateway Service, or its ClusterIP from inside the cluster
# <model>: the value of modelserver.model; the chart default is MiniMaxAI/MiniMax-M2.7
curl http://<gateway-address>/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"<model>","max_tokens":32,"messages":[{"role":"user","content":"Say hello."}]}'

If the response has a choices field, the install is complete. If the Pods stay Pending or the request returns an error, see Failure response › Diagnosis by symptom.

Next steps¶

After the first inference succeeds, continue with the following pages.

• Operations
• Failure response