Pod spec requirements¶

Goal: Check the permissions, ports, volumes, CPU, and memory that this stack's Pods require, and confirm that cluster policy and the nodes' allocatable resources admit them.

Permissions and run mode¶

1
2
3

helm upgrade --install <release-name> ./coding-assistant-stack -n <ns> \
  -f ./coding-assistant-stack/examples/root-privileged.yaml \
  <values to fill in>

Container ports¶

On a cluster where NetworkPolicy or a firewall restricts Pod-to-Pod traffic, allow Pod-to-Pod traffic to the ports in the following table and to the NIXL side-channel ports.

The following table is based on the reference configuration. In the reference configuration, prefill runs with pipeline parallel (PP) degree 4, decode runs with data parallel (DP) degree 4, and a routing sidecar sits in front of decode. The Name column shows the container port names the chart declares. With different parallelism, the number of rank ports changes (see Key chart settings).

The port shim receives prefill's 8001, 8002, and 8003. The Endpoint Picker (EPP) and the routing sidecar send requests to prefill on the same port list as decode (8000, 8001, 8002, 8003). However, the API server of a prefill that runs under PP opens only 8000. The chart therefore runs a port shim that forwards requests on 8001, 8002, and 8003 to 8000.

Volumes¶

The stack mounts the following volumes into each modelserver Pod.

The reference configuration places Pods on multiple nodes. storage.model and storage.compileCache therefore need a StorageClass that supports ReadWriteMany (RWX). Set the two volumes as follows.

• Always set storage.model. Where the weights live differs per cluster, so the chart has no default, and the render fails if storage.model is not set. The reference configuration sets the 250Gi PVC created in Secrets and the model store with kind: existingClaim. Size it to the model's weights plus headroom.
• The chart creates the PVC for storage.compileCache. The chart default size is 200Gi. If storageClass is left empty, the cluster's default StorageClass is used. Decide the size and accessModes before the first install (see PVC shape, capacity, and accessModes).

The volume source is picked with kind alone. The allowed kind values differ per volume.

The chart does not allow emptyDir for compileCache. The modelserver workloads are Deployments, so there is no per-replica persistent volume, and with emptyDir each Pod restart recompiles.

storage.dshm has no kind, and setting kind fails the render. storage.dshm is always a memory-backed emptyDir. What is written to /dev/shm counts toward the modelserver memory limit. When you raise storage.dshm.size, raise the memory requests and limits in modelserver.resources by the same amount.

1
2
3
4
5

storage:
  modelOffload:
    enabled: true
    storageClass: <sc-name>   # Required with kind: pvc. Use a StorageClass with reclaimPolicy: Delete.
    size: 512Gi

Resources and the node budget¶

modelserver.resources defaults to 40 CPU cores and 630Gi of memory, with requests equal to limits. Prefill and decode use the same modelserver.resources value. The memory default is 630Gi because decode uses additional host memory when NIXL registers the KV buffers.

Before installing, check that the allocatable memory and CPU of each node satisfy the following formulas.

allocatable(memory) ≥ modelserver memory × (pods on that node) + routing sidecar 256Mi × (decode pods on that node) + EPP 8Gi + gateway ~0.2Gi
allocatable(CPU)    ≥ modelserver CPU    × (pods on that node) + routing sidecar 0.2  × (decode pods on that node) + EPP 4 + gateway 0.1

Memory: the term most easily missed is the EPP that the chart brings up on its own. Leaving the EPP out of the calculation can overshoot allocatable memory by a few GB and block scheduling. The pre-install memory check is in Node memory budget check.

CPU: for serving, the default of 40 cores is enough. The first compile, however, runs more threads than serving. If the CPU limit is too low, the cgroup delays the compile threads (throttling). The throttling writes nothing to the log and only slows down startup. If the compile runs past the startup budget (see Startup and readiness probes), the startup probe fails and the kubelet restarts the Pod.

On a node with cpuManagerPolicy: static, the modelserver receives dedicated CPUs and is not throttled. For the CPU value of each model, use the values file of that model's recipe (see Chart preparation and the default configuration).

The per-node hardware counts are on the Hardware page.

1
2

kubectl get nodes -o json | jq -r '.items[]
  | "\(.metadata.name)\t\(.status.allocatable.memory)"'