Cluster foundation¶

Goal: Prepare the cluster foundation for the serving stack, together with the Secrets and model store the stack uses.

Gateway API and Gateway API Inference Extension CRDs¶

Install the Kubernetes Gateway API CustomResourceDefinitions (CRDs) and the Kubernetes Gateway API Inference Extension (GAIE) CRDs, which define InferencePool. Without the Gateway API CRDs and the GAIE CRDs, the chart install fails immediately with no matches for kind ....

GATEWAY_API_VERSION=v1.5.1
GAIE_VERSION=v1.5.0
kubectl apply -f "https://github.com/kubernetes-sigs/gateway-api/releases/download/${GATEWAY_API_VERSION}/standard-install.yaml"
kubectl apply -f "https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/${GAIE_VERSION}/v1-manifests.yaml"

GatewayClass provider¶

The reference configuration uses Istio as the GatewayClass provider. Install Istio with GAIE enabled, as in llm-d's Istio guide. Target Configuration validation used Istio 1.30.0.

The chart uses the GatewayClass istio by default. With Istio, you therefore do not need to set a GatewayClass in the chart values.

The chart install succeeds even without a provider or without GAIE enabled in Istio. Instead, one of the following states appears.

State	Cause
The Gateway stays `Accepted=Unknown` (`Waiting for controller`) and serves no requests	No GatewayClass provider is installed
An HTTPRoute that points at an InferencePool reports `ResolvedRefs=False`	GAIE is not enabled in Istio

The chart creates the Endpoint Picker (EPP), InferencePool, and HTTPRoute when you run helm install. On this page, you therefore prepare only the CRDs and the GatewayClass provider.

agentgateway (Unverified): install agentgateway with GAIE support and apply examples/gateway-agentgateway.yaml to the chart with -f.

Secrets and the model store¶

Create the Image Pull Secret, the HF token Secret, and the model PersistentVolumeClaim (PVC) in the namespace created in Device allocation. The chart does not create the Image Pull Secret, the HF token Secret, or the model PVC, and references them by name. For the symptom when the Image Pull Secret, the HF token Secret, or the model PVC is missing, see Failure response › Diagnosis by symptom.

# Image Pull Secret
kubectl -n <ns> create secret docker-registry <pull-secret-name> \
  --docker-server=<registry> --docker-username=<user> --docker-password=<token>

# HF token Secret. The EPP references the Secret by the name llm-d-hf-token
kubectl -n <ns> create secret generic llm-d-hf-token --from-literal=HF_TOKEN=<token>

Secret	Contents
Image Pull Secret	Container registry credentials
`llm-d-hf-token` (key `HF_TOKEN`)	The HuggingFace token injected into the EPP Pod as the environment variable `HF_TOKEN`

To change the name of the HF token Secret, overwrite llm-d-router-gateway.router.epp.env as a whole. The llm-d-router-gateway.router.epp.env array carries the Secret name.

# model store: the PVC that holds the weights
kubectl -n <ns> apply -f - <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: model-cache
spec:
  accessModes: ["ReadWriteMany"]
  storageClassName: <sc-name>
  resources:
    requests:
      storage: 250Gi
EOF

The 250Gi in the PVC example is the size of the MiniMax-M2.7 weights (about 234 GB) plus headroom. The capacity and accessModes requirements per volume are in Volumes.

Load the model weights in advance. The chart assumes that the model weights are already on the model volume. Mount the model-cache PVC in a temporary Pod and download the model under .cache/huggingface on the volume, in the HuggingFace cache layout. The chart sets HF_HOME to <mountPath>/.cache/huggingface. For the download command, see the HuggingFace CLI guide.

Make the whole model directory on the volume readable by the modelserver container. In the default non-root mode, the chart sets runAsUser: 2000, runAsGroup: 0, and fsGroup: 2000 on the modelserver Pod. If the modelserver container cannot read the weights, the engine cannot load them, and startup fails.

Cluster checks and key settings¶

Run the following commands to confirm that the cluster foundation is ready.

# Kubernetes version and DRA API
kubectl version
kubectl get --raw /apis/resource.k8s.io/v1 >/dev/null && echo "DRA ok"

# Kubernetes Gateway API CRD / Kubernetes GAIE CRD / provider
kubectl get crd gatewayclasses.gateway.networking.k8s.io
kubectl get crd inferencepools.inference.networking.k8s.io
kubectl get gatewayclass

# workload resources
kubectl -n <ns> get secret,pvc
kubectl get storageclass

When every check command succeeds and DRA ok is printed, the cluster foundation is ready. If a result differs, act as follows.

Result	Cause	Action
`DRA ok` is not printed	The cluster does not serve the Dynamic Resource Allocation (DRA) API `resource.k8s.io/v1`	Use a Kubernetes version from Software Stack and Versions › Pinned versions
A CRD is not found	The Gateway API CRDs or the GAIE CRDs are missing	Install them as described in Gateway API and Gateway API Inference Extension CRDs
`kubectl get gatewayclass` prints nothing	No GatewayClass provider is installed	Install one as described in GatewayClass provider

Set the names of the Image Pull Secret and the model PVC in the following chart values. storage.model has no default, so set storage.model. Set the namespace with helm install -n <ns>.

imagePullSecrets:
  - <pull-secret-name>
storage:
  model:
    kind: existingClaim
    existingClaim: model-cache
  compileCache:
    storageClass: <sc-name>  # empty: the cluster's default StorageClass