Startup and readiness probes¶

Goal: know when a Pod is judged Ready and how long its startup may take.

For the meaning and behavior of probes, see the Kubernetes documentation on Liveness, Readiness and Startup Probes. This page covers the following.

The chart's probe defaults and how to tune them
The Deployment progress deadline derived from the startup budget
The commands that show a failing probe
The termination grace period

Chart defaults and tuning¶

modelserver.probes.{startup,readiness,liveness} tunes the probes of the modelserver container. The chart defaults of the three probes are as follows.

Probe	Path	Period (s)	Failure threshold (count)	Timeout (s)
`startupProbe`	`/v1/models`	30	240	10
`readinessProbe`	`/v1/models`	15	4	5
`livenessProbe`	`/health`	30	10	10

The time that startupProbe allows for a Pod's startup is the startup budget. With the chart defaults, the startup budget is 30 seconds × 240 = 2 hours. Model loading and the first compile must finish within the startup budget.

If compilation takes longer than the startup budget, raise modelserver.probes.startup.failureThreshold. The engine itself waits for startup for at most VLLM_ENGINE_READY_TIMEOUT_S seconds (modelserver.serveEnv, 9600 by default); if you set the startup budget above that, raise VLLM_ENGINE_READY_TIMEOUT_S as well.
For the measured first compile time, see First startup time.
For how the modelserver container's CPU limit affects compile time, see Resources and the node budget.

Deployment progress deadline¶

The chart sets the progressDeadlineSeconds of every modelserver Deployment to the startup budget.

Formula: initialDelaySeconds + periodSeconds × failureThreshold (the values of modelserver.probes.startup)
Chart default: 0 + 30 × 240 = 7200 (2 hours)
Tuning: changing modelserver.probes.startup changes progressDeadlineSeconds too. No separate values key sets progressDeadlineSeconds.

If progressDeadlineSeconds were left at the Kubernetes default of 600 seconds, a Deployment that is still compiling normally would be marked Progressing=False (ProgressDeadlineExceeded) after 10 minutes.

Check the current value with the following command.

kubectl -n <ns> get deploy <release-name>-decode -o jsonpath='{.spec.progressDeadlineSeconds}{"\n"}'   # 7200

Probe checks¶

If a Pod does not become Ready, check it with the following commands.

kubectl -n <ns> get events --field-selector involvedObject.name=<pod>   # Shows the failing probe
kubectl -n <ns> get pods -l 'llm-d.ai/role' -w                        # Shows when each Pod becomes Ready

If a starting Pod restarts repeatedly, identify the cause of the restarts from the kubectl describe pod output as follows.

Symptom	Cause	Action
The Pod's events show `Startup probe failed`	The Pod's startup exceeded the startup budget	Raise `modelserver.probes.startup.failureThreshold`
`Last State: Terminated` shows an exit code, with no `Startup probe failed` event	The container exited on its own. For example, the entrypoint exits when it cannot raise the container's memlock limit to `unlimited`	Read the previous container's log with `kubectl logs --previous`

Termination grace period¶

The chart does not set terminationGracePeriodSeconds. The Kubernetes default of 30 seconds therefore applies.

To change the grace period, set terminationGracePeriodSeconds in modelserver.podOverrides.<role>. The chart's examples/pod-overrides.yaml has an example.

When a Pod terminates, the requests that the Pod was handling fail with a 5xx error.