Skip to content

Failure response

Goal: reach the cause from the symptom by the shortest path.

This page covers failures in deployment and operation: the stack does not start, a value you set is not applied and no error is shown, or a NIC or NPU device fault interrupts serving. This page does not cover failures inside the vLLM engine (compile errors, warm-up failures, timeouts). For engine startup failures, see RBLN vLLM Troubleshooting. For the definitions of runtime error codes, see Error Codes.

Response order

Do not redeploy a failed stack repeatedly. If you redeploy before the previous Pod returns its NPU claim, RDMA queue pair, and port, the new Pod also fails while requesting the same resources. The new failure makes the original cause harder to find. Proceed in the following order.

  1. Find the cause that matches the symptom in the table under Diagnosis by symptom.
  2. Uninstall the existing Helm release so that the previous Pods return the NPU claim, RDMA queue pair, and port.
  3. Fix the cause, then install the Helm release again.

Diagnosis by symptom

The following table maps each symptom to its cause and the check that confirms the cause.

Symptom Cause Check
Permanently Pending The Dynamic Resource Allocation (DRA) ResourceSlice is not published because RBLN NPU Operator is not in DRA mode, the ResourceClaimTemplate (RCT) is absent, or the StorageClass does not match Tell the causes apart by the kubectl describe pod events. An absent RCT is covered in the next row. A StorageClass mismatch shows ProvisioningFailed on the PersistentVolumeClaim (PVC) and FailedScheduling on the Pod. Then run kubectl get resourceslices, get resourceclaimtemplate npu-rdma, and get pvc
FailedResourceClaimCreation that persists for several minutes The RCT is absent, or devices.claimTemplateName names a different RCT Apply the RCT, or set devices.claimTemplateName to the name of the applied RCT
ImagePullBackOff The Image Pull Secret is absent or its name does not match Compare the output of get secret with imagePullSecrets
llm-d Endpoint Picker (EPP) CreateContainerConfigError The Secret llm-d-hf-token is absent kubectl -n <ns> get secret llm-d-hf-token
no matches for kind ... immediately on apply The CustomResourceDefinition (CRD) of one of the following is absent: Kubernetes Gateway API, Kubernetes Gateway API Inference Extension (GAIE, InferencePool), the GatewayClass provider (agentgateway), the VictoriaMetrics Operator (default) or Prometheus Operator Install the missing CRD, or disable the component that uses the CRD
Install succeeds but the Gateway stays Accepted=Unknown (Waiting for controller) No GatewayClass provider is running Install the provider: Istio or agentgateway (see Cluster foundation › GatewayClass provider)
HTTPRoute ResolvedRefs=False on Istio GAIE is not enabled in Istio Reinstall Istio with GAIE enabled, as in llm-d's Istio guide
Requests fail and the modelserver log carries SYS_EIO / SYS_TASK_ABORTED An NPU or Rebellions Direct Storage (RDS) device fault is possible. The error name alone cannot tell which device failed Follow the fw.log procedure in NPU and RDS device faults
LoadBalancer <pending> No load balancer address is available to the Gateway Service: the address pool is exhausted or not allocated to this namespace (AllocationFailed … no available IPs) With the default GatewayClass (Istio), set networking.istio.io/service-type: ClusterIP in gateway.annotations to switch the Service type. On agentgateway, set gateway.agentgateway.parameters.enabled: true and gateway.agentgateway.parameters.service.spec.type: ClusterIP
Pending with the event cannot allocate all claims while the NPU ResourceSlice is published The RDMA driver has not recognized the NIC on the node where the Pod should run. The reference RCT (numa-aligned.yaml) requests only NICs that can use RDMA Check ls /sys/class/infiniband/ on the node
CrashLoopBackOff (rbln-ccl / RDMA) The new Pod contends with the processes, RDMA queue pairs, or ports that the previous Pod left behind Uninstall the Helm release, install the Helm release again after the previous Pods are gone, and check whether the error recurs
Not enough memory for N blocks of KV cache or SYS_EBUSY: Lack of device memory during startup The block count the engine derived at startup does not fit the device memory of the node Lower --gpu-memory-utilization through modelserver.extraServeArgs.<role> (the later flag wins)
--set not taking effect The --set key has a typo inside a free-form map (serveEnv, podOverrides.<role>, gateway.annotations, and the like) or under the subchart (llm-d-router-gateway.*). The schema does not check these locations, so the chart ignores the typo without an error. A typo in a key that the schema defines fails the render instead Compare the output of helm get values with the keys you set

Right after a DRA driver is installed, the Pod may still be waiting for the driver to publish its ResourceSlice.

If a Pod is scheduled onto a node but does not become Ready, check which probe the Pod fails. For the cause and the check for each probe, see Startup and readiness probes.

Failure impact by P/D role

The following table lists what a stop of a prefill Pod or a decode Pod affects and how the Pod recovers.

Situation Impact Recovery
One prefill Pod stops The EPP sends new requests to the remaining prefill Pods, which raises the time to first token (TTFT). With no prefill Pod left, requests do not fail: decode runs the prefill itself, and P/D disaggregation is lost until a prefill Pod is Ready again The Deployment recreates the Pod. For a device fault, follow NPU and RDS device faults instead. If the NPU is not returned, the new Pod stays Pending (see Diagnosis by symptom)
One decode Pod stops Token generation stops for the requests that the stopped decode Pod was handling. Requests being handled by other decode Pods are unaffected. The EPP sends new requests to the remaining decode Pods Same as the prefill row

When a prefill Pod stops, its in-flight requests can fail their key-value (KV) cache transfer. Prefill keeps the KV it computed in high-bandwidth memory (HBM), so KV that decode has not yet fetched disappears with the prefill Pod. For the behavior on transfer failure, see P/D transfer failure behavior.

When a Pod restarts, it uses the compile results stored in the compile-cache PVC, so it does not compile the model again. However, the restarted Pod does not reuse the KV cache stored before the restart, in HBM or in tier 2 (RDS). The restarted Pod therefore computes the prefill from the start, even for a prompt it processed before the restart.

P/D transfer failure behavior

Decode fetches the KV that prefill produced over RDMA READ. When that transfer fails, the behavior depends on the kv_load_failure_policy value of decode.

kv_load_failure_policy Request whose transfer failed
fail (default) Ends with an error
recompute Decode recomputes the prefill. The client sees a slower response instead of an error

The chart does not set kv_load_failure_policy. The upstream default fail therefore applies.

To change kv_load_failure_policy to recompute, add "kv_load_failure_policy":"recompute" to the --kv-transfer-config JSON of decode.

For the definition of each kv_load_failure_policy value, see vLLM NIXL Connector Usage and the Fault Tolerance section of llm-d Disaggregated Serving: Operations (vLLM).

RoCE NIC faults

In the reference configuration, each prefill Pod and each decode Pod is allocated 2 RoCE NICs (Device allocation). The engine uses one NIC for the KV transfer and keeps the other NIC as the spare. The following table shows the request results and the recovery measured after taking NIC links down on a stack installed with the chart defaults. This was measured apart from the Target Configuration validation.

Fault Requests in flight New requests Recovery
The link of the NIC in use is down Fail Wait or fail until the engine switches to the spare NIC, then succeed. Requests succeed again after about 30 seconds for a decode NIC and about a minute and a half for a prefill NIC Automatic. No restart
Every NIC link of the Pod is down Fail Most end with an error (500) Transfers resume automatically when one link comes back. No restart

When every NIC link is down, decode logs one of the following messages.

  • Every decode NIC is down: every local RDMA link is down
  • Every prefill NIC is down: engine <id> reports every RDMA link down

To restart the prefill container when every local RDMA link is down for a set time, set link_down_exit_s. The chart does not set link_down_exit_s, and with the default of 0, prefill never exits.

  1. In the prefill RblnNixlConnector entry of --kv-transfer-config, change kv_role to kv_producer. The chart default is kv_both, and the engine refuses to start when link_down_exit_s is set on any other kv_role.
  2. Add link_down_exit_s (seconds) to kv_connector_extra_config of the same entry.

When the set time passes, the prefill process exits. The Pod is kept and only the container restarts.

NPU and RDS device faults

This stack does not recover from an NPU or RDS device fault on its own. A container restart does not guarantee that the device recovers. Confirm the fault, then move the Pods off the faulted node.

Confirming the fault

Confirm the fault from logs, not from the Pod status. A device fault shows one of the following symptoms.

  • The modelserver log records a runtime error such as SYS_EIO or SYS_TASK_ABORTED (Error Codes).
  • Requests do not fail, but the KV cache stored in tier 2 (RDS) is not reused. LMCache treats a failed RDS lookup as a cache miss.

The same error names also cover NPU and memory errors, so an error name alone cannot identify an RDS fault. Confirm an RDS fault with fw.log in the coredump directory on the node.

  • When a device is reset, RSMD (rbln-smd), the NPU system management daemon, creates the diagnostic directory /var/log/rebellions/<device>.<time>.coredump/ on the node. This page calls that directory a coredump.
  • The fw.log file in the coredump holds the NPU firmware log (RBLN System Management Daemon).
  • An RDS command failure that aborts the NPU work (err_code 0x10015) resets the device, so it leaves a coredump. A fault that does not lead to a device reset leaves no coredump.
  • The meta file in the coredump records the situation that created the coredump as a situation value. When the device reset fails, situation = RESET_FAIL is recorded, and that coredump may have no fw.log. In that case, check fw.log in a coredump with situation = RESET.
1
2
3
# on the node
ls -dt /var/log/rebellions/*/ | head -5
grep -nE '\[rds\]|NVMe command failed|guilty ctx|err_code 0x10015' <coredump-dir>/fw.log

When an RDS command failure aborts the NPU work, fw.log contains the following log. err_code 0x10015 in the last line is the signal of an RDS fault.

[cl0-cp0 ...] ERR  /0 0 [rds]   -> Command failed: cid=17 status=0x0281
[cl0-cp0 ...] ERR  /0 0 [rds] rblnfs_wait_batch: qid=0 NVMe command failed after 0/1 completions
[cl0-cp0 ...] INFO /0 0 hils_set_guilty_ctx: guilty ctx(0x80000001)
[cl0-cp0 ...] ERR  /0 0 hils_set_err_reason: err_code 0x10015

If there is no coredump directory, check the node kernel log for rblnfs and nvme messages. Without shell access to the node, start a debug Pod with kubectl debug node/<node> and run the same commands.

1
2
3
# on the node
dmesg -T | grep -iE 'rblnfs|nvme'
journalctl -k | grep -iE 'rblnfs|nvme'

Moving the Pods

Move the Pods off the faulted node in the following order.

  1. Preserve the evidence. Recreating a Pod discards the Pod log. Copy the Pod log (kubectl -n <ns> logs <pod>, adding --previous when the container has restarted), the node kernel log, and the coredump directory.
  2. Keep new Pods off the node. Run kubectl cordon <node>.
  3. Move the Pods. Run kubectl -n <ns> delete pod <pod>. Another node needs spare NPU and RoCE NIC capacity. In nodePlacement mode, change the node of the nodePlacement entry that names the faulted node, then uninstall and install (Scaling and Upgrades › Change rules).
  4. Replace the device, then allow Pods on the node again. Run kubectl uncordon <node>.

Serving with RDS disabled

If no other node can take the Pods, you can keep serving with RDS disabled. The stack then uses tier 1 (HBM) only. To disable RDS, set LMCACHE_STORAGE_PLUGINS and LMCACHE_EXTRA_CONFIG to null in the following key.

Install Key
Default values, P/D recipes (examples/models/<model>/pdd.yaml) modelserver.extraServeEnv.prefill
AGG recipes (examples/models/<model>/agg.yaml) modelserver.extraServeEnv.decode

For example, with the default values, run helm upgrade with the following options. The Deployment recreates only the Pods of the P/D role whose extraServeEnv changed.

1
2
--set modelserver.extraServeEnv.prefill.LMCACHE_STORAGE_PLUGINS=null \
--set modelserver.extraServeEnv.prefill.LMCACHE_EXTRA_CONFIG=null