Software Stack and Versions¶

Request routing¶

The GatewayClass provider is Istio by default, or agentgateway; it implements the Gateway, HTTPRoute, and InferencePool resources of the Gateway API and Gateway API Inference Extension (GAIE) CRDs. The Endpoint Picker (EPP) of the llm-d router picks a prefill and a decode instance for each request by prefix-cache affinity and load, and a routing sidecar in the decode Pod sends the request to prefill first. The platform installs the CRDs and the provider; the serving chart brings up the EPP and the routing sidecar.

Model serving¶

RBLN vLLM (vllm-rbln) runs vLLM on RBLN NPUs. It serves MiniMax-M2.7 in both the prefill and the decode instances and exposes the OpenAI-compatible API described in Client Integration. Two plugins handle the KV cache: lmcache-rbln is the RBLN extension of LMCache, which offloads KV from NPU device memory to host memory or to NVMe (KV Cache Offloading); in this design it runs on prefill only and offloads KV to Rebellions Direct Storage (RDS) as tier 2, and nixl-rbln is the RBLN backend for NIXL (Disaggregated Prefill / Decode), which moves the KV produced by prefill to decode over RDMA through a NUMA-local NIC. All three ship in the modelserver image.

Runtime and compiler¶

RBLN PyTorch (torch-rbln) is the PyTorch extension for RBLN NPUs, and the RBLN Compiler (rebel-compiler) compiles the model graphs. The compiler output is kept on the compile-cache volume and is reused only when the compiler version matches; see Compile cache. Both ship in the modelserver image.

Device allocation¶

The RBLN NPU Operator publishes the NPUs as DRA devices. dranet publishes the RoCE NICs as DRA devices. For how the two are deployed, see Node preparation; for how the devices are allocated to Pods, see Device allocation.

Node¶

The RBLN Driver is the kernel driver for RBLN NPUs. The RoCE NIC driver is the NIC vendor's driver in the host OS. For how the drivers are deployed and how RDS is set up, see Node preparation.

Observability is optional and off by default. With monitoring.enabled, the VictoriaMetrics Operator scrapes the metrics by default. Prometheus Operator is an option. With tracing.enabled, an OpenTelemetry collector receives the traces. See Monitoring.

Component interactions¶

The request path, prefix-cache prediction, KV transfer through nixl-rbln, and KV offload through lmcache-rbln are described in Coding Assistant on Rebel100™ › Architecture. Device publication is the one interaction outside the request path:

• Device publication: the RBLN NPU Operator publishes the NPUs and dranet publishes the RoCE NICs as DRA devices (see Device allocation).

Model server image¶

The model server image rbln-serve carries RBLN SDK 0.12.0, so its tag pins every SDK package the prefill and decode Pods run. Each tab below lists the Python and RBLN SDK package versions inside that image; vLLM, LMCache, and NIXL inside it are at the versions in the table above.

The tag reads <SDK_VERSION>-<OS>-py<PYTHON_VERSION>. The validation ran 0.12.0-ubi10.2-py3.12, and the image is published for the base operating systems below. To use another one, set modelserver.image.tag to its tag.

1

docker pull repo.rebellions.ai/rebellions/rbln-serve:0.12.0-ubi10.2-py3.12

1

docker pull repo.rebellions.ai/rebellions/rbln-serve:0.12.0-ubi10.0-py3.12

1

docker pull repo.rebellions.ai/rebellions/rbln-serve:0.12.0-ubi9.8-py3.12

1

docker pull repo.rebellions.ai/rebellions/rbln-serve:0.12.0-ubi9.6-py3.12

1

docker pull repo.rebellions.ai/rebellions/rbln-serve:0.12.0-ubuntu24.04-py3.12

1

docker pull repo.rebellions.ai/rebellions/rbln-serve:0.12.0-ubuntu22.04-py3.10

1

docker pull repo.rebellions.ai/rebellions/rbln-serve:0.12.0-alma9.8-py3.12

1

docker pull repo.rebellions.ai/rebellions/rbln-serve:0.12.0-rocky9.8-py3.12