Coding Assistant on Rebel100™¶

Overview¶

This reference design describes a serving stack for coding agents on Rebel100™: MiniMax-M2.7 served by RBLN vLLM and deployed on Kubernetes with llm-d, using prefill/decode (P/D) disaggregation, prefix-cache-aware routing, KV offload to Rebellions Direct Storage (RDS), and speculative decoding. It fixes one configuration end to end, from the hardware and software versions to the Kubernetes resources and serving arguments, and explains why each value was chosen, so that you can reproduce the design and use it as the starting point for your own deployment. It states what was validated and what was not, and describes how the Target Configuration was validated (see Validation approach).

Target audience¶

• AI engineers who tune the modelserver. Start with Serving Configuration for the parallelism, KV cache, and speculative decoding arguments and the effect of changing them.
• System administrators who build and operate the cluster. Start with Hardware, Deployment on Kubernetes, and Operations.
• Developers who connect a coding agent to the endpoint. Start with Client Integration.

Customer problem¶

A coding agent does not use a model the way a chat client does. It sends prompts of tens of thousands of tokens built from the same repository context, receives short answers, and calls the model again after every tool call, so a single task becomes a chain of requests whose latencies add up. A modelserver behind a plain load balancer serves this pattern poorly: long prefills delay token generation for the other requests on the same server, the load balancer does not know which server already holds a prompt's prefix, and KV cache that could be reused is evicted from device memory.

The workload characteristics below drive the choices in this design.

Design goals¶

• Latency first. Keep per-request latency low instead of raising the batch size for aggregate throughput.
• Isolate prefill from decode. Run the two phases on separate instances so that long prompts do not interrupt token generation, and move the KV between them over RDMA.
• Reuse the prompt prefix. Route each request to an instance that already holds its prefix, and keep KV beyond device memory in tier 2 (RDS).
• Speed up generation. Use EAGLE-3 speculative decoding by default, with a published draft head for this model; suffix decoding and n-gram need no draft model and are validated alternatives.
• Deploy with one chart on standard Kubernetes APIs. Bring up the gateway, the router, and the modelservers with a single Helm chart, using DRA for devices and the Gateway API for routing.
• State the validation boundary. Validate on the four-node Target Configuration itself, and give each component a status that says whether it was exercised there.

Authentication and authorization of the endpoint are outside the scope of this design and belong in front of the gateway (see Client Integration).

Architecture¶

A request passes through the gateway to the Endpoint Picker (EPP) of the llm-d router, which picks a prefill and a decode instance by prefix-cache affinity and load. The prefill instance computes the KV of the prompt, and the decode instance pulls that KV and generates the output tokens.

flowchart TB
  C["Client or coding agent"] --> GW["Gateway"]
  GW --> EPP["Router (EPP)<br/>picks prefill and decode"]
  EPP --> SC
  subgraph DPOD["Decode pod"]
    SC["Routing sidecar"]
    D["Decode instance<br/>DP4 + expert parallel, 4 NPUs"]
  end
  SC -->|"1. prefill"| P["Prefill instance<br/>PP4, 4 NPUs"]
  D -->|"2. pulls KV over NIXL (RDMA)"| P
  SC -->|"3. decode"| D
  P <-->|"KV tier 2 (LMCache)"| RDS[("RDS on NVMe")]

1. The routing sidecar in front of the decode instance first sends the request to the selected prefill instance.
2. The decode instance pulls the prompt's KV from the prefill instance over NIXL with RDMA. Prefill also stores KV in RDS through LMCache and looks it up on later requests.
3. The decode instance generates the output tokens, and the response returns through the routing sidecar and the gateway.

The router predicts prefix-cache hits from the requests it has already sent to each Pod. In the Target Configuration (4P4D) each node runs one prefill and one decode instance, and any prefill instance can pair with any decode instance across nodes (see Hardware › Network). For the Kubernetes resources behind each box, see Deployment on Kubernetes.

Model¶

MiniMaxAI/MiniMax-M2.7 is an MoE model with 229B total parameters and roughly 10B active parameters.

The recommended sampling parameters are in Client Integration › Recommended sampling parameters.

Configuration summary¶

The status badges used below follow the vocabulary defined in Reference Design › Status badges.

The validation exercised the serving arguments on Serving Configuration with two exceptions: KV moved from prefill to decode over NIXL without LMCache and RDS, and the P/D layout comparison spread requests round-robin without the router, which ran only for the agent workload and with a configuration that differs from Serving Configuration › Router (see Target Configuration validation). The parts of the canonical configuration that the validation did not exercise carry Unverified on the page that defines them, and are listed in Validation scope and limits.

Design alternatives¶

The design fixes one path for each of the choices below. The alternatives are supported, but they change what you have to prepare or what you give up.

Validation approach¶

Evaluate the design on your own workload before sizing a deployment.

The Target Configuration was validated with a synthetic workload based on SWE-bench Verified that reproduces the shape of coding-agent traffic; the workload, the P/D layouts, the concurrency range, the functional checks, and the accuracy results are in Validation and Limits.

Validation scope and limits¶

The table below lists the main items the Target Configuration validation did not use. It is not exhaustive; each page marks the status of its own items too.

Next steps¶