Skip to content

Hardware

Overview

This document describes the hardware requirements for deploying Coding Assistant on Rebel100™, and the rationale for each requirement.

Status badges on this page follow Reference Design › Status badges: a statement without a badge is not Validated.

Category Configuration Status
Target RebelRack™ (4 servers plus switches and auxiliary equipment), 32 NPUs in total, 4P4D (one prefill and one decode instance per node) Validated

The RebelRack™ is the product deployment unit, so the Target Configuration is described and validated on a 4-node basis.

Requirements summary

The design needs the following regardless of the server model. The sections below and Deployment on Kubernetes give the reasons.

Requirement Value Details
NPUs per instance 4 RBLN-CR13 cards, for prefill and decode alike Rationale for four cards per instance
RoCE NICs per instance 2 RoCE NICs on the same NUMA node as the instance's 4 NPUs; the transfer uses one, and the other is the spare the instance falls back to after a NIC failure Network, NIC failure behavior, Device allocation
P/D reachability Every prefill instance can reach every decode instance over RDMA, within and across nodes Network
Local NVMe for Rebellions Direct Storage (RDS) Raw NVMe with no filesystem on the nodes that run the Pods using RDS. In P/D the prefill Pods use RDS, and in AGG the decode Pods use it. A filesystem and RDS cannot share one disk Storage, Node preparation › RDS setup
Shared storage A ReadWriteMany StorageClass for the model weights (250 GB or more) and the compile cache (200Gi per role) when Pods span nodes Storage, Volumes
Node memory and CPU Per modelserver Pod: 630Gi and 40 CPU cores. Each node needs the sum of the Pods placed on it, including the routing sidecar, the EPP, and the gateway, plus headroom Resources and the node budget

Servers

Each node is a RebelServer™, a 5U chassis that holds 8 RBLN-CR13 NPU cards.

Item Value
Form factor 5U
NPU cards 8 per node
CPU 2 x AMD EPYC 9355 (32C/64T, 3.55 GHz, 280 W)
DRAM 1.5 TB (24 x 64 GB DDR5)
Local storage OS 1.92 TB NVMe x 2, data (KV cache) 1.92 TB NVMe x 2
Network 4 x 400G QSFP112-DD 1-port, 1 x 100G QSFP56 2-port, 1 x 10G/25G SFP28 2-port (OCP 3.0)
PCIe generation Gen5 (2 x PCIe Gen5 x16 per card)
Power supply 6 x 2,700 W
Power consumption 4 kW typical, 7 kW maximum
Operating temperature 10 °C to 35 °C

Rack configuration

The Target Configuration is one 42U air-cooled rack holding four of these servers, which is the RebelRack™ NPU rack.

Item Value
Servers 4 x RebelServer™ (5U)
NPU cards 32
CPU 8 x AMD EPYC 9355 (256C/512T)
System memory 6 TB DDR5
NPU memory 4.61 TB total, 153.6 TB/s aggregate HBM bandwidth
Compute 64 PFLOPS (FP8), 32 PFLOPS (FP16)
Backend fabric 400G QSFP112-DD
Cooling Air-cooled, 105,094 BTU/hour maximum
Power consumption 28 kW maximum
Dimensions 42U, 1,991 mm (H) x 600 mm (W) x 1,200 mm (D)

The management rack that accompanies the NPU rack is a separate unit and is not part of the serving capacity described here.

NPU card

RBLN-CR13 carries one Rebel100™ chip.

Item Value
Architecture 4-homogeneous-chiplet SoC based on UCIe-Advanced
Compute (dense) 1,024 TFLOPS (FP16), 2,048 TFLOPS (FP8)
HBM capacity 144 GB (HBM3E)
HBM bandwidth 4.8 TB/s
Chiplet interface (UCIe-A) 16 Gbps, 1 TB/s per channel
Host connection 2 x PCIe Gen5 x16 (64 GB/s + 64 GB/s)
Power consumption Up to 600 W

The four chiplets are connected as a single chip-wide mesh, at 1 TB/s bidirectional per channel and 11 ns of latency, so they behave as one virtual die. A software-controlled predictive DMA engine preloads KV data over that mesh and reaches about 2.7 TB/s of effective bandwidth.

Rationale for four cards per instance

The model determines the number of devices required. The FP8 checkpoint of MiniMax-M2.7 is about 234 GB and the HBM per card is 144 GB, so it cannot be loaded on a single card. Two cards are enough to hold the weights, but the KV cache, the activations, and the compile runtime use the same HBM, so the instance unit was set to 4 cards. With 4 cards the weights drop to about 59 GB per card, and the remaining capacity can be allocated to the KV cache.

The way the weights are distributed across the 4 cards differs by role. Prefill uses PP4 to split the layers into 4 parts, and decode uses DP4 with expert parallel to distribute the experts. For details, see Serving Configuration.

Network

Item Value Status
NIC Four 400 Gbps NICs per node (2 per NUMA node)
Transport RoCE v2
NIC selection nixl-rbln selects one NUMA-local NIC Validated
P/D interconnect Every P/D pair can be connected, including NICs in different subnets Validated

KV transfers run both within and across nodes, so every P/D pair has to be mutually reachable in order for the router to form pairs freely.

Even though there are 4 NICs per node, a single process uses one NUMA-local NIC. The upper bound on KV transfer bandwidth therefore has to be calculated as 400 Gbps per instance, not as the 1.6 Tbps node total. Each instance still claims 2 NUMA-local NICs: the second NIC carries no traffic during normal operation and lets the engine move the transfer to a surviving NIC after a NIC failure without restarting (see NIC failure behavior).

Item Value
KV transfer volume of an ISL 50K request About 6 GB
Theoretical lower bound at 400 Gbps About 130 ms

The theoretical lower bound is calculated from the two values above, using the per-token size in Coding Assistant on Rebel100™ › KV cache size (50,000 tokens × 124 KiB = 5.9 GiB; 5.9 GiB × 8 bit / 400 Gbps ≈ 127 ms), and does not include protocol overhead or transfer startup delay.

NIC failure behavior

After a NIC failure the engine moves the transfer to the second NUMA-local NIC the instance claims, without restarting. What happens to requests during the switch, the measured behavior for each fault, and the link_down_exit_s option are in Failure response › RoCE NIC faults.

Storage

Purpose Requirement Notes
Model weights About 234 GB, 250 GB or more recommended Preloading on a shared volume is recommended
Compile cache A 200Gi ReadWriteMany PVC per role Required for prefill and for decode separately
RDS (KV offload) Raw NVMe on the nodes that run the Pods using RDS (prefill in P/D, decode in AGG). The cache is bounded in chunks (rds_max_chunks x LMCACHE_CHUNK_SIZE; 4,096,000 tokens per rank at the defaults); see Serving Configuration Default configuration

Preloading the weights on the shared volume is recommended; an empty model volume makes startup attempt a download. See Secrets and the model store.

A Pod without RDS still becomes Ready, without any alert; how to detect it is in Serving Configuration › LMCache and RDS.

The compile cache holds the results of the first, cold compilation. The measured times and the conditions under which the cache is reused are in Compile cache.

Sizing guidance

Size a deployment in instances rather than cards. Each prefill or decode instance takes 4 NPUs and 2 NICs (one active, one failover spare), so a RebelServer™ holds two instances and a RebelRack™ holds eight.

Measure before changing the P:D ratio. Watch TTFT, the number of waiting requests on each role, and the prefix-cache hit metrics described in Monitoring, and add capacity to the role that is the bottleneck.

P:D ratio

Configuration Status
4P4D Validated. Default
5P3D, 3P5D Validated

The choice of ratio depends on the ISL/OSL ratio of the workload and on the prefix hit rate. This workload is heavily prefill-weighted at ISL 50K / OSL 1K, but when Automatic Prefix Caching hits, the prefill load decreases significantly, so the ratio cannot be decided from a simple token ratio.

On the validation workload, where 80% of each prompt is a cached prefix, 4P4D is the balanced layout, and the two neighboring layouts each run out of a different resource first as concurrency rises:

  • 5P3D runs out of decode. The decode instances stay nearly full, and requests wait for a free decode slot while prefill has spare capacity.
  • 3P5D runs out of prefill. Decode slots stay partly empty, and requests wait in the prefill queue.
  • 4P4D stays clear of both limits up to concurrency 64. At concurrency 128 its prefill instances saturate as well, and no split of eight instances serves that load; adding instances helps there, and changing the ratio does not.

Without prefix caching, prefill computes the whole prompt and does about three times as much work per request. Prefill then becomes the scarce resource, and a layout with more prefill instances, such as 5P3D, serves the same workload better than 4P4D.

To find which role limits a deployment, compare the queue time on the prefill instances with the queue time on the decode instances (vllm:request_queue_time_seconds). The prefill compute time per request does not grow when prefill saturates; the saturation shows only as queue time.