Serving Configuration¶

Common arguments¶

The Status column records whether each value was exercised in the validation of the Target Configuration.

The three tool-call and reasoning arguments were exercised by the agent workload in Validation and Limits, in which mini-swe-agent makes tool calls over many turns.

Role-specific arguments¶

LMCache environment variables (prefill only)¶

Default configuration: the feature and accuracy validation confirmed the behavior; the effect was not measured. These are configured through environment variables only, without a configuration file; the chart's default values carry them in modelserver.extraServeEnv.prefill.

The RDS capacity is a number of chunks per rank, not a disk size: each rank (engine process) keeps at most rds_max_chunks chunks (default 1000, set by adding rds_max_chunks to LMCACHE_EXTRA_CONFIG) on the device bound to RDS, and evicts the least recently used chunk before storing a new one. At the default and LMCACHE_CHUNK_SIZE 4096, each rank holds up to 4,096,000 tokens (rds_max_chunks × LMCACHE_CHUNK_SIZE) in RDS. Each stored chunk holds 2 MiB of chiplet 0 memory per NVMe drive, so rds_max_chunks × 2 MiB × the number of drives, plus the staging buffer, has to fit in the chiplet 0 memory that --gpu-memory-utilization leaves. Change the two values together. lmcache-rbln older than 0.12.0 does not read rds_max_chunks. The device requirement is in Node preparation › RDS setup.

Speculative decoding (both roles)¶

1
2
3
4
5

{
  "method": "eagle3",
  "model": "asherszhang/MiniMax-M2.7-EAGLE3-draft-vocab200k",
  "num_speculative_tokens": 3
}

The same value goes on both roles: EAGLE-3 needs --speculative-config on prefill too, because the draft head fills its own KV cache during the prefill forward pass.

Router (llm-d EPP/GAIE)¶

The canonical configuration uses prefix-cache-aware routing, with the plugins below. The scorer weights follow the upstream pd-epp-config, which is what the serving chart deploys. Unverified as a whole. Routing chooses among several Pods, so the feature and accuracy validation, with one prefill and one decode, cannot confirm it; the Target Configuration validation did not use this configuration.

The P/D deployments of the feature and accuracy validation used the same plugins and weights as this table.

The router records which Pod it sent a request to; it does not ask whether the engine actually cached it. A repeated prompt therefore skips prefill. For a measurement in which every request must pass through prefill, use always-disagg-pd-decider instead of prefix-based-pd-decider.

The AGG recipes have no prefill Pods and use different plugins: approx-prefix-cache-producer, inflight-load-producer, prefix-cache-affinity-filter, and token-load-scorer in a single scheduling profile. The peakPrefillThroughput of prefix-cache-affinity-filter is 2256 (tokens per second), measured on this model's AGG configuration.

PP and DP exclusivity¶

The entrypoint rejects the combination of pipeline-parallel-size > 1 and data-parallel-size > 1. This constraint effectively determines the per-role configuration: prefill becomes PP4/DP1 and decode becomes DP4/PP1.

Prefill: PP4¶

PP4 splits the 62 layers four ways, so the weights are distributed over four cards and each stage exchanges only activations at layer boundaries.

Decode: DP4, expert parallel enabled¶

Decode generates one token at a time per request, so only a small number of experts activate. DP4 is combined with expert parallel to distribute the 256 experts over four cards. Decode cannot use PP, so without expert parallel every card would need the full set of weights.

Effects of changing the values¶

--block-size 4096¶

Validated.

The KV cache is allocated in blocks of 4,096 tokens. Because sub-block prefix caching is enabled, the unit in which a prefix is reused is smaller than this block; see Prefix reuse unit of 512 tokens.

Prefix reuse unit of 512 tokens¶

Default configuration: the feature and accuracy validation confirmed the behavior; the effect of the sub-block size on performance was not measured.

With sub-block prefix caching (VLLM_RBLN_SUB_BLOCK_CACHE=1), the sub-block size becomes the --max-num-batched-tokens value (when VLLM_RBLN_SUB_BLOCK_SIZE is unset, as in the chart defaults), so the prefix reuse unit is 512 rather than --block-size (4096).

The router's hashing unit depends on autoTune. With autoTune: true (the default, kept by the chart and the recipes), the router replaces approx-prefix-cache-producer.blockSizeTokens and lruCapacityPerServer with the block size and block count in vllm:cache_config_info. vllm-rbln reports --block-size (4096) there even with sub-block prefix caching, so the router hashes prefixes in 4,096-token units and does not track a prefix shorter than one block, even though the engine can reuse it. The blockSizeTokens: 512 and lruCapacityPerServer: 1024 in the chart's values are fallbacks that apply only until the router first reads a Pod's metric; if a later read fails, the router keeps the last value it read. With autoTune: false, the router uses the configured blockSizeTokens, where 512 matches the sub-block unit, and lruCapacityPerServer must be set to the per-Pod block count in that unit. That configuration is Unverified.

KV block count¶

The engine derives the KV block count at startup from --gpu-memory-utilization. Use a smaller --gpu-memory-utilization only when a node reports Not enough memory for N blocks of KV cache or SYS_EBUSY: Lack of device memory during startup.

KV capacity referenced by the router¶

Default configuration: the behavior was confirmed; the effect was not measured.

The Endpoint Picker (EPP) of the llm-d router keeps an index of the prefixes each Pod holds. In approx-prefix-cache-producer, lruCapacityPerServer sets how many entries that index keeps per Pod, counted in units of blockSizeTokens. The canonical configuration sets both values only as fallbacks for the time before the metric is read (blockSizeTokens: 512, lruCapacityPerServer: 1024). autoTune (on by default) reads each Pod's vllm:cache_config_info metric (CacheBlockSize, CacheNumBlocks) and replaces both per Pod; the fallbacks apply only before the router first reads that Pod's metric. With dynamic KV the block count is a runtime answer, so the metrics decide it. Because the derivation is per Pod, prefill and decode Pods each get a capacity that matches their own HBM, even though they differ.

For reference, in 4P4D on the Target Configuration with decode --max-num-seqs 1, the engines reported the HBM capacities below: num_gpu_blocks × --block-size in vllm:cache_config_info, after the dynamic KV resize at startup. The KV cache size line that the engine logs before warm-up is an estimate taken before that resize and reports a larger value.

The EAGLE-3 draft head and its KV cache take part of the HBM, so the capacity is lower than with a method that loads no draft model. A larger decode --max-num-seqs lowers the decode capacity further, because each additional decode graph reserves memory: with EAGLE-3 at --max-num-seqs 8, a decode engine holds 282,624 tokens.

One limit: the metric covers HBM only. The RDS capacity of a prefill Pod (rds_max_chunks × LMCACHE_CHUNK_SIZE tokens per rank) is not reflected in the derived value, so with LMCache/RDS enabled the index undercounts what prefill actually holds: the router misses some prefixes that are still cached in RDS, which lowers the hit rate but produces no error. Unverified, because the Target Configuration validation ran without LMCache.

--kv-cache-dtype fp8¶

The bundled MiniMax-M2.7 and DeepSeek-V3.2 recipes use fp8 for the KV cache, and this design uses it too: the KV cache is stored in 8 bits, and because model weights and activations run as W8A16, the overall configuration is W8A16KV8. The per-token KV cache size of approximately 124 KiB assumes this state, and changing this argument requires recalculating the KV cache size in Coding Assistant on Rebel100™ › KV cache size and the KV capacity referenced by the router.

CPU tier¶

Optional: on nodes where RDS is not available, host memory can serve as the KV offloading tier. Default configuration: the lmcache cpu mode of the feature and accuracy validation confirmed accuracy and cache reuse; the effect was not measured. The KV connector configuration (MultiConnector) is the same as for RDS; change only these values in modelserver.extraServeEnv.prefill.

Keep LMCACHE_ENABLE_KV_EVENTS and LMCACHE_PRE_CACHING_HASH_ALGORITHM at the same values as the RDS configuration. Prefill's --gpu-memory-utilization 0.8 leaves chiplet 0 memory for RDS, so it is not needed when only the CPU tier is used. The CPU tier uses the prefill Pod's host memory, so it has to fit within the Pod's memory limit.