Troubleshooting KV Cache Block Estimation¶

Quick reference¶

Symptom¶

The model compiles and runs correctly on its own, but the compiled artifact can occupy the entire NPU. Nothing else fits on the same NPU, including a second model, another submodule of the same pipeline, or a later increase in batch size.

Root cause¶

kvcache_num_blocks defaults to 0, which means "decide after compilation". The estimator subtracts the allocations of the compilation it runs in from the DRAM the NPU reports, then checks whether the full pool (num_full_blocks, see Step 2) fits in the rest. If it does, it takes the full pool; if not, it searches for the largest pool that still fits. It never goes above the full pool, but either way the KV cache takes as much of the remaining DRAM as it can use, and nothing reserves headroom for anything else.

Resolution¶

Set memory_budget to cap the DRAM the estimate may assume. The budget applies per NPU and resolves against the DRAM that NPU actually offers.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

from optimum.rbln import RBLNQwen3ForCausalLM

model = RBLNQwen3ForCausalLM.from_pretrained(
    "Qwen/Qwen3-8B",
    export=True,
    rbln_batch_size=8,
    rbln_max_seq_len=40_960,
    rbln_num_devices=1,
    rbln_kvcache_partition_len=8_192,
    rbln_memory_budget=0.3,
    rbln_create_runtimes=False,
)
model.save_pretrained("rbln-Qwen3-8B")

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

from optimum.rbln import RBLNQwen3ForCausalLM

model = RBLNQwen3ForCausalLM.from_pretrained(
    "Qwen/Qwen3-8B",
    export=True,
    rbln_batch_size=8,
    rbln_max_seq_len=40_960,
    rbln_num_devices=4,
    rbln_kvcache_partition_len=8_192,
    rbln_memory_budget=0.5,
    rbln_create_runtimes=False,
)
model.save_pretrained("rbln-Qwen3-8B")

The budget accepts three forms:

The budget covers everything the compiled model loads onto the NPU, not the KV cache alone. Weights and workspace are charged against it first, and the KV cache takes whatever remains. This is why the KV cache block count falls faster than the budget itself.

1
2

$ python -c "import json; print(json.load(open('rbln-Qwen3-8B/rbln_config.json'))['kvcache_num_blocks'])"
21

1
2

$ python -c "import json; print(json.load(open('rbln-Qwen3-8B/rbln_config.json'))['kvcache_num_blocks'])"
14

A compiled artifact never records 0: an explicit kvcache_num_blocks is saved as given, and estimation writes the count it resolved. Under eager attention an unset count is saved as the full pool, whatever memory_budget says.

The same option is available from the compile CLI, where all three forms work:

1
2
3

optimum-rbln-cli --model-id Qwen/Qwen3-8B --output-dir ./rbln-Qwen3-8B \
  --batch_size 8 --max_seq_len 40960 --num_devices 1 \
  --kvcache_partition_len 8192 --memory-budget 0.3

1
2
3

optimum-rbln-cli --model-id Qwen/Qwen3-8B --output-dir ./rbln-Qwen3-8B \
  --batch_size 8 --max_seq_len 40960 --num_devices 4 \
  --kvcache_partition_len 8192 --memory-budget 0.5

Symptom¶

Setting the budget too low makes compilation fail:

1
2
3

ValueError: Insufficient memory for the required KV cache: only 8 blocks can be allocated,
but at least 9 blocks are required. Consider reducing `max_seq_len` or `batch_size`
(currently 8) to lower the memory requirement.

1
2
3

ValueError: Insufficient memory for the required KV cache: only 3 blocks can be allocated,
but at least 9 blocks are required. Consider reducing `max_seq_len` or `batch_size`
(currently 8) to lower the memory requirement.

Root cause¶

A block pool below a certain size can neither hold one full sequence nor give every item in the batch a block of its own. RBLN Optimum therefore rejects the estimate instead of settling for a smaller pool. Under flash attention the floor and the full pool are:

1
2

num_min_blocks  =  min(max((max_seq_len / kvcache_block_size), batch_size) + 1, num_full_blocks)
num_full_blocks =  (max_seq_len / kvcache_block_size) × batch_size

The + 1 is the spare block that fills the unused slots of a block table while a sequence is still growing, and it is needed whichever of the two terms sets the floor.

In the reference configurations:

The room to tighten the budget therefore comes from the gap between the two values, and batch_size is what opens that gap: the full pool grows by one full sequence per batch item, while the floor follows only the larger of the two terms. At batch_size=1 the same configuration has a full pool of 5 blocks and a floor of 5, so the two coincide and no budget below the full requirement can succeed.

Resolution¶

Raise the budget until the estimate clears the floor, then verify the resolved count as in Step 1. When the workload needs a tighter budget than the floor allows, shrink the per-sequence reservation instead, since lowering max_seq_len lowers the floor along with it:

Symptom¶

Either of two validation errors:

1

ValueError: memory_budget (1099511627776 bytes) exceeds the target NPU's available DRAM (150323855360 bytes).

1

ValueError: memory_budget (107374182400 bytes) exceeds the target NPU's available DRAM (16877879296 bytes).

1

ValueError: `memory_budget` and an explicit `kvcache_num_blocks` are mutually exclusive. `memory_budget` only guides automatic block estimation, which runs when `kvcache_num_blocks` is unset.

Root cause¶

The first error compares a byte count against the DRAM the target NPU offers, 150,323,855,360 bytes (140 GiB) on Rebel100™ and 16,877,879,296 bytes (15.72 GB) on ATOM™-Max, as shown in Step 1. A value above that is rejected rather than silently clamped, because it asks for capacity the device does not have.

The second error guards the estimation path. Automatic estimation is what consumes memory_budget, and it runs only while kvcache_num_blocks is unset. Setting both would skip estimation and silently ignore the budget, so the combination is rejected at configuration time.

Resolution¶

Express the budget as a fraction rather than a byte count when the artifact is meant to be portable, since a fraction resolves against whichever NPU it is compiled for. Keep an absolute byte count below the available DRAM of the target NPU.

Choose one of the two controls, not both: