Troubleshooting KV Cache Block Estimation

When kvcache_num_blocks is left unset, Optimum RBLN sizes the paged-attention KV cache after compilation: it grows the block pool until the compiled model fills the NPU. This guide covers how to bound that estimate with memory_budget, using the Qwen3-8B Model Zoo example as the reference. The same option applies to any decoder-only model compiled through Optimum RBLN.

Prerequisites

The measurements below come from Qwen3-8B compiled for 4 ATOM™-Max (RBLN-CA25) with the Model Zoo settings max_seq_len=40_960 and kvcache_partition_len=8_192, at batch_size=8. Block counts depend on the model, the compile configuration, and the NPU, so read them as the shape of the trade-off rather than as values to copy.

Run rbln-smi to list the NPUs on the host, and adjust num_devices to match your hardware.

Quick reference

Symptom	Root cause	Go to
The compiled model reserves the whole NPU, leaving no room for anything else on it	Automatic block estimation grows the KV cache until the NPU is full	Step 1
Insufficient memory for the required KV cache after a budget is set	The budget starves the estimate below the minimum block count the configuration needs	Step 2
memory_budget ... exceeds the target NPU's available DRAM	The budget is larger than the DRAM the NPU actually offers	Step 3
memory_budget and an explicit kvcache_num_blocks are mutually exclusive	Both were set, and estimation runs only when the block count is unset	Step 3

---

Step 1: Bound the automatic KV cache estimate

Symptom

The model compiles and runs correctly on its own, but the compiled artifact occupies the entire NPU. Nothing else fits on the same NPU — a second model, another submodule of the same pipeline, or a later increase in batch size.

Root cause

kvcache_num_blocks defaults to 0, which means "decide after compilation". The estimator then searches for the largest block pool that still fits, bounded only by the DRAM the NPU reports. It subtracts the allocations of the compilation it runs in and hands the rest to the KV cache, so by construction the result leaves no headroom.

For the reference configuration that search ends at the full pool of 40 blocks, which reserves 11.25 GB of KV cache on each of the 4 NPUs.

Resolution

Set memory_budget to cap the DRAM the estimate may assume:

from optimum.rbln import RBLNQwen3ForCausalLM

model = RBLNQwen3ForCausalLM.from_pretrained(
    "Qwen/Qwen3-8B",
    export=True,
    rbln_batch_size=8,
    rbln_max_seq_len=40_960,
    rbln_num_devices=4,
    rbln_kvcache_partition_len=8_192,
    rbln_memory_budget=0.5,
    rbln_create_runtimes=False,
)
model.save_pretrained("rbln-Qwen3-8B")

The budget applies per NPU and resolves against the DRAM that NPU actually offers. ATOM™-Max (RBLN-CA25) offers 16,877,879,296 bytes (15.72 GB) once the system reserve is set aside, which is what rbln-smi reports as its total memory.

The budget accepts three forms:

Form	Example	Meaning
Fraction	0.5	Half of the available DRAM. A float in the range (0, 1], and the recommended form.
Percent string	"50%"	Identical to 0.5. Convenient when the value passes through a command line.
Byte count	"8GB", "512MB", 8 * 2**30	An absolute budget. Units are binary (KB = 1024 bytes, up to TB) and case-insensitive; a bare int is a byte count.

The budget covers everything the compiled model loads onto the NPU, not the KV cache alone. Weights and workspace are charged against it first, and the KV cache takes whatever remains — which is why the KV cache block count falls faster than the budget itself. Varying only memory_budget in the example above:

memory_budget	Estimated kvcache_num_blocks	KV cache per NPU
unset	40	11.25 GB
0.9	37	10.41 GB
0.8	31	8.72 GB
"12GB"	29	8.16 GB
0.7	25	7.03 GB
0.6	20	5.62 GB
"8GB"	15	4.22 GB
0.5	14	3.94 GB
0.4	9	2.53 GB

Confirm the resolved count in the saved artifact:

$ python -c "import json; print(json.load(open('rbln-Qwen3-8B/rbln_config.json'))['kvcache_num_blocks'])"
14

If the value is 0, estimation did not run — the model was compiled with an explicit block count instead (see Step 3).

The same option is available from the compile CLI, where all three forms work:

optimum-rbln-cli --model-id Qwen/Qwen3-8B --output-dir ./rbln-Qwen3-8B \
  --batch_size 8 --max_seq_len 40960 --num_devices 4 \
  --kvcache_partition_len 8192 --memory-budget 0.5

---

Step 2: Keep the budget above the minimum block count

Symptom

Setting the budget too low makes compilation fail:

ValueError: Insufficient memory for the required KV cache: only 3 blocks can be allocated,
but at least 9 blocks are required. Consider reducing `max_seq_len` or `batch_size`
(currently 8) to lower the memory requirement.

Root cause

A block pool below a certain size can neither hold one full sequence nor give every item in the batch a block of its own. Optimum RBLN therefore rejects the estimate instead of settling for a smaller pool. Under flash attention the floor and the full pool are:

num_min_blocks  =  min(max((max_seq_len / kvcache_block_size), batch_size) + 1, num_full_blocks)
num_full_blocks =  (max_seq_len / kvcache_block_size) × batch_size

The + 1 is the spare block that fills the unused slots of a block table while a sequence is still growing, and it is needed whichever of the two terms sets the floor.

kvcache_block_size is kvcache_partition_len here, so the sequence term of the reference configuration is 5 blocks, batch_size=8 raises it to 8, and the floor is 9 against a full pool of 40. 0.4 lands exactly on the floor at 9 blocks; 0.3 estimates 3 blocks and is rejected with the error above.

The room to tighten the budget therefore comes from the gap between the two values, and batch_size is what opens that gap: the full pool grows by one full sequence per batch item, while the floor follows only the larger of the two terms. At batch_size=1 the same configuration has a full pool of 5 blocks and a floor of 5, so the two coincide and no budget below the full requirement can succeed.

Resolution

Raise the budget until the estimate clears the floor, then verify the resolved count as in Step 1. When the workload needs a tighter budget than the floor allows, shrink the per-sequence reservation instead, since lowering max_seq_len lowers the floor along with it:

Change	Effect on the budget range
Raise batch_size	Widens the gap between floor and full pool, so the budget has more room
Lower max_seq_len	Lowers both the floor and the full pool, so the same DRAM still holds a sequence
Lower kvcache_partition_len	Raises the sequence term of the floor, but each block is smaller

---

Step 3: Resolve budget configuration errors

Symptom

Either of two validation errors:

ValueError: memory_budget (107374182400 bytes) exceeds the target NPU's available DRAM (16877879296 bytes).

ValueError: `memory_budget` and an explicit `kvcache_num_blocks` are mutually exclusive. `memory_budget` only guides automatic block estimation, which runs when `kvcache_num_blocks` is unset.

Root cause

The first error compares a byte count against the DRAM the target NPU offers, which is 16,877,879,296 bytes (15.72 GB) on the reference NPU as shown in Step 1. A value above that is rejected rather than silently clamped, because it asks for capacity the device does not have.

The second error guards the estimation path. Automatic estimation is what consumes memory_budget, and it runs only while kvcache_num_blocks is unset. Setting both would skip estimation and discard the budget without a word, so the combination is rejected at configuration time.

Resolution

Express the budget as a fraction rather than a byte count when the artifact is meant to be portable, since a fraction resolves against whichever NPU it is compiled for. Keep an absolute byte count below the available DRAM of the target NPU.

Choose one of the two controls, not both:

Goal	Set
Let the estimate decide the block count within a memory bound	memory_budget
Pin the block count exactly	kvcache_num_blocks

Note

memory_budget is a compile-time input and is not written to rbln_config.json. The artifact records only the resolved kvcache_num_blocks, which is what inference reads. Recompile to change the memory_budget.

Related resources

• Optimum RBLN overview
• Multi-Module Models
• Model API reference
• RBLN Model Zoo — Qwen3-8B