KV Cache Offloading¶

Overview¶

Prefix caching reuses a KV cache only while it stays in NPU device memory. Device memory is the smallest tier available, so a prefix that is evicted must be recomputed when a later request contains the same prefix.

KV cache offloading stores reusable KV chunks in a larger, slower tier while requests are processed. After the NPU cache evicts a prefix, a later request can restore its KV from that tier instead of recomputing it. RBLN vLLM supports this through LMCache and its RBLN extension, lmcache-rbln.

Architecture¶

lmcache-rbln is a hardware-specific extension of LMCache for RBLN NPUs. It depends on the upstream lmcache package for cache management (chunking, lookup, and eviction policy) and replaces the device-specific components: the KV cache transfer between NPU memory and the offload tier, and the storage backend for NVMe direct storage.

RBLN vLLM registers the extension's connector, RBLNLMCacheConnectorV1, with its KV connector factory, so it is selected by name through --kv-transfer-config like any other vLLM KV connector.

Prerequisites¶

Check the packages in that environment:

1
2

python -m pip show vllm-rbln lmcache-rbln lmcache torch-rbln rebel-compiler
python -m pip check

The connector runs on the vllm model implementation, so the server is launched with --model-impl vllm. That is also what puts the KV cache in device tensors, which is the only mode the connector supports: it transfers the cache as torch tensors on device='rbln'.

• Unless the run resolves to the vllm model implementation, RBLN vLLM never imports the module that registers RBLNLMCacheConnectorV1, so engine startup fails while validating the connector name with Unsupported connector type: RBLNLMCacheConnectorV1. Set --model-impl vllm explicitly, since auto picks transformers for the architectures optimum-rbln supports.
• PYTHONHASHSEED has to be set to a fixed value. Without it, vLLM derives the base of its block hash chain from a per-process random value, the chain diverges between the scheduler and the workers, and every cache lookup misses. The connector refuses to construct with RuntimeError instead of failing silently. vLLM hashes the variable's string to seed the chain, so any fixed value works.

1
2

systemctl status rebellions-rds-binding.service --no-pager
journalctl -u rebellions-rds-binding.service -b --no-pager

1

python -c 'from rebel import rds; print("RDS available:", rds.is_available()); print("RDS devices:", rds.query_available_device_count())'

Configuration¶

The cache is configured through LMCache's LMCACHE_* environment variables, and the connector is selected through --kv-transfer-config. Choose a storage tier, review its settings below, then copy the complete block in Server launch.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

export PYTHONHASHSEED=0
export LMCACHE_LOCAL_CPU=True
export LMCACHE_MAX_LOCAL_CPU_SIZE=5.0
export LMCACHE_CHUNK_SIZE=1024
export LMCACHE_USE_LAYERWISE=False

vllm serve Qwen/Qwen3-1.7B \
  --model-impl vllm \
  --max-model-len 8192 \
  --block-size 1024 \
  --enable-prefix-caching \
  --max-num-batched-tokens 128 \
  --max-num-seqs 4 \
  --kv-transfer-config \
  '{"kv_connector":"RBLNLMCacheConnectorV1", "kv_role":"kv_both"}'

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

export PYTHONHASHSEED=0
export LMCACHE_LOCAL_CPU=False
export LMCACHE_MAX_LOCAL_CPU_SIZE=0
export LMCACHE_CHUNK_SIZE=1024
export LMCACHE_USE_LAYERWISE=False
export LMCACHE_STORAGE_PLUGINS=RDSBackend
export LMCACHE_EXTRA_CONFIG='{
    "storage_plugin.RDSBackend.module_path": "lmcache_rbln.storage_plugin.rds_backend",
    "storage_plugin.RDSBackend.class_name": "RDSBackend",
    "rds_buffer_size": 2048,
    "rds_failure_policy": "ignore"
}'

vllm serve Qwen/Qwen3-1.7B \
  --model-impl vllm \
  --max-model-len 8192 \
  --block-size 1024 \
  --enable-prefix-caching \
  --max-num-batched-tokens 128 \
  --max-num-seqs 4 \
  --kv-transfer-config \
  '{"kv_connector":"RBLNLMCacheConnectorV1", "kv_role":"kv_both"}'

Verify offloading¶

First check startup, store, and restore logs:

To check a restore, start the server with --no-enable-prefix-caching and send the same prompt, longer than one chunk, twice. If vllm:external_prefix_cache_hits_total in /metrics is higher after the second request than before the first, the second request reused KV from the offload tier. For RDS, also check that the log shows Retrieved N out of N required tokens with no RDS batched write failed error. Turn prefix caching back on afterward. If no tokens are retrieved, see Troubleshooting.

Troubleshooting¶

1

RDS batched write failed for 1 keys

Limitations¶

• Sub-block prefix caching and a KV connector both extend a full-block prefix cache hit, and the scheduler runs only one of them per request, taking whichever offers more tokens, with sub-block winning a tie. See Prefix Caching.