Prefix Caching¶

Overview¶

Prefix caching reuses cached key-value (KV) states for shared request prefixes, which can reduce prefill computation and time to first token.

Sub-block prefix caching¶

Upstream vLLM matches cached prefixes at full KV cache block boundaries. With RBLN’s large blocks, this leaves the shared tokens in the final, incomplete block to be recomputed. RBLN vLLM adds sub-block prefix caching to reuse those tokens in smaller units called sub-blocks.

For a 4,096-token KV cache block, a sub-block size of 128, and a cached 3,000-token prefix shared with a longer prompt:

New tokens in the request require computation in both cases.

Configuration¶

sub_block_size sets the sub-block size in tokens. It defaults to the prefill chunk size, max_num_batched_tokens.

Set additional_config["sub_block_size"] in Python as shown below, or use --rbln-sub-block-size on the CLI.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

from vllm import LLM

llm = LLM(
    model="<MODEL_ID>",
    model_impl="<MODEL_IMPL>",  # "vllm" or "transformers"
    enable_prefix_caching=True,
    max_num_batched_tokens=128,
    block_size=4096,
    additional_config={
        "sub_block_size": 128,
    },
)

Limitation: KV cache allocation and copying¶

Upstream vLLM reuses full KV cache blocks directly, without copying. RBLN vLLM's vllm model implementation does the same for full-block matches.

For sub-block matches, however, both model implementations allocate separate blocks and copy cached KV data into them. This avoids recomputation but adds allocation and copy overhead.

Prefix cache-aware routing¶

A prefix cache-aware router sends each request to an engine that already caches its prefix.

With sub-block caching enabled, KV cache events are published at sub-block granularity. To configure routing:

1. Enable event publishing with --kv-events-config using upstream vLLM's KVEventsConfig options, and configure the router to receive the events.
2. Match the router's token-processing block size to the engine's sub-block size. If --rbln-sub-block-size is not specified, the engine uses --max-num-batched-tokens as its sub-block size.

If sub-block caching is inactive, use --block-size to match the ordinary KV cache events.

Interaction with KV connectors¶

For each request, the scheduler uses either sub-block caching or a KV connector, without combining them. It chooses the one that provides more additional cached tokens. On a tie, it uses sub-block caching.

KV connectors are used for KV cache offloading and disaggregated prefill / decode.

For upstream prefix caching behavior, see the vLLM APC guide and prefix caching design.