Skip to content

Automatic Prefix Caching

Overview

vLLM RBLN supports Automatic Prefix Caching (APC), which utilizes the KV cache from previous requests when a new request shares the same prefix. This allows the model to skip computation for the overlapping prefix segment, improving efficiency and throughput.

Enabling APC in vLLM RBLN

APC is configured the same way as in vLLM. It is enabled by default. To disable it, set enable_prefix_caching=False.

Advanced Configuration: Prefix Cache Hit Granularity

By default, the prefix cache hit granularity is the prefill chunk size, that is max_num_batched_tokens. To change the prefix cache hit granularity, set prefix_block_size in additional_config when initializing the LLM Engine. prefix_block_size must be a multiple of the prefill chunk size.

The example below sets a prefill chunk size of 128, so prefix_block_size is set to 256 (a multiple of 128).

1
2
3
4
5
6
7
8
9
from vllm import LLM

llm = LLM(
    model=MODEL,
    max_num_batched_tokens=128,
    additional_config={
        "prefix_block_size": 256,
    },
)