Compile Cache¶

Cache behavior¶

The cache is saved only after warm-up completes successfully. If you interrupt warm-up, the cache won’t be saved.

Set VLLM_DISABLE_COMPILE_CACHE=1 to disable cache reuse and recompile on each run.

Cache format¶

Compiled graphs are stored in a mega cache bundle (.bin).

Cache location¶

$VLLM_CACHE_ROOT/rbln

Cache key¶

Included:

• vLLM configuration, including max_model_len, block_size, max_num_seqs, and max_num_batched_tokens.
• RBLN options, except those listed under Excluded below.

Excluded:

• use_custom_sampler. Sampler graphs are compiled separately and are not cached.
• enable_sub_block_cache and sub_block_size. These options control prefix-cache reuse and do not affect model compilation.

Cache behavior¶

• Precompiled model directory: The engine loads the model directly, without compilation.
• HuggingFace model ID: The engine loads a cached model if available; otherwise, it compiles and caches the model at startup.

Cache format¶

Compiled models are stored as .rbln files.

Cache location¶

$VLLM_CACHE_ROOT/compiled_models/<name>

Cache key¶

When using a HuggingFace model ID:

Included:

• model, dtype, max_model_len, block_size, and max_num_seqs.
• The user-specified max_num_batched_tokens (prefill chunk size).
• num_devices_per_local_rank and gpu_memory_utilization.
• optimum_overrides, except the entries listed below.

Excluded:

• device, create_runtimes, and kvcache_num_blocks in optimum_overrides. These options control runtime behavior and do not affect model compilation.