Skip to content

Dynamic Batch Sizes

Overview

The number of requests in a decode batch can vary during serving. Preparing multiple batch sizes lets the engine use a size suited to the current batch, reducing padding compared with always using the maximum batch size.

Both the vllm and transformers (optimum) model implementations support this approach. The configuration differs by model implementation: vllm uses the decode_batch_bucket_* options, while transformers (optimum) uses optimum_overrides["decoder_batch_sizes"]. The sections below describe how to configure each implementation.

vllm model implementation

Configuring bucket sizes

A bucket is a batch size used for decoding. You can generate these sizes automatically or specify them directly. The maximum bucket size is max_num_seqs divided by the pipeline-parallel size. By default, --rbln-decode-batch-bucket-limit is 1, so the engine creates one bucket whose size equals the maximum batch size. For example, if the maximum batch size is 8, --rbln-decode-batch-bucket-strategy is exponential, --rbln-decode-batch-bucket-step is 2, and --rbln-decode-batch-bucket-limit is 4, the engine creates four buckets with sizes 8, 4, 2, and 1.

See the Configuration Guide for detailed option descriptions and constraints.

Flag Default Purpose
--rbln-decode-batch-bucket-strategy exponential Strategy: exponential, linear, or manual
--rbln-decode-batch-bucket-limit 1 Maximum number of generated buckets
--rbln-decode-batch-bucket-min 1 Minimum generated bucket size
--rbln-decode-batch-bucket-step 2 Division factor or subtraction step
--rbln-decode-batch-bucket-manual-buckets [] Explicit bucket sizes for manual

Configuration examples

Automatic generation

Generate four batch-size buckets: 8, 4, 2, and 1.

1
2
3
4
5
6
vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
  --model-impl vllm \
  --max-model-len 8192 \
  --block-size 8192 \
  --max-num-seqs 8 \
  --rbln-decode-batch-bucket-limit 4

Explicit sizes

Specify bucket sizes 1, 4, and 8 directly:

1
2
3
4
5
6
7
vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
  --model-impl vllm \
  --max-model-len 8192 \
  --block-size 8192 \
  --max-num-seqs 8 \
  --rbln-decode-batch-bucket-strategy manual \
  --rbln-decode-batch-bucket-manual-buckets 1 4 8

With these bucket sizes, a decode batch of 3 requests is padded to 4, and a batch of 5 is padded to 8.

transformers (optimum) model implementation

Set model_impl="transformers" and specify the decoder batch sizes in additional_config["optimum_overrides"]["decoder_batch_sizes"]. The example below compiles decoder graphs for batch sizes 1, 4, and 8 when the engine starts.

Note

  • All values must be less than or equal to max_num_seqs.
  • Batch sizes can be listed in any order. If the list does not include max_num_seqs, it is added automatically.
  • For a precompiled model, configure the decoder batch sizes during compilation. This override does not change the batch sizes of an existing compiled model.
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
from vllm import LLM

llm = LLM(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    model_impl="transformers",
    max_model_len=8192,
    block_size=8192,
    max_num_seqs=8,
    additional_config={
        "optimum_overrides": {
            "decoder_batch_sizes": [1, 4, 8],
        },
    },
)