Skip to content

Serving features

This page summarizes major RBLN vLLM capabilities supported in Red Hat AI Enterprise (RHAIE) with Rebellions.

OpenAI-compatible API serving

RBLN vLLM exposes the same OpenAI-compatible HTTP API as upstream vLLM. Currently /v1/completions and /v1/chat/completions endpoints are fully tested.

Chunked prefill

Chunked prefill breaks a long prompt's prefill computation into smaller chunks that can be scheduled alongside decode steps of other requests. This helps reduce latency spikes and improves inter-token latency consistency by preventing a single large prefill from monopolizing device time. For more detail, refer to Chunked Prefill in vLLM.

Chunked prefill is always enabled in RBLN vLLM. RBLN vLLM currently offers limited support for chunked prefill. Mixed scheduling of prefill and decode phases is not yet supported; these operations are processed independently and cannot be interleaved within a single scheduling cycle. These limitations are planned to be addressed in a future release.

Automatic prefix caching

Prefix caching improves the efficiency of the LLM engine by eliminating redundant prefill computation. It does this by caching and reusing KV blocks, matched via hash, for shared prompt prefixes across requests. This can reduce time-to-first-token for workloads with repeated prefixes. For more detail, refer to Automatic Prefix Caching.

In RBLN vLLM, automatic prefix caching is always enabled by including --enable-prefix-caching in args of the ServingRuntime. Prefix caching is most effective when long and identical system prompts are reused across many requests, because this increases both compilation reuse and prefix-cache hits in RBLN vLLM. Prefix caching in RBLN vLLM will be improved in the following release.

  • Prefix caching in RBLN vLLM is currently optimized for larger block sizes. Improvements for smaller block sizes are planned for a future release.
  • Prefix caching in RBLN vLLM currently supports full attention only. Support for additional attention types is planned for a future release.

Structured output

Structured output (guided decoding) constrains the model's token generation so that model output follows a specified schema, such as JSON, regex, or grammar, using techniques such as FSM or bitmask-based logit masking. This helps ensure that generated output is always syntactically valid and parseable without post-processing retries. For more detail, refer to Structured Output in vLLM.

RBLN vLLM currently includes tested support for JSON schema, regex, grammar, and structural tag type of fields and formats.

Model parallelism

RBLN vLLM supports model parallelism at two levels: vLLM-level parallelism (DP, TP, PP, EP) and Rebellions Scalable Design (RSD), the RBLN compiler's own device partitioning. The two are orthogonal and can be composed. For the arguments, constraints, and launch examples of each, see Model Parallelism.