gpt-oss-120b on RBLN NPU¶

{"attn": "flash_attn", "axes": [{"axis": "lmcache", "options": [{"label": "Off", "value": "off"}, {"label": "LMCache (CPU)", "value": "cpu"}, {"label": "LMCache (RDS)", "value": "rds"}], "title": "KV Offload"}, {"axis": "kv_connector", "options": [{"label": "Off", "value": "off"}, {"label": "LMCache (CPU)", "value": "lmcache_cpu"}, {"label": "LMCache (RDS)", "value": "lmcache_rds"}], "title": "KV Offload"}, {"axis": "prefix_caching", "options": [{"label": "On", "value": "on"}, {"label": "On (sub-block)", "value": "on_sub_block"}, {"label": "Off", "value": "off"}], "title": "Prefix Caching"}, {"axis": "spec_decode", "options": [{"label": "Off", "value": "off"}, {"label": "N-gram", "value": "ngram"}, {"label": "EAGLE-3", "value": "eagle3"}, {"label": "Suffix", "value": "suffix"}, {"label": "MTP", "value": "mtp"}], "title": "Speculative Decoding"}, {"axis": "on_device_sampler", "options": [{"label": "On", "value": "on"}, {"label": "Off", "value": "off"}], "title": "On-device Sampler"}, {"axis": "structured_output", "options": [{"label": "On", "value": "on"}, {"label": "Off", "value": "off"}], "title": "Structured Output"}], "badges": ["MoE", "120B", "131,072 ctx", {"hl": true, "text": "vllm-rbln"}, "flash_attn", "Text"], "client": "curl http://localhost:8000/v1/chat/completions \\\n    -H \"Content-Type: application/json\" \\\n    -d \u0027{\n      \"model\": \"openai/gpt-oss-120b\",\n      \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}],\n      \"max_tokens\": 32\n    }\u0027", "features": {"api": true, "mm": false, "text": true}, "hf": "https://huggingface.co/openai/gpt-oss-120b", "installLine": "pip install rebel-compiler \"vllm-rbln[kv_connectors]\"", "model": "openai/gpt-oss-120b", "name": "gpt-oss-120b", "provider": "openai", "strategies": [{"label": "Aggregated", "nodes": null, "options": [{"attn": null, "features": {}, "grid": {"base": [{"args": {"--block-size": "--block-size 4096", "--distributed-executor-backend": "--distributed-executor-backend mp", "--enable-auto-tool-choice": "--enable-auto-tool-choice", "--max-model-len": "--max-model-len 131072", "--max-num-batched-tokens": "--max-num-batched-tokens 512", "--max-num-seqs": "--max-num-seqs 1", "--model-impl": "--model-impl vllm", "--no-enable-prefix-caching": "--no-enable-prefix-caching", "--reasoning-parser": "--reasoning-parser openai_gptoss", "--tool-call-parser": "--tool-call-parser openai", "--trust-remote-code": "--trust-remote-code"}, "envs": {"VLLM_ENGINE_READY_TIMEOUT_S": "VLLM_ENGINE_READY_TIMEOUT_S=9600"}, "launcher": "vllm serve", "ranks": null, "role": "main", "scale": null, "varies": null}], "deltas": {"lmcache=cpu": {"main": {"add_args": {"--kv-transfer-config": "--kv-transfer-config \u0027{\n      \"kv_connector\": \"RBLNLMCacheConnectorV1\",\n      \"kv_role\": \"kv_both\"\n    }\u0027"}, "add_envs": {"LMCACHE_CHUNK_SIZE": "LMCACHE_CHUNK_SIZE=256", "LMCACHE_MAX_LOCAL_CPU_SIZE": "LMCACHE_MAX_LOCAL_CPU_SIZE=10.0", "PYTHONHASHSEED": "PYTHONHASHSEED=42"}, "del_args": [], "del_envs": []}}, "lmcache=rds": {"main": {"add_args": {"--gpu-memory-utilization": "--gpu-memory-utilization 0.8", "--kv-transfer-config": "--kv-transfer-config \u0027{\n      \"kv_connector\": \"RBLNLMCacheConnectorV1\",\n      \"kv_role\": \"kv_both\"\n    }\u0027"}, "add_envs": {"LMCACHE_CHUNK_SIZE": "LMCACHE_CHUNK_SIZE=4096", "LMCACHE_EXTRA_CONFIG": "LMCACHE_EXTRA_CONFIG=\u0027{\n  \"storage_plugin.RDSBackend.module_path\": \"lmcache_rbln.storage_plugin.rds_backend\",\n  \"storage_plugin.RDSBackend.class_name\": \"RDSBackend\"\n}\u0027", "LMCACHE_LOCAL_CPU": "LMCACHE_LOCAL_CPU=False", "LMCACHE_MAX_LOCAL_CPU_SIZE": "LMCACHE_MAX_LOCAL_CPU_SIZE=0", "LMCACHE_STORAGE_PLUGINS": "LMCACHE_STORAGE_PLUGINS=RDSBackend", "PYTHONHASHSEED": "PYTHONHASHSEED=42"}, "del_args": [], "del_envs": []}}, "prefix_caching=on": {"main": {"add_args": {}, "add_envs": {"VLLM_RBLN_SUB_BLOCK_CACHE": "VLLM_RBLN_SUB_BLOCK_CACHE=0"}, "del_args": ["--no-enable-prefix-caching"], "del_envs": []}}, "prefix_caching=on_sub_block": {"main": {"add_args": {}, "add_envs": {}, "del_args": ["--no-enable-prefix-caching"], "del_envs": []}}, "spec_decode=eagle3": {"main": {"add_args": {"--speculative-config": "--speculative-config \u0027{\n      \"method\": \"eagle3\",\n      \"model\": \"nvidia/gpt-oss-120b-Eagle3-v3\",\n      \"num_speculative_tokens\": 3\n    }\u0027"}, "add_envs": {}, "del_args": [], "del_envs": []}}, "spec_decode=suffix": {"main": {"add_args": {"--speculative-config": "--speculative-config \u0027{\n      \"method\": \"suffix\",\n      \"num_speculative_tokens\": 3,\n      \"suffix_decoding_max_cached_requests\": 0\n    }\u0027"}, "add_envs": {}, "del_args": [], "del_envs": []}}, "tool_calling=off": {"main": {"add_args": {}, "add_envs": {}, "del_args": ["--enable-auto-tool-choice", "--tool-call-parser"], "del_envs": []}}}, "excluded": [], "model": "openai/gpt-oss-120b", "modes": {"lmcache": ["off", "cpu", "rds"], "prefix_caching": ["off", "on", "on_sub_block"], "spec_decode": ["off", "eagle3", "suffix"]}, "option_base": {"tool_calling": "on"}, "options": {"tool_calling": ["off", "on"]}, "recommended": {"lmcache": "rds", "prefix_caching": "on_sub_block", "spec_decode": "eagle3", "tool_calling": "on"}, "router": null}, "label": null, "modelImpl": "vllm", "note": null, "npus": 1, "skus": ["RBLN-CR13"], "target": "Latency", "verified": "2026-10-07"}, {"attn": null, "features": {}, "grid": {"base": [{"args": {"--block-size": "--block-size 4096", "--distributed-executor-backend": "--distributed-executor-backend mp", "--enable-auto-tool-choice": "--enable-auto-tool-choice", "--max-model-len": "--max-model-len 131072", "--max-num-batched-tokens": "--max-num-batched-tokens 512", "--max-num-seqs": "--max-num-seqs 8", "--model-impl": "--model-impl vllm", "--no-enable-prefix-caching": "--no-enable-prefix-caching", "--reasoning-parser": "--reasoning-parser openai_gptoss", "--tool-call-parser": "--tool-call-parser openai", "--trust-remote-code": "--trust-remote-code"}, "envs": {"VLLM_ENGINE_READY_TIMEOUT_S": "VLLM_ENGINE_READY_TIMEOUT_S=9600"}, "launcher": "vllm serve", "ranks": null, "role": "main", "scale": null, "varies": null}], "deltas": {"lmcache=cpu": {"main": {"add_args": {"--kv-transfer-config": "--kv-transfer-config \u0027{\n      \"kv_connector\": \"RBLNLMCacheConnectorV1\",\n      \"kv_role\": \"kv_both\"\n    }\u0027"}, "add_envs": {"LMCACHE_CHUNK_SIZE": "LMCACHE_CHUNK_SIZE=256", "LMCACHE_MAX_LOCAL_CPU_SIZE": "LMCACHE_MAX_LOCAL_CPU_SIZE=10.0", "PYTHONHASHSEED": "PYTHONHASHSEED=42"}, "del_args": [], "del_envs": []}}, "lmcache=rds": {"main": {"add_args": {"--gpu-memory-utilization": "--gpu-memory-utilization 0.8", "--kv-transfer-config": "--kv-transfer-config \u0027{\n      \"kv_connector\": \"RBLNLMCacheConnectorV1\",\n      \"kv_role\": \"kv_both\"\n    }\u0027"}, "add_envs": {"LMCACHE_CHUNK_SIZE": "LMCACHE_CHUNK_SIZE=4096", "LMCACHE_EXTRA_CONFIG": "LMCACHE_EXTRA_CONFIG=\u0027{\n  \"storage_plugin.RDSBackend.module_path\": \"lmcache_rbln.storage_plugin.rds_backend\",\n  \"storage_plugin.RDSBackend.class_name\": \"RDSBackend\"\n}\u0027", "LMCACHE_LOCAL_CPU": "LMCACHE_LOCAL_CPU=False", "LMCACHE_MAX_LOCAL_CPU_SIZE": "LMCACHE_MAX_LOCAL_CPU_SIZE=0", "LMCACHE_STORAGE_PLUGINS": "LMCACHE_STORAGE_PLUGINS=RDSBackend", "PYTHONHASHSEED": "PYTHONHASHSEED=42"}, "del_args": [], "del_envs": []}}, "prefix_caching=on": {"main": {"add_args": {}, "add_envs": {"VLLM_RBLN_SUB_BLOCK_CACHE": "VLLM_RBLN_SUB_BLOCK_CACHE=0"}, "del_args": ["--no-enable-prefix-caching"], "del_envs": []}}, "prefix_caching=on_sub_block": {"main": {"add_args": {}, "add_envs": {}, "del_args": ["--no-enable-prefix-caching"], "del_envs": []}}, "spec_decode=eagle3": {"main": {"add_args": {"--speculative-config": "--speculative-config \u0027{\n      \"method\": \"eagle3\",\n      \"model\": \"nvidia/gpt-oss-120b-Eagle3-v3\",\n      \"num_speculative_tokens\": 3\n    }\u0027"}, "add_envs": {}, "del_args": [], "del_envs": []}}, "spec_decode=suffix": {"main": {"add_args": {"--speculative-config": "--speculative-config \u0027{\n      \"method\": \"suffix\",\n      \"num_speculative_tokens\": 3,\n      \"suffix_decoding_max_cached_requests\": 0\n    }\u0027"}, "add_envs": {}, "del_args": [], "del_envs": []}}, "tool_calling=off": {"main": {"add_args": {}, "add_envs": {}, "del_args": ["--enable-auto-tool-choice", "--tool-call-parser"], "del_envs": []}}}, "excluded": [], "model": "openai/gpt-oss-120b", "modes": {"lmcache": ["off", "cpu", "rds"], "prefix_caching": ["off", "on", "on_sub_block"], "spec_decode": ["off", "eagle3", "suffix"]}, "option_base": {"tool_calling": "on"}, "options": {"tool_calling": ["off", "on"]}, "recommended": {"lmcache": "rds", "prefix_caching": "on_sub_block", "spec_decode": "eagle3", "tool_calling": "on"}, "router": null}, "label": null, "modelImpl": "vllm", "note": null, "npus": 1, "skus": ["RBLN-CR13"], "target": "Throughput", "verified": "2026-10-07"}], "slug": "aggregated", "tag": "Single device", "variant": "MXFP4"}], "text": {"en": {"strategyNote": "Each server takes one device. See \u003cstrong\u003eConfiguration\u003c/strong\u003e in the guide for the engine arguments.", "sub": "gpt-oss-120b \u2014 mixture-of-experts (MoE) decoder-only LLM served on RBLN NPUs with RBLN vLLM, 131,072-token context (Flash Attention).", "tag": "Aggregated vllm serve, verified on RBLN-CR13."}, "ko": {"strategyNote": "\uc11c\ubc84 \ud558\ub098\uac00 \ub514\ubc14\uc774\uc2a4 \ud558\ub098\ub97c \uc501\ub2c8\ub2e4. \uc5d4\uc9c4 \uc778\uc790\ub294 \uac00\uc774\ub4dc\uc758 \u003cstrong\u003eConfiguration\u003c/strong\u003e\uc5d0\uc11c \ud655\uc778\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4.", "sub": "gpt-oss-120b \u2014 mixture-of-experts(MoE) \ub514\ucf54\ub354 \uc804\uc6a9 LLM, RBLN vLLM\uc73c\ub85c RBLN NPU\uc5d0\uc11c \uc11c\ube59. 131,072 \ud1a0\ud070 \ucee8\ud14d\uc2a4\ud2b8(Flash Attention).", "tag": "RBLN-CR13\uc73c\ub85c \uac80\uc99d\ub41c aggregated vllm serve \uad6c\uc131\uc785\ub2c8\ub2e4."}}, "variant": "MXFP4"}

Install the RBLN Driver on the host. An RBLN Portal account is required to install the RBLN Compiler (rebel-compiler). See the installation matrix for compatible package versions.

$ pip install \
  --extra-index-url https://pypi.rbln.ai/simple \
  rebel-compiler==0.12.0

The KV offload (LMCache) commands need the kv_connectors extra, which installs lmcache-rbln and nixl-rbln. Without it those servers fail at startup.

$ pip install \
  --extra-index-url https://pypi.rbln.ai/simple \
  --extra-index-url https://wheels.vllm.ai/0.30.0/cpu \
  --extra-index-url https://download.pytorch.org/whl/cpu \
  "vllm-rbln[kv_connectors]==0.12.0"

Suffix decoding (the Suffix option under Speculative Decoding) needs arctic-inference, which vllm-rbln does not install by default. Without it that server fails at startup with an ImportError.

$ pip install arctic-inference==0.1.1

Guide — overview and configuration

Overview¶

openai/gpt-oss-120b is a Mixture-of-Experts, decoder-only model. On RBLN NPUs, the RBLN vLLM plugin compiles it on first run for a fixed batch size, sequence length, and NPU count, then serves it over vLLM's OpenAI-compatible API.

Configuration¶

Verified on RBLN-CR13. Every strategy in the recipe above is aggregated: one server takes every request on one device.

Every command runs on the vllm model implementation, selected with --model-impl vllm.

Recipes for other model sizes:

gpt-oss-20b

See the Configuration Guide for all engine arguments.

Notes and limitations¶

The batch size, sequence length, and NPU count are fixed at compilation; changing any of these settings triggers recompilation at the next startup.
The latency strategies serve with --max-num-seqs 1 and the throughput strategies with a larger value. Changing it recompiles the model.