gpt-oss-120b on RBLN NPU¶
{"attn": "flash_attn", "axes": [{"axis": "lmcache", "options": [{"label": "Off", "value": "off"}, {"label": "LMCache (CPU)", "value": "cpu"}, {"label": "LMCache (RDS)", "value": "rds"}], "title": "KV Offload"}, {"axis": "kv_connector", "options": [{"label": "Off", "value": "off"}, {"label": "LMCache (CPU)", "value": "lmcache_cpu"}, {"label": "LMCache (RDS)", "value": "lmcache_rds"}], "title": "KV Offload"}, {"axis": "prefix_caching", "options": [{"label": "On", "value": "on"}, {"label": "On (sub-block)", "value": "on_sub_block"}, {"label": "Off", "value": "off"}], "title": "Prefix Caching"}, {"axis": "spec_decode", "options": [{"label": "Off", "value": "off"}, {"label": "N-gram", "value": "ngram"}, {"label": "EAGLE-3", "value": "eagle3"}, {"label": "Suffix", "value": "suffix"}, {"label": "MTP", "value": "mtp"}], "title": "Speculative Decoding"}, {"axis": "on_device_sampler", "options": [{"label": "On", "value": "on"}, {"label": "Off", "value": "off"}], "title": "On-device Sampler"}, {"axis": "structured_output", "options": [{"label": "On", "value": "on"}, {"label": "Off", "value": "off"}], "title": "Structured Output"}], "badges": ["MoE", "120B", "131,072 ctx", {"hl": true, "text": "vllm-rbln"}, "flash_attn", "Text"], "client": "curl http://localhost:8000/v1/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -d \u0027{\n \"model\": \"openai/gpt-oss-120b\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}],\n \"max_tokens\": 32\n }\u0027", "features": {"api": true, "mm": false, "text": true}, "hf": "https://huggingface.co/openai/gpt-oss-120b", "installLine": "pip install rebel-compiler \"vllm-rbln[kv_connectors]\"", "model": "openai/gpt-oss-120b", "name": "gpt-oss-120b", "provider": "openai", "strategies": [{"label": "Aggregated", "nodes": null, "options": [{"attn": null, "features": {}, "grid": {"base": [{"args": {"--block-size": "--block-size 4096", "--distributed-executor-backend": "--distributed-executor-backend mp", "--enable-auto-tool-choice": "--enable-auto-tool-choice", "--max-model-len": "--max-model-len 131072", "--max-num-batched-tokens": "--max-num-batched-tokens 512", "--max-num-seqs": "--max-num-seqs 1", "--model-impl": "--model-impl vllm", "--no-enable-prefix-caching": "--no-enable-prefix-caching", "--reasoning-parser": "--reasoning-parser openai_gptoss", "--tool-call-parser": "--tool-call-parser openai", "--trust-remote-code": "--trust-remote-code"}, "envs": {"VLLM_ENGINE_READY_TIMEOUT_S": "VLLM_ENGINE_READY_TIMEOUT_S=9600"}, "launcher": "vllm serve", "ranks": null, "role": "main", "scale": null, "varies": null}], "deltas": {"lmcache=cpu": {"main": {"add_args": {"--kv-transfer-config": "--kv-transfer-config \u0027{\n \"kv_connector\": \"RBLNLMCacheConnectorV1\",\n \"kv_role\": \"kv_both\"\n }\u0027"}, "add_envs": {"LMCACHE_CHUNK_SIZE": "LMCACHE_CHUNK_SIZE=256", "LMCACHE_MAX_LOCAL_CPU_SIZE": "LMCACHE_MAX_LOCAL_CPU_SIZE=10.0", "PYTHONHASHSEED": "PYTHONHASHSEED=42"}, "del_args": [], "del_envs": []}}, "lmcache=rds": {"main": {"add_args": {"--gpu-memory-utilization": "--gpu-memory-utilization 0.8", "--kv-transfer-config": "--kv-transfer-config \u0027{\n \"kv_connector\": \"RBLNLMCacheConnectorV1\",\n \"kv_role\": \"kv_both\"\n }\u0027"}, "add_envs": {"LMCACHE_CHUNK_SIZE": "LMCACHE_CHUNK_SIZE=4096", "LMCACHE_EXTRA_CONFIG": "LMCACHE_EXTRA_CONFIG=\u0027{\n \"storage_plugin.RDSBackend.module_path\": \"lmcache_rbln.storage_plugin.rds_backend\",\n \"storage_plugin.RDSBackend.class_name\": \"RDSBackend\"\n}\u0027", "LMCACHE_LOCAL_CPU": "LMCACHE_LOCAL_CPU=False", "LMCACHE_MAX_LOCAL_CPU_SIZE": "LMCACHE_MAX_LOCAL_CPU_SIZE=0", "LMCACHE_STORAGE_PLUGINS": "LMCACHE_STORAGE_PLUGINS=RDSBackend", "PYTHONHASHSEED": "PYTHONHASHSEED=42"}, "del_args": [], "del_envs": []}}, "prefix_caching=on": {"main": {"add_args": {}, "add_envs": {"VLLM_RBLN_SUB_BLOCK_CACHE": "VLLM_RBLN_SUB_BLOCK_CACHE=0"}, "del_args": ["--no-enable-prefix-caching"], "del_envs": []}}, "prefix_caching=on_sub_block": {"main": {"add_args": {}, "add_envs": {}, "del_args": ["--no-enable-prefix-caching"], "del_envs": []}}, "spec_decode=eagle3": {"main": {"add_args": {"--speculative-config": "--speculative-config \u0027{\n \"method\": \"eagle3\",\n \"model\": \"nvidia/gpt-oss-120b-Eagle3-v3\",\n \"num_speculative_tokens\": 3\n }\u0027"}, "add_envs": {}, "del_args": [], "del_envs": []}}, "spec_decode=suffix": {"main": {"add_args": {"--speculative-config": "--speculative-config \u0027{\n \"method\": \"suffix\",\n \"num_speculative_tokens\": 3,\n \"suffix_decoding_max_cached_requests\": 0\n }\u0027"}, "add_envs": {}, "del_args": [], "del_envs": []}}, "tool_calling=off": {"main": {"add_args": {}, "add_envs": {}, "del_args": ["--enable-auto-tool-choice", "--tool-call-parser"], "del_envs": []}}}, "excluded": [], "model": "openai/gpt-oss-120b", "modes": {"lmcache": ["off", "cpu", "rds"], "prefix_caching": ["off", "on", "on_sub_block"], "spec_decode": ["off", "eagle3", "suffix"]}, "option_base": {"tool_calling": "on"}, "options": {"tool_calling": ["off", "on"]}, "recommended": {"lmcache": "rds", "prefix_caching": "on_sub_block", "spec_decode": "eagle3", "tool_calling": "on"}, "router": null}, "label": null, "modelImpl": "vllm", "note": null, "npus": 1, "skus": ["RBLN-CR13"], "target": "Latency", "verified": "2026-10-07"}, {"attn": null, "features": {}, "grid": {"base": [{"args": {"--block-size": "--block-size 4096", "--distributed-executor-backend": "--distributed-executor-backend mp", "--enable-auto-tool-choice": "--enable-auto-tool-choice", "--max-model-len": "--max-model-len 131072", "--max-num-batched-tokens": "--max-num-batched-tokens 512", "--max-num-seqs": "--max-num-seqs 8", "--model-impl": "--model-impl vllm", "--no-enable-prefix-caching": "--no-enable-prefix-caching", "--reasoning-parser": "--reasoning-parser openai_gptoss", "--tool-call-parser": "--tool-call-parser openai", "--trust-remote-code": "--trust-remote-code"}, "envs": {"VLLM_ENGINE_READY_TIMEOUT_S": "VLLM_ENGINE_READY_TIMEOUT_S=9600"}, "launcher": "vllm serve", "ranks": null, "role": "main", "scale": null, "varies": null}], "deltas": {"lmcache=cpu": {"main": {"add_args": {"--kv-transfer-config": "--kv-transfer-config \u0027{\n \"kv_connector\": \"RBLNLMCacheConnectorV1\",\n \"kv_role\": \"kv_both\"\n }\u0027"}, "add_envs": {"LMCACHE_CHUNK_SIZE": "LMCACHE_CHUNK_SIZE=256", "LMCACHE_MAX_LOCAL_CPU_SIZE": "LMCACHE_MAX_LOCAL_CPU_SIZE=10.0", "PYTHONHASHSEED": "PYTHONHASHSEED=42"}, "del_args": [], "del_envs": []}}, "lmcache=rds": {"main": {"add_args": {"--gpu-memory-utilization": "--gpu-memory-utilization 0.8", "--kv-transfer-config": "--kv-transfer-config \u0027{\n \"kv_connector\": \"RBLNLMCacheConnectorV1\",\n \"kv_role\": \"kv_both\"\n }\u0027"}, "add_envs": {"LMCACHE_CHUNK_SIZE": "LMCACHE_CHUNK_SIZE=4096", "LMCACHE_EXTRA_CONFIG": "LMCACHE_EXTRA_CONFIG=\u0027{\n \"storage_plugin.RDSBackend.module_path\": \"lmcache_rbln.storage_plugin.rds_backend\",\n \"storage_plugin.RDSBackend.class_name\": \"RDSBackend\"\n}\u0027", "LMCACHE_LOCAL_CPU": "LMCACHE_LOCAL_CPU=False", "LMCACHE_MAX_LOCAL_CPU_SIZE": "LMCACHE_MAX_LOCAL_CPU_SIZE=0", "LMCACHE_STORAGE_PLUGINS": "LMCACHE_STORAGE_PLUGINS=RDSBackend", "PYTHONHASHSEED": "PYTHONHASHSEED=42"}, "del_args": [], "del_envs": []}}, "prefix_caching=on": {"main": {"add_args": {}, "add_envs": {"VLLM_RBLN_SUB_BLOCK_CACHE": "VLLM_RBLN_SUB_BLOCK_CACHE=0"}, "del_args": ["--no-enable-prefix-caching"], "del_envs": []}}, "prefix_caching=on_sub_block": {"main": {"add_args": {}, "add_envs": {}, "del_args": ["--no-enable-prefix-caching"], "del_envs": []}}, "spec_decode=eagle3": {"main": {"add_args": {"--speculative-config": "--speculative-config \u0027{\n \"method\": \"eagle3\",\n \"model\": \"nvidia/gpt-oss-120b-Eagle3-v3\",\n \"num_speculative_tokens\": 3\n }\u0027"}, "add_envs": {}, "del_args": [], "del_envs": []}}, "spec_decode=suffix": {"main": {"add_args": {"--speculative-config": "--speculative-config \u0027{\n \"method\": \"suffix\",\n \"num_speculative_tokens\": 3,\n \"suffix_decoding_max_cached_requests\": 0\n }\u0027"}, "add_envs": {}, "del_args": [], "del_envs": []}}, "tool_calling=off": {"main": {"add_args": {}, "add_envs": {}, "del_args": ["--enable-auto-tool-choice", "--tool-call-parser"], "del_envs": []}}}, "excluded": [], "model": "openai/gpt-oss-120b", "modes": {"lmcache": ["off", "cpu", "rds"], "prefix_caching": ["off", "on", "on_sub_block"], "spec_decode": ["off", "eagle3", "suffix"]}, "option_base": {"tool_calling": "on"}, "options": {"tool_calling": ["off", "on"]}, "recommended": {"lmcache": "rds", "prefix_caching": "on_sub_block", "spec_decode": "eagle3", "tool_calling": "on"}, "router": null}, "label": null, "modelImpl": "vllm", "note": null, "npus": 1, "skus": ["RBLN-CR13"], "target": "Throughput", "verified": "2026-10-07"}], "slug": "aggregated", "tag": "Single device", "variant": "MXFP4"}], "text": {"en": {"strategyNote": "Each server takes one device. See \u003cstrong\u003eConfiguration\u003c/strong\u003e in the guide for the engine arguments.", "sub": "gpt-oss-120b \u2014 mixture-of-experts (MoE) decoder-only LLM served on RBLN NPUs with RBLN vLLM, 131,072-token context (Flash Attention).", "tag": "Aggregated vllm serve, verified on RBLN-CR13."}, "ko": {"strategyNote": "\uc11c\ubc84 \ud558\ub098\uac00 \ub514\ubc14\uc774\uc2a4 \ud558\ub098\ub97c \uc501\ub2c8\ub2e4. \uc5d4\uc9c4 \uc778\uc790\ub294 \uac00\uc774\ub4dc\uc758 \u003cstrong\u003eConfiguration\u003c/strong\u003e\uc5d0\uc11c \ud655\uc778\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4.", "sub": "gpt-oss-120b \u2014 mixture-of-experts(MoE) \ub514\ucf54\ub354 \uc804\uc6a9 LLM, RBLN vLLM\uc73c\ub85c RBLN NPU\uc5d0\uc11c \uc11c\ube59. 131,072 \ud1a0\ud070 \ucee8\ud14d\uc2a4\ud2b8(Flash Attention).", "tag": "RBLN-CR13\uc73c\ub85c \uac80\uc99d\ub41c aggregated vllm serve \uad6c\uc131\uc785\ub2c8\ub2e4."}}, "variant": "MXFP4"} Install the RBLN Driver on the host. An RBLN Portal account is required to install the RBLN Compiler (rebel-compiler). See the installation matrix for compatible package versions.
The KV offload (LMCache) commands need the kv_connectors extra, which installs lmcache-rbln and nixl-rbln. Without it those servers fail at startup. The upstream lmcache package that this extra installs supports Python 3.13 or earlier, so the installation fails on Python 3.14.
Suffix decoding (the Suffix option under Speculative Decoding) needs arctic-inference, which vllm-rbln does not install by default. Without it that server fails at startup with an ImportError.
Guide — overview and configuration
Overview¶
openai/gpt-oss-120b is a Mixture-of-Experts, decoder-only model. On RBLN NPUs, the RBLN vLLM plugin compiles it on first run for a fixed batch size, sequence length, and NPU count, then serves it over vLLM's OpenAI-compatible API.
Configuration¶
Verified on RBLN-CR13. Every strategy in the recipe above is aggregated: one server takes every request on one device.
Every command runs on the vllm model implementation, selected with --model-impl vllm.
Recipes for other model sizes:
See the Configuration Guide for all engine arguments.
Notes and limitations¶
- The batch size, sequence length, and NPU count are fixed at compilation; changing any of these settings triggers recompilation at the next startup.
- The latency strategies serve with
--max-num-seqs 1and the throughput strategies with a larger value. Changing it recompiles the model.