RBLN NPU에서 gpt-oss-120b 서빙¶

{"attn": "flash_attn", "axes": [{"axis": "lmcache", "options": [{"label": "Off", "value": "off"}, {"label": "LMCache (CPU)", "value": "cpu"}, {"label": "LMCache (RDS)", "value": "rds"}], "title": "KV Offload"}, {"axis": "kv_connector", "options": [{"label": "Off", "value": "off"}, {"label": "LMCache (CPU)", "value": "lmcache_cpu"}, {"label": "LMCache (RDS)", "value": "lmcache_rds"}], "title": "KV Offload"}, {"axis": "prefix_caching", "options": [{"label": "On", "value": "on"}, {"label": "On (sub-block)", "value": "on_sub_block"}, {"label": "Off", "value": "off"}], "title": "Prefix Caching"}, {"axis": "spec_decode", "options": [{"label": "Off", "value": "off"}, {"label": "N-gram", "value": "ngram"}, {"label": "EAGLE-3", "value": "eagle3"}, {"label": "Suffix", "value": "suffix"}, {"label": "MTP", "value": "mtp"}], "title": "Speculative Decoding"}, {"axis": "on_device_sampler", "options": [{"label": "On", "value": "on"}, {"label": "Off", "value": "off"}], "title": "On-device Sampler"}, {"axis": "structured_output", "options": [{"label": "On", "value": "on"}, {"label": "Off", "value": "off"}], "title": "Structured Output"}], "badges": ["MoE", "120B", "131,072 ctx", {"hl": true, "text": "vllm-rbln"}, "flash_attn", "Text"], "client": "curl http://localhost:8000/v1/chat/completions \\\n    -H \"Content-Type: application/json\" \\\n    -d \u0027{\n      \"model\": \"openai/gpt-oss-120b\",\n      \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}],\n      \"max_tokens\": 32\n    }\u0027", "features": {"api": true, "mm": false, "text": true}, "hf": "https://huggingface.co/openai/gpt-oss-120b", "installLine": "pip install rebel-compiler \"vllm-rbln[kv_connectors]\"", "model": "openai/gpt-oss-120b", "name": "gpt-oss-120b", "provider": "openai", "strategies": [{"label": "Aggregated", "nodes": null, "options": [{"attn": null, "features": {}, "grid": {"base": [{"args": {"--block-size": "--block-size 4096", "--distributed-executor-backend": "--distributed-executor-backend mp", "--enable-auto-tool-choice": "--enable-auto-tool-choice", "--max-model-len": "--max-model-len 131072", "--max-num-batched-tokens": "--max-num-batched-tokens 512", "--max-num-seqs": "--max-num-seqs 1", "--model-impl": "--model-impl vllm", "--no-enable-prefix-caching": "--no-enable-prefix-caching", "--reasoning-parser": "--reasoning-parser openai_gptoss", "--tool-call-parser": "--tool-call-parser openai", "--trust-remote-code": "--trust-remote-code"}, "envs": {"VLLM_ENGINE_READY_TIMEOUT_S": "VLLM_ENGINE_READY_TIMEOUT_S=9600"}, "launcher": "vllm serve", "ranks": null, "role": "main", "scale": null, "varies": null}], "deltas": {"lmcache=cpu": {"main": {"add_args": {"--kv-transfer-config": "--kv-transfer-config \u0027{\n      \"kv_connector\": \"RBLNLMCacheConnectorV1\",\n      \"kv_role\": \"kv_both\"\n    }\u0027"}, "add_envs": {"LMCACHE_CHUNK_SIZE": "LMCACHE_CHUNK_SIZE=256", "LMCACHE_MAX_LOCAL_CPU_SIZE": "LMCACHE_MAX_LOCAL_CPU_SIZE=10.0", "PYTHONHASHSEED": "PYTHONHASHSEED=42"}, "del_args": [], "del_envs": []}}, "lmcache=rds": {"main": {"add_args": {"--gpu-memory-utilization": "--gpu-memory-utilization 0.8", "--kv-transfer-config": "--kv-transfer-config \u0027{\n      \"kv_connector\": \"RBLNLMCacheConnectorV1\",\n      \"kv_role\": \"kv_both\"\n    }\u0027"}, "add_envs": {"LMCACHE_CHUNK_SIZE": "LMCACHE_CHUNK_SIZE=4096", "LMCACHE_EXTRA_CONFIG": "LMCACHE_EXTRA_CONFIG=\u0027{\n  \"storage_plugin.RDSBackend.module_path\": \"lmcache_rbln.storage_plugin.rds_backend\",\n  \"storage_plugin.RDSBackend.class_name\": \"RDSBackend\"\n}\u0027", "LMCACHE_LOCAL_CPU": "LMCACHE_LOCAL_CPU=False", "LMCACHE_MAX_LOCAL_CPU_SIZE": "LMCACHE_MAX_LOCAL_CPU_SIZE=0", "LMCACHE_STORAGE_PLUGINS": "LMCACHE_STORAGE_PLUGINS=RDSBackend", "PYTHONHASHSEED": "PYTHONHASHSEED=42"}, "del_args": [], "del_envs": []}}, "prefix_caching=on": {"main": {"add_args": {}, "add_envs": {"VLLM_RBLN_SUB_BLOCK_CACHE": "VLLM_RBLN_SUB_BLOCK_CACHE=0"}, "del_args": ["--no-enable-prefix-caching"], "del_envs": []}}, "prefix_caching=on_sub_block": {"main": {"add_args": {}, "add_envs": {}, "del_args": ["--no-enable-prefix-caching"], "del_envs": []}}, "spec_decode=eagle3": {"main": {"add_args": {"--speculative-config": "--speculative-config \u0027{\n      \"method\": \"eagle3\",\n      \"model\": \"nvidia/gpt-oss-120b-Eagle3-v3\",\n      \"num_speculative_tokens\": 3\n    }\u0027"}, "add_envs": {}, "del_args": [], "del_envs": []}}, "spec_decode=suffix": {"main": {"add_args": {"--speculative-config": "--speculative-config \u0027{\n      \"method\": \"suffix\",\n      \"num_speculative_tokens\": 3,\n      \"suffix_decoding_max_cached_requests\": 0\n    }\u0027"}, "add_envs": {}, "del_args": [], "del_envs": []}}, "tool_calling=off": {"main": {"add_args": {}, "add_envs": {}, "del_args": ["--enable-auto-tool-choice", "--tool-call-parser"], "del_envs": []}}}, "excluded": [], "model": "openai/gpt-oss-120b", "modes": {"lmcache": ["off", "cpu", "rds"], "prefix_caching": ["off", "on", "on_sub_block"], "spec_decode": ["off", "eagle3", "suffix"]}, "option_base": {"tool_calling": "on"}, "options": {"tool_calling": ["off", "on"]}, "recommended": {"lmcache": "rds", "prefix_caching": "on_sub_block", "spec_decode": "eagle3", "tool_calling": "on"}, "router": null}, "label": null, "modelImpl": "vllm", "note": null, "npus": 1, "skus": ["RBLN-CR13"], "target": "Latency", "verified": "2026-10-07"}, {"attn": null, "features": {}, "grid": {"base": [{"args": {"--block-size": "--block-size 4096", "--distributed-executor-backend": "--distributed-executor-backend mp", "--enable-auto-tool-choice": "--enable-auto-tool-choice", "--max-model-len": "--max-model-len 131072", "--max-num-batched-tokens": "--max-num-batched-tokens 512", "--max-num-seqs": "--max-num-seqs 8", "--model-impl": "--model-impl vllm", "--no-enable-prefix-caching": "--no-enable-prefix-caching", "--reasoning-parser": "--reasoning-parser openai_gptoss", "--tool-call-parser": "--tool-call-parser openai", "--trust-remote-code": "--trust-remote-code"}, "envs": {"VLLM_ENGINE_READY_TIMEOUT_S": "VLLM_ENGINE_READY_TIMEOUT_S=9600"}, "launcher": "vllm serve", "ranks": null, "role": "main", "scale": null, "varies": null}], "deltas": {"lmcache=cpu": {"main": {"add_args": {"--kv-transfer-config": "--kv-transfer-config \u0027{\n      \"kv_connector\": \"RBLNLMCacheConnectorV1\",\n      \"kv_role\": \"kv_both\"\n    }\u0027"}, "add_envs": {"LMCACHE_CHUNK_SIZE": "LMCACHE_CHUNK_SIZE=256", "LMCACHE_MAX_LOCAL_CPU_SIZE": "LMCACHE_MAX_LOCAL_CPU_SIZE=10.0", "PYTHONHASHSEED": "PYTHONHASHSEED=42"}, "del_args": [], "del_envs": []}}, "lmcache=rds": {"main": {"add_args": {"--gpu-memory-utilization": "--gpu-memory-utilization 0.8", "--kv-transfer-config": "--kv-transfer-config \u0027{\n      \"kv_connector\": \"RBLNLMCacheConnectorV1\",\n      \"kv_role\": \"kv_both\"\n    }\u0027"}, "add_envs": {"LMCACHE_CHUNK_SIZE": "LMCACHE_CHUNK_SIZE=4096", "LMCACHE_EXTRA_CONFIG": "LMCACHE_EXTRA_CONFIG=\u0027{\n  \"storage_plugin.RDSBackend.module_path\": \"lmcache_rbln.storage_plugin.rds_backend\",\n  \"storage_plugin.RDSBackend.class_name\": \"RDSBackend\"\n}\u0027", "LMCACHE_LOCAL_CPU": "LMCACHE_LOCAL_CPU=False", "LMCACHE_MAX_LOCAL_CPU_SIZE": "LMCACHE_MAX_LOCAL_CPU_SIZE=0", "LMCACHE_STORAGE_PLUGINS": "LMCACHE_STORAGE_PLUGINS=RDSBackend", "PYTHONHASHSEED": "PYTHONHASHSEED=42"}, "del_args": [], "del_envs": []}}, "prefix_caching=on": {"main": {"add_args": {}, "add_envs": {"VLLM_RBLN_SUB_BLOCK_CACHE": "VLLM_RBLN_SUB_BLOCK_CACHE=0"}, "del_args": ["--no-enable-prefix-caching"], "del_envs": []}}, "prefix_caching=on_sub_block": {"main": {"add_args": {}, "add_envs": {}, "del_args": ["--no-enable-prefix-caching"], "del_envs": []}}, "spec_decode=eagle3": {"main": {"add_args": {"--speculative-config": "--speculative-config \u0027{\n      \"method\": \"eagle3\",\n      \"model\": \"nvidia/gpt-oss-120b-Eagle3-v3\",\n      \"num_speculative_tokens\": 3\n    }\u0027"}, "add_envs": {}, "del_args": [], "del_envs": []}}, "spec_decode=suffix": {"main": {"add_args": {"--speculative-config": "--speculative-config \u0027{\n      \"method\": \"suffix\",\n      \"num_speculative_tokens\": 3,\n      \"suffix_decoding_max_cached_requests\": 0\n    }\u0027"}, "add_envs": {}, "del_args": [], "del_envs": []}}, "tool_calling=off": {"main": {"add_args": {}, "add_envs": {}, "del_args": ["--enable-auto-tool-choice", "--tool-call-parser"], "del_envs": []}}}, "excluded": [], "model": "openai/gpt-oss-120b", "modes": {"lmcache": ["off", "cpu", "rds"], "prefix_caching": ["off", "on", "on_sub_block"], "spec_decode": ["off", "eagle3", "suffix"]}, "option_base": {"tool_calling": "on"}, "options": {"tool_calling": ["off", "on"]}, "recommended": {"lmcache": "rds", "prefix_caching": "on_sub_block", "spec_decode": "eagle3", "tool_calling": "on"}, "router": null}, "label": null, "modelImpl": "vllm", "note": null, "npus": 1, "skus": ["RBLN-CR13"], "target": "Throughput", "verified": "2026-10-07"}], "slug": "aggregated", "tag": "Single device", "variant": "MXFP4"}], "text": {"en": {"strategyNote": "Each server takes one device. See \u003cstrong\u003eConfiguration\u003c/strong\u003e in the guide for the engine arguments.", "sub": "gpt-oss-120b \u2014 mixture-of-experts (MoE) decoder-only LLM served on RBLN NPUs with RBLN vLLM, 131,072-token context (Flash Attention).", "tag": "Aggregated vllm serve, verified on RBLN-CR13."}, "ko": {"strategyNote": "\uc11c\ubc84 \ud558\ub098\uac00 \ub514\ubc14\uc774\uc2a4 \ud558\ub098\ub97c \uc501\ub2c8\ub2e4. \uc5d4\uc9c4 \uc778\uc790\ub294 \uac00\uc774\ub4dc\uc758 \u003cstrong\u003eConfiguration\u003c/strong\u003e\uc5d0\uc11c \ud655\uc778\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4.", "sub": "gpt-oss-120b \u2014 mixture-of-experts(MoE) \ub514\ucf54\ub354 \uc804\uc6a9 LLM, RBLN vLLM\uc73c\ub85c RBLN NPU\uc5d0\uc11c \uc11c\ube59. 131,072 \ud1a0\ud070 \ucee8\ud14d\uc2a4\ud2b8(Flash Attention).", "tag": "RBLN-CR13\uc73c\ub85c \uac80\uc99d\ub41c aggregated vllm serve \uad6c\uc131\uc785\ub2c8\ub2e4."}}, "variant": "MXFP4"}

호스트에 RBLN Driver가 설치되어 있어야 하며, RBLN Compiler(rebel-compiler)를 설치하려면 RBLN Portal 계정이 필요합니다. 호환되는 패키지 버전은 설치 매트릭스에서 확인할 수 있습니다.

$ pip install \
  --extra-index-url https://pypi.rbln.ai/simple \
  rebel-compiler==0.12.0

KV 오프로드(LMCache) 명령에는 lmcache-rbln과 nixl-rbln을 설치하는 kv_connectors extra가 필요합니다. 없으면 해당 서버가 시작 단계에서 실패합니다.

$ pip install \
  --extra-index-url https://pypi.rbln.ai/simple \
  --extra-index-url https://wheels.vllm.ai/0.30.0/cpu \
  --extra-index-url https://download.pytorch.org/whl/cpu \
  "vllm-rbln[kv_connectors]==0.12.0"

Suffix decoding(Speculative Decoding의 Suffix)에는 arctic-inference가 필요합니다. vllm-rbln 기본 설치에는 포함되지 않으며, 없으면 해당 서버가 시작 단계에서 ImportError로 실패합니다.

$ pip install arctic-inference==0.1.1

가이드 — 개요 및 설정

개요¶

openai/gpt-oss-120b는 Mixture-of-Experts(MoE) 디코더 전용 모델입니다. RBLN NPU에서는 RBLN vLLM 플러그인이 최초 실행 시 배치 크기·시퀀스 길이·NPU 개수를 고정해 아티팩트로 컴파일하고, 이를 vLLM OpenAI 호환 API로 서빙합니다.

Configuration¶

RBLN-CR13에서 검증했습니다. 위 recipe의 strategy는 모두 aggregated이며, 서버 하나가 디바이스 하나로 모든 요청을 처리합니다.

모든 명령은 --model-impl vllm으로 선택하는 vllm model implementation으로 동작합니다.

이 모델 라인의 다른 크기이며, 각각 별도의 recipe가 있습니다:

gpt-oss-20b

전체 엔진 인자는 설정 가이드에서 확인할 수 있습니다.

참고 및 제약¶

배치 크기·시퀀스 길이·NPU 개수는 컴파일 시 고정되며, 변경하면 다음 실행 시 재컴파일됩니다.
latency strategy는 --max-num-seqs 1로, throughput strategy는 더 큰 값으로 서빙합니다. 값을 바꾸면 모델을 다시 컴파일합니다.