RBLN NPU에서 A.X-K2 서빙¶

{"attn": "Sparse MLA", "axes": [{"axis": "lmcache", "options": [{"label": "Off", "value": "off"}, {"label": "LMCache (CPU)", "value": "cpu"}, {"label": "LMCache (RDS)", "value": "rds"}], "title": "KV Offload"}, {"axis": "kv_connector", "options": [{"label": "Off", "value": "off"}, {"label": "LMCache (CPU)", "value": "lmcache_cpu"}, {"label": "LMCache (RDS)", "value": "lmcache_rds"}], "title": "KV Offload"}, {"axis": "prefix_caching", "options": [{"label": "On", "value": "on"}, {"label": "On (sub-block)", "value": "on_sub_block"}, {"label": "Off", "value": "off"}], "title": "Prefix Caching"}, {"axis": "spec_decode", "options": [{"label": "Off", "value": "off"}, {"label": "N-gram", "value": "ngram"}, {"label": "EAGLE-3", "value": "eagle3"}, {"label": "Suffix", "value": "suffix"}, {"label": "MTP", "value": "mtp"}], "title": "Speculative Decoding"}, {"axis": "on_device_sampler", "options": [{"label": "On", "value": "on"}, {"label": "Off", "value": "off"}], "title": "On-device Sampler"}, {"axis": "structured_output", "options": [{"label": "On", "value": "on"}, {"label": "Off", "value": "off"}], "title": "Structured Output"}], "badges": ["MoE", "688B", "262,144 ctx", {"hl": true, "text": "vllm-rbln"}, "Text"], "client": "curl http://localhost:8000/v1/chat/completions \\\n    -H \"Content-Type: application/json\" \\\n    -d \u0027{\n      \"model\": \"skt/A.X-K2-NVFP4\",\n      \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}],\n      \"max_tokens\": 32\n    }\u0027", "features": {"api": true, "mm": false, "text": true}, "hf": "https://huggingface.co/skt/A.X-K2", "installLine": "pip install rebel-compiler \"vllm-rbln[kv_connectors]\"", "model": "skt/A.X-K2", "name": "A.X-K2", "provider": "skt", "strategies": [{"label": "Aggregated \u00b7 Data + Expert Parallel", "nodes": null, "options": [{"attn": null, "features": {}, "grid": {"base": [{"args": {"--block-size": "--block-size 2048", "--data-parallel-size": "--data-parallel-size 8", "--enable-auto-tool-choice": "--enable-auto-tool-choice", "--enable-expert-parallel": "--enable-expert-parallel", "--max-model-len": "--max-model-len 262144", "--max-num-batched-tokens": "--max-num-batched-tokens 512", "--max-num-seqs": "--max-num-seqs 1", "--model-impl": "--model-impl vllm", "--no-enable-prefix-caching": "--no-enable-prefix-caching", "--prefix-caching-hash-algo": "--prefix-caching-hash-algo sha256_cbor", "--reasoning-parser": "--reasoning-parser deepseek_v3", "--revision": "--revision 9e2e804e80f8d1b3afba5d7938173cec1ed46b49", "--safetensors-load-strategy": "--safetensors-load-strategy prefetch", "--tool-call-parser": "--tool-call-parser hermes", "--trust-remote-code": "--trust-remote-code"}, "envs": {"VLLM_ENGINE_READY_TIMEOUT_S": "VLLM_ENGINE_READY_TIMEOUT_S=9600", "VLLM_WORKER_MULTIPROC_METHOD": "VLLM_WORKER_MULTIPROC_METHOD=fork"}, "launcher": "vllm serve", "ranks": null, "role": "main", "scale": null, "varies": null}], "deltas": {"lmcache=cpu": {"main": {"add_args": {"--kv-transfer-config": "--kv-transfer-config \u0027{\n      \"kv_connector\": \"RBLNLMCacheConnectorV1\",\n      \"kv_role\": \"kv_both\"\n    }\u0027"}, "add_envs": {"LMCACHE_CHUNK_SIZE": "LMCACHE_CHUNK_SIZE=256", "LMCACHE_MAX_LOCAL_CPU_SIZE": "LMCACHE_MAX_LOCAL_CPU_SIZE=10.0", "PYTHONHASHSEED": "PYTHONHASHSEED=42"}, "del_args": [], "del_envs": []}}, "lmcache=rds": {"main": {"add_args": {"--gpu-memory-utilization": "--gpu-memory-utilization 0.85", "--kv-transfer-config": "--kv-transfer-config \u0027{\n      \"kv_connector\": \"RBLNLMCacheConnectorV1\",\n      \"kv_role\": \"kv_both\"\n    }\u0027"}, "add_envs": {"LMCACHE_CHUNK_SIZE": "LMCACHE_CHUNK_SIZE=2048", "LMCACHE_EXTRA_CONFIG": "LMCACHE_EXTRA_CONFIG=\u0027{\n  \"storage_plugin.RDSBackend.module_path\": \"lmcache_rbln.storage_plugin.rds_backend\",\n  \"storage_plugin.RDSBackend.class_name\": \"RDSBackend\",\n  \"rds_max_chunks\": 800\n}\u0027", "LMCACHE_LOCAL_CPU": "LMCACHE_LOCAL_CPU=False", "LMCACHE_MAX_LOCAL_CPU_SIZE": "LMCACHE_MAX_LOCAL_CPU_SIZE=0", "LMCACHE_STORAGE_PLUGINS": "LMCACHE_STORAGE_PLUGINS=RDSBackend", "PYTHONHASHSEED": "PYTHONHASHSEED=42"}, "del_args": [], "del_envs": []}}, "prefix_caching=on": {"main": {"add_args": {}, "add_envs": {"VLLM_RBLN_SUB_BLOCK_CACHE": "VLLM_RBLN_SUB_BLOCK_CACHE=0"}, "del_args": ["--no-enable-prefix-caching"], "del_envs": []}}, "prefix_caching=on_sub_block": {"main": {"add_args": {}, "add_envs": {}, "del_args": ["--no-enable-prefix-caching"], "del_envs": []}}, "reasoning_parser=off": {"main": {"add_args": {}, "add_envs": {}, "del_args": ["--reasoning-parser"], "del_envs": []}}, "spec_decode=eagle3": {"main": {"add_args": {"--speculative-config": "--speculative-config \u0027{\n      \"method\": \"eagle3\",\n      \"model\": \"skt/A.X-K2-EAGLE3\",\n      \"revision\": \"24958e91737d760908f73a8af4b6e06080fc5c1d\",\n      \"num_speculative_tokens\": 3\n    }\u0027"}, "add_envs": {}, "del_args": [], "del_envs": []}}, "tool_calling=off": {"main": {"add_args": {}, "add_envs": {}, "del_args": ["--enable-auto-tool-choice", "--tool-call-parser"], "del_envs": []}}}, "excluded": [], "model": "skt/A.X-K2-NVFP4", "modes": {"lmcache": ["off", "cpu", "rds"], "prefix_caching": ["off", "on", "on_sub_block"], "spec_decode": ["off", "eagle3"]}, "option_base": {"reasoning_parser": "on", "tool_calling": "on"}, "options": {"reasoning_parser": ["off", "on"], "tool_calling": ["off", "on"]}, "recommended": {"lmcache": "rds", "prefix_caching": "on_sub_block", "reasoning_parser": "on", "spec_decode": "eagle3", "tool_calling": "on"}, "router": null}, "label": null, "modelImpl": "vllm", "note": null, "npus": 8, "skus": ["RBLN-CR13"], "target": "Latency", "verified": "2026-10-07"}], "slug": "aggregated-data-expert-parallel", "tag": "Data + Expert Parallel", "variant": "NVFP4"}, {"label": "Aggregated \u00b7 Pipeline Parallel", "nodes": null, "options": [{"attn": null, "features": {}, "grid": {"base": [{"args": {"--block-size": "--block-size 2048", "--enable-auto-tool-choice": "--enable-auto-tool-choice", "--max-model-len": "--max-model-len 262144", "--max-num-batched-tokens": "--max-num-batched-tokens 512", "--max-num-seqs": "--max-num-seqs 8", "--model-impl": "--model-impl vllm", "--no-enable-prefix-caching": "--no-enable-prefix-caching", "--pipeline-parallel-size": "--pipeline-parallel-size 8", "--prefix-caching-hash-algo": "--prefix-caching-hash-algo sha256_cbor", "--reasoning-parser": "--reasoning-parser deepseek_v3", "--revision": "--revision 9e2e804e80f8d1b3afba5d7938173cec1ed46b49", "--safetensors-load-strategy": "--safetensors-load-strategy prefetch", "--tool-call-parser": "--tool-call-parser hermes", "--trust-remote-code": "--trust-remote-code"}, "envs": {"VLLM_ENGINE_READY_TIMEOUT_S": "VLLM_ENGINE_READY_TIMEOUT_S=9600", "VLLM_WORKER_MULTIPROC_METHOD": "VLLM_WORKER_MULTIPROC_METHOD=fork"}, "launcher": "vllm serve", "ranks": null, "role": "main", "scale": null, "varies": null}], "deltas": {"lmcache=cpu": {"main": {"add_args": {"--kv-transfer-config": "--kv-transfer-config \u0027{\n      \"kv_connector\": \"RBLNLMCacheConnectorV1\",\n      \"kv_role\": \"kv_both\"\n    }\u0027"}, "add_envs": {"LMCACHE_CHUNK_SIZE": "LMCACHE_CHUNK_SIZE=256", "LMCACHE_MAX_LOCAL_CPU_SIZE": "LMCACHE_MAX_LOCAL_CPU_SIZE=10.0", "PYTHONHASHSEED": "PYTHONHASHSEED=42"}, "del_args": [], "del_envs": []}}, "lmcache=rds": {"main": {"add_args": {"--gpu-memory-utilization": "--gpu-memory-utilization 0.85", "--kv-transfer-config": "--kv-transfer-config \u0027{\n      \"kv_connector\": \"RBLNLMCacheConnectorV1\",\n      \"kv_role\": \"kv_both\"\n    }\u0027"}, "add_envs": {"LMCACHE_CHUNK_SIZE": "LMCACHE_CHUNK_SIZE=2048", "LMCACHE_EXTRA_CONFIG": "LMCACHE_EXTRA_CONFIG=\u0027{\n  \"storage_plugin.RDSBackend.module_path\": \"lmcache_rbln.storage_plugin.rds_backend\",\n  \"storage_plugin.RDSBackend.class_name\": \"RDSBackend\",\n  \"rds_max_chunks\": 800\n}\u0027", "LMCACHE_LOCAL_CPU": "LMCACHE_LOCAL_CPU=False", "LMCACHE_MAX_LOCAL_CPU_SIZE": "LMCACHE_MAX_LOCAL_CPU_SIZE=0", "LMCACHE_STORAGE_PLUGINS": "LMCACHE_STORAGE_PLUGINS=RDSBackend", "PYTHONHASHSEED": "PYTHONHASHSEED=42"}, "del_args": [], "del_envs": []}}, "prefix_caching=on": {"main": {"add_args": {}, "add_envs": {"VLLM_RBLN_SUB_BLOCK_CACHE": "VLLM_RBLN_SUB_BLOCK_CACHE=0"}, "del_args": ["--no-enable-prefix-caching"], "del_envs": []}}, "prefix_caching=on_sub_block": {"main": {"add_args": {}, "add_envs": {}, "del_args": ["--no-enable-prefix-caching"], "del_envs": []}}, "reasoning_parser=off": {"main": {"add_args": {}, "add_envs": {}, "del_args": ["--reasoning-parser"], "del_envs": []}}, "spec_decode=eagle3": {"main": {"add_args": {"--speculative-config": "--speculative-config \u0027{\n      \"method\": \"eagle3\",\n      \"model\": \"skt/A.X-K2-EAGLE3\",\n      \"revision\": \"24958e91737d760908f73a8af4b6e06080fc5c1d\",\n      \"num_speculative_tokens\": 3\n    }\u0027"}, "add_envs": {}, "del_args": [], "del_envs": []}}, "tool_calling=off": {"main": {"add_args": {}, "add_envs": {}, "del_args": ["--enable-auto-tool-choice", "--tool-call-parser"], "del_envs": []}}}, "excluded": [], "model": "skt/A.X-K2-NVFP4", "modes": {"lmcache": ["off", "cpu", "rds"], "prefix_caching": ["off", "on", "on_sub_block"], "spec_decode": ["off", "eagle3"]}, "option_base": {"reasoning_parser": "on", "tool_calling": "on"}, "options": {"reasoning_parser": ["off", "on"], "tool_calling": ["off", "on"]}, "recommended": null, "router": null}, "label": null, "modelImpl": "vllm", "note": null, "npus": 8, "skus": ["RBLN-CR13"], "target": "Throughput", "verified": "2026-10-07"}], "slug": "aggregated-pipeline-parallel", "tag": "Pipeline Parallel", "variant": "NVFP4"}, {"label": "Prefill/Decode disaggregated", "nodes": "2", "options": [{"attn": null, "features": {}, "grid": {"base": [{"args": {"--block-size": "--block-size 2048", "--enable-auto-tool-choice": "--enable-auto-tool-choice", "--kv-transfer-config": "--kv-transfer-config \u0027{\n      \"kv_connector\": \"RblnNixlConnector\",\n      \"kv_role\": \"kv_both\",\n      \"kv_buffer_device\": \"rbln\",\n      \"kv_connector_extra_config\": {\n        \"name\": \"nixl\"\n      }\n    }\u0027", "--max-model-len": "--max-model-len 262144", "--max-num-batched-tokens": "--max-num-batched-tokens 512", "--max-num-seqs": "--max-num-seqs 8", "--model-impl": "--model-impl vllm", "--no-enable-prefix-caching": "--no-enable-prefix-caching", "--pipeline-parallel-size": "--pipeline-parallel-size 8", "--prefix-caching-hash-algo": "--prefix-caching-hash-algo sha256_cbor", "--reasoning-parser": "--reasoning-parser deepseek_v3", "--revision": "--revision 9e2e804e80f8d1b3afba5d7938173cec1ed46b49", "--safetensors-load-strategy": "--safetensors-load-strategy prefetch", "--tool-call-parser": "--tool-call-parser hermes", "--trust-remote-code": "--trust-remote-code"}, "envs": {"VLLM_ENGINE_READY_TIMEOUT_S": "VLLM_ENGINE_READY_TIMEOUT_S=9600", "VLLM_WORKER_MULTIPROC_METHOD": "VLLM_WORKER_MULTIPROC_METHOD=fork"}, "launcher": "vllm serve", "ranks": [{"args": {"--kv-events-config": "--kv-events-config \u0027{\n      \"enable_kv_cache_events\": true,\n      \"publisher\": \"zmq\",\n      \"endpoint\": \"tcp://\u003cEPP_HOST\u003e:5557\",\n      \"topic\": \"kv@\u003cPOD_IP\u003e:8000@skt/A.X-K2-NVFP4\"\n    }\u0027", "--port": "--port 8000"}, "envs": {"VLLM_NIXL_SIDE_CHANNEL_HOST": "VLLM_NIXL_SIDE_CHANNEL_HOST=\u0027\u003cPOD_IP\u003e\u0027", "VLLM_NIXL_SIDE_CHANNEL_PORT": "VLLM_NIXL_SIDE_CHANNEL_PORT=5559"}}], "role": "prefill", "scale": "1 ranks", "varies": []}, {"args": {"--block-size": "--block-size 2048", "--data-parallel-address": "--data-parallel-address \u0027\u003cPOD_IP\u003e\u0027", "--data-parallel-rpc-port": "--data-parallel-rpc-port 5555", "--data-parallel-size": "--data-parallel-size 8", "--enable-auto-tool-choice": "--enable-auto-tool-choice", "--enable-expert-parallel": "--enable-expert-parallel", "--kv-transfer-config": "--kv-transfer-config \u0027{\n      \"kv_connector\": \"RblnNixlConnector\",\n      \"kv_role\": \"kv_both\",\n      \"kv_buffer_device\": \"rbln\",\n      \"kv_connector_extra_config\": {\n        \"name\": \"nixl\"\n      }\n    }\u0027", "--max-model-len": "--max-model-len 262144", "--max-num-batched-tokens": "--max-num-batched-tokens 512", "--max-num-seqs": "--max-num-seqs 1", "--model-impl": "--model-impl vllm", "--no-enable-prefix-caching": "--no-enable-prefix-caching", "--prefix-caching-hash-algo": "--prefix-caching-hash-algo sha256_cbor", "--reasoning-parser": "--reasoning-parser deepseek_v3", "--revision": "--revision 9e2e804e80f8d1b3afba5d7938173cec1ed46b49", "--safetensors-load-strategy": "--safetensors-load-strategy prefetch", "--tool-call-parser": "--tool-call-parser hermes", "--trust-remote-code": "--trust-remote-code"}, "envs": {"VLLM_ENGINE_READY_TIMEOUT_S": "VLLM_ENGINE_READY_TIMEOUT_S=9600", "VLLM_WORKER_MULTIPROC_METHOD": "VLLM_WORKER_MULTIPROC_METHOD=fork"}, "launcher": "vllm serve", "ranks": [{"args": {"--data-parallel-rank": "--data-parallel-rank 0", "--kv-events-config": "--kv-events-config \u0027{\n      \"enable_kv_cache_events\": true,\n      \"publisher\": \"zmq\",\n      \"endpoint\": \"tcp://\u003cEPP_HOST\u003e:5557\",\n      \"topic\": \"kv@\u003cPOD_IP\u003e:8200@skt/A.X-K2-NVFP4\"\n    }\u0027", "--port": "--port 8200"}, "envs": {"RBLN_VISIBLE_DEVICES": "RBLN_VISIBLE_DEVICES=0", "VLLM_NIXL_SIDE_CHANNEL_HOST": "VLLM_NIXL_SIDE_CHANNEL_HOST=\u0027\u003cPOD_IP\u003e\u0027", "VLLM_NIXL_SIDE_CHANNEL_PORT": "VLLM_NIXL_SIDE_CHANNEL_PORT=5659"}}, {"args": {"--data-parallel-rank": "--data-parallel-rank 1", "--kv-events-config": "--kv-events-config \u0027{\n      \"enable_kv_cache_events\": true,\n      \"publisher\": \"zmq\",\n      \"endpoint\": \"tcp://\u003cEPP_HOST\u003e:5556\",\n      \"topic\": \"kv@\u003cPOD_IP\u003e:8201@skt/A.X-K2-NVFP4\"\n    }\u0027", "--port": "--port 8201"}, "envs": {"RBLN_VISIBLE_DEVICES": "RBLN_VISIBLE_DEVICES=1", "VLLM_NIXL_SIDE_CHANNEL_HOST": "VLLM_NIXL_SIDE_CHANNEL_HOST=\u0027\u003cPOD_IP\u003e\u0027", "VLLM_NIXL_SIDE_CHANNEL_PORT": "VLLM_NIXL_SIDE_CHANNEL_PORT=5660"}}, {"args": {"--data-parallel-rank": "--data-parallel-rank 2", "--kv-events-config": "--kv-events-config \u0027{\n      \"enable_kv_cache_events\": true,\n      \"publisher\": \"zmq\",\n      \"endpoint\": \"tcp://\u003cEPP_HOST\u003e:5555\",\n      \"topic\": \"kv@\u003cPOD_IP\u003e:8202@skt/A.X-K2-NVFP4\"\n    }\u0027", "--port": "--port 8202"}, "envs": {"RBLN_VISIBLE_DEVICES": "RBLN_VISIBLE_DEVICES=2", "VLLM_NIXL_SIDE_CHANNEL_HOST": "VLLM_NIXL_SIDE_CHANNEL_HOST=\u0027\u003cPOD_IP\u003e\u0027", "VLLM_NIXL_SIDE_CHANNEL_PORT": "VLLM_NIXL_SIDE_CHANNEL_PORT=5661"}}, {"args": {"--data-parallel-rank": "--data-parallel-rank 3", "--kv-events-config": "--kv-events-config \u0027{\n      \"enable_kv_cache_events\": true,\n      \"publisher\": \"zmq\",\n      \"endpoint\": \"tcp://\u003cEPP_HOST\u003e:5554\",\n      \"topic\": \"kv@\u003cPOD_IP\u003e:8203@skt/A.X-K2-NVFP4\"\n    }\u0027", "--port": "--port 8203"}, "envs": {"RBLN_VISIBLE_DEVICES": "RBLN_VISIBLE_DEVICES=3", "VLLM_NIXL_SIDE_CHANNEL_HOST": "VLLM_NIXL_SIDE_CHANNEL_HOST=\u0027\u003cPOD_IP\u003e\u0027", "VLLM_NIXL_SIDE_CHANNEL_PORT": "VLLM_NIXL_SIDE_CHANNEL_PORT=5662"}}, {"args": {"--data-parallel-rank": "--data-parallel-rank 4", "--kv-events-config": "--kv-events-config \u0027{\n      \"enable_kv_cache_events\": true,\n      \"publisher\": \"zmq\",\n      \"endpoint\": \"tcp://\u003cEPP_HOST\u003e:5553\",\n      \"topic\": \"kv@\u003cPOD_IP\u003e:8204@skt/A.X-K2-NVFP4\"\n    }\u0027", "--port": "--port 8204"}, "envs": {"RBLN_VISIBLE_DEVICES": "RBLN_VISIBLE_DEVICES=4", "VLLM_NIXL_SIDE_CHANNEL_HOST": "VLLM_NIXL_SIDE_CHANNEL_HOST=\u0027\u003cPOD_IP\u003e\u0027", "VLLM_NIXL_SIDE_CHANNEL_PORT": "VLLM_NIXL_SIDE_CHANNEL_PORT=5663"}}, {"args": {"--data-parallel-rank": "--data-parallel-rank 5", "--kv-events-config": "--kv-events-config \u0027{\n      \"enable_kv_cache_events\": true,\n      \"publisher\": \"zmq\",\n      \"endpoint\": \"tcp://\u003cEPP_HOST\u003e:5552\",\n      \"topic\": \"kv@\u003cPOD_IP\u003e:8205@skt/A.X-K2-NVFP4\"\n    }\u0027", "--port": "--port 8205"}, "envs": {"RBLN_VISIBLE_DEVICES": "RBLN_VISIBLE_DEVICES=5", "VLLM_NIXL_SIDE_CHANNEL_HOST": "VLLM_NIXL_SIDE_CHANNEL_HOST=\u0027\u003cPOD_IP\u003e\u0027", "VLLM_NIXL_SIDE_CHANNEL_PORT": "VLLM_NIXL_SIDE_CHANNEL_PORT=5664"}}, {"args": {"--data-parallel-rank": "--data-parallel-rank 6", "--kv-events-config": "--kv-events-config \u0027{\n      \"enable_kv_cache_events\": true,\n      \"publisher\": \"zmq\",\n      \"endpoint\": \"tcp://\u003cEPP_HOST\u003e:5551\",\n      \"topic\": \"kv@\u003cPOD_IP\u003e:8206@skt/A.X-K2-NVFP4\"\n    }\u0027", "--port": "--port 8206"}, "envs": {"RBLN_VISIBLE_DEVICES": "RBLN_VISIBLE_DEVICES=6", "VLLM_NIXL_SIDE_CHANNEL_HOST": "VLLM_NIXL_SIDE_CHANNEL_HOST=\u0027\u003cPOD_IP\u003e\u0027", "VLLM_NIXL_SIDE_CHANNEL_PORT": "VLLM_NIXL_SIDE_CHANNEL_PORT=5665"}}, {"args": {"--data-parallel-rank": "--data-parallel-rank 7", "--kv-events-config": "--kv-events-config \u0027{\n      \"enable_kv_cache_events\": true,\n      \"publisher\": \"zmq\",\n      \"endpoint\": \"tcp://\u003cEPP_HOST\u003e:5550\",\n      \"topic\": \"kv@\u003cPOD_IP\u003e:8207@skt/A.X-K2-NVFP4\"\n    }\u0027", "--port": "--port 8207"}, "envs": {"RBLN_VISIBLE_DEVICES": "RBLN_VISIBLE_DEVICES=7", "VLLM_NIXL_SIDE_CHANNEL_HOST": "VLLM_NIXL_SIDE_CHANNEL_HOST=\u0027\u003cPOD_IP\u003e\u0027", "VLLM_NIXL_SIDE_CHANNEL_PORT": "VLLM_NIXL_SIDE_CHANNEL_PORT=5666"}}], "role": "decode", "scale": "8 ranks", "varies": ["--data-parallel-rank", "--kv-events-config", "--port", "RBLN_VISIBLE_DEVICES", "VLLM_NIXL_SIDE_CHANNEL_PORT"]}], "deltas": {"lmcache=cpu": {"decode": {"add_args": {}, "add_envs": {}, "del_args": [], "del_envs": []}, "prefill": {"add_args": {"--kv-transfer-config": "--kv-transfer-config \u0027{\n      \"kv_connector\": \"MultiConnector\",\n      \"kv_role\": \"kv_both\",\n      \"kv_connector_extra_config\": {\n        \"connectors\": [\n          {\n            \"kv_connector\": \"RblnNixlConnector\",\n            \"kv_role\": \"kv_both\",\n            \"kv_buffer_device\": \"rbln\",\n            \"kv_connector_extra_config\": {\n              \"name\": \"nixl\"\n            }\n          },\n          {\n            \"kv_connector\": \"RBLNLMCacheConnectorV1\",\n            \"kv_connector_module_path\": \"lmcache_rbln.integration.vllm.connector\",\n            \"kv_role\": \"kv_both\",\n            \"kv_connector_extra_config\": {\n              \"name\": \"lmcache\"\n            }\n          }\n        ]\n      }\n    }\u0027"}, "add_envs": {"LMCACHE_CHUNK_SIZE": "LMCACHE_CHUNK_SIZE=256", "LMCACHE_ENABLE_KV_EVENTS": "LMCACHE_ENABLE_KV_EVENTS=True", "LMCACHE_MAX_LOCAL_CPU_SIZE": "LMCACHE_MAX_LOCAL_CPU_SIZE=10.0", "LMCACHE_PRE_CACHING_HASH_ALGORITHM": "LMCACHE_PRE_CACHING_HASH_ALGORITHM=sha256_cbor_64bit", "PYTHONHASHSEED": "PYTHONHASHSEED=42"}, "del_args": [], "del_envs": []}}, "lmcache=rds": {"decode": {"add_args": {}, "add_envs": {}, "del_args": [], "del_envs": []}, "prefill": {"add_args": {"--gpu-memory-utilization": "--gpu-memory-utilization 0.85", "--kv-transfer-config": "--kv-transfer-config \u0027{\n      \"kv_connector\": \"MultiConnector\",\n      \"kv_role\": \"kv_both\",\n      \"kv_connector_extra_config\": {\n        \"connectors\": [\n          {\n            \"kv_connector\": \"RblnNixlConnector\",\n            \"kv_role\": \"kv_both\",\n            \"kv_buffer_device\": \"rbln\",\n            \"kv_connector_extra_config\": {\n              \"name\": \"nixl\"\n            }\n          },\n          {\n            \"kv_connector\": \"RBLNLMCacheConnectorV1\",\n            \"kv_connector_module_path\": \"lmcache_rbln.integration.vllm.connector\",\n            \"kv_role\": \"kv_both\",\n            \"kv_connector_extra_config\": {\n              \"name\": \"lmcache\"\n            }\n          }\n        ]\n      }\n    }\u0027"}, "add_envs": {"LMCACHE_CHUNK_SIZE": "LMCACHE_CHUNK_SIZE=2048", "LMCACHE_ENABLE_KV_EVENTS": "LMCACHE_ENABLE_KV_EVENTS=True", "LMCACHE_EXTRA_CONFIG": "LMCACHE_EXTRA_CONFIG=\u0027{\n  \"storage_plugin.RDSBackend.module_path\": \"lmcache_rbln.storage_plugin.rds_backend\",\n  \"storage_plugin.RDSBackend.class_name\": \"RDSBackend\",\n  \"rds_max_chunks\": 800\n}\u0027", "LMCACHE_LOCAL_CPU": "LMCACHE_LOCAL_CPU=False", "LMCACHE_MAX_LOCAL_CPU_SIZE": "LMCACHE_MAX_LOCAL_CPU_SIZE=0", "LMCACHE_PRE_CACHING_HASH_ALGORITHM": "LMCACHE_PRE_CACHING_HASH_ALGORITHM=sha256_cbor_64bit", "LMCACHE_STORAGE_PLUGINS": "LMCACHE_STORAGE_PLUGINS=RDSBackend", "PYTHONHASHSEED": "PYTHONHASHSEED=42"}, "del_args": [], "del_envs": []}}, "prefix_caching=on": {"decode": {"add_args": {}, "add_envs": {"VLLM_RBLN_SUB_BLOCK_CACHE": "VLLM_RBLN_SUB_BLOCK_CACHE=0"}, "del_args": ["--no-enable-prefix-caching"], "del_envs": []}, "prefill": {"add_args": {}, "add_envs": {"VLLM_RBLN_SUB_BLOCK_CACHE": "VLLM_RBLN_SUB_BLOCK_CACHE=0"}, "del_args": ["--no-enable-prefix-caching"], "del_envs": []}}, "prefix_caching=on_sub_block": {"decode": {"add_args": {}, "add_envs": {}, "del_args": ["--no-enable-prefix-caching"], "del_envs": []}, "prefill": {"add_args": {}, "add_envs": {}, "del_args": ["--no-enable-prefix-caching"], "del_envs": []}}, "reasoning_parser=off": {"decode": {"add_args": {}, "add_envs": {}, "del_args": ["--reasoning-parser"], "del_envs": []}, "prefill": {"add_args": {}, "add_envs": {}, "del_args": ["--reasoning-parser"], "del_envs": []}}, "spec_decode=eagle3": {"decode": {"add_args": {"--speculative-config": "--speculative-config \u0027{\n      \"method\": \"eagle3\",\n      \"model\": \"skt/A.X-K2-EAGLE3\",\n      \"revision\": \"24958e91737d760908f73a8af4b6e06080fc5c1d\",\n      \"num_speculative_tokens\": 3\n    }\u0027"}, "add_envs": {}, "del_args": [], "del_envs": []}, "prefill": {"add_args": {"--speculative-config": "--speculative-config \u0027{\n      \"method\": \"eagle3\",\n      \"model\": \"skt/A.X-K2-EAGLE3\",\n      \"revision\": \"24958e91737d760908f73a8af4b6e06080fc5c1d\",\n      \"num_speculative_tokens\": 3\n    }\u0027"}, "add_envs": {}, "del_args": [], "del_envs": []}}, "tool_calling=off": {"decode": {"add_args": {}, "add_envs": {}, "del_args": ["--enable-auto-tool-choice", "--tool-call-parser"], "del_envs": []}, "prefill": {"add_args": {}, "add_envs": {}, "del_args": ["--enable-auto-tool-choice", "--tool-call-parser"], "del_envs": []}}}, "excluded": [], "model": "skt/A.X-K2-NVFP4", "modes": {"lmcache": ["off", "cpu", "rds"], "prefix_caching": ["off", "on", "on_sub_block"], "spec_decode": ["off", "eagle3"]}, "option_base": {"reasoning_parser": "on", "tool_calling": "on"}, "options": {"reasoning_parser": ["off", "on"], "tool_calling": ["off", "on"]}, "recommended": {"lmcache": "rds", "prefix_caching": "on_sub_block", "reasoning_parser": "on", "spec_decode": "eagle3", "tool_calling": "on"}, "router": {"nodes": [{"cmd": "cat \u003e router-values.yaml \u003c\u003c\u0027EOF\u0027\nendpoint_picker:\n  image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.10.0\n  ext_proc_port: 9002\n  plugins:\n  - type: prefix-based-pd-decider\n    parameters:\n      nonCachedTokens: 1\n  - type: disagg-profile-handler\n    parameters:\n      deciders:\n        prefill: prefix-based-pd-decider\n  - type: prefill-filter\n  - type: decode-filter\n  - type: approx-prefix-cache-producer\n    parameters:\n      maxPrefixTokensToMatch: 131072\n      lruCapacityPerServer: 1024\n  - type: prefix-cache-scorer\n    parameters:\n      prefixMatchInfoProducerName: approx-prefix-cache-producer\n  - type: queue-scorer\n  - type: max-score-picker\n  scheduling_profiles:\n  - name: prefill\n    plugins:\n    - ref: prefill-filter\n    - ref: max-score-picker\n    - ref: prefix-cache-scorer\n      weight: 2\n    - ref: queue-scorer\n      weight: 1\n  - name: decode\n    plugins:\n    - ref: decode-filter\n    - ref: max-score-picker\n    - ref: prefix-cache-scorer\n      weight: 2\n    - ref: queue-scorer\n      weight: 1\ninference_pool:\n  failure_mode: FailOpen\n  match_labels:\n    llm-d.ai/guide: pd-disaggregation\n  target_ports:\n  - 8000\n  - 8001\n  - 8002\n  - 8003\n  - 8004\n  - 8005\n  - 8006\n  - 8007\nEOF", "node": "router-values.yaml"}, {"cmd": "helm install router llm-d-router-gateway \\\n    --version v0.10.0 \\\n    --values router-values.yaml", "node": "helm"}], "role": "router"}}, "label": null, "modelImpl": "vllm", "note": null, "npus": 8, "skus": ["RBLN-CR13"], "target": "Latency", "verified": "2026-10-07"}], "slug": "prefill-decode-disaggregated", "tag": "P Pipeline / D Data + Expert", "variant": "NVFP4"}], "text": {"en": {"strategyNote": "Data + Expert Parallel replicates the dense path per device (\u003ccode\u003e--data-parallel-size\u003c/code\u003e) and shards experts (\u003ccode\u003e--enable-expert-parallel\u003c/code\u003e). Pipeline Parallel splits the layers across devices (\u003ccode\u003e--pipeline-parallel-size\u003c/code\u003e). See \u003cstrong\u003eConfiguration\u003c/strong\u003e in the guide for the engine arguments.", "sub": "A.X-K2 \u2014 Mixture-of-Experts LLM served on RBLN-CR13 with RBLN vLLM (NVFP4), 262,144-token context.", "tag": "Aggregated and prefill/decode-disaggregated vllm serve, verified on RBLN-CR13."}, "ko": {"strategyNote": "Data + Expert Parallel\uc740 dense \uacbd\ub85c\ub97c \ub514\ubc14\uc774\uc2a4\ub9c8\ub2e4 \ubcf5\uc81c\ud558\uace0(\u003ccode\u003e--data-parallel-size\u003c/code\u003e) expert\ub97c \ubd84\ud560\ud569\ub2c8\ub2e4(\u003ccode\u003e--enable-expert-parallel\u003c/code\u003e). Pipeline Parallel\uc740 layer\ub97c \ub514\ubc14\uc774\uc2a4\uc5d0 \ub098\ub220 \uc62c\ub9bd\ub2c8\ub2e4(\u003ccode\u003e--pipeline-parallel-size\u003c/code\u003e). \uc5d4\uc9c4 \uc778\uc790\ub294 \uac00\uc774\ub4dc\uc758 \u003cstrong\u003eConfiguration\u003c/strong\u003e\uc5d0\uc11c \ud655\uc778\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4.", "sub": "A.X-K2 \u2014 mixture-of-experts(MoE) LLM, RBLN-CR13\uc5d0\uc11c RBLN vLLM\uc73c\ub85c \uc11c\ube59(NVFP4). 262,144 \ud1a0\ud070 \ucee8\ud14d\uc2a4\ud2b8.", "tag": "RBLN-CR13\uc73c\ub85c \uac80\uc99d\ub41c aggregated \ubc0f prefill/decode disaggregated vllm serve \uad6c\uc131\uc785\ub2c8\ub2e4."}}, "variant": "NVFP4", "variants": ["FP8", "NVFP4"]}

호스트에 RBLN Driver가 설치되어 있어야 하며, RBLN Compiler(rebel-compiler)를 설치하려면 RBLN Portal 계정이 필요합니다. 호환되는 패키지 버전은 설치 매트릭스에서 확인할 수 있습니다.

$ pip install \
  --extra-index-url https://pypi.rbln.ai/simple \
  rebel-compiler==0.12.0

KV 오프로드(LMCache)와 prefill/decode disaggregated 명령에는 lmcache-rbln과 nixl-rbln을 설치하는 kv_connectors extra가 필요합니다. 없으면 해당 서버가 시작 단계에서 실패합니다.

$ pip install \
  --extra-index-url https://pypi.rbln.ai/simple \
  --extra-index-url https://wheels.vllm.ai/0.30.0/cpu \
  --extra-index-url https://download.pytorch.org/whl/cpu \
  "vllm-rbln[kv_connectors]==0.12.0"

가이드 — 개요 및 설정

개요¶

skt/A.X-K2는 688B 파라미터 Mixture-of-Experts 디코더 전용 모델입니다(routed expert 256개 + 공유 expert 1개, 토큰당 8개 활성화). Multi-head Latent Attention(MLA)과 Sparse Gated Attention(SGA)을 함께 사용합니다. RBLN-CR13에서는 RBLN vLLM 플러그인으로 서빙되며, aggregated 또는 prefill/decode disaggregated 구성을 고를 수 있습니다. 서빙되는 체크포인트는 NVFP4로 양자화된 skt/A.X-K2-NVFP4이며, 262,144 토큰 컨텍스트 전체를 서빙합니다.

Configuration¶

RBLN-CR13으로 검증된 구성입니다. 위 recipe에서 strategy를 고르면 됩니다. 서버 하나가 디바이스 8개를 씁니다.

Data + Expert Parallel: --data-parallel-size는 모델 복제를 하나씩 올릴 CR13 디바이스 수이고, --enable-expert-parallel이 expert를 그 디바이스들에 분할합니다.
Pipeline Parallel: --pipeline-parallel-size가 layer를 디바이스에 나눠 올립니다.
Prefill/Decode disaggregated: prefill 서버와 decode 서버를 서로 다른 노드에서 실행하고 llm-d router 뒤에 둡니다. router의 helm 명령은 recipe에서 두 서버 명령 다음에 나옵니다. Kubernetes에서 llm-d router와 serving stack을 배포하는 방법은 Coding Assistant reference design의 Kubernetes 배포를 참고하세요.

모든 명령은 --model-impl vllm으로 선택하는 vllm model implementation으로 동작합니다. 커스텀 아키텍처라 --trust-remote-code가 필요하며, 모든 명령에 들어 있습니다.

전체 엔진 인자는 설정 가이드에서 확인할 수 있습니다.

참고 및 제약¶

RBLN-CR13으로 검증되었으며, serve 인자는 변경될 수 있습니다.
이 recipe에는 FP8 원본 체크포인트(skt/A.X-K2)를 사용하는 설정이 포함되어 있지 않습니다. 262,144 토큰에서는 --block-size 2048 기준 KV cache가 rank당 21.1 GiB입니다. RBLN-CR13 디바이스는 이를 weight와 함께 담지 못해 서버가 SYS_EBUSY: Lack of device memory로 실패합니다. 컨텍스트를 줄이면 서빙할 수 있습니다. --max-model-len 163840에서는 KV cache가 rank당 13.3 GiB입니다.
배치 크기·시퀀스 길이·디바이스 수는 컴파일 시 고정되며, 변경하면 다음 실행 시 재컴파일됩니다.