DeepSeek-V3.2 on RBLN NPU¶
{"attn": "Sparse MLA", "axes": [{"axis": "lmcache", "options": [{"label": "Off", "value": "off"}, {"label": "LMCache (CPU)", "value": "cpu"}, {"label": "LMCache (RDS)", "value": "rds"}], "title": "KV Offload"}, {"axis": "kv_connector", "options": [{"label": "Off", "value": "off"}, {"label": "LMCache (CPU)", "value": "lmcache_cpu"}, {"label": "LMCache (RDS)", "value": "lmcache_rds"}], "title": "KV Offload"}, {"axis": "prefix_caching", "options": [{"label": "On", "value": "on"}, {"label": "On (sub-block)", "value": "on_sub_block"}, {"label": "Off", "value": "off"}], "title": "Prefix Caching"}, {"axis": "spec_decode", "options": [{"label": "Off", "value": "off"}, {"label": "N-gram", "value": "ngram"}, {"label": "EAGLE-3", "value": "eagle3"}, {"label": "Suffix", "value": "suffix"}, {"label": "MTP", "value": "mtp"}], "title": "Speculative Decoding"}, {"axis": "on_device_sampler", "options": [{"label": "On", "value": "on"}, {"label": "Off", "value": "off"}], "title": "On-device Sampler"}, {"axis": "structured_output", "options": [{"label": "On", "value": "on"}, {"label": "Off", "value": "off"}], "title": "Structured Output"}], "badges": ["MoE", "685B", "163,840 ctx", {"hl": true, "text": "vllm-rbln"}, "Text"], "client": "curl http://localhost:8000/v1/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -d \u0027{\n \"model\": \"deepseek-ai/DeepSeek-V3.2\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}],\n \"max_tokens\": 32\n }\u0027", "features": {"api": true, "mm": false, "text": true}, "hf": "https://huggingface.co/deepseek-ai/DeepSeek-V3.2", "installLine": "pip install rebel-compiler \"vllm-rbln[kv_connectors]\"", "model": "deepseek-ai/DeepSeek-V3.2", "name": "DeepSeek-V3.2", "provider": "deepseek-ai", "strategies": [{"label": "Aggregated \u00b7 Data + Expert Parallel", "nodes": null, "options": [{"attn": null, "features": {}, "grid": {"base": [{"args": {"--block-size": "--block-size 2048", "--data-parallel-size": "--data-parallel-size 8", "--enable-auto-tool-choice": "--enable-auto-tool-choice", "--enable-expert-parallel": "--enable-expert-parallel", "--kv-cache-dtype": "--kv-cache-dtype fp8", "--max-model-len": "--max-model-len 163840", "--max-num-batched-tokens": "--max-num-batched-tokens 512", "--max-num-seqs": "--max-num-seqs 1", "--model-impl": "--model-impl vllm", "--no-enable-prefix-caching": "--no-enable-prefix-caching", "--prefix-caching-hash-algo": "--prefix-caching-hash-algo sha256_cbor", "--reasoning-parser": "--reasoning-parser deepseek_v3", "--safetensors-load-strategy": "--safetensors-load-strategy prefetch", "--tokenizer-mode": "--tokenizer-mode deepseek_v32", "--tool-call-parser": "--tool-call-parser deepseek_v32", "--trust-remote-code": "--trust-remote-code"}, "envs": {"VLLM_ENGINE_READY_TIMEOUT_S": "VLLM_ENGINE_READY_TIMEOUT_S=9600", "VLLM_WORKER_MULTIPROC_METHOD": "VLLM_WORKER_MULTIPROC_METHOD=fork"}, "launcher": "vllm serve", "ranks": null, "role": "main", "scale": null, "varies": null}], "deltas": {"lmcache=cpu": {"main": {"add_args": {"--kv-transfer-config": "--kv-transfer-config \u0027{\n \"kv_connector\": \"RBLNLMCacheConnectorV1\",\n \"kv_role\": \"kv_both\"\n }\u0027"}, "add_envs": {"LMCACHE_CHUNK_SIZE": "LMCACHE_CHUNK_SIZE=256", "LMCACHE_MAX_LOCAL_CPU_SIZE": "LMCACHE_MAX_LOCAL_CPU_SIZE=10.0", "PYTHONHASHSEED": "PYTHONHASHSEED=42"}, "del_args": [], "del_envs": []}}, "lmcache=rds": {"main": {"add_args": {"--gpu-memory-utilization": "--gpu-memory-utilization 0.9", "--kv-transfer-config": "--kv-transfer-config \u0027{\n \"kv_connector\": \"RBLNLMCacheConnectorV1\",\n \"kv_role\": \"kv_both\"\n }\u0027"}, "add_envs": {"LMCACHE_CHUNK_SIZE": "LMCACHE_CHUNK_SIZE=2048", "LMCACHE_EXTRA_CONFIG": "LMCACHE_EXTRA_CONFIG=\u0027{\n \"storage_plugin.RDSBackend.module_path\": \"lmcache_rbln.storage_plugin.rds_backend\",\n \"storage_plugin.RDSBackend.class_name\": \"RDSBackend\",\n \"rds_max_chunks\": 300\n}\u0027", "LMCACHE_LOCAL_CPU": "LMCACHE_LOCAL_CPU=False", "LMCACHE_MAX_LOCAL_CPU_SIZE": "LMCACHE_MAX_LOCAL_CPU_SIZE=0", "LMCACHE_STORAGE_PLUGINS": "LMCACHE_STORAGE_PLUGINS=RDSBackend", "PYTHONHASHSEED": "PYTHONHASHSEED=42"}, "del_args": [], "del_envs": []}}, "prefix_caching=on": {"main": {"add_args": {}, "add_envs": {"VLLM_RBLN_SUB_BLOCK_CACHE": "VLLM_RBLN_SUB_BLOCK_CACHE=0"}, "del_args": ["--no-enable-prefix-caching"], "del_envs": []}}, "prefix_caching=on_sub_block": {"main": {"add_args": {}, "add_envs": {}, "del_args": ["--no-enable-prefix-caching"], "del_envs": []}}, "reasoning_parser=off": {"main": {"add_args": {}, "add_envs": {}, "del_args": ["--reasoning-parser"], "del_envs": []}}, "spec_decode=mtp": {"main": {"add_args": {"--speculative-config": "--speculative-config \u0027{\n \"method\": \"mtp\",\n \"num_speculative_tokens\": 3\n }\u0027"}, "add_envs": {}, "del_args": [], "del_envs": []}}, "tool_calling=off": {"main": {"add_args": {}, "add_envs": {}, "del_args": ["--enable-auto-tool-choice", "--tool-call-parser"], "del_envs": []}}}, "excluded": [], "model": "deepseek-ai/DeepSeek-V3.2", "modes": {"lmcache": ["off", "cpu", "rds"], "prefix_caching": ["off", "on", "on_sub_block"], "spec_decode": ["off", "mtp"]}, "option_base": {"reasoning_parser": "on", "tool_calling": "on"}, "options": {"reasoning_parser": ["off", "on"], "tool_calling": ["off", "on"]}, "recommended": {"lmcache": "rds", "prefix_caching": "on_sub_block", "reasoning_parser": "on", "spec_decode": "mtp", "tool_calling": "on"}, "router": null}, "label": null, "modelImpl": "vllm", "note": null, "npus": 8, "skus": ["RBLN-CR13"], "target": "Latency", "verified": "2026-10-07"}], "slug": "aggregated-data-expert-parallel", "tag": "Data + Expert Parallel", "variant": "FP8"}, {"label": "Aggregated \u00b7 Pipeline Parallel", "nodes": null, "options": [{"attn": null, "features": {}, "grid": {"base": [{"args": {"--block-size": "--block-size 2048", "--enable-auto-tool-choice": "--enable-auto-tool-choice", "--kv-cache-dtype": "--kv-cache-dtype fp8", "--max-model-len": "--max-model-len 163840", "--max-num-batched-tokens": "--max-num-batched-tokens 512", "--max-num-seqs": "--max-num-seqs 8", "--model-impl": "--model-impl vllm", "--no-enable-prefix-caching": "--no-enable-prefix-caching", "--pipeline-parallel-size": "--pipeline-parallel-size 8", "--prefix-caching-hash-algo": "--prefix-caching-hash-algo sha256_cbor", "--reasoning-parser": "--reasoning-parser deepseek_v3", "--safetensors-load-strategy": "--safetensors-load-strategy prefetch", "--tokenizer-mode": "--tokenizer-mode deepseek_v32", "--tool-call-parser": "--tool-call-parser deepseek_v32", "--trust-remote-code": "--trust-remote-code"}, "envs": {"VLLM_ENGINE_READY_TIMEOUT_S": "VLLM_ENGINE_READY_TIMEOUT_S=9600", "VLLM_WORKER_MULTIPROC_METHOD": "VLLM_WORKER_MULTIPROC_METHOD=fork"}, "launcher": "vllm serve", "ranks": null, "role": "main", "scale": null, "varies": null}], "deltas": {"lmcache=cpu": {"main": {"add_args": {"--kv-transfer-config": "--kv-transfer-config \u0027{\n \"kv_connector\": \"RBLNLMCacheConnectorV1\",\n \"kv_role\": \"kv_both\"\n }\u0027"}, "add_envs": {"LMCACHE_CHUNK_SIZE": "LMCACHE_CHUNK_SIZE=256", "LMCACHE_MAX_LOCAL_CPU_SIZE": "LMCACHE_MAX_LOCAL_CPU_SIZE=10.0", "PYTHONHASHSEED": "PYTHONHASHSEED=42"}, "del_args": [], "del_envs": []}}, "lmcache=rds": {"main": {"add_args": {"--gpu-memory-utilization": "--gpu-memory-utilization 0.9", "--kv-transfer-config": "--kv-transfer-config \u0027{\n \"kv_connector\": \"RBLNLMCacheConnectorV1\",\n \"kv_role\": \"kv_both\"\n }\u0027"}, "add_envs": {"LMCACHE_CHUNK_SIZE": "LMCACHE_CHUNK_SIZE=2048", "LMCACHE_EXTRA_CONFIG": "LMCACHE_EXTRA_CONFIG=\u0027{\n \"storage_plugin.RDSBackend.module_path\": \"lmcache_rbln.storage_plugin.rds_backend\",\n \"storage_plugin.RDSBackend.class_name\": \"RDSBackend\",\n \"rds_max_chunks\": 300\n}\u0027", "LMCACHE_LOCAL_CPU": "LMCACHE_LOCAL_CPU=False", "LMCACHE_MAX_LOCAL_CPU_SIZE": "LMCACHE_MAX_LOCAL_CPU_SIZE=0", "LMCACHE_STORAGE_PLUGINS": "LMCACHE_STORAGE_PLUGINS=RDSBackend", "PYTHONHASHSEED": "PYTHONHASHSEED=42"}, "del_args": [], "del_envs": []}}, "prefix_caching=on": {"main": {"add_args": {}, "add_envs": {"VLLM_RBLN_SUB_BLOCK_CACHE": "VLLM_RBLN_SUB_BLOCK_CACHE=0"}, "del_args": ["--no-enable-prefix-caching"], "del_envs": []}}, "prefix_caching=on_sub_block": {"main": {"add_args": {}, "add_envs": {}, "del_args": ["--no-enable-prefix-caching"], "del_envs": []}}, "reasoning_parser=off": {"main": {"add_args": {}, "add_envs": {}, "del_args": ["--reasoning-parser"], "del_envs": []}}, "spec_decode=mtp": {"main": {"add_args": {"--speculative-config": "--speculative-config \u0027{\n \"method\": \"mtp\",\n \"num_speculative_tokens\": 3\n }\u0027"}, "add_envs": {}, "del_args": [], "del_envs": []}}, "tool_calling=off": {"main": {"add_args": {}, "add_envs": {}, "del_args": ["--enable-auto-tool-choice", "--tool-call-parser"], "del_envs": []}}}, "excluded": [], "model": "deepseek-ai/DeepSeek-V3.2", "modes": {"lmcache": ["off", "cpu", "rds"], "prefix_caching": ["off", "on", "on_sub_block"], "spec_decode": ["off", "mtp"]}, "option_base": {"reasoning_parser": "on", "tool_calling": "on"}, "options": {"reasoning_parser": ["off", "on"], "tool_calling": ["off", "on"]}, "recommended": null, "router": null}, "label": null, "modelImpl": "vllm", "note": null, "npus": 8, "skus": ["RBLN-CR13"], "target": "Throughput", "verified": "2026-10-07"}], "slug": "aggregated-pipeline-parallel", "tag": "Pipeline Parallel", "variant": "FP8"}, {"label": "Prefill/Decode disaggregated", "nodes": "2", "options": [{"attn": null, "features": {}, "grid": {"base": [{"args": {"--block-size": "--block-size 2048", "--enable-auto-tool-choice": "--enable-auto-tool-choice", "--kv-cache-dtype": "--kv-cache-dtype fp8", "--kv-transfer-config": "--kv-transfer-config \u0027{\n \"kv_connector\": \"RblnNixlConnector\",\n \"kv_role\": \"kv_both\",\n \"kv_buffer_device\": \"rbln\",\n \"kv_connector_extra_config\": {\n \"name\": \"nixl\"\n }\n }\u0027", "--max-model-len": "--max-model-len 163840", "--max-num-batched-tokens": "--max-num-batched-tokens 512", "--max-num-seqs": "--max-num-seqs 8", "--model-impl": "--model-impl vllm", "--no-enable-prefix-caching": "--no-enable-prefix-caching", "--pipeline-parallel-size": "--pipeline-parallel-size 8", "--prefix-caching-hash-algo": "--prefix-caching-hash-algo sha256_cbor", "--reasoning-parser": "--reasoning-parser deepseek_v3", "--safetensors-load-strategy": "--safetensors-load-strategy prefetch", "--tokenizer-mode": "--tokenizer-mode deepseek_v32", "--tool-call-parser": "--tool-call-parser deepseek_v32", "--trust-remote-code": "--trust-remote-code"}, "envs": {"VLLM_ENGINE_READY_TIMEOUT_S": "VLLM_ENGINE_READY_TIMEOUT_S=9600", "VLLM_WORKER_MULTIPROC_METHOD": "VLLM_WORKER_MULTIPROC_METHOD=fork"}, "launcher": "vllm serve", "ranks": [{"args": {"--kv-events-config": "--kv-events-config \u0027{\n \"enable_kv_cache_events\": true,\n \"publisher\": \"zmq\",\n \"endpoint\": \"tcp://\u003cEPP_HOST\u003e:5557\",\n \"topic\": \"kv@\u003cPOD_IP\u003e:8000@deepseek-ai/DeepSeek-V3.2\"\n }\u0027", "--port": "--port 8000"}, "envs": {"VLLM_NIXL_SIDE_CHANNEL_HOST": "VLLM_NIXL_SIDE_CHANNEL_HOST=\u0027\u003cPOD_IP\u003e\u0027", "VLLM_NIXL_SIDE_CHANNEL_PORT": "VLLM_NIXL_SIDE_CHANNEL_PORT=5559"}}], "role": "prefill", "scale": "1 ranks", "varies": []}, {"args": {"--block-size": "--block-size 2048", "--data-parallel-address": "--data-parallel-address \u0027\u003cPOD_IP\u003e\u0027", "--data-parallel-rpc-port": "--data-parallel-rpc-port 5555", "--data-parallel-size": "--data-parallel-size 8", "--enable-auto-tool-choice": "--enable-auto-tool-choice", "--enable-expert-parallel": "--enable-expert-parallel", "--kv-cache-dtype": "--kv-cache-dtype fp8", "--kv-transfer-config": "--kv-transfer-config \u0027{\n \"kv_connector\": \"RblnNixlConnector\",\n \"kv_role\": \"kv_both\",\n \"kv_buffer_device\": \"rbln\",\n \"kv_connector_extra_config\": {\n \"name\": \"nixl\"\n }\n }\u0027", "--max-model-len": "--max-model-len 163840", "--max-num-batched-tokens": "--max-num-batched-tokens 512", "--max-num-seqs": "--max-num-seqs 1", "--model-impl": "--model-impl vllm", "--no-enable-prefix-caching": "--no-enable-prefix-caching", "--prefix-caching-hash-algo": "--prefix-caching-hash-algo sha256_cbor", "--reasoning-parser": "--reasoning-parser deepseek_v3", "--safetensors-load-strategy": "--safetensors-load-strategy prefetch", "--tokenizer-mode": "--tokenizer-mode deepseek_v32", "--tool-call-parser": "--tool-call-parser deepseek_v32", "--trust-remote-code": "--trust-remote-code"}, "envs": {"VLLM_ENGINE_READY_TIMEOUT_S": "VLLM_ENGINE_READY_TIMEOUT_S=9600", "VLLM_WORKER_MULTIPROC_METHOD": "VLLM_WORKER_MULTIPROC_METHOD=fork"}, "launcher": "vllm serve", "ranks": [{"args": {"--data-parallel-rank": "--data-parallel-rank 0", "--kv-events-config": "--kv-events-config \u0027{\n \"enable_kv_cache_events\": true,\n \"publisher\": \"zmq\",\n \"endpoint\": \"tcp://\u003cEPP_HOST\u003e:5557\",\n \"topic\": \"kv@\u003cPOD_IP\u003e:8200@deepseek-ai/DeepSeek-V3.2\"\n }\u0027", "--port": "--port 8200"}, "envs": {"RBLN_VISIBLE_DEVICES": "RBLN_VISIBLE_DEVICES=0", "VLLM_NIXL_SIDE_CHANNEL_HOST": "VLLM_NIXL_SIDE_CHANNEL_HOST=\u0027\u003cPOD_IP\u003e\u0027", "VLLM_NIXL_SIDE_CHANNEL_PORT": "VLLM_NIXL_SIDE_CHANNEL_PORT=5659"}}, {"args": {"--data-parallel-rank": "--data-parallel-rank 1", "--kv-events-config": "--kv-events-config \u0027{\n \"enable_kv_cache_events\": true,\n \"publisher\": \"zmq\",\n \"endpoint\": \"tcp://\u003cEPP_HOST\u003e:5556\",\n \"topic\": \"kv@\u003cPOD_IP\u003e:8201@deepseek-ai/DeepSeek-V3.2\"\n }\u0027", "--port": "--port 8201"}, "envs": {"RBLN_VISIBLE_DEVICES": "RBLN_VISIBLE_DEVICES=1", "VLLM_NIXL_SIDE_CHANNEL_HOST": "VLLM_NIXL_SIDE_CHANNEL_HOST=\u0027\u003cPOD_IP\u003e\u0027", "VLLM_NIXL_SIDE_CHANNEL_PORT": "VLLM_NIXL_SIDE_CHANNEL_PORT=5660"}}, {"args": {"--data-parallel-rank": "--data-parallel-rank 2", "--kv-events-config": "--kv-events-config \u0027{\n \"enable_kv_cache_events\": true,\n \"publisher\": \"zmq\",\n \"endpoint\": \"tcp://\u003cEPP_HOST\u003e:5555\",\n \"topic\": \"kv@\u003cPOD_IP\u003e:8202@deepseek-ai/DeepSeek-V3.2\"\n }\u0027", "--port": "--port 8202"}, "envs": {"RBLN_VISIBLE_DEVICES": "RBLN_VISIBLE_DEVICES=2", "VLLM_NIXL_SIDE_CHANNEL_HOST": "VLLM_NIXL_SIDE_CHANNEL_HOST=\u0027\u003cPOD_IP\u003e\u0027", "VLLM_NIXL_SIDE_CHANNEL_PORT": "VLLM_NIXL_SIDE_CHANNEL_PORT=5661"}}, {"args": {"--data-parallel-rank": "--data-parallel-rank 3", "--kv-events-config": "--kv-events-config \u0027{\n \"enable_kv_cache_events\": true,\n \"publisher\": \"zmq\",\n \"endpoint\": \"tcp://\u003cEPP_HOST\u003e:5554\",\n \"topic\": \"kv@\u003cPOD_IP\u003e:8203@deepseek-ai/DeepSeek-V3.2\"\n }\u0027", "--port": "--port 8203"}, "envs": {"RBLN_VISIBLE_DEVICES": "RBLN_VISIBLE_DEVICES=3", "VLLM_NIXL_SIDE_CHANNEL_HOST": "VLLM_NIXL_SIDE_CHANNEL_HOST=\u0027\u003cPOD_IP\u003e\u0027", "VLLM_NIXL_SIDE_CHANNEL_PORT": "VLLM_NIXL_SIDE_CHANNEL_PORT=5662"}}, {"args": {"--data-parallel-rank": "--data-parallel-rank 4", "--kv-events-config": "--kv-events-config \u0027{\n \"enable_kv_cache_events\": true,\n \"publisher\": \"zmq\",\n \"endpoint\": \"tcp://\u003cEPP_HOST\u003e:5553\",\n \"topic\": \"kv@\u003cPOD_IP\u003e:8204@deepseek-ai/DeepSeek-V3.2\"\n }\u0027", "--port": "--port 8204"}, "envs": {"RBLN_VISIBLE_DEVICES": "RBLN_VISIBLE_DEVICES=4", "VLLM_NIXL_SIDE_CHANNEL_HOST": "VLLM_NIXL_SIDE_CHANNEL_HOST=\u0027\u003cPOD_IP\u003e\u0027", "VLLM_NIXL_SIDE_CHANNEL_PORT": "VLLM_NIXL_SIDE_CHANNEL_PORT=5663"}}, {"args": {"--data-parallel-rank": "--data-parallel-rank 5", "--kv-events-config": "--kv-events-config \u0027{\n \"enable_kv_cache_events\": true,\n \"publisher\": \"zmq\",\n \"endpoint\": \"tcp://\u003cEPP_HOST\u003e:5552\",\n \"topic\": \"kv@\u003cPOD_IP\u003e:8205@deepseek-ai/DeepSeek-V3.2\"\n }\u0027", "--port": "--port 8205"}, "envs": {"RBLN_VISIBLE_DEVICES": "RBLN_VISIBLE_DEVICES=5", "VLLM_NIXL_SIDE_CHANNEL_HOST": "VLLM_NIXL_SIDE_CHANNEL_HOST=\u0027\u003cPOD_IP\u003e\u0027", "VLLM_NIXL_SIDE_CHANNEL_PORT": "VLLM_NIXL_SIDE_CHANNEL_PORT=5664"}}, {"args": {"--data-parallel-rank": "--data-parallel-rank 6", "--kv-events-config": "--kv-events-config \u0027{\n \"enable_kv_cache_events\": true,\n \"publisher\": \"zmq\",\n \"endpoint\": \"tcp://\u003cEPP_HOST\u003e:5551\",\n \"topic\": \"kv@\u003cPOD_IP\u003e:8206@deepseek-ai/DeepSeek-V3.2\"\n }\u0027", "--port": "--port 8206"}, "envs": {"RBLN_VISIBLE_DEVICES": "RBLN_VISIBLE_DEVICES=6", "VLLM_NIXL_SIDE_CHANNEL_HOST": "VLLM_NIXL_SIDE_CHANNEL_HOST=\u0027\u003cPOD_IP\u003e\u0027", "VLLM_NIXL_SIDE_CHANNEL_PORT": "VLLM_NIXL_SIDE_CHANNEL_PORT=5665"}}, {"args": {"--data-parallel-rank": "--data-parallel-rank 7", "--kv-events-config": "--kv-events-config \u0027{\n \"enable_kv_cache_events\": true,\n \"publisher\": \"zmq\",\n \"endpoint\": \"tcp://\u003cEPP_HOST\u003e:5550\",\n \"topic\": \"kv@\u003cPOD_IP\u003e:8207@deepseek-ai/DeepSeek-V3.2\"\n }\u0027", "--port": "--port 8207"}, "envs": {"RBLN_VISIBLE_DEVICES": "RBLN_VISIBLE_DEVICES=7", "VLLM_NIXL_SIDE_CHANNEL_HOST": "VLLM_NIXL_SIDE_CHANNEL_HOST=\u0027\u003cPOD_IP\u003e\u0027", "VLLM_NIXL_SIDE_CHANNEL_PORT": "VLLM_NIXL_SIDE_CHANNEL_PORT=5666"}}], "role": "decode", "scale": "8 ranks", "varies": ["--data-parallel-rank", "--kv-events-config", "--port", "RBLN_VISIBLE_DEVICES", "VLLM_NIXL_SIDE_CHANNEL_PORT"]}], "deltas": {"lmcache=cpu": {"decode": {"add_args": {}, "add_envs": {}, "del_args": [], "del_envs": []}, "prefill": {"add_args": {"--kv-transfer-config": "--kv-transfer-config \u0027{\n \"kv_connector\": \"MultiConnector\",\n \"kv_role\": \"kv_both\",\n \"kv_connector_extra_config\": {\n \"connectors\": [\n {\n \"kv_connector\": \"RblnNixlConnector\",\n \"kv_role\": \"kv_both\",\n \"kv_buffer_device\": \"rbln\",\n \"kv_connector_extra_config\": {\n \"name\": \"nixl\"\n }\n },\n {\n \"kv_connector\": \"RBLNLMCacheConnectorV1\",\n \"kv_connector_module_path\": \"lmcache_rbln.integration.vllm.connector\",\n \"kv_role\": \"kv_both\",\n \"kv_connector_extra_config\": {\n \"name\": \"lmcache\"\n }\n }\n ]\n }\n }\u0027"}, "add_envs": {"LMCACHE_CHUNK_SIZE": "LMCACHE_CHUNK_SIZE=256", "LMCACHE_ENABLE_KV_EVENTS": "LMCACHE_ENABLE_KV_EVENTS=True", "LMCACHE_MAX_LOCAL_CPU_SIZE": "LMCACHE_MAX_LOCAL_CPU_SIZE=10.0", "LMCACHE_PRE_CACHING_HASH_ALGORITHM": "LMCACHE_PRE_CACHING_HASH_ALGORITHM=sha256_cbor_64bit", "PYTHONHASHSEED": "PYTHONHASHSEED=42"}, "del_args": [], "del_envs": []}}, "lmcache=rds": {"decode": {"add_args": {}, "add_envs": {}, "del_args": [], "del_envs": []}, "prefill": {"add_args": {"--gpu-memory-utilization": "--gpu-memory-utilization 0.9", "--kv-transfer-config": "--kv-transfer-config \u0027{\n \"kv_connector\": \"MultiConnector\",\n \"kv_role\": \"kv_both\",\n \"kv_connector_extra_config\": {\n \"connectors\": [\n {\n \"kv_connector\": \"RblnNixlConnector\",\n \"kv_role\": \"kv_both\",\n \"kv_buffer_device\": \"rbln\",\n \"kv_connector_extra_config\": {\n \"name\": \"nixl\"\n }\n },\n {\n \"kv_connector\": \"RBLNLMCacheConnectorV1\",\n \"kv_connector_module_path\": \"lmcache_rbln.integration.vllm.connector\",\n \"kv_role\": \"kv_both\",\n \"kv_connector_extra_config\": {\n \"name\": \"lmcache\"\n }\n }\n ]\n }\n }\u0027"}, "add_envs": {"LMCACHE_CHUNK_SIZE": "LMCACHE_CHUNK_SIZE=2048", "LMCACHE_ENABLE_KV_EVENTS": "LMCACHE_ENABLE_KV_EVENTS=True", "LMCACHE_EXTRA_CONFIG": "LMCACHE_EXTRA_CONFIG=\u0027{\n \"storage_plugin.RDSBackend.module_path\": \"lmcache_rbln.storage_plugin.rds_backend\",\n \"storage_plugin.RDSBackend.class_name\": \"RDSBackend\",\n \"rds_max_chunks\": 300\n}\u0027", "LMCACHE_LOCAL_CPU": "LMCACHE_LOCAL_CPU=False", "LMCACHE_MAX_LOCAL_CPU_SIZE": "LMCACHE_MAX_LOCAL_CPU_SIZE=0", "LMCACHE_PRE_CACHING_HASH_ALGORITHM": "LMCACHE_PRE_CACHING_HASH_ALGORITHM=sha256_cbor_64bit", "LMCACHE_STORAGE_PLUGINS": "LMCACHE_STORAGE_PLUGINS=RDSBackend", "PYTHONHASHSEED": "PYTHONHASHSEED=42"}, "del_args": [], "del_envs": []}}, "prefix_caching=on": {"decode": {"add_args": {}, "add_envs": {"VLLM_RBLN_SUB_BLOCK_CACHE": "VLLM_RBLN_SUB_BLOCK_CACHE=0"}, "del_args": ["--no-enable-prefix-caching"], "del_envs": []}, "prefill": {"add_args": {}, "add_envs": {"VLLM_RBLN_SUB_BLOCK_CACHE": "VLLM_RBLN_SUB_BLOCK_CACHE=0"}, "del_args": ["--no-enable-prefix-caching"], "del_envs": []}}, "prefix_caching=on_sub_block": {"decode": {"add_args": {}, "add_envs": {}, "del_args": ["--no-enable-prefix-caching"], "del_envs": []}, "prefill": {"add_args": {}, "add_envs": {}, "del_args": ["--no-enable-prefix-caching"], "del_envs": []}}, "reasoning_parser=off": {"decode": {"add_args": {}, "add_envs": {}, "del_args": ["--reasoning-parser"], "del_envs": []}, "prefill": {"add_args": {}, "add_envs": {}, "del_args": ["--reasoning-parser"], "del_envs": []}}, "spec_decode=mtp": {"decode": {"add_args": {"--speculative-config": "--speculative-config \u0027{\n \"method\": \"mtp\",\n \"num_speculative_tokens\": 3\n }\u0027"}, "add_envs": {}, "del_args": [], "del_envs": []}, "prefill": {"add_args": {"--speculative-config": "--speculative-config \u0027{\n \"method\": \"mtp\",\n \"num_speculative_tokens\": 3\n }\u0027"}, "add_envs": {}, "del_args": [], "del_envs": []}}, "tool_calling=off": {"decode": {"add_args": {}, "add_envs": {}, "del_args": ["--enable-auto-tool-choice", "--tool-call-parser"], "del_envs": []}, "prefill": {"add_args": {}, "add_envs": {}, "del_args": ["--enable-auto-tool-choice", "--tool-call-parser"], "del_envs": []}}}, "excluded": [], "model": "deepseek-ai/DeepSeek-V3.2", "modes": {"lmcache": ["off", "cpu", "rds"], "prefix_caching": ["off", "on", "on_sub_block"], "spec_decode": ["off", "mtp"]}, "option_base": {"reasoning_parser": "on", "tool_calling": "on"}, "options": {"reasoning_parser": ["off", "on"], "tool_calling": ["off", "on"]}, "recommended": {"lmcache": "rds", "prefix_caching": "on_sub_block", "reasoning_parser": "on", "spec_decode": "mtp", "tool_calling": "on"}, "router": {"nodes": [{"cmd": "cat \u003e router-values.yaml \u003c\u003c\u0027EOF\u0027\nendpoint_picker:\n image: ghcr.io/llm-d/llm-d-router-endpoint-picker:v0.10.0\n ext_proc_port: 9002\n plugins:\n - type: prefix-based-pd-decider\n parameters:\n nonCachedTokens: 1\n - type: disagg-profile-handler\n parameters:\n deciders:\n prefill: prefix-based-pd-decider\n - type: prefill-filter\n - type: decode-filter\n - type: approx-prefix-cache-producer\n parameters:\n maxPrefixTokensToMatch: 131072\n lruCapacityPerServer: 1024\n - type: prefix-cache-scorer\n parameters:\n prefixMatchInfoProducerName: approx-prefix-cache-producer\n - type: queue-scorer\n - type: max-score-picker\n scheduling_profiles:\n - name: prefill\n plugins:\n - ref: prefill-filter\n - ref: max-score-picker\n - ref: prefix-cache-scorer\n weight: 2\n - ref: queue-scorer\n weight: 1\n - name: decode\n plugins:\n - ref: decode-filter\n - ref: max-score-picker\n - ref: prefix-cache-scorer\n weight: 2\n - ref: queue-scorer\n weight: 1\ninference_pool:\n failure_mode: FailOpen\n match_labels:\n llm-d.ai/guide: pd-disaggregation\n target_ports:\n - 8000\n - 8001\n - 8002\n - 8003\n - 8004\n - 8005\n - 8006\n - 8007\nEOF", "node": "router-values.yaml"}, {"cmd": "helm install router llm-d-router-gateway \\\n --version v0.10.0 \\\n --values router-values.yaml", "node": "helm"}], "role": "router"}}, "label": null, "modelImpl": "vllm", "note": null, "npus": 8, "skus": ["RBLN-CR13"], "target": "Latency", "verified": "2026-10-07"}], "slug": "prefill-decode-disaggregated", "tag": "P Pipeline / D Data + Expert", "variant": "FP8"}], "text": {"en": {"strategyNote": "Data + Expert Parallel replicates the dense path per device (\u003ccode\u003e--data-parallel-size\u003c/code\u003e) and shards experts (\u003ccode\u003e--enable-expert-parallel\u003c/code\u003e). Pipeline Parallel splits the layers across devices (\u003ccode\u003e--pipeline-parallel-size\u003c/code\u003e). See \u003cstrong\u003eConfiguration\u003c/strong\u003e in the guide for the engine arguments.", "sub": "DeepSeek-V3.2 \u2014 Mixture-of-Experts LLM served on RBLN-CR13 with RBLN vLLM (FP8), 163,840-token context.", "tag": "Aggregated and prefill/decode-disaggregated vllm serve, verified on RBLN-CR13."}, "ko": {"strategyNote": "Data + Expert Parallel\uc740 dense \uacbd\ub85c\ub97c \ub514\ubc14\uc774\uc2a4\ub9c8\ub2e4 \ubcf5\uc81c\ud558\uace0(\u003ccode\u003e--data-parallel-size\u003c/code\u003e) expert\ub97c \ubd84\ud560\ud569\ub2c8\ub2e4(\u003ccode\u003e--enable-expert-parallel\u003c/code\u003e). Pipeline Parallel\uc740 layer\ub97c \ub514\ubc14\uc774\uc2a4\uc5d0 \ub098\ub220 \uc62c\ub9bd\ub2c8\ub2e4(\u003ccode\u003e--pipeline-parallel-size\u003c/code\u003e). \uc5d4\uc9c4 \uc778\uc790\ub294 \uac00\uc774\ub4dc\uc758 \u003cstrong\u003eConfiguration\u003c/strong\u003e\uc5d0\uc11c \ud655\uc778\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4.", "sub": "DeepSeek-V3.2 \u2014 mixture-of-experts(MoE) LLM, RBLN-CR13\uc5d0\uc11c RBLN vLLM\uc73c\ub85c \uc11c\ube59(FP8). 163,840 \ud1a0\ud070 \ucee8\ud14d\uc2a4\ud2b8.", "tag": "RBLN-CR13\uc73c\ub85c \uac80\uc99d\ub41c aggregated \ubc0f prefill/decode disaggregated vllm serve \uad6c\uc131\uc785\ub2c8\ub2e4."}}, "variant": "FP8"} Install the RBLN Driver on the host. An RBLN Portal account is required to install the RBLN Compiler (rebel-compiler). See the installation matrix for compatible package versions.
The KV offload (LMCache) and prefill/decode-disaggregated commands need the kv_connectors extra, which installs lmcache-rbln and nixl-rbln. Without it those servers fail at startup.
Guide — overview and configuration
Overview¶
deepseek-ai/DeepSeek-V3.2 is a 685B-parameter Mixture-of-Experts, decoder-only model (256 routed experts + 1 shared, 8 activated per token) using DeepSeek Sparse Attention (DSA). On RBLN-CR13 it is served (fp8) through the RBLN vLLM plugin, aggregated or with prefill and decode disaggregated.
Configuration¶
Verified on RBLN-CR13. Select a configuration in the recipe above; each server uses 8 devices.
- Data + Expert Parallel:
--data-parallel-sizesets the CR13 devices that each hold a model replica, and--enable-expert-parallelshards the experts across them. - Pipeline Parallel:
--pipeline-parallel-sizesplits the layers across the devices. - Prefill/Decode disaggregated: a prefill server and a decode server run on separate nodes behind the llm-d router, whose helm command follows the two servers in the recipe. For how to deploy the llm-d router and the serving stack on Kubernetes, see Deployment on Kubernetes in the Coding Assistant reference design.
Every command runs on the vllm model implementation, selected with --model-impl vllm.
See the Configuration Guide for all engine arguments.
Notes and limitations¶
- Verified on RBLN-CR13; the exact serve flags may change.
- The batch size, sequence length, and device count are fixed at compilation; changing any of these settings triggers recompilation at the next startup.