opt-66b on RBLN NPU¶

{"attn": "eager", "axes": [{"axis": "lmcache", "options": [{"label": "Off", "value": "off"}, {"label": "LMCache (CPU)", "value": "cpu"}, {"label": "LMCache (RDS)", "value": "rds"}], "title": "KV Offload"}, {"axis": "kv_connector", "options": [{"label": "Off", "value": "off"}, {"label": "LMCache (CPU)", "value": "lmcache_cpu"}, {"label": "LMCache (RDS)", "value": "lmcache_rds"}], "title": "KV Offload"}, {"axis": "prefix_caching", "options": [{"label": "On", "value": "on"}, {"label": "On (sub-block)", "value": "on_sub_block"}, {"label": "Off", "value": "off"}], "title": "Prefix Caching"}, {"axis": "spec_decode", "options": [{"label": "Off", "value": "off"}, {"label": "N-gram", "value": "ngram"}, {"label": "EAGLE-3", "value": "eagle3"}, {"label": "Suffix", "value": "suffix"}, {"label": "MTP", "value": "mtp"}], "title": "Speculative Decoding"}, {"axis": "on_device_sampler", "options": [{"label": "On", "value": "on"}, {"label": "Off", "value": "off"}], "title": "On-device Sampler"}, {"axis": "structured_output", "options": [{"label": "On", "value": "on"}, {"label": "Off", "value": "off"}], "title": "Structured Output"}], "badges": ["Dense", "66B", "2,048 ctx", {"hl": true, "text": "vllm-rbln"}, "eager", "Text"], "client": "curl http://localhost:8000/v1/completions \\\n    -H \"Content-Type: application/json\" \\\n    -d \u0027{\n      \"model\": \"opt-66b\",\n      \"prompt\": \"Hello, my name is\",\n      \"max_tokens\": 32\n    }\u0027", "features": {"api": true, "mm": false, "text": true}, "hf": "https://huggingface.co/facebook/opt-66b", "installLine": "pip install rebel-compiler vllm-rbln", "model": "facebook/opt-66b", "name": "opt-66b", "provider": "facebook", "strategies": [{"label": "RSD", "nodes": null, "options": [{"attn": "eager", "features": {}, "label": null, "modelImpl": "transformers", "note": null, "npus": 16, "roles": [{"cmd": "vllm serve facebook/opt-66b \\\n    --block-size 2048 \\\n    --max-model-len 2048 \\\n    --max-num-seqs 1 \\\n    --model-impl transformers \\\n    --rbln-num-devices-per-local-rank 16 \\\n    --served-model-name opt-66b", "role": "main"}], "skus": ["RBLN-CA22", "RBLN-CA25"], "target": null, "verified": "2026-10-16"}], "slug": "rsd", "tag": "Aggregated", "variant": "BF16"}], "text": {"en": {"strategyNote": "Rebellions Scalable Design (RSD): the RBLN Compiler partitions the model across NPU devices, or across the chiplets within one device. See \u003ca href=\"../features/model-parallelism.html#rebellions-scalable-design-rsd\"\u003eModel Parallelism\u003c/a\u003e for how RSD applies on each NPU, and \u003cstrong\u003eConfiguration\u003c/strong\u003e in the guide for the engine arguments.", "sub": "opt-66b \u2014 dense decoder-only LLM served on RBLN NPUs with RBLN vLLM, 2,048-token context (Naive Attention).", "tag": "The vllm serve command compiles the model on first run and starts an OpenAI-compatible server."}, "ko": {"strategyNote": "Rebellions Scalable Design(RSD): RBLN \ucef4\ud30c\uc77c\ub7ec\uac00 \ubaa8\ub378\uc744 NPU \ub514\ubc14\uc774\uc2a4 \uac04 \ub610\ub294 \ub514\ubc14\uc774\uc2a4 \ud558\ub098\uc758 \uce69\ub81b \uac04\uc5d0 \ubd84\ud560\ud569\ub2c8\ub2e4. NPU\ubcc4 RSD \uc801\uc6a9 \ubc29\uc2dd\uc740 \u003ca href=\"../features/model-parallelism.html#rebellions-scalable-design-rsd\"\u003e\ubaa8\ub378 \ubcd1\ub82c\ud654\u003c/a\u003e\uc5d0\uc11c, \uc5d4\uc9c4 \uc778\uc790\ub294 \uac00\uc774\ub4dc\uc758 \u003cstrong\u003eConfiguration\u003c/strong\u003e\uc5d0\uc11c \ud655\uc778\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4.", "sub": "opt-66b \u2014 dense \ub514\ucf54\ub354 \uc804\uc6a9 LLM, RBLN vLLM\uc73c\ub85c RBLN NPU\uc5d0\uc11c \uc11c\ube59. 2,048 \ud1a0\ud070 \ucee8\ud14d\uc2a4\ud2b8(Naive Attention).", "tag": "vllm serve \uba85\ub839\uc740 \ucd5c\ucd08 \uc2e4\ud589 \uc2dc \ubaa8\ub378\uc744 \ucef4\ud30c\uc77c\ud558\uace0 OpenAI \ud638\ud658 \uc11c\ubc84\ub97c \uc2dc\uc791\ud569\ub2c8\ub2e4."}}, "variant": "BF16"}

Install the RBLN Driver on the host. An RBLN Portal account is required to install the RBLN Compiler (rebel-compiler). See the installation matrix for compatible package versions.

$ pip install \
  --extra-index-url https://pypi.rbln.ai/simple \
  rebel-compiler==0.12.0

$ pip install \
  --extra-index-url https://wheels.vllm.ai/0.30.0/cpu \
  --extra-index-url https://download.pytorch.org/whl/cpu \
  vllm-rbln==0.12.0

Guide — overview and configuration

Overview¶

facebook/opt-66b is a dense, decoder-only model. On RBLN NPUs, the RBLN vLLM plugin compiles it on first run for a fixed batch size, sequence length, and NPU count, then serves it over vLLM's OpenAI-compatible API.

Configuration¶

--rbln-num-devices-per-local-rank sets how many NPU devices one engine process uses, and the RBLN Compiler partitions the model for those devices with Rebellions Scalable Design (RSD). See Model Parallelism for how RSD applies on each NPU. Changing these settings recompiles the model on the next start. Compile Cache lists every setting that does.

Recipes for other model sizes:

opt-125m · opt-2.7b · opt-6.7b

See the Configuration Guide for all engine arguments.

Notes and limitations¶

The batch size, sequence length, and NPU count are fixed at compilation; changing any of these settings triggers recompilation at the next startup.
The recipe serves with --max-num-seqs 1. Increase it for higher throughput; the model recompiles to match.