blip2-opt-6.7b on RBLN NPU¶

{"attn": "eager", "axes": [{"axis": "lmcache", "options": [{"label": "Off", "value": "off"}, {"label": "LMCache (CPU)", "value": "cpu"}, {"label": "LMCache (RDS)", "value": "rds"}], "title": "KV Offload"}, {"axis": "kv_connector", "options": [{"label": "Off", "value": "off"}, {"label": "LMCache (CPU)", "value": "lmcache_cpu"}, {"label": "LMCache (RDS)", "value": "lmcache_rds"}], "title": "KV Offload"}, {"axis": "prefix_caching", "options": [{"label": "On", "value": "on"}, {"label": "On (sub-block)", "value": "on_sub_block"}, {"label": "Off", "value": "off"}], "title": "Prefix Caching"}, {"axis": "spec_decode", "options": [{"label": "Off", "value": "off"}, {"label": "N-gram", "value": "ngram"}, {"label": "EAGLE-3", "value": "eagle3"}, {"label": "Suffix", "value": "suffix"}, {"label": "MTP", "value": "mtp"}], "title": "Speculative Decoding"}, {"axis": "on_device_sampler", "options": [{"label": "On", "value": "on"}, {"label": "Off", "value": "off"}], "title": "On-device Sampler"}, {"axis": "structured_output", "options": [{"label": "On", "value": "on"}, {"label": "Off", "value": "off"}], "title": "Structured Output"}], "badges": ["Dense", "6.7B", "2,048 ctx", {"hl": true, "text": "vllm-rbln"}, "eager", "Multimodal"], "client": "curl http://localhost:8000/v1/chat/completions \\\n    -H \"Content-Type: application/json\" \\\n    -d \u0027{\n      \"model\": \"blip2-opt-6.7b\",\n      \"messages\": [{\"role\": \"user\", \"content\": [\n        {\"type\": \"text\", \"text\": \"What is in this image?\"},\n        {\"type\": \"image_url\", \"image_url\": {\"url\": \"https://rbln-public.s3.ap-northeast-2.amazonaws.com/images/tabby.jpg\"}}\n      ]}],\n      \"max_tokens\": 64\n    }\u0027", "features": {"api": true, "mm": true, "text": true}, "hf": "https://huggingface.co/Salesforce/blip2-opt-6.7b", "installLine": "pip install rebel-compiler vllm-rbln", "model": "Salesforce/blip2-opt-6.7b", "name": "blip2-opt-6.7b", "provider": "Salesforce", "strategies": [{"label": "RSD", "nodes": null, "options": [{"attn": "eager", "features": {}, "label": null, "modelImpl": "transformers", "note": null, "npus": 1, "roles": [{"cmd": "vllm serve Salesforce/blip2-opt-6.7b \\\n    --block-size 2048 \\\n    --max-model-len 2048 \\\n    --max-num-seqs 1 \\\n    --model-impl transformers \\\n    --rbln-num-devices-per-local-rank 1 \\\n    --served-model-name blip2-opt-6.7b", "role": "main"}], "skus": ["RBLN-CR13"], "target": null, "verified": "2026-10-16"}, {"attn": "eager", "features": {}, "label": null, "modelImpl": "transformers", "note": null, "npus": 4, "roles": [{"cmd": "vllm serve Salesforce/blip2-opt-6.7b \\\n    --block-size 2048 \\\n    --max-model-len 2048 \\\n    --max-num-seqs 1 \\\n    --model-impl transformers \\\n    --rbln-num-devices-per-local-rank 4 \\\n    --served-model-name blip2-opt-6.7b", "role": "main"}], "skus": ["RBLN-CA22", "RBLN-CA25"], "target": null, "verified": "2026-10-16"}], "slug": "rsd", "tag": "Aggregated", "variant": "BF16"}], "text": {"en": {"strategyNote": "Vision-language serving \u2014 the language model and vision encoder are compiled together. See \u003cstrong\u003eConfiguration\u003c/strong\u003e in the guide for the engine arguments.", "sub": "blip2-opt-6.7b \u2014 vision-language model served on RBLN NPUs with RBLN vLLM; accepts images and text through an OpenAI-compatible chat API.", "tag": "The vllm serve command compiles the model on first run and starts an OpenAI-compatible server."}, "ko": {"strategyNote": "\ube44\uc804-\uc5b8\uc5b4 \uc11c\ube59 \u2014 \uc5b8\uc5b4 \ubaa8\ub378\uacfc \ube44\uc804 \uc778\ucf54\ub354\ub97c \ud568\uaed8 \ucef4\ud30c\uc77c\ud569\ub2c8\ub2e4. \uc5d4\uc9c4 \uc778\uc790\ub294 \uac00\uc774\ub4dc\uc758 \u003cstrong\u003eConfiguration\u003c/strong\u003e\uc5d0\uc11c \ud655\uc778\ud560 \uc218 \uc788\uc2b5\ub2c8\ub2e4.", "sub": "blip2-opt-6.7b \u2014 \ube44\uc804-\uc5b8\uc5b4 \ubaa8\ub378, RBLN vLLM\uc73c\ub85c RBLN NPU\uc5d0\uc11c \uc11c\ube59. OpenAI \ud638\ud658 chat API\ub85c \uc774\ubbf8\uc9c0\uc640 \ud14d\uc2a4\ud2b8\ub97c \ud568\uaed8 \uc785\ub825\ud569\ub2c8\ub2e4.", "tag": "vllm serve \uba85\ub839\uc740 \ucd5c\ucd08 \uc2e4\ud589 \uc2dc \ubaa8\ub378\uc744 \ucef4\ud30c\uc77c\ud558\uace0 OpenAI \ud638\ud658 \uc11c\ubc84\ub97c \uc2dc\uc791\ud569\ub2c8\ub2e4."}}, "variant": "BF16"}

Install the RBLN Driver on the host. An RBLN Portal account is required to install the RBLN Compiler (rebel-compiler). See the installation matrix for compatible package versions.

$ pip install \
  --extra-index-url https://pypi.rbln.ai/simple \
  rebel-compiler==0.12.0

$ pip install \
  --extra-index-url https://wheels.vllm.ai/0.30.0/cpu \
  --extra-index-url https://download.pytorch.org/whl/cpu \
  vllm-rbln==0.12.0

Guide — overview and configuration

Overview¶

Salesforce/blip2-opt-6.7b is a vision-language model. On RBLN NPUs, the RBLN vLLM plugin compiles the language model and vision encoder on first run, then serves them over vLLM's OpenAI-compatible API. Include images using image_url content items in chat completion requests.

Configuration¶

--rbln-num-devices-per-local-rank sets how many NPU devices one engine process uses, and the RBLN Compiler partitions the model for those devices with Rebellions Scalable Design (RSD). See Model Parallelism for how RSD applies on each NPU. The vision encoder is configured through --additional-config (optimum_overrides.visual). Changing these settings recompiles the model on the next start. Compile Cache lists every setting that does.

Recipes for other model sizes:

blip2-opt-2.7b

See the Configuration Guide for all engine arguments.

Notes and limitations¶

The sequence length and NPU count are fixed at compilation; changing any of these settings triggers recompilation at the next startup.