Skip to content

Troubleshooting

This page describes common RBLN vLLM errors and their solutions.

1. Failed to import the vllm or vllm-rbln package

Symptoms

  • import vllm; print(vllm.__path__) is None;
  • The vllm-rbln plugin is not registered in vllm, so UnspecifiedPlatform is initialized.

Causes

In versions prior to v0.8.1, the vllm-rbln package included a modified copy of vllm. Starting with v0.8.1, vllm-rbln uses the vLLM plugin system. This change may cause installation conflicts, which could prevent vllm from being installed properly.

Solution

If you were using vllm earlier than version 0.9.1 or vllm-rbln earlier than version 0.8.0, uninstall both packages, then reinstall them.

2. Model loading failed

Symptoms

  • [rank0]: AttributeError: 'ModelConfig' object has no attribute 'compiled_model_dir'
  • Cannot find .rbln files in model

Causes

For now, vllm-rbln supports only precompiled models. Support for online compilation using torch.compile in vLLM will be added soon.

Solution

Before running inference with vLLM, you need to compile the model using optimum-rbln and utilize the compiled model in vLLM.

3. ImportError

Symptoms

1
2
ImportError: libcudart.so.12: cannot open shared object file: No such file or directory
ImportError: libc10_cuda.so: cannot open shared object file: No such file or directory

Causes

The xformers package installed with vLLM's PyPI package attempts to load CUDA-related shared libraries (e.g., libcudart.so.12, libc10_cuda.so). This is because vLLM PyPI packages are pre-built for CUDA. In RBLN environments, attempts to load CUDA libraries result in import errors.

Solution

Use one of the following methods:

  1. Manually remove the xformers package. The xformers package is not used by RBLN vLLM, so it is safe to uninstall.
  2. Build and install vLLM from source instead of using the pre-built packages available on PyPI. This approach is recommended to prevent CUDA libraries from being loaded. For detailed instructions, refer to Installation Guide (Source Build).