Skip to content

Troubleshooting

This page provides solutions to common problems, tips for quick fixes, and guidance to help you get back on track.

1. Failed to import the vllm or vllm-rbln package

Symptoms

  • import vllm; print(vllm.__path__) is None;
  • The vllm-rbln plugin is not registered in vllm, so UnspecifiedPlatform is initialized.

Causes

In versions prior to v0.8.1, the vllm-rbln package included a modified copy of vllm. As of v0.8.1, vllm-rbln has built on the new plugin system. This change may cause installation conflicts, which could prevent vllm from being installed properly.

Solution

If you were using vllm earlier than version 0.9.1 or vllm-rbln earlier than version 0.8.0, please uninstall both packages before reinstalling.

2. Model loading failed

Symptoms

  • [rank0]: AttributeError: 'ModelConfig' object has no attribute 'compiled_model_dir'
  • Cannot find .rbln files in model

Causes

For now, vllm-rbln supports only pre-compiled models. Support for online compilation using torch.compile in vLLM will be added soon.

Solution

Before running inference with vLLM, you need to compile the model using optimum-rbln and utilize the compiled model in vLLM.

3. ImportError

Symptoms

1
2
ImportError: libcudart.so.12: cannot open shared object file: No such file or directory
ImportError: libc10_cuda.so: cannot open shared object file: No such file or directory

Causes

The xformers package installed with vLLM's PyPI package attempts to load CUDA-related shared libraries (e.g., libcudart.so.12, libc10_cuda.so). This is because vLLM PyPI packages are pre-built for CUDA. In RBLN environments, attempts to load CUDA libraries will result in import errors.

Solution

You can resolve this issue by choosing either of the following options: 1. Manually remove the xformers package. The xformers package is not used by vLLM RBLN, so it is safe to uninstall. 2. Build and install vLLM from source instead of using the pre-built packages available on PyPI. This approach is recommended to prevent CUDA libraries from being loaded. For detailed instructions, refer to Installation Guide (Source Build).