Skip to content

NVIDIA Triton Inference Server

NVIDIA Triton Inference Server is open-source software for serving machine learning models. You can serve AI models on RBLN NPUs with NVIDIA Triton Inference Server using the Python backend or the vLLM backend. For LLM serving, the vLLM backend integrates with RBLN vLLM and supports continuous batching.

Getting started

To utilize the RBLN NPU, the rebel-compiler package must be installed.

To run NVIDIA Triton Inference Server in Docker, use the nvcr.io/nvidia/tritonserver:26.07-vllm-python-py3 image. For detailed information, refer to Step 3, On-premises section in the Llama3-8B tutorial.

Tutorial

The following tutorials show how to serve models on RBLN NPUs with NVIDIA Triton Inference Server.

  • ResNet50 shows how to serve the image classification model with NVIDIA Triton Inference Server
  • YOLOv8 demonstrates how to serve the object detection model with NVIDIA Triton Inference Server
  • Llama3-8B introduces how to serve the LLM model with vLLM-enabled NVIDIA Triton Inference Server
  • Llama3.1-8B with Flash Attention shows how to serve the model with Flash Attention using NVIDIA Triton Inference Server