Skip to content

Llama-3-8B (Chatbot)

Overview

This tutorial explains how to compile and deploy the Llama 3 model from HuggingFace on a single Rebel100™ (RBLN-CR13) NPU. This guide uses the meta-llama/Meta-Llama-3-8B-Instruct model.

Serving on Rebel100™

On Rebel100™, use RBLN vLLM (vllm serve) as the main way to serve LLMs. It compiles the model and exposes an OpenAI-compatible API in one step; see the Meta-Llama-3-8B-Instruct recipe. This tutorial covers the optimum-rbln Python API, which is useful for checking compilation and running inference directly.

Note

Rebel100™ does not support RSD (see the version matrix), and every Rebel100™ example in the Model Zoo compiles for one NPU, so rbln_num_devices is set to 1 in this tutorial.

Note

Llama 3 is licensed under the LLAMA Community License, Copyright (c) Meta Platforms, Inc. All Rights Reserved.

Setup & Installation

Before you begin, ensure that your system environment is properly configured and that all required packages are installed. This includes:

Note

  • rebel-compiler requires an RBLN Portal account.
  • The commands above are intended for a default pip install on Debian-based Linux such as Ubuntu. For all other configurations, refer to the Installation Guide for the supported install matrix and the applicable commands.

Note

The meta-llama/Meta-Llama-3-8B-Instruct model on HuggingFace has restricted access. Once access is granted, you can log in using the hf (huggingface-cli) command as shown below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
$ hf auth login

    _|    _|  _|    _|    _|_|_|    _|_|_|  _|_|_|  _|      _|    _|_|_|      _|_|_|_|    _|_|      _|_|_|  _|_|_|_|
    _|    _|  _|    _|  _|        _|          _|    _|_|    _|  _|            _|        _|    _|  _|        _|
    _|_|_|_|  _|    _|  _|  _|_|  _|  _|_|    _|    _|  _|  _|  _|  _|_|      _|_|_|    _|_|_|_|  _|        _|_|_|
    _|    _|  _|    _|  _|    _|  _|    _|    _|    _|    _|_|  _|    _|      _|        _|    _|  _|        _|
    _|    _|    _|_|      _|_|_|    _|_|_|  _|_|_|  _|      _|    _|_|_|      _|        _|    _|    _|_|_|  _|_|_|_|

    To login, `huggingface_hub` requires a token generated from [https://huggingface.co/settings/tokens](https://huggingface.co/settings/tokens) .
Enter your token (input will not be visible):

Using RBLN Optimum

This step demonstrates how to compile the meta-llama/Meta-Llama-3-8B-Instruct model to run on a single Rebel100™ NPU.

Model Compilation

To begin, import the RBLNLlamaForCausalLM class from optimum-rbln. This class's from_pretrained() method downloads the Llama 3 model from the HuggingFace Hub and compiles it using the RBLN Compiler. When exporting the model, specify the following parameters:

  • export: Must be True to compile the model.
  • rbln_batch_size: Defines the batch size for compilation.
  • rbln_max_seq_len: Defines the maximum sequence length.
  • rbln_num_devices: Defines the number of NPUs to be used for inference. Set it to 1 on Rebel100™.

After compilation, save the model artifacts to disk using the save_pretrained() method. This creates a directory (e.g., rbln-Llama-3-8B-Instruct) containing the compiled model.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
from optimum.rbln import RBLNLlamaForCausalLM

# Define the HuggingFace model ID
model_id = "meta-llama/Meta-Llama-3-8B-Instruct"

# Compile the model for a single Rebel100™ NPU
compiled_model = RBLNLlamaForCausalLM.from_pretrained(
    model_id=model_id,
    export=True,
    rbln_batch_size=1,
    rbln_max_seq_len=8192,
    rbln_num_devices=1,
)

compiled_model.save_pretrained("rbln-Llama-3-8B-Instruct")

Load the Compiled RBLN Model

Load the compiled RBLN model using RBLNLlamaForCausalLM.from_pretrained(). Pass the saved directory path as model_id and set the export parameter to False.

1
2
3
4
5
6
7
from optimum.rbln import RBLNLlamaForCausalLM

# Load the compiled RBLN model from the specified directory
compiled_model = RBLNLlamaForCausalLM.from_pretrained(
    model_id="rbln-Llama-3-8B-Instruct",
    export=False
)

Prepare Inputs

Use the AutoTokenizer from the transformers library to tokenize the input sequences. For instruction-tuned models such as Llama 3, apply the chat template.

1
2
3
4
5
6
7
8
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct")
tokenizer.pad_token = tokenizer.eos_token

conversation = [{"role": "user", "content": "Hey, are you conscious? Can you talk to me?"}]
text = tokenizer.apply_chat_template(conversation, add_generation_prompt=True, tokenize=False)
inputs = tokenizer(text, return_tensors="pt", padding=True)

Model Inference

You can now generate a response using the generate() method.

1
2
3
4
5
6
7
8
9
output_sequence = compiled_model.generate(
    inputs.input_ids,
    attention_mask=inputs.attention_mask,
    do_sample=False,
    max_length=8192,
)

out = tokenizer.batch_decode(output_sequence, skip_special_tokens=True, clean_up_tokenization_spaces=True)
print(out[0])

Example Output: 

1
2
3
4
5
Hello there! I'm an AI, which means I'm a computer program designed to simulate conversation and answer questions to the best of my ability. I don't have consciousness in the way that humans do, but I'm designed to be very conversational and can understand and respond to language in a way that might seem like I'm conscious.

I can certainly chat with you and answer your questions to the best of my ability. I can provide information on a wide range of topics, from science and history to entertainment and culture. I can also help with tasks like language translation, text summarization, and even generating creative writing.

So, what's on your mind? Is there something specific you'd like to talk about or ask me? I'm all ears (or rather, all text)!

References