Skip to content

Getting Started with RBLN Profiler

This section outlines the basic steps for using the RBLN Profiler. It first explains the RBLN Profiler API with a PyTorch BERT model from the RBLN Model Zoo, and then shows how to extract profiling data with optimum-rbln and vllm-rbln.

To use torch.profiler with RBLN PyTorch, see Profiling on Rebel100™.

Prerequisites

Before starting, ensure the following pip packages are installed on your system.

Note

The RBLN Profiler is a feature in the rebel-compiler package. Therefore, it is installed when you install rebel-compiler and does not need to be installed separately.

RBLN Profiler API

The RBLN Profiler works through the runtime API of rebel-compiler, and this API is the foundation of every way to use the RBLN Profiler. This section explains the API with an example that uses rebel-compiler to compile a BERT-base model trained for question answering and extracts profiling data by calling the RBLN Runtime API directly.

The RBLN Profiler API has two roles.

  • Choosing what to profile: setting activate_profiler=True when creating a runtime makes the inference runs of that runtime the target of profiling. The environment variable RBLN_PROFILER=1 does the same.
  • Choosing when to record: start() and done() of rebel.profiler mark where the recorded window begins and ends. with profile(...) wraps a section of code with these two calls.

Compile Model

The BERT-base model can be compiled with the following command. When the model compilation is completed, the bert-base.rbln file is created in the directory.

1
2
3
$ cd rbln-model-zoo/ATOM/pytorch/nlp/bert/qa
$ pip3 install -r requirements.txt
$ python3 compile.py --model_name base

Activate RBLN Profiler

Profile the model by either using the argument activate_profiler=True in rebel.Runtime when creating the runtime module, or setting the environment variable RBLN_PROFILER=1 while running inference. A runtime created with the profiler on loads the profiling information in the compiled model and prepares for profiling; every inference it runs from then on is recorded. A runtime created with the profiler off is not profiled.

  • Using the RBLN Runtime API: set activate_profiler=True to activate the RBLN Profiler. 

    1
2
3
4
5
6
7
    # Revise `inference.py` as follows.
...
# Load compiled model to RBLN runtime module
module = rebel.Runtime(
    f"bert-{args.model_name}.rbln", tensor_type="pt", activate_profiler=True
)
...

  • Using the environment variable: setting the environment variable RBLN_PROFILER=1 is the same as setting activate_profiler=True on every runtime the process creates, so the model is profiled without code changes. 

    1
    $ RBLN_PROFILER=1 python3 inference.py --model_name base

Either setting is enough on its own. The profiler is on when either one is on, so activate_profiler=False does not turn it off while RBLN_PROFILER=1 is set.

Note

The RBLN Profiler is not available on the asynchronous runtime (rebel.AsyncRuntime).

Extract Profiled Data with Inference

  • Generally, you can obtain the profiling data without any additional setting, as shown below. The data the profiler collects is then written to the default path (./) when the process exits normally: 
    1
2
3
4
5
    # Run inference with RBLN runtime module.
...
# Save to default path("./")
out = module.run(**inputs)
...
  • with profile(...) profiles only the section of code it wraps: 
    1
2
3
4
5
6
7
    ...
# Configure profiling scope and save path as "./profile_out".
from rebel.profiler import profile

with profile(output_dir="./profile_out"):
    out = module.run(**inputs)
...

To profile a specific section, turn the profiler on as described in Activate RBLN Profiler and wrap that section with with profile(...). Call with profile(...) in the process where the RBLN runtime runs.

Warning

The RBLN Profiler writes the collected data to *.pb files when the process exits normally. If the process is terminated forcibly, for example with os._exit(), or crashes, no result file is produced.

Setting a profiling scope with with profile(...) produces the result file when the scope ends.

Check the Profiling Result

After you run one of the above commands, the profiling results appear as follows:

Terminal output of the generation status report: three category tables, every pass succeeded. Terminal output of the generation status report: three category tables, every pass succeeded.

The RBLN Profiler Generation Status Report shows all status information generated during the tracing and generation process of the RBLN Profiler, displayed across three categories.

  • System Passes: monitor general system issues and verify memory allocation status as well as the profiling data generated by the Command Processor.

  • Compilation Passes: verify and modify model information generated by the RBLN Compiler based on system constraints.

  • Tracing Passes: collect and verify profiling data from the Command Processor, analyze it, and preprocess it for visualization by reorganizing its layout and transforming its structure for Protocol Buffer file generation and serialization.

Note

If the RBLN Profiler Generation Status Report consistently reports Failed, contact client_support@rebellions.ai.

  • Extracted Filename
    • Multi Stream (Full log): rbln_{date}_{time}.pb
      • This single *.pb file represents all tracing information from all inference runs within a single profiler instance in one plot.
    • Single Stream (Log by sequence): rbln_{date}_{time}_{sequence_index}.pb
      • These files represent tracing information for a single module inference within a single profiler instance in one plot. Specifically, one *.pb file is generated for each inference run of each module.

The generated *.pb files can be visualized using Perfetto.

Next Steps

Profiling with RBLN Optimum and RBLN vLLM

RBLN Optimum (optimum-rbln) and RBLN vLLM (vllm-rbln) create and run runtimes through the RBLN Profiler API internally. Choosing what to profile and when to record therefore works as described above, and each package exposes them in a form that fits how the package is used.

Option 1: Using optimum-rbln

Extract the profiling data with optimum-rbln as follows.

Compile Model

The BERT-base model from optimum-rbln can be compiled with the following command. When the model compilation is completed, the bert-base-cased-squad2 directory is created.

1
2
3
$ cd rbln-model-zoo/ATOM/huggingface/transformers/question-answering/bert
$ pip3 install -r requirements.txt
$ python3 compile.py --model_name base

Activate RBLN Profiler

Profile the model by either setting the environment variable RBLN_PROFILER=1 while running inference, or using the argument rbln_activate_profiler=True in optimum-rbln.

  • Using the environment variable: set the environment variable RBLN_PROFILER=1 to enable the RBLN Profiler. 

    1
    $ RBLN_PROFILER=1 python3 inference.py --model_name base

  • Using the optimum-rbln API: set rbln_activate_profiler=True to activate the RBLN Profiler. 

    1
2
3
4
5
6
7
8
9
    # Revise `inference.py` as follows.
...
# Load compiled model
model = RBLNBertForQuestionAnswering.from_pretrained(
    model_id=os.path.basename(model_id),
    export=False,
    rbln_activate_profiler=True,
)
...

Extract Profiled Data with Inference

  • You can obtain the profiling data without any additional setting: 

    1
2
3
4
    # Run inference with optimum-rbln API.
...
# Save to default path("./")
answer = pipe(question=args.question, context=args.context)

  • The optimum-rbln API also supports scoped profiling methods as shown below: 

    1
2
3
4
5
6
7
    ...
# Configure profiling scope and save path as "./profile_out".
from rebel.profiler import profile

with profile(output_dir="./profile_out"):
    answer = pipe(question=args.question, context=args.context)
...

Next Steps

The following case studies analyze the traces of models run with optimum-rbln.

Option 2: Using vllm-rbln

Extract the profiling data with vllm-rbln as follows. The vllm-rbln package follows vLLM's own profiling workflow — set the window with start_profile() and stop_profile() — and the RBLN Profiler can be enabled with RBLN_PROFILER=1. For installation and model preparation, refer to RBLN vLLM.

Activate RBLN Profiler

Setting the environment variable RBLN_PROFILER=1 while running inference enables the RBLN Profiler. The RBLN Profiler runs on the vllm model implementation, and passing model_impl="vllm" to LLM selects it.

1
2
$ cd vllm-rbln
$ RBLN_PROFILER=1 python3 examples/vllm/features/profiling/simple_profiling_offline.py

The example script already sets "profiler": "torch" in profiler_config. Which profiler start_profile() and stop_profile() control depends on the profiler_config setting.

  • Without it: the RBLN Profiler runs on its own and writes a *.pb file. 
    1
2
    # Leave profiler_config unset.
llm = LLM(model="/path/to/model", model_impl="vllm")
  • With "profiler": "torch": the PyTorch profiler owns the profiling session, and the device windows the RBLN Profiler collects are merged into the PyTorch profiler's output chrometrace. This merge works on Rebel100™ only; it is not supported on ATOM™. 
    1
2
3
4
5
6
7
8
9
    # Turn the PyTorch profiler on as well and set where its trace is written.
llm = LLM(
    model="/path/to/model",
    model_impl="vllm",
    profiler_config={
        "profiler": "torch",
        "torch_profiler_dir": "/path/to/save/trace",
    },
)

For the remaining profiler_config options, refer to the Profiling Guide.

Extract Profiled Data with Inference

Wrapping generate() with start_profile() and stop_profile() keeps only that window in the trace. The first generate() call initializes the compiled models, so run it before the measured window to keep it out of the trace.

1
2
3
4
5
6
7
8
9
# vllm-rbln/examples/vllm/features/profiling/simple_profiling_offline.py
...
# Exclude first-use initialization from the trace.
llm.generate(prompts, SamplingParams(temperature=0.0, max_tokens=2))

llm.start_profile()
outputs = llm.generate(prompts, sampling_params)
llm.stop_profile()
...

stop_profile() returns only after every worker has finished writing its trace. Where the workers run in their own processes, they are torn down when the engine shuts down, so without this window, any trace data that has not been written by then may be lost.

A trace file is written per worker, in the format described in Check the Profiling Result.

Next Steps

See the following trace analysis methods and case studies for models run with vllm-rbln.