Profiling Guide¶

1.1 Offline Inference Profiling¶

For offline inference using Python scripts, pass the profiling parameters to LLM and call start_profile() before the code to profile and stop_profile() after it.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

from vllm import LLM, SamplingParams

# Initialize LLM
llm = LLM(
    model="/path/to/model",
    profiler_config={
        "profiler": "torch",
        "torch_profiler_dir": "/path/to/save/trace",
    },
    # ... other configurations
)

# Start Profiling
llm.start_profile()

# Run Inference
outputs = llm.generate(prompts, sampling_params)

# Stop Profiling
llm.stop_profile()

1.2 Online Inference Profiling¶

For online inference via the API server, start the OpenAI-compatible server with the --profiler-config option:

1

vllm serve /path/to/model --profiler-config '{"profiler": "torch", "torch_profiler_dir": "/path/to/save/trace"}'

Use the following endpoints to control profiling.

1. Call the /start_profile endpoint to begin profiling.
2. Send the actual inference request (e.g., /v1/chat/completions).
3. Call the /stop_profile endpoint to stop profiling and save the results.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

# 1. Start Profiling
curl -X POST http://localhost:8000/start_profile

# 2. Call Model Inference (Generate)
curl -X POST http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "/path/to/model",
        "messages": [
            {
                "role": "user",
                "content": "Seoul is a"
            }
        ]
    }'

# 3. Stop Profiling
curl -X POST http://localhost:8000/stop_profile

1.3 Additional Options¶

For a more detailed analysis, set the following additional keys in profiler_config:

• torch_profiler_record_shapes: Records tensor shape information, off by default.
• torch_profiler_with_memory: Profiles memory usage as well, off by default.
• torch_profiler_with_stack: Records source code stack traces, on by default.
• torch_profiler_with_flops: Records FLOPs (Floating Point Operations), off by default.
• torch_profiler_use_gzip: Compresses the profiling files with gzip, on by default.

1.4 Visualizing Profiling Results¶

The generated profiling trace files can be visualized using the Perfetto UI.

2.1 Configuration¶

Set the environment variable as shown below and run RBLN vLLM to generate profiling results.

• RBLN_PROFILER=1: Enables the RBLN Profiler.

1

export RBLN_PROFILER=1

This variable turns on the RBLN Profiler. The RBLN Profiler runs on the vllm model implementation, and model_impl="vllm" on LLM selects it. On that path, passing no profiler_config to LLM leaves the RBLN Profiler running on its own, writing its own trace file.

1
2
3
4

llm = LLM(
    model="/path/to/model",
    model_impl="vllm",
)

Rebel100™ additionally supports integration with the PyTorch Profiler. With "profiler": "torch" in profiler_config, the PyTorch Profiler controls the profiling session and the RBLN Profiler's trace is merged into the PyTorch Profiler's trace. This is the run described in section 1.

1
2
3
4
5
6
7
8

llm = LLM(
    model="/path/to/model",
    model_impl="vllm",
    profiler_config={
        "profiler": "torch",
        "torch_profiler_dir": "/path/to/save/trace",
    },
)

2.2 Offline Inference Profiling¶

Wrap the section you want to measure with start_profile() and stop_profile(). This follows the same vLLM usage as section 1, and the RBLN Profiler can be enabled with RBLN_PROFILER=1.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# Initialize LLM
llm = LLM(
    model="/path/to/model",
    # ... other configurations
)

# Start Profiling
llm.start_profile()

# Run Inference
outputs = llm.generate(prompts, sampling_params)

# Stop Profiling
llm.stop_profile()

stop_profile() returns only after every worker has finished writing its trace, so the result files are on disk by the time it returns.

2.3 Visualizing Profiling Results¶

A trace file is written per worker, so a run with multiple ranks produces multiple trace files. These files can also be visualized using the Perfetto UI.

For more details on trace analysis, refer to the Perfetto Analysis Guide, and for the profiling procedure with vllm-rbln, to Option 2: Using vllm-rbln.