How to analyze the RBLN Profiler data using Perfetto¶

Task Details¶

1. Name Structure

   The RBLN Profiler uses a naming convention for tasks structured as i{instance_index}_{sequence_index}_W{workload_index}_{layer_name}_{command_index}. This format helps identify the specific stage in a neural network layer's operation and evaluate its performance at that stage.

   • i{instance_index} identifies the runtime instance that executed the command. It appears only in the Multi Stream file, which merges the records of several runtime instances; Single Stream files carry no prefix.
   • sequence_index represents the order in which the module was executed on that runtime instance, starting from 0 for each instance. In a merged file, therefore, it takes both the prefix and this index to identify one execution.
   • W{workload_index} tells which runtime call (Device Workload slice) the command ran in. Host commands carry no token. See Understanding Chiplet, Workload, and CCL Tracks.
   • layer_name refers to the specific tensor-level operation being performed. In cases where multiple operations are fused, they are grouped under the label "fused()" and joined with "+".
   • command_index identifies the specific command generated by the RBLN Compiler when the operation is divided into multiple commands, distinguishing each individual command within that division.

   For example, i1_0_W2_fused(bn1+conv1+relu)_1 in a Multi Stream file is the second command of the fused bn1, conv1, and relu operation, run in Device Workload_2 of the first sequence of runtime instance 1.

2. Category

   The RBLN Profiler groups commands into the following categories. For more detailed information on each category, refer to the RBLN NPU Architecture page.

   • Host
   • Neural Engine Clusters
   • Neural DMA
   • Task DMA
   • External HDMA
   • Device HDMA
   • Device Sync
   • General Compute Engine

   The numbers next to the command names in the Perfetto UI are internal ID values assigned by the RBLN Profiler. These numbers are unrelated to the actual performance of the RBLN NPU.

3. Flow Connection (Preceding Flows & Following Flows)

   The Preceding Flows and Following Flows displayed at the bottom of the Perfetto UI represent dependencies among commands. During model compilation, these dependencies are determined to ensure that the Command Processor handles them correctly, taking the device's configuration and allocation status into account. Dependencies related to Neural DMAs are managed by the Task Manager within the Neural Engine, and, therefore, are not displayed in Perfetto.

   • Between External HDMA and Task DMA

     

     External HDMA transfers weight and input data from host DRAM to the device memory, creating a dependency with the initial Task DMA command that sends data from the device memory to the on-chip Shared Memory. Similarly, since the final output data retrieved by Task DMA must be sent back to host DRAM via External HDMA, it is essential to enforce a dependency ensuring that External HDMA executes only after Task DMA completes.

     These visualized flows demonstrate explicit dependencies among commands. However, this does not imply that a command without connections has no sequential dependencies. For instance, Task Managers in each Neural Engine preside over synchronization on a local hardware level. Also, there is a barrier that prevents the first Neural Engine Clusters commands from executing until related External HDMA commands are done.

   • Between Neural Engine Clusters and Task DMA

     

     The connection between the Neural Engine Clusters and Task DMA mainly shows dependencies related to transferring weights or input/output data between neural network layers, and it is designed to support each task accordingly. In this example, the dependency is determined by the compiler because the kernel weights need to be loaded for the given Neural Engine Clusters operation 0_fused(bn1+conv1+relu)_1. Additionally, if the compiler determines a connection is needed between the Neural Engine Clusters and Task DMA, it is considered a dependency. For example, data generated after neural network layer computation is temporarily stored in the device memory due to its large size.

   • Between Device HDMA and Device Sync

     

     In large models using multiple devices, layers are divided through sharding, allowing them to run across multiple devices.

     In case 1, the root device (node 0) synchronizes with the leaf devices (nodes 1, 2, 3) via Device HDMA to receive data. Figure (1) illustrates how data are gathered into the root device (node 0) through Device HDMA, after which the synchronization on each leaf device (nodes 1, 2, 3) is released.

     Conversely, in case 2, the root device (node 0) also synchronizes with the leaf devices (nodes 1, 2, 3) via Device HDMA to send data. Figure (2) illustrates how data are distributed to the leaf devices (nodes 1, 2, 3) through Device HDMA. Afterward, the leaf devices send a synchronization signal to confirm they have correctly received the data.

     For these tensor-parallel models, synchronization commands are generated during compilation to establish dependencies with the Device HDMA.

     Note

     ATOM™: The screenshots above are ATOM™ traces. Data transfers and synchronization between devices appear as Device HDMA and Device Sync commands.

     Rebel100™: Device HDMA and Device Sync commands appear inside each Chiplet group. When the model uses collective communication (CCL), as in tensor-parallel serving with RBLN vLLM, communication between devices appears on the CCL tracks; see CCL Communication.

Device Workload Matching¶

Slices on the Device Workload track are named {sequence_index}_Device Workload_{workload_index} (with the same i{instance_index}_ prefix as commands in the Multi Stream file), and each covers the device-side execution window of one runtime call. As in the image above, when consecutive runtime calls overlap, their Device Workload slices are stacked within the track.

Commands that belong to a window carry the matching W{workload_index} token in their name, so search the trace for that token to list them. The image above shows Device Workload_23 selected, with the commands named 0_W23_… running on the Neural Engine Clusters and Task DMA tracks of Chiplet 0. Overlapping windows stay distinguishable because each command carries its own workload_index.

Chiplet Groups¶

Commands are recorded under the Chiplet group of the chiplet that executed them, so that group is how the trace identifies where a command ran. A layer distributed across chiplets appears as commands with the same name under several Chiplet groups, and each of those slices is the part its chiplet executed. Rebel100™ comprises four chiplets, so Chiplet 0 through Chiplet 3 are all present; a group with no slices means that the chiplet executed no command of the profiled model.

CCL Communication¶

When a model runs across multiple Rebel100™ devices using collective communication (CCL), as in tensor-parallel serving with RBLN vLLM, the collective operations appear in the trace at two levels. A run without collective communication has no CCL tracks. The CCL Workload track at the device level holds one slice per collective, covering its whole execution window. Expanding the CCL group reveals the CCL Command track, where a slice with the same name covers only the portion in which the command stream of that collective is running.

Profiling Overhead¶

The Host track also contains Profile Wait, Profile Readback, and Profile Workload slices. The image above shows a Profile Readback slice. These slices measure the time the RBLN Profiler spent waiting for a runtime call to finish and reading its counters back from the device. Exclude these slices when you measure the latency of the model; they do not exist when profiling is disabled.

Case 1. Matching Names with Model (PyTorch)¶

The structure of a model can be recognized from a printed specification or code-level implementation. Comparing that information with the visualized slices shows how an operation is converted into a sequence of commands. In the above example, model.layers[0].self_attn.q_proj of a LlamaForCausalLM instance gets converted into 0_causal_lm.model.0.self_attn.q_proj.

The structure of a model and the order of its layers may differ from the actual sequence in which commands are processed. Several commands can be omitted during compilation. In that case, command indices of a single layer can be noncontiguous. Additional commands are implicitly generated during the compilation process for executing various operations on the RBLN NPU efficiently. In the above example, 0_causal_lm.model_0 is added to process a rotary positional embedding layer (RoPE).

Case 2. Command Generation¶

To improve performance and efficiency, the RBLN Compiler supports operation fusion, which combines a sequence of operations into a single function. The command_name of a fused function is fused(name0+name1...). In the above example, a convolution layer, a batch normalization layer and an activation function layer are merged into a single function named fused(layer1.2+layer1.2.bn3+layer1.2.conv3).

A sequence of commands for processing a module shares the same command_name, and the command_index distinguishes each of them. Several commands for running fused(layer1.2+layer1.2.bn3+layer1.2.conv3), such as device memory accesses on Task DMA and executions on the Neural Engine Clusters, are expressed in the form 0_fused(layer1.2+layer1.2.bn3+layer1.2.conv3)_{command_index}.

Case 3. Reusing Layers (PyTorch)¶

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

# YOUR_PYTHON_PACKAGE_PATH/torchvision/models/resnet.py
class Bottleneck(nn.Module):
    ...
    def __init__(...) -> None:
    super().__init__()
        ...
        self.relu = nn.ReLU(inplace=True)
        ...

    def forward(self, x: Tensor) -> Tensor:
        ...
        out = self.relu(out)
        ...
        out = self.relu(out)
        ...
        out = self.relu(out)
        ...
        return out

Inline operations such as torch.add, torch.nn.functional.relu are compiled without name because the model does not have their attribute names. This makes it hard to identify which commands correspond to the operations in Perfetto. To prevent this, use an explicit class attribute inheriting torch.nn.Module instead of inline operations.

Additionally, repeatedly using a single class attribute may also create difficulties in mapping commands in Perfetto to specific operations. For example, in the ResNet50 model, as shown in the PyTorch code, the Bottleneck module has only one explicit ReLU attribute, self.relu, which holds an nn.ReLU instance. In this case, as seen in the profiling results above, relu is omitted from the command names of the second and third Neural Engine Clusters commands, because the same self.relu attribute is repeatedly used in the forward function. Only the first command retains relu.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

# YOUR_PYTHON_PACKAGE_PATH/torchvision/models/resnet.py
class Bottleneck(nn.Module):
    ...
    def __init__(...) -> None:
        super().__init__()
        ...
        # self.relu = nn.ReLU(inplace=True)
        ...
        self.relu1 = nn.ReLU(inplace=True)
        self.relu2 = nn.ReLU(inplace=True)
        self.relu3 = nn.ReLU(inplace=True)

    def forward(self, x: Tensor) -> Tensor:
        ...
        out = self.relu1(out)
        ...
        out = self.relu2(out)
        ...
        out = self.relu3(out)

        return out

If you modify the forward function in the Bottleneck module to use different attributes (self.relu1, self.relu2, self.relu3), as shown in the PyTorch code above, relu is explicitly included not only in the first Neural Engine Clusters command name (1) but also in subsequent ones (2) and (3), without being omitted.