Skip to content

Model API

optimum API

Generic model classes

Classes

RBLNBaseModel

An abstract base class for compiling, loading, and saving neural network models from the huggingface transformers and diffusers libraries to run on RBLN NPU devices.

This class supports loading and saving models using the from_pretrained and save_pretrained methods, similar to the huggingface libraries.

The from_pretrained method loads a model corresponding to the given model_id from a local repository or the HuggingFace Hub onto the NPU. If the model is a PyTorch model and export=True is passed as a kwarg, it compiles the PyTorch model corresponding to the given model_id before loading. If model_id is an already rbln-compiled model, it can be directly loaded onto the NPU with export=False.

rbln_npu is a kwarg required for compilation, specifying the name of the NPU to be used. If this keyword is not specified, the NPU installed on the host machine is used. If no NPU is installed on the host machine, an error occurs.

rbln_device specifies the device to be used at runtime. If not specified, device 0 is used.

rbln_create_runtimes indicates whether to create runtime objects. If False, the runtime does not load the model onto the NPU. This option is particularly useful when you want to perform compilation only on a host machine without an NPU.

Models compiled in this way can be saved to a local repository using save_pretrained or uploaded to the huggingface hub.

It also supports generation through generate (for transformers models that support generation).

RBLNBaseModel is a class for models consisting of an arbitrary number of torch.nn.Modules, and therefore is an abstract class without explicit implementations of forward or export functions. To inherit from this class, forward, export, etc. must be implemented.

Functions
__init__(models, config, preprocessors=None, rbln_config=None, rbln_device=None, **kwargs)

Parameters:

Name Type Description Default
models List[RBLNCompiledModel]

List of RBLNCompiledModels

required
config PretrainedConfig

PretrainedConfig

required
preprocessors Optional[List]

List of Preprocessors

None
rbln_config Optional[RBLNConfig]

RBLNConfig

None
rbln_device Optional[List[int]]

List of device IDs

None
RBLNModel

A class that inherits from RBLNBaseModel for models consisting of a single torch.nn.Module.

This class supports all the functionality of RBLNBaseModel, including loading and saving models using the from_pretrained and save_pretrained methods, compiling PyTorch models for execution on RBLN NPU devices.

model = RBLNModel.from_pretrained("model_id", export=True, rbln_npu="npu_name")
outputs = model(**inputs)

Natural Language Processing

Classes

RBLNLlamaForCausalLM

The Llama model transformer with a language modeling head (linear layer) on top. This model inherits from [RBLNBaseModel]. Check the superclass documentation for the generic methods the library implements for all its models. A class to convert and run pre-trained HuggingFace transformer-based LlamaForCausalLM. It implements the methods to convert a pre-trained transformers LlamaForCausalLM into a RBLNLlamaForCausalLM by:

  • transferring the checkpoint weights of the original into an optimized RBLN graph,
  • compiling the resulting graph using the RBLN compiler.
Functions
from_pretrained(model_id, export=False, rbln_npu=None, rbln_device=0, rbln_create_runtimes=None, rbln_batch_size=1, rbln_max_seq_len=None, rbln_tensor_parallel_size=1, rbln_batching=None)

The from_pretrained() function is utilized in its standard form as in the HuggingFace transformers library. User can use this function to load a pre-trained model from the library.

Parameters:

Name Type Description Default
model_id Union[str, Path]

The model id of the pre-trained model to be loaded. It can be downloaded from the HuggingFace model hub or a local path, or a model id of a compiled model using the RBLN compiler.

required
export bool

A boolean flag to indicate if the model should be exported to a .rbln file.

False
rbln_npu Optional[str]

The name of the NPU to be used. If not specified, the NPU installed on the host machine is used. If no NPU is installed on the host machine, an error occurs.

None
rbln_device Optional[Union[int, List[int]]]

the device(s) to be used at runtime. If an integer is provided, it specifies the single device to use. If a list of integers is provided, it specifies the devices to use for tensor parallelism across multiple NPUs.

0
rbln_create_runtimes Optional[bool]

A flag to indicate whether to create runtime objects. If False, the runtime does not load the model onto the NPU. This option is particularly useful when you want to perform compilation only on a host machine without an NPU.

None
rbln_batch_size Optional[int]

The batch size of the model.

1
rbln_max_seq_len Optional[int]

The maximum sequence length of the model.

None
rbln_tensor_parallel_size Optional[int]

Compile and execute the model using multiple NPUs. This feature is only available on ATOM+ (RBLN-CA12). You can check the type of your current RBLN NPU using the rbln-stat command.

1
rbln_batching Optional[str]

The batching mode to be used for model execution. Can be either "static" or "vllm".

  • "static" (default): Uses the standard transformer behavior.
  • "vllm": Compiles the model for continuous batching mode, which integrates with the vllm-rbln library.

"vllm" mode can provide significant performance benefits in LLM serving scenarios, particularly for handling multiple requests concurrently. It optimizes memory usage and computational efficiency, potentially leading to higher throughput and lower latency in production environments. For more information on continuous batching and its benefits, refer to this blog post

None
save_pretrained(save_directory)

Saves a model and its configuration file to a directory, so that it can be re-loaded using the [from_pretrained] class method.

Parameters:

Name Type Description Default
save_directory Union[str, PathLike]

The directory to save the model and its configuration files. Will be created if it doesn't exist.

required
generate(input_ids, attention_mask=None, max_length=None)

The generate function is utilized in its standard form as in the HuggingFace transformers library. User can use this function to generate text from the model.

Parameters:

Name Type Description Default
input_ids LongTensor

The sequence used as a prompt for the generation

required
attention_mask Optional[Tensor]

The attention mask to apply on the sequence

None
max_length Optional[int]

The maximum length of the sequence to be generated

None

Returns:

Type Description

torch.Tensor

RBLNGPT2LMHeadModel

The GPT2 Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings).

This model inherits from [RBLNBaseModel]. Check the superclass documentation for the generic methods the library implements for all its model.

It implements the methods to convert a pre-trained GPT2LMHeadModel into RBLNGPT2LMHeadModel by:

  • transferring the checkpoint weights of the original into an optimized RBLN graph,
  • compiling the resulting graph using the RBLN compiler.
Functions
from_pretrained(model_id, export=False, rbln_npu=None, rbln_device=0, rbln_create_runtimes=None, rbln_batch_size=1, rbln_max_seq_len=None)

The from_pretrained function is utilized in its standard form as in the HuggingFace transformers library. User can use this function to load a pre-trained model from the library.

Parameters:

Name Type Description Default
model_id Union[str, Path]

The model id of the pre-trained model to be loaded. It can be downloaded from the HuggingFace model hub or a local path, or a model id of a compiled model using the RBLN compiler.

required
export bool

A boolean flag to indicate if the model should be exported to a .rbln file.

False
rbln_npu Optional[str]

The name of the NPU to be used. If not specified, the NPU installed on the host machine is used. If no NPU is installed on the host machine, an error occurs.

None
rbln_device Optional[int]

The device to be used at runtime. If not specified, device 0 is used.

0
rbln_create_runtimes Optional[bool]

A flag to indicate whether to create runtime objects. If False, the runtime does not load the model onto the NPU. This option is particularly useful when you want to perform compilation only on a host machine without an NPU.

None
rbln_batch_size Optional[int]

The batch size of the model.

1
rbln_max_seq_len Optional[int]

The maximum sequence length of the model.

None
save_pretrained(save_directory)

Saves a model and its configuration file to a directory, so that it can be re-loaded using the [from_pretrained] class method.

Parameters:

Name Type Description Default
save_directory Union[str, PathLike]

The directory to save the model and its configuration files. Will be created if it doesn't exist.

required
generate(input_ids, attention_mask=None, max_length=None)

The generate function is utilized in its standard form as in the HuggingFace transformers library. User can use this function to generate text from the model. Custom configuration is available like input_ids, attention_mask, max_length, etc.

Parameters:

Name Type Description Default
input_ids LongTensor

The sequence used as a prompt for the generation

required
attention_mask Optional[Tensor]

The attention mask to apply on the sequence

None
max_length Optional[int]

The maximum length of the sequence to be generated

None

Returns:

Type Description

torch.Tensor

RBLNT5ForConditionalGeneration

RBLN implementation of T5ForConditionalGeneration, optimized for NPU execution.

This class provides an interface compatible with HuggingFace's T5ForConditionalGeneration, but with RBLN-specific optimizations. It implements three key methods:

  • from_pretrained: Loads a pre-trained T5 model and converts it into an optimized RBLN graph.
  • save_pretrained: Saves the compiled RBLN model for efficient reuse.
  • generate: Generates new text sequences based on input prompts, similar to the HuggingFace implementation.
Functions
from_pretrained(model_id, export=False, rbln_npu=None, rbln_device=0, rbln_create_runtimes=None, rbln_batch_size=1, rbln_enc_max_seq_len=None, rbln_dec_max_seq_len=None)

The from_pretrained function is utilized in its standard form as in the HuggingFace transformers library. User can use this function to load a pre-trained model from the library.

Parameters:

Name Type Description Default
model_id Union[str, Path]

The model id of the pre-trained model to be loaded. It can be downloaded from the HuggingFace model hub or a local path, or a model id of a compiled model using the RBLN compiler.

required
export bool

A boolean flag to indicate if the model should be exported to a .rbln file.

False
rbln_npu Optional[str]

The name of the NPU to be used. If not specified, the NPU installed on the host machine is used. If no NPU is installed on the host machine, an error occurs.

None
rbln_device Optional[int]

The device to be used at runtime. If not specified, device 0 is used.

0
rbln_create_runtimes Optional[bool]

A flag to indicate whether to create runtime objects. If False, the runtime does not load the model onto the NPU. This option is particularly useful when you want to perform compilation only on a host machine without an NPU.

None
rbln_batch_size Optional[int]

The batch size of the model.

1
rbln_enc_max_seq_len Optional[int]

The maximum sequence length of the encoder. If not specified, model config's value is used.

None
rbln_dec_max_seq_len Optional[int]

The maximum sequence length of the decoder. If not specified, model config's value is used.

None
save_pretrained(save_directory)

Saves a model and its configuration file to a directory, so that it can be re-loaded using the [from_pretrained] class method.

Parameters:

Name Type Description Default
save_directory Union[str, PathLike]

The directory to save the model and its configuration files. Will be created if it doesn't exist.

required
generate(input_ids, attention_mask=None, max_length=None)

The generate function is utilized in its standard form as in the HuggingFace transformers library. User can use this function to generate text from the model.

Parameters:

Name Type Description Default
input_ids LongTensor

The sequence used as a prompt for the generation

required
attention_mask Optional[Tensor]

The attention mask to apply on the sequence

None
max_length Optional[int]

The maximum length of the sequence to be generated

None

Returns:

Type Description

torch.Tensor

RBLNBartForConditionalGeneration

RBLN implementation for BART (Bidirectional and Auto-Regressive Transformers), optimized for NPU execution.

This class provides an interface compatible with HuggingFace's BartForConditionalGeneration, but with RBLN-specific optimizations. It implements three key methods:

  • from_pretrained: Loads a pre-trained BartForConditionalGeneration model and converts it into an optimized RBLN graph.
  • save_pretrained: Saves the compiled RBLN model for efficient reuse.
  • generate: Generates new text sequences based on input prompts, similar to the HuggingFace implementation.

Note: As of now, beam search in the generate method is not supported.

Functions
from_pretrained(model_id, export=False, rbln_npu=None, rbln_device=0, rbln_create_runtimes=None, rbln_batch_size=1, rbln_enc_max_seq_len=None, rbln_dec_max_seq_len=None)

The from_pretrained function is utilized in its standard form as in the HuggingFace transformers library. User can use this function to load a pre-trained model from the library.

Parameters:

Name Type Description Default
model_id Union[str, Path]

The model id of the pre-trained model to be loaded. It can be downloaded from the HuggingFace model hub or a local path, or a model id of a compiled model using the RBLN compiler.

required
export bool

A boolean flag to indicate if the model should be exported to a .rbln file.

False
rbln_npu Optional[str]

The name of the NPU to be used. If not specified, the NPU installed on the host machine is used. If no NPU is installed on the host machine, an error occurs.

None
rbln_device Optional[int]

The device to be used at runtime. If not specified, device 0 is used.

0
rbln_create_runtimes Optional[bool]

A flag to indicate whether to create runtime objects. If False, the runtime does not load the model onto the NPU. This option is particularly useful when you want to perform compilation only on a host machine without an NPU.

None
rbln_batch_size Optional[int]

The batch size of the model.

1
rbln_enc_max_seq_len Optional[int]

The maximum sequence length of the encoder. If not specified, model config's value is used.

None
rbln_dec_max_seq_len Optional[int]

The maximum sequence length of the decoder. If not specified, model config's value is used.

None
save_pretrained(save_directory)

Saves a model and its configuration file to a directory, so that it can be re-loaded using the [from_pretrained] class method.

Parameters:

Name Type Description Default
save_directory Union[str, PathLike]

The directory to save the model and its configuration files. Will be created if it doesn't exist.

required
generate(input_ids, attention_mask=None, max_length=None)

The generate function is utilized in its standard form as in the HuggingFace transformers library. User can use this function to generate text from the model.

Parameters:

Name Type Description Default
input_ids LongTensor

The sequence used as a prompt for the generation

required
attention_mask Optional[Tensor]

The attention mask to apply on the sequence

None
max_length Optional[int]

The maximum length of the sequence to be generated

None

Returns:

Type Description

torch.Tensor

RBLNBertForQuestionAnswering

RBLN implementation for BertForQuestionAnswering, optimized for execution on NPU devices.

This class provides an interface compatible with HuggingFace's BertForQuestionAnswering, but with optimizations for RBLN NPUs. It implements two key methods:

  • from_pretrained: Loads a pre-trained BertForQuestionAnswering model and converts it into an optimized RBLN graph.
  • save_pretrained: Saves the compiled RBLN model for efficient reuse.
Functions
from_pretrained(model_id, export=False, rbln_npu=None, rbln_device=0, rbln_create_runtimes=None, rbln_batch_size=1, rbln_max_seq_len=None)

The from_pretrained function is utilized in its standard form as in the HuggingFace transformers library. User can use this function to load a pre-trained model from the library.

Parameters:

Name Type Description Default
model_id Union[str, Path]

The model id of the pre-trained model to be loaded. It can be downloaded from the HuggingFace model hub or a local path, or a model id of a compiled model using the RBLN compiler.

required
export bool

A boolean flag to indicate if the model should be exported to a .rbln file.

False
rbln_npu Optional[str]

The name of the NPU to be used. If not specified, the NPU installed on the host machine is used. If no NPU is installed on the host machine, an error occurs.

None
rbln_device Optional[int]

The device to be used at runtime. If not specified, device 0 is used.

0
rbln_create_runtimes Optional[bool]

A flag to indicate whether to create runtime objects. If False, the runtime does not load the model onto the NPU. This option is particularly useful when you want to perform compilation only on a host machine without an NPU.

None
rbln_batch_size Optional[int]

The batch size of the model.

1
rbln_max_seq_len Optional[int]

The maximum sequence length of the model.

None
save_pretrained(save_directory)

Saves a model and its configuration file to a directory, so that it can be re-loaded using the [from_pretrained] class method.

Parameters:

Name Type Description Default
save_directory Union[str, PathLike]

The directory to save the model and its configuration files. Will be created if it doesn't exist.

required

Stable Diffusion

Classes

RBLNStableDiffusionXLPipeline

Pipeline for text-to-image generation using Stable Diffusion XL.

This model inherits from [StableDiffusionXLPipeline]. Check the superclass documentation for the generic methods the library implements for all the pipelines (such as downloading or saving, etc.)

It implements the methods to convert a pre-trained StableDiffusionXLPipeline into a RBLNStableDiffusionXLPipeline by:

  • transferring the checkpoint weights of the original into an optimized RBLN graph,
  • compiling the resulting graph using the RBLN compiler.
Functions
from_pretrained(model_id, export=False, rbln_npu=None, rbln_device=0, rbln_create_runtimes=None, rbln_guidance_scale=5.0)

The from_pretrained() function is utilized in its standard form as in the HuggingFace diffusers library. User can use this function to load a pre-trained model from the library.

Parameters:

Name Type Description Default
model_id `Union[str, Path]`

Can be either:

  • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
  • A path to a directory containing a model saved using save_pretrained,
required
export bool

A boolean flag to indicate if the model should be exported to a .rbln file.

False
rbln_npu Optional[str]

The name of the NPU to be used. If not specified, the NPU installed on the host machine is used. If no NPU is installed on the host machine, an error occurs.

None
rbln_device Optional[int]

The device to be used at runtime. If not specified, device 0 is used.

0
rbln_create_runtimes Optional[bool]

A flag to indicate whether to create runtime objects. If False, the runtime does not load the model onto the NPU. This option is particularly useful when you want to perform compilation only on a host machine without an NPU.

None
rbln_guidance_scale float

The guidance scale value, which should be specified at compile time as it affects the input shape of the unet submodule in Stable Diffusion by determining whether classifier-free guidance is used. A higher value encourages the model to generate images closely linked to the text prompt, potentially at the expense of lower image quality.

5.0
save_pretrained(save_directory)

Saves a model and its configuration file to a directory, so that it can be re-loaded using the [from_pretrained] class method.

Parameters:

Name Type Description Default
save_directory Union[str, PathLike]

The directory to save the model and its configuration files. Will be created if it doesn't exist.

required
__call__(prompt=None, num_inference_steps=50, guidance_scale=5.0, generator=None)

Function invoked when calling the pipeline for generation.

Parameters:

Name Type Description Default
prompt Union[str, List[str]]

The prompt or prompts to guide the image generation. If not defined, one has to pass prompt_embeds. instead.

None
num_inference_steps int

The number of denoising steps. More denoising steps usually lead to a higher quality image at the expense of slower inference.

50
guidance_scale float

Guidance scale as defined in Classifier-Free Diffusion Guidance. guidance_scale is defined as w of equation 2. of Imagen Paper. Guidance scale is enabled by setting guidance_scale > 1. Higher guidance scale encourages to generate images that are closely linked to the text prompt, usually at the expense of lower image quality.

5.0
generator Optional[Union[Generator, List[Generator]]]

One or a list of torch generator(s) to make generation deterministic.

None

Returns:

Type Description
StableDiffusionPipelineOutput

Generated Images and bools indicating whether the corresponding generated image contains "not-safe-for-work" (nsfw) content.

RBLNStableDiffusionXLImg2ImgPipeline

Pipeline for image-to-image generation using Stable Diffusion XL.

This model inherits from [StableDiffusionXLPipeline]. Check the superclass documentation for the generic methods the library implements for all the pipelines (such as downloading or saving, etc.)

It implements the methods to convert a pre-trained StableDiffusionXLPipeline into a RBLNStableDiffusionXLPipeline by:

  • transferring the checkpoint weights of the original into an optimized RBLN graph,
  • compiling the resulting graph using the RBLN compiler.
Functions
from_pretrained(model_id, rbln_img_width, rbln_img_height, export=False, rbln_npu=None, rbln_device=0, rbln_create_runtimes=None, rbln_guidance_scale=5.0)

The from_pretrained() function is utilized in its standard form as in the HuggingFace diffusers library. User can use this function to load a pre-trained model from the library.

Parameters:

Name Type Description Default
model_id `Union[str, Path]`

Can be either:

  • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
  • A path to a directory containing a model saved using save_pretrained,
required
rbln_img_width int

The width of the image to be generated.

required
rbln_img_height int

The height of the image to be generated.

required
export bool

A boolean flag to indicate if the model should be exported to a .rbln file.

False
rbln_npu Optional[str]

The name of the NPU to be used. If not specified, the NPU installed on the host machine is used. If no NPU is installed on the host machine, an error occurs.

None
rbln_device Optional[int]

The device to be used at runtime. If not specified, device 0 is used.

0
rbln_create_runtimes Optional[bool]

A flag to indicate whether to create runtime objects. If False, the runtime does not load the model onto the NPU. This option is particularly useful when you want to perform compilation only on a host machine without an NPU.

None
rbln_guidance_scale float

The guidance scale value, which should be specified at compile time as it affects the input shape of the unet submodule in Stable Diffusion by determining whether classifier-free guidance is used. A higher value encourages the model to generate images closely linked to the text prompt, potentially at the expense of lower image quality.

5.0
save_pretrained(save_directory)

Saves a model and its configuration file to a directory, so that it can be re-loaded using the [from_pretrained] class method.

Parameters:

Name Type Description Default
save_directory Union[str, PathLike]

The directory to save the model and its configuration files. Will be created if it doesn't exist.

required
__call__(prompt=None, image=None, strength=0.8, num_inference_steps=50, guidance_scale=7.5, generator=None)

The call function to the pipeline for generation.

Parameters:

Name Type Description Default
prompt Union[str, List[str]]

The prompt or prompts to guide image generation. If not defined, you need to pass prompt_embeds.

None
image PipelineImageInput

Image, numpy array or tensor representing an image batch to be used as the starting point. For both numpy array and pytorch tensor, the expected value range is between [0, 1] If it's a tensor or a list or tensors, the expected shape should be (B, C, H, W) or (C, H, W). If it is a numpy array or a list of arrays, the expected shape should be (B, H, W, C) or (H, W, C) It can also accept image latents as image, but if passing latents directly it is not encoded again.

None
strength `float`, *optional*, defaults to 0.8

Indicates extent to transform the reference image. Must be between 0 and 1. image is used as a starting point and more noise is added the higher the strength. The number of denoising steps depends on the amount of noise initially added. When strength is 1, added noise is maximum and the denoising process runs for the full number of iterations specified in num_inference_steps. A value of 1 essentially ignores image.

0.8
num_inference_steps int

The number of denoising steps. More denoising steps usually lead to a higher quality image at the expense of slower inference.

50
guidance_scale Optional[float]

A higher guidance scale value encourages the model to generate images closely linked to the text prompt at the expense of lower image quality. Guidance scale is enabled when guidance_scale > 1.

7.5
generator Optional[Union[Generator, List[Generator]]]

A torch.Generator to make generation deterministic.

None

Returns:

Type Description
StableDiffusionPipelineOutput

Generated Images and bools indicating whether the corresponding generated image contains "not-safe-for-work" (nsfw) content.

RBLNStableDiffusionPipeline

Pipeline for text-to-image generation using Stable Diffusion.

This model inherits from [StableDiffusionPipeline]. Check the superclass documentation for the generic methods implemented for all pipelines (downloading, saving, etc.).

It implements the methods to convert a pre-trained StableDiffusionPipeline into a RBLNStableDiffusionPipeline by:

  • transferring the checkpoint weights of the original into an optimized RBLN graph,
  • compiling the resulting graph using the RBLN compiler.
Functions
from_pretrained(model_id, export=False, rbln_npu=None, rbln_device=0, rbln_create_runtimes=None, rbln_guidance_scale=5.0)

The from_pretrained() function is utilized in its standard form as in the HuggingFace diffusers library. User can use this function to load a pre-trained model from the library.

Parameters:

Name Type Description Default
model_id `Union[str, Path]`

Can be either:

  • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
  • A path to a directory containing a model saved using save_pretrained,
required
export bool

A boolean flag to indicate if the model should be exported to a .rbln file.

False
rbln_npu Optional[str]

The name of the NPU to be used. If not specified, the NPU installed on the host machine is used. If no NPU is installed on the host machine, an error occurs.

None
rbln_device Optional[int]

The device to be used at runtime. If not specified, device 0 is used.

0
rbln_create_runtimes Optional[bool]

A flag to indicate whether to create runtime objects. If False, the runtime does not load the model onto the NPU. This option is particularly useful when you want to perform compilation only on a host machine without an NPU.

None
rbln_guidance_scale float

The guidance scale value, which should be specified at compile time as it affects the input shape of the unet submodule in Stable Diffusion by determining whether classifier-free guidance is used. A higher value encourages the model to generate images closely linked to the text prompt, potentially at the expense of lower image quality.

5.0
save_pretrained(save_directory)

Saves a model and its configuration file to a directory, so that it can be re-loaded using the [from_pretrained] class method.

Parameters:

Name Type Description Default
save_directory Union[str, PathLike]

The directory to save the model and its configuration files. Will be created if it doesn't exist.

required
__call__(prompt=None, num_inference_steps=50, guidance_scale=7.5, generator=None)

The call function to the pipeline for generation.

Parameters:

Name Type Description Default
prompt Union[str, List[str]]

The prompt or prompts to guide image generation. If not defined, you need to pass prompt_embeds.

None
num_inference_steps int

The number of denoising steps. More denoising steps usually lead to a higher quality image at the expense of slower inference.

50
guidance_scale float

A higher guidance scale value encourages the model to generate images closely linked to the text prompt at the expense of lower image quality. Guidance scale is enabled when guidance_scale > 1.

7.5
generator Optional[Union[Generator, List[Generator]]]

A torch.Generator to make generation deterministic.

None

Returns:

Type Description
StableDiffusionPipelineOutput

Generated Images and bools indicating whether the corresponding generated image contains "not-safe-for-work" (nsfw) content.

RBLNStableDiffusionImg2ImgPipeline

Pipeline for image-to-image generation using Stable Diffusion.

This model inherits from [StableDiffusionPipeline]. Check the superclass documentation for the generic methods implemented for all pipelines (downloading, saving, etc.).

It implements the methods to convert a pre-trained StableDiffusionImg2ImgPipeline into a RBLNStableDiffusionImg2ImgPipeline by:

  • transferring the checkpoint weights of the original into an optimized RBLN graph,
  • compiling the resulting graph using the RBLN compiler.
Functions
from_pretrained(model_id, rbln_img_width, rbln_img_height, export=False, rbln_npu=None, rbln_device=0, rbln_create_runtimes=None, rbln_guidance_scale=5.0)

The from_pretrained() function is utilized in its standard form as in the HuggingFace diffusers library. User can use this function to load a pre-trained model from the library.

Parameters:

Name Type Description Default
model_id `Union[str, Path]`

Can be either:

  • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
  • A path to a directory containing a model saved using save_pretrained,
required
rbln_img_width int

The width of the image to be generated.

required
rbln_img_height int

The height of the image to be generated.

required
export bool

A boolean flag to indicate if the model should be exported to a .rbln file.

False
rbln_npu Optional[str]

The name of the NPU to be used. If not specified, the NPU installed on the host machine is used. If no NPU is installed on the host machine, an error occurs.

None
rbln_device Optional[int]

The device to be used at runtime. If not specified, device 0 is used.

0
rbln_create_runtimes Optional[bool]

A flag to indicate whether to create runtime objects. If False, the runtime does not load the model onto the NPU. This option is particularly useful when you want to perform compilation only on a host machine without an NPU.

None
rbln_guidance_scale float

The guidance scale value, which should be specified at compile time as it affects the input shape of the unet submodule in Stable Diffusion by determining whether classifier-free guidance is used. A higher value encourages the model to generate images closely linked to the text prompt, potentially at the expense of lower image quality.

5.0
save_pretrained(save_directory)

Saves a model and its configuration file to a directory, so that it can be re-loaded using the [from_pretrained] class method.

Parameters:

Name Type Description Default
save_directory Union[str, PathLike]

The directory to save the model and its configuration files. Will be created if it doesn't exist.

required
__call__(prompt=None, image=None, strength=0.8, num_inference_steps=50, guidance_scale=7.5, generator=None)

The call function to the pipeline for generation.

Parameters:

Name Type Description Default
prompt Union[str, List[str]]

The prompt or prompts to guide image generation. If not defined, you need to pass prompt_embeds.

None
image PipelineImageInput

Image, numpy array or tensor representing an image batch to be used as the starting point. For both numpy array and pytorch tensor, the expected value range is between [0, 1] If it's a tensor or a list or tensors, the expected shape should be (B, C, H, W) or (C, H, W). If it is a numpy array or a list of arrays, the expected shape should be (B, H, W, C) or (H, W, C) It can also accept image latents as image, but if passing latents directly it is not encoded again.

None
strength float

Indicates extent to transform the reference image. Must be between 0 and 1. image is used as a starting point and more noise is added the higher the strength. The number of denoising steps depends on the amount of noise initially added. When strength is 1, added noise is maximum and the denoising process runs for the full number of iterations specified in num_inference_steps. A value of 1 essentially ignores image.

0.8
num_inference_steps Optional[int]

The number of denoising steps. More denoising steps usually lead to a higher quality image at the expense of slower inference. This parameter is modulated by strength.

50
guidance_scale Optional[float]

A higher guidance scale value encourages the model to generate images closely linked to the text prompt at the expense of lower image quality. Guidance scale is enabled when guidance_scale > 1.

7.5
generator Optional[Union[Generator, List[Generator]]]

A torch.Generator to make generation deterministic.

None

Returns:

Type Description
StableDiffusionPipelineOutput

Generated Images and bools indicating whether the corresponding generated image contains "not-safe-for-work" (nsfw) content.

RBLNStableDiffusionControlNetPipeline

Pipeline for text-to-image generation using Stable Diffusion and ContorlNet.

This model inherits from [StableDiffusionControlNetPipeline]. Check the superclass documentation for the generic methods implemented for all pipelines (downloading, saving, etc.).

It implements the methods to convert a pre-trained StableDiffusionControlNetPipeline into a RBLNStableDiffusionControlNetPipeline by:

  • transferring the checkpoint weights of the original into an optimized RBLN graph,
  • compiling the resulting graph using the RBLN compiler.
Functions
from_pretrained(model_id, rbln_img_width, rbln_img_height, controlnet, export=False, rbln_npu=None, rbln_device=0, rbln_create_runtimes=None, rbln_guidance_scale=5.0)

The from_pretrained() function is utilized in its standard form as in the HuggingFace diffusers library. User can use this function to load a pre-trained model from the library.

Parameters:

Name Type Description Default
model_id `Union[str, Path]`

Can be either:

  • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
  • A path to a directory containing a model saved using save_pretrained,
required
rbln_img_width int

The width of the image to be generated.

required
rbln_img_height int

The height of the image to be generated.

required
controlnet ControlNetModel

Provides additional conditioning to the unet during the denoising process.

required
export bool

A boolean flag to indicate if the model should be exported to a .rbln file.

False
rbln_npu Optional[str]

The name of the NPU to be used. If not specified, the NPU installed on the host machine is used. If no NPU is installed on the host machine, an error occurs.

None
rbln_device Optional[int]

The device to be used at runtime. If not specified, device 0 is used.

0
rbln_create_runtimes Optional[bool]

A flag to indicate whether to create runtime objects. If False, the runtime does not load the model onto the NPU. This option is particularly useful when you want to perform compilation only on a host machine without an NPU.

None
rbln_guidance_scale float

The guidance scale value, which should be specified at compile time as it affects the input shape of the unet submodule in Stable Diffusion by determining whether classifier-free guidance is used. A higher value encourages the model to generate images closely linked to the text prompt, potentially at the expense of lower image quality.

5.0
save_pretrained(save_directory)

Saves a model and its configuration file to a directory, so that it can be re-loaded using the [from_pretrained] class method.

Parameters:

Name Type Description Default
save_directory Union[str, PathLike]

The directory to save the model and its configuration files. Will be created if it doesn't exist.

required
__call__(prompt=None, image=None, num_inference_steps=50, guidance_scale=7.5, negative_prompt=None, controlnet_conditioning_scale=1.0, generator=None)

The call function to the pipeline for generation.

Parameters:

Name Type Description Default
prompt Optional[Union[str, List[str]]]

The prompt or prompts to guide image generation. If not defined, you need to pass prompt_embeds.

None
image Optional[PipelineImageInput]

The ControlNet input condition to provide guidance to the unet for generation. If the type is specified as torch.Tensor, it is passed to ControlNet as is. PIL.Image.Image can also be accepted as an image. The dimensions of the output image defaults to image's dimensions. If height and/or width are passed, image is resized accordingly. If multiple ControlNets are specified in init, images must be passed as a list such that each element of the list can be correctly batched for input to a single ControlNet. When prompt is a list, and if a list of images is passed for a single ControlNet, each will be paired with each prompt in the prompt list. This also applies to multiple ControlNets, where a list of image lists can be passed to batch for each prompt and each ControlNet.

None
num_inference_steps int

The number of denoising steps. More denoising steps usually lead to a higher quality image at the expense of slower inference.

50
guidance_scale float

A higher guidance scale value encourages the model to generate images closely linked to the text prompt at the expense of lower image quality. Guidance scale is enabled when guidance_scale > 1.

7.5
negative_prompt Optional[Union[str, List[str]

The prompt or prompts to guide what to not include in image generation. If not defined, you need to pass negative_prompt_embeds instead. Ignored when not using guidance (guidance_scale < 1).

None
controlnet_conditioning_scale Optional[Union[float, List[float]

The outputs of the ControlNet are multiplied by controlnet_conditioning_scale before they are added to the residual in the original unet. If multiple ControlNets are specified in init, you can set the corresponding scale as a list.

1.0
generator Optional[Union[Generator, List[Generator]]]

A torch.Generator to make generation deterministic.

None
RBLNStableDiffusionControlNetImg2ImgPipeline

Pipeline for image-to-image generation using Stable Diffusion and ControlNet.

This model inherits from [StableDiffusionControlNetImg2ImgPipeline]. Check the superclass documentation for the generic methods implemented for all pipelines (downloading, saving, etc.).

It implements the methods to convert a pre-trained StableDiffusionControlNetImg2ImgPipeline into a RBLNStableDiffusionControlNetImg2ImgPipeline by:

  • transferring the checkpoint weights of the original into an optimized RBLN graph,
  • compiling the resulting graph using the RBLN compiler.
Functions
from_pretrained(model_id, rbln_img_width, rbln_img_height, controlnet, export=False, rbln_npu=None, rbln_device=0, rbln_create_runtimes=None, rbln_guidance_scale=5.0)

The from_pretrained() function is utilized in its standard form as in the HuggingFace diffusers library. User can use this function to load a pre-trained model from the library.

Parameters:

Name Type Description Default
model_id `Union[str, Path]`

Can be either:

  • A string, the model id of a pretrained model hosted inside a model repo on huggingface.co.
  • A path to a directory containing a model saved using save_pretrained,
required
rbln_img_width int

The width of the image to be generated.

required
rbln_img_height int

The height of the image to be generated.

required
controlnet ControlNetModel

Provides additional conditioning to the unet during the denoising process.

required
export bool

A boolean flag to indicate if the model should be exported to a .rbln file.

False
rbln_npu Optional[str]

The name of the NPU to be used. If not specified, the NPU installed on the host machine is used. If no NPU is installed on the host machine, an error occurs.

None
rbln_device Optional[int]

The device to be used at runtime. If not specified, device 0 is used.

0
rbln_create_runtimes Optional[bool]

A flag to indicate whether to create runtime objects. If False, the runtime does not load the model onto the NPU. This option is particularly useful when you want to perform compilation only on a host machine without an NPU.

None
rbln_guidance_scale float

The guidance scale value, which should be specified at compile time as it affects the input shape of the unet submodule in Stable Diffusion by determining whether classifier-free guidance is used. A higher value encourages the model to generate images closely linked to the text prompt, potentially at the expense of lower image quality.

5.0
save_pretrained(save_directory)

Saves a model and its configuration file to a directory, so that it can be re-loaded using the [from_pretrained] class method.

Parameters:

Name Type Description Default
save_directory Union[str, PathLike]

The directory to save the model and its configuration files. Will be created if it doesn't exist.

required
__call__(prompt=None, image=None, control_image=None, strength=0.8, num_inference_steps=50, guidance_scale=7.5, negative_prompt=None, controlnet_conditioning_scale=0.8, generator=None)

The call function to the pipeline for generation.

Parameters:

Name Type Description Default
prompt Optional[Union[str, List[str]]]

The prompt or prompts to guide image generation. If not defined, you need to pass prompt_embeds.

None
image Optional[PipelineImageInput]

The initial image to be used as the starting point for the image generation process. Can also accept image latents as image, and if passing latents directly they are not encoded again.

None
control_image Optional[PipelineImageInput]

The ControlNet input condition to provide guidance to the unet for generation. If the type is specified as torch.Tensor, it is passed to ControlNet as is. PIL.Image.Image can also be accepted as an image. The dimensions of the output image defaults to image's dimensions. If height and/or width are passed, image is resized accordingly. If multiple ControlNets are specified in init, images must be passed as a list such that each element of the list can be correctly batched for input to a single ControlNet.

None
strength float

Indicates extent to transform the reference image. Must be between 0 and 1. image is used as a starting point and more noise is added the higher the strength. The number of denoising steps depends on the amount of noise initially added. When strength is 1, added noise is maximum and the denoising process runs for the full number of iterations specified in num_inference_steps. A value of 1 essentially ignores image.

0.8
num_inference_steps int

The number of denoising steps. More denoising steps usually lead to a higher quality image at the expense of slower inference.

50
guidance_scale float

A higher guidance scale value encourages the model to generate images closely linked to the text prompt at the expense of lower image quality. Guidance scale is enabled when guidance_scale > 1.

7.5
negative_prompt Optional[Union[str, List[str]

The prompt or prompts to guide what to not include in image generation. If not defined, you need to pass negative_prompt_embeds instead. Ignored when not using guidance (guidance_scale < 1).

None
controlnet_conditioning_scale Optional[Union[float, List[float]

The outputs of the ControlNet are multiplied by controlnet_conditioning_scale before they are added to the residual in the original unet. If multiple ControlNets are specified in init, you can set the corresponding scale as a list.

0.8
generator Optional[Union[Generator, List[Generator]]]

A torch.Generator to make generation deterministic.

None

Audio

Classes

RBLNASTForAudioClassification

Audio Spectrogram Transformer model with an audio classification head on top (a linear layer on top of the pooled output) e.g. for datasets like AudioSet, Speech Commands v2. This model inherits from [RBLNModel]. Check the superclass documentation for the generic methods the library implements for all its models.

A class to convert and run pre-trained transformer-based ASTForAudioClassification models on RBLN devices. It implements the methods to convert a pre-trained transformers ASTForAudioClassification model into a RBLN transformer model by:

  • transferring the checkpoint weights of the original into an optimized RBLN graph,
  • compiling the resulting graph using the RBLN compiler.

Currently, this model class only supports the 'AST' model from the transformers library. Future updates may include support for additional model types.

Functions
from_pretrained(model_id, export=False, rbln_npu=None, rbln_device=0, rbln_create_runtimes=None, rbln_batch_size=1)

The from_pretrained function is utilized in its standard form as in the HuggingFace transformers library. User can use this function to load a pre-trained model from the library.

Parameters:

Name Type Description Default
model_id Union[str, Path]

The model id of the pre-trained model to be loaded. It can be downloaded from the HuggingFace model hub or a local path, or a model id of a compiled model using the RBLN compiler.

required
export bool

A boolean flag to indicate if the model should be exported to a .rbln file.

False
rbln_npu Optional[str]

The name of the NPU to be used. If not specified, the NPU installed on the host machine is used. If no NPU is installed on the host machine, an error occurs.

None
rbln_device Optional[int]

The device to be used at runtime. If not specified, device 0 is used.

0
rbln_create_runtimes Optional[bool]

A flag to indicate whether to create runtime objects. If False, the runtime does not load the model onto the NPU. This option is particularly useful when you want to perform compilation only on a host machine without an NPU.

None
rbln_batch_size Optional[int]

The batch size of the model.

1
save_pretrained(save_directory)

Saves a model and its configuration file to a directory, so that it can be re-loaded using the [from_pretrained] class method.

Parameters:

Name Type Description Default
save_directory Union[str, PathLike]

The directory to save the model and its configuration files. Will be created if it doesn't exist.

required
RBLNWav2Vec2ForCTC

Wav2Vec2 Model with a language modeling head on top for Connectionist Temporal Classification (CTC).

This model inherits from [RBLNModel]. Check the superclass documentation for the generic methods the library implements for all its model.

It implements the methods to convert a pre-trained Wav2Vec2ForCTC into a RBLNWav2Vec2ForCTC by:

  • transferring the checkpoint weights of the original into an optimized RBLN graph,
  • compiling the resulting graph using the RBLN compiler.
Functions
from_pretrained(model_id, export=False, rbln_npu=None, rbln_device=0, rbln_create_runtimes=None, rbln_batch_size=1)

The from_pretrained function is utilized in its standard form as in the HuggingFace transformers library. User can use this function to load a pre-trained model from the library.

Parameters:

Name Type Description Default
model_id Union[str, Path]

The model id of the pre-trained model to be loaded. It can be downloaded from the HuggingFace model hub or a local path, or a model id of a compiled model using the RBLN compiler.

required
export bool

A boolean flag to indicate if the model should be exported to a .rbln file.

False
rbln_npu Optional[str]

The name of the NPU to be used. If not specified, the NPU installed on the host machine is used. If no NPU is installed on the host machine, an error occurs.

None
rbln_device Optional[int]

The device to be used at runtime. If not specified, device 0 is used.

0
rbln_create_runtimes Optional[bool]

A flag to indicate whether to create runtime objects. If False, the runtime does not load the model onto the NPU. This option is particularly useful when you want to perform compilation only on a host machine without an NPU.

None
rbln_batch_size Optional[int]

The batch size of the model.

1
save_pretrained(save_directory)

Saves a model and its configuration file to a directory, so that it can be re-loaded using the [from_pretrained] class method.

Parameters:

Name Type Description Default
save_directory Union[str, PathLike]

The directory to save the model and its configuration files. Will be created if it doesn't exist.

required
RBLNWhisperForConditionalGeneration

The Whisper Model with a language modeling head. Can be used for automatic speech recognition.

This model inherits from [RBLNBaseModel]. Check the superclass documentation for the generic methods the library implements for all its models.

A class to convert and run pre-trained transformer-based WhisperForConditionalGeneration model on RBLN devices. It implements the methods to convert a pre-trained transformers WhisperForConditionalGeneration into a RBLNWhisperForConditionalGeneration by:

  • transferring the checkpoint weights of the original into an optimized RBLN graph,
  • compiling the resulting graph using the RBLN compiler.
Functions
from_pretrained(model_id, export=False, rbln_npu=None, rbln_device=0, rbln_create_runtimes=None, rbln_batch_size=1)

The from_pretrained function is utilized in its standard form as in the HuggingFace transformers library. User can use this function to load a pre-trained model from the library.

Parameters:

Name Type Description Default
model_id Union[str, Path]

The model id of the pre-trained model to be loaded. It can be downloaded from the HuggingFace model hub or a local path, or a model id of a compiled model using the RBLN compiler.

required
export bool

A boolean flag to indicate if the model should be exported to a .rbln file.

False
rbln_npu Optional[str]

The name of the NPU to be used. If not specified, the NPU installed on the host machine is used. If no NPU is installed on the host machine, an error occurs.

None
rbln_device Optional[int]

The device to be used at runtime. If not specified, device 0 is used.

0
rbln_create_runtimes Optional[bool]

A flag to indicate whether to create runtime objects. If False, the runtime does not load the model onto the NPU. This option is particularly useful when you want to perform compilation only on a host machine without an NPU.

None
rbln_batch_size Optional[int]

The batch size of the model.

1
save_pretrained(save_directory)

Saves a model and its configuration file to a directory, so that it can be re-loaded using the [from_pretrained] class method.

Parameters:

Name Type Description Default
save_directory Union[str, PathLike]

The directory to save the model and its configuration files. Will be created if it doesn't exist.

required
generate(input_features)

The generate function is utilized in its standard form as in the HuggingFace transformers library. User can use this function to generate text from the model.

Parameters:

Name Type Description Default
input_features LongTensor

Float values of log-mel features extracted from the raw speech waveform.

required

Returns:

Type Description

torch.Tensor

Computer Vision

Classes

RBLNResNetForImageClassification

ResNet Model with an image classification head on top (a linear layer on top of the pooled features), e.g. for ImageNet. This model inherits from [RBLNModel]. Check the superclass documentation for the generic methods the library implements for all its models.

Functions
from_pretrained(model_id, export=False, rbln_npu=None, rbln_device=0, rbln_create_runtimes=None, rbln_batch_size=1)

The from_pretrained() function is utilized in its standard form as in the HuggingFace transformers library. User can use this function to load a pre-trained model from the library.

Parameters:

Name Type Description Default
model_id Union[str, Path]

The model id of the pre-trained model to be loaded. It can be downloaded from the HuggingFace model hub or a local path, or a model id of a compiled model using the RBLN compiler.

required
export bool

A boolean flag to indicate if the model should be exported to a .rbln file.

False
rbln_npu Optional[str]

The name of the NPU to be used. If not specified, the NPU installed on the host machine is used. If no NPU is installed on the host machine, an error occurs.

None
rbln_device Optional[int]

The device to be used at runtime. If not specified, device 0 is used.

0
rbln_create_runtimes Optional[bool]

A flag to indicate whether to create runtime objects. If False, the runtime does not load the model onto the NPU. This option is particularly useful when you want to perform compilation only on a host machine without an NPU.

None
rbln_batch_size Optional[int]

The batch size of the model.

1
save_pretrained(save_directory)

Saves a model and its configuration file to a directory, so that it can be re-loaded using the [from_pretrained] class method.

Parameters:

Name Type Description Default
save_directory Union[str, PathLike]

The directory to save the model and its configuration files. Will be created if it doesn't exist.

required