RBLN System Management Daemon¶

Functionality¶

rbln-smd performs the following:

• Monitors kernel events via netlink sockets and persists them as CSV logs
• Exposes device telemetry (temperature, power, memory, clock, utilization) through the gRPC interface
• Reports which host processes and execution contexts currently occupy the NPUs
• Handles device control operations such as resets
• Manages RSD groups (creation and destruction)
• Runs on-demand hardware validation diagnostics
• Enforces mTLS and IP/CN-based access control

Features¶

• Operates as a systemd service (rbln-smd.service) for automatic startup and lifecycle management
• Supports both TCP (default port 50051) and Unix domain sockets for client connections
• Configurable event and coredump retention with automatic rotation

Configuration¶

The following environment variables control daemon behavior:

• RBLN_SMD_URI: gRPC listen URI (default: [::]:50051). Supplies --uri when that flag is absent, so passing the flag overrides this variable. Takes precedence over RBLN_SMD_PORT. Accepts a Unix domain socket, for example unix:///var/run/rbln.sock.
• RBLN_SMD_PORT (default: 50051): TCP port for the gRPC server. Ignored when RBLN_SMD_URI is set.
• RSMD_TLS_CA_CERT / RSMD_TLS_SERVER_CERT / RSMD_TLS_SERVER_KEY: Certificate trio (CA certificate, server certificate, server private key). All three must be set together.
• RSMD_MAX_EVENTS_PER_DEV (default: 1000): Maximum events retained per device. The oldest events are removed when the limit is reached.
• RSMD_MAX_COREDUMPS_PER_DEV (default: 200): Maximum coredump directories retained per device. The oldest entries are removed when the limit is reached.
• RSMD_FWLOG_MAX_LINES_PER_CP (default: 10000, range 1-32768): Maximum firmware-log entries captured per Command Processor into a coredump. A higher value reaches further back into a wrapped ring, at the cost of a longer device read while the kernel driver is blocked during a reset.
• RSMD_LOG_LEVEL (default: 6): Log verbosity. 0-2 off, 3 error, 4-5 warning, 6 info, 7 debug, 8 verbose.
• RSMD_REQUIRE_OPENAT2: Require openat2 for certificate reads, so the daemon does not start on a kernel without it (before Linux 5.6). Unset, the daemon falls back to the older path check on such a kernel.
• RSMD_SKIP_CERT_DIR_PERM_CHECK: Skip the certificate ownership and permission check at startup. Set it only where the deployment enforces those permissions by other means.

Command-line Flags¶

The daemon accepts the following flags (append to ExecStart= in the systemd unit, or override with systemctl edit):

• --mtls: Enable mTLS mode.
• --allow-reset: Allow device reset calls. Without this flag, reset is always denied.
• --ca-cert <file>: CA certificate used to verify client certificates.
• --cert <file>: Server certificate.
• --key <file>: Server private key.
• -a, --allowlist <file>: Per-client access restriction file (IP / CN based).
• --uri <uri>: Alternate binding (e.g., unix:///var/run/rbln.sock).
• --pid-file <path>: PID file path. Precedence: this flag, then RBLN_SMD_PID_PATH, then /run/rbln-smd.pid.

--ca-cert, --cert, and --key form one tier and must be supplied together. See Access Control and mTLS for how the daemon picks a certificate tier and for the behavior of --mtls, --allow-reset, and -a, --allowlist.

Service Management¶

The daemon runs as a systemd service. Service status can be checked with:

1

sudo systemctl status rbln-smd

Logs are available through the systemd journal:

1

sudo journalctl -u rbln-smd -f

The daemon can be configured to listen on a Unix domain socket instead of the TCP port by specifying the --uri option in the systemd service configuration. The two are mutually exclusive: with a Unix socket, no TCP listener is opened.

1

rbln-smd --uri unix:///var/run/rbln.sock

Event Logs¶

Device events are stored in /var/log/rebellions/rsmd_<device>.event as CSV files. These logs contain:

• Event type and source
• Timestamps (UTC and kernel time)
• Event data and sub-values

Available Services¶

Each service, with the RPCs this page documents. The proto file is the authoritative, complete list.

Operation Types¶

Every RPC declares an operation type as a method option, so a client can reason about a method's effect without reading prose:

Queries and streams are observational; commands are state-mutating. The daemon may require a stricter privilege tier for commands — see Access Control and mTLS.

Concurrency¶

Queries and streams never block and are never rejected for contention. Commands serialize against one another, and how widely a command serializes depends on what it touches. The table covers the commands this page documents; the proto file is the authoritative, complete list.

For the other commands, a client issuing commands concurrently should read ABORTED as "the daemon is busy with a conflicting command, retry" rather than as a failure of the request itself.

Message Types¶

These types appear across requests and responses, independent of any one service:

These are the entries a service returns:

Protocol Definition¶

The protocol buffer definition files are typically installed in /opt/rebellions/etc/ or a similar system location. Client applications use them to generate language-specific bindings for gRPC communication with the daemon.

Common Options¶

Most subcommands accept:

• --ip <address>: gRPC server IP (default: localhost)
• --port <number>: gRPC server port (overrides ~/.rbln-smdi/config)
• --jsons: Output in JSON format instead of tables

Global flags appear before the subcommand:

• --ca-cert <file>: CA certificate used to verify the server certificate.
• --cert <file>: Client certificate.
• --key <file>: Client private key.
• --uri <URI>: Daemon connection URI (e.g., unix:///var/run/rbln.sock). Overrides --ip and --port.
• -V, --version: Print the CLI version.

--ca-cert, --cert, and --key form one tier and must be supplied together. See Access Control and mTLS for how the client picks a certificate tier.

Device and Group Filters¶

status, topo, journal, and ps accept the same two filter options:

• -d <SPEC>: Device filter. Default: all devices.
• -g <SPEC>: RSD group filter. Ignored when -d is also specified.

-d accepts a device name, a range, a comma-separated list, a glob, or all:

-g accepts a group id, a range, or a comma-separated list — for example 1, 1..3, or 1,2.

A device named in the spec that is not present is reported as a warning and dropped, so a typo never silently widens the query to every device.

Environment Variables¶

• RSMD_IF_CA_PATH / RSMD_IF_CERT_PATH / RSMD_IF_KEY_PATH: Certificate trio (CA certificate, client certificate, client private key). All three must be set together.
• RBLN_SMDI_VERBOSE: Set to 1, true, yes, or on to print the resolved certificate tier and the three certificate paths to stderr at connection time. Off by default, so stderr stays clean for scripts.

Persistent Configuration¶

The file ~/.rbln-smdi/config accepts the following keys:

• secure=true|false: Whether to use TLS (default false).
• port=<number>: Default port used when --port is not specified.
• ca_path=<file> / cert_path=<file> / key_path=<file>: Certificate trio. All three must be set together.

Access Control Flags¶

Operating Modes¶

Allowlist¶

The allowlist is an INI-style file with [basic] and [privileged] sections. When the specified file does not exist, the daemon creates a template with guidance comments.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# rbln-smd allowlist configuration
# Changes take effect on the next gRPC request (no restart needed).

[basic]
# Queries and streams (ListDevices, GetVersion, GetHardware, etc.)
192.168.1.0/24
::1
monitoring-service

[privileged]
# All operations, including commands such as ResetDevice
# [privileged] implies [basic] access.
10.0.0.1
admin-client

On the v1 API, [basic] covers queries and streams, and every command requires [privileged]; ResetDevice also requires --allow-reset. The legacy API restricts only reset to [privileged].

Supported entry types:

• IPv4 address: 192.168.1.100
• IPv4 CIDR: 192.168.10.0/24
• IPv6 address: ::1
• IPv6 CIDR: fe80::/10
• CN string: client-service-name (evaluated only on mTLS channels)

Channel rules:

• Insecure channel: Only IP / CIDR entries are evaluated.

• mTLS channel: Entries are evaluated in this order, and the first matching step decides the result:

  1. CN listed in [privileged] → privileged
  2. CN listed in [basic] → basic
  3. IP / CIDR matches [privileged] → privileged
  4. IP / CIDR matches [basic] → basic
  5. No match → denied

  When a client is registered under both CN and IP, the earlier step wins. For example, a client whose CN appears in [basic] while its IP appears in [privileged] resolves to basic.

The allowlist file is checked for modifications via mtime on each gRPC request. Changes take effect on the next request without restarting the daemon. Entities that do not appear in any section are denied all access, including monitoring APIs.

Server Certificates¶

With --mtls enabled, the daemon needs three components: a CA certificate to verify client certificates, its own server certificate, and the matching private key.

The three are resolved as one tier — all three come from the same source. The daemon takes the first complete tier:

1. CLI: --ca-cert, --cert, --key
2. Environment: RSMD_TLS_CA_CERT, RSMD_TLS_SERVER_CERT, RSMD_TLS_SERVER_KEY
3. OS default directories: the daemon searches for known filenames, splitting public certificates from the private key per the Linux PKI convention

Mixing tiers, or setting only one or two components of a tier, is rejected when the daemon starts rather than at the first connection, and the diagnostic names the missing component. Certificates are resolved only when --mtls is set; insecure mode does not require them.

For the third tier, the directories are:

Within those directories the daemon takes the first filename that exists, in this order:

Restrict the private key so users other than the daemon cannot read it (for example, the daemon's user account with mode 0600).

At startup the daemon also checks each certificate file and the directory that holds it: both must be owned by root or by the daemon's user, and neither may be group- or other-writable. If the check fails, the daemon does not start. Set RSMD_SKIP_CERT_DIR_PERM_CHECK=1 to skip the check where the deployment enforces these permissions by other means, such as a read-only mount.

A symbolic link is accepted as long as it resolves within the certificate directory — on the same mount point and under that directory. A link pointing outside it is refused. This keeps the indirect layout that Kubernetes presents for a mounted certificate Secret working.

On Linux 5.6 and later the daemon reads the trio through openat2, which applies a stricter rule: it refuses an absolute link even when its target sits inside the directory. Write in-directory links as relative ones — the Kubernetes layout already does. On a kernel without openat2 the daemon falls back to a path check that does not draw that distinction, and an absolute in-directory link loads. Set RSMD_REQUIRE_OPENAT2=1 to require openat2; the daemon then does not start on such a kernel.

TLS Activation¶

The client needs three components: a CA certificate to verify the server certificate, its own client certificate, and the matching private key. As on the daemon side, the three are resolved as one tier, and the first complete tier wins:

1. CLI: --ca-cert, --cert, --key — applies to the current invocation.
2. Environment: RSMD_IF_CA_PATH, RSMD_IF_CERT_PATH, RSMD_IF_KEY_PATH — applies per shell session.
3. Config file: the ca_path, cert_path, and key_path keys in ~/.rbln-smdi/config, used when secure=true — a persistent setting that applies to every subsequent invocation.

Mixing tiers is rejected when the client starts. A partly-filled tier is not always rejected, though: three older forms are still accepted and resolved into the CLI tier, each with a one-time warning. When TLS is off — secure=false and no certificate flag or variable is set — the client connects over the insecure channel. When TLS is on but no tier is complete, the client exits with an error at startup rather than falling back to the insecure channel.

Set RBLN_SMDI_VERBOSE=1 to have the client print which tier it resolved and the three paths it uses to stderr, which is the quickest way to tell a config-file certificate from an environment one.

Client Certificates¶

When per-client access control is required, register the CN of the client certificate in the server-side allowlist.

A symbolic link is accepted under the same rule as on the daemon side, including the openat2 caveat: it must resolve within the certificate directory, and on a kernel that provides openat2 it must also be relative.

Before it connects, the client applies the same ownership and write-permission check to its own certificates, with the client's user in place of the daemon's. RSMD_SKIP_CERT_DIR_PERM_CHECK skips it on this side too.