ros2_profiling

A set of tools and techniques for evaluating performance of ROS 2 based systems

Goals

• Provide a tool for launching arbitrary ROS 2 systems instrumented for important performance metrics:
  • System Metrics
    • CPU Usage
    • Memory Usage
    • IO Statistics
  • Instrumentation Points
    • Callback durations
    • Message latency/jitter
• Provide tooling for analyzing the results of the launched system
  • Via pytest-style assertings
  • Via plots or tool-based introspection
• Minimize the load on the system under test

Methodology

In order to minimize the system load, it was decided to use efficient formats for real-time collection and defer a majority of the processing to after the execution of the test. As a result, the implementation can be considered in three phases:

• Collection: efficiently recording instrumented data
• Post-processing: expanding the recorded formats into a more easy-to-use format
• Analysis: Perform testing/assertions on available data as well as more detailed interactive exploration.

Collection

Selected data is collected from a specified system-under-test. To collection system-wide metrics, we utilize the topnode, a component node that will instrument the parent container to track various information available through /proc endpoints. In practice, the data available from topnode largely mirrors what would be retrieved via top.

In addition to system-wide metrics, individual metrics are derived from ros2_tracing instrumentation points. ros2_tracing adds instrumentation points across the ROS 2 stack from the DDS layer up through the rclcpp client library. As these instrumentation points are reached, the relevant information is recorded in a performant manner to be analyzed after execution.

In order to accurately connect metrics across nodes, this also makes use of the message flow instrumentation points that are currently in development. These additional trace points and their use are described in Message Flow Analysis with Complex Causal Links for Distributed ROS 2 Systems.

Post-processing

Analysis

Usage

Installation

Setup LTTng and ros2_tracing

Since ros2_tracing depends on LTTng, we must first install that:

sudo apt-get update
sudo apt-get install lttng-tools liblttng-ust-dev
sudo apt-get install python3-babeltrace python3-lttng

# Also install lttng kernel tracing for additional memory/cpu/threading informatoin
sudo apt-get install lttng-modules-dkms

Additionally, if you’re using kernel tracing with a non-root user, make sure that the tracing group exists and that your user is added to it.

# Create group if it doesn't exist
sudo groupadd -r tracing
# Add user to the group
sudo usermod -aG tracing $USER

Note for Ubuntu 22.04

LTTng-UST is not built with SDT support in the package repositories. Since message flow analysis makes use of SDT support, it is necessary to build a custom copy of lttng-ust

sudo apt install systemtap-sdt-dev
git clone https://github.com/lttng/lttng-ust -b stable-2.13
cd lttng-ust
./bootstrap
./configure --with-sdt
make && make install

Building the workspace

With lttng setup, we can create and build a workspace to use the reference system

# Create Workspace
mkdir -p ~/safe_ros/src
cd ~/safe_ros

# Download and import repos file
wget https://raw.githubusercontent.com/safe-ros/ros2_profiling/main/ros2_profiling_demo/demo.repos
vcs import src < demo.repos 
rosdep install --from-paths src --ignore-src -r -y

File truncated at 100 lines see the full file