performance_test

[TOC]

The performance_test tool tests latency and other performance metrics of various middleware implementations that support a pub/sub pattern. It is used to simulate non-functional performance of your application.

The performance_test tool allows you to quickly set up a pub/sub configuration, e.g. number of publisher/subscribers, message size, QOS settings, middleware. The following metrics are automatically recorded when the application is running:

• latency: corresponds to the time a message takes to travel from a publisher to subscriber. The latency is measured by timestamping the sample when it’s published and subtracting the timestamp (from the sample) from the measured time when the sample arrives at the subscriber (only logged when a subscriber is created)
• CPU usage: percentage of the total system wide CPU usage (logged separately for each instance of perf_test)
• resident memory: heap allocations, shared memory segments, stack (used for system’s internal work) (logged separately for each instance of perf_test)
• sample statistics: number of samples received, sent, and lost per experiment run.

This master branch is compatible with the following ROS 2 versions

• rolling
• jazzy
• iron
• humble
• galactic
• foxy
• eloquent
• dashing
• Apex.OS

How to use this document

1. Start here for a quick example of building and running the performance_test tool with the Cyclone DDS plugin.
2. If needed, find more detailed information about building and running
3. Or, if the quick example is good enough, skip ahead to the list of supported middleware plugins to learn how to test a specific middleware implementation.
4. Check out the tools for visualizing the results
5. If desired, read about the design and architecture of the tool.

Example

This example shows how to test the non-functional performance of the following configuration:

1. Install ROS 2

2. Install Cyclone DDS to /opt/cyclonedds

3. Build performance_test with the CMake build flag for Cyclone DDS:

    source /opt/ros/rolling/setup.bash
    cd ~/perf_test_ws
    colcon build --cmake-args -DPERFORMANCE_TEST_PLUGIN=CYCLONEDDS
    source ./install/setup.bash
    

1. Run with the communication plugin option for Cyclone DDS:

mkdir experiment
./install/performance_test/lib/performance_test/perf_test --communication CycloneDDS
                                                          --msg Array1k
                                                          --rate 100
                                                          --topic test_topic
                                                          --max-runtime 30
                                                          --logfile experiment/log.csv

At the end of the experiment, a CSV log file will be generated in the experiment folder with a name that starts with log.

Building the performance_test tool

For a simple example, see Dockerfile.rclcpp.

The performance_test tool is structured as a ROS 2 package, so colcon is used to build it. Therefore, you must source a ROS 2 installation:

source /opt/ros/rolling/setup.bash

Select a middleware plugin from this list. Then build the performance_test tool with the selected middleware:

```bash mkdir -p ~/perf_test_ws/src

File truncated at 100 lines see the full file

Changelog for package performance_test

X.Y.Z (YYYY/MM/DD)

2.3.0 (2024/09/24)

Removed

• Moved apex_performance_plotter to its own package here

2.2.0 (2024/05/15)

Added

• performance_test can be built with ROS 2 Iron and Jazzy

  Changed

• Renamed the --dds-domain_id CLI arg to --dds-domain-id
• When --dds-domain-id is unspecified, fall back to the ROS_DOMAIN_ID environment variable
• --zero-copy has been separated into two flags:
  • --shared-memory: Enable shared-memory transfer in the plugin. This is meant to replace the need to manually set runtime flags via CYCLONEDDS_URI, APEX_MIDDLEWARE_SETTINGS, etc.
  • --loaned-samples: When publishing messages in the plugin, borrow loaned samples instead of publishing by copy
  • --zero-copy is now an alias for --shared-memory --loaned-samples
  • Supported plugins include:
    • -c CycloneDDS
    • -c CycloneDDS-CXX
    • -c ApexOSPollingSubscription
    • -c rclcpp-* with RMW_IMPLEMENTATION=rmw_cyclonedds_cpp
    • -c rclcpp-* with RMW_IMPLEMENTATION=rmw_fastrtps_cpp

2.1.0 (2024/04/17)

Added

• Add new function prepare() to the Publisher and Subscriber API, intended to allow participant discovery without blocking the main thread

  Changed

• Change the default --history arg from KEEP_ALL to KEEP_LAST
• Change the default --history-depth arg from 1000 to 16
• If --expected-num-pubs is unspecified, set it to the same value as -p
• If --expected-num-subs is unspecified, set it to the same value as -s

  Fixed

• Removed an unused variable to fix a Clang build
• Remove unused variable names in the Plugin abstract class
• Fix a potential lockup in PublisherTask on QNX

2.0.0 (2024/03/19)

Added

• Add experimental bazel support
  • bazel build //performance_test --//:plugin_implementation=//path/to/a/plugin
• Add a rudimentary socket-based plugin for testing the bazel support
  • bazel run //performance_test --//:plugin_implementation=//performance_test/plugins/demo:demo_plugin -- --help

    Changed

• Instead of enabling/disabling each plugin, you select exactly one with a CMake string option, for example:
  • colcon build --cmake-args -DPERFORMANCE_TEST_PLUGIN=ROS2
• Renamed the --communication CLI arg to --communicator. The short -c is unchanged.

  Removed

• Removed the deprecated CLI flags for QOS settings:
  • Instead of --reliable, use --reliability RELIABLE
  • Instead of --transient, use --durability TRANSIENT_LOCAL
  • Instead of --keep-last, use --history KEEP_LAST
• Removed the obsolete BoundedSequenceFlat messages
• Removed the superfluous --msg-list CLI flag. The --help message already lists the available messages.

  Fixed

• Update the Apex.OS Runner to use executor_runner::deferred instead of executor_runner::deferred_tag()
• Ensure that the first few published samples are sent at the expected rate

1.5.2 (YYYY/MM/DD)

Added

• --prevent-cpu-idle is available on QNX

  Changed

• JSON log files will contain all values in the APEX_PERFORMANCE_TEST dictionary, instead of the five specific values used previously
• Switch to build as C++17 by default

  Fixed

• Zero copy transfer is again enabled for the rclcpp publisher

1.5.0 (2023/06/14)

Added

• New CLI switch --prevent-cpu-idle (linux only). When specified, perf_test will use /dev/cpu_dma_latency to request that the CPU not enter any sleep states, to potentially give more consistent results
• Some smaller Array messages, down to 32 bits
• Added support to the FastDDS plugin for bounded and unbounded sequences

  Changed

• Update the README to better explain how to use this tool with Apex.OS
• In the Runner, allocate the AnalysisResults on the stack instead of using shared_ptr
• Subscriber methods accept a callback parameter, instead of returning a vector of results, to reduce heap usage
• Refactored the interaction between SubscriberStats and AnalysisResult to remove the need for a std::vector of latency samples, to reduce heap usage
• Adjusted the Array message sizes to make the name match the contents
• Updated apex_os_communicator to use the new zero-copy API

1.4.2 (2023/03/15)

Added

File truncated at 100 lines see the full file