The reference system

With the distributed development of ROS across many different organizations it is sometimes hard to benchmark and concretely show how a certain change to a certain system improves or reduces the performance of that system. For example did a change from one executor to another actually reduce the CPU or was it something else entirely?

The reference_system package was developed to provide the fundamental building blocks to create complex systems that then can be used to evaluate features or performance in a standardized and repeatable way.

The first project to use this reference_system is the autoware_reference_system.

Future reference systems could be proposed that are more complex using the same basic node building blocks within the reference_system package.

Defining a reference system

A reference system is defined by:

• A fixed number of nodes
  • each node with:
    • a fixed number of publishers and subscribers
    • a fixed processing time or a fixed publishing rate
  • the reference_system base package defines reusable base node types to be used in various reference systems
• A fixed message type of fixed size to be used for every node
• For simplicity and ease of benchmarking, all nodes must run on a single process

Reference systems can run on what is referred to as a platform. A platform is defined by: - Hardware (e.g. an off-the-shelf single-board computer, embedded ECU, etc.) - if there are multiple configurations available for such hardware, ensure it is specified - Operating System (OS) like RT Linux, QNX, etc. along with any special configurations made

With these defined attributes the reference system can be replicated across many different possible configurations to be used to benchmark each configuration against the other in a reliable and fair manner.

With this approach portable and repeatable tests can also be defined to reliably confirm if a given reference system meets the requirements.

Supported Platforms

To enable as many people as possible to replicate this reference system, the platform(s) were chosen to be easily accessible (inexpensive, high volume), have lots of documentation, large community use and will be supported well into the future.

Platforms were not chosen for performance of the reference system - we know we could run “faster” with a more powerful CPU or GPU but then it would be harder for others to validate findings and test their own configurations. Accessibility is the key here and will be considered if more platforms want to be added to this benchmark list.

Platforms:

• Raspberry Pi 4B:
  • 4 GB RAM version is the assumed default
    • other versions could also be tested / added by the community
  • real-time Linux kernel

Note: create an issue to add more platforms to the list, keeping in mind the above criteria

!!! warning Each reference system can be run on other targets as well however the results will change drastically depending on the specifications of the target hardware.

Implemented reference systems

The first reference system benchmark proposed is based on the Autoware.Auto LiDAR data pipeline as stated above and shown in the node graph image above as well.

1. Autoware Reference System
   • ROS2
     • Executors
       • Single Threaded
       • Static Single Threaded
       • Multithreaded
       • Callback Group
       • Prioritized

Results below show various characteristics of the same simulated system (Autoware.Auto).

Setup Raspberry Pi 4 for the test

The goal is to provide a clean computation environment for the test avoiding an interference of other Ubuntu components.

Setup a constant CPU frequency

Frequency is setup to 1.50 GHz for all CPUs

```console

run it as root

sudo su

File truncated at 100 lines see the full file

Any contribution that you make to this repository will be under the Apache 2 License, as dictated by that license:

5. Submission of Contributions. Unless You explicitly state otherwise,
   any Contribution intentionally submitted for inclusion in the Work
   by You to the Licensor shall be under the terms and conditions of
   this License, without any additional terms or conditions.
   Notwithstanding the above, nothing herein shall supersede or modify
   the terms of any separate license agreement you may have executed
   with Licensor regarding such Contributions.