performance_report

[TOC]

This package serves two purposes:

1. Run multiple performance_test experiments
2. Visualize the combined results of those experiments

Quick start

Install the required dependencies:

python3 -m pip install -r third_party/python/requirements.txt
sudo apt-get install firefox-geckodriver

Note: all the commands below are run from the colcon workspace where performance_test/performance_report is installed:

# Build performance_test and performance_report
colcon build

# Set up the environment
source install/setup.bash

# Run perf_test for each experiment in the yaml file
ros2 run performance_report runner \
  --log-dir perf_logs \
  --test-name experiments \
  --configs src/performance_test/performance_report/cfg/runner/run_one_experiment.yaml

# The runner generates log files to the specified directory: `./perf_logs/experiements/`

# Generate the plots configured in the specified yaml file
ros2 run performance_report plotter \
  --log-dir perf_logs \
  --configs src/performance_test/performance_report/cfg/plotter/plot_one_experiment.yaml

# The generated plots will be saved in `./perf_logs`

# Generate the reports configured in the specified yaml file
ros2 run performance_report reporter \
  --log-dir perf_logs \
  --configs src/performance_test/performance_report/cfg/reporter/report_one_experiment.yaml

runner

The performance_report runner tool is a wrapper around performance_test perf_test. It executes one or more perf_test experiments defined in a yaml file:

---
experiments:
  -
    com_mean: ApexOSPollingSubscription  # or rclcpp-single-threaded-executor for ROS 2
    msg: Array1k
    rate: 20
  -
    com_mean: ApexOSPollingSubscription
    msg: Array4k
    rate: 20

To run all experiments in the config file, only a single command is required:

ros2 run performance_report runner \
  --configs input/path/to/config.yaml \
  --log-dir output/path/to/log/files \
  --test-name custom_name_for_this_set_of_tests

runner will invoke perf_test for each experiment, in sequence. The results for each experiment will be stored in a json log file in the directory output/path/to/log/files/custom_name_for_this_set_of_tests/.

For a list of all experiment configuration options, and their default values, see any of the example yaml configuration files in cfg/runner.

runner will by default skip any experiments that already have log files generated in the output directory. This can be overridden by adding -f or --force to the command.

Reducing duplication in configuration files

All of the experiment values can be a single value or an array:

```yaml

experiments: - com_mean: ApexOSPollingSubscription msg: - Array1k - Array4k - Array16k

File truncated at 100 lines see the full file

Changelog for package performance_report

X.Y.Z (YYYY/MM/DD)

2.3.0 (2024/09/24)

2.2.0 (2024/05/15)

Changed

• Plugins are now responsible for enabling shared memory transfer, so runner and commander will no longer set the related runtime flags (e.g. CYCLONEDDS_URI)

  Fixed

• For categorical plots, coerce the x_range to a string

2.1.0 (2024/04/17)

2.0.0 (2024/03/19)

Removed

• Removed the special handling for the BoundedSequenceFlat messages, because the messages are removed in performance_test

1.5.2 (2024/01/24)

Fixed

• Elegantly handle a failure to parse JSON log files

1.5.0 (2023/06/14)

Added

• The reporter box-and-whisker latency plots now support latency_mean_ms for the y-axis, in addition to the previously-supported latency_mean
• Added a new option prevent_cpu_idle (bool) for experiment configurations, which corresponds to the --prevent-cpu-idle switch in perf_test

  Changed

• Update the README to better explain the purpose and usage of runner, commander, plotter, and reporter

1.4.2 (2023/03/15)

1.4.1 (2023/02/23)

1.4.0 (2023/02/20)

Added

• Figures have a new x_range option: ru_maxrss_mb

  Changed

• BoundedSequenceFlatXYZ will be mapped to BoundedSequenceXYZ for categorical plots, so that both message types can be compared directly on a single plot

1.3.7 (2023/01/04)

Added

• The reporter templates can access os environment variables:
  • {{ env['SOME_ENVIRONMENT_VARIABLE'] }}
• For error detection, the exit code for performance_report reporter is the number of missing datasets

Fixed

• Add a workaround for bug in bokeh

1.3.6 (2023/01/03)

1.3.5 (2022/12/05)

1.3.4 (2022/11/28)

1.3.3 (2022/11/28)

Fixed

• Do not try to create a box-and-whisker for a file that contains no measurements

1.3.2 (2022/11/21)

1.3.1 (2022/11/21)

1.3.0 (2022/08/25)

Added

• The reporter configuration supports box-and-whisker latency plots:
  • set the x_range to Experiment
  • set the y_range to latency_mean
  • set datasets to one or more datasets, each containing a single experiment
  • an example can be found in cfg/reporter/report_many_experiments.yaml

    Changed

• Expanded the transport setting into the following two settings:
  • process_configuration:
    • INTRA_PROCESS
    • INTER_PROCESS
  • sample_transport:
    • BY_COPY
    • SHARED_MEMORY
    • LOANED_SAMPLES

1.2.1 (2022/06/30)

1.2.0 (2022/06/28)

Changed

• In the reporter configuration, the template_name value may be an array

File truncated at 100 lines see the full file