Planning Evaluator

Purpose

This package provides nodes that generate various metrics to evaluate the quality of planning.

Metrics can be published in real time and saved to a JSON file when the node is shut down:

• metrics_for_publish:
  • Metrics listed in metrics_for_publish are calculated and published to the topic.
• metrics_for_output:
  • Metrics listed in metrics_for_output are saved to a JSON file when the node shuts down (if output_metrics is set to true).
  • These metrics include statistics derived from metrics_for_publish and additional information such as parameters and descriptions.

Metrics

All possible metrics are defined in the Metric enumeration located in:

• include/autoware/planning_evaluator/metrics/metric.hpp
• include/autoware/planning_evaluator/metrics/output_metric.hpp

These files also provide string conversions and human-readable descriptions for use in output files.

Metric Classifications

By Data Type

1. Statistics-based Metrics:
   • Calculated using autoware_utils::Accumulator, which tracks minimum, maximum, mean, and count values.
   • Sub-metrics: /mean, /min, /max, and /count.
2. Value-based Metrics:
   • Metrics with a single value.
   • Sub-metrics: /value.
   • Some metrics with older implementations use the statistics-based format of /mean, /min, /max, but all values are the same.
3. Count-based Metrics:
   • Count occurrences of specific events over a time period.
   • Sub-metrics: /count, or /count_in_duration.

By Purpose

1. Trajectory Metrics
2. Trajectory Deviation Metrics
3. Trajectory Stability Metrics
4. Trajectory Obstacle Metrics
5. Modified Goal Metrics
6. Planning Factor Metrics
7. Steering Metrics
8. Blinker Metrics
9. Other Information

Detailed Metrics

Trajectory Metrics

Evaluates the current trajectory T(0) itself. Metrics are calculated and published when a trajectory is received.

Implemented metrics

• curvature: Statistics of curvature at each trajectory point.
  • Sub-metrics to publish: /mean, /min, /max.
  • Sub-metrics to output: The same as above but take the published data as data point instead of each trajectory point.
• point_interval: Statistics of distances between consecutive trajectory points.
  • Sub-metrics to publish: /mean, /min, /max.
  • Sub-metrics to output: The same as above but take the published data as data point instead of each trajectory point.
• relative_angle: Statistics of angles between consecutive trajectory points.
  • Parameters: trajectory.min_point_dist_m (minimum distance between points).
  • Sub-metrics to publish: /mean, /min, /max.
  • Sub-metrics to output: The same as above but take the published data as data point instead of each trajectory point.

• resampled_relative_angle: Similar to relative_angle, but considers a point at a fixed distance (e.g., half the vehicle length) for angle calculation from the current point as the next point to calculate the relative angle, instead of using the immediately adjacent point.

• length: Total trajectory length.
  • Sub-metrics: Value-based metric, but using the statistics-based format of /mean, /min, /max with the same value.
  • Sub-metrics to output: /mean, /min, /max for the published data.
• duration: Expected driving time to travel the trajectory.
  • Sub-metrics: Value-based metric, but using the statistics-based format of /mean, /min, /max with the same value.
  • Sub-metrics to output: /mean, /min, /max for the published data.
• velocity: Statistics of velocity at each trajectory point.
  • Sub-metrics to publish: /mean, /min, /max.
  • Sub-metrics to output: The same as above but take the published data as data point instead of each trajectory point.
• acceleration: Statistics of acceleration at each trajectory point.
  • Sub-metrics to publish: /mean, /min, /max.
  • Sub-metrics to output: The same as above but take the published data as data point instead of each trajectory point.
• jerk: Statistics of jerk at each trajectory point.
  • Sub-metrics to publish: /mean, /min, /max.
  • Sub-metrics to output: The same as above but take the published data as data point instead of each trajectory point.

Trajectory Deviation Metrics

Evaluates the trajectory deviation by comparing the trajectory T(0) and the reference trajectory.

File truncated at 100 lines see the full file

Changelog for package autoware_planning_evaluator

0.47.0 (2025-08-11)

• feat: change planning output topic name to /planning/trajectory (#11135)

  • change planning output topic name to /planning/trajectory

  * style(pre-commit): autofix ---------Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

• fix(planning_evaluaotr): prevent abnormal value for ttc (#11138)

• style(pre-commit): autofix (#10982) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

• Contributors: Kosuke Takeuchi, Ryohsuke Mitsudome, Yukihiro Saito

0.46.0 (2025-06-20)

• Merge remote-tracking branch 'upstream/main' into tmp/TaikiYamada/bump_version_base

• feat!: remove obstacle_stop_planner and obstacle_cruise_planner (#10695)

  • feat: remove obstacle_stop_planner and obstacle_cruise_planner
  • update

  * fix

• fix(planning_evaluator): fix bug of abnormal_stop metric, and turn its threshold (#10628)

  • fix bug
  • change threeshold
  • update abnormal_deceleration_threshold_mps2
  • change back threshold
  • add takeuchi san as maintainer

  * rename func ---------

• Contributors: Kem (TiankuiXian), TaikiYamada4, Takayuki Murooka

0.45.0 (2025-05-22)

0.44.2 (2025-06-10)

0.44.1 (2025-05-01)

0.44.0 (2025-04-18)

• Merge remote-tracking branch 'origin/main' into humble

• chore(autoware_planning_evaluator): record goal_stop_deviation only when ego stop (#10429)

  • record modified_goal related output_metric only when the ego stop close to goal
  • change vel thr

  * pre-commit ---------

• feat(autoware_planning_evaluator): refactor planning_evaluator for new metrics (#10368)

  • tmp save.
  • tmp save.
  • WIP add accumulator-based metrics.
  • pre-commit
  • add unit test.
  • pre-commit
  • fix cppcheck
  • update readme.
  • pre-commit
  • polish readme.
  • pre-commit
  • change count to size_t
  • update config.
  • publish count.
  • fix stop decision bug
  • update parameters.

  * fix typo ---------Co-authored-by: t4-adc <<grp-rd-1-adc-admin@tier4.jp>>

• Contributors: Kem (TiankuiXian), Ryohsuke Mitsudome

0.43.0 (2025-03-21)

• Merge remote-tracking branch 'origin/main' into chore/bump-version-0.43

File truncated at 100 lines see the full file