autoware_camera_streampetr

Purpose

The autoware_camera_streampetr package is used for 3D object detection based on images only.

Inner-workings / Algorithms

This package implements a TensorRT powered inference node for StreamPETR [1]. This is the first camera-only 3D object detection node in autoware.

This node has been optimized for multi-camera systems where the camera topics are published in a sequential manner, not all at once. The node takes advantage of this by preprocessing (resize, crop, normalize) the images and storing them appropriately on GPU, so that delay due to preprocessing can be minimized.

Topic for image_i arrived                                     -------------------------
  |                                                                                   |
  |                                                                                   |
  |                                                                                   |
  v                                                                                   |
Is image distorted?                                                                   |
  |              \                                                                    |
  |               \                                                                   |
Yes               No                                                                  |Image Updates
  |                |                                                                  |done in parallel, if multitheading is on
  v                |                                                                  |otherwise done sequentially in FIFO order
Undistort          |                                                                  |
  |                |                                                                  |
  v                v                                                                  |
Load image into GPU memory                                                            |
  |                                                                                   |
  v                                                                                   |
Preprocess image (scale & crop ROI & normalize)                                       |
  |                                                                                   |
  v                                                                                   |
Store in GPU memory binding location for model input                                  |
  |                                                          -------------------------|
  v                                                                                   |
Is image the `anchor_image`?                                                          |
  |                \                                                                  |
  |                 \                                                                 |
No                  Yes                                                               |
  |                  |                                                                |
  v                  v                                                                | If multithreading is on
(Wait)     Are all images synced within `max_time_difference`?                        | image Updates are temporarily frozen
                      |                           \                                   | until this part completes.
                      |                            \                                  |
                    Yes                             No                                |
                      |                             |                                 |
                      v                             v                                 |
         Perform model forward pass            (Sync failed! Skip prediction)         |
                      |                                                               |
                      v                                                               |
         Postprocess (NMS + ROS2 format)                                              |
                      |                                                               |
                      v                                                               |
             Publish predictions                             -------------------------|

Inputs / Outputs

Input

Output

All latency/* topics are published only when debug_mode is enabled.

Diagnostics

Published on /diagnostics regardless of debug_mode. Both statuses are tasks of one timer-driven updater (period diagnostics.validation_callback_interval_ms, hardware_id = the node name), so they keep reporting when the cameras or the inference stop — precisely the conditions they exist for — and always arrive in the same /diagnostics message. One status per failure mode, so a diagnostic graph can route input trouble and processing-time trouble to different paths.

camera_status reports three key/values per camera — the resolved input cameraN/topic (the

File truncated at 100 lines see the full file

Changelog for package autoware_camera_streampetr

0.53.0 (2026-09-29)

• Merge remote-tracking branch 'origin/main' into prepare-0.53.0-changelog

• fix(design): align the perception node designs with the packages they describe (#13339) The lanelet filter components register under the lanelet_filter:: namespace, and the apollo instance segmentation and transfusion nodes are built with the autoware_ prefix on the executable name. BEVFusion, StreamPetr and LidarFRNet name param file defaults that resolve to nothing: the first two are missing the config/ prefix, and the three files LidarFRNet names are called frnet.param.yaml, ml_package_frnet_ot128.param.yaml and diagnostics_frnet.param.yaml. ElevationMapLoader pins its map_hash input with global:, which keeps the port out of the design graph: link_manager skips any connection whose target is a global input port and the exporter emits no remap. remap_target: keeps the same fixed topic name while letting the port take part in the graph.

• refactor(camera-streampetr): replace NMS with perception_utils::IouBevNms (#13179) refactor: replace NMS with perception_utils::IouBevNms

• fix(autoware_camera_streampetr): report the preprocess time with sub-millisecond precision (#13212)

  * fix(autoware_camera_streampetr): report the preprocess time with sub-millisecond precision The per-image preprocessing takes about 2 ms, but the data store truncated it through duration_cast<milliseconds> to whole milliseconds, quantizing away most of the signal before it reached the latency/preprocess debug topic and the diagnostics. Measure it as a double in milliseconds instead.

  • chore: clean up comment

  * feat(autoware_camera_streampetr): report per-camera input status diagnostics The node consumes five cameras but nothing outside its own log reported whether they are actually usable: a camera that never arrives, publishes an encoding the preprocessing rejects, or silently stops publishing mid-run leaves /diagnostics empty while the node stops inferring. The motivating incident was a subscription that failed to be created at all: every camera published normally and the sensing-side diagnostics stayed green, but the node received nothing -- only a consumer-side status can report that class of failure. Publish a timer-driven camera_status (period diagnostics.validation_callback_interval_ms) so the reporting survives the cameras dying. Each camera collapses into one state, most-specific-cause first: rejected (ERROR, dropped by input validation), waiting_camera_info / waiting_image (WARN, normal during start-up), stale (ERROR, newest frame older than diagnostics.max_image_age_ms) and active. Per camera it also carries the resolved input topic (the model index and the physical camera differ per deployment) and the image age; summary keys (num_waiting/num_stale/num_rejected, stalest camera, inter-camera stamp spread) make the worst offender readable without per-camera digging. Ages are clamped at zero so a rosbag loop does not read as a stall, and printed as fixed-point strings so epoch-sized values do not degrade to scientific notation.

  * feat(autoware_camera_streampetr): add a processing-time watchdog with per-stage breakdown Add a processing_time_status task to the diagnostics updater, mirroring the lidar detectors: WARN once a cycle exceeds diagnostics.max_allowed_processing_time_ms, escalating to ERROR when it stays over budget for longer than diagnostics.max_acceptable_consecutive_delay_ms. Timer driven, so a node that has stopped inferring altogether keeps reporting -- exactly when the per-cycle path stops running -- and 'waiting' is reported until the first inference completes. The stopwatch becomes a plain always-on member: the watchdog needs the per-cycle total even with the debug topics disabled, which makes every null check on it dead code. Beyond the thresholds shared with the other detectors, the status carries:

  - preprocess/inference/postprocess_time_ms: the same cycle's per-stage breakdown, latched together with the total, so an over-budget cycle can be localized without enabling debug_mode. Not-yet-inferred cycles report 'n/a' rather than 0.0, which would read as a 0 ms cycle.

  - last_frame_timestamp / last_published_timestamp: the newest published detection under both clocks (sensing instant vs node clock), plus their difference output_latency_ms (end-to-end latency including camera transport and decode queueing, which processing_time_ms cannot see) and time_since_last_publish_ms (how long since the node last produced objects). Observational only; the level is decided by the thresholds. Timestamps are fixed-point strings: the epoch-sized values degrade to scientific notation through the double overload.

  * refactor(autoware_camera_streampetr): split postprocessing out of the inference call inference_detector() ran the model and then decoded the detections inside the same call, so the postprocess stage was timed inside the inference window: the reported stage

File truncated at 100 lines see the full file