autoware_traffic_light_multi_camera_fusion

Overview

This node fuses traffic light recognition results from multiple cameras to produce a single, reliable traffic light state. By integrating information from different viewpoints and ROIs, it ensures robust performance even in challenging scenarios, such as partial occlusions or recognition errors from an individual camera.

graph LR
    subgraph "Multi Camera Feeds"
        direction TB
        Cam1[" <br> <b>Camera 1</b> <br> State: GREEN <br> Confidence: 0.95"]
        Cam2[" <br> <b>Camera 2</b> <br> State: GREEN <br> Confidence: 0.94"]
        Cam3[" <br> <b>Camera 3</b> <br> State: RED <br> Confidence: 0.95"]
    end

    subgraph "Processing"
        direction TB
        Fusion["<b>Multi-Camera Fusion Node</b> <br><i>Fuses evidence using <br> Bayesian updating</i>"]
    end

    subgraph "Unified & Robust State"
        direction TB
        Result[" <br> <b>Final State: GREEN</b>"]
    end

    Cam1 --> Fusion
    Cam2 --> Fusion
    Cam3 --> Fusion
    Fusion --> Result

    style Fusion fill:#e0f7fa,stroke:#00796b,stroke-width:2px,color:#004d40
    style Result fill:#e8f5e9,stroke:#2e7d32,stroke-width:3px,color:#1b5e20

How It Works

The fusion algorithm operates in two main stages.

graph TD
    subgraph "Input: Multiple Camera Results"
        A["Camera 1<br>Recognition Result"]
        B["Camera 2<br>Recognition Result"]
        C["..."]
    end

    subgraph "Stage 1: Per-Camera Fusion"
        D{"Best ROIs Selection<br><br>For each ROI,<br>select the single most<br>reliable detection result."}
    end

    E["Best Detection per ROIs"]

    subgraph "Stage 2: Group Fusion"
        F{"Group Consensus<br><br>Fuse all 'best detections'<br>into a single state for<br>the entire traffic light group<br>using Bayesian updating."}
    end

    subgraph "Final Output"
        G["Final Group State<br>(e.g., GREEN)"]
    end

    A --> D
    B --> D
    C --> D
    D --> E
    E --> F
    F --> G

    style D fill:#e0f7fa,stroke:#00796b,stroke-width:2px,color:black
    style F fill:#e0f7fa,stroke:#00796b,stroke-width:2px,color:black
    style E fill:#fff,stroke:#333,stroke-width:2px,stroke-dasharray: 5 5,color:black
    style G fill:#e8f5e9,stroke:#2e7d32,stroke-width:3px,color:black

Stage 1: Best View Selection (Per-Camera Fusion)

First, for each individual ROIs, the node selects the single most reliable detection—the “best shot”—from all available camera views.

This selection is based on a strict priority queue:

• Latest Timestamp: Detections with the most recent timestamp are prioritized for the same sensor.
• Known State: Results with a known color (Red, Green, etc.) are prioritized over ‘Unknown’.
• Full Visibility: Detections from non-truncated ROIs (fully visible ROIs) are prioritized.
• Highest Confidence: The result with the highest detection confidence score is prioritized.

This process yields the single most plausible recognition for every ROIs.

Stage 2: Group Consensus (Bayesian Fusion)

Next, the “best shot” detections from Stage 1 are fused to determine a single, coherent state for the entire traffic light group. Instead of simple voting or averaging, this node employs a more principled method: Bayesian updating.

• Belief Score: Each color (Red, Green, Yellow) maintains a “belief score” represented in log-odds for numerical stability and ease of updating.
• Evidence Update: Each selected detection from Stage 1 is treated as a piece of “evidence.” Its confidence score is converted into a log-odds value representing the strength of that evidence.
• Score Accumulation: This evidence is added to the corresponding color’s belief score.
• Final Decision: After accumulating all evidence, the color with the highest final score is chosen as the definitive state for the group.

Cross Camera Validation

This node includes an option to compare and validate detected traffic light signals within the same regulatory element.

File truncated at 100 lines see the full file

Changelog for package autoware_traffic_light_multi_camera_fusion

0.53.0 (2026-09-29)

• Merge remote-tracking branch 'origin/main' into prepare-0.53.0-changelog

• build(traffic_light_perception): split core/node CMake targets and expose public core headers (#13325)

  * refactor(autoware_traffic_light_category_merger): expose core header, split core/node CMake targets Move traffic_light_category_merger.hpp into include/autoware/traffic_light_category_merger/, and split the CMake target into ${PROJECT_NAME} (core) and ${PROJECT_NAME}_node (Node adapter linking the core). The Node header stays in src/. No behavior change; colcon build + colcon test pass (7 tests, 0 failures).

  * refactor(autoware_traffic_light_selector): expose core header, split core/node CMake targets Move traffic_light_selector.hpp into include/autoware/traffic_light_selector/, keeping traffic_light_selector_utils.hpp private in src/ (not part of the public API). Split the CMake target into ${PROJECT_NAME} (core: selector + utils) and ${PROJECT_NAME}_node (Node adapter linking the core). No behavior change; colcon build + colcon test pass (7 tests, 0 failures).

  * refactor(autoware_traffic_light_map_based_detector): expose core header, split core/node CMake targets Move traffic_light_map_based_detector.hpp into include/autoware/traffic_light_map_based_detector/, keeping traffic_light_map_based_detector_process.hpp private in src/ (not part of the public API). Since the public header declares functions taking image_geometry::PinholeCameraModel, add that include directly to the header instead of pulling it in transitively via the private process header. Split the CMake target into ${PROJECT_NAME} (core: detector + process) and ${PROJECT_NAME}_node (Node adapter linking the core). No behavior change; colcon build + colcon test pass (7 tests, 0 failures).

  * refactor(autoware_tensorrt_yolox): build TrtYoloXDetector as part of the core library tensorrt_yolox_detector.cpp implements core detection logic and was being compiled into ${PROJECT_NAME}_node (the Node adapter target) instead of ${PROJECT_NAME} (the core library). Move it to the core target so the core library does not depend on the node target for its own logic.

  • refactor(autoware_traffic_light_classifier): expose TrafficLightClassifier as the sole public header

  - Move traffic_light_classifier.hpp to include/autoware/traffic_light_classifier/ and forward-declare ClassifierInterface instead of including classifier/classifier_interface.hpp, so the classifier/*.hpp backends and classifier_params.hpp / traffic_light_classifier_node.hpp stay private under src/.

  - Split the CMake library target into ${PROJECT_NAME} (ROS-free classification core) and ${PROJECT_NAME}_node (rclcpp::Node adapter layer), mirroring the core/node separation already present in src/.

  - single_image_debug_inference_node and the node-level integration test now link against both libraries instead of recompiling every source file.

  • refactor(autoware_traffic_light_multi_camera_fusion): move headers under include/, split core/node CMake targets
  • refactor(autoware_traffic_light_arbiter): split core/node CMake targets
  • refactor(autoware_image_transport_decompressor): move core logic header to include, node header to src

  * refactor(autoware_crosswalk_traffic_light_estimator): move core logic header to include, node header to src, split core/node CMake targets ---------

• feat(autoware_traffic_light_multi_camera_fusion): add map based signal filter (#13046)

  • add map based signal filter
  • fixed text
  • fix text
  • filter signals has unknown
  • fix documents
  • change to return by value
  • fix documents
  • move filter before multi_camera_fusion
  • add test for valid vs invalid input of same camera and id

  * add warning message when no map available ---------

• feat(traffic_light_multi_camera_fusion): apply [agnocast_wrapper::Node]{.title-ref} to [traffic_light_multi_camera_fusion]{.title-ref} (#12785)

File truncated at 100 lines see the full file