autoware_tensorrt_vad

Overview

The autoware_tensorrt_vad is a ROS 2 component that implements end-to-end autonomous driving using the TensorRT-optimized Vectorized Autonomous Driving (VAD) model. It leverages the VAD model (Jiang et al., 2023), optimized for deployment using NVIDIA’s DL4AGX TensorRT framework.

This module replaces traditional localization, perception, and planning modules with a single neural network, trained on the Bench2Drive benchmark (Jia et al., 2024) using CARLA simulation data. It integrates seamlessly with Autoware and is designed to work within the Autoware framework.

Features

• Monolithic End-to-End Architecture: Single neural network directly maps camera inputs to trajectories, replacing the entire traditional perception-planning pipeline with one unified model - no separate detection, tracking, prediction, or planning modules
• Multi-Camera Perception: Processes 6 surround-view cameras simultaneously for 360° awareness
• Vectorized Scene Representation: Efficient scene encoding using vector maps for reduced computational overhead
• Real-time TensorRT Inference: Optimized for embedded deployment with ~20ms inference time
• Integrated Perception Outputs: Generates both object predictions (with future trajectories) and map elements as auxiliary outputs
• Temporal Modeling: Leverages historical features for improved temporal consistency and prediction accuracy

Visualization

Lane Following Demo

Turn Right Demo

Parameters

Parameters can be set via configuration files:

• Deployment configuration (node and interface parameters): config/vad_carla_tiny.param.yaml
• Model architecture parameters: vad-carla-tiny.param.json (downloaded with model to ~/autoware_data/ml_models/vad/v0.1/)

Inputs

*Image transport supports both raw and compressed formats. Configure per-camera via use_raw parameter (default: compressed).

Outputs

Building

Build the package with colcon:

colcon build --symlink-install --cmake-args -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DCMAKE_BUILD_TYPE=Release --packages-up-to autoware_tensorrt_vad

Testing

Unit Tests

Unit tests are provided and can be run with:

colcon test --packages-select autoware_tensorrt_vad
colcon test-result --all

For verbose output:

colcon test --packages-select autoware_tensorrt_vad --event-handlers console_cohesion+

CARLA Simulator Testing

First, setup CARLA following the autoware_carla_interface instructions.

Then launch the E2E VAD system:

```bash

File truncated at 100 lines see the full file

Changelog for package autoware_tensorrt_vad

0.53.0 (2026-09-29)

• Merge remote-tracking branch 'origin/main' into prepare-0.53.0-changelog

• fix: declare the dependencies the simulator, e2e and map packages use (#13209)

  * fix: declare the dependencies the simulator, e2e and map packages use Three packages use headers or symbols of packages that they never declare. Add the 12 missing <depend> entries.

  * fix: declare tf2 in tier4_dummy_object_rviz_plugin The plugin uses tf2::Transform and tf2::Quaternion directly, but only tf2_geometry_msgs and tf2_ros were declared. ---------

• docs: replace retired model hosting URLs with Hugging Face links (#13204)

  * docs: replace retired model hosting URLs with Hugging Face links The model artifacts moved from awf.ml.dev.web.auto and the autoware-files S3 bucket to Hugging Face repositories under the AutowareFoundation org (autowarefoundation/autoware#7223). The READMEs still pointed manual downloads at the old hosts, and the yabloc README still gave wget instructions for the retired archive. The two dataset links on the S3 bucket stay: the bucket keeps serving datasets, maps and rosbags. Only the model objects are retired. The centerpoint v0 and v1 files were never migrated and stop being distributed, so their changelog rows lose the download links.

  * docs: name the ML package configs in the model download notes The launch files read transfusion_ml_package.param.yaml and ml_package_camera_streampetr.param.yaml from the model directory. The download notes did not name them, so a manual download missed two required files. ---------

• refactor(e2e): move node design files into each package (#13095) Co-authored-by: Claude Opus 5 (1M context) <<noreply@anthropic.com>>

• Contributors: Mete Fatih Cırıt, Ryohsuke Mitsudome, Taekjin LEE

0.52.0 (2026-06-30)

• Merge remote-tracking branch 'origin/main' into tmp/bot/bump_version_base

• fix(clang-tidy): re-enable bugprone-multi-level-implicit-pointer-conversion (#12578) fix(clang-tidy): re-enable multi-level pointer conversion check

• fix(autoware_tensorrt_vad): bind prev_bev via setTensorAddress (#12608)

  • fix(autoware_tensorrt_vad): bind prev_bev via setTensorAddress

  * fix(autoware_tensorrt_vad): abort frame on load_inputs failure to keep ping-pong in sync ---------

• feat: remove deprecated planning interface (#12636) remove deprecated planning interface

• docs(autoware_tensorrt_vad): fix link in docs (#12624)

• Contributors: Max-Bin, Shin, Vishal Chauhan, Yukihiro Saito, github-actions

0.51.0 (2026-05-01)

• Merge remote-tracking branch 'origin/main' into tmp/bot/bump_version_base

• feat: default artifact paths to ~/autoware_data/ml_models (#12523) feat(launches,configs): default artifact paths to ~/autoware_data/ml_models Roll every per-package [data_path]{.title-ref} / [model_path]{.title-ref} launch-arg default from [$(env HOME)/autoware_data[/...]]{.title-ref} to [$(env HOME)/autoware_data/ml_models[/...]]{.title-ref} so standalone universe launches resolve artifacts under the new [~/autoware_data/ml_models/]{.title-ref} layout (autowarefoundation/autoware#7068). When invoked through autoware_launch the parent overrides cascade and already pin the new root (autowarefoundation/autoware_launch#1835); this commit closes the gap for users who launch a perception / localization / sensing / planning component directly with [ros2 launch <pkg>]{.title-ref}. 22 launch files updated (one-line default change each):

  • e2e/autoware_tensorrt_vad/launch/vad_carla_tiny.launch.xml

File truncated at 100 lines see the full file