autoware_ptv3

Purpose

The autoware_ptv3 package is used for 3D lidar segmentation.

Inner-workings / Algorithms

This package implements a TensorRT powered inference node for Point Transformers V3 (PTv3) [1]. The sparse convolution backend corresponds to spconv. Autoware installs it automatically in its setup script. If needed, the user can also build it and install it following the following instructions.

The network input is a densified point cloud: the current lidar frame is concatenated with ego-motion-compensated past frames (densification_num_past_frames), and every point carries five features (x, y, z, intensity, time_lag), where time_lag is the point age in seconds relative to the current frame. The densified cloud is voxelized on the GPU into padded voxels (max_points_per_voxel slots holding the first points of each voxel in input order) together with the valid point count per voxel, and the encoder graph averages the points of every voxel itself, so training and inference share one voxel feature definition. Past sweeps provide geometric and temporal context to the network, while every segmentation output cloud describes only the current frame’s points.

Inputs / Outputs

Input

Output

Segmentation point cloud format

The ~/output/pointcloud/segmentation topic uses the naturally aligned PointXYZCPE point type. Its point_step is 24 bytes.

Bytes 13 through 15 are alignment padding. The output is intentionally not packed so that the floating-point fields remain naturally aligned on both the CPU and GPU.

Points with an invalid label (class_id = PointCloudClassification::INVALID, that is 255) have no class distribution: their probability is 0.0 and their entropy is NaN, which is the default value of autoware::point_types::PointXYZCPE::entropy and means “not available”. Consumers must therefore guard against NaN before using the entropy field.

Parameters

PTv3Node node

{{ json_to_markdown(“perception/autoware_ptv3/schema/ptv3.schema.json”) }}

PTv3Node model

{{ json_to_markdown(“perception/autoware_ptv3/schema/ml_package_ptv3.schema.json”) }}

filter.* and class_mapping.* parameters are configured in config/ptv3.param.yaml, while class metadata and the visualization palette are configured in config/ml_package_ptv3_seg3d_head.param.yaml.

segmentation3d.class_mapping maps each segmentation class name to the autoware::point_types::PointCloudClassification value published in the segmentation point cloud’s class_id field. Its keys must match segmentation3d.class_names exactly, and every entry of class_names must be present — the node fails to start otherwise, naming the missing class. This makes the consolidation of model classes (for example traffic_cone, debris and vertical_thin into HAZARD) a configuration choice rather than a compile-time constant.

The build_only option

The autoware_ptv3 node has a build_only option to build the TensorRT engine file from the specified ONNX file, after which the program exits.

ros2 launch autoware_ptv3 ptv3.launch.xml build_only:=true

The log_level option

The default logging severity level for autoware_ptv3 is info. For debugging purposes, the developer may decrease severity level using log_level parameter:

File truncated at 100 lines see the full file

Changelog for package autoware_ptv3

0.53.0 (2026-09-29)

• Merge remote-tracking branch 'origin/main' into prepare-0.53.0-changelog

• feat(autoware_ptv3): support multisweep densification with time-lag features (#13300)

  • feat(autoware_ptv3): support multisweep densification with time-lag features
  • feat(autoware_ptv3): support none source reconstruction mode
  • fix(autoware_ptv3): publish every output cloud in input order
  • perf(autoware_ptv3): drop the per-frame stream sync from sweep aggregation
  • fix(autoware_ptv3): mark the current sweep explicitly instead of inferring it from the lag
  • fix(autoware_ptv3): reject unusable frames when they are cached instead of throwing mid-inference
  • refactor(autoware_ptv3): create the TF buffer and listener only when densification is enabled
  • fix(autoware_ptv3): advertise the filtered cloud only when filter classes are configured
  • refactor(autoware_ptv3): drop the duplicated frame count, config copies and cache walks
  • fix(autoware_ptv3): pass identity for the current frame instead of a pose round trip

  * chore: add voxelizer to the cspell dictionary ---------

• feat(autoware_ptv3)!: per-stage voxel-count maximums and input truncation to the stage bounds (#13390) TensorRT sizes an engine's activation memory for the profile maximum, and until now every pooled encoder level was bounded by the input maximum (voxels_num[2]) or, at best, its grid capacity, which only binds at the coarsest levels. Since the channel width doubles per level, the deepest levels dominated the encoder's and segmentation head's memory even though stride-2 pooling merges at least half of a lidar sweep's voxels at every level in practice. BREAKING CHANGE: the new required parameter encoder.pooled_voxels_num_max gives one voxel-count maximum per pooled encoder stage (one entry per pooling stride). encoder.voxels_num keeps its [min, opt, max] meaning for the input level. Existing parameter files must add the parameter; existing engines are rebuilt because the profiles change. PTv3Config::stage_voxel_capacity takes the smaller of a stage's configured maximum and its grid bound, and stage_profile_counts (moved from PTv3TRT) derives every stage's [min, opt, max] profile from it, so the encoder and head profiles and the per-stage feature buffers all follow the parameter. Validation requires one entry per pooling stage, each positive and at most the previous stage's maximum. The shipped defaults are measured per-stage peaks with a margin. A frame whose level would exceed its maximum is truncated instead of failing. The levels are in order-0 serialization order and each parent's children are contiguous, so keeping the first [max]{.title-ref} voxels of a pooled level keeps a prefix of every finer level down to the input. generateSerializedPoolingMetadata builds the levels, finds the longest input prefix whose levels all fit (a single-thread device walk over the indptr arrays) and rebuilds from that prefix when it is shorter, reading the codes from the caller's buffer at their original stride. The walk's result returns with the level counts in the one synchronization the caller already needed, so the common path adds no sync and the truncation path adds one. PTv3TRT feeds the encoder the truncated count and warns. This drops the voxels with the largest codes (a region at the end of the curve); choosing what to prune is left for later. Co-authored-by: Claude Fable 5.1 <<noreply@anthropic.com>>

• feat(autoware_ptv3): declare and bind the encoder IO the artifact exposes (#13290)

  * feat(autoware_ptv3): offer the encoder IO the artifact may omit LitePT gates convolution and attention off per encoder level, so nothing consumes the base serialization order and levels without convolution read no head_indices. The ONNX exporter drops inputs the traced graph never consumes, so a LitePT encoder declares 20 of the 27 inputs this node hardcoded and TrtCommon rejected it on an IO count mismatch: "Failed to setup encoder TRT engine." Everything except grid_coord, feat and the point features is now offered as optional, and TrtCommon drops the optional entries the artifact omits. Address binding and per-frame shape setting follow the surviving set. A PTv3 artifact declares the full contract and is unaffected. Two checks keep this a subset rather than a free-for-all:

  - a tensor the artifact declares that this node cannot supply is rejected by name, instead of reaching enqueue with no address ever bound;

  - grid_coord, feat and every point_feat_i must be present, since no encoder variant can drop them. Pooling metadata is still generated for every stage, so this changes what is offered and bound, not what is computed. Co-Authored-By: Claude Opus 5 (1M context) <<noreply@anthropic.com>>

File truncated at 100 lines see the full file