autoware_tensorrt_common

This package provides a high-level API to work with TensorRT. This library simplifies the process of loading, building, and executing TensorRT inference engines using ONNX models. It also includes utilities for profiling and managing TensorRT execution contexts, making it easier to integrate TensorRT-based packages in Autoware.

Usage

Here is an example usage of the library. For the full API documentation, please refer to the doxygen documentation (see header file).

#include <autoware/tensorrt_common/tensorrt_common.hpp>

#include <memory>
#include <utility>
#include <vector>

using autoware::tensorrt_common::TrtCommon;
using autoware::tensorrt_common::TrtCommonConfig;
using autoware::tensorrt_common::TensorInfo;
using autoware::tensorrt_common::NetworkIO;
using autoware::tensorrt_common::ProfileDims;

std::unique_ptr<TrtCommon> trt_common_;

Create a tensorrt common instance and setup engine

• With minimal configuration.

trt_common_ = std::make_unique<TrtCommon>(TrtCommonConfig("/path/to/onnx/model.onnx"));
trt_common_->setup();

• With full configuration.

trt_common_ = std::make_unique<TrtCommon>(TrtCommonConfig("/path/to/onnx/model.onnx", "fp16", "/path/to/engine/model.engine", (1ULL << 30U), -1, false));

std::vector<NetworkIO> network_io{
  NetworkIO("sample_input", {3, {-1, 64, 512}}), NetworkIO("sample_output", {1, {50}})};
std::vector<ProfileDims> profile_dims{
  ProfileDims("sample_input", {3, {1, 64, 512}}, {3, {3, 64, 512}}, {3, {9, 64, 512}})};

auto network_io_ptr = std::make_unique<std::vector<NetworkIO>>(network_io);
auto profile_dims_ptr = std::make_unique<std::vector<ProfileDims>>(profile_dims);

trt_common_->setup(std::move(profile_dims_ptr), std::move(network_io_ptr));

By defining network IO names and dimensions, an extra shapes validation will be performed after building / loading engine. This is useful to ensure the model is compatible with current code for preprocessing as it might consists of operations dependent on tensor shapes.

Profile dimension is used to specify the min, opt, and max dimensions for dynamic shapes.

Network IO or / and profile dimensions can be omitted if not needed.

Setting input and output tensors

bool success = true;
success &= trt_common_->setTensor("sample_input", sample_input_d_.get(), nvinfer1::Dims{3, {var_size, 64, 512}});
success &= trt_common_->setTensor("sample_output", sample_output_d_.get());
return success;

Execute inference

auto success = trt_common_->enqueueV3(stream_);
return success;

Changelog for package autoware_tensorrt_common

0.53.0 (2026-09-29)

• Merge remote-tracking branch 'origin/main' into prepare-0.53.0-changelog

• feat(autoware_tensorrt_common): let a caller offer IO the model may omit (#13347)

  * feat(autoware_tensorrt_common): let a caller offer IO the model may omit A caller declares the IO its node can serve, and setup() compares that list against the model. An architecture variant that gates blocks off declares less than the full contract, and the ONNX exporter drops inputs the traced graph never consumes, so such a model arrives narrower than the offered list and validateNetworkIO rejects it on the count. Every node facing that has to enumerate the model itself and pre-filter its own lists before calling setup(). NetworkIO and ProfileDims entries can now be marked optional, and setup() drops the optional ones the model does not declare before profiling or comparing them. An all-required list behaves exactly as it does today. Reconciliation then requires the offer and the model to account for each other in both directions, refusing a mismatch before an engine is built: Model does not declare the required IO tensor(s): [...] Model declares IO tensor(s) that were not offered: [...] The second direction takes over from validateNetworkIO's cardinality check for caller error. That check compares counts rather than sets, so one tensor the model declares and one surplus offered entry cancelled out and setup() could succeed leaving a tensor unbound; and because it runs after the engine is loaded, a mismatch discarded a good cached engine and burned a full rebuild before failing. The count check stays, now guarding only against the engine's IO set diverging from the parsed network's, which reconciliation runs too early to see. On success setup() says which of the two happened, once: Model declares all 27 offered IO tensors Model declares 20 of 27 offered IO tensors; skipping the optional tensors it does not declare: [serialized_code, ...] so a narrower model is visible at load instead of being inferred from a shape error later. Name resolution for index-addressed entries moves ahead of the profile loop and now covers NetworkIO too, since the name is what the comparison and the log report; an index that cannot be named is an error rather than a crash assigning nullptr to std::string. Co-Authored-By: Claude Opus 5 (1M context) <<noreply@anthropic.com>>

  * test(autoware_tensorrt_common): cover IO reconciliation against a dummy network Adds the package's first unit tests. A 256-byte ONNX fixture declares three dynamically shaped inputs feeding one output, which is enough to exercise every branch of the offered-IO reconciliation: an artifact declaring everything, an optional tensor the artifact omits (dropped, and both by-name setters no-op for it while a never-offered name still fails), the same when only the profile list offers it, a required tensor the artifact omits (refused before an engine is built, asserted by the absent engine file), the same via the profile list alone, an out-of-range tensor index, a caller offering fewer tensors than the model declares, equal counts covering different sets, and an offer of nothing but optional tensors the artifact omits. The last two would pass a size comparison alone. The tensor-metadata cases need no device and always run. The rest build an engine, so they use autoware_cuda_utils' SKIP_TEST_IF_CUDA_UNAVAILABLE() to skip where there is no GPU, as the CUDA CI job has none. Co-Authored-By: Claude Opus 5 (1M context) <<noreply@anthropic.com>> ---------Co-authored-by: Claude Opus 5 (1M context) <<noreply@anthropic.com>>

• feat(autoware_tensorrt_common): add getIOTensorNames (#13346) Callers that need to know what a model declares had to enumerate getNbIOTensors/getIOTensorName themselves, or ask getTensorIOMode one name at a time. The latter is a trap: TensorRT logs an error for every name the engine does not have, so probing a contract against a narrower model fills the log with errors that report nothing wrong. getIOTensorNames enumerates once and returns the set, and its documentation says why it is the right way to ask. Like the accessors it wraps, it reads the parsed network before setup() and the engine afterwards. Co-authored-by: Claude Opus 5 (1M context) <<noreply@anthropic.com>>

• fix(autoware_tensorrt_common): stop warning on the documented network fallback (#13288) getIOTensorName, getNbIOTensors and getTensorShape(index) fall back to the parsed network when the engine is not built yet. That is the documented contract - the header already says "with fallback from TensorRT network" - and it is how a caller inspects the IO a model declares before setup() builds or loads the engine. Logging kWARNING on every such call turns intended use into noise: enumerating a 27-input encoder emits 27 warnings before the engine exists, which buries the warnings that matter. The log is dropped; a missing network stays an error. The three fallbacks are also flattened to early returns - network missing is an error, an engine short-circuits, otherwise read the network - instead of nesting the network path inside [if (!engine_)]{.title-ref}. network_ is created by initialize() from the constructor and never reset, so testing it first is safe. Every member that reads only state established by the constructor now says so: "Valid immediately after construction. Valid to call before [setup()]{.title-ref}." Co-authored-by: Claude Opus 5 (1M context)

File truncated at 100 lines see the full file