ZED + Isaac ROS Object Detection Example

This example detects objects in the images of a ZED camera with the Isaac ROS 5 object detection packages, accelerated with TensorRT. Two models are supported:

• RT-DETR (default): the NVIDIA SyntheticaDETR sdetr_grasp model, trained on synthetic data to detect objects that a robot arm can grasp.
• YOLOv8: an Ultralytics YOLOv8 model, by default trained on the 80 classes of the COCO dataset (people, vehicles, animals, everyday objects).

The example is designed for the monocular ZED X One cameras, which do not provide depth and so cannot use the Object Detection module of the ZED SDK. It works with all the ZED cameras: with stereo cameras, the objects are detected in the left image.

The images stay in GPU memory from the ZED node to the TensorRT inference, thanks to the CUDA zero-copy transport of ROS 2 Lyrical (rosidl::Buffer with the cuda buffer backend).

ZED node ─► Resize ─► Pad ─► BGR to RGB ─► Image to tensor ─► Interleaved to planar ─► Reshape
 (bgr8)    (keep aspect   (640x640,                                                        │
            ratio)         bottom-right)            ┌──────────────────────────────────────┘
                                                    ▼
             [RT-DETR only] RT-DETR preprocessor ─► TensorRT ─► RT-DETR / YOLOv8 decoder
                                                                        │ detections (640x640 network image)
                                                                        ▼
                                                              Detection rescaler ─► detections, annotations
                                                                        │           (ZED image coordinates)
                                                                        ▼
                                                              NVIDIA visualizer ─► processed_image

The ZED node runs in the /zed_isaac/zed_container component container, and the Isaac ROS nodes in the /zed_isaac/object_detection/object_detection_container component container. The ZED images are shared between the two processes through CUDA IPC, without copies. The ZED node and the Isaac ROS nodes are not loaded in the same container: with cuda_buffer_backend 0.1.2, the same-process CUDA IPC path can deliver an image buffer already reused by the ZED node, which stops the container.

The Isaac ROS ResizeNode, which receives the ZED images, uses input_queue_size: 1 to always take the latest image: the cuda buffer backend can reuse the GPU memory of an image about 100 ms after it is published, so an image taken late from a longer queue could no longer be read. This queue depth does not replace the two containers: in our tests on a Jetson AGX Orin, with all the nodes in the ZED container and input_queue_size: 1, the container stopped in 9 of 16 runs (Docker and Bare Metal), while no error occurred in 18 runs with two containers.

Requirements

• A Jetson Thor or Orin device with JetPack 7.2, or an x86_64 machine with an NVIDIA GPU (Ampere or newer), Ubuntu 24.04 and CUDA 13.2
• NVIDIA Isaac ROS 5 with ROS 2 Lyrical, in Docker or Bare Metal mode, using the rmw_fastrtps_cpp (default) or the rmw_zenoh_cpp (v0.10.5 or newer) middleware
• ZED SDK v5.1 or newer
• ZED ROS 2 Wrapper v5.6.0 or newer, built with the cuda_buffer package
• The cuda_buffer_backend plugin, built from source

Follow these guides, in Docker or in Bare Metal mode:

1. Setting up Isaac ROS 5, to install Isaac ROS 5.
2. Use ZED with Isaac ROS, to install the ZED SDK and the ZED ROS 2 Wrapper.
3. Set up the CUDA zero-copy transport, to install the cuda_buffer package, build the cuda_buffer_backend plugin and rebuild the ZED ROS 2 Wrapper.

[!WARNING] Do not install the ros-lyrical-cuda-buffer-backend Debian package: it is built for the NVIDIA desktop drivers and the CUDA 12 runtime. On Jetson, it tries to install desktop driver packages that conflict with JetPack and leaves the package manager in a broken state.

Without the cuda_buffer_backend plugin, the example works, but the ZED node publishes the images in CPU memory and the Isaac ROS nodes upload them to the GPU. The ZED node reports the transport in use at startup, for example * Buffer transport: CUDA zero-copy.

Installation

The installation steps are the same in Docker and in Bare Metal mode, with the differences described in each step. Run all the commands in the Isaac ROS environment:

• Docker: run isaac-ros activate to open a shell in the Isaac ROS container. The workspace ${ISAAC_ROS_WS} of the host is mounted in the container, so the source code, the build artifacts and the models persist across container restarts.
• Bare Metal: run isaac-ros activate in your terminal.

1. Get the source code

Clone the ZED ROS 2 Examples repository in the workspace, next to the ZED ROS 2 Wrapper:

cd ${ISAAC_ROS_WS}/src
git clone https://github.com/stereolabs/zed-ros2-examples.git

2. Install the dependencies

Install the Isaac ROS packages used by the example, the tool to download the RT-DETR model, and the optional Foxglove bridge:

cd ${ISAAC_ROS_WS}
sudo apt-get update
rosdep update
rosdep install --from-paths src/zed-ros2-wrapper src/zed-ros2-examples/isaac_ros/isaac_ros_5/zed_isaac_ros_object_detection --ignore-src -r -y
sudo apt-get install -y ros-lyrical-isaac-ros-rtdetr-models-install ros-lyrical-foxglove-bridge

rosdep installs the Isaac ROS image processing, tensor processing, TensorRT, RT-DETR and YOLOv8 packages, vision_msgs and foxglove_msgs. The ZED ROS 2 Wrapper folder is listed so that rosdep uses the wrapper built in your workspace, instead of looking for its Debian packages.

Docker: the packages installed with apt-get in the container are lost when the container is removed, for example after rebuilding the image with isaac-ros activate --build-local, and you must repeat this step. To keep them, add them to a custom image layer, as you did for the ZED SDK:

1. On the host, create the ${ISAAC_ROS_WS}/docker/Dockerfile.zed_object_detection file:

   ARG BASE_IMAGE
   FROM ${BASE_IMAGE}

   RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
       apt-get update && apt-get install -y --no-install-recommends \
           ros-${ROS_DISTRO}-isaac-ros-image-proc \
           ros-${ROS_DISTRO}-isaac-ros-tensor-proc \
           ros-${ROS_DISTRO}-isaac-ros-tensor-rt \
           ros-${ROS_DISTRO}-isaac-ros-rtdetr \
           ros-${ROS_DISTRO}-isaac-ros-rtdetr-models-install \
           ros-${ROS_DISTRO}-isaac-ros-yolov8 \
           ros-${ROS_DISTRO}-vision-msgs \
           ros-${ROS_DISTRO}-foxglove-msgs \
           ros-${ROS_DISTRO}-foxglove-bridge \
       && rm -rf /var/lib/apt/lists/*
   

File truncated at 100 lines see the full file