Detect objects with depth estimation

Application for detecting and estimating the depth of detected objects using stereo cameras. The application is built on ROS2 framework that can be incorporated into robotics systems as their perception capabilities. In the application, YOLO-v4 Tiny model is used as the object detection and classification model. It has been tested to be able to run on the Jetson Nano at ~5 FPS. Below lists some of the application features:

• Continuous publishing of live stereo-camera video feeds into ROS2 architecture (DDS-RTPS)
• Stereo image processor to obtain disparity maps and depth maps, for depth estimation
• Image segmentation, image processing techniques and blob detector on disparity frames
• Obstacle detection capabilities using Scaled-YOLOv4 (tiny) model
• Implementation of depth estimation capabilities with obstacles detected

Author and owner: Niven Sie (sieniven@gmail.com)

Table of Contents

1. Getting started
2. Setup
3. Installing
   1. Installing on computer
   2. Installing using Docker environment
4. Run
   1. Run software locally
   2. Run software on Docker
5. Acknowledgments

1. Getting started

The software has been deployed on a Jetson Nano, and can be deployed in a Docker container or locally. For the full setup, installation and deployment guide, refer to the steps below.

2. Setup

Pull this repository

git clone https://github.com/sieniven/detect-objects-with-depth-estimation-ros.git

3. Installing

3.1 Installing on computer

The following step-by-step instructions will guide you on the installation process on your computer.

1. Install ROS2 Eloquent. Refer to: https://index.ros.org/doc/ros2/Installation/Eloquent/Linux-Install-Debians/
   • NOTE: please skip the environment setup options in the installation process.
2. Install ROS2 launch packages

	sudo apt-get update && sudo apt-get install -y ros-eloquent-launch*
	

1. Install required dependencies. Check Tensorflow website for more information on this.
   • CUDA: 11.0
   • CUDnn: 8.0
2. Install Tensorflow-GPU (2.4.0).
   • NOTE: Ensure that all dependencies are installed as well.
3. Install Boost libraries

	sudo apt-get update && sudo apt-get install libboost-all-dev
	

1. Install required dependencies from requirements.txt file

	cd detect-obstacles-ros/detector/bin
	python3 -m pip install -r requirements.txt
	

1. Install OpenCV dependencies

	sudo sudo apt-get purge *libopencv*
	sudo apt-get install build-essential cmake git unzip pkg-config
	sudo apt-get install libjpeg-dev libpng-dev libtiff-dev
	sudo apt-get install libavcodec-dev libavformat-dev libswscale-dev
	sudo apt-get install libgtk2.0-dev libcanberra-gtk*
	sudo apt-get install python3-dev python3-numpy python3-pip
	sudo apt-get install libxvidcore-dev libx264-dev libgtk-3-dev
	sudo apt-get install libtbb2 libtbb-dev libdc1394-22-dev
	sudo apt-get install libv4l-dev v4l-utils
	sudo apt-get install libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev
	sudo apt-get install libavresample-dev libvorbis-dev libxine2-dev
	sudo apt-get install libfaac-dev libmp3lame-dev libtheora-dev
	sudo apt-get install libopencore-amrnb-dev libopencore-amrwb-dev
	sudo apt-get install libopenblas-dev libatlas-base-dev libblas-dev
	sudo apt-get install liblapack-dev libeigen3-dev gfortran
	sudo apt-get install libhdf5-dev protobuf-compiler
	sudo apt-get install libprotobuf-dev libgoogle-glog-dev libgflags-dev
	

1. Install OpenCV and OpenCV_Contrib libraries
   • Install version 4.5.2, refer to installation guide here: https://docs.opencv.org/master/d7/d9f/tutorial_linux_install.html.
   • Install OpenCV and OpenCV_Contrib

``` bash wget https://github.com/opencv/opencv/archive/4.5.2.zip -O opencv-4.5.2.zip wget https://github.com/opencv/opencv_contrib/archive/4.5.2.zip -O opencv-contrib-4.5.2.zip unzip opencv-4.5.2.zip unzip opencv-contrib-4.5.2.zip mv opencv-4.5.2 opencv

File truncated at 100 lines see the full file