Simultaneous Positioning, Observing, Tracking, Identifying Targets in 3D (SPOT-IT 3D)

1. Introduction

SPOT-IT 3D (Simultaneous Positioning, Observing, Tracking, Identifying Targets in 3D) is a project by Temasek Laboratories @ National University of Singapore. This project aims to develop an application that utilizes a multi-camera surveillance system for real-time multiple target tracking capabilities, focusing on small and fast moving targets. This software capability is highly applicable for monitoring specific areas, and some use cases include monitoring airspaces, traffic junctions, etc.

2. Table of Contents

• 3. Publications
• 4. Aim
• 5. Capabilities
• 6. Installation Guide
  • 6.1 Install and build locally
  • 6.2 Install and build using Docker
• 7. Configurations
• 8. Run
  • 8.1 Run software locally
  • 8.2 Run software on Docker
  • 1. Ensure that camera ports are configured correctly
• 9. Acknowledgements

3. Publications

1. Paper on trajectory-based target matching and re-identification between cameras:
   • Niven Sie Jun Liang and Sutthiphong Srigrarom. “Multi-camera multi-target tracking systems with trajectory-based target matching and re-identification.” In 2021 IEEE International Conference on Unmanned Aerial Systems (ICUAS), IEEE, Athens, Greece, 2021.
   • Link to paper: (To be added into IEEE Xplore soon)
2. Paper on field test validations for using trajectory-based tracking with a multiple camera system for target tracking and 3-dimensional localization:
   • Niven Sie Jun Liang, Sutthiphong Srigrarom and Sunan Huang. “Field test validations of vision-based multi-camera multi-drone tracking and 3D localizing, using concurrent camera pose estimation.” In 2021 IEEE 6th International Conference on Control and Robotics Engineering (ICCRE), IEEE, Beijing, China, 2021.
   • Link to paper: https://ieeexplore.ieee.org/abstract/document/9358454
3. Paper on state estimation filters and proposed use of implementing multiple state estimation filters in parrallel (Integrated Multiple Model):
   • Sutthiphong Srigrarom, Niven Sie Jun Liang, Jiahe Yi, Kim Hoe Chew, Floorian Holzapfel, Henrik Hesse, Teng Hooi Chan and Jalvin Jiaxiang Chen. “Vision-based drones tracking using correlation filters and Linear Integrated Multiple Model.” In 2021 IEEE International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology (ECTI-CON), IEEE, Chiang Mai, Thailand, 2021.
   • Link to paper: (To be added into IEEE Xplore soon)
4. Paper on integrated kinematic-based detection tracking estimation system for dynamic localization of small aerial vehicles:
   • Sutthiphong Srigrarom, Shawndy Michael Lee, Mengda Lee, Foong Shaohui and Photchara Ratsamee. “An integrated vision-based detection-tracking-estimation system for dynamic localization of small aerial vehicles.” In 2020 IEEE 5th International Conference on Control and Robotics Engineering (ICCRE), IEEE, Osaka, Japan, 2020.
   • Link to paper: https://ieeexplore.ieee.org/abstract/document/9096259
5. Paper on binocular and stereo cameras for multiple drone detection and 3-dimensional localization:
   • Yi, Jiahe, and Sutthiphong Srigrarom. “Near-Parallel Binocular-Like Camera Pair for Multi-Drone Detection and 3D Localization.” In 2020 16th International Conference on Control, Automation, Robotics and Vision (ICARCV), pp. 204-210. IEEE, Shenzhen, China, 2020.
   • Link to paper: https://ieeexplore.ieee.org/abstract/document/9305485

4. Aim

This project aims to develop a methodology of identification, localization, and tracking of small and fast moving targets, such as flying drones, using an integrated multiple camera monitoring system. Our study focuses on using motion-based features to track targets, instead of traditional tracking algorithms that use appearance-based features by incorporating deep convolutional neural networks for target tracking. As we focus on small and fast moving targets, such as drones, using appearance-based features in this specific case may be difficult, as these targets often appear as small black “dots/blobs” in video frames. This would specifically mean that we use targets’ trajectory features, such as the derivatives of displacement, heading and turning angle, to identify and track targets instead.

Thus, our software incorporates the use of a multiple camera surveillance system to track and localize these moving targets. We aim to be able to continuously track drones in monitored airspaces, re-identify these targets between the cameras, and montor their 3-dimensional coordinates when they are inside the monitored space.

5. Capabilities

The current software runs in a Dockerized environment and it is built on Robot Operating System (ROS2) framework as our message passing ecosystem. We use ROS2’s Data Distribution Service (DDS) as our centralized system of communications for message passing. DDS uses RTPS (Real-Time Publish-Subscribe) protocol and acts as a central broker for data distribution. The software is built in C++ for high runtime performance capabilities, and we use Python for data management and matching of tracks’ feature values.

Our software mainly runs on two process and are handled in the respective ROS2 nodes. They are the detection and tracking process, and the re-identification and trackplot process. They are built on the two main ROS2 packages in our software stack, “MCMT Detect” and “MCMT Track”.

Our software capabilities include:

1. Open and read from single / double camera sensors, and obtain live camera frames.

2. Apply image processing techniques to remove background noise, by distinguisihing between foreground and background. We apply background subtraction in our image processing pipeline, and subsequently identify contours that do not conform to a minimum level of circularity. We deem these contours as background, and subtract them out of our image frames.

3. Apply morphological operations such as dilation and erosion to remove noise and enhance our detection.

4. Apply thresholding and binarization of our frames to obtained masked frames to detect targets using blob detection.

5. Use of Extemded Kalman filtering (KF) and Discriminative Correlation Filter (DCF) as our state estimation techniques to predict our targets’ next known location in the following frames.Both filters are implemented in our tracking pipeline, as these targets move at fast and erratic speeds, and using these filters allow us to better predict their positions for continuous tracking.

6. Implements the use of Hungarian / Munkre’s algorithm to match our detections to tracks. The matching algorithm is based on a 2 dimensional cost matrix of tracks and detections, where the cost is computed by comparing every detection’s euclidean distance away from each tracked target predicted location from our DCF and EKF filters.

7. The software has re-identification capabilities of targets between cameras. This would mean that every camera will be able to know that they are tracking the same target, purely based on their kinematic features. We implement cross-correlation matching of tracks’ trajectory features, and obtain a 2-dimensional correlation score between the cameras’ tracks.

8. Apply graph matching algorithm for geomentry-based identification using relative coordinates. The software initializes a complete bipartite graph, that calculates the maximum sum of the weights of the edges that span across the two disjoint groups of the complete bipartite graph. We obtain a geometry-based identification using relative coordinates, and serves as a secondary verification in our re-identification process.

9. Estimation of targets’ location in 3-dimensional space using triangulation and stereo camera methodology. We use disparities between the two camera frames to obtain depth estimations of the tracked targets.

6. Installation Guide

The following step-by-step processing will guide you on the installation process. Our software runs on Linux environment with ROS2 Eloquent. To run it on other OS, please install and run using Docker. The following step-by-step instructions will guide you on the installation process.

6.1 Install and build locally

1. Pull spot-it-3d repository

	get clone https://github.com/sieniven/spot-it-3d.git
	

1. Install ROS2 Eloquent (Bare Bones version). Refer to: https://index.ros.org/doc/ros2/Installation/Eloquent/Linux-Install-Debians/

2. Install required dependencies.

``` bash cd ~/spot-it-3d/mcmt-tracking/bin/

File truncated at 100 lines see the full file