tag_based_pnp_calibrator

A tutorial for this calibrator can be found here.

Purpose

The package tag_based_pnp_calibrator performs extrinsic calibration between camera and 3d lidar sensors used in autonomous driving and robotics.

Note that the 3d lidar should have a resolution that is high enough to detect the lidartag [1] or be among the compatible models.

Inner-workings / Algorithms

The tag_based_pnp_calibrator package utilizes the PnP (Perspective-n-Point) algorithm, a computer vision technique that is used to estimate the transformation between a set of correspondences of 2d and 3d points (in this case 2d points come from the camera and 3d points come from the lidar). This package also makes use of the apriltag_ros and lidartag packages, which are executed automatically by this package’s launcher files.

The apriltag_ros package detects apriltag markers from an image and outputs the detection results. Conversely, the lidartag package detects lidartag markers and outputs its detection results. apriltag and lidartag differ in that in every lidartag there is an apriltag payload, but the opposite is not guaranteed. It follows that for the same physical marker, the detection’s sizes are different, which is compensated during calibration.

The tag_based_pnp_calibrator utilizes the detections from both the apriltag_ros and lidartag packages, employing a Kalman filter to monitor these detections. If the detections converge, the calibrator applies the SQPnP [2] algorithm provided by OpenCV to estimate the transformation between the image points from the apriltag markers and the object points from the lidartag markers, which is effectively the transformation from camera to lidar.

Diagram

The following figure presents a diagram of the nodes, launchers and relevant methods involved in the calibration process:

ROS Interfaces

Input

Output

Services

Parameters

Core Parameters

Requirements

lidartag

To perform camera-lidar calibration using this tool, it is necessary to prepare lidartags and lidars with intensity measures. To ensure that no object obstructs the tag detection and to achieve the most stable detection possible, it is highly recommended to also prepare fixed mounts for these tags, as shown below.

Note that the lidartags we used have a size of 0.8 meters (edge size). This means their apriltag payload is 0.6 meters (black frame’s edge size). We have also tried with lidartags with smaller sizes (e.g., edge size of 0.6 meters), but to use them the user is required to set several parameters by himself.

Calibration process

A complete example with real data is provided in our tutorial, so we recommend users interested in this tools to get hands-on-experience directly with that example. Nevertheless, the calibration process consists of the following steps:

• Place the available tags within the field-of-view of the sensors (see the Pro tips/recommendations for more information).
• Ensure that both sensors detect the tags, and wait until the detections converge.
• Once the detections are added to the calibration data, move the tags to a new location, making sure to fulfill the new_hypothesis_distance criteria.
• If needed, just after moving the tag, stop its oscillations with your hand.
• Repeat the process until calibration_convergence_min_pairs pairs have been obtained and the calibration process finishes.

Known issues/limitations

• The tool uses a basic OpenCV camera model for calibration.
• Only lidars with an intensity channel can be used (lidartag requirement).

Pro tips/recommendations

• During calibration, ensure that the lidar scan covers the tag similar to the first example shown in the image below. However, if the tag resolution is low, as in the second example, and the lidar detects the tag, it may still be acceptable, yet should be avoided when possible. The third example demonstrates a scenario where the lidar scan fails to cover the tag, resulting in the inability to detect the lidartag.

• It is highly recommended to place the tag perpendicular to the lidar as shown in the following image:

• As in the previous image, it is is important to spread the tag samples throughout the calibration area, making sure it covers the field-of-view of the camera, and encompasses multiple distances to minimize the error incurred due to imperfect camera models.

Regarding the reprojection error and its relation to the calibration error

This method formulates camera-lidar as an optimization problem that minimizes the reprojection error between camera and lidar points. Achieving perfect reprojection error would imply a perfect calibration (pose between sensors), but that can only be achieved when both detectors and the camera model are also perfect. If this is not the case, even though the reprojection errors acts as a proxy for the real 3d pose error, there is no direct correspondence / translation.

In our experiments using lenses with an horizontal field-of-view between 30 and 90 degrees, up to 2 radial distortion coefficients, and HD resolution, we can achieve sub pixel reprojection error upon convergence.

For higher resolution cameras, usually the reprojection error increases almost linearly with the resolution, indicating that the bottleneck lies between the lidar detection accuracy and the lens model. When using wider lenses, more complex camera models are needed. If this requisite is not met, the reprojection error will likely not converge to an adequate value or may even cause the optimizer to fail.

References

[1] Jiunn-Kai (Bruce) Huang, Shoutian Wang, Maani Ghaffari, and Jessy W. Grizzle, “LiDARTag: A Real-Time Fiducial Tag System for Point Clouds,” in IEEE Robotics and Automation Letters. Volume: 6, Issue: 3, July 2021.

[2] G. Terzakis and M. Lourakis, “A Consistently Fast and Globally Optimal Solution to the Perspective-n-Point Problem” in ECCV 2020. Lecture Notes in Computer Science, vol 12346.