Franka Teleoperation

This is a ROS2 Iron package implementing teleoperation on the Emika Franka Panda 7 DOF robot. The package uses Google’s MediaPipe hand tracking and gesture recognition in combination with MoveIt Servo to control the robot’s movement.

Check out the post for this project on my portfolio website: Franka Teleoperation

Introduction

This repository consists of several ROS packages

• franka_teleop: this custom package contains a ROS2 node which implements the MoveIt! Servo package. It’s responsible for controlling to the movement of the robot using linear and angular increments.
• handcv: a Python ROS2 node combining Google Mediapipe gesture recognition and hand tracking to provide hand waypoints and state information.
• cv_franka_bridge: A Python ROS2 node that subscribes to the information handcv provides, and processes it into commands for the franka_teleop node.
• numsr_franka_moveit_config: this custom package contains necessary configuration files for the Franka robot.
• hand_interfaces: A package containing custom ROS2 messages and services.

How to Run

Necessary packages:

• Media Pipe:

pip install mediapipe

• RealSense Camera
  • follow setup instructions here: https://github.com/IntelRealSense/realsense-ros/tree/ros2-development
• MoveIt

sudo apt install ros-{ROS_DISTRO}-moveit

Setup

Your robot may or may not be connected to the open internet. If it is, you can install moveit_servo and it’s dependencies by running:

sudo apt install ros-<distro>-moveit-servo

If you are like me and have a robot not connected to the internet, you’ll have to clone the necessary repos on your host system and then copy them over to your workspace. Here’s how to do that:

Install Necessary Packages

git clone git@github.com:gjcliff/FrankaTeleop.git

git clone git@github.com:ros-planning/moveit2.git

git clone git@github.com:ros2/ros_testing.git

• install these to some temporary location on your host machine.

Copy all packages over to the Franka control computer

Now we’re going to copy these three packages into a ros workspace on the robot’s computer in a specific order. We have to start with the lowest dependencies first and work up.

SSH into the robot in one terminal, and create the workspace on the robot if you haven’t already:

# on robot
ssh username@robot
source /opt/ros/<distro>/setup.bash
mkdir -p ~/Documents/ws/src/
cd ~/Documents/ws/

ros_testing is a dependency of moveit_servo, so we need to build it first.

Create a new terminal, and follow the next series of commands to copy ros_testing to the robot and build it:

# on host
scp ros_testing/ user@robot:/home/user/Documents/ws/src/

# on robot
colcon build
source install/setup.bash

Now that’s done, we can do the same thing to moveit_servo:

# on host
scp moveit_servo/* user@robot:/home/Documents/your/directory/here

# on robot
colcon build
source install/setup.bash

And finally, we can copy over the FrankaTeleop package:

# on host
scp FrankaTeleop/* user@robot:/home/Documents/ws/FrankaTeleop

```sh

on robot

File truncated at 100 lines see the full file