Stretch AI

Stretch AI is designed to help researchers and developers build intelligent behaviors for the Stretch 3 mobile manipulator from Hello Robot. It contains code for:

• grasping
• manipulation
• mapping
• navigation
• LLM agents
• text to speech and speech to text
• visualization and debugging

Much of the code is licensed under the Apache 2.0 license. See the LICENSE file for more information. Parts of it are derived from the Meta HomeRobot project and are licensed under the MIT license.

Hardware Requirements

We recommend the following hardware to run Stretch AI. Other GPUs and other versions of Stretch may support some of the capabilities found in this repository, but our development and testing have focused on the following hardware.

• Stretch 3 from Hello Robot
  • When Checking Hardware, stretch_system_check.py should report that all hardware passes.
• Computer with an NVIDIA GPU
  • The computer should be running Ubuntu 22.04. Later versions might work, but have not been tested.
  • Most of our testing has used a high-end CPU with an NVIDIA GeForce RTX 4090.
• Dedicated WiFi access point
  • Performance depends on high-bandwidth, low-latency wireless communication between the robot and the GPU computer.
  • The official Stretch WiFi Access Point provides a tested example.
• (Optional) Stretch Dexterous Teleop Kit.
  • To use the learning-from-demonstration (LfD) code you’ll need the Stretch Dexterous Teleop Kit.

Quick-start Guide

Artificial intelligence (AI) for robots often has complex dependencies, including the need for trained models. Consequently, installing stretch-ai from source can be challenging.

First, you will need to install software on your Stretch robot and another computer with a GPU (GPU computer). Use the following link to go to the installation instructions: Instructions for Installing Stretch AI

Once you’ve completed this installation, you can start the server on your Stretch robot. Prior to running the script, you need to have homed your robot with stretch_robot_home.py. Then, run the following command:

./scripts/run_stretch_ai_ros2_bridge_server.sh

After this, we recommend trying the Language-Directed Pick and Place demo.

Experimental support for Older Robots

The older model of Stretch, the Stretch RE2, did not have an camera on the gripper. If you want to use this codebase with an older robot, you can purchase a Stretch 2 Upgrade Kit to give your Stretch 2 the capabilities of a Stretch 3. Alternatively, you can run a version of the server with no d405 camera support on your robot.

Note that many demos will not work with this script (including the Language-Directed Pick and Place demo) and learning from demonstration. However, you can still run the simple motions demo and view images with this script.

./scripts/run_stretch_ai_ros2_bridge_server.sh --no-d405

Optional: Docker Quickstart

To help you get started more quickly, we provide two pre-built Docker images that you can download and use with two shell scripts.

On your remote machine, you can install docker as normal, then, you can start the client on your GPU computer:

./scripts/run_stretch_ai_gpu_client.sh

This script will download the Docker image and start the container. You will be able to run Stretch AI applications from within the container.

Language-Directed Pick and Place

Now that you have the server running on Stretch, we recommend you try a demonstration of language-directed pick and place.

For this application, Stretch will attempt to pick up an object from the floor and place it inside a nearby receptacle on the floor. You will use words to describe the object and the receptacle that you’d like Stretch to use.

While attempting to perform this task, Stretch will speak to tell you what it is doing. So, it is a good idea to make sure that you have the speaker volume up on your robot. Both the physical knob on Stretch’s head and the volume settings on Stretch’s computer should be set so that you can hear what Stretch says.

Now, on your GPU computer, run the following commands in the Docker container that you started with the script above.

You need to let the GPU computer know the IP address (#.#.#.#) for your Stretch robot.

./scripts/set_robot_ip.sh #.#.#.#

Please note that it’s important that your GPU computer and your Stretch robot be able to communicate via the following ports 4401, 4402, 4403, and 4404. If you’re using a firewall, you’ll need to open these ports.

Next, run the application on your GPU computer:

python -m stretch.app.ai_pickup

It will first spend time downloading various models that it depends on. Once the program starts, you will be able to bring up a Rerun-based GUI in your web browser.

File truncated at 100 lines see the full file

Contributing to Stretch AI

We welcome contributions to Stretch AI! Please read the following guidelines before submitting a pull request.

This repository is in an early state of development. Guidelines are subject to change and tests will be by necessity incomplete.

Before contributing, you will need to sign a Contributor License Agreement. For more information, you can check out the Stretch Contributor License Agreement page for more information.

We use the SAP CLA Assistant bot, which will open a comment and give you instructions when you open a pull request.

Setup

Install the code and set up the pre-commit hooks:

git clone https://github.com/hello-robot/stretch_ai.git --recursive
cd stretch_ai/src
pip install -e .[dev]
pre-commit install

Style

We use black and flake8 to format our code. In addition, we use isort for sorting imports, mypy for static type checking, and codespell for spell checking, among other things.

You can run the pre-commit hooks on all files with:

pre-commit run --all-files

Please make sure that all changes are made and that the pre-commit hooks pass before submitting a pull request.

If you need to temporarily commit something that is not passing, use:

git commit --no-verify

However, pull requests with failing pre-commit hooks will not be merged.

Pull Requests

We follow a squash-and-merge strategy for pull requests, which means that all commits in a PR are squashed into a single commit before merging. This keeps the git history clean and easy to read.

Please make sure your PR is up-to-date with the latest changes in the main branch before submitting. You can do this by rebasing your branch on the main branch:

git checkout main
git pull
git checkout <your-branch>
git rebase main

Draft PRs

If a PR is still a work-in-progress and not ready for review, please open it with “WIP: (final PR title)” in the title to indicate that it is still a work in progress. This will indicate to reviewers that the PR is not ready for review yet. In addition, use the “Draft” PR status on github to indicate that the PR is not ready yet.

Documentation

Please make sure to update the documentation if you are adding new features or changing existing ones. This includes docstrings, README, and any other relevant documentation. Use type hints to make the code more readable and maintainable.

For example:

def add(a: int, b: int) -> int:
    return a + b

This shows what a and b are expected to be and what the function returns – in this case, all are int variables.

Testing

We use pytest for testing. Please make sure to add tests for new features or changes to existing code. You can run the tests with:

cd src
pytest

File Structure

The code is organized as follows. Inside the core package src/stretch:

• core is basic tools and interfaces
• app contains individual endpoints, runnable as python -m stretch.app.<app_name>, such as mapping, discussed above.
• motion contains motion planning tools, including algorithms like RRT.
• mapping is broken up into tools for voxel (3d / ok-robot style), instance mapping
• perception contains tools for perception, such as object detection and pose estimation.
  • perception/encoders contains tools for encoding vision and language features, such as the SiglipEncoder class.
  • perception/captioners contains tools for generating captions from images, such as the MoonbeamCaptioner class.
• agent is aggregate functionality, particularly robot_agent which includes lots of common tools including motion planning algorithms.
  • In particular, agent/zmq_client.py is specifically the robot control API, an implementation of the client in core/interfaces.py. there’s another ROS client in stretch_ros2_bridge.
  • agent/robot_agent.py is the main robot agent, which is a high-level interface to the robot. It is used in the app scripts.
  • agent/base contains base classes for creating common and sequentially executed robot operations through the ManagedOperation class.
  • agent/task contains task-specific code, such as for the pickup task. This is divided between “Managers” like pickup_manager.py which are composed of “Operations.” Each operation is a composable state machine node with pre- and post-conditions.
  • agent/operations contains the individual operations, such as move_to_pose.py which moves the robot to a given pose.

The stretch_ros2_bridge package is a ROS2 bridge that allows the Stretch AI code to communicate with the ROS2 ecosystem. It is a separate package that is symlinked into the ament_ws workspace on the robot.

File truncated at 100 lines see the full file