Technical Paper | Documentation | X | Discord

OpenMind’s OM1 is a modular AI runtime that empowers developers to create and deploy multimodal AI agents across digital environments and physical robots, including Humanoids, Phone Apps, websites, Quadrupeds, and educational robots such as TurtleBot 4. OM1 agents can process diverse inputs like web data, social media, camera feeds, and LIDAR, while enabling physical actions including motion, autonomous navigation, and natural conversations. The goal of OM1 is to make it easy to create highly capable human-focused robots, that are easy to upgrade and (re)configure to accommodate different physical form factors.

Capabilities of OM1

• Modular Architecture: Designed with Python for simplicity and seamless integration.
• Data Input: Easily handles new data and sensors.
• Hardware Support via Plugins: Supports new hardware through plugins for API endpoints and specific robot hardware connections to ROS2, Zenoh, and CycloneDDS. (We recommend Zenoh for all new development).
• Web-Based Debugging Display: Monitor the system in action with WebSim (available at http://localhost:8000/) for easy visual debugging.
• Pre-configured Endpoints: Supports Voice-to-Speech, OpenAI’s gpt-4o, DeepSeek, and multiple Visual Language Models (VLMs) with pre-configured endpoints for each service.

Architecture Overview

Getting Started - Hello World

To get started with OM1, let’s run the Spot agent. Spot uses your webcam to capture and label objects. These text captions are then sent to OpenAI 4o, which returns movement, speech and face action commands. These commands are displayed on WebSim along with basic timing and other debugging information.

Package Management and VENV

You will need the uv package manager

Clone the Repo

git clone https://github.com/OpenmindAGI/OM1.git
cd OM1
git submodule update --init
uv venv

Install Dependencies

For MacOS

brew install portaudio
brew install ffmpeg

For Linux

sudo apt-get update
sudo apt-get install portaudio19-dev python-all-dev
sudo apt-get install ffmpeg

Obtain an OpenMind API Key

Obtain your API Key at OpenMind Portal. Copy it to config/spot.json5, replacing the openmind_free placeholder. Or, cp env.example .env and add your key to the .env.

Launching OM1

Run

uv run src/run.py spot

After launching OM1, the Spot agent will interact with you and perform (simulated) actions. For more help connecting OM1 to your robot hardware, see getting started.

What’s Next?

• Try out some examples
• Add new inputs and actions.
• Design custom agents and robots by creating your own json5 config files with custom combinations of inputs and actions.
• Change the system prompts in the configuration files (located in /config/) to create new behaviors.

Interfacing with New Robot Hardware

OM1 assumes that robot hardware provides a high-level SDK that accepts elemental movement and action commands such as backflip, run, gently pick up the red apple, move(0.37, 0, 0), and smile. An example is provided in actions/move_safe/connector/ros2.py:

...
elif output_interface.action == "shake paw":
    if self.sport_client:
        self.sport_client.Hello()
...

If your robot hardware does not yet provide a suitable HAL (hardware abstraction layer), traditional robotics approaches such as RL (reinforcement learning) in concert with suitable simulation environments (Unity, Gazebo), sensors (such as hand mounted ZED depth cameras), and custom VLAs will be needed for you to create one. It is further assumed that your HAL accepts motion trajectories, provides battery and thermal management/monitoring, and calibrates and tunes sensors such as IMUs, LIDARs, and magnetometers.

OM1 can interface with your HAL via USB, serial, ROS2, CycloneDDS, Zenoh, or websockets. For an example of an advanced humanoid HAL, please see Unitree’s C++ SDK. Frequently, a HAL, especially ROS2 code, will be dockerized and can then interface with OM1 through DDS middleware or websockets.

Recommended Development Platforms

OM1 is developed on:

• Jetson AGX Orin 64GB (running Ubuntu 22.04 and JetPack 6.1)
• Mac Studio with Apple M2 Ultra with 48 GB unified memory (running macOS Sequoia)
• Mac Mini with Apple M4 Pro with 48 GB unified memory (running macOS Sequoia)
• Generic Linux machines (running Ubuntu 22.04)

OM1 should run on other platforms (such as Windows) and microcontrollers such as the Raspberry Pi 5 16GB.

Detailed Documentation

More detailed documentation can be accessed at docs.openmind.org.

File truncated at 100 lines see the full file

Contributing to OM1

We welcome contributions from the community! OM1 is an open-source project, and we appreciate your help in making it better. Whether you’re fixing bugs, adding features, improving documentation, or suggesting new ideas, your contributions are valuable.

Before contributing, please take a moment to read through the following guidelines. This helps streamline the process and ensures everyone is on the same page.

Ways to Contribute:

• Report Bugs: If you find a bug, please open an issue on GitHub. Be sure to include:
  • A clear and concise description of the bug.
  • Steps to reproduce the bug.
  • Your operating system and Python version.
  • Relevant error messages or stack traces.
  • Screenshots (if applicable).

• Suggest Features: Have an idea for a new feature or improvement? Open an issue on GitHub and describe your suggestion. Explain the motivation behind the feature and how it would benefit OM1 users. We encourage discussion on feature requests before implementation.

• Improve Documentation: Good documentation is crucial. If you find anything unclear, incomplete, or outdated in the documentation, please submit a pull request with your changes. This includes the README, docstrings, and any other documentation files. Visit OM1 docs, and source code.

• Fix Bugs: Browse the open issues and look for bugs labeled “bug” or “help wanted.” If you want to tackle a bug, comment on the issue to let us know you’re working on it.

• Implement Features: Check the open issues for features labeled “enhancement” or “bounty” or “help wanted”. It’s best to discuss your approach in the issue comments before starting significant development.

• Write Tests: OM1 aims for high test coverage. If you’re adding new code, please include corresponding tests. If you find areas with insufficient test coverage, adding tests is a great contribution.

• Code Review: Reviewing pull requests is a valuable way to contribute. It helps ensure code quality and maintainability.

Contribution Workflow (Pull Requests):

1. Fork the Repository: Click the “Fork” button on the top-right of the OM1 repository page to create your own copy.

2. Clone Your Fork with CLI:

    git clone [https://github.com/](https://github.com/)<your-username>/OM1.git
    cd OM1
    

(Replace `<your-username>` with your GitHub username.)

1. Create a Branch: Create a new branch for your work. Use a descriptive name that reflects the purpose of your changes (e.g., fix-bug-xyz, add-feature-abc, docs-improve-readme).

    git checkout -b your-branch-name
    

1. Make Changes: Make your code changes, add tests, and update documentation as needed.

2. Commit Changes: Commit your changes with clear and concise commit messages. Follow the Conventional Commits specification if possible (e.g., feat: Add new feature, fix: Correct bug in module X, docs: Update README).

    git commit -m "feat: Add support for XYZ"
    

1. Push Changes: Push your branch to your forked repository.

    git push origin your-branch-name
    

1. Create a Pull Request (PR): Go to the original OM1 repository on GitHub. You should see a prompt to create a pull request from your newly pushed branch. Click “Compare & pull request.”

2. Write a Clear PR Description:
   • Describe the purpose of your pull request.
   • Link to any relevant issues it addresses (e.g., “Closes #123”).
   • Explain your changes and your design choices.
   • Include any relevant screenshots or GIFs (if applicable).

3. Request Review: Your pull request will be reviewed by the maintainers. Be prepared to address any feedback or make further changes.

4. Merge: Once your pull request is reviewed and approved, it will be merged into the main branch.

Coding Style and Conventions:

• Code Style: Follow the PEP 8 style guide. We may use a code formatter like black or ruff (check the pyproject.toml or setup.cfg for project-specific configuration). Run pre-commit run --all-files before committing.
• Docstrings: Write clear and comprehensive docstrings for all functions, classes, and modules. We may use a specific docstring format (e.g., Google style, NumPy style).
• Tests: Write unit tests to ensure your code works as expected. Use a testing framework like pytest.
• Type Hints: Use type hints (PEP 484) to improve code readability and maintainability.

Code of Conduct:

Please review and adhere to our Code of Conduct. We expect all contributors to be respectful and inclusive.

Getting Help:

If you have any questions or need help with the contribution process, feel free to:

• Open an issue on GitHub.
• Ask questions in the comments of relevant issues or pull requests.
• Join our developer telegram group.

Thank you for contributing to OM1!