ROS Index
|
rai_open_set_vision package from rai reporai_bringup rai_nomad rai_open_set_vision |
Package Summary
| Tags | No category tags. |
| Version | 0.1.0 |
| License | Apache-2.0 |
| Build type | AMENT_PYTHON |
| Use | RECOMMENDED |
Repository Summary
| Description | RAI is a vendor-agnostic agentic framework for robotics, utilizing ROS 2 tools to perform complex actions, defined scenarios, free interface execution, log summaries, voice interaction and more. |
| Checkout URI | https://github.com/robotecai/rai.git |
| VCS Type | git |
| VCS Version | development |
| Last Updated | 2025-06-04 |
| Dev Status | UNKNOWN |
| CI status | No Continuous Integration |
| Released | UNRELEASED |
| Tags | ai robotics ros2 vlm multimodal embodied-artificial-intelligence embodied-agent embodied-ai o3de llm generative-ai ai-agents-framework embodied-agents robotec |
| Contributing |
Help Wanted (0)
Good First Issues (0) Pull Requests to Review (0) |
Package Description
Additional Links
Maintainers
- Kajetan Rachwał
Authors
RAI Open Set Vision
This package provides a ROS2 Node which is an interface to the Idea-Research GroundingDINO Model. It allows for open-set detection.
Installation
In your workspace you need to have an src folder containing this package rai_open_set_vision and the rai_interfaces package.
Preparing the GroundingDINO
Add required ROS dependencies:
rosdep install --from-paths src --ignore-src -r
Build and run
In the base directory of the RAI package install dependencies:
poetry install --with openset
Source the ros installation
source /opt/ros/${ROS_DISTRO}/setup.bash
Run the build process:
colcon build --symlink-install
Source the environment
source setup_shell.sh
Run the GroundedSamAgent and GroundingDinoAgent agents.
python run_vision_agents.py
Agents create two ROS 2 Nodes: grounding_dino and grounded_sam using ROS2Connector.
These agents can be triggered by ROS2 services:
-
grounding_dino_classify:rai_interfaces/srv/RAIGroundingDino -
grounded_sam_segment:rai_interfaces/srv/RAIGroundedSam
[!TIP]
If you wish to integrate open-set vision into your ros2 launch file, a premade launch file can be found in
rai/src/rai_bringup/launch/openset.launch.py
[!NOTE] The weights will be downloaded to
~/.cache/raidirectory.
RAI Tools
rai_open_set_vision package contains tools that can be used by RAI LLM agents
enhance their perception capabilities. For more information on RAI Tools see
Tool use and development tutorial.
GetDetectionTool
This tool calls the grounding dino service to use the model to see if the message from the provided camera topic contains objects from a comma separated prompt.
[!TIP]
you can try example below with rosbotxl demo binary. The binary exposes
/camera/camera/color/image_rawand/camera/camera/depth/image_rawtopics.
Example call
from rai_open_set_vision.tools import GetDetectionTool
from rai.communication.ros2 import ROS2Connector, ROS2Context
with ROS2Context():
connector=ROS2Connector(node_name="test_node")
x = GetDetectionTool(connector=connector)._run(
camera_topic="/camera/camera/color/image_raw",
object_names=["chair", "human", "plushie", "box", "ball"],
)
Example output
I have detected the following items in the picture - chair, human
GetDistanceToObjectsTool
This tool calls the grounding dino service to use the model to see if the message from the provided camera topic contains objects from a comma separated prompt. Then it utilises messages from depth camera to create an estimation of distance to a detected object.
Example call
from rai_open_set_vision.tools import GetDetectionTool
from rai.communication.ros2 import ROS2Connector, ROS2Context
with ROS2Context():
connector=ROS2Connector(node_name="test_node")
connector.node.declare_parameter("conversion_ratio", 1.0) # scale parameter for the depth map
x = GetDistanceToObjectsTool(connector=connector)._run(
camera_topic="/camera/camera/color/image_raw",
depth_topic="/camera/camera/depth/image_rect_raw",
object_names=["chair", "human", "plushie", "box", "ball"],
)
Example output
I have detected the following items in the picture human: 3.77m away
Simple ROS2 Client Node Example
An example client is provided with the package as rai_open_set_vision/talker.py
You can see it working by running:
python run_vision_agents.py
cd rai # rai repo BASE directory
ros2 run rai_open_set_vision talker --ros-args -p image_path:=src/rai_extensions/rai_open_set_vision/images/sample.jpg
If everything was set up properly you should see a couple of detections with classes dinosaur, dragon, and lizard.
Wiki Tutorials
Package Dependencies
| Deps | Name |
|---|---|
| vision_msgs | |
| rai_interfaces | |
| ament_copyright | |
| ament_pep257 |
System Dependencies
| Name |
|---|
| python3-pytest |
Dependant Packages
Launch files
- launch/gsam_launch.xml
- ~ Copyright 2024 Robotec.ai ~ ~ Licensed under the Apache License, Version 2.0 (the "License"); ~ you may not use this file except in compliance with the License. ~ You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License.
-
- weights_path [default: ]
- launch/gdino_launch.xml
- ~ Copyright 2024 Robotec.ai ~ ~ Licensed under the Apache License, Version 2.0 (the "License"); ~ you may not use this file except in compliance with the License. ~ You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License.
-
- weights_path [default: ]
- launch/example_communication_launch.xml
- ~ Copyright 2024 Robotec.ai ~ ~ Licensed under the Apache License, Version 2.0 (the "License"); ~ you may not use this file except in compliance with the License. ~ You may obtain a copy of the License at ~ ~ http://www.apache.org/licenses/LICENSE-2.0 ~ ~ Unless required by applicable law or agreed to in writing, software ~ distributed under the License is distributed on an "AS IS" BASIS, ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. ~ See the License for the specific language governing permissions and ~ limitations under the License.
-
- dino_weights_path [default: ]
- sam_weights_path [default: ]
- image_path