GaussianRPG: 3D Gaussian Rendering PlayGround
This is an open-source autonomous driving closed-loop simulator demo using 3D Gaussian Splatting tech. The whole project is base on the codes from this paper:
Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splatting
Yunzhi Yan, Haotong Lin, Chenxu Zhou, Weijie Wang, Haiyang Sun, Kun Zhan, Xianpeng Lang, Xiaowei Zhou, Sida Peng
ECCV 2024
In this project, a simulator has been developed. Street_Gaussians is responsible for rendering images based on the camera position. A dummy AEB controller applies YOLOv5 to recognize cars in the images and determine the distance, issuing braking commands when appropriate. The car dynamics component within the simulator adjusts the camera position according to the braking commands and provides this information to Street_Gaussians as the basis for rendering the next frame. For convenience, ROS2 serves as the middleware providing the framework for the simulator.
GaussianRPG v2.0 is launched!!
Examples:
Waymo Scene 002:
Waymo Scene 124:
Waymo Scene 149:
Installation
Environments of the repo:
ubuntu 22.04
GeForce RTX 3090
Clone this repository
```
git clone https://github.com/GimpelZhang/GaussianRPG.git
```
Set up the python environment
```
# First make sure there is g++-11 in your system, if not:
sudo apt install g++
# Install mamba to manage environments:
conda install mamba -c conda-forge
# Set conda environment
mamba create -n street-gaussian-ros python=3.8
mamba activate street-gaussian-ros
conda config --env --add channels conda-forge
conda config --env --add channels robostack-staging
conda config --env --remove channels defaults
# Install ROS2 humble:
mamba install ros-humble-desktop
mamba deactivate
mamba activate street-gaussian-ros
mamba install compilers cmake pkg-config make ninja colcon-common-extensions catkin_tools rosdep
pip install setuptools==58.2.0
# Install torch (corresponding to your CUDA version)
pip install torch==2.1.2+cu118 torchvision==0.16.2+cu118 --extra-index-url https://download.pytorch.org/whl/cu118
# Install requirements
pip install -r requirements.txt
conda install ninja
# Install submodules
# Because of the ROS2 env, g++ is in version 12 in this conda env.
# So next we can link to the g++-11 in our system directly:
CC=/usr/bin/gcc CXX=/usr/bin/g++ pip install ./submodules/diff-gaussian-rasterization
CC=/usr/bin/gcc CXX=/usr/bin/g++ pip install ./submodules/simple-knn
CC=/usr/bin/gcc CXX=/usr/bin/g++ pip install ./submodules/simple-waymo-open-dataset-reader
CC=/usr/bin/gcc CXX=/usr/bin/g++ python script/test_gaussian_rasterization.py
# Build ROS2 nodes:
cd nodes
colcon build
# Further, if you want to train or to render using Street_Gaussians in this street-gaussian-ros conda env,
# don't forget to add the g++-11 path before every command:
CC=/usr/bin/gcc CXX=/usr/bin/g++ python train.py --config configs/xxxx.yaml
CC=/usr/bin/gcc CXX=/usr/bin/g++ python render.py --config configs/xxxx.yaml mode {evaluate, trajectory}
```
Prepare Waymo Open Dataset and others to run the simulator.
To run the GaussianRPG simulator, a trained Street_Gaussians model is needed. To render Street_Gaussians models, the training data folders are also needed.
I provide all the data for the **examples** including: Waymo scene training data, trained outputs, and some **special files** inside them.
| baidunetdisk | google drive |
|-------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------|
File truncated at 100 lines [see the full file](https://github.com/gimpelzhang/gaussianrpg/tree/main/README.md)
ROS Index