Overview

This repo provides a simulation stack for Stretch 4, built on MuJoCo. The simulation for Stretch 3 can be found in the stretch_mujoco repo. The simulation includes position control for the arm and gripper joints, velocity control for mobile base, calibrated camera RGB + depth imagery, 3D lidar clouds, and more. There is a visualizer that supports user interaction, or a more efficient headless mode. There is a ROS2 package, built on this library, that works with Nav2 and more. There is 100s of permutations of Robocasa-provided kitchen environments that Stretch can spawn into. The MuJoCo API can be used for features like deformables, procedural model generation, SDF collisions, cloth simulation, and more.

Check out a video of Stretch 4 in Robocasa environments in Mujoco:

mujoco_grid_4x.mp4

Getting Started

First, install uv. Uv is a package manager that we'll use to run this project.

Then, clone this repo:

git clone https://github.com/hello-robot/stretch4_mujoco --recurse-submodules
cd stretch4_mujoco

If you've already cloned the repo without --recurse-submodules, run git submodule update --init to pull the submodule.

Then, install this repo:

uv venv
uv pip install -e .

Lastly, run the simulation:

uv run launch_sim.py

To exit, press Ctrl+C in the terminal.

On MacOS, if mjpython fails to locate libpython3.10.dylib and libz.1.dylib, run these commands:

# Before proceeding, please reload your terminal and/or IDE window, to make sure the correct UV environment variables are loaded.

source .venv/bin/activate

# When `libpython3.12.dylib` is missing, run:
PYTHON_LIB_DIR=$(python3 -c 'from distutils.sysconfig import get_config_var; print(get_config_var("LIBDIR"))')
ln -s "$PYTHON_LIB_DIR/libpython3.12.dylib" ./.venv/lib/libpython3.12.dylib

# When `libz.1.dylib` is missing, run:
export DYLD_LIBRARY_PATH=/usr/lib:$DYLD_LIBRARY_PATH

GPU Acceleration

On Linux, by default, the Python code will attempt to use GPU acceleration by setting the necessary environment variables at runtime. However, if you are running examples outside the typical Python execution flow or need to ensure these are set system-wide, you can manually export the following environment variables:

export MUJOCO_GL=egl
export PYOPENGL_PLATFORM=egl
export XLA_FLAGS=--xla_gpu_triton_gemm_any=true

These variables ensure that MuJoCo and any associated rendering libraries utilize hardware acceleration, which is highly recommended for performance when working with cameras or complex scenes.

Example Scripts

Keyboard teleop

uv run examples/keyboard_teleop.py

Gamepad teleop

Control Stretch in simulation using any xbox type gamepad (read through pygame/SDL2, so it works on Linux, macOS and Windows)

uv run examples/gamepad_teleop.py

Digital twin

Mirror motion between the sim and a real Stretch 4 over the network. Commands issued in sim are forwarded to the robot as the same RobotClient command, and the robot's joint status is followed back into the sim, so moving either side moves the other.

uv pip install -e ".[digital-twin]"
uv run examples/digital_twin.py --robot_ip 192.168.1.10

Use --controller sim or --controller robot to mirror in one direction only (the default is bidirectional), and --joints to limit which joint groups are mirrored.

Velocity Control

Demonstrate base and joint velocity control with set_velocity()

uv run examples/velocity_control.py

Robocasa environments

# Setup
uv pip install -e "robocasa@third_party/robocasa"
uv pip install -e "robosuite@third_party/robosuite"
uv run third_party/robosuite/robosuite/scripts/setup_macros.py
uv run third_party/robocasa/robocasa/scripts/setup_macros.py
uv run third_party/robocasa/robocasa/scripts/download_kitchen_assets.py

# Run sim
uv run examples/keyboard_teleop.py --select_env
uv run examples/gamepad_teleop.py --select_env
uv run examples/robocasa_environment.py

Ignore any warnings.

Writing Code

Use the Stretch4MujocoSimulator class to:

• start the simulation
• position control the robot's ranged joints
• velocity control the robot's mobile base
• read joint states
• read camera imagery
• read 2D and 3D LiDAR point clouds

Try the code below using uv run ipython. For advanced Mujoco users, the class also exposes the mjModel and mjData. See the official Mujoco documentation.

from stretch4_mujoco import Stretch4MujocoSimulator

if __name__ == "__main__":
    sim = Stretch4MujocoSimulator()
    sim.start(headless=False) # This will open a Mujoco-Viewer window

    # Poses
    sim.stow()
    sim.home()

    # Position Control (Subsystem API)
    sim.lift.move_to(1.0)
    sim.end_of_arm.wrist_yaw.move_by(0.2)
    sim.base.translate_by(0.1)

    sim.wait_command()

    # Base Velocity control
    sim.base.set_velocity(vx_m=0.3, vy_m=0.0, w_r=-0.1)

    # Get Joint Status
    from pprint import pprint
    pprint(sim.pull_status())

    # Get Camera Frames
    camera_data = sim.pull_camera_data()
    pprint(camera_data)

    # Get 2D LiDAR Scan Data (from status sensors)
    sensor_data = sim.pull_sensor_data()
    from stretch4_mujoco.enums.stretch_sensors import StretchSensors
    pprint(sensor_data.get_data(StretchSensors.base_lidar))

    # Get 3D LiDAR Point Clouds (from left and right lidars)
    lidar_data = sim.pull_lidar_points()
    pprint(lidar_data["left"])
    pprint(lidar_data["right"])

    # Kills simulation process
    sim.stop()

Loading Robocasa Kitchen Scenes

The stretch4_mujoco.robocasa_gen.model_generation_wizard() method gives you:

• Wizard/API to generate a kitchen model for a given task, layout, and style.
• If layout and style are not provided, it will take you through a wizard to choose them in the terminal.
• If robot_spawn_pose is not provided, it will spawn the robot to the default pose from robocasa fixtures.
• You can also write the generated xml model with absolutepaths to a file.

from stretch4_mujoco import Stretch4MujocoSimulator
from stretch4_mujoco.robocasa_gen import model_generation_wizard

# Use the wizard:
model, xml, objects_info = model_generation_wizard(stretch_xml_absolute=StretchMujocoSimulator.get_robot_xml_path(),)

# Or, launch a specific task/layout/style
model, xml = model_generation_wizard(
    stretch_xml_absolute=StretchMujocoSimulator.get_robot_xml_path(),
    task=<task_name>,
    layout=<layout_id>,
    style=<style_id>,
    wrtie_to_file=<filename>,
)

sim = Stretch4MujocoSimulator(model=model, use_full_center_camera_resolution=False)
sim.start()

ROS2

You can use this simulation in ROS2 using the stretch_simulation package in stretch4_ros2.

Docs

Check out the following documentation resources:

• Architecture
• Stretch 4 MJCF and URDF

Feature Requests and Bug reporting

All the enhancements/bugfixes are tracked by Github Issues filed. Please feel free to file an issue if you would like to report a bug or request a feature addition.

Acknowledgment

The assets in this repository contain significant contributions and efforts from Kevin Zakka and Google Deepmind, along with others in Hello Robot Inc. who helped us in modeling Stretch in Mujoco. Thank you for your contributions.

License

The license covering the code and assets in this repo can be found in the LICENSE file.