Skip to content

feat(sim): add xarm6-worldbelief-sim blueprint - #3242

Open
mustafab0 wants to merge 1 commit into
mainfrom
mustafa/sim/l1-worldbelief-sim-blueprint
Open

feat(sim): add xarm6-worldbelief-sim blueprint#3242
mustafab0 wants to merge 1 commit into
mainfrom
mustafa/sim/l1-worldbelief-sim-blueprint

Conversation

@mustafab0

@mustafab0 mustafab0 commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Motivation

The WorldBelief perception stack can only be exercised with an xArm6 on the desk. This makes the sim a first-class dev surface: dimos run xarm6-worldbelief-sim brings up the same perception/planning/MCP stack against MuJoCo, so scan, recall, TF, gripper, and the agent path can be developed and regression-tested without hardware.

What it wires

xarm6-worldbelief-sim mirrors xarm6-worldbelief with the RealSense wrist camera replaced by MujocoSimModule (wrist_camera, base_frame_id=link6, dof=6, XARM6_SIM_PATH). It renders the MJCF camera and publishes the same color/depth/camera-info topics plus link6 -> wrist_camera_*_optical_frame from sim ground truth; world -> link6 comes from ManipulationModule FK.

  • The rerun bridge, recorder, belief module, and MCP server were byte-identical between the two blueprints, so they move into a worldbelief_stack(state_subdir) factory. xarm6_worldbelief is otherwise untouched — verified by diffing every atom's module + kwargs before/after (only function object addresses differ).
  • XARM6_SIM_HOME puts wrist_camera at world (0.45, 0, 0.52) looking straight down. Solved against data/xarm6/scene.xml as shipped, i.e. with link_base on the 0.12 m pedestal — the same convention XARM7_SIM_HOME already follows (verified: main's xarm7 home lands the camera at 0.44991, 0, 0.51979).
  • make_xarm6_sim_hardware / make_xarm6_sim_module_kwargs mirror the existing xarm7 factories.

No changes outside blueprint, config, registration, and tests.

Coordination with #3275 (worldbelief → experimental)

#3275 (approved) relegates worldbelief to dimos/experimental/world_belief/, including moving xarm/blueprints/worldbelief.pyexperimental/world_belief/xarm6_blueprint.py.

By design this PR stays green against main and does not pre-empt those paths. worldbelief_sim.py intentionally stays in the xarm package. Once #3275 lands, this blueprint needs exactly one import line changed:

-from dimos.robot.manipulators.xarm.blueprints.worldbelief import worldbelief_stack
+from dimos.experimental.world_belief.xarm6_blueprint import worldbelief_stack

plus the same substitution for the two lazy imports in test_worldbelief_sim_blueprint.py. Note #3275 renames worldbelief.py at 96% similarity, so its move and this PR's edits to that file will need a merge resolution whichever order they land in.

Rebase notes (onto #3183)

Rebased onto current main. Two API changes absorbed:

  • dimos/robot/manipulators/test_blueprints.py was deleted by feat: extract manipulation plan execution manager #3183, so the blueprint-composition tests moved to a scoped dimos/robot/manipulators/xarm/test_worldbelief_sim_blueprint.py rather than resurrecting the deleted file.
  • RobotModelConfig.coordinator_task_name was removed by feat: extract manipulation plan execution manager #3183; the test now derives the expected task name from trajectory_task_name(robot.name).
  • xarm6-planner-only was dropped from SELF_HOSTED_BLUEPRINTS on main; that removal is preserved.

Validation

Re-run after the rebase, so these reflect #3183's new plan-execution manager.

result
Launch 7/7 modules deploy, 0 errors; camera transports wired; arm reaches home
Trajectory move_to_joints → "Reached target joint configuration"; 51-point trajectory completes; arm converges toward target (0.530 → 0.546 → 0.558 → 0.6)
Scan "Scan complete: 1 object(s) present" (49 s, 12 frames folded)
MCP server lists 14 skills; go_home / move_to_joints / get_robot_state / scan all invoke
Agent McpClient deploys and discovers all 14 tools over MCP
Tests 280 pass (blueprint suites + generation + kwargs checks); lint/format clean

Detected red ball vs MJCF ground truth (apple at 0.40, 0.08, 0.17):

axis detected truth error
X 0.4094 0.40 +0.94 cm
Y 0.0697 0.08 −1.03 cm
Z 0.0746 0.17 −9.54 cm

Known issues (the PR ladder)

Documented, not fixed — each is its own rung.

  1. Base-frame disagreement → scan Z reads ~9.5 cm low; pick will fail IK. data/xarm6/scene.xml spawns link_base at pos="0 0 .12" (plus a pedestal geom), but make_xarm_model_config places base_pose at the world origin. The sim renders from the true camera pose while perception unprojects using the FK pose. Measured −9.54 cm above, matching prediction. Scan/detect/recall work; pick does not. Next rung.
  2. Gripper direction inverted. hand.xml's driver joint grows as the jaws close; DiMOS positions grow as they open. Needs gripper_position_inverted support in MujocoSimModule, which main does not have.
  3. Pink IK aborts retries on QP exception. Needed before pick converges.
  4. scan / recall exceed the 120 s default RPC timeout. Server-side the skill succeeds (SKILL scan result=OK duration_ms=156804.7) but the MCP layer reports an error at 120 s. Aggravated by (5).
  5. Detector runs on CPU on this boxFailed to initialize NVML: Driver/library version mismatch (NVML 595.84), so torch.cuda.is_available() is False. Environment, not code, but it is what pushes scan past the RPC limit.
  6. dimos mcp call hardcodes a 30 s HTTP read timeout, below the scan's runtime even on the happy path.
  7. Trajectories converge asymptotically. move_to_joints/go_home report success while the arm is still travelling; sampling immediately after the call shows it near its start. Repeated go_home calls converge (joint 0: 0.417 → 0.021 → 0.013 → 0.008).
  8. Scan prompt sensitivity. With ["red ball", "orange ball", "cup"] only the red ball is detected; the orange and cup are missed.

Not verified

The agent's natural-language exchange could not complete: the available OpenAI key is rejected with 401 invalid_api_key (confirmed with a direct curl to api.openai.com, so it is the credential, not the plumbing). Everything up to the LLM call works — the agent deploys, connects, and discovers all 14 MCP tools.

@codecov

codecov Bot commented Jul 28, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ All tests successful. No failed tests found.

@@            Coverage Diff             @@
##             main    #3242      +/-   ##
==========================================
+ Coverage   75.20%   75.22%   +0.01%     
==========================================
  Files        1128     1130       +2     
  Lines      107954   108020      +66     
  Branches     9751     9752       +1     
==========================================
+ Hits        81188    81254      +66     
- Misses      23955    23956       +1     
+ Partials     2811     2810       -1     
Flag Coverage Δ
OS-ubuntu-24.04-arm 68.85% <56.89%> (-0.01%) ⬇️
OS-ubuntu-latest 70.91% <56.89%> (-0.01%) ⬇️
Py-3.10 70.91% <56.89%> (-0.02%) ⬇️
Py-3.11 70.90% <56.89%> (-0.02%) ⬇️
Py-3.12 70.91% <56.89%> (-0.01%) ⬇️
Py-3.13 70.91% <56.89%> (-0.01%) ⬇️
Py-3.14 70.91% <56.89%> (-0.01%) ⬇️
Py-3.14t 70.90% <56.89%> (-0.02%) ⬇️
SelfHosted-Large 29.10% <29.62%> (+0.01%) ⬆️
SelfHosted-Linux 35.64% <83.58%> (-0.02%) ⬇️
SelfHosted-macOS 34.71% <83.58%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
dimos/robot/all_blueprints.py 100.00% <ø> (ø)
.../robot/manipulators/xarm/blueprints/worldbelief.py 87.09% <100.00%> (+1.38%) ⬆️
...ot/manipulators/xarm/blueprints/worldbelief_sim.py 100.00% <100.00%> (ø)
dimos/robot/manipulators/xarm/config.py 90.47% <100.00%> (+0.82%) ⬆️
...anipulators/xarm/test_worldbelief_sim_blueprint.py 100.00% <100.00%> (ø)
dimos/robot/test_all_blueprints.py 87.50% <ø> (ø)

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions github-actions Bot added the ready-to-merge Required CI checks have passed on this PR label Jul 28, 2026
WorldBelief stack against MuJoCo: mirrors xarm6_worldbelief with the
RealSense replaced by MujocoSimModule (wrist_camera, base_frame_id=link6,
dof=6, XARM6_SIM_PATH), giving the perception/agent stack a first-class
dev surface that needs no arm on the desk.

The rerun bridge, recorder, belief module, and MCP server are identical
for both camera sources, so they move into a shared worldbelief_stack()
factory parameterized by state subdir; xarm6_worldbelief is unchanged
apart from calling it.

XARM6_SIM_HOME puts wrist_camera at world (0.45, 0, 0.52) looking down
at the table, solved against data/xarm6/scene.xml as shipped -- i.e.
with link_base on the 0.12m pedestal, the same convention XARM7_SIM_HOME
already follows.

Known limitation: ManipulationModule FK places link_base at the world
origin while the scene spawns it 0.12m up, so scanned poses read low in
Z and pick-derived IK targets are unreachable. Scan/detect/recall work;
pick does not. Tracked as the next rung.
@mustafab0
mustafab0 force-pushed the mustafa/sim/l1-worldbelief-sim-blueprint branch from ec3d6c7 to b3b95d7 Compare July 30, 2026 00:06
@github-actions github-actions Bot added ready-to-merge Required CI checks have passed on this PR and removed ready-to-merge Required CI checks have passed on this PR labels Jul 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-to-merge Required CI checks have passed on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant