feat(sim): add xarm6-worldbelief-sim blueprint - #3242
Open
mustafab0 wants to merge 1 commit into
Open
Conversation
mustafab0
requested review from
Dreamsorcerer,
leshy,
paul-nechifor and
spomichter
as code owners
July 28, 2026 05:56
Codecov Report✅ All modified and coverable lines are covered by tests. @@ Coverage Diff @@
## main #3242 +/- ##
==========================================
+ Coverage 75.20% 75.22% +0.01%
==========================================
Files 1128 1130 +2
Lines 107954 108020 +66
Branches 9751 9752 +1
==========================================
+ Hits 81188 81254 +66
- Misses 23955 23956 +1
+ Partials 2811 2810 -1
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 2 files with indirect coverage changes 🚀 New features to boost your workflow:
|
WorldBelief stack against MuJoCo: mirrors xarm6_worldbelief with the RealSense replaced by MujocoSimModule (wrist_camera, base_frame_id=link6, dof=6, XARM6_SIM_PATH), giving the perception/agent stack a first-class dev surface that needs no arm on the desk. The rerun bridge, recorder, belief module, and MCP server are identical for both camera sources, so they move into a shared worldbelief_stack() factory parameterized by state subdir; xarm6_worldbelief is unchanged apart from calling it. XARM6_SIM_HOME puts wrist_camera at world (0.45, 0, 0.52) looking down at the table, solved against data/xarm6/scene.xml as shipped -- i.e. with link_base on the 0.12m pedestal, the same convention XARM7_SIM_HOME already follows. Known limitation: ManipulationModule FK places link_base at the world origin while the scene spawns it 0.12m up, so scanned poses read low in Z and pick-derived IK targets are unreachable. Scan/detect/recall work; pick does not. Tracked as the next rung.
mustafab0
force-pushed
the
mustafa/sim/l1-worldbelief-sim-blueprint
branch
from
July 30, 2026 00:06
ec3d6c7 to
b3b95d7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The WorldBelief perception stack can only be exercised with an xArm6 on the desk. This makes the sim a first-class dev surface:
dimos run xarm6-worldbelief-simbrings up the same perception/planning/MCP stack against MuJoCo, so scan, recall, TF, gripper, and the agent path can be developed and regression-tested without hardware.What it wires
xarm6-worldbelief-simmirrorsxarm6-worldbeliefwith the RealSense wrist camera replaced byMujocoSimModule(wrist_camera,base_frame_id=link6,dof=6,XARM6_SIM_PATH). It renders the MJCF camera and publishes the same color/depth/camera-info topics pluslink6 -> wrist_camera_*_optical_framefrom sim ground truth;world -> link6comes from ManipulationModule FK.worldbelief_stack(state_subdir)factory.xarm6_worldbeliefis otherwise untouched — verified by diffing every atom's module + kwargs before/after (only function object addresses differ).XARM6_SIM_HOMEputswrist_cameraat world (0.45, 0, 0.52) looking straight down. Solved againstdata/xarm6/scene.xmlas shipped, i.e. withlink_baseon the 0.12 m pedestal — the same conventionXARM7_SIM_HOMEalready follows (verified: main's xarm7 home lands the camera at 0.44991, 0, 0.51979).make_xarm6_sim_hardware/make_xarm6_sim_module_kwargsmirror the existing xarm7 factories.No changes outside blueprint, config, registration, and tests.
Coordination with #3275 (worldbelief → experimental)
#3275 (approved) relegates worldbelief to
dimos/experimental/world_belief/, including movingxarm/blueprints/worldbelief.py→experimental/world_belief/xarm6_blueprint.py.By design this PR stays green against
mainand does not pre-empt those paths.worldbelief_sim.pyintentionally stays in the xarm package. Once #3275 lands, this blueprint needs exactly one import line changed:plus the same substitution for the two lazy imports in
test_worldbelief_sim_blueprint.py. Note #3275 renamesworldbelief.pyat 96% similarity, so its move and this PR's edits to that file will need a merge resolution whichever order they land in.Rebase notes (onto #3183)
Rebased onto current main. Two API changes absorbed:
dimos/robot/manipulators/test_blueprints.pywas deleted by feat: extract manipulation plan execution manager #3183, so the blueprint-composition tests moved to a scopeddimos/robot/manipulators/xarm/test_worldbelief_sim_blueprint.pyrather than resurrecting the deleted file.RobotModelConfig.coordinator_task_namewas removed by feat: extract manipulation plan execution manager #3183; the test now derives the expected task name fromtrajectory_task_name(robot.name).xarm6-planner-onlywas dropped fromSELF_HOSTED_BLUEPRINTSon main; that removal is preserved.Validation
Re-run after the rebase, so these reflect #3183's new plan-execution manager.
move_to_joints→ "Reached target joint configuration"; 51-point trajectory completes; arm converges toward target (0.530 → 0.546 → 0.558 → 0.6)go_home/move_to_joints/get_robot_state/scanall invokeMcpClientdeploys and discovers all 14 tools over MCPDetected
red ballvs MJCF ground truth (apple at 0.40, 0.08, 0.17):Known issues (the PR ladder)
Documented, not fixed — each is its own rung.
data/xarm6/scene.xmlspawnslink_baseatpos="0 0 .12"(plus a pedestal geom), butmake_xarm_model_configplacesbase_poseat the world origin. The sim renders from the true camera pose while perception unprojects using the FK pose. Measured −9.54 cm above, matching prediction. Scan/detect/recall work; pick does not. Next rung.hand.xml's driver joint grows as the jaws close; DiMOS positions grow as they open. Needsgripper_position_invertedsupport inMujocoSimModule, which main does not have.scan/recallexceed the 120 s default RPC timeout. Server-side the skill succeeds (SKILL scan result=OK duration_ms=156804.7) but the MCP layer reports an error at 120 s. Aggravated by (5).Failed to initialize NVML: Driver/library version mismatch (NVML 595.84), sotorch.cuda.is_available()is False. Environment, not code, but it is what pushes scan past the RPC limit.dimos mcp callhardcodes a 30 s HTTP read timeout, below the scan's runtime even on the happy path.move_to_joints/go_homereport success while the arm is still travelling; sampling immediately after the call shows it near its start. Repeatedgo_homecalls converge (joint 0: 0.417 → 0.021 → 0.013 → 0.008).["red ball", "orange ball", "cup"]only the red ball is detected; the orange and cup are missed.Not verified
The agent's natural-language exchange could not complete: the available OpenAI key is rejected with 401
invalid_api_key(confirmed with a directcurltoapi.openai.com, so it is the credential, not the plumbing). Everything up to the LLM call works — the agent deploys, connects, and discovers all 14 MCP tools.