Skip to content

feat(agents): add Jupyter-backed code policy module - #3259

Open
TomCC7 wants to merge 6 commits into
mainfrom
cc/feat/code-as-policy-interface
Open

feat(agents): add Jupyter-backed code policy module#3259
TomCC7 wants to merge 6 commits into
mainfrom
cc/feat/code-as-policy-interface

Conversation

@TomCC7

@TomCC7 TomCC7 commented Jul 29, 2026

Copy link
Copy Markdown
Member

Contribution path

Problem

DimOS agents can call predefined skills, but they cannot submit a Python program that processes native observations, branches or retries, and composes deployed RPCs as one synchronous policy rollout.

Solution

  • Add a persistent CodePolicyModule exposing one synchronous python_exec(code, timeout_s) MCP skill.
  • Use the standard jupyter_client + ipykernel stack instead of a custom worker protocol.
  • Lazily bootstrap the parent PR's connected Dimos app handle and a Memory2 SqliteStore attached to the explicitly configured recorder database.
  • Preserve the Jupyter namespace across calls and ordinary Python errors.
  • On timeout, interrupt first and preserve the namespace when the kernel recovers; restart only an unresponsive or dead kernel.
  • Define a private, pure Recorder beside the xArm simulation-agent blueprint. It records the enabled joint-state and color-image streams to a stable database path without changing Memory2 or perception.
  • Add soft prompt guidance that prefers code execution for observation processing and multi-RPC control flow while retaining direct skills for atomic actions.
  • Use standard DimOS structured logs and Jupyter execution identity for submitted source, output, duration, timeouts, interrupts, and restarts.

ipykernel and jupyter-client are explicit dependencies of the agents extra and are imported lazily. The v1 interface is foreground-only, text-only, single-call-at-a-time, and intentionally not a security sandbox. Real-hardware blueprints are unchanged.

How to Test

Install and run the xArm simulation:

uv sync --extra agents --extra manipulation
uv run dimos --simulation --viewer none run xarm-perception-sim-agent \
  -o pickandplacemodule.visualization.backend=none

From another terminal:

uv run dimos mcp call python_exec --json-args \
  '{"code":"print(memory.list_streams()); print(app.skills.get_robot_state())"}'

Focused automated validation:

uv run pytest \
  dimos/agents/test_code_policy.py \
  dimos/codebase_checks/test_no_all.py \
  dimos/robot/manipulators/xarm/blueprints/test_agentic.py \
  dimos/robot/test_all_blueprints_generation.py

The focused suite passes 19 tests. Ruff, strict mypy over the changed source files, blueprint registry generation, staged pre-commit hooks, and git diff --check also pass. A live MCP smoke test verified that python_exec can read the latest joint state and color image from the active Recorder database and query bounded observation history. Earlier validation covered DimOS RPC calls, namespace persistence, and interrupt-first timeout recovery.

AI assistance

OpenAI Codex with GPT-5 was substantially involved in design discussion, implementation, tests, documentation, and validation. The author reviewed and approved the direction interactively.

Checklist

  • I have read and approved the CLA.

@codecov

codecov Bot commented Jul 29, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 88.95184% with 39 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
dimos/agents/code_policy.py 80.88% 31 Missing and 8 partials ⚠️
@@            Coverage Diff             @@
##             main    #3259      +/-   ##
==========================================
+ Coverage   75.20%   75.26%   +0.06%     
==========================================
  Files        1128     1131       +3     
  Lines      107954   108307     +353     
  Branches     9751     9771      +20     
==========================================
+ Hits        81188    81521     +333     
- Misses      23955    23965      +10     
- Partials     2811     2821      +10     
Flag Coverage Δ
OS-ubuntu-24.04-arm 69.10% <88.95%> (+0.24%) ⬆️
OS-ubuntu-latest 71.01% <88.95%> (+0.09%) ⬆️
Py-3.10 71.01% <88.95%> (+0.08%) ⬆️
Py-3.11 71.01% <88.95%> (+0.09%) ⬆️
Py-3.12 71.01% <88.95%> (+0.09%) ⬆️
Py-3.13 71.01% <88.95%> (+0.09%) ⬆️
Py-3.14 71.02% <88.95%> (+0.09%) ⬆️
Py-3.14t 71.01% <88.95%> (+0.09%) ⬆️
SelfHosted-Large 29.16% <27.19%> (+0.07%) ⬆️
SelfHosted-Linux 35.62% <27.19%> (-0.04%) ⬇️
SelfHosted-macOS 34.66% <27.19%> (-0.06%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
dimos/agents/test_code_policy.py 100.00% <100.00%> (ø)
dimos/robot/all_blueprints.py 100.00% <ø> (ø)
dimos/robot/manipulators/common/agent_prompts.py 100.00% <100.00%> (ø)
...imos/robot/manipulators/xarm/blueprints/agentic.py 100.00% <100.00%> (ø)
...robot/manipulators/xarm/blueprints/test_agentic.py 100.00% <100.00%> (ø)
dimos/agents/code_policy.py 80.88% <80.88%> (ø)

... and 3 files with indirect coverage changes

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Comment thread dimos/agents/code_policy/policy_kernel.py Outdated
Comment thread dimos/agents/code_policy/policy_kernel.py Outdated
Comment thread dimos/agents/code_policy/policy_kernel.py Outdated
Comment thread dimos/agents/code_policy/policy_kernel.py Outdated
Comment thread dimos/agents/code_policy/policy_kernel.py Outdated
Comment thread dimos/agents/code_policy/policy_kernel.py Outdated
Comment thread examples/code_policy_xarm_sim.sh Outdated
@TomCC7 TomCC7 changed the title feat(agents): add persistent code policy kernel feat(agents): add Jupyter-backed code policy module Jul 29, 2026
@TomCC7
TomCC7 marked this pull request as ready for review July 29, 2026 03:46
Comment thread dimos/perception/detection/type/detection3d/object.py
Comment thread dimos/perception/manipulation_policy_recorder.py Outdated
@TomCC7

TomCC7 commented Jul 29, 2026

Copy link
Copy Markdown
Member Author

cleaned up, can check again @leshy

@TomCC7
TomCC7 requested a review from leshy July 29, 2026 20:19
Base automatically changed from cc/feat/repl-module to main July 29, 2026 22:45
@mintlify

mintlify Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
dimensional 🟢 Ready View Preview Jul 30, 2026, 1:01 AM

💡 Tip: Enable Workflows to automatically generate PRs for you.

@TomCC7

TomCC7 commented Jul 30, 2026

Copy link
Copy Markdown
Member Author

@mintlify

mintlify Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
dimensional 🟡 Building Jul 30, 2026, 1:00 AM

💡 Tip: Enable Workflows to automatically generate PRs for you.

@github-actions github-actions Bot added the ready-to-merge Required CI checks have passed on this PR label Jul 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-to-merge Required CI checks have passed on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants