Skip to content

Harden replay repository operations - #3188

Closed
xiaoyaoqilan wants to merge 15 commits into
dimensionalOS:mainfrom
xiaoyaoqilan:codex/replay-repository-hardening
Closed

Harden replay repository operations#3188
xiaoyaoqilan wants to merge 15 commits into
dimensionalOS:mainfrom
xiaoyaoqilan:codex/replay-repository-hardening

Conversation

@xiaoyaoqilan

Copy link
Copy Markdown
Contributor

Summary

  • require TLS 1.2+ when TLS is enabled
  • add per-object and per-repository quotas
  • expose Prometheus-compatible operational metrics
  • support CDN download base URLs
  • audit upload operations
  • recover interrupted .part uploads safely on service start
  • document production deployment and operational settings

Motivation

This is PR 6 of the #3158 implementation stack. It adds the bounded resource use, observability, recovery, and CDN integration required to move the validated MVP toward a reusable service.

Stack dependency

Validation

  • quota, TLS configuration, metrics, CDN URL, audit, and interrupted-upload recovery paths are covered by tests
  • full six-PR targeted suite: 56 passed
  • ruff check and ruff format --check passed
  • targeted mypy checks passed

@codecov

codecov Bot commented Jul 25, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 79.85005% with 430 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
dimos/protocol/pubsub/impl/webrtc/mvp_cli.py 16.94% 195 Missing and 1 partial ⚠️
...s/protocol/pubsub/impl/webrtc/replay_repository.py 80.74% 67 Missing and 42 partials ⚠️
...rotocol/pubsub/impl/webrtc/replay_repository_s3.py 74.64% 23 Missing and 13 partials ⚠️
dimos/protocol/pubsub/impl/webrtc/replay_nodes.py 76.19% 19 Missing and 16 partials ⚠️
dimos/memory2/remote_replay.py 79.46% 15 Missing and 8 partials ⚠️
dimos/memory2/remote_dataset.py 87.36% 6 Missing and 6 partials ⚠️
...otocol/pubsub/impl/webrtc/replay_repository_cli.py 86.36% 6 Missing and 3 partials ⚠️
dimos/protocol/pubsub/impl/webrtc/mvp_metrics.py 88.46% 3 Missing and 3 partials ⚠️
...os/protocol/pubsub/impl/webrtc/providers/broker.py 80.00% 1 Missing and 1 partial ⚠️
.../pubsub/impl/webrtc/test_memory2_remote_dataset.py 98.97% 1 Missing ⚠️
... and 1 more
@@            Coverage Diff             @@
##             main    #3188      +/-   ##
==========================================
- Coverage   73.77%   73.71%   -0.07%     
==========================================
  Files        1109     1125      +16     
  Lines      104383   108053    +3670     
  Branches     9567    10147     +580     
==========================================
+ Hits        77013    79648    +2635     
- Misses      24695    25591     +896     
- Partials     2675     2814     +139     
Flag Coverage Δ
OS-ubuntu-24.04-arm 68.23% <79.85%> (+0.52%) ⬆️
OS-ubuntu-latest 70.24% <79.85%> (+0.47%) ⬆️
Py-3.10 70.24% <79.85%> (+0.47%) ⬆️
Py-3.11 70.24% <79.85%> (+0.47%) ⬆️
Py-3.12 70.24% <79.85%> (+0.47%) ⬆️
Py-3.13 70.24% <79.85%> (+0.47%) ⬆️
Py-3.14 70.25% <79.85%> (+0.47%) ⬆️
Py-3.14t ?

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
dimos/core/global_config.py 88.63% <100.00%> (+5.68%) ⬆️
dimos/core/test_global_config.py 100.00% <100.00%> (ø)
dimos/memory2/cli/app.py 84.21% <100.00%> (+14.21%) ⬆️
...otocol/pubsub/impl/webrtc/providers/video_track.py 24.39% <ø> (-2.28%) ⬇️
...os/protocol/pubsub/impl/webrtc/test_mvp_metrics.py 100.00% <100.00%> (ø)
.../protocol/pubsub/impl/webrtc/test_remote_replay.py 100.00% <100.00%> (ø)
...s/protocol/pubsub/impl/webrtc/test_replay_nodes.py 100.00% <100.00%> (ø)
...tocol/pubsub/impl/webrtc/test_replay_repository.py 100.00% <100.00%> (ø)
...l/pubsub/impl/webrtc/test_replay_repository_cli.py 100.00% <100.00%> (ø)
...ol/pubsub/impl/webrtc/test_replay_repository_s3.py 100.00% <100.00%> (ø)
... and 12 more

... and 31 files with indirect coverage changes

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copy link
Copy Markdown
Contributor Author

Public end-to-end validation (2026-07-26)

The six-PR stack is now deployed on an Alibaba Cloud node in Beijing and has been validated from an external Windows client, not only through server-local requests.

  • Public health check: GET http://39.106.170.53:8765/healthz returned HTTP 200 with node cn-beijing, region china.
  • H.264 object: 640x360, 24 FPS, 2 seconds, 16,763 bytes. External download SHA-256 matched object id 22d3353f196c7ebca06979e03dc68e952d0b92f5162fbe105dc5fdab8655c8c7.
  • Official Go2 memory2 dataset: 229,855,232 bytes. External download completed in about 60 seconds and SHA-256 matched object id 32c2971034e449136a9196a2ec50dadd046c04a2783cf56513e2b613f92af0bd.
  • SQLite PRAGMA integrity_check: ok.
  • Streams discovered from _streams: color_image (855 rows), color_image_embedded (108), lidar (461), and odom (1122).
  • Public object pages:
  • The memory2 upload path created a WAL-consistent snapshot plus JSON index; the remote resolver downloaded, content-hash verified, SQLite-validated, and cached the dataset successfully.

CI follow-up was also pushed: project logging now uses setup_logger(), and Typer/Rich CLI error assertions normalize ANSI styling. The targeted replay suite passes (56 passed), Ruff passes, and scoped mypy passes.

The repository/data path is therefore validated end to end. Running the complete dimos --replay run unitree-go2 navigation graph remains a robot/developer-host test because the repository VM is intentionally a small storage service rather than a full Open3D/navigation runtime.

@xiaoyaoqilan

Copy link
Copy Markdown
Contributor Author

Superseded by #3201. The implementation is consolidated into the dimos/hosted_data package and unified dimos data CLI.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant