Skip to content

Add Cosmos3-Edge variant of the LIBERO-10 action-policy SFT recipe - #278

Open
filipemartinsubrobotics wants to merge 2 commits into
NVIDIA:mainfrom
ubrobotics-ai:libero-edge-recipe
Open

filipemartinsubrobotics wants to merge 2 commits into
NVIDIA:mainfrom
ubrobotics-ai:libero-edge-recipe

Conversation

@filipemartinsubrobotics

@filipemartinsubrobotics filipemartinsubrobotics commented Oct 2, 2026 •

Copy link
Copy Markdown

What

action_policy_libero_edge: the libero_10 action-policy SFT recipe on the public
nvidia/Cosmos3-Edge base. It mirrors action_policy_libero_nano and differs only in

  • the model config (EDGE_MODEL_CONFIG, dense Nemotron-2B-Dense-VL backbone), and
  • one extra trainable key, k_norm_und_for_gen, as in vision_sft_edge.

Adds the experiment, a run TOML and launcher mirroring Nano preset A (HSDP 2x8,
global batch 2048), and a docs section covering the one-node setting
(replicate 1, grad_accum 2) and running outside the Docker image.

Validation

  • 50-iteration single-node smoke run, 8x RTX PRO 6000 Blackwell 96 GB, global batch 2048:
    ~79 s/iteration, loss 15.3 -> 3.6, DCP checkpoint saved.

  • Full 2000-iteration run of this recipe (same experiment, lr, schedule and global batch 2048)
    on 4x RTX PRO 6000 Blackwell 96 GB: shard 4, replicate 1, grad_accum 4; ~220-310 s/iteration,
    final loss 0.90.

  • Closed-loop libero_10 eval of iter_000002000, following the docs (section 3): server with
    --action-normalization quantile_rot, the bundled libero_native_frame_wise_relative_rot6d.json
    stats, --raw-action-dim 10 --fps 20, 30 sampling steps; client --camera agentview,wrist --image_size 256 --action_space frame_wise_relative --rotation_space 6d --action_dim 10,
    50 trials per task, 8 envs, seed 0.

    Success rate: 407/500 = 81.4 %

Closed-loop LIBERO-10 rollouts (final checkpoint iter_000002000)

libero10_cosmos3_edge_iter2000.mp4

One episode per libero_10 task, agent-view camera, instruction shown on top. The 81.4 % above comes from the separate 500-episode run (50 trials per task). Full page: https://huggingface.co/spaces/ubr-physical-ai/cosmos3-edge-libero10

Task Instruction Success
0 put both the alphabet soup and the tomato sauce in the basket 17/50
1 put both the cream cheese box and the butter in the basket 49/50
2 turn on the stove and put the moka pot on it 49/50
3 put the black bowl in the bottom drawer of the cabinet and close it 49/50
4 put the white mug on the left plate and put the yellow and white mug on the right plate 44/50
5 pick up the book and place it in the back compartment of the caddy 46/50
6 put the white mug on the plate and put the chocolate pudding to the right of the plate 39/50
7 put both the alphabet soup and the cream cheese box in the basket 46/50
8 put both moka pots on the stove 25/50
9 put the yellow and white mug in the microwave and close it 43/50

Notes for running outside the container

torchcodec needs NPP (nvidia/cu13/lib on LD_LIBRARY_PATH) and an FFmpeg with an AV1
decoder (dav1d) for LIBERO's AV1 videos; opencv-python's bundled FFmpeg lacks it.
For the LIBERO sim venv, egl-probe (pulled in by robomimic) fails to build with CMake 4;
CMAKE_POLICY_VERSION_MINIMUM=3.5 fixes it.

action_policy_libero_edge is action_policy_libero_nano on the public
nvidia/Cosmos3-Edge base. It differs only in the model config
(EDGE_MODEL_CONFIG, the dense Nemotron-2B-Dense-VL backbone) and one extra
trainable key, k_norm_und_for_gen, which trains with the generation pathway
as in vision_sft_edge.

Adds the experiment, a run TOML and launcher mirroring the Nano preset A
(HSDP 2x8, global batch 2048), and a docs section with the one-node setting
(replicate 1, grad_accum 2) and notes for running outside the Docker image
(NPP and an AV1-capable FFmpeg for torchcodec).

Validated with a 50-iteration single-node smoke run on 8x RTX PRO 6000
Blackwell: about 79 s/iteration at global batch 2048, loss 15.3 -> 3.6,
DCP checkpoint saved. A full 2000-iteration run and the closed-loop libero_10
success rate have not been measured for Edge yet.

Signed-off-by: Filipe Martins <293984334+filipemartinsubrobotics@users.noreply.github.com>
Signed-off-by: Filipe Martins <293984334+filipemartinsubrobotics@users.noreply.github.com>
@filipemartinsubrobotics
filipemartinsubrobotics marked this pull request as ready for review October 2, 2026 10:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant