test: [2/3] add L2 Gym-to-RL rollout acceptance - #4014
Conversation
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
|
/ok to test 0341f3b |
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
|
/ok to test e4a9d932dc447d48f6c88c725405c8c4d393d9dd |
@yaoyu-33, there was an error processing your request: See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/ |
|
/ok to test e4a9d93 |
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
yaoyu-33
changed the title
|
/ok to test d5251d9 |
yaoyu-33
changed the title
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
|
/ok to test 85e137e |
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
|
/ok to test bce2879 |
|
Targeted L2 CI for the reviewed Ray-serializable-tokenizer head: https://github.com/NVIDIA-NeMo/RL/actions/runs/33984301832 |
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
|
/ok to test 5533d83 |
|
The previous targeted run passed the first six acceptance environments and exposed a test-harness limit on workplace_assistant's 90,056-token continuation. Reviewed fix raises the CI-only byte-token limit to 131,072. Rerun: https://github.com/NVIDIA-NeMo/RL/actions/runs/33989903365 |
|
Optimized L2 rerun reuses the compatible image from the prior successful build while mounting head 5533d83; the redundant rebuild run was cancelled. Active run: https://github.com/NVIDIA-NeMo/RL/actions/runs/33989961341 |
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
|
/ok to test 4368454 |
|
All 7 L2 environment rollouts passed in the prior run; its only error was the shared actor's strict 10-second teardown. Reviewed fix allows 60 seconds (15 subprocesses have a ~45-second internal worst-case bound), still fails on a real shutdown hang, and always releases the Ray actor. Rerun: https://github.com/NVIDIA-NeMo/RL/actions/runs/33991595090 |
|
Latest targeted CI is green: https://github.com/NVIDIA-NeMo/RL/actions/runs/33991595090
Keeping this PR in Draft for review. |
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
|
/ok to test 8954ff2 |
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
|
/ok to test b77e034 |
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
|
/ok to test 220faac |
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
|
/ok to test |
@yaoyu-33, there was an error processing your request: See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/1/ |
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
|
/ok to test b897b7a |
|
/ok to test 94a595d |
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
|
/ok to test b8588ae |
Two conflicts, both pure adjacency -- independent additions landing at the same file position, so both sides are kept: - nemo_rl/algorithms/single_controller.py: this branch's _SUPERVISOR_DRAIN_TIMEOUT_S constant sits where #3924 (periodic rollout checkpointing) added _MAX_CONSECUTIVE_ROLLOUT_CHECKPOINT_FAILURES and the _RolloutCheckpointCut dataclass. Neither refers to the other. - tests/unit/environments/test_nemo_gym_utils.py: #4014 (L2 Gym-to-RL rollout acceptance) appended a spinup-cleanup test to the same end-of-file this branch appended TestUnresolvedAgentRefsAreDiagnosable to. main's function first, this branch's class after it. The other three commits (#3730 colocated MInf, #4002 actor venv list from a Python leaf module, #4048 partial rerun routing) merged clean. Verified after the merge: the restart keys are still in both SC exemplars, on_dead_shard is still absent from the config and both yamls, and 1287 tests pass across single_controller, fleet_health, weight_sync and the gym utils. Signed-off-by: asolergibert <asolergibert@nvidia.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
## Purpose Add focused L1 standalone environment/verifier acceptance coverage for three environments in the active RL training mix whose prior tests were empty or helper-only. ## Coverage - `jailbreak_detection`: validates all five checked-in rows, standard and Nemotron verdicts, combined reward, all five production policy mappings, both reward polarities for all six policy verifiers, final-verdict precedence, empty responses, and judge failures - `over_refusal_detection`: validates all five checked-in rows and all represented categories, complied/refused/unclear goldens, standard and Nemotron verdicts, final-verdict precedence, empty responses, and judge failures - `multichallenge`: validates all five checked-in rows and represented challenge shapes, production-config judge behavior, full verification, context construction, rubric aggregation, negative pass criteria, empty rubrics, and judge failures All external judge calls are mocked at the service boundary. The tests exercise real checked-in data schemas and verifier reward logic on CPU without adding test-specific APIs to production code. ## Validation - local focused tests: 33 jailbreak + 11 over-refusal + 12 MultiChallenge = 56 passed - scoped pre-commit: passed - Ruff and `git diff --check`: passed - multiple independent review/fix rounds; closure review found no P0-P2 issues - latest full Gym CI matrix passed: core unit tests and all 8/8 environment server shards ## Three-PR test stack 1. this PR - L1 standalone Gym environment and verifier behavior 2. NVIDIA-NeMo/RL#4014 - L2 Gym-to-RL rollout acceptance 3. NVIDIA-NeMo/RL#4015 - L3 Nano 4B Gym-backed training E2E Draft: the author will review before this is ready for merge. --------- Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
3 participants
Purpose
Add the L2 acceptance gate for bumping NeMo Gym inside NeMo RL: representative Gym data must complete a rollout through real Gym services and the RL consumer boundary.
Coverage
Seven representative environments are covered, each with accepted and rejected behavior (14 rollout paths):
math_with_judgecode_genmcqaequivalence_llm_judgesingle_step_tool_use_with_argument_comparisonstructured_outputs_v4workplace_assistantEach case uses one SHA256-pinned Gym example and a deterministic CPU OpenAI-compatible model endpoint. The real path is
NemoGymDataset -> nemo_gym_data_processor/collate -> Gym services -> run_nemo_gym_rollout_sync -> RL postprocess.Assertions cover positive/zero reward goldens, verifier-derived outputs, preservation of source request parameters plus RL generation overrides, exact prompt/generation token IDs and logprobs, policy-versus-judge request contracts, agent routing, tool schemas and outputs, and multi-turn token/history continuity.
The PR also hardens actor lifecycle behavior: failed Gym startup or tokenizer installation attempts graceful shutdown with a bounded 120-second wait, always attempts
ray.kill, and preserves the original startup exception even if cleanup fails. Dedicated unit tests cover_spinup,set_tokenizer,shutdown.remote, shutdownray.get, andray.killfailures.L0in file and job names is the explicitly selected RL CI tier because the rollout acceptance test is CPU-only. Conceptually this is L2 in the three-PR series because it verifies the Gym-to-RL integration boundary. No smoke-specific abstraction is added to production code.Validation
Current head:
b897b7a3eae1826e8c269876007820e9f27b0a03L0_Unit_Tests_Environments: 128 passed, 1 deselected; all startup and cleanup failure cases passed in https://github.com/NVIDIA-NeMo/RL/actions/runs/34016740347L0_Unit_Tests_Nemo_Gym: 89 passed, 13 skipped, 2 deselected; all 14 accepted/rejected rollout paths passed in https://github.com/NVIDIA-NeMo/RL/actions/runs/34017650139git diff --check, and DCO sign-off: passedLocal macOS pytest is unavailable because the lockfile supports Linux x86_64/aarch64; Linux CI is authoritative. EOS launchers and clean detached worktrees are prepared, but exact-head local execution currently requires pull access or an approved mirror for the private CI ECR image; older NGC images were rejected before tests because they use Python 3.13.13 while current main requires 3.13.14.
Three-PR series
Draft: the author will review before this is ready for merge.