feat(telemetry)!: instrument the single-controller path and propagate traces by rrs45 · Pull Request #4052 · NVIDIA-NeMo/RL

rrs45 · 2026-09-08T21:22:12Z

What does this PR do ?

Extends NeMo-RL's OpenTelemetry/nemo-lens layer to cover the async single-controller path, propagates trace context across Ray actor boundaries so a run reads as one trace, and adopts nemo-lens's SpanRegistry and consumer-driven metric registry.

Before this, the sync GRPO path was instrumented and everything else was not. A recent run's 1,200 data-plane spans and the entire NeMo-Gym leg were unreachable from the job trace, because Ray does not carry OTel context across an actor boundary and every worker span started its own trace.

What lands here:

Single-controller instrumentation — 13 rl.sc.* spans covering the actor's phases (step, logprobs, value inference, advantage, training, optimizer step, checkpointing) plus the per-prompt dispatch span. The run lives inside SingleControllerActor, so that is the process that opens the job span.
Trace-context propagation — a dispatch_with_trace_context / @accepts_trace_context pair wired through the TQPolicy, TQValue and teacher presharded entrypoints and NemoGym.run_rollouts, plus aiohttp client instrumentation so the HTTP hop into the Gym service keeps the trace. Ray validates remote signatures after inspect.unwrap, which hides a wrapper's **kwargs, so the decorator also advertises the carrier on __signature__ — Ray hits the same problem with its own _ray_trace_ctx and fixes it the same way.
Startup spans — rl.startup over init_ray() and setup(), with rl.setup.<phase> beneath, so the startup phases arrive as one waterfall instead of unrelated root traces.
vLLM engine metrics — the per-step Prometheus read is widened from the spec-decode family to the engine's token, sequence-length and request-outcome series, at no extra RPC. Absent series are omitted rather than reported as 0.0, so a vLLM rename leaves a gap in the dashboard instead of a plausible-looking zero.
Optional vLLM native tracing (telemetry.vllm_native_tracing, default false) — one span per request, so it is a debugging tool you switch on for a few steps, not something to leave on.
nv.dl.campaign.stage="RL" on every process, so a backend collecting several stages of a model's lifecycle (pretrain → SFT → RL) can select this stage without matching on service names.

Breaking change

TelemetryConfig loses export_strategy, export_rank, export_sample_rate and sampler_enabled, following nemo-lens deleting the rank-gating machinery they drove. Every process that enables telemetry now exports and labels itself by rank. Narrowing a fleet down is now a collector-side filter on nv.dl.rank, or telemetry.enabled: false on the ranks that should stay quiet.

A stale key in an existing YAML still parses (TelemetryConfig allows extras) and is not projected into the worker environment, so it reaches nothing rather than erroring. Migration note in docs/observability/configuration.md.

Issues

None.

Usage

Off by default. Enable per run:

telemetry:
  enabled: true
  exporter: otlp
  span_groups: per_step          # or `all`, or an explicit `per_step,per_prompt`

export OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:4318
uv run examples/run_grpo_single_controller.py --config examples/configs/grpo_math_1B.yaml

Then filter by the nemo.run.id resource attribute in your backend to isolate the run. Adding a span to a new phase:

from nemo_rl.telemetry.instrumentation import managed_span, umbrella_span
from nemo_rl.telemetry.span_groups import RLSpanGroup

# A leaf span, bucketed for goodput accounting.
with managed_span(RLSpanGroup.POLICY_UPDATE, "rl.grpo.policy_training", tracer=_tracer):
    ...

# An umbrella: carries trace shape, no `rl.bucket`. Use whenever a span can
# overlap another instance of itself, since concurrent spans sum past the wall
# clock they happened in.
with umbrella_span(RLSpanGroup.U_PER_PROMPT, "rl.sc.generate_and_push", tracer=_tracer):
    ...

See docs/observability/ — configuration, span groups, metrics, extending, and vLLM tracing.

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

On the unit-test box: the suite could not run locally — uv run --group test pytest refuses on macOS arm64 because the lockfile's supported environments are Linux-only. Locally verified instead: ruff check and ruff format clean on all changed files, py_compile on all 54 changed Python files, and the seven test_source_drift.py guards run green (by putting the pinned nemo-lens src tree on sys.path). The rest of the telemetry suite is on CI.

Additional Information

Test coverage. ~2,600 lines of new unit tests across tests/unit/telemetry/, tests/unit/data_plane/, tests/unit/models/generation/, tests/unit/distributed/ and tests/unit/environments/. test_source_drift.py is worth a look during review: it parses the sources and fails the build when a declaration drifts from the call sites that use it — every emitted span name must be documented, every registered span group must have an emitter, every efficiency timer must be declared, every @accepts_trace_context method must be dispatched with a carrier, and every teed logger key must be emitted somewhere. Those are the failure modes nothing at runtime can catch, since a missing metric is indistinguishable from a step that did not report one.

Dependency. nemo-lens moves to rev b0f977d for SpanRegistry and the metric registry, via [tool.uv.sources] plus a [tool.uv] override-dependencies entry — megatron-core pins v0.2.0 in its own sources and the workspace resolves a single nemo-lens, so without the override the two git URLs are a hard conflict. Drop the override once Megatron-LM bumps to the same or newer rev. uv.lock changes are 52 insertions: the nemo-lens rev bump plus the three OpenTelemetry aiohttp packages the extra pulls in, and nothing else.

Known gaps, documented rather than hidden:

async_ppo_train in ppo.py is still uninstrumented (timer-only). Pre-existing, untouched here.
On the async single-controller path the productive generation itself has no span, so generation is absent from goodput attribution there. Explained in docs/observability/span-groups.md.
With token_capture finalizers configured, the single controller's per-prompt dispatch commits through the finalizer pool rather than generate_and_push, and that branch opens no rl.sc.generate_and_push span. It landed on main after this branch was written; recorded in the coverage-gap table and left for a follow-up.
idle/validation is idle as a metric and overhead on the spans covering the same seconds. Both are true of different fleets; attributing it properly needs per-fleet accounting.

Performance. Every instrumentation site gates on its span group before allocating. The data-plane wrapper — the most frequent call site in the repo, once per prompt on the rollout path — checks is_span_group_enabled before building the span name or the attribute dict, so a run with telemetry off pays nothing.

copy-pr-bot · 2026-09-08T21:22:17Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

rrs45 · 2026-09-08T22:07:01Z

/ok to test 493d853

rrs45 · 2026-09-08T23:49:06Z

/ok to test de7ab13

rrs45 · 2026-09-08T23:59:23Z

/ok to test de7ab13

rrs45 · 2026-09-09T17:35:29Z

/ok to test bec17bd

rrs45 · 2026-09-09T18:17:46Z

/ok to test 130756c

rrs45 · 2026-09-10T04:51:35Z

/ok to test 45c2002

rrs45 · 2026-09-10T14:29:07Z

/ok to test 7bf922c

rrs45 · 2026-09-10T15:38:36Z

/ok to test 88cb61d

rrs45 · 2026-09-10T20:45:01Z

/ok to test 246b047

rrs45 · 2026-09-10T20:54:23Z

/ok to test 246b047

rrs45 · 2026-09-23T19:36:29Z

/ok to test e2235e6 CI:Lfast

rrs45 · 2026-09-23T19:37:39Z

/ok to test e2235e6 CI:Lfast

rrs45 · 2026-09-23T21:04:36Z

/ok to test e2235e6

terrykong

Thanks for turning this around so quickly. I checked each reply against the code at e2235e6, and ran the fixes where I could (the Gym cancel path, the telemetry-only wrapper, the stage per algorithm, the counter filter, the real-Ray tests).

Of the 29 threads, 21 are fixed; I replied with the commit and resolved them. The other 8 have a follow-up reply in this review: 7 are partly fixed, and 1 is fixed with a small docs leftover.

Before this can merge:

Four unit tests fail at this head (three in test_metrics.py, one in test_vllm_utils.py). CI has not shown them because the unit lanes have not run yet.
CI Lint fails on the return type of streaming_umbrella_span.
The new Ray test's fixture shuts down the shared session cluster.
metric_utils.py's shared key constants are not used by the producers yet (reply on the _TRAIN_SCALARS thread).

In the PR description: the nemo-lens rev is still given as b71263c (the pin is b0f977d).

Generated by Claude Code

Signed-off-by: Raj Singh <rajsin@nvidia.com>

… traces Open job, step, and rollout spans on the single-controller path, add startup and vLLM engine metrics, and carry W3C trace context across the Ray hops into the policy, value, and NeMo-Gym workers so one training step reads as a single trace. Signed-off-by: Raj Singh <rajsin@nvidia.com>

Signed-off-by: Raj Singh <rajsin@nvidia.com>

…lity stack Signed-off-by: Raj Singh <rajsin@nvidia.com>

startup_span opened a span and yielded nothing, so `with startup_span() as span:` bound None even with the setup group enabled. It now yields the span, matching setup_span. Also from review: - shutdown_telemetry narrows its except to (ImportError, AttributeError) and warns instead of logging at debug, matching ensure_metric_group_registered. - Annotate tracer as Optional[Tracer] on the span helpers, type the trace_fn / umbrella_trace_fn decorators via the existing _F, widen safe_set_span_attributes to dict[str, Any], and spell _metric_specs' real return type. - SingleControllerActor uses the handle init_telemetry_worker returns rather than discarding it and re-reading the global, matching trajectory_collector. Signed-off-by: Raj Singh <rajsin@nvidia.com>

Moves both the `[tool.uv.sources]` pin and the matching `override-dependencies` entry from b71263cc to b0f977d4, picking up two upstream commits: - `feat: Add single trace id for the entire job run based on config` (#61) adds `NemoLensConfig.single_trace_id` (default False, opt in via `ENABLE_SINGLE_TRACEID`) and a `DeterministicTraceIdGenerator`. The provider only selects it when the flag is set and a job seed exists, so our runs keep the existing `SeedIndependentIdGenerator`. - `refactor(semconv): organize attribute definitions and resource mappings` (#66) splits `semconv.py` into a package. `semconv/__init__.py` star-imports `attributes.py`, so `NV_DL_RANK` / `NV_DL_WORLD_SIZE` stay importable from `nemo.lens.semconv` as before. No module we import was touched other than those, so every lens call site keeps its signature. Relocked with the uv version pinned in docker/Dockerfile (0.11.28) to keep the lockfile at `revision = 3`; the re-resolution also consolidates gitpython onto 3.1.62 for the sglang extra combination, which previously resolved to 3.1.59. Signed-off-by: Raj Singh <rajsin@nvidia.com>

The single-controller train pump retries every 5ms while the replay buffer is empty, and wrapped each sleep in its own `rl.idle.buffer_starvation` span. A startup stall therefore arrived as thousands of near-identical spans -- ~10.5k for a minute of waiting -- which buries the trace it was meant to explain. Async GRPO shares the category name but polls at 0.5s, so it needs no coalescing and keeps wrapping its bare sleep. `start_efficiency_span` returns a started span the caller ends by hand, since a wait that spans loop iterations cannot be a `with`. It pointedly does not attach the span to the context: held open across `await` points, a current span is copied into every task created during the wait, which would reparent unrelated rollout work under an idle span. It returns None when the group is off, so the `is not None` guards a hand-managed span needs are also the telemetry-off no-op. The pump opens the span on the first starved poll, closes it when a batch becomes selectable, and closes it again after the loop for the two exits that continue the run (the `break` on a target that fell to what is already dispatched, and the condition going false mid-wait). The exits that end the run are deliberately uncovered, since dropping the span costs nothing there; a try/finally would have re-indented 383 lines of loop body to say the same thing. Widening the span past the sleeps is only sound because a *starved* poll never reaches the data plane: evict() returns 0 before calling remove() when nothing is stale, and select() gives up in _finalize_selection before claiming. So the span still has no bucketed children for a rollup summing by rl.bucket to count twice. The new rl.idle.polls attribute is what keeps the wider duration readable -- ten seconds is two thousand clean polls or two hundred whose selection ran long, opposite diagnoses. The bucket and category rules move into a helper shared with efficiency_span so the unbucketed-category list cannot drift between the two ways of opening one of these spans. Signed-off-by: Raj Singh <rajsin@nvidia.com>

The stub restated the adapter's constructor keywords, so the `checkpointing` argument the factory now passes made every test using it fail on an unexpected keyword. Which keywords reach the adapter is already checked against the adapter itself, so the stub takes whatever it is given instead of keeping a copy that goes stale. Signed-off-by: Raj Singh <rajsin@nvidia.com>

Move the teed rows out of nemo_rl/telemetry/metrics.py and next to the code that produces their keys, so a rename touches one place: the training scalars to algorithms/metric_utils.py, kl_penalty/approx_entropy to the loss that computes them, critic/loss to ppo.py, and the vllm/* family to a torch-free module beside compute_engine_step_metrics. Telemetry keeps the tee and the registration. nemo_rl/telemetry/vocabulary.py holds what both sides need and imports nothing, which is what lets a torch-heavy owner declare a row and lets telemetry stay importable without the training stack. The registry key is derived from the series name rather than typed a second time, and the init/total category, the timing/setup prefix and the setup field-naming rule each get a single home. ALL_GROUPS and UMBRELLA_GROUPS are read off RLSpanGroup instead of restated, so a group added to the class cannot go unregistered. Signed-off-by: Raj Singh <rajsin@nvidia.com>

…ollout path - accepts_trace_context attached the caller's context across the generator's yield. Ray abandons a cancelled streaming call without closing the generator, so the token was detached later by the garbage collector in a different context and OTel logged 'Failed to detach context' twice per cancelled shard. Attach around each __anext__ instead, and give NemoGym the same shape through streaming_umbrella_span. - rl.gym.run_rollouts is dispatched once per prompt by the single controller, so it now takes U_PER_PROMPT there and keeps U_ROLLOUT on the sync batch path, restoring per_step's promise to scale with steps. - telemetry alone installed the data-plane wrapper, and is_metrics_client answered on type, so observability.enabled: false still paid for per-worker stat polling and a payload walk per put. - nv.dl.campaign.stage was RL for every process, SFT runs included. - the vLLM step-metric reads swallowed RayActorError, so the single controller's handlers never fired; the two methods also shared one warn key. - the carrier-refusal warning keyed on a repr for .options() handles, so it repeated per dispatch and never named the method. Signed-off-by: Raj Singh <rajsin@nvidia.com>

…histogram - rl.sc.generate_and_push built its attribute dict before checking whether PER_PROMPT was on, once per prompt. Gate first and fall back to a shared NO_SPAN, now exported from instrumentation so the data plane and the single controller hold one instance. per_prompt_scope() is still entered either way, since the data-plane put inside reads it. - the batch generate() wall clock went to gen_ai.server.request.duration, which semconv defines per inference request; one call here covers a batch across every data-parallel shard. It gets rl.vllm.batch.duration instead, declared next to the code that records it. - startup in run_grpo_single_controller ran outside the try, so a setup failure skipped shutdown_telemetry() and dropped the buffered spans that would have explained it. - run_grpo's startup timer labels did not match their span names. - docs: adding a metric now means declaring it next to its producer; drop the claims that telemetry.enabled is per-rank, that lens names the rl.* attributes, and that the engine series need _total aliases. Signed-off-by: Raj Singh <rajsin@nvidia.com>

…passes Each algorithm's validate() repeated the same four things: resolve a tracer, open an U_EVALUATE umbrella, build the span name, and scope the enclosed generation to OVERHEAD because validation tokens are scored and discarded. Three of the six had already dropped the bucket scope, so their validation passes read as goodput. Folding the pairing into the helper makes that property part of what an eval span *is*. Signed-off-by: Raj Singh <rajsin@nvidia.com>

…M gaps - the carrier tests all ran against fakes, which cannot catch the thing that actually broke: Ray validates .remote() arguments on the caller, against a signature it gets by unwrapping. Put the methods on a real actor, across all three shapes plus an .options() handle, and assert each runs under the caller's trace. - a counter that went backwards, which is what a mid-step engine restart looks like. - the engine series names are copies of vLLM's; check them against vLLM's own declarations so a rename fails a test rather than dropping a dashboard line. Signed-off-by: Raj Singh <rajsin@nvidia.com>

Nineteen comment blocks ran five lines or longer inside a function body, several of them twelve to nineteen. Keep the constraint each one records and drop the argument around it; the durable rationale for the span groups already lives in the RLSpanGroup docstrings, which is where a reader looking for it will go. Signed-off-by: Raj Singh <rajsin@nvidia.com>

Use the metric key constants at the sites that build the logged dicts instead of retyping the strings, across grpo, grpo_sync, the single controller utils and the two policy workers. Enter per_prompt_scope() around the token-capture finalizer dispatch, so the gym and data-plane spans under that branch are gated on PER_PROMPT like the sibling branch already is. Widen the streaming umbrella helper's yield to ContextManager[Any]: use_span yields the Span, which failed pyrefly's invalid-yield check. Add vocabulary.py and metric_names.py to pyrefly.toml. Both type-check clean, so the CI whitelist step fails while they are missing from it. Guard the four tests that build a lens handle with requires_lens, and drop test_run_window_categories_match_the_efficiency_summary: utils.py now imports the category set rather than restating it, so there is no second copy left to drift. Reword the docs and the drift-guard docstring that called rl.vllm.generate worker-side; it is driver-side on the sync path, and the async path has no generation span yet. Signed-off-by: Raj Singh <rajsin@nvidia.com>

rrs45 · 2026-09-24T00:48:20Z

/ok to test 5b4217c

rrs45 · 2026-09-24T01:39:37Z

Thanks for the detailed pass — the reproductions and the suggested one-liners made this quick to act on.

All 13 open threads are fixed in 5b4217c (rebased onto ebf2f3adb). I replied on each thread individually, but 12 of the 13 are now outdated because the fixes moved the anchored lines, so GitHub collapses them behind Show outdated in Files changed. Summarising here so the state is visible without expanding them.

The five merge blockers

1. Four failing unit tests. All fixed.

In test_metrics.py, test_map_teed_scalars_reads_training_and_vllm_keys and test_every_declared_scalar_has_a_unique_key_and_name now call a _import_teed_metric_owners() helper first, so they name the owning modules rather than inheriting whatever an earlier test imported, and the second iterates teed_metrics() instead of the removed _TEED_SCALARS. The helper uses pytest.importorskip to match the convention of the other tests in that file.

test_run_window_categories_match_the_efficiency_summary I deleted rather than repaired. Its premise was that print_efficiency_summary keeps its own copy of the category set, but algorithms/utils.py now imports the single definition from vocabulary.py, so there is no second copy left to drift. It was also AST-scanning utils.py for a constant that no longer appears in that file, so it could not have passed in any form. The shared algorithms_utils_categories conftest helper is still used by two other tests and stays.

In test_vllm_utils.py, the stale *_total test is deleted, and I added test_a_counter_that_went_backwards_is_omitted_rather_than_negative to cover the filter that replaced those aliases.

2. CI Lint. Fixed, and green on this head: run 35940079731. The yield is ContextManager[Any], with _no_span_activation's return widened to match so both arms agree. I reproduced invalid-yield with pyrefly@0.24.2 before and after to confirm.

Worth flagging: fixing that exposed the step immediately after it. telemetry/vocabulary.py and vllm/metric_names.py are both new, both type-check clean, and neither was in pyrefly.toml, so the "zero errors but not in whitelist" check would have failed as soon as lint got past pre-commit. Both are added in the same commit. I checked every file this PR touches for the same trap; those two were the only ones.

3. The Ray fixture. The module-scoped ray_cluster fixture is deleted and its argument dropped from all five tests, which now rely on the session-scoped autouse init_ray_cluster.

4. metric_utils.py key constants. The constants stay where they are, and the producers now use them at all seven sites: grpo.py (three dicts plus MEAN_GEN_TOKENS_PER_SAMPLE_KEY on both sides of the assignment), grpo_sync.py, single_controller_utils/utils.py, and the two "lr" writes in dtensor_policy_worker_v2.py and megatron_policy_worker.py. Same shape as loss_functions.py and ppo.py.

5. PR description rev. Corrected to b0f977d.

The other eight threads

The docs and comment leftovers are all done: vllm-tracing.md now says two RPCs per step; extending.md L224 says the constant goes on RLSpanGroup and there is nothing to add it to; both surviving copies of the "worker-side rl.vllm.generate" claim (in span-groups.md and the test_source_drift.py docstring) now say driver-side on the sync path, with no generation span yet on the async path.

The token-capture branch enters per_prompt_scope() around the finalizer dispatch, scope only with no span, in the form you gave — so the gym and data-plane spans under it are gated on PER_PROMPT like the normal dispatch.

test_engine_metric_names_match_vllm now also asserts OK_FINISH_REASONS <= set(FINISH_REASON_STRINGS), so a reason that disappears upstream fails the test instead of being counted as failed forever.

Registration now closes: ensure_metric_group_registered calls freeze_metrics() on the success path, and both register functions check first, so a late row raises a RuntimeError naming it rather than being recorded against a key lens never saw. The freeze call sits outside the try body to keep it minimal. Covered by a new tests/unit/telemetry/test_vocabulary.py with 13 tests.

On the comment-length thread I took all nine suggested one-liners as written, then re-ran an AST scan restricted to PR-added lines (comment runs over 4 lines whose first line is inside a function body, intersected with the diff against the merge-base). It found four more the table had not listed — grpo.py L4849, single_controller.py L3293, setup.py L267, test_factory.py L113 — which I also trimmed. The scan now reports zero.

One more fix, not from your review

Four tests this PR added in test_instrumentation.py build a lens handle but were missing the @requires_lens guard that every sibling carries, so they hard-fail instead of skipping where nemo-lens is absent. Added.

Verification

Locally: ruff 0.9.9 clean on check, isort and format; taplo clean on the edited TOML; uv lock --check in sync; the seven test_source_drift.py guards and the 13 new test_vocabulary.py tests green.

The macOS limitation from the description still applies — the lockfile is Linux-only, so anything needing a real lens handle, ray or tensordict cannot run here. On CI, 20 jobs have passed with no failures, including Lint, both coverage checks and four functional lanes. The unit lanes are queued behind Docs_Tests and have not started, so the three test fixes above are not yet confirmed by CI.

`_stream_rows` now passes `per_prompt=in_per_prompt_scope()` when it dispatches `run_rollouts`, so the fakes standing in for that actor method reject the call with a TypeError. Seven doubles across four files take the kwarg with the same `False` default the real method has. The eighth, in test_run_async_nemo_gym_rollout_streams_complete_prompt_groups, is left alone: it stands in for the sharded dispatch in `rollouts.py`, which passes its arguments positionally and never sends `per_prompt`. Only one of these failed CI, because `addopts` carries `-x` and the run stopped at the first failure. The other three files had not been reached. Signed-off-by: Raj Singh <rajsin@nvidia.com>

rrs45 · 2026-09-24T02:35:33Z

/ok to test b3552d8

Brings #4052 (single-controller telemetry; moves nemo-lens to b0f977d and adds the aiohttp extra), #4252 (reward-model environment expected score, which fixes `L0_Unit_Tests_Environments` on the previous merge), #4248, #4234, #4254 and #4171. Only uv.lock conflicted: regenerated from this branch's lock; nemo-lens moves to main's rev and the three opentelemetry-instrumentation packages its aiohttp extra needs are added. Nothing else moved, and nothing resolves below main except the deliberate numpy / llguidance / flash-attn pins. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>

rrs45 requested review from a team as code owners September 8, 2026 21:22

github-actions Bot added the Documentation Improvements or additions to documentation label Sep 8, 2026

rrs45 changed the title ~~feat(telemetry)!: instrument the single-controller path and propagate trace context~~ feat(telemetry)!: instrument the single-controller path and propagate traces Sep 8, 2026

rrs45 added the CI:L1 Run doctests, unit tests, and functional tests label Sep 8, 2026

github-actions Bot removed the CI:L1 Run doctests, unit tests, and functional tests label Sep 8, 2026

rrs45 added the CI:L1 Run doctests, unit tests, and functional tests label Sep 8, 2026

rrs45 added CI:L2 Run doctests, unit tests, functional tests, and convergence tests and removed CI:L1 Run doctests, unit tests, and functional tests labels Sep 10, 2026

rrs45 added CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) and removed CI:L2 Run doctests, unit tests, functional tests, and convergence tests labels Sep 10, 2026

terrykong reviewed Sep 24, 2026

View reviewed changes

rrs45 added 15 commits September 23, 2026 17:46

build(telemetry): move nemo-lens to the rev with the metric registry

373fbfe

Signed-off-by: Raj Singh <rajsin@nvidia.com>

test(telemetry): cover span groups, trace propagation, and source drift

6ece26a

Signed-off-by: Raj Singh <rajsin@nvidia.com>

docs(observability): document span groups, metrics, and the observabi…

5a2fd03

…lity stack Signed-off-by: Raj Singh <rajsin@nvidia.com>

rrs45 force-pushed the raj/lens-sc branch from e2235e6 to 5b4217c Compare September 24, 2026 00:48

terrykong approved these changes Sep 24, 2026

View reviewed changes

terrykong merged commit f99036d into main Sep 24, 2026
96 checks passed

terrykong deleted the raj/lens-sc branch September 24, 2026 05:12

yfw mentioned this pull request Sep 24, 2026

build: bump vllm to 0.29.0 #4080

Open

terrykong mentioned this pull request Sep 25, 2026

feat: track Bridge-preferred MFU in SFT v2 and single-controller #4268

Open

Conversation

rrs45 commented Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do ?

Breaking change

Issues

Usage

Before your PR is "Ready for review"

Additional Information

Uh oh!

copy-pr-bot Bot commented Sep 8, 2026

Uh oh!

rrs45 commented Sep 8, 2026

Uh oh!

rrs45 commented Sep 8, 2026

Uh oh!

rrs45 commented Sep 8, 2026

Uh oh!

rrs45 commented Sep 9, 2026

Uh oh!

rrs45 commented Sep 9, 2026

Uh oh!

rrs45 commented Sep 10, 2026

Uh oh!

rrs45 commented Sep 10, 2026

Uh oh!

rrs45 commented Sep 10, 2026

Uh oh!

rrs45 commented Sep 10, 2026

Uh oh!

rrs45 commented Sep 10, 2026

Uh oh!

rrs45 commented Sep 23, 2026

Uh oh!

rrs45 commented Sep 23, 2026

Uh oh!

rrs45 commented Sep 23, 2026

Uh oh!

terrykong left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rrs45 commented Sep 24, 2026

Uh oh!

rrs45 commented Sep 24, 2026

The five merge blockers

The other eight threads

One more fix, not from your review

Verification

Uh oh!

rrs45 commented Sep 24, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rrs45 commented Sep 8, 2026 •

edited

Loading