fix(e2e): remove stale eagle VL waives

[Moviw]

Background

tests/e2e/waives.txt skipped two models for a 404 Hugging Face repo:

eagle-embed-vl-1b-v2     SKIP   (HF repo 404)
eagle-rerank-vl-1b-v2    SKIP   (HF repo 404)

No manifest declares either name. The eagle_vlm manifests are
nemotron-embed-vl-1b-v2 and nemotron-rerank-vl-1b-v2 (plus their -tp4
variants), pointing at nvidia/llama-nemotron-embed-vl-1b-v2 and
nvidia/llama-nemotron-rerank-vl-1b-v2. Both are listed 🟢 Green in
website/data/model-support-matrix.md. The repo is reachable and the waived
names are gone, so both the subjects and the reason are stale.

_load_waives matches purely by model name and skips any line it cannot
parse, so these entries have been silently inert rather than failing — which is
exactly why they rotted unnoticed. If either name were ever reused, the waive
would spring back and skip a Green model with no one asking for it.

Precedent: 19316ed9 fix(e2e): remove stale Qwen TorchTRT waive.

No originating issue; found while surveying eagle_vlm coverage.

Exit Criteria

• tests/e2e/waives.txt contains no entry naming a model with no declared
  manifest.
• A CPU test reports the next such entry instead of letting it rot.
• Existing _load_waives platform-selection coverage is unchanged.

Non-goals: no change to _load_waives itself, to the per-family
waives.txt files, or to any manifest.

Implementation

Remove the two stale entries. Add
test_every_waive_names_a_declared_manifest to the existing
tests/tools/test_e2e_waives.py: every waived name must resolve to a manifest
declared through the E2E harness's own iter_manifest_paths(), collected
across every platform prefix the file mentions so platform-scoped waives are
also checked.

The guard reuses the harness loader rather than globbing, so it follows the
same indexed/flat/nested manifest layouts the harness supports.

Affected components: tests/e2e/waives.txt, tests/tools/test_e2e_waives.py.
No public API, ABI, bundle-format, or dependency change.

Change categories

• Model or runtime behavior
• Public API
• ABI
• Bundle or artifact format
• Dependencies
• Documentation only
• CI or developer tooling

Validation

Commands and Results

Test-first. With the guard added and tests/e2e/waives.txt still at
upstream/main, it reproduces exactly the two stale names:

$ git checkout upstream/main -- tests/e2e/waives.txt
$ PYTHONPATH=python:. pytest tests/tools/test_e2e_waives.py -q
E   assert not ['eagle-embed-vl-1b-v2', 'eagle-rerank-vl-1b-v2']
FAILED tests/tools/test_e2e_waives.py::test_every_waive_names_a_declared_manifest
1 failed, 2 passed

After removing the two entries:

$ PYTHONPATH=python:. pytest tests/tools/test_e2e_waives.py -v
tests/tools/test_e2e_waives.py::test_load_waives_filters_with_explicit_platform PASSED
tests/tools/test_e2e_waives.py::test_load_waives_without_platform_ignores_prefixed_entries PASSED
tests/tools/test_e2e_waives.py::test_every_waive_names_a_declared_manifest PASSED
3 passed

The two pre-existing tests in this file are unchanged; the diff on it is
purely additive (0 deleted lines).

Full CPU tests/tools, on this branch and on upstream/main, using the same
marker set and the same deselect the workflow applies for filesystems without
reflink support:

$ PYTHONPATH=python:. pytest tests/tools -q \
    -m "not gpu and not trt and not e2e and not model_proof_allocator" \
    --deselect tests/tools/test_model_proof_runner.py::test_distinct_explicit_hf_cache_paths_reach_both_containers

upstream/main:  2986 passed, 22 deselected
this branch:    2987 passed, 22 deselected

The single extra pass is the new guard. No failures on either side.

The two public gates that run without a GPU:

$ python3 -m tools.community_ci source-quality --base upstream/main
[ccm] PASS
All checks passed!            # ruff on tests/tools/test_e2e_waives.py
160 passed in 18.72s          # model architecture contracts
exit 0

$ python3 -m tools.community_ci impact --base upstream/main
{
  "changed_paths": ["tests/e2e/waives.txt", "tests/tools/test_e2e_waives.py"],
  "classifications": ["platform", "unit_tests"],
  "run_unit_tests": true,
  "unit_scope": "all"
}
exit 0

Evidence for the staleness claim:

$ ls tests/e2e/models/eagle_vlm/manifests/
nemotron-embed-vl-1b-v2-tp4.json   nemotron-embed-vl-1b-v2.json
nemotron-rerank-vl-1b-v2-tp4.json  nemotron-rerank-vl-1b-v2.json
$ grep -i "nemotron-embed-vl\|nemotron-rerank-vl" website/data/model-support-matrix.md
| `nvidia/llama-nemotron-embed-vl-1b-v2`  | `nemotron-embed-vl-1b-v2`  | `FP16` | None | — | 🟢 Green |
| `nvidia/llama-nemotron-rerank-vl-1b-v2` | `nemotron-rerank-vl-1b-v2` | `FP16` | None | — | 🟢 Green |

Consistency checks:

$ PYTHONPATH=python:. python3 tools/test_impact.py --validate
Validation passed. 229 models, 13 core, 89 families.
$ PYTHONPATH=python:. python3 tools/legal_headers.py
legal headers: tracked=6226 managed=5321 excepted=4 ignored=901 changed=0 findings=0
$ git diff --check

Unit evidence only. No inference, parity, performance, or qualification claim.

Hardware, Environment, and Revisions

• Repository head: e9f45793 (upstream/main), branch rebased onto it.
• Host: Ubuntu 22.04.5, Python 3.13.9, pytest 8.4.2 and ruff 0.16.4 from
  requirements/community-ci.txt, plus numpy, PyYAML, torch 2.5.1+cu121,
  safetensors, transformers, tensorrt 11.2.1.2 and cuda-python so the full
  tests/tools set collects.
• No GPU execution: this change is data and a Python test.

Not Run / Remaining Gaps

• No E2E run: removing a waive for a name nothing declares cannot change E2E
  selection, since _load_waives never matched it.
• The Community CPU / Docs job was not run locally; this change touches
  nothing under website/.
• The unit stage was not run through tools.community_ci unit --scope all,
  which requires Docker; Docker on this host needs privileges the contributor
  account does not have. The same test set was run directly instead, as
  recorded above.
• The guard covers the repo-wide tests/e2e/waives.txt only. The five
  per-family waives.txt files are loaded by each family's own runner.py
  and are currently header-only; extending the guard to them is model-owned
  work and is left to those owners.

Notes For Future Readers

The silent-continue behavior in every waive loader is the root of this: a
waive that stops matching stops doing anything, and nothing reports it. The
guard turns the next occurrence into a CPU failure instead of a slow rot.

Promoting the loader itself to reject unparseable lines would be a stronger
fix, but it changes _load_waives behavior for all callers and belongs in a
separate change.

Risk level

• Low
• Medium
• High

Deletes two entries that provably matched nothing, and adds one test. No
production code touched.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 5225b658-496a-4887-98d8-abb9649597cb

📥 Commits

Reviewing files that changed from the base of the PR and between c95a8d8 and 5399cbd.

📒 Files selected for processing (1)

• tests/tools/test_e2e_waives.py

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.

---

📝 Summary

Summary

• Removed stale waivers for eagle-embed-vl-1b-v2 and eagle-rerank-vl-1b-v2.
• Added test_every_waive_names_a_declared_manifest().
• The test checks unprefixed and platform-prefixed waiver names against manifests discovered by iter_manifest_paths().
• Reported validation passed for the three waiver tests and 2,987 CPU tool tests.

Architecture impact

• Family-owned files: tests/e2e/waives.txt and tests/tools/test_e2e_waives.py.
• Shared surfaces: E2E waiver validation and existing manifest discovery through iter_manifest_paths().
• Dependency directions: No new dependencies. The test uses the existing E2E harness.
• Affected consumers: E2E waiver maintenance and CPU tool-test validation.
• Unresolved blast-radius questions: The test validates manifest declaration only. It does not validate waiver behavior or test coverage for each waiver.

HUMAN REVIEW REQUIRED — Confirm the name-resolution correction and trigger CI for the current PR head before merge.

Walkthrough

The change removes two obsolete E2E waivers and adds validation that every remaining waiver references a declared E2E test case across supported platforms.

Changes

E2E waiver validation

Layer / File(s)	Summary
Waiver-to-manifest validation tests/tools/test_e2e_waives.py, tests/e2e/waives.txt	The test discovers declared E2E case names, loads unprefixed and platform-prefixed waivers, and reports orphaned waiver names. The waiver file removes entries for eagle-embed-vl-1b-v2 and eagle-rerank-vl-1b-v2.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: ⚪ Minimal · up to 5399c

This change removes obsolete E2E waivers and adds validation that remaining waiver names map to declared test cases. No current merge-blocking risk remains.

Suggested reviewers: yifeif-nv

🚥 Pre-merge checks | ✅ 8 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 25.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 4 functions across 1 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (8 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely identifies the primary change: removing stale Eagle VL E2E waivers.
Description check	✅ Passed	The description follows the repository template and provides complete background, exit criteria, implementation details, change classification, validation results, environment information, remaining g…
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Family Ownership Boundary	✅ Passed	PASS. The pull request changes only the central waiver file and the central tests/tools/test_e2e_waives.py test. The new import at line 11 uses the shared tests.e2e_harness.manifest_loader API. Li…
Shared Semantic Neutrality	✅ Passed	PASS. The diff changes only tests/e2e/waives.txt and tests/tools/test_e2e_waives.py. The added test makes one model-agnostic repository check: it compares waiver names returned by the existing `_l…
Benchmark Validation Integrity	✅ Passed	PASS — The committed diff changes only tests/e2e/waives.txt and tests/tools/test_e2e_waives.py. It removes two waiver records and adds a namespace-consistency test that compares _load_waives() n…
Shared Change Blast Radius	✅ Passed	The PR changes a shared E2E waiver validation surface, but it documents the required blast radius. The description identifies the model-agnostic need: stale names in the centralized waiver file become…

Full details: Description check

Explanation

The description follows the repository template and provides complete background, exit criteria, implementation details, change classification, validation results, environment information, remaining gaps, notes, and risk rationale.

Full details: Family Ownership Boundary

Explanation

PASS. The pull request changes only the central waiver file and the central tests/tools/test_e2e_waives.py test. The new import at line 11 uses the shared tests.e2e_harness.manifest_loader API. Lines 58 and 71–89 read shared manifest and waiver data; they do not import or reuse any family-owned implementation, fixture, comparator, reference, or validation artifact. The removed entries at tests/e2e/waives.txt do not add a family dependency. No family directory, family registry, switch, source list, or strategy map changes in the pull-request diff.

Full details: Shared Semantic Neutrality

Explanation

PASS. The diff changes only tests/e2e/waives.txt and tests/tools/test_e2e_waives.py. The added test makes one model-agnostic repository check: it compares waiver names returned by the existing _load_waives contract with case names from the existing manifest loader for each mentioned platform. It adds no model, family, tensor, topology, runtime, reference, metric, threshold, dataset, or preprocessing semantics. The removed model names are confined to the E2E waiver data file.

Full details: Benchmark Validation Integrity

Explanation

PASS — The committed diff changes only tests/e2e/waives.txt and tests/tools/test_e2e_waives.py. It removes two waiver records and adds a namespace-consistency test that compares _load_waives() names with get_case_names(). It does not add or alter benchmark, performance, metric, workload, timing, or implementation-comparison paths. _load_waives() remains unchanged. No compared paths exist with different timed regions, asymmetric synchronization or transfer, task-unit accounting, evidence aggregation, or output validation. Therefore no explicit failure condition in this check is introduced.

Full details: Shared Change Blast Radius

Explanation

The PR changes a shared E2E waiver validation surface, but it documents the required blast radius. The description identifies the model-agnostic need: stale names in the centralized waiver file become silently inert and may later match a case. Repository evidence shows that tests/test_e2e.py loads this shared file, passes waivers to tests/e2e_harness/model_runner.py, and applies them by child case.name across discovered manifests. The description identifies the affected platform-prefixed and unprefixed consumers, states that _load_waives and existing platform tests remain unchanged, and provides CPU and quality validation results. The centralized file cannot be checked only by an eagle_vlm owner because it is consumed by the shared E2E entrypoint; the repository also distinguishes separate model-local waiver files. The diff adds only the cross-manifest guard and removes two inert entries. No explicit shared-change blast-radius failure condition is present.

---

_{Comment @coderabbitai help to get the list of available commands.}

[Moviw]

@yifeif-nv This PR is ready for CI. Please trigger CI for the current head.

[yifeif-nv]

Thanks for contribution @Moviw there is some minor issue with the name resolution. Please help to correct that and I'm happy to trigger the internal CI for you! Looking forward to your future contribution

[Moviw]

@yifeif-nv Thanks — you were right, and it turned out to be a real defect rather than wording.

The runtime resolves waives against case.name, and 50 of the 272 case names are not manifest names (bark-small-tts-probe01, canary-1b-v2-asr-probe01, act-aloha-sim-transfer-cube-recorded-episode-0-frame-0, ...), so the guard would have falsely reported every one of those as silently inert. A guard against waives rotting that itself checked the wrong namespace was not much of a guard.

Fixed as suggested in 5399cbd4: _declared_case_names() now returns set(get_case_names()), with the import, call site, docstring and assertion updated to match. Mutation-checked — reverting the source back to manifest names makes the test fail as expected.

Public CI is green (7/7) on the current head. Ready for the internal CI whenever you have a slot.