fix(perf): align BERT reference timing scope

[chaofengw-nv]

Background

The BERT embedding task reference prepared tokenizer inputs before its timed
operation while the release suite declared end-to-end pipeline timing. The
runtime timing guard therefore rejected bert.embed before collecting a valid
baseline or candidate comparison.

The same hf-transformers-embedding adapter is also used by Eagle VLM, whose
declared boundary intentionally excludes input preparation. The fix must
preserve both family contracts.

Exit Criteria

• BERT embedding reference timing includes tokenization and device input
  preparation and matches the declared pipeline contract.
• Eagle VLM embedding continues to prepare inputs outside its model-call timing
  boundary.
• Model loading and warmup remain excluded from reported latency.
• A regression test covers both family contracts and their invocation
  boundaries.

Implementation

• Resolve the authoritative task-reference timing contract for the selected
  model family inside the shared embedding adapter.
• Prepare inputs lazily inside each BERT invocation when the contract includes
  input preparation; otherwise prepare them once outside the timed invocation.
• Populate the session timing metadata from that same contract.
• Add CPU-safe regression coverage using fake external framework boundaries.

There are no public API, ABI, artifact format, dependency, compatibility,
migration, or rollout changes.

Change categories

• Model or runtime behavior
• Public API
• ABI
• Bundle or artifact format
• Dependencies
• Documentation only
• CI or developer tooling

Validation

Commands and Results

• env PYTHONPATH=/tmp/trtmc-bert-perf-timing/python python3 -m pytest -q tests/tools/test_perf_matrix.py tests/e2e/models/bert/test_bert_embedding_runner.py: 149 passed.
• python3 -m ruff check benchmarks/performance/baselines/task_reference.py tests/tools/test_perf_matrix.py: passed.
• git diff --check: passed.
• Claude SDK/DevToolkit GB300 campaign for all-minilm-l6-v2: container
  environment ready, native build succeeded, and check, hf_preflight,
  preview, and formal all passed without retries or recoveries. The single
  bert.embed case was green.

Hardware, Environment, and Revisions

• Repository head: 6ca45e757bdda1302902b4677e61c9a95d540fba.
• GPU: NVIDIA GB300; Linux aarch64; container image digest
  sha256:2ec7d8b70819c36d290225d5c56a4b3d6c9ff250d0f4115c32b674a14fe84fd6.
• CUDA 13.0; TensorRT 11.0.0.114; PyTorch 2.12.0+cu130; Transformers 5.2.0.
• Model: sentence-transformers/all-MiniLM-L6-v2, resolved revision
  1110a243fdf4706b3f48f1d95db1a4f5529b4d41.
• Reference FP32 and candidate FP16, 3 warmups and 10 measured samples each.
  Reference recorded task-pipeline-call-wall with input preparation included;
  candidate recorded public_pipeline_call_wall with input preparation
  included. Both sample sets were stable. P50 latency was 3.4004 ms reference
  and 0.3420 ms candidate.

Not Run / Remaining Gaps

• Eagle VLM was not run on GPU for this change. Its existing model-only timing
  behavior is preserved and covered by the focused CPU regression test.
• Other GPU platforms and embedding models were not run because the change is
  limited to the shared reference timing boundary and the GB300 BERT case
  exercises the corrected path.

Notes For Future Readers

• This does not change the release YAML, timing contracts, performance
  thresholds, model selection, or comparison criteria.
• The GB300 model engine was a cache hit; DevToolkit still produced a native
  worker from the exact repository head above.
• Review the family-aware input preparation boundary first, then the
  parameterized regression test.

Risk level

• Low
• Medium
• High

The change is isolated to reference timing for one shared adapter, derives its
behavior from the existing contract source, and is covered for both consumers.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 54a3bc7b-fdc4-4e5f-b206-0eae86b7142f

📥 Commits

Reviewing files that changed from the base of the PR and between ead6801 and 6ca45e7.

📒 Files selected for processing (2)

• benchmarks/performance/baselines/task_reference.py
• tests/tools/test_perf_matrix.py

Included review availability: Your plan provides up to 12 included reviews per hour; 10 remain after this review.

---

📝 Summary

Summary

The BERT embedding reference now measures the declared end-to-end pipeline operation. Each invocation includes tokenizer input preparation and device input preparation. Model loading, tokenizer loading, and warmup remain outside reported latency. The reference reports timing as task-pipeline-call-wall.

The PR adds CPU-safe regression coverage for timing metadata and invocation boundaries. The tests use fake Torch and Transformers boundaries to verify that session creation performs no pipeline work and that invocation performs tokenization before model inference.

Validation reports 148 passing tests, successful Ruff checks, and matching reference and candidate pipeline timing on GB300.

Architecture impact

• Family-owned files: benchmarks/performance/baselines/task_reference.py owns BERT embedding reference timing behavior. tests/tools/test_perf_matrix.py owns its regression coverage.
• Changed shared surfaces: The Session invocation boundary and timing metadata now represent the full repeatable pipeline call. Consumers that compare or display reference timing must continue to accept task-pipeline-call-wall.
• Dependency directions: The reference keeps model and tokenizer loading outside the timed operation. Each invocation now performs tokenizer and device input preparation before model inference through the existing Torch and Transformers boundaries.
• Affected consumers: Performance-matrix execution and reporting or comparison paths that read reference timing metadata.
• Unresolved blast-radius questions: Confirm that all downstream timing consumers interpret task-pipeline-call-wall as the intended end-to-end metric. Confirm that no consumer depends on inputs being prepared during session creation.

Status: HUMAN REVIEW REQUIRED

Walkthrough

The embedding reference now prepares tokenizer inputs according to the timing contract and propagates contract settings to the session. Tests cover BERT and Eagle VLM timing scopes, call order, metadata, and embedding output.

Changes

Embedding timing

Layer / File(s)	Summary
Invocation measurement and validation benchmarks/performance/baselines/task_reference.py, tests/tools/test_perf_matrix.py	The reference prepares inputs per invocation when preprocessing is measured and prepares them during loading otherwise. The session adopts timing-contract settings. Tests verify both timing modes, call order, metadata, asset-loading behavior, and one embedding vector.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: ⚪ Minimal · up to 6ca45

Embedding reference timing now follows declared input-preparation boundaries while keeping asset loading and warmup outside measurement. Coverage validates the affected BERT and Eagle VLM behaviors, with no current merge-blocking risk identified.

Suggested reviewers: zhenshanx-nv

🚥 Pre-merge checks | ✅ 7 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 23 functions across 2 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Benchmark Validation Integrity	⚠️ Warning	The PR enables the BERT comparison, but the two timed paths still measure different work. The candidate path calls pipeline.embed inside the public-pipeline timer (`examples/trtmc_benchmark_worker.c…	Make the reference and candidate use the same returned-output boundary. For the current candidate contract, materialize the reference embedding on the host inside the timed invoke and perform the same finite-output validation inside that …

✅ Passed checks (7 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Family Ownership Boundary	✅ Passed	No family-ownership violation is introduced. The changed loader at task_reference.py:1200-1240 uses the shared timing_contract contract. That contract is model-agnostic and was already imported an…
Shared Semantic Neutrality	✅ Passed	The changed production code is generic. _load_embedding uses the existing timing_contract(runner="task-reference", family=...) contract. It does not add a BERT or Eagle conditional, model tensor r…
Shared Change Blast Radius	✅ Passed	The shared-change rationale is identifiable from the description and repository. The changed hf-transformers-embedding loader serves both bert.embed and eagle_vlm.embed, while `timing_contract()…
Title check	✅ Passed	The title is concise and clearly identifies the primary change: aligning BERT reference timing scope.
Description check	✅ Passed	The description completes all required sections, explains the problem and implementation, identifies change categories, records validation evidence and environment details, documents remaining gaps, a…

Full details: Benchmark Validation Integrity

Explanation

The PR enables the BERT comparison, but the two timed paths still measure different work. The candidate path calls pipeline.embed inside the public-pipeline timer (examples/trtmc_benchmark_worker.cpp), and the TensorRT forward() performs device-to-host output copies before returning (src/runtime/backend/trt_module_impl.cpp:560-583). The reference path now includes tokenization and host-to-device preparation in Session.invoke, but it keeps the pooled vector on the GPU and only runs _tensor_summary there (benchmarks/performance/baselines/task_reference.py:1202-1231). Therefore, output transfer is timed only on the candidate. Finite-output validation is also timed only on the reference: _tensor_summary(...isfinite().all().item()) is inside the reference invoke, while the candidate's summarize runs after its timer (benchmarks/performance/baselines/hf_transformers.py:193-244, 392-415). The PR changes BERT from a pre-existing timing-contract rejection to an executable comparison, so it exposes this integrity failure. The new regression test checks tokenizer/model call order and metadata, but it does not check transfer or validation boundaries.

Resolution

Make the reference and candidate use the same returned-output boundary. For the current candidate contract, materialize the reference embedding on the host inside the timed invoke and perform the same finite-output validation inside that boundary, or change the candidate to retain device output and apply equivalent validation. Apply the same decision to the model_call_wall Eagle path, where reference input transfer is outside invoke but TensorRT records module input transfer from forward_async. Add a regression test that records host-transfer and validation events for both paths, not only tokenizer and model events.

---

_{Comment @coderabbitai help to get the list of available commands.}