-
Notifications
You must be signed in to change notification settings - Fork 41
test(phi4-multimodal): cover the runtime config boundary #1144
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Moviw
wants to merge
1
commit into
NVIDIA:main
Choose a base branch
from
Moviw:test/phi4-multimodal-runtime-config-contract
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+100
−0
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
99 changes: 99 additions & 0 deletions
99
tests/cpp/models/phi4_multimodal/test_phi4_multimodal_runtime_config_contract.cpp
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,99 @@ | ||
| /* | ||
| * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| * SPDX-License-Identifier: Apache-2.0 | ||
| */ | ||
|
|
||
| // CPU-only consumer contract for Phi-4-multimodal's serialized runtime config. | ||
| // | ||
| // Unlike the families that nest their decoder under "text_config", Phi-4 | ||
| // multimodal is already flat at the top level, so it needs no | ||
| // get_bundle_config_overrides(). What it does carry is a nested | ||
| // "img_processor" block holding the image tower's own hidden size, head count | ||
| // and layer count. Those keys survive into the bundle's config.json, so pin | ||
| // that the decoder contract is read from the top level and the vision tower's | ||
| // dimensions never leak into it. | ||
|
|
||
| #include "runtime/models/phi4_multimodal/image_preprocessor.h" | ||
| #include "trtmc/runtime/pipeline_plugin.h" | ||
|
|
||
| #include <iostream> | ||
| #include <string> | ||
|
|
||
| int main() { | ||
| // Shape follows the checkpoint config: flat decoder geometry at the top | ||
| // level, image tower geometry nested under "img_processor". The nested | ||
| // values are deliberately small so a consumer that picked them up instead | ||
| // would fail these checks. | ||
| const std::string config = R"({ | ||
| "model_type": "phi4mm", | ||
| "img_processor": { | ||
| "image_size": 448, | ||
| "patch_size": 14, | ||
| "hidden_size": 64, | ||
| "num_attention_heads": 4, | ||
| "num_hidden_layers": 2, | ||
| "intermediate_size": 128, | ||
| "image_token_id": 999, | ||
| "vision_output_dim": 777, | ||
| "num_image_pad_tokens": 888 | ||
| }, | ||
| "vocab_size": 200064, | ||
| "hidden_size": 3072, | ||
| "num_hidden_layers": 32, | ||
| "num_attention_heads": 24, | ||
| "num_key_value_heads": 8, | ||
| "head_dim": 128, | ||
| "bos_token_id": 199999, | ||
| "eos_token_id": 199999, | ||
| "max_position_embeddings": 131072, | ||
| "image_token_id": 200010, | ||
| "fixed_image_size": 448, | ||
| "patch_size": 14, | ||
| "num_image_pad_tokens": 721, | ||
| "vision_output_dim": 3072, | ||
| "preprocessor_type": "phi4_hd_chw", | ||
| "interpolation": "bilinear", | ||
| "image_token_str": "<|endoftext10|>", | ||
| "runtime_strategy": "phi4_multimodal_vision_language" | ||
| })"; | ||
|
|
||
| const auto parsed = trtmc::parse_base_config(config, 768); | ||
| bool ok = true; | ||
| const auto check = [&](bool condition, const char* name) { | ||
| if (!condition) { | ||
| std::cerr << "FAIL: " << name << '\n'; | ||
| ok = false; | ||
| } | ||
| }; | ||
|
|
||
| // Decoder geometry, read from the top level. | ||
| check(parsed.runtime_strategy == "phi4_multimodal_vision_language", "runtime strategy"); | ||
| check(parsed.vocab_size == 200064, "vocabulary size"); | ||
| check(parsed.num_layers == 32, "layer count"); | ||
| check(parsed.num_kv_heads == 8, "KV head count"); | ||
| check(parsed.head_dim == 128, "head dimension"); | ||
| check(parsed.id_bos == 199999, "BOS token"); | ||
| check(parsed.id_eos == 199999, "EOS token"); | ||
| check(parsed.max_cache_length == 768, "max cache length override"); | ||
|
|
||
| // The image tower's own dimensions must not reach the decoder contract. | ||
| check(parsed.hidden_size == 3072, "hidden size is the decoder's, not img_processor's"); | ||
| check(parsed.num_heads == 24, "head count is the decoder's, not img_processor's"); | ||
| check(parsed.attention_size == 3072, "attention width"); | ||
|
|
||
| // The VL contract, through the family's own consumer: plugin.cpp builds | ||
| // its Phi4MultimodalPreprocessConfig with exactly this call, so a | ||
| // regression that stops reading a key, reaches into img_processor | ||
| // instead, or silently falls back to a default fails here. | ||
| const auto vl = trtmc::phi4_multimodal_parse_preprocess_config(config, ""); | ||
| check(vl.image_token_id == 200010, "image token id reaches the VL config"); | ||
| check(vl.vision_output_dim == 3072, "vision output dim is the decoder width"); | ||
| check(vl.num_image_pad_tokens == 721, "image pad token count"); | ||
| check(vl.fixed_image_size == 448, "fixed image size"); | ||
| check(vl.patch_size == 14, "patch size"); | ||
| check(vl.preprocessor_type == "phi4_hd_chw", "preprocessor strategy"); | ||
| check(vl.interpolation == "bilinear", "interpolation"); | ||
| check(vl.image_token_str == "<|endoftext10|>", "image placeholder token"); | ||
|
|
||
| return ok ? 0 : 1; | ||
| } | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Cover all multimodal fields in the contract test.
The fixture does not contain
image_token_id,vision_output_dim, orhas_vision_engine, and the test does not assert the correspondingparsedfields. A parser regression that drops or misreads any of these values would still pass. Add explicit, non-default fixture values and strict assertions for all three.As per path instructions, keep assertions strict and do not weaken expected values. The PR objective requires coverage for these three fields.
🤖 Prompt for AI Agents
Source: Path instructions