Skip to content

Fix per-layer rope_parameters misparsing when layer_types omits keys - #48411

Closed
Noxtimo wants to merge 1 commit into
huggingface:mainfrom
Noxtimo:fix/per-layer-rope-partial-layer-types
Closed

Fix per-layer rope_parameters misparsing when layer_types omits keys#48411
Noxtimo wants to merge 1 commit into
huggingface:mainfrom
Noxtimo:fix/per-layer-rope-partial-layer-types

Conversation

@Noxtimo

@Noxtimo Noxtimo commented Aug 29, 2026

Copy link
Copy Markdown

CPU CI GPU run-slow

What does this PR do?

Fixes #48392.

Problem

Models that use heterogeneous attention layers (e.g. Gemma3, Olmo3 mixing sliding and full attention) populate rope_parameters with entries for every possible layer type (sliding_attention and full_attention). However, when a model configuration is instantiated with fewer layers than the sliding window period (such as Gemma3TextConfig(num_hidden_layers=2) or Olmo3Config(num_hidden_layers=2)), config.layer_types contains only ["sliding_attention", "sliding_attention"].

Previously, standardize_rope_params and validate_rope checked set(rope_parameters.keys()).issubset(layer_types). When not all layer types were instantiated, this check returned False, causing the nested dictionary to be misclassified as a single flat config dict:

  1. Emitting a spurious warning: Unrecognized keys in rope_parameters for 'rope_type'='default': {'full_attention', 'sliding_attention'}.
  2. Polluting config.rope_parameters with top-level "rope_type": "default" and "rope_theta": None which was then persisted to disk on save_pretrained.
  3. Bypassing validation of actual per-layer RoPE parameters (e.g., yarn, linear) for non-instantiated layer types.

Solution

  • In src/transformers/modeling_rope_utils.py, validate that rope_parameters keys are a subset of allowed_layer_types (checking _rope_type_labels if present, or ALLOWED_LAYER_TYPES from configuration_utils.py) rather than only currently instantiated self.layer_types.
  • In standardize_rope_params, iterate over set(rope_parameters.keys()) so all configured layer types are standardized.
  • Added comprehensive unit tests in tests/utils/test_modeling_rope_utils.py verifying clean standardization, warning suppression, validation, and save_pretrained/from_pretrained round-tripping for partial layer configurations.

Before submitting checklist


Who can review?

@zucchini-nlp @ArthurZucker @Cyrilvallez

@github-actions

Copy link
Copy Markdown
Contributor

CI recap

Dashboard: View test results in Grafana
Latest run: 33267969929:2
Result: success | Jobs: 16 | Tests: 179,213 | Failures: 0 | Duration: 14h 32m

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Nested per-layer rope_parameters is misparsed when layer_types omits one of its keys

2 participants