Skip to content

fix: restore layerwise MiniCPM checkpoint loading on Transformers 5 - #1600

Open
feng1201 wants to merge 1 commit into
FlagOpen:masterfrom
feng1201:investigate/minicpm-rope-config
Open

fix: restore layerwise MiniCPM checkpoint loading on Transformers 5#1600
feng1201 wants to merge 1 commit into
FlagOpen:masterfrom
feng1201:investigate/minicpm-rope-config

Conversation

@feng1201

@feng1201 feng1201 commented Sep 6, 2026

Copy link
Copy Markdown

Layerwise MiniCPM loading fails under the supported Transformers 5.x range. Loading the official BAAI/bge-reranker-v2-minicpm-layerwise configuration with Transformers 5.7.0 changes rope_scaling=None into a default rope_type dictionary; _init_rope() then raises KeyError: 'type'. This reproduces the exception reported in #1581 (the report did not specify its Transformers version).

Fix the complete checkpoint-loading path in the layerwise finetuning model:

  • Interpret both legacy and normalized RoPE fields, including the unscaled default and the configured base frequency.
  • Declare the existing tied-weight relationship as a mapping and materialize its alias on head containers before the loader inspects parameters.
  • Use guarded nn.init operations so Transformers 5 does not overwrite loaded parameters, and rebuild the non-persistent rotary buffers after loading from the meta device.

The latter steps are necessary to load successfully and preserve model outputs after resolving the initial configuration error. Model weights, biases, padding rows, normalization weights and sharing relationships are checked across checkpoint round trips.

Validation on CPU, Python 3.11.15 / Torch 2.14.0:

  • New 21-case regression on unchanged source: 21 failures with Transformers 5.7.0; 21 passes with 4.44.2.
  • Same regression after the patch: 21 passes on each version. Covers analytic sine/cosine values for default/linear/dynamic RoPE beyond the original context, config round trips, the real get_model loader, one optimizer step, and all head types with shared/unshared embeddings.
  • Regression + existing import and Trainer compatibility suites on 5.7.0: 30 passed.
  • The same expanded suite on 4.44.2: 27 passed, 1 version-specific skip, 2 existing failures. Both failures assert the newer Trainer.processing_class attribute and reproduce identically on unchanged source.
  • Official checkpoint config loaded through AutoConfig.from_pretrained at revision 47b5332b296c4d8cb6ee2c60502cc62a0d708881, resized to a tiny model: construction and shared-weight check pass.
  • New-test Ruff import/format checks, git diff --check and dependency consistency checks pass.

Full-size GPU finetuning and Flash Attention were not run. No dependency changes. Developed with AI assistance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant