Switch the NER reference parser to UltraBERT-150M (v5.1) and take its encoder from the checkpoint

[thesanogoeffect]

Before: the reference parser is scienceverse/bibr-parser-v4-5-gold, and the torch path always builds ModernBERT-base.

After: the default parser is scienceverse/bibr-parser-v5-1-ultrabert @ 97b06cc7 (UltraBERT-150M + CRF, public). It is trained only on permissive-licence data: Europe PMC and SciELO papers under CC BY / CC0, LinkedBooks dev, and the v4.5 gold set. EXCITE is fully held out. On out-of-domain reference strings it scores about 17 points higher on RenoBench and about 33 points higher on the sealed refsets test:

	v4.5	v5.1
refsets sealed test, ref_exact	0.424	0.751
RenoBench 10k, ref_exact	0.557	0.726
JATS 2k test, record_exact	0.005	0.788
v4.5 gold val, entity F1	0.966	0.975

End to end on all 175 EXCITE test PDFs (score_refsets_e2e.py --set excite --split test; main cd7150a vs this branch, shared OCR and LLM caches, so only the parser differs):

	main (v4.5)	v5.1
ref_matching_f1	0.734	0.767
ref_journal_acc	0.681	0.725
ref_pages_acc	0.618	0.853
ref_year_acc	0.906	0.919
ref_doi_recall	0.70	0.70

Per paper, matching F1 is better on 85 papers, worse on 27 and equal on 63. The mean is +0.031 (bootstrap 95% CI +0.016 to +0.047).

The parser checkpoint now carries its own encoder. resolve_encoder loads the encoder's config, modeling code and tokenizer from the parser repo at the pinned NER_PARSER_REVISION. A repo without them, such as v4.5, keeps ModernBERT-base, so NER_PARSER_CKPT=scienceverse/bibr-parser-v4-5-gold still works.

How: trust_remote_code is set only for a 40-hex commit revision or a local directory, so a remote-code parser loads under PyTorch only at a commit SHA, never at main. AutoModel.from_config gets code_revision so the modeling code always comes from the same pinned commit (vendored, Apache-2.0, from dmis-lab/UltraBERT-150M@545ac9d9). NER_PARSER_ENCODER / NER_PARSER_ENCODER_REVISION override it and are part of the parser cache key. The ONNX path needs no remote code, and ONNX matches torch on the pinned checkpoint. This PR adds no dependency changes; main's lock refresh (#208, transformers 5.17.0) is merged in. The docs runtime table and the dry-run size table mirror the pin, and test_model_pins checks both.

Training code: scienceverse/bibr-training#32.

[thesanogoeffect]

Primary suite (Linux, Python 3.12) fails in two tests:

• tests/test_extractor.py::TestRefExtractionStrategy::test_llm_extracts_references
• tests/test_extractor.py::TestRefExtractionStrategy::test_completeness_filter

Both run the real NER recovery path, which downloads the default parser from the Hub. The pin on this head, scienceverse/bibr-parser-v5-ultrabert@f508ada9, is still a private repo, so the download returns 401 (RepositoryNotFoundError) and the recovered references are missing.

This failure is caused by this PR, but no code change fixes it. The PR will be re-pinned to the permissive-licence retrain once that model is published, and the next push carries that pin. A re-run before then would fail the same way, so I'm not re-running. Every other check on this head has passed or is still running.

---

Generated by Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 97.29730% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 90.56%. Comparing base (c18eb85) to head (175e80e).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #209   +/-   ##
=======================================
  Coverage   90.55%   90.56%           
=======================================
  Files         309      309           
  Lines       57289    57321   +32     
  Branches    10376    10380    +4     
=======================================
+ Hits        51880    51911   +31     
  Misses       3450     3450           
- Partials     1959     1960    +1     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.