Skip to content

fix(load): skip hand-submitted CDS records on migrate - #603

Open
TahaKhan998 wants to merge 1 commit into
CERNDocumentServer:masterfrom
TahaKhan998:fix/skip-native-cds-duplicates
Open

TahaKhan998 wants to merge 1 commit into
CERNDocumentServer:masterfrom
TahaKhan998:fix/skip-native-cds-duplicates

Conversation

@TahaKhan998

@TahaKhan998 TahaKhan998 commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Closes CERNDocumentServer/cds-rdm#347
Right now the migrator only skips if we already minted an lrecid. That misses the manually submitted new CDS papers that later come back through the old CDS dump (new CDS → INSPIRE → old CDS → migrate), so we would create a second copy.
Before create, _should_skip_recid now checks if this is already on the new CDS under another id.
It looks for that record from the dump in this order: structured CDSRDM ids, repository.cern urls, CERN DOI suffixes (10.17181/), then DOI pidstore (including arXiv DataCite DOIs), then arXiv related identifiers. If exactly one parent matches we skip, mint lrecid on that parent, and log it. Dry run does the same lookups but does not mint. If two parents match we raise and do not pick one.

EP load uses the same path, so it now passes entry into _should_skip_recid instead of only the recid.

@TahaKhan998
TahaKhan998 force-pushed the fix/skip-native-cds-duplicates branch 3 times, most recently from 18fe587 to 80a8bf6 Compare September 21, 2026 08:32
Comment thread cds_migrator_kit/rdm/records/load/load.py Outdated
Comment thread cds_migrator_kit/rdm/records/load/load.py Outdated
Comment thread cds_migrator_kit/rdm/records/load/load.py
@TahaKhan998
TahaKhan998 force-pushed the fix/skip-native-cds-duplicates branch 2 times, most recently from ea85477 to 29065c5 Compare September 22, 2026 11:54
Comment thread cds_migrator_kit/rdm/records/load/load.py Outdated
Comment thread cds_migrator_kit/rdm/records/load/load.py
@TahaKhan998
TahaKhan998 force-pushed the fix/skip-native-cds-duplicates branch 3 times, most recently from adb9d89 to cb060ed Compare September 23, 2026 14:21
@TahaKhan998
TahaKhan998 force-pushed the fix/skip-native-cds-duplicates branch from cb060ed to 1916e16 Compare September 23, 2026 14:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

INSPIRE to CDS-RDM harvester: sync document types already in CDS

2 participants