SOLR-18249: Stage shard backup metadata before publication

[nick-boss-tech]

🤖 AI text below 🤖 (posted on behalf of Nick Shanin)

https://issues.apache.org/jira/browse/SOLR-18249

What happens today

ShardBackupMetadata.store overwrites the previous metadata file in place: it deletes the existing file and then writes the new one through createOutput. A crash or failure mid-write can destroy the only good copy of a shard's backup metadata.

What this change does

The new metadata is serialized to a buffer and published in one step (BackupRepository.writeBytes, overridden in LocalFileSystemRepository):

• the local repository stages the bytes in a sibling temp file and publishes them with a requested atomic move (ATOMIC_MOVE with REPLACE_EXISTING); it no longer falls back to a non-atomic move, so an unsupported filesystem fails closed instead of silently losing atomicity;
• after any staging or publication failure the temp file is cleaned up (cleanup failures are suppressed onto the original exception), leaving the previous metadata byte-for-byte intact;
• DelegatingBackupRepository forwards writeBytes, so a wrapped local repository keeps the atomic path;
• DeleteBackupCmd now skips directory names it cannot parse as shard metadata files, so a staged temp file left behind by a JVM death no longer makes backup deletion and retention throw.

Proof

Verified at head c4cd38c: tidy, Error Prone compile, and the module check pass.

• ShardBackupMetadataOverwriteTest (1 of 1): a recording repository shows an overwrite never deletes the previous metadata file. The class uses only the pre-change API and fails against base production code with "overwrite must not delete the previous metadata file" (fork test runner run 37238130723), because base store deletes the existing file before writing.
• ShardBackupMetadataTest (4 of 4): overwrite reads back, a failed overwrite keeps the previous bytes, an unsupported atomic move preserves the existing metadata and cleans the temp file, and the interface default writeBytes routes through createOutput. In the failed-overwrite case the repository throws from writeBytes before the store code runs, so it pins the harness more than the production path. These cases do not compile against base code (they exercise the writeBytes method this PR adds), so no fail-on-base proof is possible for them.
• DeleteBackupCmdTest (5 of 5): two new cases place a leftover .tmp. staging file in the shard metadata directory and show that delete and retention both complete; with only DeleteBackupCmd.java swapped to base, both fail with IllegalArgumentException from the unguarded filename parse.

A choice to check

When the filesystem cannot move a file atomically, this PR fails the backup rather than completing it non-atomically. The alternative is a logged fallback to a non-atomic move, which keeps backups running on such filesystems but silently gives up the guarantee this change exists to provide. Failing was chosen; please say if you would rather see the logged fallback.

Separately, the deletion skip logs at DEBUG only; whether it should be WARN is left open.

Limits

• Atomic here means rename-atomic publication on filesystems that support an atomic move: readers see either the previous metadata or the new metadata, and a failed write leaves the previous file intact. It does not by itself establish power-loss durability, which would require syncing the file and its directory.
• The interface default writeBytes (via createOutput) makes no atomicity guarantee; only repositories whose backing store supports it override writeBytes to stage the bytes and publish them atomically.
• store no longer deletes before writing, so a third-party BackupRepository whose createOutput refuses to overwrite an existing object worked on base and fails with this change. The bundled S3 and GCS repositories overwrite in place and are unaffected.
• Backup deletion now skips every directory entry whose name does not parse as a shard metadata filename (a leftover staged temp file, an operator's .bak copy without the .json suffix, a .json name with the wrong prefix or too few parts, a non-numeric id), logging each skip at DEBUG only. On base each such name stopped the delete or retention operation before anything was deleted, so a genuinely damaged directory now gets a quiet partial cleanup where base failed loudly. Names the backup store writes always parse, so referenced backups are unaffected. Deletion does not remove a skipped file; a purge deletes it as an unknown file.

Changelog: changelog/unreleased/SOLR-18249.yml (fixed)

AI assistance

AI agents assisted with research, implementation, review, and drafting. Nick Shanin directed the work and takes responsibility for this contribution.