[upstream PR Ephemeral-AI-Lab#495] feat(persistence): export a Store's changed rows as bundles for replicas

• Upstream: feat(persistence): export a Store's changed rows as bundles for replicas Ephemeral-AI-Lab/layerfs#495
• Type: pull request
• State: OPEN
• Draft: False
• Mergeable: MERGEABLE
• Updated: 2026-10-28
• Upstream labels: -

Original body (excerpt)

I would like to know whether this direction fits the roadmap. I am happy to rework or split it, and the open questions below are where I most need your call.

What

A new module, layerfs_persistence::replication, keeps a second copy of a Store up to date by moving only the rows that changed:

• mark_base(store, token) makes an offline Store the start of an export chain and returns its cursor. A plain copy of the file taken at that point is the replica.
• export_changes(store, since, bundle) writes every row changed after since into a new SQLite bundle file, from one read snapshot of the live Store. It takes no write lock and does not stop writers.
• apply_changes(replica, bundle) installs bundles in order into the offline replica. The replica then holds exactly the live Store's rows and opens through the normal open path.

Nothing here moves a file anywhere. A bundle is a local file; the caller decides whether it goes to another disk, another machine or object storage.

Why

Today the only way to bring a second copy of a Store up to date is to copy the whole file again, or a consistent snapshot of it. That cost follows the size of the Store, not the size of the change: a few new Commits on a multi-GiB Store mean copying multi-GiB again. Cases where I need a second copy:

• keep a backup of a Store current while the Store is in use;
• move a Store to another machine and keep the copy in step;
• the push side of item R in LayerFS 2.0 roadmap Ephemeral-AI-Lab/layerfs#446: with bundles, a push could send only the rows written since the last push instead of the whole file.

How

• Change log. Two new tables, change_state and change_log, and AFTER INSERT/UPDATE/DELETE triggers on the 18 replicated tables (plus pack_unit in the group-row layouts). Every trigger is WHEN EXISTS(SELECT 1 FROM change_state), so a Store that was never marked logs nothing. The log holds row keys only. Export reads the current values, so a row written many times is exported once, and a deleted row becomes a deletion.
• Chained, verified bundles. Each bundle carries a BLAKE3 digest of its rows and deletions and a state chained from the base: state' = H(catalog, state, seq, seq', digest). Before it installs anything, apply_changes refuses a meta row that is not exactly one well-formed row (it does not trust the bundle's own constraints), an unknown bundle format, another catalog, a cursor that is not the replica's (out of order or repeated), a replica written since its last apply (Diverged), a bundle that deletes an immutable row, and a changed bundle (digest or ending state). One bundle installs in one transaction, with foreign keys checked at commit.
• Publishing a bundle. An export writes the bundle under a temporary name beside the bundle path and hard-links it into place, so the path names a complete bundle or nothing, also after a process crash. An existing path is refused with Exists.
• Checkpoints. mark_base and apply_changes checkpoint after they commit, so that the Store file alone holds the change. When another connection's read blocks that checkpoint, or it fails, they return ChangeError::NotCheckpointed: the change committed, and the file holds it only together with its WAL.
• Connections. The module opens Stores through the Disposable profile (profile::apply: WAL, synchronous=OFF) with no busy handler. A locked Store is refused at once with SQLITE_BUSY instead of waiting 5 s in rusqlite's default handler; a Store that is not in WAL mode is refused as ChangeError::Profile(PersistenceError::Malformed). ChangeError implements source() for the wrapped SQLite, I/O and profile errors.
• Migration. add_change_log is an additive open-time migration (REBUILD.md kind 2), added under the same conditions as add_file_digests.
• Layout. src/replication/ has one file per concern: base.rs, export.rs and apply.rs for the three calls, and tables.rs, digest.rs, connection.rs, lookup.rs, records.rs and error.rs for what they share. The largest file, export.rs, is 227 lines.
• Second commit. apply_changes installed rows with logging on and then deleted that log, which left free pages in the replica file (22 in the new test). It now removes change_state for the install and restores it with the ending state, in the same transaction.
• Architecture doc 84 lists the two tables and describes export, apply and the checkpoint report.

Not changed

• No change to object formats, pack layout, history semantics, the publication lane or any layerfs-api / SDK / CLI surface. Nothing outside the module's tests calls it yet.
• The Store is still one SQLite database. A bundle is a transfer file that only apply_changes reads; it is not part of the Store.
• Cargo.lock is unchanged. blake3 =1.8.5 moves from dev-dependencies to dependencies of layerfs-persistence; six other workspace crates already depend on it.

Cost

Measured with a temporary test that is not part of this PR: save 3000 distinct 2 KiB objects into a fresh Store, 7 rounds per tree, Linux aarch64 container, test profile (opt-level = 1). I measured the first revision of this branch; later revisions changed export and apply only, not the triggers or the write path.

An unmarked Store shows no difference beyond run-to-run noise; its file is 48 KiB larger, which is the new schema. A marked Store was 0.9% slower than main and 5.6% slower than the unmarked run at the median. It logged 6 rows per saved object, about 19 bytes per row in the file. I can turn this into a proper benchmark if you want one in the tree.

Open questions

1. Writer admission. mark_base and apply_changes rely on SQLite's BEGIN IMMEDIATE with no wait. They do not take the Linux directory lock in writer_admission.rs. Both require an offline Store, so I left them outside the lane. Should they go through it, or through Handles?
2. Cursor check above the base. For a cursor above the base, export_changes checks the sequence but not the state, because the live Store keeps no record of the states it exported. A cursor from a separately written copy is therefore not detected at export; apply_changes still refuses the result, because the replica's state does not match. The doc comment says this. Should the Store record the states it exports instead?
3. Log growth. The log grows until the next mark_base. Nothing prunes entries that were already exported, because export does not write to the live Store. A prune_changes(cursor) writer call would fix that; I left it out to keep this PR small.
4. Do you want this in layerfs-persistence at all, or behind a feature, before anything

…（正文截断；完整内容见上游链接）

Sync note

这是只读镜像，不是上游仓库的替代。上游 issue/PR 仍是唯一真源；本镜像不复制评论、不复制代码分支。