Add the rustic filesystem snapshot store

[kmatasfp]

resolves GOL-604
stacked on #3940

This change adds RusticSnapshotStore, the production adapter of FilesystemSnapshotStore. It keeps one rustic repository for each scope on the blob storage of the executor. It also adds the [filesystem_snapshots] configuration block, and it sets a time zone in the executor image.

Configuration

[filesystem_snapshots] has type = "Disabled" (the default) or "Managed". [filesystem_snapshots.config] holds repository_key, storage_call_deadline (default 60 s), restore_reader_threads (default 6) and save_threads (default 2). The key has 128 hex characters, and the safe display and Debug hide it. Parsing refuses a zero deadline, a zero thread count, and a key that is empty, too short, too long or not hex. The block follows the Raw* and custom Deserialize pattern of FilesystemPressureConfig.

The 60 s default is in one place, DEFAULT_FILESYSTEM_SNAPSHOT_STORAGE_CALL_DEADLINE. The constant STORAGE_CALL_DEADLINE of the bridge goes, and the benchmark takes the configuration default. The executor start builds no store yet. A later step builds it and awaits shut_down before the runtime drops.

The debugging service builds a GolemConfig field by field, so it gets the new field with its default. The reference configuration files of the executor change with the new block.

The time zone of the image

rustic asks for the system time zone for each timestamp that a save writes. The image has no time zone database and no TZ, so each save wrote about 6 warnings for each file. The final stage of golem-worker-executor/docker/Dockerfile now sets ENV TZ=UTC0, a POSIX rule that jiff reads without a database. A test requires that line.

A save makes a snapshot visible in one step

The backend of a save keeps the snapshot file in a stage and does not write it. rustic hashes the file and returns its id, and it reads nothing back. After the blocking work returns, the save future writes the file with one put_raw_if_absent. That write is the commit point. So no blocking thread can publish a snapshot after the caller drops the save, which the contract of GOL-602 requires.

A publish that fails, also by the deadline, deletes the file before the save gives its error. A drop during the publish starts the same delete as a task of the task tracker of the store. One case stays open: S3 completes a dropped or timed-out PUT after that delete, or the delete fails. The result is a complete snapshot under a name that the caller gave up. Retention removes it. The store adds no persistent tombstone for this case. The filesystem backend has the same window: its put_raw_if_absent writes on a blocking thread that goes on after the call gives up. GOL-622 fixes that case in the filesystem backend.

Cancellation and shutdown

Each operation gets a child of the root cancellation token of the store, and its drop cancels that token. BlobBackend::request refuses a call of a cancelled operation, and the token also ends a call that runs. So the rustic threads of a dropped restore, stat, list, delete or save stop at their next storage call. The calls that the store makes itself, through SnapshotFiles, hold the token of their operation too: the calls of copy_scope and delete_scope, the ledger calls and the listing of data/. A call of a cancelled operation does not start, and a cancel ends a call that runs.

The store counts each blocking task, each backend, each publish and each delete of a dropped publish in a TaskTracker. A backend holds a tracker token until its last owner drops it, so a rustic thread that outlives its operation is counted too. shut_down cancels the root token, closes the tracker and waits. The publish is the commit point, so no cancel ends it. A publish that starts before the cancel runs to its end, and shut_down waits for it. The deadline limits that wait. The tracker counts the publish before the save checks the root token. So a save that reaches its publish after the cancel publishes nothing and gives Storage, and its packs stay for a later prune. After shut_down returns, no storage call polls a timer, and each operation gives Storage. A timer that is polled after the runtime stops its time driver panics, and the executor builds with panic = "abort".

Repository creation, index files and lookup

The backend writes the config file with put_raw_if_absent. When two stores make one repository at the same time, the store that loses sees a private marker in the error chain, and open_or_create opens the repository of the winner. init with a master key writes only the config file, so the store that loses leaves nothing. This change is in the bridge, so the benchmark gets it too.

Index files also use put_raw_if_absent, and AlreadyExists is a success, because the name of an index file is the hash of its content. On the filesystem backend, put_raw can leave a part of a file after the process stops, and a partial index makes the repository unreadable.

The label of a snapshot is its name. Of the snapshot files with the name, the one with the least time and id wins. The name is inside an encrypted file. So when no file has the name and a snapshot file of the scope fails its integrity check, stat and restore give Corrupt, because that file can have the name. list leaves such a file out. A snapshot file that a delete removes after the listing is not counted as a failed check.

Two saves of one name at the same time can both succeed, and the lookup rule then gives the older one. The upload service admits one upload for each agent, so this does not happen in use.

What a snapshot keeps

• files and bytes come from one metadata walk of the tree, and the store keeps them in the description of the snapshot. rustic counts a symlink as a file.
• The time of a save is snapshot_time of the module, from the first whole millisecond that is not before the call, because a snapshot keeps its time in milliseconds.
• A save records no device id, so a restore gives each name of a hard-linked file as its own file, as the contract says. With the default device id, rustic restores the hard link.
• The store gives rustic the canonical path of the tree. rustic strips the backup root from the path of each entry with an unwrap. A root through a symlink, such as /proc/self/fd/N, made that unwrap panic. as_path("/") keeps each stored path below /.

Options

• Backup: the options of the bridge, with fail_on_read_error(true), threads(save_threads) and no device id. A save that cannot read an entry gives Source and writes no snapshot file.
• Restore: fail_on_metadata_error(true), no_ownership(true) and reader_threads(restore_reader_threads). A snapshot does not keep the owner, so a restore does not set it. After GOL-650, a failed set of a security. attribute is one warning, so the strict option works on a volume with seclabel.
• Repository: no_cache(true) and the master key.

The benchmark keeps its own options.

Prune

Prune runs in delete, after the forget. The store adds the packed bytes that the deleted snapshots added to a ledger blob at golem/prune-ledger in the scope. The pure function prune_due decides from the ledger, the time, the size of the repository and the policy. A prune is due when the freed bytes reach about 10% of the size of the repository (10%, rounded down to a whole byte), or when the last prune marked packs, and at most one time in each keep_delete. The size of the repository is the sum of the sizes of the blobs below data/, from one listing. The store makes that listing only when the size can make a prune due: the grace period passed, the ledger has freed bytes, and no packs wait for removal. So a delete inside the grace period makes no listing. A failed listing gives Storage, and the ledger keeps its freed bytes. The store writes the ledger before the prune, so a delete that runs again after a failed prune prunes again. instant_delete stays off. A prune then marks a pack that no index lists, and it removes the pack only after keep_delete. The report of the prune counts these packs, and the ledger records that marked packs wait for removal, so the next prune after the grace period removes them. After the prune, an index lists a marked pack, so a later prune does not count it again. So a save that is shorter than the grace period keeps its packs. Two deletes at the same time can lose a count, which only makes a prune come later.

Two deletes that read the same ledger must not both prune. So a delete whose prune is due first takes a claim, a blob at golem/prune-claims/<generation>/<n> that it writes with put_raw_if_absent. The generation is the time of the last prune in the ledger that the delete read, so two such deletes claim in the same directory, and only one write succeeds. A newest claim that is younger than keep_delete holds the generation. A newest claim that is older belongs to a prune that stopped without an end, and the delete claims the next number. After it takes the claim, the delete reads the ledger again. A prune writes its ledger before it deletes the claims, so a delete that claims after another prune ended sees the new ledger, deletes its claim and does not prune. A prune that fails, and a second read that fails, delete the claim, so a retry of the delete prunes again. A prune that succeeds deletes the claims of its generation.

Copy and delete of a scope

copy_scope lists the snapshot files, the index files, the keys and the packs. It writes them in the reverse order of the listing, and the config file last with put_raw_if_absent. So the target holds a repository only when all its blobs are there, and a snapshot file in the target always has its data. It does not copy the ledger. delete_scope deletes the config file first, and then each directory by name, the ledger directory included.

The parent of a save

save takes parent: Option<(&SnapshotName, ChangeDetection)>, as section 3.1 of the project describes. A reflink capture gives each file a new ctime, so rustic's default check read the whole tree at each save. The caller knows when the size and the modification time can be trusted, so the caller chooses the mode. A rename over a file of the same size and modification time shows that the store cannot choose it. The rustic store finds the named parent with the lookup rule and gives its id to rustic. With SizeMtime, a file whose size and modification time equal those in the parent is not read. Full, no parent, and a parent that the scope does not hold make rustic read every file. The store no longer takes the newest snapshot of the scope as the parent, so a save without a parent now reads every file. The in-memory store ignores the parent.

Tree packs of one operation

Without the rustic cache, rustic reads each tree blob with its own ranged read, one after another, and a restore reads each tree two times. The GOL-603 benchmark measured a warm save of 77 s and a restore of 152 s for a tree of 2,775 directories. The backend now keeps each pack of tree blobs after its first read and gives the later ranges from memory. It reads a pack in full one time, through the same call with its deadline and cancellation. When two threads miss one pack, one reads it and the other waits. A failed read keeps nothing. The packs live in the backend of one operation, in memory only, up to 32 MiB, and a range is a slice of the kept bytes. Data blobs do not change. A restore of a tree with 60 directories now makes one full read of its tree pack in place of 121 ranged reads.

Low CPU priority for saves and prunes

A save is bound by the CPU, and its thread count does not limit its CPU use, so agent work comes first. Saves and prunes run at nice 19, and a restore keeps the normal priority, because the start of an agent waits for it. A thread without privilege cannot raise its priority again after it lowers it, so no thread of the tokio blocking pool changes its priority. Inside the task of execute_native, a save or a prune starts its own thread. That thread sets nice 19, builds a rayon pool of its own with save_threads threads, runs the rustic work in that pool, and ends. The threads that rustic starts get nice 19 from it, and the parallel parts of rustic, such as the repack of a prune, run in that pool and not in the global rayon pool. The store starts the global rayon pool when it is made, at the normal priority, so no nice-19 thread ever makes it. A failed step gives a warning, and the work still runs. The benchmark keeps the normal priority.

Errors

rustic gives no public kind for an error. So classify reads the chain of sources:

• A failed blob storage call gives Storage. It is retryable unless a name error of the blob storage caused it.
• An I/O error gives Source in a save and Destination in a restore.
• Each other error gives Storage that is not retryable in a save or a prune, and Corrupt in a restore or a read of the repository.

Tests check this with real chains: a file without read permission, an extended attribute that the destination refuses, and failed pack reads.

Values that the user approved in GOL-603

The user approved these values on 2026-09-25, and the code uses them:

• the Rabin chunker with the rustic sizes, compression unset, and extra_verify on;
• a prune when the freed bytes reach about 10% of the size of the repository;
• a keep_delete of 15 minutes, which is also the shortest time between two prunes of one scope;
• fast_repack on, with the rustic repack limits (max_unused 5%, max_repack 10%);
• a storage_call_deadline of 60 s, 6 restore reader threads and 2 save threads.

The threshold is a share of the repository and not a fixed byte count, so a small scope prunes after a small delete and a large scope does not prune after each delete. The values of the upload service, such as max_concurrent_uploads, belong to a later step.

How to read the diff

• golem-worker-service/src/gateway_server/tests.rs: a fix of a flaky gateway test, in its own commit. LiveBodies::guard did not wake the waits for a count, so a wait that started before the websocket handler made its guard never ended. The guard now wakes them, and the timeout of the test is the only limit of a wait.
• filesystem_snapshot/contract_tests/mod.rs: a fix of a flaky contract case of GOL-602, in its own commit. The in-memory save can end in its first poll on a busy host, so the case tries saves in new scopes until one does not end in its first poll. It proves the same rule as before.
• rustic/mod.rs: the bridge changes little. open_or_create opens the repository of the winner, and restore moves its writing part into restore_snapshot, which the store also uses.
• rustic/store.rs: the store composes the bridge functions.
• backend.rs with backend/kept.rs, files.rs, publish.rs, fault.rs, prune.rs, scope.rs, priority.rs: they hold the parts above.
• scripted.rs: a test blob storage that records calls and can refuse, lose an answer, never answer, or wait at a gate.
• The rustic store registers the contract suite of GOL-602 through the configuration path.