From b231eeaf6c47cd044f021a4eee4f57410e77fd9d Mon Sep 17 00:00:00 2001 From: Josh Vaughen Date: Thu, 8 Oct 2026 17:46:41 -0700 Subject: [PATCH 1/3] docs(claude): rewrite CLAUDE.md in STE in the mirror shape --- CLAUDE.md | 186 ++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 131 insertions(+), 55 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 7388d69..9989c06 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,63 +1,139 @@ # dropbox -A nightly, self-chaining mirror of a Dropbox account into one Proton Drive folder, with -its state in an age-encrypted SQLite database in a bucket. `README.md` says what it -mirrors, how it works step by step, how to fork it and the runbook; -[katoptra/lib](https://github.com/katoptra/lib)'s README is the manual for the toolbox -this mirror includes. This file is what a change must not break. - -Nothing in this repo starts a run: an external scheduler dispatches `sync.yml` nightly. -This mirror includes the toolbox and the proton engine. The engine supplies the session -verbs, `empty-trash`, and the bucket's `pull` and `push`. `Taskfile.yml` owns the order, -one `python -m migrator ` per step of the mirror, and `src/migrator/` owns every -decision but when to walk Proton, which is lib's `due`. The infrastructure modules there -are from donphi/dropbox_proton at `cfd0e57`, MIT; the phases under `src/migrator/phases/` -and the Taskfile are this repository's own. +This repository has the pipeline of a nightly mirror of a Dropbox account in one Proton +Drive folder. If a run has more work than its budget, it starts the next run through the +chain. The state of the mirror is an SQLite database in a bucket, encrypted with age. +`README.md` tells: + +- The files that the mirror contains +- How each step operates +- How to fork the mirror +- How to operate it, with the runbook. + +The README of [katoptra/lib](https://github.com/katoptra/lib) is the manual for the toolbox +and the proton engine that this mirror includes. This file gives the rules that each change +must obey. + +No part of this repository starts a run. An external scheduler dispatches `sync.yml` +nightly, at 11:42 UTC. + +The proton engine supplies the session verbs and `empty-trash`. It also supplies the `pull` +and `push` of the bucket. `Taskfile.yml` sets the sequence of the steps, with one +`python -m migrator ` for each step of the mirror. The code in `src/migrator/` +makes each decision, but the `due` verb of lib finds if a run does the Proton walk. + +The infrastructure modules in `src/migrator/` are from donphi/dropbox_proton at `cfd0e57`, +with the MIT license. The phases in `src/migrator/phases/` and the Taskfile are from this +repository. + +## Constraints + +- **Where a change goes.** `Taskfile.yml` has these verbs: + - `pipeline` and `plan-pipeline` + - The steps of the pipeline, each one `python -m migrator `: `clock-phase`, + `state`, `inventory`, `delta`, `plan-phase`, `batches`, `trash`, `reconcile` with + `record-walk`, and `report-phase` + - `status`, `offline`, `fmt` and `state-rollback` + - `report-mirror`, `report-engine` and `empty-trash-pipeline`. + + Make all other changes to verbs in lib: in the toolbox or in the proton engine. Then each + mirror that includes that file gets the change. +- **Excludes.** The `excludes:` of the two includes contain these verbs: + - On the toolbox include: `report-mirror` + - On the proton include: `pipeline`, `plan-pipeline`, `trash`, `empty-trash-pipeline` and + `report-engine`. + + `report-engine` has no commands, because the pipeline does not use the `stage` and the + `confirm` of the engine. +- **Root vars.** Root vars hold only the values of this mirror. Do not put an engine default + in a root var, because then the command line cannot set it + ([lib README, Rules a mirror keeps](https://github.com/katoptra/lib#rules-a-mirror-keeps)). + `RECONCILE_HOURS` (168) is the only root var. +- **The run budget.** `run_budget_minutes` (335) in `config/mirror.toml` must stay 20 + minutes less than `timeout-minutes` (355) in `sync.yml`. Thus, the migrator can upload + the last batch and make the report before the timeout. The chain is a step of the sync + workflow of lib: it starts the next run if the pipeline wrote `.run/chain`. +- **`config/mirror.toml` is strict.** It rejects a key that it does not know. The + environment gives the three account identifiers, and they override the keys of the file. +- **Storage.** For each change that adds storage, calculate the new storage. Compare it + with these values: + - The baseline: 161,650 files and 610 GiB in Proton Drive (the state on 2026-10-09) + - The ceiling: `ceiling_gb` (4,000 GiB), and the storage limit of the Proton plan. +- **Writing.** Use ASD-STE100 and the rules in the + [Writing section of the org CONTRIBUTING](https://github.com/katoptra/.github/blob/main/CONTRIBUTING.md#writing). + Read that section before you write. ## Must knows -- **Plan by default.** `batches`, `trash`, `reconcile`, `report` and `empty-trash` mutate - only with `--apply`, which `task pipeline` passes and `task plan-pipeline` never does. - No mutation is trusted on its exit status: the state push is trusted once the object - lands, and nothing is recorded as uploaded on a command's exit code alone. -- **A missing state beside history is refused.** A lost state must never look like an - empty mirror. The rollback is `task state-rollback`; never delete the history to make - a run start fresh. -- **A truncated listing can never become a trash list.** `delta` refuses a listing under - `listing_floor_ratio` of the mirrored count, and `trash` runs only when every planned - batch landed. `trash` takes a folder whole only when `mirror_objects` holds nothing - live under it, otherwise its files by name, 50 paths per call; it works inside the run - budget, checkpoints every 50 units and chains the rest; a run cut off mid-phase - repeats at most 50 units. -- **`checkpoint` is the last step of a batch**, a dated copy first and then a server-side - copy to the canonical key, so a killed run repeats at most one batch. A batch that - recorded nothing fails the run rather than chaining, so an identical failure cannot loop. -- **The session is this mirror's own.** Two processes holding it race its rotating - refresh token. Never run `sync` or `empty-trash` from a laptop while an Actions run may - be going. The reconcile walk's workers each run from their own copy of the session, and - the copy a refresh rewrote is adopted. -- **`run_budget_minutes` stays 20 minutes under `sync.yml`'s `timeout-minutes`**, so the - last batch's upload and the report finish. The chain is the workflow's, on `.run/chain`. -- **Files are keyed by lowercased path**, so a case-only rename changes nothing in the - delta; display paths are rebuilt from each ancestor's own name because Dropbox cases - parent segments inconsistently. -- **The logs are public.** The report is built from the state and carries counts only; - errors print as their class unless `MIRROR_VERBOSE=1`; `op run` masks every value. -- **Run flags go after the double dash** (`task sync -- RUN_BUDGET_MIN=30 RECONCILE=true`, - or the workflow's `vars` input). The Taskfile maps `RUN_BUDGET_MIN` to the environment - the migrator reads; `RECONCILE` (`true`, `false` or `auto`) is lib's `due`'s, which - leaves `.run/reconcile` for the migrator when the last complete walk is - `RECONCILE_HOURS` (168) old. -- **`config/mirror.toml` is strict** and rejects unknown keys; the three account - identifiers come from the environment and override its keys. +- **Without `--apply`, a step only makes a plan.** These steps change data only with + `--apply`: + - `batches` + - `trash` + - `reconcile` + - `report` + - `empty-trash`. + + `task pipeline` gives `--apply`, and `task plan-pipeline` does not. For a command that + changes data, an exit code of 0 is not a sufficient indication that the change occurred. + The mirror accepts a state push only when the object is in the bucket. It records a file + as uploaded only when the summary of `upload` includes that file. +- **If the bucket has history but no state, the run stops.** The run must not accept a + missing state as an empty mirror. To repair the state, use `task state-rollback`. Do not + delete the history to make a run start with no state. +- **A short listing cannot become a trash list.** + - `delta` rejects a listing that has less than `listing_floor_ratio` of the number of + mirrored files. + - `trash` runs only if all the planned batches are in Proton. + - `trash` moves a folder to the trash in one call only if `mirror_objects` has no + remaining file in it. In the other folders, it gives the path of each deleted file to + `filesystem trash`, 50 paths in each call. + - `trash` operates in the run budget. It makes a checkpoint after each 50 units, and the + chain does the remaining units. If a run stops in the middle of the phase, the next run + does a maximum of 50 units again. +- **`checkpoint` is the last step of a batch.** It pushes a dated copy of the state first. + Then it makes a server-side copy of it at the canonical key. Thus, if the runner stops a + run, the next run does a maximum of one batch again. If a batch records no file, the run + stops with a failure and does not start the next run. Thus, the same failure cannot occur + in a loop. +- **This mirror has a session that no other mirror uses.** + [lib, The session](https://github.com/katoptra/lib#the-session) gives the rules for a + session. After each CLI call, the migrator compares `auth-session.json` with the digest in + `.run/session.sha`. If the token changed, it sends the session back to the bucket. Each + worker of the reconcile walk uses a copy of the session that no other worker uses. The + session directory then gets the copy that a refresh changed, also if the walk stops with + an error. +- **The key of each file is its path in lowercase letters.** Thus, if a new name changes + only uppercase and lowercase letters, the delta does not change. The mirror makes each + display path from the name of each folder above the file. The cause is that Dropbox does + not always give a parent folder the same uppercase and lowercase letters. +- **The logs are public.** The report uses only the data in the state, and it contains only + counts. An error shows only its class, unless `MIRROR_VERBOSE=1`. `op run` removes each + secret value from the output. +- **Put the flags of a run after the double dash** (`task sync -- RUN_BUDGET_MIN=30 + RECONCILE=true`), or in the `vars` input of the workflow. The Taskfile gives + `RUN_BUDGET_MIN` to the migrator in its environment. `RECONCILE` (`true`, `false` or + `auto`) is for the `due` verb of lib. `due` writes `.run/reconcile` for the migrator when + the last completed walk is `RECONCILE_HOURS` (168) or more before the run. +- **A run failure is the only alert.** The healthcheck in healthchecks.io has the cron + `42 11 * * *` UTC and a grace time of 3 hours. On Thursdays, the weekly Proton walk adds + approximately 95 minutes to the run, and the grace time includes it. ## Verifying a change -- `task test` runs the pytest suite inside the image, offline; `task lint` is ruff. -- `task check` renders every command of the pipeline inside the image and diffs it - against `render.txt`; `task render-update` accepts a change. The `check` workflow runs - check, test and lint on every pull request, with no secret. -- `task plan` runs the read path against the real account and bucket, changes nothing in - Proton, and prints the report it would make. -- `task status` fetches the state and prints counts; both leave `.run/state.sqlite` for - inspection. +- `task run -- task offline` runs these checks in the image, with no network and no + credentials: + - pytest + - `ruff check` + - `ruff format --check` + - A dry run of `task empty-trash-pipeline` from the command line, as `task empty-trash` + starts it. +- `task check` makes a render of each command of the pipeline in the image. Then it compares + the render with `render.txt`. `task render-update` accepts a change. +- The `check` workflow uses the check workflow of lib, with `offline: true`. Thus, on each + pull request, it runs `task check` and then `task run -- task offline`, with no secrets. +- `task plan` does the read-only part of a run with the account and the bucket of the + mirror. It makes no change in Proton, and it prints the report of the plan. +- `task status` gets the state and prints counts. After `task plan` or `task status`, the + state is in `.run/state.sqlite`, where you can examine it. +- The checks of the engine are in lib: + `cd ../lib/examples/proton && task run -- task offline`. From 1c9140176717e9ed7975f205668fa44aca3a237a Mon Sep 17 00:00:00 2001 From: Josh Vaughen Date: Thu, 8 Oct 2026 17:46:41 -0700 Subject: [PATCH 2/3] docs(readme): rewrite the README in STE and apply the review items --- README.md | 477 +++++++++++++++++++++++++++++++++--------------------- 1 file changed, 295 insertions(+), 182 deletions(-) diff --git a/README.md b/README.md index bfc419e..572d637 100644 --- a/README.md +++ b/README.md @@ -17,42 +17,58 @@ mirror

-A nightly, self-chaining mirror of a Dropbox account into one Proton Drive folder. After -each run Proton Drive holds what Dropbox held at listing time. Changed files become new -Proton revisions, files that left Dropbox move to Proton's trash, and nothing is recorded -as mirrored until Proton's own upload summary has accounted for it. The only durable -state is one SQLite database, age-encrypted in a bucket; no run is ever told where to -start. Dropbox stays the primary; nothing written in Proton Drive flows back. - -The repository is public and holds no account: every credential and every account -identifier lives in one vault and reaches a run by name. The infrastructure modules under -`src/migrator/` (the SQLite evidence schema, the one-pass hasher, atomic writes, the -redacting logger, path guards, and the two providers for the Dropbox API and the -`proton-drive` CLI) come from -[donphi/dropbox_proton](https://github.com/donphi/dropbox_proton) at commit `cfd0e57`, -MIT, whose copyright notice is retained in [LICENSE](LICENSE). The mirror phases and the -Taskfile are this repository's own. +This is a nightly mirror of a Dropbox account in one Proton Drive folder. If a run has more +work than its budget, it starts the next run through the chain. After each run, Proton Drive +has the files that Dropbox had at the time of the listing: + +- A file that changed becomes a new revision in Proton. +- The mirror moves a file that is not in Dropbox to the trash of Proton. +- The mirror records a file only after the summary of the Proton CLI includes that file. + +The only state that the mirror keeps from one run to the next is one SQLite database. It is +in a bucket, encrypted with age. Each run finds its start point in the state, and no input +tells a run where to start. Dropbox is the primary copy. The mirror writes no change from +Proton Drive back to Dropbox. + +This repository is public, and it contains no account data. All the credentials and the +account identifiers are in one vault. A run uses the name of each value to get it. + +The infrastructure modules in `src/migrator/` are from +[donphi/dropbox_proton](https://github.com/donphi/dropbox_proton), at commit `cfd0e57`, with +the MIT license. [LICENSE](LICENSE) keeps the copyright notice of that project. These are +the modules: + +- The SQLite schema of the evidence tables +- The hasher, which reads a file one time to calculate all its hashes +- The module that writes files atomically +- The logger, which removes the secret values from its lines +- The path guards +- The two providers: one for the Dropbox API, and one for the `proton-drive` CLI. + +The phases of the mirror and the Taskfile are from this repository. ## How to use -The files are under the destination folder in Proton Drive, at their Dropbox paths and -with Dropbox's casing. An edit in Dropbox is a new revision in Proton the next morning; -a delete is in Proton's trash; a file too large for the runner to stage is counted as -oversized in the run's report and can be uploaded by hand to its Dropbox path, where the -weekly walk leaves it alone. +The files are in the destination folder in Proton Drive. Each file has its Dropbox path, +with the uppercase and lowercase letters that Dropbox gives. + +- If you edit a file in Dropbox, the next nightly run makes a new revision of it in Proton. +- If you delete a file in Dropbox, the mirror moves it to the trash of Proton. +- A file can be too large for the disk of the runner. The report of the run counts such a + file as oversized. You can upload it manually to its Dropbox path in Proton. The weekly + walk does not move that file to the trash. ## How it works -Once a night a GitHub Actions job runs this pipeline inside the toolbox image from -[katoptra/lib](https://github.com/katoptra/lib). The verbs of the toolbox and the proton -engine are the solid boxes; every dashed box is one `python -m migrator `. This -mirror includes the toolbox and the proton engine, and supplies its own pipeline: the -Taskfile owns the order, the Python owns every decision. +Nightly, an external scheduler starts a GitHub Actions job. The job runs this pipeline in +the toolbox image of [katoptra/lib](https://github.com/katoptra/lib). Each solid box is a +verb of the toolbox or of the proton engine. Each dashed box is a step of this mirror: one +`python -m migrator `. ```mermaid flowchart LR - clock --> cp["clock"] --> session --> state --> inventory --> delta --> plan --> batches - subgraph b["batches: each until the run's budget"] + clock --> due --> cp["clock"] --> session --> state --> inventory --> delta --> plan --> batches + subgraph b["batches: one at a time, until the end of the run budget"] direction LR fetch --> verify --> upload --> confirm --> checkpoint end @@ -61,47 +77,63 @@ flowchart LR class cp,state,inventory,delta,plan,batches,fetch,verify,upload,confirm,checkpoint,trash,reconcile,rp own ``` -| Step | Does | +This mirror includes the toolbox and the proton engine, and `Taskfile.yml` gives the +pipeline. The Taskfile sets the sequence of the steps. The Python makes each decision, but +the `due` verb of the toolbox finds if a run does the Proton walk. + +| Step | Function | |---|---| -| `clock` | The migrator's own stamp after the toolbox's: the run's start to `.run/clock.json`; clears staging, the report and the chain marker | -| `session` | Fetches the encrypted Proton CLI session from the bucket and unpacks it for every later CLI call | -| `state` | Fetches the state database and starts the run row. A missing state is accepted as an empty mirror only when the history is empty too; a missing state beside history is refused, because a lost state must never look like an empty mirror | -| `inventory` | Walks Dropbox over the API, each page committed with its cursor. Entries with no content hash are recorded as non-downloadable and excluded. Display paths are rebuilt from each ancestor's own name, because Dropbox cases parent segments inconsistently | -| `delta` | Compares the inventory against `mirror_objects` on path, size and content hash. Refuses a listing under half the mirrored file count, so a truncated listing can never become a trash list | -| `plan` | Refuses a tree over `ceiling_gb` or a batch the disk cannot stage. Leaves out files over `max_file_gb` and counts them as oversized. Packs the rest into batches of `batch_gb` and `batch_files` | -| `batches` | Runs each batch through the five steps below, stopping before a batch that would pass the run's budget; stopping with batches left is a success that chains the next run | -| `fetch` | Downloads the batch from Dropbox into staging under its display paths; a path that vanished since listing is counted and skipped | -| `verify` | Recomputes every staged file's Dropbox content hash and records SHA-1 and SHA-256. A mismatch is a file edited since listing: removed and counted, never recorded | -| `upload` | One `proton-drive filesystem upload` of the staging tree; Proton skips files whose content it already holds and makes revisions of changed ones | -| `confirm` | The upload summary must account for every verified file plus every folder, and every failure must name a file in the batch. Those alone are recorded as failed; the rest confirm | -| `checkpoint` | Merges the confirmed rows into `mirror_objects` and pushes the state to the bucket, a dated copy first and then the canonical key. Always the last step of a batch, so a killed run repeats at most one | -| `trash` | Only when every planned batch landed. A topmost folder the mirror holds nothing live under goes in one `filesystem trash` call, subtree and all; a folder still holding live files gets its deleted files trashed by name, 50 paths per call. Each unit's `mirror_objects` rows are dropped as it lands. Checkpoints every 50 units, stops at the run budget and chains the next run for the rest | -| `reconcile` | Once the last complete walk is `RECONCILE_HOURS` (168, a week) old, by lib's `due`, or with `RECONCILE=true`: a full Proton walk compared against `mirror_objects`. Rows Proton lacks or mis-sizes are dropped so they re-upload; nodes neither Dropbox nor the state knows are trashed. A walk that does not fit one run resumes on the next, and a partial walk drops and trashes nothing | -| `report` | Builds the step summary from the state alone, finishes the run row, writes the chain marker, and returns the run's status | - -Every step is plan-by-default: `batches`, `trash`, `reconcile`, `report` and -`empty-trash` change anything only with `--apply`, which `task pipeline` passes and -`task plan-pipeline` never does. No mutation is trusted on its exit status. The evidence -has three layers: the upload summary per batch, matched item by item; Proton's own -server-side block hashes, checked at upload and out of this repository's hands; and the -weekly walk, which compares Proton's own listing against the state independently of -anything a batch claimed. - -The run around those steps, from the image and the secrets to the report and the ping, -is the toolbox's and is documented once in -[lib's README](https://github.com/katoptra/lib#the-toolbox). +| `clock` | The `clock` of the toolbox writes the start epoch of the run to `.run/start.txt`, and it deletes the chain marker. Then this step reads the epoch and writes it to `.run/clock.json`. It removes all the files from `.run/staging`, and it deletes the report and the walked marker of the previous run | +| `session` | This is a verb of the proton engine. It gets the encrypted session of the Proton CLI from the bucket, and it puts the session in `.run/session` for each subsequent CLI call. It writes the digest of the session to `.run/session.sha`. After each CLI call, the migrator compares the session with that digest, and it sends a changed session back to the bucket | +| `state` | It gets the state database from the bucket, and it starts the row of the run. If the bucket has no state and no history, the run starts with an empty mirror. If the bucket has history but no state, the run stops, because it must not accept a missing state as an empty mirror | +| `inventory` | It makes a listing of Dropbox with the API, and it commits each page with its cursor. It records an entry with no content hash as non-downloadable, and the mirror does not download that entry. It makes each display path from the name of each folder above the file. The cause is that Dropbox does not always give a parent folder the same uppercase and lowercase letters | +| `delta` | It compares the inventory with `mirror_objects`: the path, the size and the content hash of each file. It rejects a listing that has less than 50% of the number of mirrored files. Thus, a short listing cannot become a trash list | +| `plan` | It rejects a tree that is larger than `ceiling_gb`. It also rejects a batch that the disk cannot contain in staging. It does not include the files that are larger than `max_file_gb`, and it counts them as oversized. It puts the other files in batches of `batch_gb` and `batch_files` | +| `batches` | It does the five steps that follow for each batch. It does not start a batch that can make the run longer than its budget. If it stops at the budget with batches to do, the run has no failure, and the chain starts the next run | +| `fetch` | It downloads the files of the batch from Dropbox into `.run/staging`, at their display paths. If a path is not in Dropbox after the listing, `fetch` counts it and does not download it | +| `verify` | It calculates the Dropbox content hash of each file in staging again, and it records the SHA-1 and the SHA-256. A different hash shows a file that changed after the listing. `verify` removes that file and counts it, and it does not record it | +| `upload` | One `proton-drive filesystem upload` call sends the staging tree, and the CLI writes a summary. The CLI does not upload a file if Proton has the same contents. Proton makes a new revision of a file that changed | +| `confirm` | The summary must include each verified file and each folder. Each failure in the summary must give the name of a file in the batch. `confirm` gives only those files the status `CONFIRM_FAILED`. It gives the other files the status `CONFIRMED` | +| `checkpoint` | It adds the confirmed rows to `mirror_objects`. Then it pushes the state to the bucket: first a dated copy, then the canonical key. It is always the last step of a batch. Thus, if the runner stops a run, the next run does a maximum of one batch again | +| `trash` | It operates only if all the planned batches are in Proton. If a topmost folder has no remaining file of the mirror, one `filesystem trash` call moves that folder to the trash, with its subtree. In a folder that has remaining files, `trash` gives the path of each deleted file to `filesystem trash`, 50 paths in each call. When a unit is complete, `trash` removes its rows from `mirror_objects`. It makes a checkpoint after each 50 units. At the end of the run budget, it stops, and the chain starts the next run for the remaining units | +| `reconcile` | It runs only if `due` wrote `.run/reconcile`: when the last completed walk is `RECONCILE_HOURS` (168, one week) or more before the run, or with `RECONCILE=true`. It does a full walk of Proton and compares it with `mirror_objects`. It removes the row of a file that Proton does not have, or has with a different size. The next run then uploads that file again. It moves a node that is not in Dropbox and not in the state to the trash. A walk that does not complete in one run removes no row and moves no node to the trash, and the next run continues it | +| `report` | It makes the step summary from the state only. It completes the run row, and it writes the chain marker. Its exit code gives the status of the run | + +Without `--apply`, a step only makes a plan. These steps change data only with `--apply`: + +- `batches` +- `trash` +- `reconcile` +- `report` +- `empty-trash`. + +`task pipeline` gives `--apply`, and `task plan-pipeline` does not. For a command that +changes data, an exit code of 0 is not a sufficient indication that the change occurred. +Three layers of checks examine the mirror: + +1. The summary of `upload` for each batch. `confirm` compares each item of the summary with + the batch. +2. The block hashes on the server of Proton. Proton examines them when the CLI uploads a + file, and this repository cannot change that check. +3. The weekly walk. It compares the listing of Proton with the state, independently of the + data that the batches recorded. + +The toolbox supplies the other parts of the run, from the image and the secrets to the +report and the ping. [lib's README](https://github.com/katoptra/lib#the-toolbox) has the +only description of these parts. ## Want your own? ### 1. Fork it -Fork [katoptra/dropbox](https://github.com/katoptra/dropbox). Nothing in `Taskfile.yml` -names an account; [`config/mirror.toml`](config/mirror.toml) is the one behaviour input -and its defaults fit a GitHub runner. The account is the environment, all of it `op.env`. +Fork [katoptra/dropbox](https://github.com/katoptra/dropbox). `Taskfile.yml` contains no +account data. [`config/mirror.toml`](config/mirror.toml) is the only file that sets how the +mirror operates, and its default values are correct for a GitHub runner. The account data is +in the environment, and all of it is from `op.env`. ### 2. Storage -The bucket holds the state and the session, nothing of the mirrored tree. +The bucket contains the state and the session. It contains no file of the mirrored tree. ``` .state/state.sqlite.xz.age the state: evidence tables, mirror_objects, runs, batches, deletions @@ -110,43 +142,55 @@ The bucket holds the state and the session, nothing of the mirrored tree. .state/reconciled the start epoch of the run that last completed a Proton walk, plain text ``` -| What | Why | +| Item | Function | |---|---| -| An R2 bucket, or any S3-compatible bucket | The state and the session | -| An API token with Object Read & Write, scoped to that bucket | The `r2` values in step 3 | -| A lifecycle rule expiring `.state/history/` after 7 days | The bucket has no object versioning; the dated copies are the rollback | +| An R2 bucket, or a different S3-compatible bucket | It contains the state and the session. | +| An API token with Object Read & Write, for that bucket only | It gives the `r2` values in step 3. | +| A lifecycle rule that deletes the objects in `.state/history/` after 7 days | The bucket has no object versioning. The dated copies are the rollback. | -The state holds every mirrored path name, which is why it is encrypted. What the toolbox -and the engines keep in a bucket: [lib, Storage](https://github.com/katoptra/lib#storage). +The state contains the name of each mirrored path. Thus, the mirror encrypts the state. +[lib, Storage](https://github.com/katoptra/lib#storage) gives the keys that the toolbox and +the engines keep in a bucket. ### 3. Secrets -Twelve values, in one vault item named `dropbox`, one section per service: +A run gets twelve values from one vault item, `dropbox`. The item has one section for each +service: -| Section | Field | What it is | Reaches the run as | +| Section | Field | Value | The run gets it as | |---|---|---|---| -| `dropbox` | `app_key`, `app_secret` | The scoped app, step 4 | `MIRROR_DROPBOX_APP_KEY`, `MIRROR_DROPBOX_APP_SECRET` | -| `dropbox` | `refresh_token` | Its permanent token | `MIRROR_DROPBOX_REFRESH_TOKEN` | -| `dropbox` | `account_id` | The `dbid:...` the run must be reading | `MIRROR_DROPBOX_ACCOUNT_ID` | +| `dropbox` | `app_key`, `app_secret` | The scoped app from step 4 | `MIRROR_DROPBOX_APP_KEY`, `MIRROR_DROPBOX_APP_SECRET` | +| `dropbox` | `refresh_token` | The permanent token of the app | `MIRROR_DROPBOX_REFRESH_TOKEN` | +| `dropbox` | `account_id` | The `dbid:...` of the account that the run must read | `MIRROR_DROPBOX_ACCOUNT_ID` | | `proton` | `destination` | The CLI path of the folder, `/my-files/Dropbox` | `MIRROR_PROTON_DESTINATION` | -| `proton` | `destination_uid` | That folder's UID | `MIRROR_PROTON_DESTINATION_UID` | +| `proton` | `destination_uid` | The UID of that folder | `MIRROR_PROTON_DESTINATION_UID` | | `r2` | `access_key_id`, `secret_access_key` | The token from step 2 | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` | | `r2` | `endpoint` | `https://.r2.cloudflarestorage.com` | `AWS_ENDPOINT_URL_S3` | -| `r2` | `bucket` | The bucket's name | `MIRROR_R2_BUCKET` | +| `r2` | `bucket` | The name of the bucket | `MIRROR_R2_BUCKET` | | `age` | `identity` | An `AGE-SECRET-KEY-...` line from `age-keygen` | `MIRROR_AGE_IDENTITY` | -| `healthcheck` | `url` | Optional: a healthchecks.io ping URL | `HEALTHCHECK_URL` | +| `healthcheck` | `url` | A healthchecks.io ping URL. This value is optional. | `HEALTHCHECK_URL` | + +1. Put the UUID of your vault in the references in [`op.env`](op.env). +2. Make a service account that can read that vault. +3. Put the token of the service account in the `OP_SERVICE_ACCOUNT_TOKEN` secret, on the + organization or on the repository. -Put your vault's UUID into the references in [`op.env`](op.env), make a service account -that can read that vault, and store its token as the `OP_SERVICE_ACCOUNT_TOKEN` secret, -on the organization or on the repository. Finding a vault's UUID and why a UUID and not -a name: [lib, Secrets](https://github.com/katoptra/lib#secrets). +Each reference uses the UUID of the vault, not its name. +[lib, Secrets](https://github.com/katoptra/lib#secrets) tells how to find the UUID, and +gives the cause of this rule. ### 4. Dropbox and Proton -**The Dropbox app.** At https://www.dropbox.com/developers/apps create a Scoped access, -Full Dropbox app with only `files.metadata.read` and `files.content.read`; the mirror -can never write to Dropbox. Then a refresh token, which never expires or rotates, and -the account id, which every run checks before trusting a listing: +**The Dropbox app.** Do these steps: + +1. At https://www.dropbox.com/developers/apps, make an app with Scoped access and Full + Dropbox access. +2. Give the app only these permissions: `files.metadata.read` and `files.content.read`. + Then the mirror cannot write to Dropbox. +3. Get a refresh token and the account ID with the commands that follow. + +The refresh token is permanent, and it does not change. Each run makes sure that the token +is for this account ID before it uses a listing. ```sh # 1. approve, with your key filled in, and copy the code: @@ -158,126 +202,195 @@ curl -X POST https://api.dropboxapi.com/2/users/get_current_account -H "Authoriz # 3. the whole dbid:... string is account_id ``` -**The session.** The Proton CLI can only be seeded by a browser sign-in, so the session -is made once on a laptop and carried to every run encrypted; the commands, the folder and -its UID, and `task session-seal -- .run/pd` are in -[lib, The session](https://github.com/katoptra/lib#the-session). Turn telemetry off in -Proton's account settings first. Store the folder's CLI path as `destination` and its -`uid` from the listing as `destination_uid`. This mirror's session is its own. +**The session.** Only a sign-in in a browser can make a session for the Proton CLI. Make the +session one time, on a laptop. Then each run gets the encrypted session from the bucket. +[lib, The session](https://github.com/katoptra/lib#the-session) gives: -### 5. Prove it, run it, schedule it +- The commands +- How to make the folder and find its UID +- `task session-seal -- .run/pd`. -On a laptop with go-task, the 1Password CLI and Docker or Apple `container`: +Put the CLI path of the folder in the field `destination`. Put the `uid` of the folder from +the listing in the field `destination_uid`. This mirror has a session that no other mirror +uses. -```sh -task test # the pytest suite, offline, inside the image -task check # every pipeline command rendered inside the image, diffed against render.txt -task plan # the real thing, read-only: the state, the Dropbox listing, the plan it would move -``` +### 5. Do the checks, run it, schedule it + +1. Use a laptop with these tools: + - go-task + - The 1Password CLI + - Docker or Apple `container`. +2. Run these commands: + + ```sh + task run -- task offline # pytest, ruff check and ruff format --check: in the image, offline + task check # render each command of the pipeline in the image, then compare it with render.txt + task plan # read-only, with the real account and bucket: the state, the Dropbox listing, the plan of a sync + ``` + +3. Before the first run, pause the healthcheck + ([lib, Monitoring](https://github.com/katoptra/lib#monitoring)). +4. In Actions, select the sync workflow. +5. Select **Run workflow**. + +The first run finds no state and no history, and it uses the full tree as the delta. Then +the chain starts one run after the other, until the full tree is in Proton. Each run does +these steps: + +- At the end of its budget, it starts no new batch. +- It makes a checkpoint of the batches that are in Proton. +- It starts the next run through the chain. + +Each run pulls the image one time and makes one listing. Thus, the default budget is long: +335 minutes. The job timeout in `sync.yml` (`timeout-minutes`) is 355 minutes, and the limit +of GitHub for a job is six hours. The chain puts a tree of 200,000 files in Proton in +approximately 45 hours, with approximately sixteen runs. + +No part of this repository starts a run. To start runs at set times, use one of these two +methods: + +- Add a `schedule:` trigger to `.github/workflows/sync.yml`. +- Dispatch the workflow from an external scheduler. This mirror uses this method. + +If a chained run is in progress, a scheduled run waits for it. -Then Actions, sync, Run workflow. The first run finds no state and no history, treats -the whole tree as the delta, and chains itself run after run until the tree is mirrored: -each run stops starting batches at its budget, checkpoints what landed, and queues the -next. The default budget is 335 minutes under a 355-minute job timeout, just under -GitHub's six-hour limit, because every run pays one image pull and one listing. A -200,000-file tree is about 45 hours, roughly sixteen chained runs. +After three nightly runs with no failure, do a test of the mirror: -Nothing in this repository schedules a run: add a `schedule:` trigger to -`.github/workflows/sync.yml`, or dispatch it from outside as this mirror is. A scheduled -run queues behind a chained one. After three green nights, edit one file and delete one -in Dropbox and confirm both appear in Proton the next morning. +1. Edit one file in Dropbox. +2. Delete one different file in Dropbox. +3. After the next nightly run, make sure that Proton has the two changes. ## Operating it -`task` alone prints the menu, grouped by effect. Everything runs inside the toolbox. +`task` with no task name prints the menu. The menu puts the tasks in groups, one group for +each effect. Each task operates in the toolbox. ```sh -task plan # read-only: the state, the listing, what a sync would move -task status # counts and the last run's figures from the state in the bucket -task test && task lint # pytest; ruff check and format check -task sync # one budgeted run, the same thing Actions runs +task plan # read-only: the state, the listing, and the files that a sync would move +task status # counts, and the figures of the last run, from the state in the bucket +task run -- task offline # pytest, ruff check and ruff format --check +task sync # one run with a budget, the same as a run in Actions task sync -- RUN_BUDGET_MIN=30 # a shorter budget -task sync -- RECONCILE=true # force the weekly Proton walk (true, false or auto) -task state-rollback # list the dated history objects +task sync -- RECONCILE=true # the weekly Proton walk, in this run (true, false or auto) +task state-rollback # a list of the dated history objects task state-rollback -- # copy one of them over the canonical state -task session-seal -- .run/pd # encrypt a laptop Proton CLI session into the bucket -task empty-trash # permanently delete Proton's trash; asks first; never scheduled +task session-seal -- .run/pd # encrypt a laptop session of the Proton CLI into the bucket +task empty-trash # delete the trash of Proton permanently, after a prompt. No schedule starts it +gh workflow run sync.yml # one run in Actions ``` -Never run `task sync` or `task empty-trash` from a laptop while an Actions run may be in -progress: both hold the one Proton session. `task sync` from a laptop is exactly one -chained run, and the next one, wherever it runs, picks up from the state in the bucket. - -**Reading a run.** The step summary is built from the state alone, so `task status` on a -laptop shows the same figures as the Actions page, and it carries counts only, never a -path name: mirror status (inventory, mirrored, percent, oversized, batches and runs -remaining); this run (budget used, batches, files fetched, vanished, mismatched, -uploaded, skipped, confirmed, trashed); throughput; throttling per provider; errors by -class; the last reconcile walk; every phase's status. Error text lives in the encrypted -state's `events` table, readable after `task status` from `.run/state.sqlite`; log rows -older than seven days are pruned, the same clock as the bucket's history copies. - -**Runbook.** - -- **`login first` in the events, or a failure at the first Proton call.** The session is - gone. Sign in again on a laptop and `task session-seal -- .run/pd`. -- **`configured Proton destination did not resolve to exactly one folder`, or `did not - exactly match the listing`.** The folder is not a direct child of its parent, or its - UID differs from the vault's. List the parent and fix the folder or the field. -- **`MIRROR_DROPBOX_ACCOUNT_ID ... must be a full dbid: identifier`.** The field is empty - or `op.env` names the wrong one. -- **`state object is missing but history exists`.** `task state-rollback`; never delete - the history to make a run start fresh. -- **The state looks wrong after a run.** `task state-rollback -- ` copies a dated - copy over the canonical state; re-uploads skip content Proton already holds. -- **Files confirm failed.** Proton refused those uploads, usually a passing server error; - the text is in the state under the batch item. The next run re-uploads them. -- **A run stops on budget every night.** Lower `batch_files` or `batch_gb`; the throughput - rows say which. A run that checkpointed nothing does not chain and fails instead. -- **Proton 429s.** The throttling table is the gauge; lower `batch_gb`. -- **A weekly reconcile does not finish in one run.** Normal on a large tree; only a - complete walk is recorded, so the next run is due again and resumes it. -- **A flag seems ignored.** Flags go after the double dash; before it they set a host-side - var that never reaches the container. `RECONCILE` takes `true`, `false` or `auto`. -- **A case-only rename in Dropbox does not reach Proton.** Files are keyed by lowercased - path. Rename to something else and back if the case matters. -- **Move the folder in Proton.** Move it anywhere under My files and change the vault's - `destination`; the UID survives and every run verifies it. -- **Switch the Dropbox account.** New `refresh_token` and `account_id` fields. The next - run trashes what the old account had and mirrors the new tree; to start clean, empty - the Proton folder and delete the state and its history first. +Do not run `task sync`, `task plan` or `task empty-trash` on a laptop while an Actions run +can be in progress. If the laptop and the run use the session at the same time, a new login +can be necessary ([lib, The session](https://github.com/katoptra/lib#the-session)). + +On a laptop, `task sync` does one run, and it does not start the next run. The next run +continues from the state in the bucket, on a laptop or in Actions. + +**The run summary.** The mirror makes the step summary from the state only. Thus, on a +laptop, `task status` shows the same numbers as the Actions page. The summary contains only +counts, and no path name. It has these parts: + +- The status of the mirror: the inventory, the mirrored files, the percent, the oversized + files, and the remaining batches and runs +- This run: the budget that it used, the batches, and the files in each of these groups: + fetched, vanished, mismatched, uploaded, skipped, confirmed and trashed +- The throughput +- The throttling of each provider +- The errors, for each class +- The last reconcile walk +- The status of each phase. + +The text of each error is in the `events` table of the encrypted state. After +`task status`, you can read it in `.run/state.sqlite`. The mirror deletes the log rows after +seven days. The lifecycle rule of the history copies in the bucket has the same limit. + +**The runbook.** If a run has a problem, find it in this list: + +- **`login first` in the `events` table, or a failure at the first Proton call.** Proton + does not accept the session. Do a new login on a laptop. Then run + `task session-seal -- .run/pd`. +- **`configured Proton destination did not resolve to exactly one folder`, or + `did not exactly match the listing`.** The folder is not directly in its parent folder, or + its UID is different from the UID in the vault. Get the listing of the parent folder. Then + correct the folder or the field. +- **`MIRROR_DROPBOX_ACCOUNT_ID ... must be a full dbid: identifier`.** The field is empty, + or the reference in `op.env` is for an incorrect field. +- **`state object is missing but history exists`.** Run `task state-rollback`. Do not delete + the history to make a run start with no state. With no state and no history, a run starts + with an empty mirror. +- **The state looks incorrect after a run.** Run `task state-rollback -- `. It puts a + dated copy at the key of the canonical state. Then the CLI does not upload again a file + that Proton has. +- **Files have the status `CONFIRM_FAILED`.** Proton did not accept these files. The cause + is usually a temporary error on the server. The text of the error is in the state, at the + batch item. The next run uploads the files again. +- **Each nightly run stops at the end of its budget.** Decrease `batch_files` or + `batch_gb`. The throughput rows show which one. If a run made no checkpoint, it does not + start the next run, and it stops with a failure. +- **Proton sends 429 errors.** The throttling table shows how many. Decrease `batch_gb`. +- **A weekly reconcile does not complete in one run.** This is usual for a large tree. The + mirror records only a completed walk. Thus, the next run must also do a walk, and it + continues the walk. +- **A flag has no effect.** Put the flags after the double dash. Before it, a flag sets a + var on the host, and that var does not go into the container. `RECONCILE` accepts `true`, + `false` or `auto`. +- **Proton does not show a new name that changes only uppercase and lowercase letters.** + The key of each file is its path in lowercase letters. If the letters are important, give + the file a different name in Dropbox. After the next run, give the file its name again. +- **Move the folder in Proton.** Move it to a different location in My files. Then change + `destination` in the vault. The UID stays the same, and each run makes sure that the UID + is correct. +- **Change the Dropbox account.** Put new values in the fields `refresh_token` and + `account_id`. The next run moves the files of the previous account to the trash, and it + uploads the new tree. To start with no data, do these steps before the next run: + 1. Remove all the files from the Proton folder. + 2. Delete the state and its history. +- **The run did not start.** This repository does not start runs. Examine the scheduler + first ([katoptra/dispatch](https://github.com/katoptra/dispatch#when-something-goes-wrong)). + Then use `gh workflow list --all`. For the sync workflow, it shows `active`, or + `disabled_manually` if a person disabled it. Until you find the cause, start each run with + `gh workflow run sync.yml`. ## Reference -**Configuration.** [`config/mirror.toml`](config/mirror.toml) is strict and rejects -unknown keys. It names no account. +**Configuration.** [`config/mirror.toml`](config/mirror.toml) is strict: it rejects a key +that it does not know. It contains no account identifier. -| Key | Meaning | +| Key | Function | |---|---| -| `mirror.id` | Name recorded in the state | -| `dropbox.root` | Subtree to mirror; empty means the whole Dropbox | -| `dropbox.page_limit`, `minimum_call_interval_seconds` | Listing page size and the serialised call spacing | -| `dropbox.download_workers` | Files fetched in flight during `fetch` | -| `budget.batch_gb`, `batch_files` | A batch's byte and file caps; a file over `batch_gb` is a batch by itself | -| `budget.max_file_gb` | Largest file a plan will stage; bigger files are counted as oversized | -| `budget.run_budget_minutes` | Wall-clock budget from the run's start; `sync.yml`'s timeout stays 20 minutes above it | -| `budget.ceiling_gb` | Refuse a Dropbox tree larger than this | -| `budget.disk_headroom_gb` | Free disk the runner must keep beyond a batch's staging | -| `budget.listing_floor_ratio` | Refuse a listing smaller than this share of the mirrored file count | -| `proton.walk_workers` | Folder listings in flight during the reconcile walk, each from its own copy of the session | - -The three account identifiers in `op.env` override the TOML keys -`dropbox.expected_account_id`, `proton.destination` and `proton.expected_destination_uid`, -which exist for a private fork that prefers a file. `MIRROR_VERBOSE=1` prints an error's -full text instead of its class. - -**What sits where.** The mirrored tree includes personal documents. During a batch its -files sit decrypted on the runner's ephemeral disk and in memory, which is inherent: -Dropbox serves plaintext and Proton encrypts client-side inside the CLI. What bounds it: -the runner is a single-tenant VM destroyed after the job; logs and the step summary carry -counts, never names; no workflow artifact is ever uploaded; the state, which holds every -path name, is age-encrypted at rest; the Dropbox credentials cannot write, the bucket -token reaches one bucket, and the service account reads one vault. +| `mirror.id` | The name of the mirror that the state records | +| `dropbox.root` | The subtree of Dropbox for the mirror. With an empty value, the mirror contains all of Dropbox | +| `dropbox.page_limit`, `minimum_call_interval_seconds` | The number of entries on a page of the listing, and the minimum interval between two Dropbox API calls. All the workers obey this one interval | +| `dropbox.download_workers` | The number of files that `fetch` downloads at the same time | +| `budget.batch_gb`, `batch_files` | The maximum GB and the maximum number of files in a batch. A file larger than `batch_gb` is a batch with only that file | +| `budget.max_file_gb` | The largest file that a plan puts in staging. The plan counts a larger file as oversized | +| `budget.run_budget_minutes` | The budget of the run, in minutes of clock time from its start. The timeout in `sync.yml` stays 20 minutes more than this value | +| `budget.ceiling_gb` | The plan rejects a Dropbox tree that is larger than this value | +| `budget.disk_headroom_gb` | The free disk that the runner must keep after it puts a batch in staging | +| `budget.listing_floor_ratio` | If the number of files in a listing is less than this value multiplied by the number of mirrored files, `delta` rejects the listing | +| `proton.walk_workers` | The number of folder listings at the same time in the reconcile walk. Each worker uses a copy of the session that no other worker uses | + +The three account identifiers in `op.env` override these TOML keys: + +- `dropbox.expected_account_id` +- `proton.destination` +- `proton.expected_destination_uid`. + +These keys are for a private fork that keeps the identifiers in a file. With +`MIRROR_VERBOSE=1`, a run prints the full text of an error, not only its class. + +**Where the data is.** The mirrored tree contains personal documents. During a batch, its +files are on the temporary disk of the runner and in memory, with no encryption. This is +necessary, because Dropbox sends plaintext, and the CLI encrypts each file on the client. +These items put limits on the risk: + +- The runner is a VM for one job only. GitHub deletes it after the job. +- The logs and the step summary contain counts, and no names. +- No run uploads a workflow artifact. +- The state contains each path name. The mirror encrypts it with age before it pushes it to + the bucket. +- The Dropbox credentials cannot write. The bucket token gives access to one bucket only. + The service account can read one vault only. Pull requests are welcome. From 44f8dada50a73d6f5f0765771292dcd603d949f0 Mon Sep 17 00:00:00 2001 From: Josh Vaughen Date: Thu, 8 Oct 2026 18:08:40 -0700 Subject: [PATCH 3/3] docs(readme): name only the verbs that use the session --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 572d637..891d2e4 100644 --- a/README.md +++ b/README.md @@ -279,8 +279,8 @@ task empty-trash # delete the trash of Proton permanently, aft gh workflow run sync.yml # one run in Actions ``` -Do not run `task sync`, `task plan` or `task empty-trash` on a laptop while an Actions run -can be in progress. If the laptop and the run use the session at the same time, a new login +Do not run `task sync` or `task empty-trash` on a laptop while an Actions run can be in +progress. If the laptop and the run use the session at the same time, a new login can be necessary ([lib, The session](https://github.com/katoptra/lib#the-session)). On a laptop, `task sync` does one run, and it does not start the next run. The next run