From f8987909ad9bdd99c082cae230e70372c62efdaa Mon Sep 17 00:00:00 2001 From: Josh Vaughen Date: Thu, 8 Oct 2026 16:07:59 -0700 Subject: [PATCH 01/12] docs(readme): rewrite the manual in ASD-STE100, with lib's items --- README.md | 1494 +++++++++++++++++++++++++++++++++-------------------- 1 file changed, 935 insertions(+), 559 deletions(-) diff --git a/README.md b/README.md index e1e0185..6715a71 100644 --- a/README.md +++ b/README.md @@ -9,30 +9,37 @@

lib

-

The toolbox every katoptra mirror includes by URL.

+

The toolbox that all katoptra mirrors include from a URL.

ci license

-The rule it enforces: the code that -starts a run, contains it, resolves its secrets, checks it and reports it lives here, -once. The code that moves bytes for a transport lives here too, once per engine: -`engines/rsync.yml` for an rsync upstream into a bucket, `engines/proton.yml` for a -staging tree into Proton Drive. A mirror holds only its identity, the hooks it fills, and -the few verbs no other mirror shares. +This repository has one rule: the code that all mirrors use is here, one time. This code +does these tasks: -This file is the manual. A mirror's README says what it mirrors and how to fork it, and -links here for everything the mirrors share. Read it top to bottom once and the system -should hold no surprises; after that, each section stands on its own. +- It starts each run in a container. +- It gets the secrets of the run. +- It does checks on the pipeline, and it reports each run. +- It moves the bytes. Each transport has one engine here: + - `engines/rsync.yml` copies an rsync upstream into a bucket. + - `engines/proton.yml` copies a staging tree into Proton Drive. -## The system in one picture +A mirror contains only its identity, the hooks that it fills, and the verbs that no other +mirror uses. + +This file is the manual. The README of a mirror tells which upstream the mirror copies, and +how to fork it. For each part that all mirrors use, it has a link to this file. Read this +file one time, from the start to the end. Then you will know the full system. After that, +you can read each section without the other sections. + +## The system in one diagram ```mermaid flowchart LR subgraph up["Upstreams"] - u1["CTAN, over rsync"] + u1["CTAN, GNU and Savannah, over rsync"] u2["GitHub, over the API and git"] u3["Dropbox, over the API"] end @@ -50,12 +57,22 @@ flowchart LR box --> hc["healthchecks.io, one ping per run"] --> site["katoptra.org, live status"] ``` -Every mirror is that picture with one upstream and one sink. A public mirror (ctan, -tlnet) copies an rsync tree into a bucket and serves the bucket; the bucket is the mirror. -A private mirror (github, dropbox) copies an account into Proton Drive and keeps only -what the next run needs in the bucket. Both run the same way: a job on GitHub Actions, -started by a dispatch, that pulls one image, resolves its secrets by name from a vault, -and runs `task pipeline` inside the image. The same command runs on a laptop. +Each mirror has the shape of this diagram, with one upstream and one sink. There are two +types of mirror: + +- A public mirror (ctan, tlnet, gnu and nongnu) copies an rsync tree into a bucket, and a + domain serves the bucket. The bucket is the mirror. +- A private mirror (github and dropbox) copies an account into Proton Drive. Its bucket + contains only the data that the next run uses. + +The two types of mirror do a run with the same steps: + +1. A dispatch starts a job on GitHub Actions. +2. The job downloads one image. +3. The job gets the secrets of the mirror by name, from a vault. +4. The job runs `task pipeline` in the image. + +The same command runs on a laptop. ## The layers @@ -80,17 +97,18 @@ flowchart TB wf -- "task sync, task check" --> tb ``` -| Layer | Lives in | Owns | A mirror touches it when | +| Layer | Location | Contents | Does a mirror change it? | |---|---|---|---| -| Toolbox | `toolbox.yml` | How a run starts, is contained, resolves secrets, is rendered, checked and reported | Never; it is included as is | -| Engine | `engines/.yml` | How bytes move for one transport, and the pipeline's order | It lists a hook under `excludes:` and defines its own | -| Image | `docker/`, `toolchain.lock.toml` | Every tool a run needs, pinned by checksum | Never; its engine names one, or it names one in `IMAGE` | -| Workflows | `.github/workflows`, `.github/actions/toolbox` | How Actions installs the tools and runs `task sync` and `task check` | Never; its callers are a few lines each | -| Mirror | The mirror's own repository | Identity, the hooks, the exceptions | Always; this is all it holds | +| Toolbox | `toolbox.yml` | How a run starts, gets its secrets and stays in its container. The render, the check and the report of a run | No. A mirror includes it with no change | +| Engine | `engines/.yml` | How bytes move for one transport, and the sequence of the pipeline | Yes, to fill a hook: it puts the hook in `excludes:`, and it writes the hook | +| Image | `docker/`, `toolchain.lock.toml` | All the tools that a run uses, each pinned by its checksum | No. The engine gives the image, or the mirror gives it in `IMAGE` | +| Workflows | `.github/workflows`, `.github/actions/toolbox` | How Actions installs the tools and runs `task sync` and `task check` | No. Each caller in a mirror has only a small number of lines | +| Mirror | The repository of the mirror | The identity, the hooks, and the verbs that no other mirror uses | Yes, always. These items are all that a mirror contains | -## What a run looks like +## The steps of a run -The same path on a laptop and in Actions. On a laptop it starts at `task sync`. +A run has the same steps on a laptop and in Actions. On a laptop, the run starts at +`task sync`. ```mermaid sequenceDiagram @@ -99,7 +117,7 @@ sequenceDiagram participant H as Host: task participant O as op run participant C as Container: task - D->>W: workflow_dispatch, vars + D->>W: workflow_dispatch of the mirror's caller, no inputs W->>W: toolbox action installs task and op from the lock W->>H: task sync -- RECONCILE=... MAX_BATCHES=... H->>H: image: pull IMAGE if missing @@ -113,31 +131,34 @@ sequenceDiagram else the runner cut the run off W->>C: task op -- task failed else .run/chain exists - W->>D: gh workflow run, the caller's own file + W->>W: gh workflow run, the caller's own file: the next run end ``` -Four things the diagram hides: - -- **Secrets cross by name, never by value.** `op run` resolves `op.env` on the host and - exports the values; `run` passes `-e NAME` for each name in `op.env` and in `PASS`, - plus `HEALTHCHECK_URL`, `GITHUB_STEP_SUMMARY` and `GITHUB_RUN_ID` always, so no value - ever appears on a command line or in a log. The whole path is under [Secrets](#secrets). -- **The vault is read once per run.** One `op run` wraps the whole pipeline, and the - failure path, `failed`, runs inside the same container, so a failed run costs no - second read. The one exception is a run the runner cut off, by the timeout or a - cancellation, which never reached its own failure path: the workflow runs `failed` - again from outside, one more read on a rare day. -- **Nothing inside the container reaches the network for task itself.** The image sets - `TASK_REMOTE_OFFLINE=1` and the mirror's `.task/remote` cache rides in with the repo. -- **`plan` is `sync` with the read-only half.** It runs `plan-pipeline` instead of - `pipeline`, inside the same image with the same secrets. +The diagram does not show these four items: + +- **Secrets go into the container by name, not by value.** On the host, `op run` gets the + value of each reference in `op.env`, and it exports the values. For each name in `op.env` + and in `PASS`, `run` gives `-e NAME` to the container. It also always gives + `HEALTHCHECK_URL`, `GITHUB_STEP_SUMMARY` and `GITHUB_RUN_ID`. Thus, no value shows on a + command line or in a log. [Secrets](#secrets) gives the full path. +- **A run reads the vault one time.** One `op run` command runs all of the pipeline, and + the failure path, `failed`, runs in the same container. Thus, a run with a failure does + not read the vault a second time. If the runner stops a run, at the timeout or because a + person cancels it, the run does not start its failure path. Then the workflow runs + `failed` again from the host, and this reads the vault one more time. This condition does + not occur frequently. +- **In the container, task does not use the network for its Taskfiles.** The image sets + `TASK_REMOTE_OFFLINE=1`. The `.task/remote` cache of the mirror goes into the container + with the repository. +- **`plan` is `sync` with only the read-only half.** It runs `plan-pipeline`, not + `pipeline`, in the same image and with the same secrets. ## The toolbox -`toolbox.yml` is every verb a mirror has whatever moves its bytes. Half of its verbs run -on the host and wrap the container; the other half run inside it and are the first and -last steps of every pipeline. +`toolbox.yml` contains the verbs that all mirrors have, with all engines. Half of these +verbs run on the host and start the container. The other half run in the container: they +are the first steps and the last steps of each pipeline. ```mermaid flowchart TB @@ -157,106 +178,136 @@ flowchart TB ### Host side -The contract the workflows call and the menu documents. A mirror does not override these. +These verbs are the contract that the workflows use and that the menu shows. A mirror does +not override them. -| Verb | Does | +| Verb | Function | |---|---| -| `default` | The grouped menu, from `NAME`, `DESC` and `MENU` | -| `sync -- [K=v ...]` | `op -- task pipeline`: one run. The words after `--` are task vars for the pipeline inside the image | +| `default` | Prints the menu in groups, from `NAME`, `DESC` and `MENU` | +| `sync -- [K=v ...]` | `op -- task pipeline`: one run. The words after `--` are task vars for the pipeline in the image | | `plan -- [K=v ...]` | `op -- task plan-pipeline`: the read-only half | -| `check` | `render`, then diff against `render.txt` | -| `render-update` | `render`, then accept it as `render.txt` | -| `render` | `task --dry --force pipeline` inside the image, over an empty `.run`, saved to `.run/render.txt`; no secrets cross | -| `run -- ` | Anything inside the image, with the repository at `/work` | -| `op -- ` | `run`, wrapped in `op run --env-file=op.env` when the file exists | -| `image` | Pull `IMAGE`; a no-op while it exists locally | -| `image-build` | Build `IMAGE` from `LIB_DIR/docker/.Dockerfile`; `LIB_DIR` defaults to `../lib` | -| `image-clean` | Remove `IMAGE` | -| `clean` | Delete `.run`, `staging` and the Taskfile cache; nothing else | - -The container engine is Apple `container` when its daemon is up, else Docker; -`ENGINE=docker` forces Docker. The host needs go-task, the container engine, and the -1Password CLI for anything that reads the vault. Nothing else is installed on a host. - -### Inside the image - -The first and last verbs of every pipeline. A mirror or engine may override these when it -keeps its own. - -| Verb | Does | +| `check` | `render`, then a diff against `render.txt` | +| `render-update` | `render`, then it copies the render to `render.txt` | +| `render` | `task --dry --force pipeline` in the image, with an empty `.run`. It writes the result to `.run/render.txt`. No secret goes into the container | +| `run -- ` | Runs a command in the image, with the repository at `/work` | +| `op -- ` | `run`, in `op run --env-file=op.env` if the repository has `op.env` | +| `image` | Downloads `IMAGE`. While the host has the image, it does not download it again | +| `image-build` | Builds `IMAGE` from `LIB_DIR/docker/.Dockerfile`. The default `LIB_DIR` is `../lib` | +| `image-clean` | Removes `IMAGE` | +| `clean` | Deletes `.run`, `staging` and the Taskfile cache, and no other file | + +The container runtime (`ENGINE`) is Apple `container` when its daemon operates. If not, it +is Docker. `ENGINE=docker` selects Docker. A host has only these tools: + +- go-task +- The container runtime +- The 1Password CLI, for a task that reads the vault. + +### In the image + +These verbs are the first verbs and the last verbs of each pipeline. A mirror or an engine +can override one of them with a verb of the same name. + +| Verb | Function | |---|---| -| `clock` | Write the run's start epoch to `.run/start.txt` and forget the last run's chain file and `.run/reconcile` | -| `due` | Decide whether this run reconciles, by the age of the last reconcile at `.state/reconciled`; leave `.run/reconcile` when it does. `due-rule` is the rule alone | -| `reconciled` | Record this run's start at `.state/reconciled`; an engine runs it once a reconcile completes | -| `report` | Append the run summary to the Actions job page (stdout elsewhere): its own rows, then `report-engine`'s, then `report-mirror`'s | -| `report-mirror` | Hook: a no-op here; the mirror's rows | -| `ping` | GET `HEALTHCHECK_URL`; skipped when unset | -| `ping-fail` | GET `HEALTHCHECK_URL/fail`; skipped when unset | +| `clock` | Writes the start epoch of the run to `.run/start.txt`. It deletes the chain file and the `.run/reconcile` of the previous run | +| `due` | Finds if this run does a reconcile, from the time of the last reconcile in `.state/reconciled`. If it does, `due` writes `.run/reconcile`. `due-rule` is the rule with no download | +| `reconciled` | Records the start of this run in `.state/reconciled`. An engine runs it when a reconcile is completed | +| `report` | Adds the run summary to the job page in Actions. If it does not run in Actions, it writes the summary to stdout. The rows of the toolbox go first, then the rows of `report-engine`, then the rows of `report-mirror` | +| `report-mirror` | A hook, empty here, for the rows of the mirror | +| `ping` | Sends a GET to `HEALTHCHECK_URL`. If `HEALTHCHECK_URL` is empty, it sends no request | +| `ping-fail` | Sends a GET to `HEALTHCHECK_URL/fail`. If `HEALTHCHECK_URL` is empty, it sends no request | | `failed` | The failure path: `report STATUS=failed`, then `ping-fail` | -`sync` runs `pipeline`, and on a failure `failed`, all in the one container. The report -is one table in three layers, each a `| Label | value |` line appended to the same file: -the toolbox's rows (when the run started and how long it took, the image, whether the -next run is queued), the engine's, the mirror's. Every row tolerates a missing file, -because the report also runs after a failed pipeline. +`sync` runs `pipeline` in one container. If the pipeline has a failure, `sync` runs +`failed` in the same container. The report is one table with three layers. Each row is a +`| Label | value |` line that `report` adds to the same file. The rows are in this +sequence: + +1. The rows of the toolbox: the start time and the length of the run, the image, and the + time of the next run (through the chain, or at the schedule) +2. The rows of the engine +3. The rows of the mirror. + +Each row accepts a missing file, because the report also runs after a pipeline with a +failure. #### Reconcile, by age -A reconcile, the pass that checks the mirror against its destination rather than -against its own state, is due by the age of the last one and never by when a run starts: -the schedule lives outside the mirror and can move. `due` runs early in a pipeline that -reconciles, and the pipeline's reconcile runs only when `.run/reconcile` is there. +A reconcile compares the mirror with its destination, not with its state. The time of the +last reconcile sets when the next reconcile is necessary. The start time of a run does not +set it, because the schedule is not in the mirror and can change. A pipeline that does a +reconcile runs `due` near its start. Its reconcile step runs only if `.run/reconcile` is +there. -| Var | Default | Meaning | +| Var | Default | Function | |---|---|---| -| `RECONCILE` | `auto` | Set per run: `task sync -- RECONCILE=true`. `true` reconciles, `false` does not, `auto` reconciles once the last is `RECONCILE_HOURS` old or none is on record | -| `RECONCILE_HOURS` | 24 | Whole hours between reconciles. A mirror that wants another interval sets it in its root vars | +| `RECONCILE` | `auto` | Each run can set it: `task sync -- RECONCILE=true`. With `true`, the run does a reconcile. With `false`, it does not. With `auto`, it does a reconcile if there is no record of a reconcile. It also does one if the last reconcile occurred `RECONCILE_HOURS` or more before the run | +| `RECONCILE_HOURS` | 24 | The number of full hours between two reconciles. For a different interval, a mirror sets it in its root vars | + +`.state/reconciled` contains the start epoch of the run that completed the last reconcile. +The `pull` and `push` of the engine read and write this file. Thus, with the rsync +engine, the file has xz compression, as the state has. -The last reconcile is the start epoch of the run that finished it, at `.state/reconciled`, -kept through the engine's `pull` and `push`, so under the rsync engine it is -xz-compressed like the state. The mark is half an hour short of `RECONCILE_HOURS`: a -daily run starts seconds either side of 24 h after the one before, and a strict 24 h -would reconcile every other day. A run that fails before its reconcile completes -records nothing, so the next run is due as well, and so does an rsync run with batches -still waiting: its reconcile holds for the chained run that lands the last one. A mirror with no engine that -reconciles defines `pull` and `push` itself. +The limit is half an hour less than `RECONCILE_HOURS`. A daily run starts some seconds +before or after the time that is 24 h after the previous run. With a limit of 24 h and no +less, a reconcile occurs only on each second day. -### What a mirror gives the toolbox +In these conditions, a run records no reconcile, and the reconcile is also necessary at the +next run: -| Name | Kind | Meaning | +- The run stops with a failure before its reconcile is completed. +- An rsync run has batches that wait for a subsequent run. Its reconcile waits for the + chained run that uploads the last batch. + +A mirror with no engine that does a reconcile writes `pull` and `push` in its Taskfile. + +### The parts that a mirror supplies + +| Name | Type | Function | |---|---|---| -| `NAME`, `DESC` | include vars, required | The menu's title and line | -| `IMAGE` | include var, required unless the engine names it | The image the run happens inside. The include's value wins over the engine's | -| `pipeline`, `plan-pipeline` | tasks, required | The full run and its read-only half, inside the image. An engine supplies both; a mirror with no engine defines them | -| `report-engine` | task, required | The engine's rows of the run summary, which `report` calls. An engine supplies it; a mirror with no engine defines it, with no commands when it has no rows | -| `op.env` | file | `op://` references, one per secret. Absent means the environment is already resolved: on a laptop, whatever is exported; in Actions, the repository's secrets, crossing by the names in `PASS` | -| `PASS` | include var | Host environment names that cross into the container beside the ones in `op.env`. Task vars are not environment: they go after `--` | -| `MENU` | include var | Extra lines for the menu, one per mirror-specific verb | -| `report-mirror` | task | The mirror's rows of the run summary. Excluded on the toolbox include | -| `LIB_DIR` | include var | Where `image-build` finds `docker/`; defaults to `../lib` | -| `RECONCILE_HOURS` | root var | Hours between reconciles, when not 24. See [Reconcile, by age](#reconcile-by-age) | -| `pull`, `push` | tasks | What `due` and `reconciled` keep `.state/reconciled` through. An engine supplies both; a mirror with no engine that reconciles defines them | -| `excludes:` | include key | Library verbs the mirror replaces. See [Changing it](#changing-it) | +| `NAME`, `DESC` | Include vars, necessary | The title and the line of the menu | +| `IMAGE` | Include var, necessary if the engine does not give it | The image that the run occurs in. If the include and the engine each give a value, the run uses the value on the include | +| `pipeline`, `plan-pipeline` | Tasks, necessary | The full run and its read-only half, in the image. An engine supplies the two tasks. A mirror with no engine, or that puts them in `excludes:`, writes them | +| `report-engine` | Task, necessary | The rows of the engine in the run summary. `report` runs it. An engine supplies it. A mirror with no engine, or that puts it in `excludes:`, writes it, with no commands if it has no rows | +| `op.env` | File | `op://` references, one for each secret. With no `op.env`, the environment has the values before the run starts. On a laptop, these are the exported values. In Actions, the secrets of the caller go into the container by the names in `PASS` | +| `PASS` | Include var | The names of host environment variables that go into the container. The names in `op.env` go in too. Task vars are not environment variables: they go after `--` | +| `MENU` | Include var | More lines for the menu, one for each verb that only this mirror has | +| `report-mirror` | Task | The rows of the mirror in the run summary. The mirror puts it in `excludes:` on the toolbox include | +| `LIB_DIR` | Include var | The location where `image-build` finds `docker/`. The default is `../lib` | +| `RECONCILE_HOURS` | Root var | The hours between two reconciles, if not 24. Refer to [Reconcile, by age](#reconcile-by-age) | +| `pull`, `push` | Tasks | The tasks that `due` and `reconciled` use to read and write `.state/reconciled`. An engine supplies the two tasks. A mirror with no engine that does a reconcile writes them | +| `excludes:` | Include key | The verbs of lib that the mirror overrides. Refer to [Changing it](#changing-it) | ## The engines -An engine is a second include with the verbs for one transport. Its verbs are the -pipeline vocabulary, reserved across every engine so a mirror's pipeline reads the same -whichever transport it uses. An engine reads the mirror's root vars and the environment; -it declares no `vars:` default for anything a mirror owns, because a value there would -shadow the mirror's. Every tunable is an inline `{{.X | default N}}`. +An engine is a second include, with the verbs for one transport. The names of these verbs +are the same in all engines, and a mirror cannot use these names for a different verb. Thus, +the pipeline of a mirror has the same verbs with all transports. + +An engine reads the root vars of the mirror and the environment. It has no `vars:` default +for a var that a mirror sets, because the run then uses the default, not the value of the +mirror. Each tunable is an inline `{{.X | default N}}`. -The hooks are the verbs an engine leaves empty for a mirror to fill. A mirror lists each -it defines under `excludes:` on the engine's include. Extension is a hook, never a copy: -a mirror that needs the engine's `smoke` and more defines `smoke-mirror`, which `smoke` -runs last. +`IMAGE` is different. The engine sets it in `vars:` as `{{.IMAGE | default "..."}}`. Thus, +the run uses an `IMAGE` on the toolbox include, not the default of the engine. If a mirror +sets `IMAGE` in its root vars, the run does not always use that value. + +The hooks are the verbs that an engine keeps empty, for a mirror to fill. For each hook +that a mirror writes, the mirror puts the hook in `excludes:` on the include of the engine. +To extend a verb, use a hook, not a copy. For example, a mirror that wants the `smoke` of +the engine and more checks writes `smoke-mirror`. `smoke` runs `smoke-mirror` last. ### The rsync engine -`engines/rsync.yml`: an rsync upstream into an S3 bucket, as a list diff and never a -local tree. The runner has 14 GB of disk and CTAN has 140; a run lists upstream, diffs -the listing against the state file the last run left in the bucket, and moves only the -delta, in batches, each committed before the next starts. +`engines/rsync.yml` copies an rsync upstream into an S3 bucket. It compares two listings, +and it does not keep a local copy of the tree. The runner has 14 GB of disk, and CTAN has +140 GB. A run does these steps: + +1. It makes a listing of upstream. +2. It compares the listing with the state file that the last run put in the bucket. +3. It copies only the delta, in batches. `checkpoint` records each batch in the state + before the next batch starts. ```mermaid flowchart LR @@ -273,39 +324,62 @@ flowchart LR class prepare,verify,index,sm,re,rm hook ``` -Dashed verbs are hooks. `prepare` and `verify` do nothing until `TL_KEY` is set, `index` -until `INDEX` is set, and `smoke-mirror` until a mirror fills it. +The verbs with a broken line are hooks. These hooks do no work until a mirror sets a value: + +- `prepare` and `verify` do no work until `TL_KEY` has a value. +- `index` does no work until `INDEX` has a value. +- `smoke-mirror` does no work until a mirror fills it. -| Verb | Does | +| Verb | Function | |---|---| -| `pipeline`, `plan-pipeline` | The verbs above in order, and the read-only half: `clock`, `list`, `state`, `diff`, `split` | -| `list` | `rsync -rL --list-only` of `SOURCE`, through `FILTER`, normalised to `.run/upstream.txt` as `path TAB size TAB mtime`, byte-sorted; a listing under `LIST_FLOOR` lines stops the run | -| `state` | Fetch `.state/applied.txt.xz` from the bucket; a missing one asks `rebuild` | -| `rebuild` | List the bucket and make the state exactly what it holds, at upstream's sizes | -| `diff` | `changed.txt` (upstream has, the state lacks, plus the file each changed signed sha512 names), `deleted.txt` (the state has, upstream lacks), `paths.txt` | -| `split` | Refuse a tree over `CEILING_GB` or a file the disk cannot hold; split the delta into `batch-NNNN.txt` of `BATCH_GB`, the decision batch last | -| `prepare` | Hook. With `TL_KEY` set and the delta touching `TL`: fetch the tlpdb and check its signature against the pinned key | -| `batches` | Work the first `MAX_BATCHES`, each `fetch`, `verify`, `publish`, `checkpoint`; touch `.run/chain` when batches remain | -| `fetch` | rsync the batch's files into `staging/`, dereferencing symlinks; a path that vanished since the listing is skipped | -| `verify` | Hook. With `TL_KEY` set: every signed file and every container in the batch against the tlpdb | -| `publish` | `label` the batch, then `aws s3 cp --recursive` once per type, each type's files in a tree of their own, one PutObject per file, never a destination listing; the `FRESH_KEY` file last | -| `label` | `.run/labels.txt`: every staged file's Content-Type, from its name and first KiB; see [Content types](#content-types) | -| `checkpoint` | `merge` what landed into the state and push it as one PutObject; empty staging | -| `delete` | Remove the keys upstream dropped, 1,000 per call, once every batch has landed, and drop them from the state | -| `reconcile` | When `due` left `.run/reconcile`: rebuild the state, delete what neither upstream, `OWN` nor the state's own directories own, then `reconciled` | -| `index` | Hook. With `INDEX` set: `pages`, then both key sets uploaded, the keys of emptied directories removed, and `.state/indexed.txt.xz` moved forward. A mirror with a landing page of its own replaces it | -| `pages` | A page for every directory the run touched, drawn from the state into staging, with a tree per depth for the slashless keys | -| `smoke` | A key of every type the run published, read back through `HOST`: size against the listing, Content-Type against the label; the tlpdb sha512 when `TL` is set; with `INDEX`, one redrawn page under both keys; with `CANARY`, that file as `libwww-perl` against the bucket's copy; no `Content-Encoding` on the first non-empty `.tar.gz`; a warning for every path upstream lists and rsync never sends; then `fresh` and `smoke-mirror` | -| `fresh` | With `FRESH_KEY` set: read upstream's clock in that file through `HOST` and fail the run past `FRESH_HOURS` | -| `smoke-mirror` | Hook. Nothing here; a mirror with more to read back defines it | -| `report-engine` | Hook. The engine's rows of the run summary | -| `retry` | Run a command, retrying rsync's transport exit codes with backoff; 23 (an unreadable path, skipped) and 24 (a file vanished mid-transfer) are successes | - -`normalise`, `pull`, `push`, `batch`, `label-trees`, `merge` and `remove` are the verbs those call; -`pull` and `push` move an xz-compressed key between the bucket and `.run`, and a -mirror's own verb may call them. - -#### The bucket is the mirror; the state is a cache +| `pipeline`, `plan-pipeline` | The verbs above, in sequence, and the read-only half: `clock`, `list`, `state`, `diff`, `split` | +| `list` | `rsync -rL --list-only` of `SOURCE`, through `FILTER`. `normalise` writes it to `.run/upstream.txt` as `path TAB size TAB mtime`, in byte order. A listing that does not have more lines than `LIST_FLOOR` stops the run. If rsync cannot read a directory, it exits 23, and `retry` accepts exit 23: the run continues. The listing then does not have the files of that directory, and `delete` removes them from the bucket, as if upstream deleted them. When rsync can read the directory again, a run fetches its files again | +| `state` | Downloads `.state/applied.txt.xz` from the bucket. If the file is missing, `state` tells `rebuild` to make it | +| `rebuild` | Makes a listing of the bucket. The new state has each key that the bucket contains with the number of bytes that upstream gives | +| `diff` | Writes `changed.txt`, `deleted.txt` and `paths.txt`. `changed.txt` has the lines that upstream has and the state does not have. It also has the file of each changed signed sha512. `deleted.txt` has the paths that the state has and upstream does not have | +| `split` | Rejects a tree that is larger than `CEILING_GB`, or a file that the disk cannot contain. It divides the delta into `batch-NNNN.txt` files of `BATCH_GB` each, with the decision batch last | +| `prepare` | A hook. If `TL_KEY` has a value and the delta has a `TL` path: it fetches the tlpdb and compares its signature with the pinned key | +| `batches` | Runs `fetch`, `verify`, `publish` and `checkpoint` for each of the first `MAX_BATCHES` batches. If there are remaining batches, it makes the file `.run/chain` | +| `fetch` | Copies the files of the batch into `staging/` with rsync, and copies the target of each symlink. If upstream deletes a path after the listing, `fetch` does not copy it | +| `verify` | A hook. If `TL_KEY` has a value: it compares each signed file and each container in the batch with the tlpdb | +| `publish` | Runs `label` on the batch. Then it runs `aws s3 cp --recursive` one time for each type, with the files of each type in a different tree. It sends each file as one PutObject, and it makes no listing of the bucket. The engine does not use `aws s3 sync`. The `FRESH_KEY` file goes last | +| `label` | Writes `.run/labels.txt`: the Content-Type of each file in staging, from its name and its first KiB. Refer to [Content types](#content-types) | +| `checkpoint` | Runs `merge` to add the uploaded files to the state. Then it uploads the state as one PutObject, and it deletes the files in staging | +| `delete` | Removes the keys in `deleted.txt` from the bucket, 1,000 in each call, and removes them from the state. It waits for the run that uploads the last batch | +| `reconcile` | If `due` wrote `.run/reconcile`, it runs `rebuild`. Then it deletes each key that is not a path of upstream, a key in `OWN`, or a directory of the state. Then it runs `reconciled` | +| `index` | A hook. If `INDEX` has a value: it runs `pages`, uploads the two sets of keys, and removes the keys of the directories that became empty. Then it moves `.state/indexed.txt.xz` to the new state. A mirror with a different landing page overrides `index` | +| `pages` | Writes a page for each directory that the run changed, from the state, into staging. The keys with no slash at the end go in one tree for each depth | +| `smoke` | Reads a sample back through `HOST`. The list below the table gives each item of the sample. Then it runs `fresh` and `smoke-mirror` | +| `fresh` | If `FRESH_KEY` has a value: it reads the clock of upstream in that file, through `HOST`. If the clock is more than `FRESH_HOURS` behind the time of the run, it stops the run | +| `smoke-mirror` | A hook, empty here. A mirror with more items to read back writes it | +| `report-engine` | A hook: the rows of the engine in the run summary | +| `retry` | Runs a command. After an rsync transport exit (5, 10, 12, 30 or 35), it tries the command again, a maximum of five times, with backoff. It accepts exit 23 and exit 24, and the run continues. rsync exits 23 if it cannot read a path, and 24 if upstream deletes a file while rsync sends it. Thus, a directory that rsync cannot read does not stop the run, and `delete` removes its files, as if upstream deleted them. Only `LIST_FLOOR` stops a large loss | + +The verbs above also run these verbs: `normalise`, `pull`, `push`, `batch`, `label-trees`, +`merge` and `remove`. `pull` and `push` move a key with xz compression between the bucket +and `.run`. A verb of a mirror can also run them. + +`smoke` reads these items back through `HOST`, and it stops the run if one of them is not +correct: + +- One key of each type that the run published. Its length in bytes must agree with the + listing, and its Content-Type must agree with the label. +- The `FRESH_KEY` file, for its length. `smoke` reads it only if the state has it. In a + first fill, the runs that stop at `MAX_BATCHES` do not upload the decision batch. The + bucket gives 404 for a key that it did not contain before. +- If `TL` has a value: `texlive.tlpdb.sha512`. It must be the same as the verified copy. +- If `INDEX` has a value: one page that `index` wrote, at its two keys. +- If `CANARY` has a value: that file, read as `libwww-perl`. It must be the same as the copy + in the bucket, byte for byte. +- The first `.tar.gz` that is not empty. It must have no `Content-Encoding`. + +`smoke` gets the length of an object from the headers of a read of one byte, not from a +HEAD. Cloudflare sends no `content-length` for `text/html`, and a HEAD then gives no +length. `smoke` also finds each path that upstream lists and rsync did not send. Each run +fetches these paths again, the report counts them, and the first 20 are warnings on the +run. + +#### The bucket is the mirror, and the state is a cache ```mermaid flowchart LR @@ -319,37 +393,52 @@ flowchart LR reconcile -- "rebuild, then delete what neither
upstream, OWN nor the state's directories own" --> bucket ``` -An hourly run costs one listing of upstream and none of the bucket. Three things follow. - -- **There is no seed.** A missing state file is rebuilt from a listing of the bucket. An - empty bucket gives an empty state, so the first run's delta is the whole tree, worked - `MAX_BATCHES` at a time; when batches remain the run chains the next one, and the fill - converges in a few chained runs with no flag. A lost state file costs the same listing - and nothing else. -- **The state records what landed.** `merge` joins the batch against what is in staging, - so a path that vanished upstream between listing and fetch never enters the state, and - the state never names a key the bucket lacks. -- **`RECONCILE` is the check on a live mirror.** It rebuilds the state from the bucket on - purpose, once every `RECONCILE_HOURS` (24) by default, and deletes keys neither upstream - nor the state owns. - -What a mirror cannot afford to lose is the bucket. Everything else, the state file and the -staging tree included, is derived from it and from upstream. +An hourly run makes one listing of upstream, and no listing of the bucket. The state line +is `path TAB size TAB mtime`, with the path first, in byte order. Each `sort`, `comm` and +`join` of a listing runs with `LC_ALL=C`, because `comm` pairs full lines incorrectly in a +different order. This has three results: + +- **A first fill has no special step.** If the state file is missing, `rebuild` makes it + from a listing of the bucket. An empty bucket gives an empty state, and the delta of the + first run is the full tree. Each run does `MAX_BATCHES` batches, and it starts the next + run through the chain until the bucket is full. No flag is necessary. The cost of a + missing state file is one listing. +- **The state records only the uploaded files.** `merge` compares the batch with the files + in staging. If upstream deletes a path between the listing and the fetch, the path does + not go into the state. Thus, the state has no key that the bucket does not have. +- **`RECONCILE` is the check on a live mirror.** A reconcile always makes the state from the + bucket again, one time in each `RECONCILE_HOURS` (24 by default). Then it deletes the keys + that upstream and the state do not have. + +The engine writes the state one time for each batch, after the upload, as one PutObject. If +a run stops before its end, the next run does one batch again, at most. A run that stops +at `MAX_BATCHES` with remaining batches is not a failure. + +The bucket is the one item that a mirror cannot make again. The engine makes all other +data from the bucket and from upstream: the state file and the staging tree too. #### The decision batch -`split` puts last whatever a client reads to decide what to fetch: the signed subtree's -`tlpkg/` and every bucket-root file, `timestamp` among them. Containers land before the -tlpdb that names them, and `delete` waits for the run in which every batch has landed, so -the live tlpdb never names a removed container and a `tlmgr` run that overlaps a publish -sees the previous tlpdb, never one naming a file that is not there. +`split` puts in the last batch the files that tell a client which files to fetch. These +files are the `tlpkg/` of the signed subtree and each file at the root of the bucket, for +example `timestamp`. Thus, each container is in the bucket before the tlpdb that gives its +name. `delete` waits for the run that uploads the last batch. These are the results: + +- The live tlpdb does not give the name of a container that `delete` removed. +- A `tlmgr` run during a publish sees the previous tlpdb. It does not see a tlpdb that gives + a file that is not in the bucket. #### A signed subtree -With `TL` and `TL_KEY` set, the engine verifies TeX Live's signatures before anything is -published. The keyring is fetched from the same mirror as the signature, so the pinned -fingerprint is the only real check; `GOODSIG` is required too, because gpgv reports an -expired or revoked key as `VALIDSIG` with exit 0. +If `TL` and `TL_KEY` have values, the engine verifies the signatures of TeX Live before it +publishes a file. `prepare` runs after `split`, before the first batch. `verify` runs on +each batch, before the upload. + +The engine fetches the keyring from the same mirror as the signature, and it reads the +keyring as a plain file. Thus, gpgv imports no key before the check, and only the pinned +fingerprint is a check that the mirror cannot change: `VALIDSIG` must end in `TL_KEY`. The +engine also must find `GOODSIG`, because gpgv shows an expired or revoked key as `VALIDSIG` +with exit 0. `tlmgr` accepts such a key, but the engine does not. ```mermaid flowchart TB @@ -364,88 +453,219 @@ flowchart TB p --> v ``` -A batch that fails any check stays local; the previous good copy stays live. `smoke` -reads `texlive.tlpdb.sha512` back through the domain afterwards and compares it with the -verified copy. `tlmgr` repeats the signature check on the client. +`verify` does these checks: + +- It verifies each signed sha512 in the batch: `texlive.tlpdb.sha512`, and each sha512 at + the root of the tree, for example the `install-tl*.sha512` and + `update-tlmgr-latest.*.sha512` files. +- It compares each container in the batch with the `containerchecksum`, + `doccontainerchecksum` or `srccontainerchecksum` in the verified tlpdb. A source container + is `.source.tar.xz`. If the tlpdb does not give a container, `verify` stops the run. +- In the decision batch, `texlive.tlpdb.xz` must decompress to the verified `texlive.tlpdb`, + byte for byte. Each container that the tlpdb gives must be in the bucket after this run. + +Upstream keeps each container with a name that has its revision, for example +`foo.r123.tar.xz`, and a symlink `foo.tar.xz` points to it. `fetch` uses `-L`. Thus, a +mirror serves the two names, unless its `FILTER` removes the names with a revision, as the +`FILTER` of tlnet does. `verify` finds the checksum for the two names. It gets the revision +from the `revision` line of the stanza in the tlpdb. + +If a check finds a problem in a batch, the engine does not upload that batch, and the +previous good copy stays live. After the upload, `smoke` reads `texlive.tlpdb.sha512` +back through the domain, and compares it with the verified copy. `tlmgr` does the +signature check again on the client. #### Directory pages -R2 serves no listings. With `INDEX` set, `index` draws a page for every directory a run -touched, from the state rather than from upstream, and writes it under two keys: -`/`, which a zone Transform Rule serves for `/dir/`, and the bare ``, -which serves `/dir` where a filesystem mirror would answer 301. A `` lets one -document serve both. `.state/indexed.txt.xz` records the state the pages last showed, so a -run that dies before moving it redraws the same pages next time. Deleting it redraws every -page once, which is also how a change to the markup or to `PAGE_FOOT` reaches directories -that have not changed. `reconcile` spares both keys. +R2 serves no directory listings. If `INDEX` has a value, `index` writes a page for each +directory that a run changed. It writes the pages from the state, not from upstream, and +each page has two keys: + +- `/`: a Transform Rule of the zone serves this key for `/dir/`. +- ``, with no slash at the end: this key serves `/dir`. A mirror on a file system + sends 301 for that URL. The URL does not show if a name is a file or a directory. Thus, + no Cloudflare rule can send 301. + +A `` lets one document serve the two URLs. A file system cannot contain `a/b` and +`a/b/` at the same time. Thus, `pages` puts the keys with no slash in one tree for each +depth, and `index` uploads one tree at a time. `index` gives these keys +`--content-type text/html`. Without it, a key with no suffix gets `binary/octet-stream`, +and a browser downloads the page and does not show it. + +`.state/indexed.txt.xz` records the state that the pages showed last. If a run stops before +it moves that file, the next run writes the same pages again. If you delete the file, the +next run writes all the pages again, one time. Thus, a change to the HTML of the page or to +`PAGE_FOOT` goes into the directories that did not change. `reconcile` does not delete the +two keys. -The zone rule, one per host: +The zone has one rule for each host: when: ends_with(http.request.uri.path, "/") rewrite path: concat(http.request.uri.path, http.host, ".directory.index.html") -It serves the root page for `/` too. ctan excludes `/`, which is CTAN's own `index.html`. +This rule also serves the page of the root for `/`. In ctan, the rule does not include `/`, +because `/` serves the `index.html` of CTAN. #### Content types -R2 serves the Content-Type an object was stored with, and the AWS CLI, left to itself, -stores the first half of Python's guess: `foo.diff.gz` guesses as a gzipped diff and is -stored as `text/x-diff`, which a browser draws as text. Its guesses also move with the -image, so one type can be stored two ways in one bucket. So `label` decides every type and -the CLI guesses none: - -1. A name ending in a `COMPRESSED` ending (`.gz`, `.tgz`, `.bz2`, `.xz`, `.Z`, `.zst`, - `.lz`, ...) is that compressed type, whatever the rest of the name says. The types are - ftp.gnu.org's. -2. An ending Python reads as compressed and `COMPRESSED` lacks gets a signature's type or - `application/octet-stream`, never text. -3. Otherwise the image's `mime.types` names it, unless the name says a type a browser draws - (text, XML, JSON, JavaScript, PDF) and the first KiB disagrees: a compression - signature, a NUL, a PDF without `%PDF-`. Then the bytes decide. -4. A name with no type is typed from the bytes: a known signature, `text/plain` without a - NUL, `application/octet-stream` with one. - -A stray control byte is not a NUL, so an old `.sty` ending in `^Z` stays `text/x-tex`. Three -checks hold it: `offline` labels a set of names and bytes and fails on any change, which is -where an image bump that moves `mime.types` shows; `smoke` reads one key of every type -back through the domain and compares the served type with the label; and the -`.tar.gz` check keeps `Content-Encoding` off tarballs. +R2 serves the Content-Type that the object has in the bucket. Without a type in the +command, the AWS CLI uses the first half of the type that Python gets from the name. For +`foo.diff.gz`, Python gives a diff with gzip compression, and the CLI then stores +`text/x-diff`. A browser then shows the file as text. + +The result of Python also changes with the image. Thus, the same type of file can have two +different Content-Types in one bucket. For these causes, `label` sets each type, and the +CLI sets no type: + +1. A name with a `COMPRESSED` extension (`.gz`, `.tgz`, `.bz2`, `.xz`, `.Z`, `.zst`, `.lz` + and others) gets that compressed type. The other parts of the name have no effect. The + types are the types of ftp.gnu.org. +2. An extension that Python reads as compressed, and that is not in `COMPRESSED`, gets the + type of its signature, or `application/octet-stream`. It does not get a text type. +3. If not, the `mime.types` of the image gives the type. But the name can give a type that + a browser shows (text, XML, JSON, JavaScript, PDF), and the first KiB can show a + different type. Then the bytes set the type. This occurs if the first KiB has a + compression signature or a NUL, or if a PDF has no `%PDF-`. +4. A name with no type gets its type from the bytes. A known signature gives its type. If + the bytes have no NUL, the type is `text/plain`. If they have a NUL, the type is + `application/octet-stream`. + +A control byte is not a NUL. Thus, a `.sty` file of CTAN with a `^Z` at its end stays +`text/x-tex`. Three checks make sure that `label` obeys these rules: + +- `offline` gives a set of names and bytes to `label`, and stops on each change. If an + update of the image changes `mime.types`, this check shows it. +- `smoke` reads one key of each type back through the domain, and compares the served type + with the label. +- The `.tar.gz` check makes sure that tarballs have no `Content-Encoding`. #### The vars -A mirror sets `SOURCE`, `BUCKET` and `HOST` in its root vars, always. Everything else -has an inline default, and a mirror sets only what differs: +A mirror always sets `SOURCE`, `BUCKET` and `HOST` in its root vars. All other vars have an +inline default, and a mirror sets only the vars that are different: + +| Var | Default | Function | +|---|---|---| +| `CEILING_GB` | 0, no ceiling | `split` rejects a tree that is larger than this number of decimal GB | +| `BATCH_GB` | 4 | Decimal GB in each batch. A larger file is a batch with no other file | +| `MAX_BATCHES` | 4 | The batches in each run. The other batches go to the next run, through the chain | +| `LIST_FLOOR` | 0, no guard | A listing that does not have more lines than this number is truncated, and the engine does not use it to delete keys. Set it to approximately 90% of the usual number of lines | +| `RECONCILE`, `RECONCILE_HOURS` | `auto`, 24 | The vars of the toolbox: refer to [Reconcile, by age](#reconcile-by-age) | +| `RETRY_BASE` | 15 | Seconds. Before retry `i`, `retry` waits `RETRY_BASE * 2^i` seconds, plus jitter | +| `TL`, `TL_KEY` | Empty | A signed TeX Live subtree, and the fingerprint of the key that signs it. If they are empty, the engine does no signature checks | +| `FILTER` | Empty | rsync filter arguments that make the listing smaller, for a mirror of a subtree | +| `OWN` | Empty | Keys at the root of the bucket that are the keys of the mirror, with a space between them. `reconcile` does not delete them | +| `INDEX` | Empty | The key suffix of the directory pages. If it has a value, `index` writes the pages. `reconcile` does not delete the pages, or a key with the name of a directory of the state | +| `PAGE_FOOT` | Empty | The HTML at the end of each directory page. `%s` is the encoded path of the directory, and `%%` gives one `%` | +| `CANARY` | Empty | A file that `smoke` reads through the domain as a Perl client, and compares with the bucket, byte for byte. All files can be the canary. A file with plain `http://` links or `mailto:` addresses also finds the HTML rewriters of the zone. The path must have no character that a URL encodes | +| `FRESH_KEY` | Empty | A file in which upstream records its clock, as Unix time or as `YYYY-MM-DD-HH-MM` (the format of CTAN). If it has a value, `fresh` stops the run when the clock is more than `FRESH_HOURS` behind | +| `FRESH_HOURS` | 24 | Hours. This value is less than 28 hours: at 28 hours behind, the monitors of GNU and CTAN show a problem for the mirror | + +A var in the root `vars:` of a mirror has the same value for all runs. In an included verb, +the run uses a root value, not a `KEY=value` on the command line. A var that the mirror +does not set keeps its default, and each run can set it: +`task sync -- MAX_BATCHES=8 RECONCILE=true`. + +Do not set a root var to the default of the engine. Then the command line cannot change +that var. + +#### When a run fails + +The job log shows the step that stopped the run, and that step gives the verb. The table +below gives each verb of the engine that can stop a run, with the cause and the repair. The +README of a mirror gives only the values of that mirror: its ceiling, its upstream, and the +steps that only that mirror has. + +To find the step, and the number of files that the mirror does not have, use these +commands: + +```sh +gh run list --workflow sync.yml --limit 5 +gh run view --log-failed | tail -50 +curl -sI https:/// | grep -i -E 'last-modified|cf-cache-status' +aws s3 cp s3:///.state/applied.txt.xz - | xz -d | wc -l +``` + +Compare the number of lines in the state with the number of lines in the listing. The +difference shows how many files the mirror does not have. + +You can always start the run again, unless a row below tells you differently. A second run +writes the same bytes again, or it writes no bytes. To start it, use +`gh workflow run sync.yml` or `task sync`. -| Var | Default | Meaning | +The state has the batches until the last checkpoint, and the second run calculates the +remaining delta again. In a first fill, this second run continues the fill. On a laptop, +`task sync` uses `op run`. Thus, sign in to the 1Password CLI first. + +| Verb | Cause | Repair | |---|---|---| -| `CEILING_GB` | 0, no ceiling | `split` refuses a tree larger than this many decimal GB | -| `BATCH_GB` | 4 | Decimal GB per batch; a larger file is a batch by itself | -| `MAX_BATCHES` | 4 | Batches per run; the rest chain the next run | -| `LIST_FLOOR` | 0, no guard | A listing under this many lines is a truncated one, never a deletion list. Set it to about 90% of the usual line count | -| `RECONCILE`, `RECONCILE_HOURS` | `auto`, 24 | The toolbox's: see [Reconcile, by age](#reconcile-by-age) | -| `RETRY_BASE` | 15 | Seconds; the retry sleeps are `RETRY_BASE * 2^i` plus jitter | -| `TL`, `TL_KEY` | empty | A signed TeX Live subtree and the fingerprint that signs it; empty, no signature checks | -| `FILTER` | empty | rsync filter arguments that narrow the listing, for a mirror of a subtree | -| `OWN` | empty | Bucket-root keys the mirror owns, space separated; `reconcile` never deletes them | -| `INDEX` | empty | The key suffix of the directory pages; set, `index` draws them and `reconcile` spares them and every bare directory of the state | -| `PAGE_FOOT` | empty | The HTML every directory page closes on; `%s` is the directory's encoded path, `%%` a literal percent | -| `CANARY` | empty | A path carrying plain `http://` links or `mailto:` addresses, no character of which a URL must encode; set, `smoke` fails if the domain serves it differently from the bucket or refuses a Perl client | -| `FRESH_KEY` | empty | A file in which upstream records its own clock, as Unix time or CTAN's `YYYY-MM-DD-HH-MM`; set, `fresh` fails the run when it is older than `FRESH_HOURS` | -| `FRESH_HOURS` | 24 | Hours; under the 28 at which GNU's and CTAN's monitors call a mirror old | - -A var the mirror puts in its root `vars:` is fixed for every run: inside an included -verb a root value shadows a `KEY=value` from the command line. One the mirror leaves to -its default is the run's to set, `task sync -- MAX_BATCHES=8 RECONCILE=true`. So a root -var that restates a default is worse than none: it silences the command line. +| `image` | The toolbox could not download the image from GHCR. The mirror has no problem: the run uploaded no file, and the state did not change | Start the run again | +| `list` | The listing does not have more lines than `LIST_FLOOR`. The engine does not use a truncated listing to delete keys | Start the run again. If the problem continues, examine upstream. If upstream became more than 10% smaller, decrease `LIST_FLOOR` | +| `retry`, in `list`, `prepare` or `fetch` | rsync gave a transport exit five times. If upstream moved, each run uses approximately ten minutes for the retries, and then it stops | Examine `SOURCE`. If upstream moved, change `SOURCE` in the Taskfile | +| `split` | The tree is larger than `CEILING_GB`, or the disk cannot contain one file. The mirror gets no new files until you increase `CEILING_GB`. A larger ceiling also increases the cost | Increase `CEILING_GB` | +| `prepare`, `verify` | A signature or a checksum did not agree. The engine uploaded no file from that batch, and the previous good copy is live | If TeX Live changed its key, compare `TL_KEY` with https://www.tug.org/texlive/verify.html. Do not use a different source for the key | +| `smoke`: the canary | The domain served bytes that are not the bytes in the bucket, or it rejected a Perl client. A zone rule is missing, or the scope of a zone rule became wider. The bucket is correct | Examine the Configuration Rule of the zone. Refer to [The parts of a bucket](#the-parts-of-a-bucket) | +| `smoke`: the Content-Encoding | A `.tar.gz` had a `Content-Encoding` when `smoke` read it. Clients then decompress it while they download it | Examine `label` and the AWS CLI version in the lock (`toolchain.lock.toml`). Refer to [Content types](#content-types) | +| `smoke`: the type or the length | A key of the sample had a Content-Type that is not its label, or a length that is not its length in the listing. The CLI, R2 or the zone changed the object | Examine `label` and the AWS CLI version in the lock. Then examine the rules of the zone | +| `fresh`: the clock stopped | The clock in the `FRESH_KEY` file did not change for `FRESH_HOURS`. Upstream stopped, or the mirror that `SOURCE` gives stopped. A secondary mirror can stop its sync, and rsync can continue to connect to it. The run copied all the bytes that it got | Examine upstream. If `SOURCE` is a secondary mirror, set `SOURCE` to a different secondary mirror. If upstream stopped, no change in the mirror is necessary | + +These procedures repair other problems: + +- **The state looks incorrect.** Use this procedure if the state file is damaged, or if + you do not know if it is correct. It makes the state again from the bucket: + + ```sh + gh workflow run sync.yml -f vars='RECONCILE=true' # or: task sync -- RECONCILE=true + ``` + + The run makes a listing of the bucket, and compares it with the listing of upstream. It + accepts each object with the length that upstream gives as correct. Thus, if a file + changed but kept its length while the state was not correct, the run does not see the + change. It sees the change when upstream changes the file again. `rebuild` uploads no + file. This procedure is not necessary for a missing state file: the next run makes it. +- **The first fill.** No flag is necessary. An empty bucket gives an empty state, and the + delta is the full tree. Each run does `MAX_BATCHES` batches (4), and then it starts the + next run through the chain. For more batches in each run, use + `gh workflow run sync.yml -f vars='MAX_BATCHES=8'`. Before a first fill, pause the + healthcheck (refer to [Monitoring](#monitoring)). +- **Delete one key.** Use `aws s3 rm s3:///`. While the Cache Rule of the zone + bypasses the cache, the edge contains no copy to remove. If upstream has the key, the next + reconcile finds that the key is missing, and the subsequent run fetches it again. Do not + edit the state manually to get the key back before the next reconcile. This is not + safe. +- **Write all directory pages again.** Use `aws s3 rm s3:///.state/indexed.txt.xz`. + The next run writes the page of each directory at its two keys. This procedure is safe: + it does not change the files in the bucket. +- **A directory URL gives 404.** The page is at its key, with or without the zone rule. Use + `curl -sI https:////`. If this command gives 200 and the directory URL + does not, the Transform Rule is missing, or its scope is not correct. +- **Change a secret.** Edit the `r2` section of the item of the mirror in the 1Password app. + Then use `gh workflow run sync.yml && gh run watch`. After a run with no failure, delete + the previous R2 token in the Cloudflare dashboard. This procedure is safe: the first call + with the new credentials is a read. +- **Abort a multipart upload that did not complete.** Find the upload. Then abort it: + + ```sh + aws s3api list-multipart-uploads --bucket --query 'Uploads[].[Key,UploadId,Initiated]' --output text + aws s3api abort-multipart-upload --bucket --key --upload-id + ``` + + This procedure is safe, and it has no cost. If you do not do it, R2 aborts the upload + after 7 days. ### The proton engine -`engines/proton.yml`: a staging tree into one Proton Drive folder, through the official -`proton-drive` CLI, for a mirror whose upstream fits in a run. The mirror fills -`staging/` and the engine does the rest. It keeps nothing of the mirror's in the bucket -but the CLI session, because the CLI skips a file whose content Proton already holds and -`-f create-new-revision` makes a revision of one that changed: Proton's version history -is the history of the mirror. +`engines/proton.yml` copies a staging tree into one Proton Drive folder, with the +`proton-drive` CLI of Proton. It is for a mirror that can copy its full upstream in one run. +The mirror fills `staging/`, and the engine does all the other steps. + +The bucket contains only the CLI session of the mirror. The engine keeps no other data +there, because of these two facts: + +- The CLI does not upload a file if Proton has its content. +- `-f create-new-revision` makes a new revision of a file that changed. + +Thus, the version history of Proton is the history of the mirror. ```mermaid flowchart LR @@ -455,37 +675,43 @@ flowchart LR class stage,prune,re,rm hook ``` -| Verb | Does | +| Verb | Function | |---|---| -| `pipeline`, `plan-pipeline` | The verbs above in order; the read-only half runs through `stage` and prints what an upload would carry | -| `session` | Pull `.state/session.tar.age` from the bucket, decrypt it with `MIRROR_AGE_IDENTITY`, extract the two session files to `.run/session` | -| `destination` | List the parent of `MIRROR_PROTON_DESTINATION` and refuse the run unless exactly one folder of that name exists and its UID is `MIRROR_PROTON_DESTINATION_UID` | -| `stage` | Hook. Every mirror defines it: fill `staging/` with what Proton should hold | -| `upload` | One `filesystem upload -f create-new-revision -d merge -t --json` of `staging/*` into the destination; the summary to `.run/upload.json` | -| `confirm` | Transferred plus skipped plus failed must equal the staged files plus folders, with no failure; the verdict to `.run/confirm.txt` | -| `prune` | Hook. Nothing here; a mirror that trashes what its upstream dropped defines it, from `list-folder` and `trash` | -| `pd` | Every CLI call: stderr to `.run/pd.err`, then the session sealed back to the bucket when its token rotated, whatever the exit | -| `list-folder`, `trash` | A folder's JSON listing to `OUT`; the nodes at `PATHS` to Proton's trash | -| `session-seal -- ` | Host side: a laptop login's two files, encrypted into the bucket | -| `empty-trash` | Host side, asks first: everything in Proton's trash, permanently | -| `report-engine` | Hook. The engine's rows of the run summary | - -The engine reads no root var. Its inputs are the environment, by the names every Proton -mirror's `op.env` carries: - -| Name | What it is | +| `pipeline`, `plan-pipeline` | The verbs above, in sequence. The read-only half runs to `stage`, and then prints the number of files and MB that an upload can send | +| `session` | Downloads `.state/session.tar.age` from the bucket, decrypts it with `MIRROR_AGE_IDENTITY`, and puts the two session files from the tar into `.run/session` | +| `destination` | Makes a listing of the parent of `MIRROR_PROTON_DESTINATION`. It stops the run unless the parent has one folder of that name, and only one, with the UID `MIRROR_PROTON_DESTINATION_UID` | +| `stage` | A hook that each mirror writes. It fills `staging/` with the files that Proton must contain | +| `upload` | Runs one `filesystem upload -f create-new-revision -d merge -t --json` of `staging/*` into the destination. It writes the summary to `.run/upload.json` | +| `confirm` | In the summary, `transferredItems` and `skippedItems` together must be equal to the files and folders in staging, and `failedItems` must be 0. It writes the result to `.run/confirm.txt` | +| `prune` | A hook, empty here. A mirror that moves to the trash the items that its upstream deleted writes it, with `list-folder` and `trash` | +| `pd` | Runs each CLI call, and writes the stderr of the call to `.run/pd.err`. After each call, with all exit codes, it examines the session. If the token changed, it seals the session back into the bucket | +| `list-folder`, `trash` | `list-folder` writes the JSON listing of a folder to `OUT`. `trash` moves the nodes at `PATHS` to the trash of Proton | +| `session-seal -- ` | On the host: it encrypts the two files of a laptop login into the bucket | +| `empty-trash` | On the host: after you agree, it deletes all the items in the trash of Proton, permanently | +| `report-engine` | A hook: the rows of the engine in the run summary | + +The stderr of `list-folder` goes to the log of the run, not to `.run/pd.err`. Its command +ends in `|| echo "[]" > OUT`. Thus, the stderr redirect of `pd` goes to the `echo`, not to +the CLI. + +The engine reads no root var. It reads only the environment, by the names that the `op.env` +of each Proton mirror has: + +| Name | Value | |---|---| -| `MIRROR_PROTON_DESTINATION` | The CLI path of the destination folder, `/my-files/GitHub` say | -| `MIRROR_PROTON_DESTINATION_UID` | Its UID, compared on every run before any write | -| `MIRROR_R2_BUCKET` | The bucket that holds the session | -| `MIRROR_AGE_IDENTITY` | The `AGE-SECRET-KEY-...` line the session is encrypted to | -| `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_ENDPOINT_URL_S3` | What boto3 reads; note `_S3` on the endpoint | +| `MIRROR_PROTON_DESTINATION` | The CLI path of the destination folder, for example `/my-files/GitHub` | +| `MIRROR_PROTON_DESTINATION_UID` | The UID of that folder. Each run compares it before the first write | +| `MIRROR_R2_BUCKET` | The bucket that contains the session | +| `MIRROR_AGE_IDENTITY` | The `AGE-SECRET-KEY-...` line that encrypts and decrypts the session | +| `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_ENDPOINT_URL_S3` | The credentials and the endpoint for `s3`. The proton mirrors use the `_S3` name for the endpoint. The AWS CLI and boto3 read the two names, `AWS_ENDPOINT_URL` and `AWS_ENDPOINT_URL_S3` | #### The session -The CLI can only be seeded by a browser sign-in, and fresh sign-ins from datacenter -addresses are blocked, so the session is made once on a laptop and carried to every run -encrypted. Its refresh token rotates on use, so every run writes it back. +Only a sign-in in a browser can start a session of the CLI. Proton does not accept a new +sign-in from a datacenter address. Thus, you make the session one time on a laptop, and +each run gets it from the bucket, encrypted. The CLI changes the refresh token when it uses +the session. Thus, after each CLI call, `pd` examines the session. If the token changed, +`pd` seals the session back into the bucket. ```mermaid flowchart LR @@ -502,40 +728,53 @@ flowchart LR r4 --> key ``` -To make one: +To make a session, do these steps on a laptop: -```sh -export PROTON_DRIVE_CACHE_DIR=.run/pd PROTON_DRIVE_CREDENTIALS_STORE=unsafe_file -proton-drive auth login # a browser opens; .run/pd holds two JSON files after -proton-drive filesystem create-folder /my-files # /my-files is the drive's own root, not a folder you make -proton-drive filesystem list -j /my-files # the entry named : its uid is the destination UID -task session-seal -- .run/pd # once the bucket and the vault exist -``` +1. On the Settings page of your Proton account, turn off telemetry. Do this for each Proton + account. +2. Run these commands: + + ```sh + export PROTON_DRIVE_CACHE_DIR=.run/pd PROTON_DRIVE_CREDENTIALS_STORE=unsafe_file + proton-drive auth login # a browser opens; .run/pd holds two JSON files after + proton-drive filesystem create-folder /my-files # /my-files is the drive's own root, not a folder you make + proton-drive filesystem list -j /my-files # the entry named : its uid is the destination UID + task session-seal -- .run/pd # once the bucket and the vault exist + ``` + +Install the CLI at the version that `toolchain.lock.toml` pins. The format of the session +file changes with the version, and the Linux binary in the image must read the files that +the laptop wrote. Put `.run/pd` in the repository, because the container gets only the +repository. git ignores `.run/`, and `task clean` deletes it. + +Obey these three rules: + +- Do not let two mirrors use one session. The CLI changes the refresh token at each call. + Then Proton rejects the token that the other mirror has, and a new login is necessary. +- Do not run `task sync`, `task plan` or `task empty-trash` from a laptop while an Actions + run can be in progress. The three commands use the one session. If the laptop and the run + use it at the same time, a new login can be necessary. +- Do not keep a history of the session in the bucket. A previous copy contains a token that + the CLI changed, and the CLI cannot use that copy. -Install the CLI at the version `toolchain.lock.toml` pins: the session file format is -tied to the version, and the Linux binary in the image must read what the laptop wrote. -`.run/pd` must be inside the repository, because only the repository is mounted into the -image; `.run/` is ignored by git and deleted by `task clean`. +### A pipeline of a mirror -Three rules. **Two mirrors never share a session**: its refresh token rotates on every -call, and the loser of a race needs a fresh login. **Never run `sync` or `plan` from a -laptop while an Actions run may be in progress**, for the same reason. **The session gets -no history in the bucket**: a stale copy holds a rotated-out token and cannot be restored. +A mirror that uses a program for its work includes only the toolbox. It writes `pipeline` +and `plan-pipeline` from the `clock`, `report` and `ping` of the toolbox, and the steps of +its program. dropbox has this shape: -### A pipeline of your own +- Each step is one `python -m migrator `. +- The Taskfile sets the sequence of the steps. +- The Python makes each decision, but the `due` of the toolbox sets when to do a reconcile. +- The `proton` image supplies the interpreter, boto3 and the CLI. -A mirror whose logic is a program of its own includes the toolbox alone and defines -`pipeline` and `plan-pipeline` from the toolbox's `clock`, `report` and `ping` and its own -steps. dropbox is that shape: each step is one `python -m migrator `, the -Taskfile owns the order, the Python owns every decision but when to reconcile, which is -the toolbox's `due`, and the `proton` image supplies the interpreter, boto3 and the CLI. -Such a mirror fills `report-mirror` with its own report and nothing else of the -toolbox's changes. +A mirror of this shape fills `report-mirror` with its report. It changes no other part of +the toolbox. ## Secrets -Every credential and every account identifier lives in one vault and reaches a run by -name. A repository holds `op://` references only. +All credentials and all account identifiers are in one vault, and a run gets each of them +by name. A repository contains only `op://` references. ```mermaid flowchart LR @@ -548,155 +787,218 @@ flowchart LR ### The vault -One vault. One item per mirror, named for its repository. One section per service in -the item (`r2`, `healthcheck`, `proton`, `age`, `github`, `dropbox`), with the field names -the mirror's README lists. Neither the vault's name nor its UUID is a secret: without the -service account's token it opens nothing. +There is one vault, with one item for each mirror. Each item has the name of its +repository. An item has one section for each system that the mirror uses (`r2`, +`healthcheck`, `proton`, `age`, `github`, `dropbox`). The README of the mirror gives the +names of the fields. The name and the UUID of the vault are not secrets: without the token +of the service account, they give access to no item. -The references address the vault by UUID, `op:////
/`, so a -renamed vault cannot break a run and a name with a slash in it cannot be written: +The references give the vault by its UUID: `op:////
/`. Thus, if +the vault gets a new name, the runs continue to operate. Also, a reference cannot contain a +vault name that has a slash. To find the UUID, use this command: ```sh op vault get --format json | jq -r .id ``` -One service account that can read that vault and nothing else. Its token is the secret -`OP_SERVICE_ACCOUNT_TOKEN`: on an organization, one organization secret that every -repository inherits; on a personal account, one repository secret per mirror. Only a -dispatched sync run holds it; a pull-request check gets no secret at all. +One service account can read that vault, and no other vault. Its token is the secret +`OP_SERVICE_ACCOUNT_TOKEN`: + +- On an organization, the token is one organization secret that all repositories inherit. +- On a personal account, the token is one repository secret for each mirror. + +Only a dispatched sync run gets the token. A check of a pull request gets no secret. ### On the host -`op run --env-file=op.env` resolves every reference and exports the values into the -environment of the one command it wraps, `task run -- ...`, masking each value in that -command's output. On a laptop `op` uses the signed-in desktop app; in Actions it uses the -service account token the reusable workflow put in its environment. The vault is read -once per run. +`op run --env-file=op.env` gets the value of each reference. It exports the values into the +environment of one command, `task run -- ...`. The output of that command does not show the +values, because `op run` removes them. On a laptop, `op` uses the desktop app, and you must +sign in to the app. In Actions, `op` uses the token of the service account, which the +reusable workflow puts in its environment. A run reads the vault one time. ### Into the container -`run` passes `-e NAME` for each name in `op.env` and in the mirror's `PASS`, plus -`HEALTHCHECK_URL`, `GITHUB_STEP_SUMMARY` and `GITHUB_RUN_ID` always. The container engine -copies each named variable from the host's environment; no value is on the command line. -A host variable the pipeline reads and `op.env` does not name arrives empty. +For each name in `op.env` and in the `PASS` of the mirror, `run` gives `-e NAME` to the +container runtime. It also always gives `HEALTHCHECK_URL`, `GITHUB_STEP_SUMMARY` and +`GITHUB_RUN_ID`. The container runtime copies the value of each variable that `run` gives +from the environment of the host. Thus, no value is on the command line. + +If the pipeline reads a host variable that `op.env` and `PASS` do not give, the variable is +empty in the container. `GITHUB_STEP_SUMMARY` is a path on the runner. Thus, `run` gives +the variable, and it also puts the file into the container at that same path. ### The names -| Name | Read by | Set by | +| Name | The tool that reads it | Its source | |---|---|---| -| `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` | the AWS CLI and boto3 | the `r2` section | -| `AWS_ENDPOINT_URL` | the AWS CLI, in the `rsync` image | the `r2` section's `endpoint` | -| `AWS_ENDPOINT_URL_S3` | boto3, in the `proton` image | the same field, under boto3's name | -| `AWS_REGION` | both | a literal `auto` in the image; R2 has one region, and masking the word `auto` would corrupt output | -| `MIRROR_R2_BUCKET` | the proton engine's `s3` | the `r2` section's `bucket` | -| `MIRROR_AGE_IDENTITY` | the proton engine's `age` | the `age` section | -| `MIRROR_PROTON_DESTINATION`, `_UID` | the proton engine's `destination` | the `proton` section | -| `HEALTHCHECK_URL` | `ping`, `ping-fail` | the `healthcheck` section; optional | -| `MIRROR_*` of a mirror's own | that mirror | a section of its own: `github`, `dropbox` | +| `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` | The AWS CLI and boto3 | The `r2` section | +| `AWS_ENDPOINT_URL` | The AWS CLI, in the `rsync` image | The `endpoint` field of the `r2` section | +| `AWS_ENDPOINT_URL_S3` | boto3, in the `proton` image. The proton mirrors use this name, and the AWS CLI and boto3 read the two names | The same field | +| `AWS_REGION` | The two tools | The value `auto`, in the image. R2 has one region. If `op run` removed the value `auto` from the output, the output can be incorrect | +| `MIRROR_R2_BUCKET` | `s3`, in the proton engine | The `bucket` field of the `r2` section | +| `MIRROR_AGE_IDENTITY` | `age`, in the proton engine | The `age` section | +| `MIRROR_PROTON_DESTINATION`, `_UID` | `destination`, in the proton engine | The `proton` section | +| `HEALTHCHECK_URL` | `ping`, `ping-fail` | The `healthcheck` section. It is optional | +| `MIRROR_*` names that one mirror uses | That mirror | A section for that mirror: `github`, `dropbox` | ### Without a vault -A mirror with no `op.env` runs on its repository secrets: the reusable workflow exports -every inherited secret into the sync step's environment by name, the ones the mirror -lists in `PASS` cross into the container, and the 1Password CLI is not installed at all. -On a laptop, whatever is exported. This is the fallback, not the pattern: it works, and -it puts the values in a second place. +A mirror with no `op.env` runs on the secrets of its caller: + +- The reusable workflow exports each inherited secret, by name, into the environment of + the sync step. +- The secrets that the mirror gives in `PASS` go into the container. +- The workflow does not install the 1Password CLI. + +On a laptop, the run uses the exported variables. This method is not the usual method. It +operates, but it puts the values in a second location. ## Storage -Every mirror has one S3-compatible bucket. What it holds depends on the engine. +Each mirror has one S3-compatible bucket. The contents of the bucket are different for each +engine. -| Engine | Key | What | +| Engine | Key | Contents | |---|---|---| -| rsync | every upstream path, at the root | The mirror. A public domain serves the bucket | -| rsync | `.state/applied.txt.xz` | The state: what the bucket holds, at upstream's size and mtime | -| rsync, with `INDEX` | `.state/indexed.txt.xz`, `/`, `` | What the directory pages last showed, and the pages under both keys | -| rsync, a mirror's own | `index.html` | tlnet's landing page, spared by `OWN` | -| proton | `.state/session.tar.age` | The CLI session, encrypted. The only key | -| a pipeline of its own | `.state/state.sqlite.xz.age`, `.state/history/-