diff --git a/CLAUDE.md b/CLAUDE.md index 00f5b57..9c3df4e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,82 +1,154 @@ # lib -The toolbox every katoptra mirror includes by URL. `README.md` is the manual: the layers, -the verbs, the engines, secrets, storage, the images, the workflows, and how a mirror -uses and changes any of it. This file is what a change must not break: each entry is a -verified go-task or platform fact that cost a session to find. +This repository is the toolbox that all katoptra mirrors include from a URL. `README.md` is +the manual. It has sections about these parts of the system: + +- The layers, the verbs and the engines +- The secrets and the storage +- The images and the workflows. + +It also tells how a mirror uses these parts and how a mirror changes them. This file gives +the items that a change must not break. Each item is a verified fact about go-task or a +platform. One session was necessary to find each fact. + +Writing: obey ASD-STE100 and the rules in the +[Writing section of the org CONTRIBUTING](https://github.com/katoptra/.github/blob/main/CONTRIBUTING.md#writing). +Read that section before you write. ## Hazards -- A global var that reads another var is rendered once, from the root's and the command - line's values, never from a call var: `GPGCHECK` takes the fingerprint as an awk `-v` - at each use, where a call var `TL_KEY` is visible. -- A var of another name that reads a tunable, `BATCHES_MAX` from `MAX_BATCHES`, still - sees the command line (verified), so each default is spelled once, inline. -- No `dir:` on an engine verb. In a flattened include read by path, task joins even an - absolute `dir:` onto the include's directory (verified, 3.53.1), so `verify` starts - each command with `cd {{.STAGING}}` instead. -- `.taskrc.yml` keys are `trusted-hosts` and `cache-expiry`; task ignores a key it does - not know, silently, and refetches on every invocation. -- Every tool in an image comes from `toolchain.lock.toml`, read through `docker/lock.py`, - the one reader the Dockerfiles and the toolbox action share, and every one is checked - before it is used. Most carry a recorded sha256. The AWS CLI zip carries none, because - upstream publishes a detached PGP signature instead, so the fetch stage verifies that - signature against `docker/aws-cli.pub` and requires the fingerprint the lock pins, - through the same GOODSIG-and-VALIDSIG awk gate the engine uses for TeX Live. -- Images set `TASK_REMOTE_OFFLINE=1`. Inside a run the include resolves from the - mirror's `.task/remote` cache, bind-mounted with the repo, never from the network. -- `run` and `render` mount the repository's git top level at `/work` and set the working - directory to the Taskfile's subdirectory. For a mirror the two coincide; for the - examples here it is what makes `../../toolbox.yml` reachable. -- `render` captures task's dry run from inside the container (`sh -c 'task ... 2>&1'`) - so the container engine's own progress lines never reach `render.txt`. -- Inside a `sh:` var, `printf -- '-e %s'` prints dashes: task's built-in shell takes - the `--` as the format. Use `printf '%s %s ' -e "$v"`. -- Task's built-in shell runs every command under `set -e`, and an assignment takes its - command substitution's exit: `x=$(cat missing)` stops the command (verified, 3.53.1). -- Task's built-in shell keeps `set -e` inside a subshell whose status `||` reads: - `( false; echo on ) || echo caught` prints only `caught` (verified, 3.53.1), where bash - and dash print `on`. A guard meant to hold under either exits each step by hand. -- Task's built-in shell has no `umask`. A file that must be born 0600 is - `install -m 600 /dev/null "$f"` and then written, as the proton engine's `age` does. -- Every Proton CLI call goes through `pd`, which pushes the session back whatever the - exit: the refresh token rotates, and a run that kept a rotated token to itself leaves - the next run unable to log in. Two mirrors never share one session for the same reason. -- Actions pinned to a full SHA with the version in a trailing comment. A mirror pins the - two reusable workflows that way; each checks this repository out at its own commit - (`github.job_workflow_sha`) for the toolbox action and the lock, so a workflow pin is - the one pin. The include and the image float at `v2` by design: moving that tag is - the rollout. -- GNU `xargs` runs its command once on empty input. A guard on the file feeding a pipe is - not a guard on what reaches `xargs`: `pages`'s `SLASH` awk drops the root, which has no - slashless key, so a run whose only dirty directory is the root sends it nothing. Every - `xargs` whose input can be filtered down to nothing takes `-r`. -- A `>-` folded block keeps the newline when a continuation line is indented further than - the lines around it. `pages`'s awk programs depend on that; a shell line meant to - continue ends with a backslash. `task check` shows what actually renders. -- A macOS disk merges directories that differ only in case. Pages drawn on a laptop from a - real listing come out short wherever upstream has two such directories (CTAN has - `obsolete/support/TeXshell/` and `texshell/`). The runner is ext4 and draws both. +- Do not read a call var in a global var. Task renders a global var that reads a different + var one time. It uses the values of the root and of the command line, and not a call + var. Thus, `GPGCHECK` gets the fingerprint as an awk `-v` in each command that uses it. In + that command, the call var `TL_KEY` is available. +- Some vars have a different name and read a var that a mirror can set, for example + `BATCHES_MAX` from `MAX_BATCHES`. Such a var also gets the value on the command line + (verified). Thus, write each default one time, inline. +- Do not put `dir:` on an engine verb. Task can read a flattened Taskfile of `includes:` + from a path. In that condition, it adds each `dir:`, also an absolute one, to the + directory of that Taskfile (verified, 3.53.1). Thus, `verify` starts each command with + `cd {{.STAGING}}`. +- Use only the keys `trusted-hosts` and `cache-expiry` in `.taskrc.yml`. Task ignores a key + that it does not know, and it shows no warning. In that condition, it downloads the remote + Taskfiles again each time that it runs. +- Each tool in an image is from `toolchain.lock.toml`. `docker/lock.py` reads that file + for the Dockerfiles and for the toolbox action, and the build examines each tool before + it uses it. Most tools have a recorded sha256. The AWS CLI zip has no sha256, because + upstream publishes only a detached PGP signature for it. Thus, the fetch stage verifies + that signature with `docker/aws-cli.pub`, and it must find the fingerprint that + `toolchain.lock.toml` pins. It uses the same GOODSIG and VALIDSIG awk check as the engine + uses for TeX Live. +- The images set `TASK_REMOTE_OFFLINE=1`. In a run, task gets the included Taskfiles from + the `.task/remote` cache of the mirror. This cache goes into the container with the + repository (a bind mount). Task does not get the Taskfiles from the network. +- `run` and `render` put the git top level of the repository at `/work` (a bind mount). + They set the working directory to the subdirectory of the Taskfile. For a mirror, the two + directories are the same. For the examples in this repository, this lets task find + `../../toolbox.yml`. +- `render` gets the dry run of task in the container, with `sh -c 'task ... 2>&1'`. Thus, + the status lines of the container runtime do not go into `render.txt`. +- In a `sh:` var, do not use `printf -- '-e %s'`. It prints dashes, because the built-in + shell of task reads the `--` as the format. Use `printf '%s %s ' -e "$v"`. +- The built-in shell of task runs each command with `set -e`. An assignment gets the exit + of its command substitution: `x=$(cat missing)` stops the command (verified, 3.53.1). +- The built-in shell of task keeps `set -e` in a subshell when `||` reads the status of + that subshell. `( false; echo on ) || echo caught` prints only `caught` (verified, + 3.53.1), but bash and dash print `on`. For a guard that must operate in all these shells, + make each step exit manually. +- The built-in shell of task has no `umask`. To make a file with the mode 0600, use + `install -m 600 /dev/null "$f"`. Then write the file. The `age` verb of the proton engine + does this. +- `pd` runs each command of the Proton CLI. After each command, with all exit codes, `pd` + examines the session, and it writes the session back to the bucket if the refresh token + changed. The token changes each time that the CLI uses the session. If a run keeps a + changed token and does not write it back, Proton rejects the token of the next run. A new + login is then necessary. For the same cause, two mirrors do not use one session. +- Pin each action to a full SHA, with the version in a comment at the end of the line. A + mirror pins the two reusable workflows in this method. Each reusable workflow does a + checkout of this repository at the commit of the workflow (`github.job_workflow_sha`) for + the toolbox action and `toolchain.lock.toml`. Thus, the pin of the workflow is the only + pin. The URLs in `includes:` use the git tag `v2`, and the image uses the tag + `-v2`. The two tags move. When they move, all mirrors get the release. +- Give `-r` to each `xargs` if a filter can make its input empty. GNU `xargs` runs its + command one time on empty input. A guard on the file that a pipe reads is not a guard on + the input of `xargs`. For example, the `SLASH` awk of `pages` removes the root, which has + no key without a slash. Thus, if the only changed directory of a run is the root, that + awk sends no line to `xargs`. +- A `>-` folded block keeps a newline if a line that continues the line before it starts + with more spaces than the lines around it. The awk programs of `pages` use this. Put a + backslash at the end of a shell line that must continue. `task check` shows how the block + renders. +- On a macOS disk, two directory names that are different only in uppercase and lowercase + letters are one directory. Thus, the pages that a laptop writes from a listing of + upstream are not complete if upstream has two such directories. For example, CTAN has + `obsolete/support/TeXshell/` and `texshell/`. The runner uses ext4, and it writes the two + directories. +- Put a YAML scalar that contains a literal `: ` between single quotation marks (`'`). + Write each `'` in it two times. Without the quotation marks, YAML reads the `: ` as a + key, also in `sed` and `awk` text in the line. +- In `verify`, read or write `.run/tl` only for a batch with a `TL` path. If not, a + redirect into a directory that no step made can stop a run. The fixtures and CI do not + show this problem. +- The `status:` check of `prepare` reads all of `changed.txt`, not the batch. Thus, + `prepare` runs on almost all runs. But if the delta of a run has no `TL` path, `prepare` + does not run, and `.run/tl` is not there. Each branch of `verify` reads the batch. The + container branch and the decision-batch branch use `.run/tl` only if the batch has a + `TL` path. +- Do not make a task internal if a host verb starts it with its name in the image. go-task + does not start an internal task from the command line. For example, `empty-trash` starts + `empty-trash-pipeline` with `task op`. Thus, `empty-trash-pipeline` is not internal. +- go-task reads the global vars of the Taskfiles that one Taskfile includes in a random + sequence at each run (verified, 3.53.1 and 3.54.0). Thus, an engine global var that reads + a toolbox var (`RUN`) can get that var with no value in the tasks of a mirror. For this + cause, `SESSION` and `UPLOAD` in `proton.yml` give `ROOT_DIR/.run` in full. Also, if a + mirror sets `IMAGE` in its root vars, a run can use the default of the engine. If the + toolbox entry of `includes:` sets `IMAGE`, the run always uses that value. ## Verifying a change ```sh cd examples/rsync && task image-build && task run -- task tools && task check && task run -- task offline cd examples/proton && task image-build && task run -- task tools && task check && task run -- task offline +sh .github/validate-vars.sh --check && sh .github/chain-file.sh --check && sh docker/gpg-gate-check.sh ``` -A verb change updates the `render.txt` files via `task render-update`; `offline` is -each engine's own check. The rsync one runs over `examples/rsync/fixtures/`: the list -diff over `run-root` and `run-empty`, `retry`'s exit codes, `pages` over `run-pages` -(whose `want/` is ctan's page set, matched byte for byte), `smoke`'s page read-back over -`run-smoke`, and `prepare` and `verify` over `tree/`, a signed subtree whose tlpdb is signed by a throwaway key pinned in the -example. `label`, `label-trees` and `fresh` run over files the check -writes itself, because their names and first bytes are the point: a change to the -image's `mime.types` that moves a label fails there, before anything uploads. Regenerate the tree with a new key only to change its shape; the private half -was never kept. The proton one runs `confirm` over `examples/proton/fixtures/`, an -accepting and a refusing upload summary, and the `age` verb round trip with a throwaway +After a change to a verb, run `task render-update` to update the `render.txt` files. +`offline` is the check of each engine. + +The rsync check runs on `examples/rsync/fixtures/`: + +- The list diff on `run-root` and `run-empty` +- The exit codes of `retry` +- `pages` on `run-pages`. Its `want/` is the page set of ctan, and the result must agree + with it byte for byte. +- The page read-back of `smoke` on `run-smoke` +- `prepare` and `verify` on `tree/`, a signed subtree. A test key, which the example pins, + signs its tlpdb. +- `label`, `label-trees` and `fresh`, on files that the check writes. The check examines + their names and their first bytes. Thus, if a change to the `mime.types` of the image + changes a label, the check stops there, before the engine uploads a file. + +Make the tree again, with a new key, only to change its shape. There is no copy of the +private half of the test key. + +The proton check runs `confirm` on `examples/proton/fixtures/`: an upload summary that it +accepts and one that it rejects. Then it does a dry run of `empty-trash-pipeline` from the +command line. Last, it encrypts and decrypts a file with the `age` verb and a test identity. -A README change is a diff `task check` cannot see; the review is reading it. Every -anchor a mirror's README links here (`#secrets`, `#storage`, `#the-rsync-engine`, -`#the-proton-engine`) is a heading in `README.md`; renaming one breaks four repositories. +`task check` does not examine a change to `README.md`. For that change, the review is the check. +Seven repositories link to headings of `README.md`: ctan, dropbox, github, gnu, nongnu, site +and tlnet. Do not change the text of these headings. If you change it, the links from +these repositories cannot find them: + +- `#secrets` +- `#storage` +- `#the-toolbox` +- `#the-rsync-engine` +- `#when-a-run-fails` +- `#content-types` +- `#the-proton-engine` +- `#the-session` +- `#r2-specifics` +- `#monitoring` +- `#rules-a-mirror-keeps`. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 73cbeaf..9d1ca03 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,24 +1,42 @@ # Contributing -The [organization's rules](https://github.com/katoptra/.github/blob/main/CONTRIBUTING.md) -apply. This repository adds three. - -- **No `vars:` default for anything a mirror owns.** A default declared in this file's - `vars:` block shadows the mirror's root value. Defaults go inline in the command as - `{{.X | default N}}`. -- **The verb names in `toolbox.yml` are reserved.** Host-side and in-container verbs - share one namespace when flattened into a mirror. Renaming one is a major version. -- **A change to a verb is a change to a `render.txt`.** A toolbox verb changes both - examples' files, an engine verb changes `examples/rsync/render.txt`. Run `task - render-update` in each example and commit the result with the change. The pull - request diff is the review. A verb that is `silent` renders nothing, so `report`'s - rows are reviewed in the diff of the file itself. +The [rules of the organization](https://github.com/katoptra/.github/blob/main/CONTRIBUTING.md) +are applicable to this repository. This repository adds three rules: + +- **Do not put a `vars:` default in `toolbox.yml` or in an engine for a var that a mirror + sets.** If a `vars:` block has a default, the run uses that default, not the root value + of the mirror. Put each default inline in the command: `{{.X | default N}}`. `IMAGE` is + different: each engine sets `IMAGE: '{{.IMAGE | default "..."}}'` in `vars:`. Thus, if + the toolbox entry of `includes:` gives `IMAGE`, the run uses that value. +- **Do not change the name of a verb in `toolbox.yml` or in an engine.** A mirror includes + these files with `flatten: true`. Then the host verbs and the verbs in the container use + one namespace. + A new name for one of these verbs is a new major version. +- **A change to a verb that a pipeline runs is a change to a `render.txt`.** `render` makes + a dry run of the `pipeline` of each example. A change to a verb changes these files: + - A toolbox verb: the `render.txt` of each example that runs it + - A verb of the rsync engine: `examples/rsync/render.txt` + - A verb of the proton engine: `examples/proton/render.txt`. + + A host verb, for example `run` or `session-seal`, is not in a render. Run + `task render-update` in each example. Commit the result with the change. The diff of the + pull request is the review. A verb with `silent: true` renders no command. Thus, for + `report` and `report-engine`, the review of their rows is in the diff of the file that + has the verb. ## Checking a change ```sh cd examples/rsync && task image-build && task run -- task tools && task check && task run -- task offline -cd examples/proton && task image-build && task run -- task tools && task check +cd examples/proton && task image-build && task run -- task tools && task check && task run -- task offline +sh .github/validate-vars.sh --check && sh .github/chain-file.sh --check && sh docker/gpg-gate-check.sh ``` -CI runs the same commands per image on every pull request. +CI does these steps for each image, on each pull request and on each push to `main`: + +1. It builds the image. +2. It runs `task run -- task tools`. +3. It runs `task check`. +4. It runs `task run -- task offline`. + +It also runs the three guards one time, in the job of the `rsync` image. diff --git a/README.md b/README.md index e1e0185..d62ec07 100644 --- a/README.md +++ b/README.md @@ -9,30 +9,37 @@

lib

-

The toolbox every katoptra mirror includes by URL.

+

The toolbox that all katoptra mirrors include from a URL.

ci license

-The rule it enforces: the code that -starts a run, contains it, resolves its secrets, checks it and reports it lives here, -once. The code that moves bytes for a transport lives here too, once per engine: -`engines/rsync.yml` for an rsync upstream into a bucket, `engines/proton.yml` for a -staging tree into Proton Drive. A mirror holds only its identity, the hooks it fills, and -the few verbs no other mirror shares. +This repository has one rule: the code that all mirrors use is here, one time. This code +does these tasks: -This file is the manual. A mirror's README says what it mirrors and how to fork it, and -links here for everything the mirrors share. Read it top to bottom once and the system -should hold no surprises; after that, each section stands on its own. +- It starts each run in a container. +- It gets the secrets of the run. +- It does a check of the pipeline, and it gives a report of each run. +- It moves the bytes. Each transport has one engine here: + - `engines/rsync.yml` copies an rsync upstream into a bucket. + - `engines/proton.yml` copies a staging tree into Proton Drive. -## The system in one picture +A mirror contains only its identity, the hooks that it fills, and the verbs that no other +mirror uses. + +This file is the manual. The README of a mirror gives the upstream that the mirror copies, +and the steps to fork it. For each part that all mirrors use, it has a link to this file. +Read this file one time, from the start to the end. This gives you the full system. After +that, you can read each section without the other sections. + +## The system in one diagram ```mermaid flowchart LR subgraph up["Upstreams"] - u1["CTAN, over rsync"] + u1["CTAN, GNU and Savannah, over rsync"] u2["GitHub, over the API and git"] u3["Dropbox, over the API"] end @@ -50,12 +57,22 @@ flowchart LR box --> hc["healthchecks.io, one ping per run"] --> site["katoptra.org, live status"] ``` -Every mirror is that picture with one upstream and one sink. A public mirror (ctan, -tlnet) copies an rsync tree into a bucket and serves the bucket; the bucket is the mirror. -A private mirror (github, dropbox) copies an account into Proton Drive and keeps only -what the next run needs in the bucket. Both run the same way: a job on GitHub Actions, -started by a dispatch, that pulls one image, resolves its secrets by name from a vault, -and runs `task pipeline` inside the image. The same command runs on a laptop. +Each mirror has the structure of this diagram, with one upstream and one sink. There are two +types of mirror: + +- A public mirror (ctan, tlnet, gnu and nongnu) copies an rsync tree into a bucket, and a + domain serves the bucket. The bucket is the mirror. +- A private mirror (github and dropbox) copies an account into Proton Drive. Its bucket + contains only the data that the next run uses. + +The two types of mirror do a run with the same steps: + +1. The scheduler starts a job on GitHub Actions. +2. The job downloads one image. +3. The job gets the secrets of the mirror from a vault, with the name of each secret. +4. The job runs `task pipeline` in the image. + +The same command runs on a laptop. ## The layers @@ -80,17 +97,18 @@ flowchart TB wf -- "task sync, task check" --> tb ``` -| Layer | Lives in | Owns | A mirror touches it when | +| Layer | Location | Contents | Does a mirror change it? | |---|---|---|---| -| Toolbox | `toolbox.yml` | How a run starts, is contained, resolves secrets, is rendered, checked and reported | Never; it is included as is | -| Engine | `engines/.yml` | How bytes move for one transport, and the pipeline's order | It lists a hook under `excludes:` and defines its own | -| Image | `docker/`, `toolchain.lock.toml` | Every tool a run needs, pinned by checksum | Never; its engine names one, or it names one in `IMAGE` | -| Workflows | `.github/workflows`, `.github/actions/toolbox` | How Actions installs the tools and runs `task sync` and `task check` | Never; its callers are a few lines each | -| Mirror | The mirror's own repository | Identity, the hooks, the exceptions | Always; this is all it holds | +| Toolbox | `toolbox.yml` | How a run starts, gets its secrets and stays in its container. The render, the check and the report of a run | No. A mirror includes it with no change | +| Engine | `engines/.yml` | How bytes move for one transport, and the sequence of the pipeline | Yes, to fill a hook: it puts the hook in `excludes:`, and it writes the hook | +| Image | `docker/`, `toolchain.lock.toml` | All the tools that a run uses. Each tool has a pin: its checksum | No. The engine gives the image, or the mirror gives it in `IMAGE` | +| Workflows | `.github/workflows`, `.github/actions/toolbox` | How Actions installs the tools and runs `task sync` and `task check` | No. Each caller in a mirror has only a small number of lines | +| Mirror | The repository of the mirror | The identity, the hooks, and the verbs that no other mirror uses | Yes, always. These items are all that a mirror contains | -## What a run looks like +## The steps of a run -The same path on a laptop and in Actions. On a laptop it starts at `task sync`. +A run has the same steps on a laptop and in Actions. On a laptop, the run starts at +`task sync`. ```mermaid sequenceDiagram @@ -99,7 +117,7 @@ sequenceDiagram participant H as Host: task participant O as op run participant C as Container: task - D->>W: workflow_dispatch, vars + D->>W: workflow_dispatch of the mirror's caller, no inputs W->>W: toolbox action installs task and op from the lock W->>H: task sync -- RECONCILE=... MAX_BATCHES=... H->>H: image: pull IMAGE if missing @@ -113,31 +131,34 @@ sequenceDiagram else the runner cut the run off W->>C: task op -- task failed else .run/chain exists - W->>D: gh workflow run, the caller's own file + W->>W: gh workflow run, the workflow file of the caller: the next run end ``` -Four things the diagram hides: - -- **Secrets cross by name, never by value.** `op run` resolves `op.env` on the host and - exports the values; `run` passes `-e NAME` for each name in `op.env` and in `PASS`, - plus `HEALTHCHECK_URL`, `GITHUB_STEP_SUMMARY` and `GITHUB_RUN_ID` always, so no value - ever appears on a command line or in a log. The whole path is under [Secrets](#secrets). -- **The vault is read once per run.** One `op run` wraps the whole pipeline, and the - failure path, `failed`, runs inside the same container, so a failed run costs no - second read. The one exception is a run the runner cut off, by the timeout or a - cancellation, which never reached its own failure path: the workflow runs `failed` - again from outside, one more read on a rare day. -- **Nothing inside the container reaches the network for task itself.** The image sets - `TASK_REMOTE_OFFLINE=1` and the mirror's `.task/remote` cache rides in with the repo. -- **`plan` is `sync` with the read-only half.** It runs `plan-pipeline` instead of - `pipeline`, inside the same image with the same secrets. +The diagram does not show these four items: + +- **Secrets go into the container with their names, not with their values.** On the host, + `op run` gets the value of each reference in `op.env`, and it exports the values. For + each name in `op.env` and in `PASS`, `run` gives `-e NAME` to the container. It also + always gives `HEALTHCHECK_URL`, `GITHUB_STEP_SUMMARY` and `GITHUB_RUN_ID`. Thus, no value + shows on a command line or in a log. [Secrets](#secrets) gives the full path. +- **A run reads the vault one time.** One `op run` command runs all of the pipeline, and + the failure path, `failed`, runs in the same container. Thus, a run with a failure does + not read the vault a second time. If the runner stops a run, at the timeout or because a + person cancels it, the run does not start its failure path. Then the workflow runs + `failed` again from the host, and this reads the vault one more time. Such a run does + not occur frequently. +- **In the container, task does not use the network for its Taskfiles.** The image sets + `TASK_REMOTE_OFFLINE=1`. The `.task/remote` cache of the mirror goes into the container + with the repository. +- **`plan` is `sync` with only the read-only half.** It runs `plan-pipeline`, not + `pipeline`, in the same image and with the same secrets. ## The toolbox -`toolbox.yml` is every verb a mirror has whatever moves its bytes. Half of its verbs run -on the host and wrap the container; the other half run inside it and are the first and -last steps of every pipeline. +`toolbox.yml` contains the verbs that all mirrors have, with all engines. Half of these +verbs run on the host and start the container. The other half run in the container: they +are the first steps and the last steps of each pipeline. ```mermaid flowchart TB @@ -157,106 +178,136 @@ flowchart TB ### Host side -The contract the workflows call and the menu documents. A mirror does not override these. +These verbs are the contract that the workflows use and that the menu shows. A mirror does +not override them. -| Verb | Does | +| Verb | Function | |---|---| -| `default` | The grouped menu, from `NAME`, `DESC` and `MENU` | -| `sync -- [K=v ...]` | `op -- task pipeline`: one run. The words after `--` are task vars for the pipeline inside the image | +| `default` | Prints the menu in groups, from `NAME`, `DESC` and `MENU` | +| `sync -- [K=v ...]` | `op -- task pipeline`: one run. The words after `--` are task vars for the pipeline in the image | | `plan -- [K=v ...]` | `op -- task plan-pipeline`: the read-only half | -| `check` | `render`, then diff against `render.txt` | -| `render-update` | `render`, then accept it as `render.txt` | -| `render` | `task --dry --force pipeline` inside the image, over an empty `.run`, saved to `.run/render.txt`; no secrets cross | -| `run -- ` | Anything inside the image, with the repository at `/work` | -| `op -- ` | `run`, wrapped in `op run --env-file=op.env` when the file exists | -| `image` | Pull `IMAGE`; a no-op while it exists locally | -| `image-build` | Build `IMAGE` from `LIB_DIR/docker/.Dockerfile`; `LIB_DIR` defaults to `../lib` | -| `image-clean` | Remove `IMAGE` | -| `clean` | Delete `.run`, `staging` and the Taskfile cache; nothing else | - -The container engine is Apple `container` when its daemon is up, else Docker; -`ENGINE=docker` forces Docker. The host needs go-task, the container engine, and the -1Password CLI for anything that reads the vault. Nothing else is installed on a host. - -### Inside the image - -The first and last verbs of every pipeline. A mirror or engine may override these when it -keeps its own. - -| Verb | Does | +| `check` | `render`, then a diff with `render.txt` | +| `render-update` | `render`, then it copies the render to `render.txt` | +| `render` | `task --dry --force pipeline` in the image, with an empty `.run`. It writes the result to `.run/render.txt`. No secret goes into the container | +| `run -- ` | Runs a command in the image, with the repository at `/work` | +| `op -- ` | `run`, in `op run --env-file=op.env` if the repository has `op.env` | +| `image` | Downloads `IMAGE`. While the host has the image, it does not download it again | +| `image-build` | Builds `IMAGE` from `LIB_DIR/docker/.Dockerfile`. The default `LIB_DIR` is `../lib` | +| `image-clean` | Removes `IMAGE` | +| `clean` | Deletes `.run`, `staging` and the Taskfile cache, and no other file | + +The container runtime (`ENGINE`) is Apple `container` when its daemon operates. If not, it +is Docker. `ENGINE=docker` selects Docker. A host has only these tools: + +- go-task +- The container runtime +- The 1Password CLI, for a task that reads the vault. + +### In the image + +These verbs are the first verbs and the last verbs of each pipeline. A mirror or an engine +can override one of them with a verb of the same name. + +| Verb | Function | |---|---| -| `clock` | Write the run's start epoch to `.run/start.txt` and forget the last run's chain file and `.run/reconcile` | -| `due` | Decide whether this run reconciles, by the age of the last reconcile at `.state/reconciled`; leave `.run/reconcile` when it does. `due-rule` is the rule alone | -| `reconciled` | Record this run's start at `.state/reconciled`; an engine runs it once a reconcile completes | -| `report` | Append the run summary to the Actions job page (stdout elsewhere): its own rows, then `report-engine`'s, then `report-mirror`'s | -| `report-mirror` | Hook: a no-op here; the mirror's rows | -| `ping` | GET `HEALTHCHECK_URL`; skipped when unset | -| `ping-fail` | GET `HEALTHCHECK_URL/fail`; skipped when unset | +| `clock` | Writes the start epoch of the run to `.run/start.txt`. It deletes the chain file and the `.run/reconcile` of the last run | +| `due` | Finds if this run does a reconcile, from the time of the last reconcile in `.state/reconciled`. If it does, `due` writes `.run/reconcile`. `due-rule` is the rule, and it does not download the file | +| `reconciled` | Records the start of this run in `.state/reconciled`. An engine runs it when it completes a reconcile | +| `report` | Adds the run summary to the job page in Actions. If it does not run in Actions, it writes the summary to stdout. The report has the rows of the toolbox first, then the rows of `report-engine`, then the rows of `report-mirror` | +| `report-mirror` | A hook, empty here, for the rows of the mirror | +| `ping` | Sends a GET to `HEALTHCHECK_URL`. If `HEALTHCHECK_URL` is empty, it sends no request | +| `ping-fail` | Sends a GET to `HEALTHCHECK_URL/fail`. If `HEALTHCHECK_URL` is empty, it sends no request | | `failed` | The failure path: `report STATUS=failed`, then `ping-fail` | -`sync` runs `pipeline`, and on a failure `failed`, all in the one container. The report -is one table in three layers, each a `| Label | value |` line appended to the same file: -the toolbox's rows (when the run started and how long it took, the image, whether the -next run is queued), the engine's, the mirror's. Every row tolerates a missing file, -because the report also runs after a failed pipeline. +`sync` runs `pipeline` in one container. If the pipeline has a failure, `sync` runs +`failed` in the same container. The report is one table with three layers. Each row is a +`| Label | value |` line that `report` adds to the same file. The rows are in this +sequence: -#### Reconcile, by age +1. The rows of the toolbox. They give the start time and the length of the run, and the + image. They also give the time of the next run: immediately with the chain, or at its + next slot. +2. The rows of the engine +3. The rows of the mirror. -A reconcile, the pass that checks the mirror against its destination rather than -against its own state, is due by the age of the last one and never by when a run starts: -the schedule lives outside the mirror and can move. `due` runs early in a pipeline that -reconciles, and the pipeline's reconcile runs only when `.run/reconcile` is there. +Each row operates if a file is missing, because the report also runs after a pipeline with +a failure. -| Var | Default | Meaning | +#### When a reconcile runs + +A reconcile compares the mirror with its destination, not with its state. The time of the +last reconcile gives the time of the next one. The start time of a run does not give it, +because the scheduler is not in the mirror, and its times can change. A pipeline that does a +reconcile runs `due` near its start. Its reconcile step runs only if `.run/reconcile` is +there. + +| Var | Default | Function | |---|---|---| -| `RECONCILE` | `auto` | Set per run: `task sync -- RECONCILE=true`. `true` reconciles, `false` does not, `auto` reconciles once the last is `RECONCILE_HOURS` old or none is on record | -| `RECONCILE_HOURS` | 24 | Whole hours between reconciles. A mirror that wants another interval sets it in its root vars | +| `RECONCILE` | `auto` | Each run can set it: `task sync -- RECONCILE=true`. With `true`, the run does a reconcile. With `false`, it does not. With `auto`, it does a reconcile if no run recorded a reconcile. It also does one if the last reconcile occurred `RECONCILE_HOURS` or more before the run | +| `RECONCILE_HOURS` | 24 | The number of full hours between two reconciles. For a different interval, a mirror sets it in its root vars | + +`.state/reconciled` contains the start epoch of the run that completed the last reconcile. +The `pull` and `push` of the engine read and write this file. Thus, with the rsync +engine, the file has xz compression, the same as the state. -The last reconcile is the start epoch of the run that finished it, at `.state/reconciled`, -kept through the engine's `pull` and `push`, so under the rsync engine it is -xz-compressed like the state. The mark is half an hour short of `RECONCILE_HOURS`: a -daily run starts seconds either side of 24 h after the one before, and a strict 24 h -would reconcile every other day. A run that fails before its reconcile completes -records nothing, so the next run is due as well, and so does an rsync run with batches -still waiting: its reconcile holds for the chained run that lands the last one. A mirror with no engine that -reconciles defines `pull` and `push` itself. +The limit is half an hour less than `RECONCILE_HOURS`. A daily run starts some seconds +before or after the time that is 24 h after the last run. With a limit of 24 h and no +less, a reconcile occurs only on each second day. -### What a mirror gives the toolbox +In these conditions, a run records no reconcile, and the reconcile is also necessary at the +next run: -| Name | Kind | Meaning | +- The run stops with a failure before it completes its reconcile. +- An rsync run has batches that wait for a subsequent run. Its reconcile waits for the + run of the chain that uploads the last batch. + +### The parts that a mirror supplies + +| Name | Type | Function | |---|---|---| -| `NAME`, `DESC` | include vars, required | The menu's title and line | -| `IMAGE` | include var, required unless the engine names it | The image the run happens inside. The include's value wins over the engine's | -| `pipeline`, `plan-pipeline` | tasks, required | The full run and its read-only half, inside the image. An engine supplies both; a mirror with no engine defines them | -| `report-engine` | task, required | The engine's rows of the run summary, which `report` calls. An engine supplies it; a mirror with no engine defines it, with no commands when it has no rows | -| `op.env` | file | `op://` references, one per secret. Absent means the environment is already resolved: on a laptop, whatever is exported; in Actions, the repository's secrets, crossing by the names in `PASS` | -| `PASS` | include var | Host environment names that cross into the container beside the ones in `op.env`. Task vars are not environment: they go after `--` | -| `MENU` | include var | Extra lines for the menu, one per mirror-specific verb | -| `report-mirror` | task | The mirror's rows of the run summary. Excluded on the toolbox include | -| `LIB_DIR` | include var | Where `image-build` finds `docker/`; defaults to `../lib` | -| `RECONCILE_HOURS` | root var | Hours between reconciles, when not 24. See [Reconcile, by age](#reconcile-by-age) | -| `pull`, `push` | tasks | What `due` and `reconciled` keep `.state/reconciled` through. An engine supplies both; a mirror with no engine that reconciles defines them | -| `excludes:` | include key | Library verbs the mirror replaces. See [Changing it](#changing-it) | +| `NAME`, `DESC` | Vars in `includes:`, necessary | The title and the line of the menu | +| `IMAGE` | Var in `includes:`, necessary if the engine does not give it | The image in which the run operates. If there is a value in `includes:` and also in the engine, the run uses the value in `includes:` | +| `pipeline`, `plan-pipeline` | Tasks, necessary | The full run and its read-only half, in the image. An engine supplies the two tasks. A mirror with no engine, or that puts them in `excludes:`, writes them | +| `report-engine` | Task, necessary | The rows of the engine in the run summary. `report` runs it. An engine supplies it. A mirror with no engine, or that puts it in `excludes:`, writes it, with no commands if it has no rows | +| `op.env` | File | `op://` references, one for each secret. With no `op.env`, the environment has the values before the run starts. On a laptop, these are the exported values. In Actions, the secrets of the caller go into the container with the names in `PASS` | +| `PASS` | Var in `includes:` | The names of host environment variables that go into the container. The names in `op.env` also go into the container. Task vars are not environment variables: they go after `--` | +| `MENU` | Var in `includes:` | More lines for the menu, one for each verb that only this mirror has | +| `report-mirror` | Task | The rows of the mirror in the run summary. The mirror puts it in `excludes:` in the toolbox entry of `includes:` | +| `LIB_DIR` | Var in `includes:` | The location where `image-build` finds `docker/`. The default is `../lib` | +| `RECONCILE_HOURS` | Root var | The hours between two reconciles, if not 24. Refer to [When a reconcile runs](#when-a-reconcile-runs) | +| `pull`, `push` | Tasks | The tasks that `due` and `reconciled` use to read and write `.state/reconciled`. An engine supplies the two tasks | +| `excludes:` | Key in `includes:` | The verbs of lib that the mirror overrides. Refer to [Changing it](#changing-it) | ## The engines -An engine is a second include with the verbs for one transport. Its verbs are the -pipeline vocabulary, reserved across every engine so a mirror's pipeline reads the same -whichever transport it uses. An engine reads the mirror's root vars and the environment; -it declares no `vars:` default for anything a mirror owns, because a value there would -shadow the mirror's. Every tunable is an inline `{{.X | default N}}`. +An engine is a second entry in `includes:`, with the verbs for one transport. The names of +these verbs are the same in all engines, and a mirror cannot use these names for a +different verb. Thus, the pipeline of a mirror has the same verbs with each engine. + +An engine reads the root vars of the mirror and the environment. It has no `vars:` default +for a var that a mirror sets. With such a default, the run uses the default, not the value +of the mirror. Each var that a mirror can set has its default inline: +`{{.X | default N}}`. -The hooks are the verbs an engine leaves empty for a mirror to fill. A mirror lists each -it defines under `excludes:` on the engine's include. Extension is a hook, never a copy: -a mirror that needs the engine's `smoke` and more defines `smoke-mirror`, which `smoke` -runs last. +`IMAGE` is different. The engine sets it in `vars:` as `{{.IMAGE | default "..."}}`. Thus, +the run uses an `IMAGE` in the toolbox entry of `includes:`, not the default of the engine. +If a mirror sets `IMAGE` in its root vars, the run does not always use that value. + +The hooks are the empty verbs of an engine, for a mirror to fill. For each hook that a +mirror writes, the mirror puts the hook in `excludes:` in the engine entry of `includes:`. +To add to a verb, use a hook, not a copy. For example, a mirror that wants the `smoke` of +the engine and more checks writes `smoke-mirror`. `smoke` runs `smoke-mirror` last. ### The rsync engine -`engines/rsync.yml`: an rsync upstream into an S3 bucket, as a list diff and never a -local tree. The runner has 14 GB of disk and CTAN has 140; a run lists upstream, diffs -the listing against the state file the last run left in the bucket, and moves only the -delta, in batches, each committed before the next starts. +`engines/rsync.yml` copies an rsync upstream into an S3 bucket. It compares two listings, +and it does not keep a local copy of the tree. The runner has 14 GB of disk, and CTAN has +140 GB. A run does these steps: + +1. It makes a listing of upstream. +2. It compares the listing with the state file that the last run put in the bucket. +3. It copies only the delta, in batches. `checkpoint` records each batch in the state + before the next batch starts. ```mermaid flowchart LR @@ -273,39 +324,62 @@ flowchart LR class prepare,verify,index,sm,re,rm hook ``` -Dashed verbs are hooks. `prepare` and `verify` do nothing until `TL_KEY` is set, `index` -until `INDEX` is set, and `smoke-mirror` until a mirror fills it. +The verbs with a broken line are hooks. These hooks do no work until a mirror sets a value: + +- `prepare` and `verify` do no work until `TL_KEY` has a value. +- `index` does no work until `INDEX` has a value. +- `smoke-mirror` does no work until a mirror fills it. -| Verb | Does | +| Verb | Function | |---|---| -| `pipeline`, `plan-pipeline` | The verbs above in order, and the read-only half: `clock`, `list`, `state`, `diff`, `split` | -| `list` | `rsync -rL --list-only` of `SOURCE`, through `FILTER`, normalised to `.run/upstream.txt` as `path TAB size TAB mtime`, byte-sorted; a listing under `LIST_FLOOR` lines stops the run | -| `state` | Fetch `.state/applied.txt.xz` from the bucket; a missing one asks `rebuild` | -| `rebuild` | List the bucket and make the state exactly what it holds, at upstream's sizes | -| `diff` | `changed.txt` (upstream has, the state lacks, plus the file each changed signed sha512 names), `deleted.txt` (the state has, upstream lacks), `paths.txt` | -| `split` | Refuse a tree over `CEILING_GB` or a file the disk cannot hold; split the delta into `batch-NNNN.txt` of `BATCH_GB`, the decision batch last | -| `prepare` | Hook. With `TL_KEY` set and the delta touching `TL`: fetch the tlpdb and check its signature against the pinned key | -| `batches` | Work the first `MAX_BATCHES`, each `fetch`, `verify`, `publish`, `checkpoint`; touch `.run/chain` when batches remain | -| `fetch` | rsync the batch's files into `staging/`, dereferencing symlinks; a path that vanished since the listing is skipped | -| `verify` | Hook. With `TL_KEY` set: every signed file and every container in the batch against the tlpdb | -| `publish` | `label` the batch, then `aws s3 cp --recursive` once per type, each type's files in a tree of their own, one PutObject per file, never a destination listing; the `FRESH_KEY` file last | -| `label` | `.run/labels.txt`: every staged file's Content-Type, from its name and first KiB; see [Content types](#content-types) | -| `checkpoint` | `merge` what landed into the state and push it as one PutObject; empty staging | -| `delete` | Remove the keys upstream dropped, 1,000 per call, once every batch has landed, and drop them from the state | -| `reconcile` | When `due` left `.run/reconcile`: rebuild the state, delete what neither upstream, `OWN` nor the state's own directories own, then `reconciled` | -| `index` | Hook. With `INDEX` set: `pages`, then both key sets uploaded, the keys of emptied directories removed, and `.state/indexed.txt.xz` moved forward. A mirror with a landing page of its own replaces it | -| `pages` | A page for every directory the run touched, drawn from the state into staging, with a tree per depth for the slashless keys | -| `smoke` | A key of every type the run published, read back through `HOST`: size against the listing, Content-Type against the label; the tlpdb sha512 when `TL` is set; with `INDEX`, one redrawn page under both keys; with `CANARY`, that file as `libwww-perl` against the bucket's copy; no `Content-Encoding` on the first non-empty `.tar.gz`; a warning for every path upstream lists and rsync never sends; then `fresh` and `smoke-mirror` | -| `fresh` | With `FRESH_KEY` set: read upstream's clock in that file through `HOST` and fail the run past `FRESH_HOURS` | -| `smoke-mirror` | Hook. Nothing here; a mirror with more to read back defines it | -| `report-engine` | Hook. The engine's rows of the run summary | -| `retry` | Run a command, retrying rsync's transport exit codes with backoff; 23 (an unreadable path, skipped) and 24 (a file vanished mid-transfer) are successes | - -`normalise`, `pull`, `push`, `batch`, `label-trees`, `merge` and `remove` are the verbs those call; -`pull` and `push` move an xz-compressed key between the bucket and `.run`, and a -mirror's own verb may call them. - -#### The bucket is the mirror; the state is a cache +| `pipeline`, `plan-pipeline` | The verbs in the diagram, in sequence, and the read-only half: `clock`, `list`, `state`, `diff`, `split` | +| `list` | `rsync -rL --list-only` of `SOURCE`, with `FILTER`. `normalise` writes it to `.run/upstream.txt` as `path TAB size TAB mtime`, in the sequence of byte values. A listing that does not have more lines than `LIST_FLOOR` stops the run. If rsync cannot read a directory, it exits 23, and `retry` accepts exit 23: the run continues. The listing then does not have the files of that directory, and `delete` removes them from the bucket, the same as files that upstream deleted. When rsync can read the directory again, a run fetches its files again | +| `state` | Downloads `.state/applied.txt.xz` from the bucket. If the file is missing, `state` tells `rebuild` to make it | +| `rebuild` | Makes a listing of the bucket. The new state has each key that the bucket contains with the number of bytes that upstream gives. It uploads the new state as one PutObject | +| `diff` | Writes `changed.txt`, `deleted.txt` and `paths.txt`. `changed.txt` has the lines that upstream has and the state does not have. It also has the file of each changed signed sha512. `deleted.txt` has the paths that the state has and upstream does not have | +| `split` | Rejects a tree that is larger than `CEILING_GB`, or a file that the disk cannot contain. It divides the delta into `batch-NNNN.txt` files of `BATCH_GB` each, with the decision batch last | +| `prepare` | A hook. If `TL_KEY` has a value and the delta has a `TL` path: it fetches the tlpdb and compares its signature with the pinned key | +| `batches` | Runs `fetch`, `verify`, `publish` and `checkpoint` for each of the first `MAX_BATCHES` batches. If there are remaining batches, it makes the file `.run/chain` | +| `fetch` | Copies the files of the batch into `staging/` with rsync, and copies the target of each symlink. If upstream deletes a path after the listing, `fetch` does not copy it | +| `verify` | A hook. If `TL_KEY` has a value: it compares each signed file and each container in the batch with the tlpdb | +| `publish` | Runs `label` on the batch. Then it runs `aws s3 cp --recursive` one time for each type, with the files of each type in a different tree. It sends each file as one PutObject, and it makes no listing of the bucket. The engine does not use `aws s3 sync`. The `FRESH_KEY` file goes last | +| `label` | Writes `.run/labels.txt`: the Content-Type of each file in staging, from its name and its first KiB. Refer to [Content types](#content-types) | +| `checkpoint` | Runs `merge` to add the uploaded files to the state. Then it uploads the state as one PutObject, and it deletes the files in staging | +| `delete` | Removes the keys in `deleted.txt` from the bucket, 1,000 in each call, and removes them from the state. It waits for the run that uploads the last batch | +| `reconcile` | If `due` wrote `.run/reconcile`, it runs `rebuild`. Then it deletes each key that is not a path of upstream, a key in `OWN`, or a directory of the state. Then it runs `reconciled` | +| `index` | A hook. If `INDEX` has a value: it runs `pages`, uploads the two sets of keys, and removes the keys of the directories that became empty. Then it moves `.state/indexed.txt.xz` to the new state. A mirror with a different landing page overrides `index` | +| `pages` | Writes a page for each directory that the run changed, from the state, into staging. The keys with no slash at the end go in one tree for each depth | +| `smoke` | Reads a sample back from `HOST`. The list below the table gives each item of the sample. Then it runs `fresh` and `smoke-mirror` | +| `fresh` | If `FRESH_KEY` has a value: it reads the clock of upstream in that file, from `HOST`. If the clock is more than `FRESH_HOURS` before the time of the run, it stops the run | +| `smoke-mirror` | A hook, empty here. A mirror with more items to read back writes it | +| `report-engine` | A hook: the rows of the engine in the run summary | +| `retry` | Runs a command a maximum of five times. After a transport error of rsync (exit 5, 10, 12, 30 or 35), it runs the command again, with backoff. It accepts exit 23 and exit 24, and the run continues. rsync exits 23 if it cannot read a path, and 24 if upstream deletes a file while rsync sends it. Thus, a directory that rsync cannot read does not stop the run, and `delete` removes its files, the same as files that upstream deleted. Only `LIST_FLOOR` stops a large loss | + +The verbs in the table also run these verbs: `normalise`, `pull`, `push`, `batch`, +`label-trees`, `merge` and `remove`. `pull` and `push` move a key with xz compression +between the bucket and `.run`. A verb of a mirror can also run them. + +`smoke` reads these items back from `HOST`, and it stops the run if one of them is not +correct: + +- One key of each type that the run published. Its length in bytes must agree with the + listing, and its Content-Type must agree with the label. +- The `FRESH_KEY` file, for its length. `smoke` reads it only if the state has it. While + the engine fills an empty bucket, the runs that stop at `MAX_BATCHES` do not upload the + decision batch. The bucket gives 404 for a key that it did not contain before. +- If `TL` has a value: `texlive.tlpdb.sha512`. It must be the same as the verified copy. +- If `INDEX` has a value: one page that `index` wrote, at its two keys. +- If `CANARY` has a value: that file, read as `libwww-perl`. It must be the same as the copy + in the bucket, byte for byte. +- The first `.tar.gz` that is not empty. It must have no `Content-Encoding`. + +`smoke` reads one byte of an object, and it gets the length of the object from the +headers. It does not use a HEAD: Cloudflare sends no `content-length` for `text/html`. +`smoke` also finds each path in the listing of upstream that rsync did not send. Each run +fetches these paths again, the report counts them, and the first 20 are warnings on the +run. + +#### The bucket is the mirror, and the state is a cache ```mermaid flowchart LR @@ -319,37 +393,55 @@ flowchart LR reconcile -- "rebuild, then delete what neither
upstream, OWN nor the state's directories own" --> bucket ``` -An hourly run costs one listing of upstream and none of the bucket. Three things follow. - -- **There is no seed.** A missing state file is rebuilt from a listing of the bucket. An - empty bucket gives an empty state, so the first run's delta is the whole tree, worked - `MAX_BATCHES` at a time; when batches remain the run chains the next one, and the fill - converges in a few chained runs with no flag. A lost state file costs the same listing - and nothing else. -- **The state records what landed.** `merge` joins the batch against what is in staging, - so a path that vanished upstream between listing and fetch never enters the state, and - the state never names a key the bucket lacks. -- **`RECONCILE` is the check on a live mirror.** It rebuilds the state from the bucket on - purpose, once every `RECONCILE_HOURS` (24) by default, and deletes keys neither upstream - nor the state owns. - -What a mirror cannot afford to lose is the bucket. Everything else, the state file and the -staging tree included, is derived from it and from upstream. +An hourly run makes one listing of upstream, and no listing of the bucket. The state line +is `path TAB size TAB mtime`, with the path first, in the sequence of byte values. Each +`sort`, `comm` and `join` of a listing runs with `LC_ALL=C`, because with a different +sequence `comm` compares the incorrect lines. This has three results: + +- **The engine fills an empty bucket with no other step.** If the state file is missing, + `rebuild` makes it from a listing of the bucket. An empty bucket gives an empty state, + and the delta of the first run is the full tree. Each run does `MAX_BATCHES` batches, and + it starts the next run with the chain until the bucket is full. No flag is necessary. The + cost of a missing state file is one listing. +- **The state records only the uploaded files.** `merge` compares the batch with the files + in staging. If upstream deletes a path after `list` runs and before `fetch` runs, the path + does not go into the state. Thus, the state has no key that the bucket does not have. +- **`RECONCILE` is the check on a mirror that serves clients.** A reconcile always makes + the state from the bucket again, one time in each `RECONCILE_HOURS` (the default is 24). + Then it deletes the keys that upstream and the state do not have. + +The engine writes the state one time for each batch, after it uploads the batch, as one +PutObject. If +a run stops before its end, the next run does one batch again, at most. A run that stops +at `MAX_BATCHES` with remaining batches is not a failure. + +The bucket is the one item that a mirror cannot make again. The engine makes all other +data again from the bucket and from upstream. This includes the state file and the staging +tree. #### The decision batch -`split` puts last whatever a client reads to decide what to fetch: the signed subtree's -`tlpkg/` and every bucket-root file, `timestamp` among them. Containers land before the -tlpdb that names them, and `delete` waits for the run in which every batch has landed, so -the live tlpdb never names a removed container and a `tlmgr` run that overlaps a publish -sees the previous tlpdb, never one naming a file that is not there. +`split` puts these files in the last batch: the files that a client reads to find the files +to fetch. These files are the `tlpkg/` of the signed subtree and each file at the root of +the bucket, for example `timestamp`. Thus, each container is in the bucket before the tlpdb +that gives its name. `delete` waits for the run that uploads the last batch. These are the +results: + +- The tlpdb that clients read does not give the name of a container that `delete` removed. +- If `tlmgr` runs while the engine publishes a batch, it gets the last tlpdb. It does + not get a tlpdb that gives a file that is not in the bucket. #### A signed subtree -With `TL` and `TL_KEY` set, the engine verifies TeX Live's signatures before anything is -published. The keyring is fetched from the same mirror as the signature, so the pinned -fingerprint is the only real check; `GOODSIG` is required too, because gpgv reports an -expired or revoked key as `VALIDSIG` with exit 0. +If `TL` and `TL_KEY` have values, the engine verifies the signatures of TeX Live before it +publishes a file. `prepare` runs after `split`, before the first batch. `verify` runs on +each batch, before the engine uploads the batch. + +The engine fetches the keyring from the same mirror as the signature, and it reads the +keyring as a plain file. Thus, gpgv imports no key before the check. Only the pinned +fingerprint is a check that the mirror cannot change: `VALIDSIG` must have `TL_KEY` at its +end. The engine also must find `GOODSIG`, because gpgv shows an expired or revoked key as +`VALIDSIG` with exit 0. `tlmgr` accepts such a key, but the engine does not. ```mermaid flowchart TB @@ -364,88 +456,227 @@ flowchart TB p --> v ``` -A batch that fails any check stays local; the previous good copy stays live. `smoke` -reads `texlive.tlpdb.sha512` back through the domain afterwards and compares it with the -verified copy. `tlmgr` repeats the signature check on the client. +`verify` does these checks: + +- It verifies each signed sha512 in the batch: `texlive.tlpdb.sha512`, and each sha512 at + the root of the tree, for example the `install-tl*.sha512` and + `update-tlmgr-latest.*.sha512` files. +- It compares each container in the batch with the `containerchecksum`, + `doccontainerchecksum` or `srccontainerchecksum` in the verified tlpdb. A source container + is `.source.tar.xz`. If the tlpdb does not give a container, `verify` stops the run. +- In the decision batch, `texlive.tlpdb.xz` must decompress to the verified `texlive.tlpdb`, + byte for byte. Each container that the tlpdb gives must be in the bucket after this run. + +Upstream keeps each container with a name that has its revision, for example +`foo.r123.tar.xz`, and a symlink `foo.tar.xz` points to it. `fetch` uses `-L`. Thus, a +mirror serves the two names, unless its `FILTER` removes the names with a revision. The +`FILTER` of tlnet removes them. `verify` finds the checksum for the two names. It gets the +revision from the `revision` line of the stanza in the tlpdb. + +If a check finds a problem in a batch, the engine does not upload that batch, and clients +continue to get the last good copy. After the engine uploads the batch, `smoke` reads +`texlive.tlpdb.sha512` back from the domain, and compares it with the verified copy. +`tlmgr` does the signature check again on the client. #### Directory pages -R2 serves no listings. With `INDEX` set, `index` draws a page for every directory a run -touched, from the state rather than from upstream, and writes it under two keys: -`/`, which a zone Transform Rule serves for `/dir/`, and the bare ``, -which serves `/dir` where a filesystem mirror would answer 301. A `` lets one -document serve both. `.state/indexed.txt.xz` records the state the pages last showed, so a -run that dies before moving it redraws the same pages next time. Deleting it redraws every -page once, which is also how a change to the markup or to `PAGE_FOOT` reaches directories -that have not changed. `reconcile` spares both keys. +R2 serves no directory listings. If `INDEX` has a value, `index` writes a page for each +directory that a run changed. It writes the pages from the state, not from upstream, and +each page has two keys: + +- `/`: a Transform Rule of the zone serves this key for `/dir/`. +- ``, with no slash at the end: this key serves `/dir`. A mirror on a file system + sends 301 for that URL. The URL does not show if a name is a file or a directory. Thus, + no Cloudflare rule can send 301. -The zone rule, one per host: +A `` lets one document serve the two URLs. A file system cannot contain `a/b` and +`a/b/` at the same time. Thus, `pages` puts the keys with no slash in one tree for each +depth, and `index` uploads one tree at a time. `index` gives these keys +`--content-type text/html`. Without it, a key with no suffix gets `binary/octet-stream`, +and a browser downloads the page and does not show it. - when: ends_with(http.request.uri.path, "/") - rewrite path: concat(http.request.uri.path, http.host, ".directory.index.html") +No page goes into the state. `merge` adds to the state only the files that are in staging +and also in the batch, and a page is in no batch. -It serves the root page for `/` too. ctan excludes `/`, which is CTAN's own `index.html`. +`.state/indexed.txt.xz` records the state that the pages showed last. If a run stops before +it moves that file, the next run writes the same pages again. If you delete the file, the +next run writes all the pages again, one time. Thus, the pages of the directories that did +not change also get a change to the HTML of the page or to `PAGE_FOOT`. `reconcile` does +not delete the two keys. + +The zone has one rule for each host: + +```text +when: ends_with(http.request.uri.path, "/") +rewrite path: concat(http.request.uri.path, http.host, ".directory.index.html") +``` + +This rule also serves the page of the root for `/`. In ctan, the rule does not include `/`, +because `/` serves the `index.html` of CTAN. #### Content types -R2 serves the Content-Type an object was stored with, and the AWS CLI, left to itself, -stores the first half of Python's guess: `foo.diff.gz` guesses as a gzipped diff and is -stored as `text/x-diff`, which a browser draws as text. Its guesses also move with the -image, so one type can be stored two ways in one bucket. So `label` decides every type and -the CLI guesses none: - -1. A name ending in a `COMPRESSED` ending (`.gz`, `.tgz`, `.bz2`, `.xz`, `.Z`, `.zst`, - `.lz`, ...) is that compressed type, whatever the rest of the name says. The types are - ftp.gnu.org's. -2. An ending Python reads as compressed and `COMPRESSED` lacks gets a signature's type or - `application/octet-stream`, never text. -3. Otherwise the image's `mime.types` names it, unless the name says a type a browser draws - (text, XML, JSON, JavaScript, PDF) and the first KiB disagrees: a compression - signature, a NUL, a PDF without `%PDF-`. Then the bytes decide. -4. A name with no type is typed from the bytes: a known signature, `text/plain` without a - NUL, `application/octet-stream` with one. - -A stray control byte is not a NUL, so an old `.sty` ending in `^Z` stays `text/x-tex`. Three -checks hold it: `offline` labels a set of names and bytes and fails on any change, which is -where an image bump that moves `mime.types` shows; `smoke` reads one key of every type -back through the domain and compares the served type with the label; and the -`.tar.gz` check keeps `Content-Encoding` off tarballs. +R2 serves the Content-Type that the object has in the bucket. Without a type in the +command, the AWS CLI uses the first half of the type that Python gets from the name. For +`foo.diff.gz`, Python gives a diff with gzip compression, and the CLI then stores +`text/x-diff`. A browser then shows the file as text. + +The result of Python also changes with the image. Thus, the same type of file can have two +different Content-Types in one bucket. For these causes, `label` sets each type, and the +CLI sets no type: + +1. A name with a `COMPRESSED` extension (for example `.gz`, `.tgz`, `.bz2`, `.xz`, `.Z`, + `.zst` and `.lz`) gets that compressed type. The other parts of the name have no effect. + The types are the types of ftp.gnu.org. +2. An extension that Python reads as compressed, and that is not in `COMPRESSED`, gets the + type of its signature, or `application/octet-stream`. It does not get a text type. +3. If not, the `mime.types` of the image gives the type. But the name can give a type that + a browser shows (text, XML, JSON, JavaScript, PDF), and the first KiB can show a + different type. Then the bytes set the type. This occurs if the first KiB has a + compression signature or a NUL, or if a PDF has no `%PDF-`. +4. A name with no type gets its type from the bytes. A known signature gives its type. If + the bytes have no NUL, the type is `text/plain`. If they have a NUL, the type is + `application/octet-stream`. + +A control byte is not a NUL. Thus, a `.sty` file of CTAN with a `^Z` at its end stays +`text/x-tex`. Three checks make sure that `label` obeys these rules: + +- `offline` gives a set of names and bytes to `label`, and stops on each change. If an + update of the image changes `mime.types`, this check shows it. +- `smoke` reads one key of each type back from the domain, and compares the served type + with the label. +- The `.tar.gz` check makes sure that tarballs have no `Content-Encoding`. #### The vars -A mirror sets `SOURCE`, `BUCKET` and `HOST` in its root vars, always. Everything else -has an inline default, and a mirror sets only what differs: +A mirror always sets `SOURCE`, `BUCKET` and `HOST` in its root vars. All other vars have an +inline default, and a mirror sets only the vars that are different: + +| Var | Default | Function | +|---|---|---| +| `CEILING_GB` | 0, no ceiling | `split` rejects a tree that is larger than this number of decimal GB | +| `BATCH_GB` | 4 | Decimal GB in each batch. A larger file is a batch with no other file | +| `MAX_BATCHES` | 4 | The batches in each run. The next run, which the chain starts, does the other batches | +| `LIST_FLOOR` | 0, no guard | A listing that does not have more lines than this number is truncated, and the engine does not use it to delete keys. Set it to approximately 90% of the usual number of lines | +| `RECONCILE`, `RECONCILE_HOURS` | `auto`, 24 | The vars of the toolbox: refer to [When a reconcile runs](#when-a-reconcile-runs) | +| `RETRY_BASE` | 15 | Seconds. Before retry `i`, `retry` waits `RETRY_BASE * 2^i` seconds, plus jitter | +| `TL`, `TL_KEY` | Empty | A signed TeX Live subtree, and the fingerprint of the key that signs it. If they are empty, the engine does no signature checks | +| `FILTER` | Empty | rsync filter arguments that make the listing smaller, for a mirror of a subtree | +| `OWN` | Empty | Keys at the root of the bucket that the mirror writes, not upstream. A space is between two keys. `reconcile` does not delete them | +| `INDEX` | Empty | The key suffix of the directory pages. If it has a value, `index` writes the pages. `reconcile` does not delete the pages, or a key with the name of a directory of the state | +| `PAGE_FOOT` | Empty | The HTML at the end of each directory page. `%s` is the encoded path of the directory, and `%%` gives one `%` | +| `CANARY` | Empty | A file that `smoke` reads from the domain as a Perl client, and compares with the bucket, byte for byte. All files can be the canary. A file with `http://` links (not `https://`) or `mailto:` addresses also finds the HTML rewriters of the zone. The path must have no character that a URL encodes | +| `FRESH_KEY` | Empty | A file in which upstream records its clock, as Unix time or as `YYYY-MM-DD-HH-MM` (the format of CTAN). If it has a value, `fresh` stops the run when the clock is more than `FRESH_HOURS` before the time of the run | +| `FRESH_HOURS` | 24 | Hours. This value is less than 28 hours: at 28 hours, the mirror checks of GNU and CTAN show a problem | + +A var in the root `vars:` of a mirror has the same value for all runs. In an included verb, +the run uses a root value, not a `KEY=value` on the command line. A var that the mirror +does not set keeps its default, and each run can set it: +`task sync -- MAX_BATCHES=8 RECONCILE=true`. + +Do not set a root var to the default of the engine. If you do, the command line cannot +change that var. + +#### When a run fails + +The job log shows the step that stopped the run. The name of that step is the verb. The +table in this section gives each verb of the engine that can stop a run, with the cause and +the repair. The README of a mirror gives only the values of that mirror: its ceiling, its +upstream, and the steps that only that mirror has. + +To find the step, and the number of files that the mirror does not have, use these +commands: + +```sh +gh run list --workflow sync.yml --limit 5 +gh run view --log-failed | tail -50 +curl -sI https:/// | grep -i -E 'last-modified|cf-cache-status' +aws s3 cp s3:///.state/applied.txt.xz - | xz -d | wc -l +``` + +Compare the number of lines in the state with the number of lines in the listing. The +difference shows how many files the mirror does not have. + +You can always start the run again, unless a row of the table tells you differently. A +second run writes the same bytes again, or it writes no bytes. To start it, use +`gh workflow run sync.yml` or `task sync`. -| Var | Default | Meaning | +The state has the batches until the last checkpoint, and the second run calculates the +remaining delta again. While the engine fills an empty bucket, this second run continues +the work. On a laptop, +`task sync` uses `op run`. Thus, sign in to the 1Password CLI first. + +| Verb | Cause | Repair | |---|---|---| -| `CEILING_GB` | 0, no ceiling | `split` refuses a tree larger than this many decimal GB | -| `BATCH_GB` | 4 | Decimal GB per batch; a larger file is a batch by itself | -| `MAX_BATCHES` | 4 | Batches per run; the rest chain the next run | -| `LIST_FLOOR` | 0, no guard | A listing under this many lines is a truncated one, never a deletion list. Set it to about 90% of the usual line count | -| `RECONCILE`, `RECONCILE_HOURS` | `auto`, 24 | The toolbox's: see [Reconcile, by age](#reconcile-by-age) | -| `RETRY_BASE` | 15 | Seconds; the retry sleeps are `RETRY_BASE * 2^i` plus jitter | -| `TL`, `TL_KEY` | empty | A signed TeX Live subtree and the fingerprint that signs it; empty, no signature checks | -| `FILTER` | empty | rsync filter arguments that narrow the listing, for a mirror of a subtree | -| `OWN` | empty | Bucket-root keys the mirror owns, space separated; `reconcile` never deletes them | -| `INDEX` | empty | The key suffix of the directory pages; set, `index` draws them and `reconcile` spares them and every bare directory of the state | -| `PAGE_FOOT` | empty | The HTML every directory page closes on; `%s` is the directory's encoded path, `%%` a literal percent | -| `CANARY` | empty | A path carrying plain `http://` links or `mailto:` addresses, no character of which a URL must encode; set, `smoke` fails if the domain serves it differently from the bucket or refuses a Perl client | -| `FRESH_KEY` | empty | A file in which upstream records its own clock, as Unix time or CTAN's `YYYY-MM-DD-HH-MM`; set, `fresh` fails the run when it is older than `FRESH_HOURS` | -| `FRESH_HOURS` | 24 | Hours; under the 28 at which GNU's and CTAN's monitors call a mirror old | - -A var the mirror puts in its root `vars:` is fixed for every run: inside an included -verb a root value shadows a `KEY=value` from the command line. One the mirror leaves to -its default is the run's to set, `task sync -- MAX_BATCHES=8 RECONCILE=true`. So a root -var that restates a default is worse than none: it silences the command line. +| `image` | The toolbox could not download the image from GHCR. The mirror has no problem: the run uploaded no file, and the state did not change | Start the run again | +| `list` | The listing does not have more lines than `LIST_FLOOR`. The engine does not use a truncated listing to delete keys | Start the run again. If the problem continues, examine upstream. If upstream became more than 10% smaller, decrease `LIST_FLOOR` | +| `retry`, in `list`, `prepare` or `fetch` | rsync gave a transport exit five times. If upstream moved, each run uses approximately ten minutes for the retries, and then it stops | Examine `SOURCE`. If upstream moved, change `SOURCE` in the Taskfile | +| `split`: the ceiling | The tree is larger than `CEILING_GB`. The mirror gets no new files until you increase `CEILING_GB`. A larger ceiling also increases the cost | Increase `CEILING_GB` | +| `split`: the disk | One file of the delta is too large for the disk of the runner, with 1 GiB free for the other data. The log shows `no room for `. `CEILING_GB` has no effect on this check | Remove the file from the listing with `FILTER`, or give the run a larger disk | +| `prepare`, `verify` | A signature or a checksum did not agree. The engine uploaded no file from that batch, and clients continue to get the last good copy | If TeX Live changed its key, compare `TL_KEY` with https://www.tug.org/texlive/verify.html. Do not use a different source for the key | +| `smoke`: the canary | The domain served bytes that are not the bytes in the bucket, or it rejected a Perl client. A zone rule is missing, or the scope of a zone rule became wider. The bucket is correct | Examine the Configuration Rule of the zone. Refer to [The parts of a bucket](#the-parts-of-a-bucket) | +| `smoke`: the Content-Encoding | A `.tar.gz` had a `Content-Encoding` when `smoke` read it. Clients then decompress it while they download it | Examine `label` and the AWS CLI version in `toolchain.lock.toml`. Refer to [Content types](#content-types) | +| `smoke`: the type or the length | A key of the sample had a Content-Type that is not its label, or a length that is not its length in the listing. The CLI, R2 or the zone changed the object | Examine `label` and the AWS CLI version in `toolchain.lock.toml`. Then examine the rules of the zone | +| `fresh`: the clock stopped | The clock in the `FRESH_KEY` file did not change for more than `FRESH_HOURS`. Upstream stopped, or the mirror that `SOURCE` gives stopped. A secondary mirror can stop its sync, and rsync can continue to connect to it. The run copied all the bytes that it got | Examine upstream. If `SOURCE` is a secondary mirror, set `SOURCE` to a different secondary mirror. If upstream stopped, no change in the mirror is necessary | + +These procedures repair other problems: + +- **The state is not correct.** Use this procedure if the state file is damaged, or if you + do not know if it is correct. It makes the state again from the bucket: + + ```sh + gh workflow run sync.yml -f vars='RECONCILE=true' # or: task sync -- RECONCILE=true + ``` + + The run makes a listing of the bucket, and compares it with the listing of upstream. It + accepts each object with the length that upstream gives as correct. Thus, if a file + changed but kept its length while the state was not correct, the run does not find the + change. It finds the change when upstream changes the file again. `rebuild` uploads only + the new state, and no file of upstream. This procedure is not necessary for a missing + state file: the next run makes it. +- **Fill an empty bucket.** No flag is necessary. An empty bucket gives an empty state, and + the delta is the full tree. Each run does `MAX_BATCHES` batches (4), and then it starts + the next run with the chain. For more batches in each run, use + `gh workflow run sync.yml -f vars='MAX_BATCHES=8'`. Before the engine fills an empty + bucket, pause the healthcheck (refer to [Monitoring](#monitoring)). +- **Delete one key.** Use `aws s3 rm s3:///`. While the Cache Rule of the zone + bypasses the cache, the edge contains no copy to remove. If upstream has the key, the next + reconcile finds that the key is missing, and the subsequent run fetches it again. Do not + edit the state manually to get the key back before the next reconcile. An error in the + state can make the engine delete or fetch the incorrect files. +- **Write all directory pages again.** Use `aws s3 rm s3:///.state/indexed.txt.xz`. + The next run writes the page of each directory at its two keys. This procedure is safe: + it does not change the files in the bucket. +- **A directory URL gives 404.** The page is at its key, with or without the zone rule. Use + `curl -sI https:////`. If this command gives 200 and the directory URL + does not, the Transform Rule is missing, or its scope is not correct. +- **Change a secret.** Edit the `r2` section of the item of the mirror in the 1Password app. + Then use `gh workflow run sync.yml && gh run watch`. After a run with no failure, delete + the previous R2 token in the Cloudflare dashboard. This procedure is safe: the first call + with the new credentials only reads. +- **Abort a multipart upload that did not complete.** Find the upload. Then abort it: + + ```sh + aws s3api list-multipart-uploads --bucket --query 'Uploads[].[Key,UploadId,Initiated]' --output text + aws s3api abort-multipart-upload --bucket --key --upload-id + ``` + + This procedure is safe, and it has no cost. If you do not do it, R2 aborts the upload + after 7 days. ### The proton engine -`engines/proton.yml`: a staging tree into one Proton Drive folder, through the official -`proton-drive` CLI, for a mirror whose upstream fits in a run. The mirror fills -`staging/` and the engine does the rest. It keeps nothing of the mirror's in the bucket -but the CLI session, because the CLI skips a file whose content Proton already holds and -`-f create-new-revision` makes a revision of one that changed: Proton's version history -is the history of the mirror. +`engines/proton.yml` copies a staging tree into one Proton Drive folder, with the +`proton-drive` CLI of Proton. It is for a mirror that can copy its full upstream in one run. +The mirror fills `staging/`, and the engine does all the other steps. + +The bucket contains only the CLI session of the mirror. The engine keeps no other data +there, because of these two facts: + +- The CLI does not upload a file if Proton has its content. +- `-f create-new-revision` makes a new revision of a file that changed. + +Thus, the version history of Proton is the history of the mirror. ```mermaid flowchart LR @@ -455,37 +686,47 @@ flowchart LR class stage,prune,re,rm hook ``` -| Verb | Does | +| Verb | Function | |---|---| -| `pipeline`, `plan-pipeline` | The verbs above in order; the read-only half runs through `stage` and prints what an upload would carry | -| `session` | Pull `.state/session.tar.age` from the bucket, decrypt it with `MIRROR_AGE_IDENTITY`, extract the two session files to `.run/session` | -| `destination` | List the parent of `MIRROR_PROTON_DESTINATION` and refuse the run unless exactly one folder of that name exists and its UID is `MIRROR_PROTON_DESTINATION_UID` | -| `stage` | Hook. Every mirror defines it: fill `staging/` with what Proton should hold | -| `upload` | One `filesystem upload -f create-new-revision -d merge -t --json` of `staging/*` into the destination; the summary to `.run/upload.json` | -| `confirm` | Transferred plus skipped plus failed must equal the staged files plus folders, with no failure; the verdict to `.run/confirm.txt` | -| `prune` | Hook. Nothing here; a mirror that trashes what its upstream dropped defines it, from `list-folder` and `trash` | -| `pd` | Every CLI call: stderr to `.run/pd.err`, then the session sealed back to the bucket when its token rotated, whatever the exit | -| `list-folder`, `trash` | A folder's JSON listing to `OUT`; the nodes at `PATHS` to Proton's trash | -| `session-seal -- ` | Host side: a laptop login's two files, encrypted into the bucket | -| `empty-trash` | Host side, asks first: everything in Proton's trash, permanently | -| `report-engine` | Hook. The engine's rows of the run summary | - -The engine reads no root var. Its inputs are the environment, by the names every Proton -mirror's `op.env` carries: - -| Name | What it is | +| `pipeline`, `plan-pipeline` | The verbs in the diagram, in sequence. The read-only half runs to `stage`, and then prints the number of files and MB that the engine can upload | +| `session` | Downloads `.state/session.tar.age` from the bucket, decrypts it with `MIRROR_AGE_IDENTITY`, and puts the two session files from the tar into `.run/session` | +| `destination` | Makes a listing of the parent of `MIRROR_PROTON_DESTINATION`. It stops the run unless the parent has one folder of that name, and only one, with the UID `MIRROR_PROTON_DESTINATION_UID` | +| `stage` | A hook that each mirror writes. It fills `staging/` with the files that Proton must contain | +| `upload` | Runs one `filesystem upload -f create-new-revision -d merge -t --json` of `staging/*` into the destination. It writes the summary to `.run/upload.json` | +| `confirm` | In the summary, `transferredItems` and `skippedItems` together must be equal to the files and folders in staging, and `failedItems` must be 0. It writes the result to `.run/confirm.txt` | +| `prune` | A hook, empty here. A mirror that moves to the trash the items that its upstream deleted writes it, with `list-folder` and `trash` | +| `pd` | Runs each CLI call, and writes the stderr of the call to `.run/pd.err`. After each call, with all exit codes, it examines the session. If the token changed, it seals the session back into the bucket | +| `list-folder`, `trash` | `list-folder` writes the JSON listing of a folder to `OUT`. `trash` moves the nodes at `PATHS` to the trash of Proton | +| `session-seal -- ` | On the host: it encrypts the two files of a laptop login into the bucket | +| `empty-trash` | On the host: it deletes all the items in the trash of Proton, permanently, but only after you tell it to continue | +| `report-engine` | A hook: the rows of the engine in the run summary | + +The verbs in the table also run these verbs: `session-push`, `seal`, `age`, `pull`, `push` +and `empty-trash-pipeline`. `pull` and `push` move a key between the bucket and `.run`, +with no compression. A verb of a mirror can also run them. + +The stderr of `list-folder` goes to the log of the run, not to `.run/pd.err`. Its command +has `|| echo "[]" > OUT` at its end. Thus, the stderr redirect of `pd` goes to the `echo`, +not to the CLI. + +The engine reads no root var. It reads only the environment, with the names that the +`op.env` of each Proton mirror has: + +| Name | Value | |---|---| -| `MIRROR_PROTON_DESTINATION` | The CLI path of the destination folder, `/my-files/GitHub` say | -| `MIRROR_PROTON_DESTINATION_UID` | Its UID, compared on every run before any write | -| `MIRROR_R2_BUCKET` | The bucket that holds the session | -| `MIRROR_AGE_IDENTITY` | The `AGE-SECRET-KEY-...` line the session is encrypted to | -| `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_ENDPOINT_URL_S3` | What boto3 reads; note `_S3` on the endpoint | +| `MIRROR_PROTON_DESTINATION` | The CLI path of the destination folder, for example `/my-files/GitHub` | +| `MIRROR_PROTON_DESTINATION_UID` | The UID of that folder. Each run compares it before it writes the first file | +| `MIRROR_R2_BUCKET` | The bucket that contains the session | +| `MIRROR_AGE_IDENTITY` | The `AGE-SECRET-KEY-...` line that encrypts and decrypts the session | +| `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_ENDPOINT_URL_S3` | The credentials and the endpoint for `s3`. The proton mirrors use the `_S3` name for the endpoint. The AWS CLI and boto3 read the two names, `AWS_ENDPOINT_URL` and `AWS_ENDPOINT_URL_S3` | #### The session -The CLI can only be seeded by a browser sign-in, and fresh sign-ins from datacenter -addresses are blocked, so the session is made once on a laptop and carried to every run -encrypted. Its refresh token rotates on use, so every run writes it back. +Only a sign-in in a browser can start a session of the CLI. Proton does not accept a new +sign-in from a datacenter address. Thus, you make the session one time on a laptop, and +each run gets it from the bucket, encrypted. The CLI changes the refresh token when it uses +the session. Thus, after each CLI call, `pd` examines the session. If the token changed, +`pd` seals the session back into the bucket. ```mermaid flowchart LR @@ -502,40 +743,57 @@ flowchart LR r4 --> key ``` -To make one: - -```sh -export PROTON_DRIVE_CACHE_DIR=.run/pd PROTON_DRIVE_CREDENTIALS_STORE=unsafe_file -proton-drive auth login # a browser opens; .run/pd holds two JSON files after -proton-drive filesystem create-folder /my-files # /my-files is the drive's own root, not a folder you make -proton-drive filesystem list -j /my-files # the entry named : its uid is the destination UID -task session-seal -- .run/pd # once the bucket and the vault exist -``` - -Install the CLI at the version `toolchain.lock.toml` pins: the session file format is -tied to the version, and the Linux binary in the image must read what the laptop wrote. -`.run/pd` must be inside the repository, because only the repository is mounted into the -image; `.run/` is ignored by git and deleted by `task clean`. - -Three rules. **Two mirrors never share a session**: its refresh token rotates on every -call, and the loser of a race needs a fresh login. **Never run `sync` or `plan` from a -laptop while an Actions run may be in progress**, for the same reason. **The session gets -no history in the bucket**: a stale copy holds a rotated-out token and cannot be restored. - -### A pipeline of your own - -A mirror whose logic is a program of its own includes the toolbox alone and defines -`pipeline` and `plan-pipeline` from the toolbox's `clock`, `report` and `ping` and its own -steps. dropbox is that shape: each step is one `python -m migrator `, the -Taskfile owns the order, the Python owns every decision but when to reconcile, which is -the toolbox's `due`, and the `proton` image supplies the interpreter, boto3 and the CLI. -Such a mirror fills `report-mirror` with its own report and nothing else of the -toolbox's changes. +To make a session, do these steps on a laptop: + +1. On the Settings page of your Proton account, set telemetry to OFF. Do this for each + Proton account. +2. Run these commands: + + ```sh + export PROTON_DRIVE_CACHE_DIR=.run/pd PROTON_DRIVE_CREDENTIALS_STORE=unsafe_file + proton-drive auth login # a browser opens; .run/pd holds two JSON files after + proton-drive filesystem create-folder /my-files # /my-files is the drive's own root, not a folder you make + proton-drive filesystem list -j /my-files # the entry named : its uid is the destination UID + task session-seal -- .run/pd # once the bucket and the vault exist + ``` + +Install the CLI at the version that `toolchain.lock.toml` pins. The format of the session +file changes with the version, and the Linux binary in the image must read the files that +the laptop wrote. Put `.run/pd` in the repository, because the container gets only the +repository. git ignores `.run/`, and `task clean` deletes it. + +Obey these three rules: + +- Do not let two mirrors use one session. The CLI changes the refresh token at each call. + Then Proton rejects the token that the other mirror has, and a new login is necessary. +- Do not run `task sync`, `task plan` or `task empty-trash` from a laptop while an Actions + run can be in progress. The three commands use the one session. If the laptop and the run + use it at the same time, a new login can be necessary. +- Do not keep a history of the session in the bucket. A previous copy contains a token that + the CLI changed, and the CLI cannot use that copy. + +### A pipeline of a mirror + +A mirror that uses a program for its work writes `pipeline` and `plan-pipeline` in its +Taskfile. Their steps are the `clock`, `report` and `ping` of the toolbox, and the steps of +the program. The mirror puts these two verbs in `excludes:` in the engine entry of +`includes:`. dropbox has this structure: + +- It includes the proton engine, which supplies `session`, `session-seal`, `empty-trash`, + `pull` and `push`. +- Each step is one `python -m migrator `. +- The Taskfile sets the sequence of the steps. +- The Python makes each decision, but the `due` of the toolbox finds if a run does a + reconcile. +- The `proton` image supplies the interpreter, boto3 and the CLI. + +A mirror of this structure fills `report-mirror` with its report. It changes no other part +of the toolbox. ## Secrets -Every credential and every account identifier lives in one vault and reaches a run by -name. A repository holds `op://` references only. +All credentials and all account identifiers are in one vault, and a run gets each of them +with its name. A repository contains only `op://` references. ```mermaid flowchart LR @@ -548,235 +806,330 @@ flowchart LR ### The vault -One vault. One item per mirror, named for its repository. One section per service in -the item (`r2`, `healthcheck`, `proton`, `age`, `github`, `dropbox`), with the field names -the mirror's README lists. Neither the vault's name nor its UUID is a secret: without the -service account's token it opens nothing. +There is one vault, with one item for each mirror. Each item has the name of its +repository. An item has one section for each system that the mirror uses (`r2`, +`healthcheck`, `proton`, `age`, `github`, `dropbox`). The README of the mirror gives the +names of the fields. The name and the UUID of the vault are not secrets: without the token +of the service account, they give access to no item. -The references address the vault by UUID, `op:////
/`, so a -renamed vault cannot break a run and a name with a slash in it cannot be written: +The references give the UUID of the vault: `op:////
/`. Thus, if +the vault gets a new name, the runs continue to operate. Also, a reference cannot contain a +vault name that has a slash. To find the UUID, use this command: ```sh op vault get --format json | jq -r .id ``` -One service account that can read that vault and nothing else. Its token is the secret -`OP_SERVICE_ACCOUNT_TOKEN`: on an organization, one organization secret that every -repository inherits; on a personal account, one repository secret per mirror. Only a -dispatched sync run holds it; a pull-request check gets no secret at all. +One service account can read that vault, and no other vault. Its token is the secret +`OP_SERVICE_ACCOUNT_TOKEN`: + +- On an organization, the token is one organization secret that all repositories inherit. +- On a personal account, the token is one repository secret for each mirror. + +Only a dispatched sync run gets the token. A check of a pull request gets no secret. ### On the host -`op run --env-file=op.env` resolves every reference and exports the values into the -environment of the one command it wraps, `task run -- ...`, masking each value in that -command's output. On a laptop `op` uses the signed-in desktop app; in Actions it uses the -service account token the reusable workflow put in its environment. The vault is read -once per run. +`op run --env-file=op.env` gets the value of each reference. It exports the values into the +environment of one command, `task run -- ...`. The output of that command does not show the +values, because `op run` removes them. On a laptop, `op` uses the desktop app, and you must +sign in to the app. In Actions, `op` uses the token of the service account, which the +reusable workflow puts in its environment. A run reads the vault one time. ### Into the container -`run` passes `-e NAME` for each name in `op.env` and in the mirror's `PASS`, plus -`HEALTHCHECK_URL`, `GITHUB_STEP_SUMMARY` and `GITHUB_RUN_ID` always. The container engine -copies each named variable from the host's environment; no value is on the command line. -A host variable the pipeline reads and `op.env` does not name arrives empty. +For each name in `op.env` and in the `PASS` of the mirror, `run` gives `-e NAME` to the +container runtime. It also always gives `HEALTHCHECK_URL`, `GITHUB_STEP_SUMMARY` and +`GITHUB_RUN_ID`. The container runtime copies the value of each variable that `run` gives +from the environment of the host. Thus, no value is on the command line. + +If the pipeline reads a host variable that `op.env` and `PASS` do not give, the variable is +empty in the container. `GITHUB_STEP_SUMMARY` is a path on the runner. Thus, `run` gives +the variable, and it also puts the file into the container at that same path. ### The names -| Name | Read by | Set by | +| Name | The tool that reads it | Its source | |---|---|---| -| `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` | the AWS CLI and boto3 | the `r2` section | -| `AWS_ENDPOINT_URL` | the AWS CLI, in the `rsync` image | the `r2` section's `endpoint` | -| `AWS_ENDPOINT_URL_S3` | boto3, in the `proton` image | the same field, under boto3's name | -| `AWS_REGION` | both | a literal `auto` in the image; R2 has one region, and masking the word `auto` would corrupt output | -| `MIRROR_R2_BUCKET` | the proton engine's `s3` | the `r2` section's `bucket` | -| `MIRROR_AGE_IDENTITY` | the proton engine's `age` | the `age` section | -| `MIRROR_PROTON_DESTINATION`, `_UID` | the proton engine's `destination` | the `proton` section | -| `HEALTHCHECK_URL` | `ping`, `ping-fail` | the `healthcheck` section; optional | -| `MIRROR_*` of a mirror's own | that mirror | a section of its own: `github`, `dropbox` | +| `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` | The AWS CLI and boto3 | The `r2` section | +| `AWS_ENDPOINT_URL` | The AWS CLI, in the `rsync` image | The `endpoint` field of the `r2` section | +| `AWS_ENDPOINT_URL_S3` | boto3, in the `proton` image. The AWS CLI also reads it | The same field. The proton mirrors use this name | +| `AWS_REGION` | The two tools | The value `auto`, in the image. R2 has one region. If `op run` removes the value `auto` from the output, the output is incorrect | +| `MIRROR_R2_BUCKET` | `s3`, in the proton engine | The `bucket` field of the `r2` section | +| `MIRROR_AGE_IDENTITY` | `age`, in the proton engine | The `age` section | +| `MIRROR_PROTON_DESTINATION`, `_UID` | `destination`, in the proton engine | The `proton` section | +| `HEALTHCHECK_URL` | `ping`, `ping-fail` | The `healthcheck` section. It is optional | +| `MIRROR_*` names that one mirror uses | That mirror | A section for that mirror: `github`, `dropbox` | ### Without a vault -A mirror with no `op.env` runs on its repository secrets: the reusable workflow exports -every inherited secret into the sync step's environment by name, the ones the mirror -lists in `PASS` cross into the container, and the 1Password CLI is not installed at all. -On a laptop, whatever is exported. This is the fallback, not the pattern: it works, and -it puts the values in a second place. +A mirror with no `op.env` runs on the secrets of its caller: + +- The reusable workflow exports each inherited secret into the environment of the sync + step, with its name. +- The secrets that the mirror gives in `PASS` go into the container. +- The workflow does not install the 1Password CLI. + +On a laptop, the run uses the exported variables. This method is not the usual method. It +operates, but it puts the values in a second location. ## Storage -Every mirror has one S3-compatible bucket. What it holds depends on the engine. +Each mirror has one S3-compatible bucket. The contents of the bucket are different for each +engine. -| Engine | Key | What | +| Engine | Key | Contents | |---|---|---| -| rsync | every upstream path, at the root | The mirror. A public domain serves the bucket | -| rsync | `.state/applied.txt.xz` | The state: what the bucket holds, at upstream's size and mtime | -| rsync, with `INDEX` | `.state/indexed.txt.xz`, `/`, `` | What the directory pages last showed, and the pages under both keys | -| rsync, a mirror's own | `index.html` | tlnet's landing page, spared by `OWN` | -| proton | `.state/session.tar.age` | The CLI session, encrypted. The only key | -| a pipeline of its own | `.state/state.sqlite.xz.age`, `.state/history/-