[Deicyde]

Problem

Backup D1 uploads a plaintext export of the production D1 database as a workflow artifact. The repository is public, so any signed-in GitHub account can download workflow artifacts (GitHub requires only read access), and anyone can list them without signing in.

• .github/workflows/backup-d1.yml#L68-L80: wrangler d1 export wikilean --remote with no table filter, then upload-artifact of the .sql file with retention-days: 30. The job runs daily.
• The export contains every table, including users.email (0002_users.sql#L5-L14), sessions.token / ip_address / user_agent and the OAuth access_token / refresh_token / id_token in accounts (0003_auth.sql#L4-L27), and flags.ip_hash (0005_events_flags_counts.sql#L35-L45). That contradicts the data notice in CONTRIBUTING.md#L151 ("never include emails or IP addresses").
• The run log leaks the dump too. wrangler d1 export --remote (wrangler 4.120.0, exportRemotely) logs You can also download your export from the following URL manually. This link will be valid for one hour: <presigned URL> at its default level. That URL serves the plaintext dump with no authentication, and Actions run logs on a public repo are public, so every run published an auth-free download link for its first hour. Each Export D1 (wikilean) step log on main shows the line, and those links have since expired.
• The second artifact (backup-d1.yml#L87-L109) republishes the output of GET /api/research/export.jsonl, an endpoint gated behind the pipeline bearer token or an admin session. Publishing it as a public artifact undoes that gate.

Change

.github/workflows/backup-d1.yml:

• New first step, Install age: installs age v1.3.2 from the upstream GitHub release. sha256sum --check --strict checks the tarball against a pinned SHA-256 before anything is extracted. Only the age binary is installed (none of the bundled plugins).
• New second step, Check backup recipients: reads the repo variable BACKUP_AGE_RECIPIENTS through vars, not secrets, because it holds public keys only. If the variable is empty, or age rejects it as a recipient list, the job fails with an ::error:: annotation before checkout, npm ci or any export. age is the validator, and it rejects malformed or typo'd keys (bech32 checksum), a pasted AGE-SECRET-KEY-…, a list where any line is invalid, and plugin recipients whose plugin is not installed. Any output from age also fails the check: age only warns about SSH key types it can't use (ECDSA, security keys, short RSA) and skips them, which would silently drop a spare key.
• Export and encrypt: wrangler d1 export now writes into $RUNNER_TEMP, outside the workspace, under umask 077.
  • wrangler runs with WRANGLER_LOG=warn and its stdout is discarded, so the presigned URL never reaches the log. Its stderr (warnings and errors) passes through a sed filter that replaces any URL with [URL redacted], so a future wrangler that logs the URL as a warning or in an error still can't publish it. pipefail keeps wrangler's exit status. WRANGLER_WRITE_LOGS=false also skips wrangler's on-disk debug log, which records every message.
  • An empty export now fails the step. Otherwise it would upload as a valid-looking .age file.
  • age -R encrypts the dump into the workspace and the plaintext is deleted immediately. An EXIT trap also deletes it when the export or the encryption fails.
  • The step then checks that the output begins with the age-encryption.org/v1 header.
• Upload: only wikilean-<ts>.sql.age is uploaded. Artifact names, retention and ordering are unchanged.
• Research export: encrypted, not dropped. The ROADMAP (/stats + research export + research plan, P2a) lists it as the nightly research-data deliverable: the pseudonymized annotation_events stream, ready for analysis. The encrypted full dump also holds every annotation_events row, but raw and keyed by user id, so the export is not redundant with it. Leaving it public would contradict the bearer gate. The curl response is piped straight into age (curl … | age -R … -o research-export-<ts>.jsonl.age), so this plaintext never touches disk. pipefail means a failed fetch fails the step, and the upload is skipped as before (continue-on-error is unchanged).
• Actions are unchanged and still SHA-pinned. Permissions stay contents: read, and no new action or secret is added.

New docs/BACKUP.md:

• generating the keypair with age-keygen (the private key stays offline; an optional spare key)
• gh variable set BACKUP_AGE_RECIPIENTS --repo Deicyde/WikiLean
• a one-time cutover from the plaintext era:
  • disable the workflow
  • never re-run a pre-encryption run (a re-run replays the old workflow file) and dispatch only from main; delete the old runs
  • delete the pre-encryption artifacts
  • treat their contents as exposed (revoke OAuth user tokens, rotate BETTER_AUTH_SECRET, decide on notification)
• re-enabling with gh workflow enable backup-d1.yml, plus an end-to-end check
• download → decrypt → restore into a local D1 first (wrangler d1 execute --local --persist-to)
• D1 Time Travel as the short-horizon production path (with the undo bookmark)
• importing a dump into a fresh D1, listing every name and id reference that must move with it
• the render-cache caveat after any restore (it rewinds articles.version)
• key rotation, and what to do if the private key leaks
• how to bump the pinned age

Local script: wiki/scripts/backup-d1.sh is left unchanged in this PR. Under site/ops/brain_release_authority.py, any change under wiki/scripts/ is release-affecting, and this fix should not wait on a Brain release. docs/BACKUP.md states that the script still writes plaintext to the gitignored wiki/backups/ and shows how to encrypt its output by hand. A follow-up PR makes the script encrypt the same way.

Release check: brain_release_authority.blocking_changes() over git diff --name-only main...HEAD returns nothing, so this PR is release-neutral (.github/ and docs/ only).

How tested

The workflow only runs on schedule and workflow_dispatch, so PR CI never executes it. Everything below ran locally on macOS.

• age release verification: the linux-amd64 tarball's SHA-256 equals the release asset digest that GitHub reports (cbe24006…ac10). The tarball also verifies against the release's Sigsum proof (sigsum-verify v0.13.1, policy sigsum-generic-2025-1, the key from age's SIGSUM.md), and a copy with one byte appended fails the proof.
• Workflow simulation: a harness parses backup-d1.yml and runs the real run: blocks in job order, under bash -e and GNU sha256sum. wrangler and curl are stubbed, the fake dump and research rows carry PII markers, and the keys are throwaway. All checks pass:
  • Round trip: the uploaded .sql.age and .jsonl.age decrypt byte-for-byte to the originals, either of two recipients can decrypt, and a non-recipient key cannot.
  • Fail closed before export: with the variable unset, empty, whitespace-only, comment-only, TODO, a key with a one-character typo, a valid key plus a garbage line, a pasted AGE-SECRET-KEY-…, or an age1yubikey1… recipient, the job stops at Check backup recipients. npm ci, wrangler and the research fetch never run, nothing is uploaded, and an ::error:: annotation is emitted. age's error text never echoes the rejected line.
  • Tampered tarball: with one byte appended, the job fails at Install age, with no export and no upload.
  • Plaintext never left behind: when wrangler fails midway, or age fails after the export, no file containing the PII markers remains in the workspace or $RUNNER_TEMP. No marker appears in any step log on any path.
  • Signed URL kept out of the log: the wrangler stub prints a presigned-URL line at log level, as 4.120.0 does, and that line never reaches the step log.
  • Empty export: the job fails at the export step, and nothing is uploaded.
  • Research export: when the fetch fails (curl exit 22), the backup is still uploaded and the research upload is skipped. With no PIPELINE_TOKEN, the endpoint is never called.
  • Old workflow: the same harness run against main's workflow fails: it uploads wikilean-<ts>.sql with no age header.
• The real wrangler 4.120.0 at WRANGLER_LOG=warn: a d1 export --local prints nothing on stdout (its default level prints 🌀 Exporting SQL… / Done!), the export bytes are unchanged, and an error still prints on stderr.
• Restore drill with the pinned wrangler 4.120.0: apply all 12 migrations to a local D1 and seed users / sessions / accounts rows, then wrangler d1 export --local, age encrypt, delete the plaintext, decrypt (bytes identical), and wrangler d1 execute wikilean --local --persist-to <fresh dir> --file=…. All 16 tables have matching row counts. The wrangler d1 time-travel and gh run download / gh variable set flags in the doc were checked against --help.
• Second review pass (410b340): the edited Check backup recipients and Export D1 blocks were extracted from the YAML and run under bash -eo pipefail with age v1.3.2 and a wrangler stub.
  • Recipient check: 24/24 cases behave as expected. Valid age1…, age1pq1…, ssh-ed25519 and RSA-3072 lists pass. An ECDSA or security-key SSH line, short RSA, malformed keys and a pasted AGE-SECRET-KEY all fail. Before this change, age1… plus an ECDSA line passed with only a warning.
  • Export: with the stub printing the presigned URL on stdout, on stderr as a warning, or on stderr before a non-zero exit, no URL reaches the step output. A non-zero wrangler exit, an empty export or a killed wrangler fails the step, and the plaintext is gone on every path.
  • actionlint and shellcheck were not re-run for this pass.
• Lint: actionlint v1.7.12 with shellcheck v0.11.0 reports nothing for backup-d1.yml.

Risks

• No Ubuntu run yet. The first real run on ubuntu-latest happens after you re-enable the workflow. The install step relies on GNU tar's --strip-components=1 age/age member selection, which I checked with bsdtar, not GNU tar. If any step before the export misbehaves, the job fails before exporting, so the failure mode is "no backup that day", never a plaintext upload.
• wrangler logging could change. wrangler 4.120.0 logs the URL through logger.log to stdout, which WRANGLER_LOG=warn and > /dev/null both drop. A future wrangler that printed it at warn or error level would go to stderr, where the URL redactor replaces it. A URL split across lines, or printed without a scheme, would still get through, so recheck the export step's log when bumping wrangler.
• The key is a single point of failure. If the private key is lost, every encrypted backup is unreadable. The doc recommends a second, separately held key listed in the same variable.
• The variable shows in public logs. BACKUP_AGE_RECIPIENTS is printed in the step's env block, which is fine for public keys. A private key pasted there by mistake is rejected but exposed, and the doc says to rotate.
• Plaintext on the runner: the D1 plaintext still exists on the ephemeral runner disk for the seconds between export and encryption, because wrangler d1 export can only write to a file. It sits outside the workspace with mode 0600 and is deleted on every exit path.
• Existing artifacts: this PR changes only future runs and does nothing about artifacts that already exist.

What Jack must do

Now, if not already done:

0. Disable the workflow (gh workflow disable backup-d1.yml --repo Deicyde/WikiLean). Then work through the cutover in docs/BACKUP.md: never re-run a pre-encryption run (re-runs replay the original commit's plaintext workflow) and delete those runs, delete the existing wikilean-d1-* / research-export-* artifacts, and decide on OAuth token revocation, BETTER_AUTH_SECRET rotation and user notification.

Before merging (or before re-enabling):

1. Generate the keypair offline and keep the private key off GitHub:
   age-keygen -o wikilean-backup.key && age-keygen -y wikilean-backup.key > backup-recipients.txt
   (optional: append a second, separately held spare key's age1… line).
2. Set the variable with public keys only:
   gh variable set BACKUP_AGE_RECIPIENTS --repo Deicyde/WikiLean < backup-recipients.txt
3. After merge, re-enable the workflow and run it once:
   gh workflow enable backup-d1.yml --repo Deicyde/WikiLean && gh workflow run backup-d1.yml --repo Deicyde/WikiLean --ref main
   (Dispatch only from main: a dispatch uses the workflow file on the chosen ref, and older branches still carry the plaintext version.)
   Then download that run's wikilean-d1-<ts> artifact and confirm it decrypts (docs/BACKUP.md, "Download and decrypt").

If the variable is missing, the re-enabled workflow fails every day without exporting anything. It never uploads plaintext.

[Deicyde]

Review status: merged in da3ea52d; all required fixes verified

Update 2026-10-14 (re-checked at f32bc5be):

• Landed (verified at f32bc5b, intact on main): in 410b340, wrangler's stderr now passes through a URL redactor while stdout is still discarded and WRANGLER_LOG=warn is kept. pipefail preserves the exit status (.github/workflows/backup-d1.yml:139-148).
• Landed: a new cutover step in docs/BACKUP.md:59-75: never re-run pre-encryption runs, delete them, and dispatch only from main. Manual-run commands now pass --ref main.
• Landed: the PR body's research-export rationale is corrected (ROADMAP P2a deliverable, not the only copy).
• Extra: the recipient check now rejects any key age would warn about and skip. This is reasonable and fails closed.
• Departure: the cutover prescribes deletion rather than offering never-re-run as an alternative, and doesn't suggest saving the run list first. The maintainer can make that call, but save gh run list --json databaseId,createdAt,headSha output before deleting.
• Still open: set the recipients variable, re-enable the workflow, decrypt one run; decide on retention and post-quantum keys; revoke OAuth tokens; decide on notification. actionlint and shellcheck were not re-run on 410b340.

Original review (2026-10-14)

Independent adversarial review of c2404981: a defect reviewer with skeptics, a design reviewer with a devil's advocate, and an arbiter.

Verdict: right approach; the age encryption is implemented correctly for the pinned wrangler 4.120.0. Three small fixes before merge.

Required before merge

• At .github/workflows/backup-d1.yml:133, pipe wrangler's stderr through a URL redactor (keep discarding stdout and WRANGLER_LOG=warn), and reword the comment at :120-123 and docs/BACKUP.md:34-36 to match. The presigned URL stays out of the public log only because 4.120.0 prints it to stdout; the job picks up any wrangler bump on main, and a change there would publish a live link daily while staying green.
• Add a cutover step to docs/BACKUP.md: re-runs replay the original commit's workflow, and dispatches use the chosen ref's file. Re-running one of the 30 plaintext-era runs (re-runnable until about 2026-11-12), or dispatching from a branch with the old file, uploads plaintext and logs a live URL. Manual runs must target main; never re-run the old runs, or delete them first.
• Fix the PR body: the research export is not the only off-Worker copy of annotation_events, since the encrypted full dump holds every row. The real reason to keep it is the ROADMAP P2a nightly deliverable, and this page is the only place that rationale lives.

Decisions for the maintainer

• Research export: keep it encrypted (recommended) or drop its two steps.
• Key type: age-keygen -pq from the start (recommended); the ciphertext is public and can be kept forever.
• Retention: 30 days or 90 (recommended); the last two silent failures took 21 and 26 days to notice.

Merge order: release-neutral; merge alone once fixed, then set BACKUP_AGE_RECIPIENTS, handle old runs, re-enable and decrypt one run on main. Don't re-enable before merging. Rebase #58 afterwards.

Fixes are in progress; this comment will be updated.

Posted by the WikiLean PR swarm (lead); updated 2026-10-14.

[Deicyde]

Status review (2026-10-14): ✅ Ready

The Export step's real run: block was exercised with stubbed wrangler and age on success, wrangler failure, empty export and age failure. In every case no plaintext stays in the workspace or $RUNNER_TEMP, neither the dump nor the presigned URL reaches the step log, and the upload is gated on the step succeeding. Only .github/ and docs/ change, so this is release-neutral.

Before re-enabling the workflow

• The BACKUP_AGE_RECIPIENTS repo variable is not set yet. Without it the re-enabled workflow fails every run before exporting. Set it with public keys only, and keep the workflow disabled until this PR is merged.
• After merging: gh workflow enable backup-d1.yml, then gh workflow run backup-d1.yml. Confirm the artifact decrypts with the offline key and, if you added one, with the spare (this also catches the dropped-spare case below).
• Confirm the one-time cutover checklist in docs/BACKUP.md is complete.

Nits

• The recipient check can silently drop a spare key. If the list holds a valid but unsupported SSH key (ecdsa-*, sk-ssh-ed25519@openssh.com, or RSA under 2048 bits), age prints a warning, skips the line, and the check passes. Every backup is then encrypted to the primary only. Reproduced with age v1.3.2. Suggest failing on any line age warns about, or allowing only age1… and ssh-ed25519 lines.
• The safety properties (no presigned URL in the log, no plaintext upload) aren't checked anywhere in CI. The workflow only runs on schedule or dispatch, so a later edit or a wrangler bump could quietly regress them. A small lint or grep check in CI would catch the edit case; a wrangler bump still needs the manual recheck the PR's Risks section describes.

Order: merge before #58, which contains this branch.

---

_{🤖 Automated status pass by Claude Code over all 18 open PRs (#46–#63), posted for the repo owner. An independent verifier re-checked each finding. All 18 heads were also merged and tested together in a scratch tree; the only test failure that comes from combining them is #47 × #55 (three other pairs had mechanical text conflicts). Reviewed head: c2404981.}

[Deicyde]

Fixes pushed (2026-10-14)

This addresses the PR swarm's three required items and the dropped-spare-key nit from the "Status review (2026-10-14)" comment.

What changed

• Export step: wrangler's stderr now goes through a sed filter that replaces URLs with [URL redacted]. stdout is still discarded and pipefail keeps wrangler's exit status. The step comment and docs/BACKUP.md are updated to match. (swarm item 1, 410b340b)
• Recipient check: the step now fails if age -R prints anything, so an SSH key age would only warn about and skip (ECDSA, security key, short RSA) is rejected. (status-review nit, 410b340b)
• docs/BACKUP.md cutover: a new step 2 says to never re-run pre-encryption runs, to delete them, and to dispatch only from main. Manual-run commands now pass --ref main. (swarm item 2, 410b340b)
• PR body: the research-export rationale is corrected, and How tested has a bullet for this pass. (swarm item 3)

Verification

• Recipient check (run from the YAML with age v1.3.2): 24/24 cases behave as expected. The old check passed age1 + ECDSA with only a warning.
• Export (with a wrangler stub): the stub's URL never reached the step output, whether printed on stdout, in a warning or before a non-zero exit. A failed or killed wrangler, an empty export and an age error all fail the step, and no plaintext is left on any path.
• Still release-neutral. actionlint and shellcheck were not re-run.
• CI on 410b340b: all checks green (worker, python, browser, required).

Not changed / still open

• The swarm's maintainer decisions (research export, -pq keys, retention) are still open; the PR keeps its current choices.
• No CI guard was added (status-review nit). Split or scheme-less URLs aren't redacted, so recheck the log after any wrangler bump (see Risks).

Merge notes

• Merge before Encrypt local D1 backups with age #58. Encrypt local D1 backups with age #58's branch now includes 410b340b through merge 02dcb6f7, with no conflicts.
• After merging, follow "What Jack must do" in the PR body.

---

_{🤖 Posted by Claude Code for the repo owner. Each fix was checked by an independent verifier before it was pushed. Head: 410b340b.}