CI import: design + implementation (TNT-208 + TNT-209)#153
Draft
cevian wants to merge 10 commits into
Draft
Conversation
Design for replacing the per-clone post-commit git hook with a GitHub-Actions-driven import on push to the default branch, covering git history and docs as one system: a scaffolded repo-agnostic workflow, a `me import ci` orchestrator, `me project ci` setup with service-account provisioning, and removal of `me import git-hook`. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KrraND2hq3NVUSS38nhP9d
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new design document (CI_IMPORT_DESIGN.md) describing how to move repo→memory imports from a local git hook to a GitHub Actions workflow that runs me import ci on pushes to the default branch, using service accounts as the team-owned identity.
Changes:
- Introduces a combined TNT-208/TNT-209 CI import architecture (workflow shim →
me import ciorchestrator → existing importers). - Specifies service-account-based identity + provisioning flow (
me project ci) including org-secret and per-repo variants. - Documents operational behaviors (default-branch gating, full-history backfill, docs prune, idempotency, and failure UX).
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| workflow_dispatch: {} # manual backfill / re-run | ||
|
|
||
| concurrency: | ||
| group: me-import |
The header described a retired identity scheme (deterministic UUIDv7 keyed by git:<tree>:<sha>, ON CONFLICT (id) DO NOTHING). The real behavior is (tree, name=sha) idempotency with onConflict: "replace" and random-tailed uuidv7At ids used only for recency ordering. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KrraND2hq3NVUSS38nhP9d
The one command the scaffolded GitHub workflow calls: run the project's configured imports from the repo toplevel — git history (incremental; first run backfills), then docs (default markdown globs, prune on, clean skip when the repo has no matching docs). Phases are fail-fast and each renders its own summary; --dry-run previews locally exactly what CI will do. An optional `import:` block in .me/config.yaml (strict, like the rest of the schema) shapes the run: git/docs phase toggles, docs_include/docs_exclude globs, and service_account (read by the upcoming `me project ci` setup command, never at import time). Also hardens the e2e harness for running inside a live agent session: cliEnv() now strips harness markers (AI_AGENT, CLAUDECODE, …) and the injected-contract vars so the spawned CLI behaves as a plain human/CI invocation instead of tripping the harness failsafe, and the git() helper uses a passed env as the BASE rather than an overlay (overlaying re-leaked the stripped markers into hook-spawned processes). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KrraND2hq3NVUSS38nhP9d
An admin-gated denial was a dead end: a non-admin can't enumerate the
roster to find whom to ask (principal.list is admin-only). Now the
server enriches such FORBIDDEN errors with the space's effective
admins — the same predicate enforce_last_admin counts, via
listSpacePrincipals — as data.admins [{email}] (a user principal's
name IS its email). Computed server-side, only on a denied attempt,
only ever reaching space members; enrichment is best-effort and never
turns a denial into an internal error.
Enriched gates: requireSpaceAdmin + requireTreeOwner (memory RPC —
grants, groups, roster, invites) and requireSpaceAdminById +
requireServiceAccountManager (user RPC — service accounts and their
keys). The CLI renders the contacts under the error in text mode and
carries them as `admins` in structured output.
For `me project ci` (next commit) this is the escalation path: the
denied provisioning run prints the admins and the exact commands to
ask them to run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KrraND2hq3NVUSS38nhP9d
The setup side of the CI import: scaffold the GitHub Actions workflow and provision the service-account credentials. - Workflow: managed-file semantics (marker line; a hand-maintained file is never silently clobbered). The scaffold is repo-agnostic — the default branch is discovered at runtime by the workflow's own job-level `if` (a scaffold-time name silently rots on rename) — so the only variable content is the secret name and, for servers a bare CI checkout couldn't resolve, a baked-in ME_SERVER. - Identity/key phases pivot on secret presence (repo secrets via gh, org-provided via the repo's organization-secrets endpoint). Provisioning is never implicit: --create-service-account or an interactive yes. A named identity (--service-account or the .me import.service_account) turns a secret-present run into verify mode (SA exists + write at the project tree; missing grant is offered/applied; missing SA is a hard inconsistency error). - Keys are minted ONLY when piped straight into `gh secret set` — never displayed or stored; without gh the exact commands are printed and nothing is minted. --rotate-key mints a fresh (collision-free) key for an existing SA. - An admin-gated denial renders the escalation block: the effective admins (from the enriched FORBIDDEN) + the two commands to ask them to run (--admin <you> seeds the bound admin group so the retry is self-serve) + "then re-run me project ci". - Requires a committed .me pinning space and a non-home tree — a service account has no `~`, so a home-rooted tree can never work in CI. - Init grows a thin `ci-workflow` step wrapping the command; a pending end-state (ask-your-admin, no secret yet) never aborts the wizard's remaining steps. e2e drives the full flow against a fake `gh` (deterministic secret state): scaffold-only denial headless, explicit provisioning with SA + grant + key-into-secret, scaffold-only maintenance re-run, committed-identity verify, and rotation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KrraND2hq3NVUSS38nhP9d
The local post-commit hook imported whatever HEAD was — unmerged and rebased commits landed in the tree keyed by (tree, sha) forever — it ran per-clone with the committing human's credentials, and it failed silently. CI imports replaced it (TNT-208). - `me import git-hook` is now a removed-command stub (hidden; accepts the old flags so its message prints instead of a parse error), pointing at `me project ci` or manual block deletion. - import-git-hook.ts is reduced to the marker-based cleanup helpers (installedHookFile / stripHookBlock / removeHookBlock). - `me project ci` migrates installed hooks: it strips the managed block once CI credentials are in place — never earlier, which would open a capture gap between "hook gone" and "CI not yet importing" — with a TTY confirm, or a note when credentials are still pending. - Init's git-import (local backfill) and git-hook steps are gone: the workflow's first run backfills main's full ancestry under the service account. - docs/cli/me-import.md's section is replaced by a removal notice with the one-liner for people who genuinely want local capture (`me import git` in their own hook). BREAKING CHANGE: `me import git-hook` and `me project init`'s --skip-git-import/--skip-git-hook flags are removed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KrraND2hq3NVUSS38nhP9d
- me-import.md: the `me import ci` orchestrator (phases, prune-on semantics, dry-run parity, the import: config block, no structured output). - me-project.md: `me project ci` (workflow scaffold + managed-file semantics, secret-presence gating, provisioning with mint-into- secret, the denied-non-admin escalation, hook migration); the init checklist's Git-history rows replaced by the CI-import row. - project-config.md: the strict `import:` block (phase toggles, docs globs, service_account and when committing it matters). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KrraND2hq3NVUSS38nhP9d
The dogfood scaffold, byte-exact from `me project ci`'s renderer. Inert until an operator completes activation out-of-band: commit a .me/config.yaml pinning this repo's space + shared tree, and run `me project ci --create-service-account` (provisions the service account, grants write at the tree, and sets the ME_API_KEY repo secret). Until then pushes to main will show the run failing loudly at `me import ci` — by design, setup-pending is visible, not silent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KrraND2hq3NVUSS38nhP9d
cevian
changed the title
Comment on lines
+361
to
+364
| output(structured, fmt, () => { | ||
| clack.log.info("No matching docs — skipping the docs import"); | ||
| }); | ||
| return; |
Comment on lines
+28
to
+33
| try { | ||
| const principals = await core.listSpacePrincipals(spaceId, "u"); | ||
| return principals.filter((p) => p.admin).map((p) => ({ email: p.name })); | ||
| } catch { | ||
| return []; | ||
| } |
Comment on lines
570
to
581
| } else { | ||
| output(code ? { error: msg, code } : { error: msg }, fmt, () => {}); | ||
| output( | ||
| { | ||
| error: msg, | ||
| ...(code ? { code } : {}), | ||
| ...(admins ? { admins } : {}), | ||
| }, | ||
| fmt, | ||
| () => {}, | ||
| ); | ||
| } | ||
| process.exit(1); |
serviceAccount.list is silently FILTERED for non-admins (only the accounts the caller administers), so a plain member verifying a committed import.service_account — the org shared-SA flow — got a false "does not exist" hard error, with advice that dead-ends in a name conflict. principal.resolve is the targeted any-member lookup built for exactly this; both the verify-phase and provisioning-phase existence checks now use it (kind 's'), so a non-admin's verify degrades honestly on the admin-gated grant check (verified:false, exit 0) instead of lying about existence, and provisioning finds an SA someone else created rather than re-attempting a doomed creation. e2e 8i grows the non-admin verify case that would have caught this. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KrraND2hq3NVUSS38nhP9d
Comment on lines
466
to
+474
| if (opts.prune) { | ||
| if (unique.length === 0) { | ||
| pruneRefused = | ||
| "empty walk — a wrong directory or over-narrowed --include must not delete the whole corpus"; | ||
| // Under skipIfEmpty (orchestrated runs) an all-empty walk skips the | ||
| // prune silently instead: deleting the whole previously-imported corpus | ||
| // because every file emptied out is exactly what the refusal guards. | ||
| if (opts.skipIfEmpty !== true) { | ||
| pruneRefused = | ||
| "empty walk — a wrong directory or over-narrowed --include must not delete the whole corpus"; | ||
| } |
"Couldn't verify '<sa>' locally (requires admin/owner authority). Run the workflow on GitHub to verify." — says where the real verification happens instead of vaguely gesturing at it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KrraND2hq3NVUSS38nhP9d
Comment on lines
+359
to
+364
| errors: [], | ||
| }; | ||
| output(structured, fmt, () => { | ||
| clack.log.info("No matching docs — skipping the docs import"); | ||
| }); | ||
| return; |
Comment on lines
+571
to
582
| output( | ||
| { | ||
| error: msg, | ||
| ...(code ? { code } : {}), | ||
| ...(admins ? { admins } : {}), | ||
| }, | ||
| fmt, | ||
| () => {}, | ||
| ); | ||
| } | ||
| process.exit(1); | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Design and full implementation for moving repo→memory imports from the per-clone post-commit git hook to a GitHub-Actions-driven import on push to the default branch, treating TNT-208 (git history) and TNT-209 (docs) as one system. The design doc (
CI_IMPORT_DESIGN.md, first commit-group) was reviewed iteratively; the implementation follows its §12 plan as sequenced, individually-reviewable commits.The commits
fix(cli): correct stale importers/git.ts module header — the header described the retired deterministic-id /DO NOTHINGscheme.feat(cli):me import ciorchestrator — the one command the workflow calls: git phase (incremental; first run backfills) then docs phase (default markdown globs, prune on, clean skip when no docs match). Fail-fast, per-phase summaries,--dry-runpreviews locally exactly what CI will do. New strictimport:block in.me/config.yaml(git/docstoggles,docs_include/docs_exclude,service_account). Also hardens the e2e harness to run inside a live agent session (strips harness markers incliEnv(); thegit()helper now uses a passed env as the base — overlaying re-leaked stripped markers into hook-spawned processes).feat(server): authority denials name the space admins — admin-gated FORBIDDEN errors now carrydata.admins: [{email}](effective admins vialistSpacePrincipals, the same predicateenforce_last_admincounts; a user principal's name IS its email). Enriched gates:requireSpaceAdmin/requireTreeOwner(memory RPC) and the two service-account gates (user RPC). CLI renders the contacts under the error; structured output carriesadmins.feat(cli):me project ci+ init step — managed-file workflow scaffold (runtime default-branch gate,--key-name), secret-presence pivot (repo + org-provided secrets viagh), never-implicit provisioning (--create-service-account/ TTY yes), verify mode for a named identity, mint-only-into-gh secret set(key never displayed),--rotate-key, and the denied-non-admin escalation block (two admin commands + self-serve retry). Init wraps it as theci-workflowstep; pending states never abort init.feat(cli)!: removeme import git-hook— removed-command stub;import-git-hook.tsreduced to marker-based cleanup helpers;me project cistrips an installed legacy block once CI credentials are in place (never earlier — avoids a capture gap); init's git-import/git-hook steps dropped.docs: CI import —me-import.md(+ removal notice),me-project.md,project-config.md.ci: dogfood workflow in this repo — byte-exact scaffold; see activation note below.Verification
./bun run check:full(typecheck + lint + unit + integration + e2e against local Postgres) green.me import ci: dry-run, toplevel anchoring, idempotent re-run, prune-on-delete,.mephase gating, no-docs skip, non-repo error,--jsonrejection), 8i (me project cifull flow against a fakeghfor deterministic secret state: headless denial, provisioning with SA+grant+key-into-secret, maintenance re-run, committed-identity verify, rotation), 8f (removed stub + legacy-hook migration), 10d (deniedme service createrenders admin contacts), plus integration tests for the enriched FORBIDDEN on both RPC endpoints and unit tests for the scaffold/parsing/grant-check helpers.The committed workflow is inert-but-loud until, on this repo:
.me/config.yamlpinning this repo's space and a shared tree is committed (I didn't guess your dogfooding space), andme project ci --create-service-accountis run once (provisions the SA, grants write at the tree, sets theME_API_KEYrepo secret).Until then, pushes to main will show the import run failing at
me import ci— visible-pending rather than silent, but if you'd rather not have red runs pre-activation, drop commit 7 or gate it.Questions / uncertainties encountered while building
AdminContactis{email}only, not{name, email}. The design doc said{name, email}with email joined fromauth.users— but core SQL never crosses into the auth schema, the RPC handlers have no auth-store handle, and for user principalscore.principal.nameIS the email. Shipping the email (the actual contact) avoided threading auth-schema access through the server for a display name. The type is an object sonamecan be added compatibly later. OK?requireSpaceAdmin/requireTreeOwnerback groups, roster, invites too. I judged admins-to-members exposure fine (it's "contact your admin", members only), but it's a broader surface than the design's minimum.me import cirejects--json/--yaml(two phases, one stdout). The alternative — refactoring both importers to return structured results for a merged document — felt like scope creep. Run the phases separately for structured output.principal.resolve(the targeted any-member lookup), so any member gets a true answer — the first cut usedserviceAccount.list, which is silently filtered for non-admins and produced a false "does not exist" for a shared org SA; e2e 8i now pins the non-admin case. The grant check (grant.liston another principal) stays admin-gated, so a non-admin gets "couldn't verify (requires authority)" +verified: false+ exit 0 rather than a hard answer — degrade-not-fail because the setup may be perfectly correct (org SA, parent-level grant) and the verifier's privileges shouldn't block it. The first workflow run is the real end-to-end verification either way.gh, not authenticated, or no repo access) blocks the no-flag path with guidance rather than guessing.gh api repos/{nwo}/actions/organization-secretsfailures are treated as "no org secrets" only after the repo-level check succeeded — is that the right 403-vs-404 heuristic for org-secret visibility?--service-accountgrant-fix applies without a prompt (a named SA = stated intent, per the design). Flagging in case that feels too eager for a verify-shaped run.cliEnv()/git()for every test — needed to run the suite from inside any live agent session at all (pre-existing gap: inheritedCLAUDECODE/AI_AGENTmarkers tripped the spawned CLI's failsafe). CI behavior is unchanged (markers absent there), but it touches shared harness code.ci-import-<rand4>(api-key names are unique per principal, so a fixed name would break rotation). Old keys accumulate until revoked — the rotate path prints the revoke commands but doesn't auto-revoke (the old key may still be in use elsewhere, e.g. a second repo sharing the SA).Tickets
🤖 Generated with Claude Code
https://claude.ai/code/session_01KrraND2hq3NVUSS38nhP9d