Skip to content

A write token may set up a base - #1244

Merged
WaylandYang merged 3 commits into
deeplethe:devfrom
gedaiu:feat/write-token-sets-up-a-base
Oct 16, 2026
Merged

WaylandYang merged 3 commits into
deeplethe:devfrom
gedaiu:feat/write-token-sets-up-a-base

Conversation

@gedaiu

@gedaiu gedaiu commented Oct 15, 2026

Copy link
Copy Markdown
Contributor

Closes #1243.

What happened

A backend that mirrors many repositories into Utopia needs a base and a statements source per repository. The push is the source key's job (0054), but creating the base and the source took a session, so the backend either held someone's seven-day JWT or a person did the setup by hand for every repository.

What changed

Five routes now also take a personal token with write scope, acting as the person who issued it: POST /workspaces/{id}/kbs, POST /kbs/{id}/sources, GET /kbs/{id}/sources/{sid}/token, POST /kbs/{id}/ontology/imports and POST /kbs/{id}/review/alignment/kind-words/{word}. GET /workspaces/{id}/kbs takes any token, so a client can find a base by name before it creates one.

  • One extractor, SessionOrToken, resolves either credential to the same User. Only these routes and document content (0052, where DocumentReader was a near-copy of it and is folded in) use it; AuthUser still reads every bearer as a JWT, so every other route stays session-only.
  • The token only narrows. Each handler calls require_write first and then the role check it already had: a read token is refused with 403 token_read_only, a token limited by kb_ids reaches only those bases and creates none (403 token_kb_out_of_scope), and the owner's role is still the ceiling. AppError::CodedForbidden carries those codes, the same shape as CodedConflict, so a client can tell "use another token" from "your role is not enough".
  • Expiry and revocation are judged in SQL on every request; a deactivated owner is refused. Audit rows name the owner, as from a session.

0014 gains decision 6 with the reasons it keeps (a living person is the actor, scope from the token, writing must be ticked, no base-level machine tokens or service accounts) and what is not done. Its status line also said can_write was hard-coded false, which #318 had fixed; a dated note corrects it.

Tests

api::token_setup_tests, against a real database: a workspace admin's write token creates a base and a source, reads the push key, passes the import and kind-word checks, and the audit rows name the admin, while an expired and a revoked token from the same person get 401; a read token lists bases and is refused with token_read_only on all five writing routes; a viewer's write token creates no base, and a token limited to base A creates a source in A but not in B and no base, and lists only A.

cargo fmt --all --check, cargo clippy --workspace --all-targets -- -D warnings and cargo test --workspace pass with UTOPIA_TEST_REQUIRE_DB=1 (utopia-server 634 passed, utopia-store 306). The web UI needs no change.

Five routes take a personal token with write scope as well as a session,
acting as the token's owner: creating a base in a workspace, creating a
source, reading a source's push key, importing an ontology, and deciding a
kind word. Listing a workspace's bases takes any token. The person's role
is still the ceiling, a read token is refused with token_read_only, and a
token limited by kb_ids reaches only those bases and creates none
(token_kb_out_of_scope). One extractor, SessionOrToken, takes either
credential; AuthUser still reads every bearer as a JWT.

0014 gains decision 6.

Closes deeplethe#1243

Signed-off-by: Bogdan Szabo <bogdan@re-cinq.com>
gedaiu added a commit to re-cinq/lore that referenced this pull request Oct 15, 2026
## Why

#2767 mirrors each repo's spec and ADR chain into Utopia, but a person
has to run the setup route once per repo with their Utopia session. The
ask was that Lore creates the knowledge base itself when it is missing.
Utopia's setup routes took a session only; deeplethe/utopia#1244 lets a
write-scoped personal token call them, acting as its owner. It is open
upstream and ships in our build `ghcr.io/re-cinq/utopia:0.1.0-re.2`
(re-dev). This is the Lore side, off by default.

## What Changed

- `withSession(jwt)` becomes `withCredential(bearer)`: the same requests
with a session JWT or a `utp_pat_…` write token.
- With `UTOPIA_API_URL`, `UTOPIA_TOKEN` and `UTOPIA_WORKSPACE_ID` set,
the first spec or ADR ingest of a repo with no stored source runs the
setup under the token
(`apps/lore-api/src/work/utopia/utopia-auto-setup.ts`) and stores the
key and settings through `storeUtopiaSetup`, the one writer the setup
route now shares. Two deltas for one repo in a process share one setup;
across processes the base is found by name and the per-repo upsert keeps
the last saved key. A failed setup is a mirror failure: logged, counted,
never raised to the ingest.
- The setup route is unchanged for deployments without a token.
- Infra behind `enable_utopia_auto_setup` (default false) and
`utopia_lore_workspace_id`: GSM container `lore-utopia-lore-api-token`,
ExternalSecret `utopia-token` in `lore-api`, `seed-secrets.sh` OPTIONAL
entry, chart env `UTOPIA_WORKSPACE_ID` and `UTOPIA_TOKEN` (secretKeyRef)
only when the flag is on, so the chart never references a secret that
does not exist. A validation refuses the flag without `enable_utopia`
and a workspace id.
- Spec FR5.5 and FR6.1–6.4, an ADR-052 amendment,
`docs/managing-secrets.md` (minting and rotating the token), CLAUDE.md.

## Testing

Client and `sourceToMirror` tests against a local HTTP Utopia: first
ingest sets the repo up and stores the key; two concurrent calls
(`re-cinq/lore`, `re-cinq/Lore`) make one setup; a stored source makes
no calls; no token means nothing happens; a 403 stores nothing and the
next call succeeds. libs/shared 3226 and lore-api 1596 tests pass; `tsc`
clean everywhere; `terraform fmt`/`validate` pass; the chart renders
with the flag off, on, and on with the token.

To turn on, once Utopia runs 0.1.0-re.2: mint the token in Utopia, seed
`lore-utopia-lore-api-token`, set both variables, `terraform apply` from
main.

Refs #2766
gedaiu added a commit to re-cinq/lore that referenced this pull request Oct 15, 2026
## Why

The image Utopia runs was pinned only in the git-ignored
`terraform.tfvars`, so main never recorded which build is deployed, and
the apply on 2026-10-15 kept running `0.1.0-re.1` because a local file
still named it. `0.1.0-re.2` carries what the Lore mirror needs:
deeplethe/utopia#1244 (a write token sets up a base, for #2768's
auto-setup) and #1245 (a pushed phrase equal to a declared relation is
typed without the model).

## What Changed

- `variables.tf` defaults: `utopia_image_repository =
"ghcr.io/re-cinq/utopia"`, `utopia_image_tag = "0.1.0-re.2"`,
`utopia_image_digest = "sha256:936141d3…"` (the pushed image, built from
gedaiu/utopia `re-dev` at tag `v0.1.0-re.2`). The descriptions say what
the build carries and that a bump is a PR here.
- `terraform.tfvars.example` no longer sets the tag; it says the pin
lives in the defaults and tfvars overrides only to test a build.

## Testing

`terraform fmt -check` and `terraform validate` pass. The image boots on
a fresh database, runs migration 0113 and answers health; the full
Utopia suite passed on the tagged commit (1,545 tests).

Refs #2766

@WaylandYang WaylandYang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, this is careful work: the token only narrows, the role checks are unchanged, and the read/expired/revoked/kb_ids cases are all covered by tests that run in CI.

One change before merge: please limit what a token can create in POST /kbs/{id}/sources to the push kinds (statements, and api if you need it), and refuse other kinds from a token with a coded 403. As it stands a write token can create url/rss/custom sources, which the server starts fetching immediately (sources_routes.rs:111), with stored credentials for custom. That turns a leaked CI token from "can read and propose memories" into "can make the server call any address on a schedule", and #1243 only needs the statements source. A session keeps creating every kind. One test case for the refusal is enough.

Also, please put the token id in the audit metadata on the rows these routes already write (source.created, alignment.kind_word_decided, ontology import). 0014 decision 5 relies on knowing which token acted; without it, an admin can't tell what a leaked token did before revoking it. A line in decision 6 noting that a system admin's write token acts as Owner in every base would also help the next reader.

A write token may create only a statements source; any other kind is
refused with token_source_kind, since url, rss and custom sources make the
server fetch on a schedule. source.created, alignment.kind_word_decided and
ontology.imported carry detail.token_id when a token acted, so an admin can
see what a token did before revoking it. Decision 6 of 0014 says so, and
that a system admin's write token acts as Owner in every base.

Signed-off-by: Bogdan Szabo <bogdan@re-cinq.com>
@gedaiu

gedaiu commented Oct 16, 2026

Copy link
Copy Markdown
Contributor Author

Thanks, both done in 0ed8e9b.

  • Source kinds: a token now creates only statements sources. Every other kind is refused with a coded 403 token_source_kind after the write check and before anything is written; sessions still create every kind. I left out api: it is also a push kind, but A write-scoped personal token could create a knowledge base and a source in its owner's workspace #1243 only needs statements, so it stays out until someone does. The refusal is tested by a_write_token_creating_a_url_source_is_refused_with_token_source_kind, which checks the 403 and code and that no source row exists.
  • Audit: source.created, alignment.kind_word_decided and ontology.imported carry detail.token_id when a token acted, through a small audit::via_token helper. This includes packs installed while creating a base. A session's detail is unchanged. a_live_write_token_of_a_workspace_admin_sets_up_a_base_and_a_dead_one_does_not now asserts the token id on the source.created row.
  • 0014 decision 6: it now says that a system admin's write token acts as Owner in every base, that tokens create only statements sources and why, and which audit rows name the token (per decision 5). I removed the old "not done" sentence about audit rows not naming the token. access-and-audit.md now says "a statements source".

Local runs: fmt and clippy (-D warnings) are clean. The utopia-server bin suite passes, except the load-sensitive the_embedding_gate_is_never_held_beyond_its_ceiling, which passes on its own.

gedaiu added a commit to re-cinq/lore that referenced this pull request Oct 16, 2026
Pins Utopia `0.1.0-re.4`
(`sha256:0697b93d1c976b02409e63f08ed9b1706d4806b31466208d8da61e2464b9e02c`)
in the committed defaults of `infra/terraform/variables.tf`.

`re.4` is `re.3` (re-dev `bc23e69`, which includes #1250) plus the fixes
the maintainer asked for in their review of:
- deeplethe/utopia#1244: a write token creates only `statements` sources
(anything else gets a 403 `token_source_kind`), and the audit rows for
source creation, kind-word decisions and ontology imports record the
token id.
- deeplethe/utopia#1245: a declared state relation is left to the
aligner instead of being bound by the ontology.

Lore pushes only `statements` sources and declares no state relations,
so neither fix changes what the mirror does.

There is no new migration: 113 is still the last one. Before tagging,
the merged re-dev (`26ea10b`) passed fmt, clippy and the `utopia-server`
/ `utopia-store` tests. The image was boot-tested on an empty database:
health returned 200 and migrations ran to 113.

Takes effect with `terraform apply` from main.

@WaylandYang WaylandYang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, both points are addressed exactly as asked: a token now creates only statements sources and is refused with token_source_kind before anything is written, sessions are unaffected, and the three audit rows carry detail.token_id so a leaked token's actions can be traced before it is revoked. Decision 6 reads well with the system-admin sentence. Merging.

@WaylandYang
WaylandYang merged commit 15921f0 into deeplethe:dev Oct 16, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants