feat: connectors and item files

CHANGELOG.md

@@ -6,6 +6,58 @@ project adheres to [Semantic Versioning](https://semver.org/).
 
 ## [Unreleased]
 
+### Changed
+
+- The browsable API's sync button reads "Sync" (was "Sync now"), as in the web UI.
+- The image ships unified-ui 0.2.0 (`UI_REF`): the federation map on edges, a source's groups with their hosts and variables, exact live durations, and the shared look with unified-gitops.
+
+- **Breaking (everyone: every configuration directory):** the configuration is item files, the way AWX configuration as code is laid out. `config.yaml` stays at the root, unchanged; every other `*.yaml`/`*.yml`, at any depth and in any folder, holds items under a typed top-level key (`sources`, `views`, `endpoints`, `enrichers`, `connectors`, `projects`, `credentials`, `api_keys`). Each item has a `name` (its identity: what references, URLs and cache keys use — the 0.34 ids) and an optional `description` (its label). A name defined twice, an unknown top-level key or an unknown key inside an item fails the load naming the file and the item. The eight fixed files of id-keyed maps are gone; a 0.34 directory is converted at startup (see Upgrading below). See `docs/configuration.md`.
+- **Breaking (anyone writing sources):** what a source said about its script moved to a new `connectors` item: `type` (`script` or `ssh`), `project`, `path` (the script inside the project — `project` + `path` always means a file in that checkout), `script_args`, `host_args`, `output_format` and a default `timeout_seconds`. A source names it with `connector:` and keeps only what is its own: credentials, `config`, schedule, TTLs and every per-source flag, plus an optional `timeout_seconds` override. Sources of the same system now share one connector instead of each carrying a copy. Two connectors are built in and need no item: `remote` (another unified-api: `config.url` and `config.source`, the source id on that instance — formerly `script_path`) and `static_inventory` (`project` + `path` on the source — formerly `project_id` + `script_path`). See `docs/connectors.md`.
+- **Breaking (anyone writing references):** reference keys lost their `_id` suffix: a source's `credentials`, a project's `credential`, an enricher's `target`, `source` and `project`, an endpoint's `sources` and `project`.
+- **Breaking (pipelines using the configuration API):** files are addressed by path, folders included: `GET/PUT/DELETE /api/v1/config/{path}` with slashes (`/api/v1/config/sources/acme/src-d42.yaml`). A whole-directory push with `prune: true` removes every file at any depth that it leaves out, and the folders it empties; `missing` only ever names `config.yaml`. A push still in the 0.34 layout is refused with a hint naming `--migrate-config`.
+- **Breaking (`GET /api/v1/sources/{id}` admin callers):** `gathering.project_id` is null for remote and ssh sources, which run nothing from a checkout (it used to repeat a project that was required and never used).
+
+### Added
+
+- Edges show the whole federation. Every minute, an instance with remote sources pushes its map to each peer (`PUT /api/v1/topology/federation`, with the key its remote sources already use); admins read it at `GET /api/v1/topology/federation`. Nothing to configure on the edges. The map an edge receives is redacted the way a restricted caller's is: its peers' names and versions, not their own maps, and no error that names the central's secrets. Only a key sent with the request may push, never a browser session. See `docs/federation.md`.
+- `GET /api/v1/projects` answers each project's `revision` (the commit its checkout is at) and `credential_id`, so the UI's project cards show the commit and the credential, as unified-gitops does.
+- `GET /api/v1/topology`: each peer source's `last_pull_age_seconds`, when data last moved along that link (its last successful sync); the UI animates a link each time it does.
+- `GET /api/v1/sources/{id}/groups/{group}`: one group's variables, child groups and host count, for sources and views, without fetching the whole dataset.
+- `server.name` (optional, reloadable): what the instance is called on the Topology page; in its own topology answer and in the map it pushes. Without it the UI derives one from the URL.
+- `server.url` (optional, reloadable): this instance's public address. A central sends it with the federation map so edges can link back to it; also shown on `GET /api/v1/settings`.
+
+- A browser that is not signed in is sent to the login form by the server when it asks for a page of the UI, so the app never loads only to redirect itself (no flash of the overview before the login page).
+
+- `GET /api/v1/connectors` and `GET /api/v1/connectors/{id}` (admin-only): each connector, the two built-ins included (`built_in: true`), what it runs, the sources that use it with the timeout each syncs under, and a health summary of those sources. Linked from the API index as `connectors`.
+- `gathering.connector` on a source's page, `used_by.connectors` on projects, and `connectors` in the configuration API's summary and reload report.
+- `unified-api --migrate-config [dir]`: converts a 0.34 directory in place, for a configuration repository to run once and commit.
+
+### Fixed
+
+- A browser session opened with an API key is ended by a reload that removes that key's `name` (its identity), not by its label. Two keys with the same label, one of them removed, left the removed key's sessions working with its old permissions until they expired.
+
+### Upgrading from 0.34
+
+Response shapes and ids are unchanged: consumers see no difference. On
+startup, a configuration directory still in the 0.34 layout is converted once,
+before it is loaded: connectors are extracted from the sources and
+de-duplicated, references renamed, one file written per item
+(`<section>/<name>.yaml`), and the old files moved to a hidden
+`.legacy-config-<timestamp>/` folder; the log says what was done. Values are
+written as they were (`1.10`, `0x1F` and `True` stay exactly that), and an
+unquoted value in a text field is still text (`script_args: ["--days", 7]`).
+A generated connector never takes a name already in use, whether another
+connector's or a built-in's (a script in a `remote/` folder becomes `remote-2`). Nothing is
+moved unless the converted configuration loads — otherwise the process refuses
+to start with the directory untouched. A read-only directory (a ConfigMap) is
+converted into a temporary copy instead, every start; the copy is removed
+when the process stops, or by the next start after a crash. Comments in the old
+files are not carried over (the backup keeps them). Convert the source of the
+files too — `unified-api --migrate-config <dir>`, or by hand — or the next
+push puts the old layout back; `--check-config` on a 0.34 directory checks the
+conversion without writing. Rolling back to 0.34 needs the old files restored
+from the backup.
+
 ## [0.34.0] - 2026-10-25
 
 ### Added

CLAUDE.md

@@ -42,10 +42,13 @@ src/
 ├── main.rs                   # Entrypoint: load config, build app, start Axum
 ├── lib.rs                    # Module tree + AppBuilder (composition root)
 ├── state.rs                  # AppState (ports as Arc<dyn Trait> + static config)
-├── config.rs                 # YAML configuration loading from config/ directory
+├── config.rs                 # Loader: config.yaml + item files at any depth (SECTIONS)
+├── config/
+│   └── migrate.rs            # one-time 0.34 → item-file conversion (startup, --migrate-config)
 ├── domain/                   # Core domain types (pure, no dependencies)
 │   ├── dataset.rs            # Dataset, Group, HostVars
-│   ├── source.rs             # Source, TtlOverrides, ConnectorType
+│   ├── source.rs             # SourceDef (file) → Source (resolved with its connector), TtlOverrides
+│   ├── connector.rs          # Connector item (how to talk to a system); built-ins remote/static_inventory
 │   ├── cache_entry.rs        # CacheEntry with TTL logic
 │   ├── sync_health.rs        # SyncHealth + registry (one per kind: sync/enrich/project/snapshot)
 │   ├── credential.rs         # Credential, CredentialType
@@ -95,6 +98,7 @@ src/
 │   │   │   ├── hosts.rs      # PUT/DELETE host
 │   │   │   ├── endpoints.rs  # Output endpoints (GET and POST)
 │   │   │   ├── projects.rs   # Git project routes (admin-only)
+│   │   │   ├── connectors.rs # GET connectors + their users and health (admin-only)
 │   │   │   ├── health.rs     # /healthz, /readyz
 │   │   │   ├── metrics.rs    # /metrics (Prometheus exporter, installed once)
 │   │   │   ├── login.rs      # login page, sessions, logout
@@ -115,8 +119,9 @@ src/
 │       └── secrets/          # env.rs: SecretsPort → env/JSON files; mock.rs: test double;
 │                             #   vault.rs; cache.rs; reloadable.rs (swappable chain);
 │                             #   build_chain() — the one place the chain is wired
-config/                       # Split YAML config (server, credentials, sources, etc.)
+config/                       # Example config: config.yaml + item files (sources/, connectors/, ...)
 tests/                        # Integration tests (*.rs), with sample scripts mirroring src/adapters/out/:
+├── fixtures/legacy-0.34/     # a 0.34-layout directory the migration tests convert
 └── adapters/
     └── out/                  # sample scripts — stand-ins for the driven adapters
         ├── connectors/       #   sample source connectors (incl. slow.py for timeout tests)
@@ -136,6 +141,19 @@ translators that call it; don't put orchestration logic in either.
 No external data dependencies (no Redis, no PostgreSQL). All cache in-memory with DashMap.
 Cache mutations must use the atomic `CachePort::update`/`merge_or_insert` operations —
 never the get → modify → set pattern (it loses concurrent writes).
+Configuration is item files (AWX style): `config.yaml` at the root, every other
+`*.yaml`/`*.yml` at any depth holds items under a typed top-level key
+(`config::SECTIONS`); `name` is the identity (what references, URLs and cache keys
+use), `description` the label. Struct `name` fields hold that LABEL — `config::take`
+moves `description` into them, which is what keeps API `name` fields unchanged.
+References are written without `_id` (`credentials`, `project`, `target`, `sources`);
+Rust fields kept their old names via `#[serde(rename)]`, because the API still
+answers `target_id` etc. A source item (`SourceDef`) is resolved with its connector
+at load into the flat runtime `Source` that everything downstream reads — never
+read a connector at sync time. A new item type touches `SECTIONS`, the loader's
+`take` calls, `AppConfig`, and `config::migrate`'s key order. A 0.34 directory is
+converted at startup (`config::migrate`): staged, validated, backed up to
+`.legacy-config-*`, never half-written; a read-only one into a temp copy.
 Configuration from YAML files, parsed STRICTLY: every config struct carries
 `deny_unknown_fields`, so a typo'd key fails startup naming the key instead of
 silently applying the default (policy comment in `config.rs`). Free-form data
@@ -222,7 +240,7 @@ Vault (KV v2, token or Kubernetes auth) per credential via `vault_path` +
   removal) and one naming an api-key env var that is not set. See
   `docs/config-api.md`.
 
-- **Views:** `views.yaml` declares read-only composites over several sources.
+- **Views:** `views` items declare read-only composites over several sources.
   A view is served on the SOURCE routes and shares their id space (config
   validation rejects a collision) — that is what makes migrating a consumer a
   one-word change. It holds no cache entry: `application::views::snapshot`
@@ -268,7 +286,7 @@ Vault (KV v2, token or Kubernetes auth) per credential via `vault_path` +
   waits for every configured source. Either way the body carries
   `sources_synced` and `sources_pending`, so a failing probe names what it is
   waiting for.
-- **Auth:** keys are declared in `api_keys.yaml`, each `role: admin` (everything)
+- **Auth:** keys are declared as `api_keys` items, each `role: admin` (everything)
   or restricted to explicit `sources`/`endpoints` id lists; secrets come from the
   env var each definition names. The legacy `UNIFIED_API_KEY` still works as one
   extra admin key. Constant-time compare; `/healthz`, `/readyz` and Swagger

README.md

@@ -19,6 +19,7 @@ and every one of them sees the same inventory.
 ## ✨ Features
 
 - **Pluggable sources**: any executable that prints inventory JSON is a connector
+- **Connectors**: one item says how to read a system; every instance of it is a source naming it
 - **SSH connector**: gathers Ansible facts from whole fleets in parallel
 - **In-memory cache with TTLs**: per-dataset, per-host and per-group freshness, no database
 - **Gzip responses**: a client that accepts gzip transfers about a tenth of the bytes
@@ -28,6 +29,7 @@ and every one of them sees the same inventory.
 - **Views**: one id over several sources, routing each host to the member that owns it
 - **Scheduled + on-demand sync**: interval sync per source, plus scoped sync over the API
 - **Refresh**: a read can bring the hosts it names up to date, bounded by the source's TTL
+- **Config as code**: AWX-style item files, one per source, in any folders you like
 - **Configuration API**: a pipeline pushes the config directory and reloads it live, no restart
 - **Web UI**: sources, endpoints, topology, metrics and host search in a browser, same address
 - **Swagger UI**: interactive OpenAPI docs served at `/swagger-ui/`
@@ -41,13 +43,15 @@ and every one of them sees the same inventory.
 |---|---|
 | [API](docs/api.md) | All routes with authentication, status code semantics and curl examples |
 | [Architecture](docs/architecture.md) | The four layers, the dependency rule and the concurrency model |
-| [CLI](docs/cli.md) | Environment variables, log tuning, health checks and common curl operations |
-| [Configuration](docs/configuration.md) | Every YAML file field by field, env vars and startup validation |
+| [Browsable API](docs/browsable-api.md) | The API's own pages in a browser: login, sessions, navigation |
+| [CLI](docs/cli.md) | `--check-config`, `--migrate-config`, env vars, log tuning and common curl operations |
+| [Configuration](docs/configuration.md) | The item-file layout, every item field by field, upgrading from 0.34 |
 | [Configuration API](docs/config-api.md) | Pushing the config directory over HTTP, validating it and reloading it live |
-| [Connectors](docs/connectors.md) | Script contracts for connectors, enrichers and output transformers |
+| [Connectors](docs/connectors.md) | Connector items, the built-ins, and the script contracts for all three kinds |
 | [Deployment](docs/deployment.md) | Container image, worked config example, CI/CD, Kubernetes and ArgoCD |
 | [Enrichers](docs/enrichers.md) | Post-processing cached data: modes, triggers, freshness rules, health |
 | [Federation](docs/federation.md) | One instance per datacenter, one central federating them, no WAN SSH |
+| [LDAP login](docs/ldap-login.md) | Signing in with a directory account, and the access rules that decide what it reaches |
 | [Observability](docs/observability.md) | Scheduling, structured logs and the metrics worth alerting on |
 | [Output endpoints](docs/endpoints.md) | Consumer-facing transformers: contract, GET vs POST, permissions |
 | [Projects](docs/projects.md) | Git checkouts: sync styles, script resolution, virtualenvs, health |

config/api_keys/example.yaml

@@ -0,0 +1,17 @@
+---
+# API keys with per-consumer permissions. The secret is never in this file:
+# `env` names the environment variable that holds it. No keys defined and no
+# UNIFIED_API_KEY = open API (a warning is logged at startup).
+#
+# api_keys:
+#   - name: key-awx
+#     description: "AWX"
+#     env: "UNIFIED_API_KEY_AWX"
+#     role: "admin"                    # everything
+#
+#   - name: key-forms
+#     description: "AnsibleForms"
+#     env: "UNIFIED_API_KEY_FORMS"
+#     # role defaults to "restricted": only what is listed below
+#     sources: ["src-section9"]        # may list/read/sync these sources
+#     endpoints: ["ep-ansible-full"]   # may list/run these endpoints

config/config.yaml

@@ -3,7 +3,7 @@ server:
   port: 8182
   # The web UI on this same address: the image ships a build of unified-ui at
   # /usr/share/unified-api/ui. Sign-in is the API's own login page (an API key,
-  # or LDAP: docs/ldap-login.md); with no api_keys.yaml the API is open and so
+  # or LDAP: docs/ldap-login.md); with no api_keys items the API is open and so
   # is the UI. Remove the block for an API without a UI. See docs/ui.md.
   ui:
     enabled: true
@@ -19,7 +19,7 @@ server:
 # Optional: let an admin key read, write and reload this directory over HTTP,
 # so a configuration pipeline can push a change instead of publishing an
 # artifact this instance has to pull. OFF by default — turning it on means an
-# admin key can rewrite every file the loader reads, api_keys.yaml included.
+# admin key can rewrite every file the loader reads, api_keys included.
 # See docs/config-api.md.
 # config_api:
 #   enabled: true

config/connectors/granular.yaml

@@ -0,0 +1,16 @@
+---
+# A script that can gather ONE host without gathering the inventory. The full
+# sync calls it with script_args, a `?host=` sync with host_args instead:
+#
+#   POST /api/v1/sources/src-granular/sync                      -> --list
+#   POST /api/v1/sources/src-granular/sync?host=motoko.section9.net
+#                                        -> --only-host motoko.section9.net
+#
+# Try it and read `called_with` on the host to see which mode ran.
+connectors:
+  - name: granular
+    description: "Granular inventory script"
+    project: "prj-connectors-infra"
+    path: "tests/adapters/out/connectors/granular.py"
+    script_args: ["--list"]
+    host_args: ["--only-host", "{target}"]

config/connectors/sample-infra.yaml

@@ -0,0 +1,6 @@
+---
+connectors:
+  - name: sample-infra
+    description: "Sample infrastructure data"
+    project: "prj-connectors-infra"
+    path: "tests/adapters/out/connectors/infra.py"

config/connectors/sample-inventory.yaml

@@ -0,0 +1,14 @@
+---
+# A connector says HOW to read a kind of system: which script, in which
+# project, with which arguments. Every source of that system names it
+# (`connector: sample-inventory`) and adds only what is its own: credentials,
+# config, schedule, TTLs. See docs/connectors.md.
+connectors:
+  - name: sample-inventory
+    description: "Sample inventory script"
+    type: "script"                      # script (default) | ssh
+    project: "prj-connectors-infra"     # the checkout `path` resolves in
+    path: "tests/adapters/out/connectors/inventory.py"
+    # script_args: ["--list"]           # for an Ansible dynamic inventory script
+    # output_format: "ansible"          # native (default) | ansible
+    # timeout_seconds: 300              # default; a source may override it

config/connectors/ssh-facts.yaml

@@ -0,0 +1,10 @@
+---
+# SSH: the native connector, nothing installed on the hosts. `path` is what
+# runs remotely — a label such as "gather_facts" for the built-in gather
+# modes, or the command itself for `gather_mode: script`. No project: nothing
+# runs from a checkout.
+connectors:
+  - name: ssh-facts
+    description: "SSH fact gathering"
+    type: "ssh"
+    path: "gather_facts"