Login: the operator commands and fleet replication (#261)

AGENTS.md

@@ -33,6 +33,7 @@ State / architecture / decisions / "why": Obsidian Vault, `AINode` (cluster ops:
 - **A fresh install requires a key, and an update never changes an installed node's access control** (`scripts/install.sh`). The installer mints one key on a node with no `config.json`, stores the SHA-256 the way `AuthConfig` does, prints the plaintext ONCE in a box, and the summary reads "API protected, one key"; `AINODE_AUTH=off` keeps the old open behaviour and prints what that choice means. Both gates matter: minting on an existing home would lock out every client the operator already pointed at the node, with a key they never saw. Because of this, `ainode update` verifies the running release on `/api/health` and not `/api/status` (health is the one keyless route, which is why it carries `version`): reading a keyed route there made every update on a protected node pull, pin, restart and then report that it had not applied.
 - **`auth.json` and `config.json` are 0600, written temp-then-replace, and re-read when they change.** Both carry credentials (`config.json` holds `cluster_secret` and may hold `hf_token`) and both were written under the default umask as root. `AuthConfig.reload_if_changed()` stats the store per request the way `ClusterSecret` stats the config per datagram, which is what makes `ainode auth enable|disable|key create|key revoke` LIVE on a running node (it used to need a restart nobody was told to do, so the CLI said "Auth enabled" about a node that went on answering everybody). A file that cannot be stat'ed or parsed keeps the state in memory: the tolerant direction is never "let everybody in". A credential-shaped config key goes in `api/server.py::SCRUBBED_CONFIG_KEYS` and is masked by `ainode config`, or it leaves the node in a `GET /api/config`.
 - **A person logs in, a machine presents a key, and the two stores sit side by side** (`ainode/auth/accounts.py`, `auth/session_routes.py`, #261). `users.json` is 0600, written temp-then-replace and re-read per request exactly as `auth.json` is, because `ainode auth user add`, a password change and a revoked session all have to land on a running node. The ONLY crypto is the standard library's: `hashlib.scrypt` (n=2**14, r=8, p=1) over a per-password salt, and a session token stored as its SHA-256 alone, so the file a backup or a support bundle picks up logs nobody in and a login page is not the reason the image grows an argon2 wheel. **A session does not expire** (in until you log out), which makes REVOCATION the whole control and it has to be complete: a password change, a disable and a removal each take that account's open sessions with them, and `session_for_token` refuses one whose account has since been removed or disabled. **A cookie-authenticated request whose method is not GET, HEAD or OPTIONS must carry `X-AINode-Client: dashboard`**, and that header is the whole CSRF defence: a browser attaches a cookie to a cross-site request on its own, and a custom header cannot be set cross-origin without a preflight this node never approves, so dropping the rule would make every node an operator is logged into writable by any page on the internet. A Bearer token is tried FIRST and the cookie only when no key matched, so automation is untouched and a stale key in a browser does not cost a good session its request. Administration (`/api/auth/users*`, `/api/auth/keys*`, `/api/auth/enable|disable`) goes through `accounts.require_admin`: an admin session, an operator key, or the fleet key, never a member; the two replication routes are the FLEET key alone because they move password hashes; and the last ENABLED admin cannot be removed or disabled, enforced in the store (`UsersStore.is_last_admin`) so the CLI cannot get around it. `/api/auth/login` and `/api/auth/me` are the newest entries in `SKIP_PATHS`, and adding one there means naming it in the middleware docstring, `api/cluster_join.py`'s docstring, the README's keyless paragraph and the dashboard's API access panel, because four tests assert each of those.
+- **Dashboard ACCOUNTS have one authority, the master, and SESSIONS are never replicated** (`ainode/auth/replication.py`). A cluster cannot hold one account list per node: an operator who adds a login on the master and then opens the dashboard of whichever node a bookmark points at would be told their password is wrong. So the master pushes its `export_users()` to `/api/auth/users/sync` on every peer after every change (`app["users_changed"]`, with `fleet_headers` like every other node-to-node call, because the export carries password hashes), retrying a peer that refused on the next change and on a 60 second tick while it is behind; a worker pulls `/api/auth/users/export` at startup and every 5 minutes, which is the safety net a push cannot be (a node that was down, or one that has just joined and has no accounts at all). "Behind" is a comparison and not a queue: the master remembers the stamp each peer accepted. A session is one browser's credential against ONE node, so nothing copies them and `ainode auth session ...` is a per-node command; a node with no `cluster_secret` or nobody to talk to does nothing at all, because it could not authenticate to a peer anyway. The FIRST account is made on the box (`ainode auth user add <name> --admin`), never over HTTP: a route that mints the first admin is a route anybody who can reach the port calls before the operator does. `--password-stdin` is not optional garnish, it is the only path with no TTY, and the installer's wrapper runs `docker exec -it`. A worker records the master's own stamp in `users-sync.json` so `ainode doctor`'s `login.sync` compares two values of the same kind rather than two hashes computed by different code. Pinned by `tests/test_auth_replication.py` and `tests/test_cli_auth_users.py`.
 - **A discovery announcement is SIGNED when `cluster_secret` is set, and a node that has one drops what it cannot verify** (`discovery/signing.py`). The signature is one extra top-level key beside the payload's own fields, never a wrapper around them, so a peer that has never heard of it drops the unknown key in `from_json` and stays in the cluster view. Both directions read the secret per datagram through `ClusterSecret`, so rotating it is a `config.json` edit and not a fleet-wide restart: never capture the value at startup. A node with NO secret behaves exactly as it did and says so once per process, because nothing on the fleet sets one and a mandatory-signing release would partition every existing cluster on upgrade.
 - **The discovery port has one home: `core/config.py::DEFAULT_DISCOVERY_PORT` (5679).** `NodeConfig`, both discovery classes and the installer read it from there. They each carried their own literal, and the dataclass said 5678 while the installer, the fleet and the docs said 5679, so a source install listened where nobody spoke and vanished from every cluster view with nothing logged (#181). The port, the cluster id and whether the wire is signed are logged together at startup.
 - **`ainode_version` travels on the announcement and into every view that lists nodes** (`/api/nodes`, `/api/cluster/resources`, `/api/version/check`, `/api/cluster/update-status`). A peer that announces none reports `""` and is counted as unknown: filling it in with the local version is how a split fleet goes on looking like a healthy one (#171).

ainode/api/server.py

@@ -635,6 +635,23 @@ def _get_master_address() -> Optional[str]:
                 )
             )
 
+    # Dashboard accounts come from the MASTER, and this is the only thing that
+    # moves them (ainode/auth/replication.py): the master pushes every change to
+    # its peers, a worker pulls at startup and every five minutes, and sessions
+    # are never replicated because a session is one browser's credential against
+    # one node. Started here, after the cluster secret and the HTTP session exist,
+    # because both are what it authenticates and talks with. A node with no
+    # cluster_secret gets no loop and says so once: it could not authenticate to a
+    # peer anyway, and pretending otherwise would fill the log with 401s.
+    try:
+        from ainode.auth.replication import start_replication
+
+        replicator = start_replication(app)
+        if replicator is not None:
+            await replicator.start()
+    except Exception:
+        logger.exception("could not start account replication")
+
 
 def _start_metrics_retention(app: web.Application, config: NodeConfig) -> None:
     """Open ``<AINODE_HOME>/metrics.db`` and start the sampler, if enabled.
@@ -883,6 +900,15 @@ async def _on_cleanup(app: web.Application) -> None:
         except asyncio.CancelledError:
             pass
 
+    # Stop the account replication loop. Nothing it does is a write this node
+    # needs to finish, so a cancel is the whole teardown.
+    try:
+        from ainode.auth.replication import stop_replication
+
+        await stop_replication(app)
+    except Exception:  # pragma: no cover - teardown must not raise
+        logger.exception("could not stop account replication")
+
     # Stop Ray autostart task
     ray_task = app.get("_ray_autostart_task")
     if ray_task: