From eb4332a4eb7e7dbe19bb6b0a04b666b16cc8569d Mon Sep 17 00:00:00 2001 From: webdevtodayjason Date: Mon, 21 Sep 2026 18:49:46 -0500 Subject: [PATCH 1/4] Operator commands and fleet replication for the dashboard login (#261) The login gives a node a second credential, a named account with a password in users.json, and two things have to exist around it before it is usable on this fleet: a way to make the first account (there is no sign-up page, and must not be: a route that mints the first admin is a route anybody who can reach port 3000 calls before the operator does), and one authority for the account list, because an operator who adds a login on the master and then opens the dashboard of whichever node a bookmark points at would otherwise be told their password is wrong. `ainode auth user add|list|remove|passwd|disable|enable` and `ainode auth session list|revoke|clear` write the store on the box, which the running node picks up through reload_if_changed exactly as `ainode auth enable` is live. Interactive add prompts twice with getpass; --password-stdin is the path with no TTY, which matters because the installer's host wrapper runs `docker exec -it` and a script, a unit or `ssh host ainode ...` has no terminal. Removing or disabling the last admin is refused: with auth on and no admin the dashboard can only be opened by pasting an API key, which is what the login exists to replace. `ainode auth status` now counts users, admins and sessions beside the keys. ainode/auth/replication.py makes the master the authority. It registers app["users_changed"], so every account mutation pushes export_users() to /api/auth/users/sync on every peer with fleet_headers (the export carries password hashes, so that key is the only one the route takes). A peer that refused is retried on the next change and on a 60 second tick while it is behind, which is a comparison and not a queue: the master remembers the stamp each peer accepted, so a peer that failed simply still disagrees. A worker pulls /api/auth/users/export at startup and every 5 minutes, because a push cannot reach a node that was down and a node that has just joined has no accounts at all. Sessions are never replicated: a session is one browser's credential against one node. `ainode doctor` gains a Login check: FAIL when auth is on with no account, WARN with accounts but no enabled admin, WARN on a users.json wider than 0600 (--fix tightens it), and on a worker WARN when its list is older than the master's stamp, comparing two of the master's own stamps rather than two hashes computed by different code. The installer prints the two lines an operator needs after a protected install, and nothing when auth is off. Co-Authored-By: Claude Fable 5.1 --- AGENTS.md | 1 + ainode/api/server.py | 26 + ainode/auth/replication.py | 856 +++++++++++++++++++++++++++++++++ ainode/cli/doctor.py | 201 ++++++++ ainode/cli/main.py | 567 +++++++++++++++++++++- scripts/install.sh | 15 + tests/test_auth_replication.py | 809 +++++++++++++++++++++++++++++++ tests/test_cli_auth_users.py | 716 +++++++++++++++++++++++++++ tests/test_doctor.py | 214 +++++++++ 9 files changed, 3404 insertions(+), 1 deletion(-) create mode 100644 ainode/auth/replication.py create mode 100644 tests/test_auth_replication.py create mode 100644 tests/test_cli_auth_users.py diff --git a/AGENTS.md b/AGENTS.md index aa5642b6..6883622e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -33,6 +33,7 @@ State / architecture / decisions / "why": Obsidian Vault, `AINode` (cluster ops: - **A fresh install requires a key, and an update never changes an installed node's access control** (`scripts/install.sh`). The installer mints one key on a node with no `config.json`, stores the SHA-256 the way `AuthConfig` does, prints the plaintext ONCE in a box, and the summary reads "API protected, one key"; `AINODE_AUTH=off` keeps the old open behaviour and prints what that choice means. Both gates matter: minting on an existing home would lock out every client the operator already pointed at the node, with a key they never saw. Because of this, `ainode update` verifies the running release on `/api/health` and not `/api/status` (health is the one keyless route, which is why it carries `version`): reading a keyed route there made every update on a protected node pull, pin, restart and then report that it had not applied. - **`auth.json` and `config.json` are 0600, written temp-then-replace, and re-read when they change.** Both carry credentials (`config.json` holds `cluster_secret` and may hold `hf_token`) and both were written under the default umask as root. `AuthConfig.reload_if_changed()` stats the store per request the way `ClusterSecret` stats the config per datagram, which is what makes `ainode auth enable|disable|key create|key revoke` LIVE on a running node (it used to need a restart nobody was told to do, so the CLI said "Auth enabled" about a node that went on answering everybody). A file that cannot be stat'ed or parsed keeps the state in memory: the tolerant direction is never "let everybody in". A credential-shaped config key goes in `api/server.py::SCRUBBED_CONFIG_KEYS` and is masked by `ainode config`, or it leaves the node in a `GET /api/config`. - **A person logs in, a machine presents a key, and the two stores sit side by side** (`ainode/auth/accounts.py`, `auth/session_routes.py`, #261). `users.json` is 0600, written temp-then-replace and re-read per request exactly as `auth.json` is, because `ainode auth user add`, a password change and a revoked session all have to land on a running node. The ONLY crypto is the standard library's: `hashlib.scrypt` (n=2**14, r=8, p=1) over a per-password salt, and a session token stored as its SHA-256 alone, so the file a backup or a support bundle picks up logs nobody in and a login page is not the reason the image grows an argon2 wheel. **A session does not expire** (in until you log out), which makes REVOCATION the whole control and it has to be complete: a password change, a disable and a removal each take that account's open sessions with them, and `session_for_token` refuses one whose account has since been removed or disabled. **A cookie-authenticated request whose method is not GET, HEAD or OPTIONS must carry `X-AINode-Client: dashboard`**, and that header is the whole CSRF defence: a browser attaches a cookie to a cross-site request on its own, and a custom header cannot be set cross-origin without a preflight this node never approves, so dropping the rule would make every node an operator is logged into writable by any page on the internet. A Bearer token is tried FIRST and the cookie only when no key matched, so automation is untouched and a stale key in a browser does not cost a good session its request. Administration (`/api/auth/users*`, `/api/auth/keys*`, `/api/auth/enable|disable`) goes through `accounts.require_admin`: an admin session, an operator key, or the fleet key, never a member; the two replication routes are the FLEET key alone because they move password hashes; and the last ENABLED admin cannot be removed or disabled, enforced in the store (`UsersStore.is_last_admin`) so the CLI cannot get around it. `/api/auth/login` and `/api/auth/me` are the newest entries in `SKIP_PATHS`, and adding one there means naming it in the middleware docstring, `api/cluster_join.py`'s docstring, the README's keyless paragraph and the dashboard's API access panel, because four tests assert each of those. +- **Dashboard ACCOUNTS have one authority, the master, and SESSIONS are never replicated** (`ainode/auth/replication.py`). A cluster cannot hold one account list per node: an operator who adds a login on the master and then opens the dashboard of whichever node a bookmark points at would be told their password is wrong. So the master pushes its `export_users()` to `/api/auth/users/sync` on every peer after every change (`app["users_changed"]`, with `fleet_headers` like every other node-to-node call, because the export carries password hashes), retrying a peer that refused on the next change and on a 60 second tick while it is behind; a worker pulls `/api/auth/users/export` at startup and every 5 minutes, which is the safety net a push cannot be (a node that was down, or one that has just joined and has no accounts at all). "Behind" is a comparison and not a queue: the master remembers the stamp each peer accepted. A session is one browser's credential against ONE node, so nothing copies them and `ainode auth session ...` is a per-node command; a node with no `cluster_secret` or nobody to talk to does nothing at all, because it could not authenticate to a peer anyway. The FIRST account is made on the box (`ainode auth user add --admin`), never over HTTP: a route that mints the first admin is a route anybody who can reach the port calls before the operator does. `--password-stdin` is not optional garnish, it is the only path with no TTY, and the installer's wrapper runs `docker exec -it`. A worker records the master's own stamp in `users-sync.json` so `ainode doctor`'s `login.sync` compares two values of the same kind rather than two hashes computed by different code. Pinned by `tests/test_auth_replication.py` and `tests/test_cli_auth_users.py`. - **A discovery announcement is SIGNED when `cluster_secret` is set, and a node that has one drops what it cannot verify** (`discovery/signing.py`). The signature is one extra top-level key beside the payload's own fields, never a wrapper around them, so a peer that has never heard of it drops the unknown key in `from_json` and stays in the cluster view. Both directions read the secret per datagram through `ClusterSecret`, so rotating it is a `config.json` edit and not a fleet-wide restart: never capture the value at startup. A node with NO secret behaves exactly as it did and says so once per process, because nothing on the fleet sets one and a mandatory-signing release would partition every existing cluster on upgrade. - **The discovery port has one home: `core/config.py::DEFAULT_DISCOVERY_PORT` (5679).** `NodeConfig`, both discovery classes and the installer read it from there. They each carried their own literal, and the dataclass said 5678 while the installer, the fleet and the docs said 5679, so a source install listened where nobody spoke and vanished from every cluster view with nothing logged (#181). The port, the cluster id and whether the wire is signed are logged together at startup. - **`ainode_version` travels on the announcement and into every view that lists nodes** (`/api/nodes`, `/api/cluster/resources`, `/api/version/check`, `/api/cluster/update-status`). A peer that announces none reports `""` and is counted as unknown: filling it in with the local version is how a split fleet goes on looking like a healthy one (#171). diff --git a/ainode/api/server.py b/ainode/api/server.py index b613771d..736e7601 100644 --- a/ainode/api/server.py +++ b/ainode/api/server.py @@ -635,6 +635,23 @@ def _get_master_address() -> Optional[str]: ) ) + # Dashboard accounts come from the MASTER, and this is the only thing that + # moves them (ainode/auth/replication.py): the master pushes every change to + # its peers, a worker pulls at startup and every five minutes, and sessions + # are never replicated because a session is one browser's credential against + # one node. Started here, after the cluster secret and the HTTP session exist, + # because both are what it authenticates and talks with. A node with no + # cluster_secret gets no loop and says so once: it could not authenticate to a + # peer anyway, and pretending otherwise would fill the log with 401s. + try: + from ainode.auth.replication import start_replication + + replicator = start_replication(app) + if replicator is not None: + await replicator.start() + except Exception: + logger.exception("could not start account replication") + def _start_metrics_retention(app: web.Application, config: NodeConfig) -> None: """Open ``/metrics.db`` and start the sampler, if enabled. @@ -883,6 +900,15 @@ async def _on_cleanup(app: web.Application) -> None: except asyncio.CancelledError: pass + # Stop the account replication loop. Nothing it does is a write this node + # needs to finish, so a cancel is the whole teardown. + try: + from ainode.auth.replication import stop_replication + + await stop_replication(app) + except Exception: # pragma: no cover - teardown must not raise + logger.exception("could not stop account replication") + # Stop Ray autostart task ray_task = app.get("_ray_autostart_task") if ray_task: diff --git a/ainode/auth/replication.py b/ainode/auth/replication.py new file mode 100644 index 00000000..eb083bfc --- /dev/null +++ b/ainode/auth/replication.py @@ -0,0 +1,856 @@ +"""Accounts come from the master. Sessions never leave the node they were made on. + +The dashboard login (#261) puts a second credential on a node: a user account with +a password, in ``/users.json``. A cluster cannot have one of those +per node. An operator who adds an account on the master and then opens the +dashboard of whichever node a browser bookmark happens to point at would be told +their password is wrong, and the fix ("log in on a different node") is not a fix. + +So there is ONE authority for accounts, the cluster's master, and this module is +the only thing that moves them: + +* **The master pushes.** Every mutation the account routes make calls + ``app["users_changed"]()``, which is registered here. It POSTs the master's + ``export_users()`` (hashes included, which is why the route takes the fleet key + and nothing else) to ``/api/auth/users/sync`` on every peer discovery knows + about, in the background. A peer that did not take the push is retried on the + next change and on a :data:`RETRY_INTERVAL_SECONDS` timer for as long as it is + behind, so an account added while a node was rebooting lands when it comes back. +* **A worker pulls**, once at startup and every :data:`PULL_INTERVAL_SECONDS` + after that, because a push cannot reach a node that was down when it happened + and because a node that has just joined has no accounts at all. +* **Sessions are NOT replicated, ever.** A session is a cookie one browser holds + against one node; copying them around would hand every node in the fleet a + credential it never issued, and revoking a session would have to be a fan-out + to be true. ``ainode auth session ...`` is therefore a per-node command, and + that is the honest shape rather than a limitation. +* **A node with no ``cluster_secret``, or with nobody to talk to, does nothing.** + The fleet key is derived from that secret (``ainode/auth/fleet.py``), so a node + without one cannot authenticate to a peer and must not pretend to: it keeps the + accounts it has and ``ainode doctor`` says why. + +Both halves share the two questions the CLI also has to answer, so they live here +as plain functions rather than in the loop: which peers are there +(:func:`peer_targets`), and is this node the authority (:func:`local_role`). +``ainode auth user ...`` writes the store on the box and then calls +:func:`replicate_from_cli`, which is the same push over the stdlib, because the +CLI runs in a different process from the server (the installer's wrapper is +``docker exec``) and has no event loop to borrow. +""" + +from __future__ import annotations + +import asyncio +import hashlib +import json +import logging +import os +import time +from pathlib import Path +from typing import Any, Optional + +from ainode.auth.fleet import cluster_secret_of, fleet_headers, fleet_key_headers + +logger = logging.getLogger(__name__) + +#: The two routes this module speaks, both fleet-key only (they carry hashes). +EXPORT_PATH = "/api/auth/users/export" +SYNC_PATH = "/api/auth/users/sync" +#: How the CLI asks a running node who the master is and who the peers are. +CLUSTER_INFO_PATH = "/api/cluster/info" + +#: A worker re-reads the master's list this often. Five minutes because the pull +#: is the SAFETY NET under the master's push, not the mechanism: a change the +#: operator makes is on every reachable peer within a second of making it. +PULL_INTERVAL_SECONDS = 300.0 +#: How often the loop wakes to retry a peer that is behind. Also the master's +#: idle tick, which costs one in-memory stamp comparison when nothing is behind. +RETRY_INTERVAL_SECONDS = 60.0 +#: One node-to-node request's budget. Short: a peer that is down must not hold +#: the loop, and the next tick is 60 seconds away. +REQUEST_TIMEOUT_SECONDS = 10.0 + +#: What a worker remembers about its last successful pull, under AINODE_HOME. It +#: holds the MASTER's own stamp for the list it imported, so ``ainode doctor`` can +#: ask the master for its current stamp and compare two values of the same kind. +#: No hashes and no passwords go in here, so it is not a credential file. +SYNC_STATE_NAME = "users-sync.json" + +#: The fallback minimum password length, used only when the account store does not +#: publish one of its own. The store is the authority and refuses a short password +#: itself; the CLI asks for the number so it can say no at the prompt instead of +#: after the operator has typed the same password twice. +DEFAULT_MIN_PASSWORD = 8 + + +# --------------------------------------------------------------------------- +# The account store, which lands on its own branch +# --------------------------------------------------------------------------- + +def users_store_class(): + """``UsersStore`` from :mod:`ainode.auth.accounts`, or ``None``. + + The import is guarded because this module, the CLI and the doctor are written + against the store's contract and must import cleanly on a tree that does not + carry it yet. Returning None (rather than raising) is also what lets a test + put its own store in place by replacing this one function. + """ + try: + from ainode.auth.accounts import UsersStore + except Exception: # pragma: no cover - only on a tree without the store + return None + return UsersStore + + +def min_password_length() -> int: + """The store's minimum password length, or :data:`DEFAULT_MIN_PASSWORD`. + + Asked of the store rather than restated here, so the CLI's refusal and the + store's refusal cannot drift apart. The class first, then the module, because + the contract fixes the number (8) and not where it is spelled. + """ + cls = users_store_class() + holders: list = [cls] + try: + from ainode.auth import accounts + + holders.append(accounts) + except Exception: # pragma: no cover - only on a tree without the store + pass + for holder in holders: + value = getattr(holder, "MIN_PASSWORD_LENGTH", None) + if isinstance(value, int) and value > 0: + return value + return DEFAULT_MIN_PASSWORD + + +def open_store(app=None): + """The node's account store, or ``None`` when this release has none. + + Prefers the one already on the application, so the CLI-free path (the server) + and this module cannot end up holding two objects over one file. Off an app + there is nothing to share, so a fresh one is opened. + + ``UsersStore.load()`` is modelled on ``AuthConfig.load()``, which is a + classmethod returning a new instance, so the result is kept when it is one and + ignored when ``load()`` is an instance method that reads in place. + """ + if app is not None: + existing = None + getter = getattr(app, "get", None) + if callable(getter): + existing = getter("users_store") + if existing is not None: + return existing + cls = users_store_class() + if cls is None: + return None + try: + store = cls() + loaded = store.load() + if isinstance(loaded, cls): + store = loaded + except Exception: + logger.exception("could not open the account store") + return None + if app is not None and callable(getattr(app, "get", None)): + try: + app["users_store"] = store + except Exception: # pragma: no cover - a mapping that refuses writes + pass + return store + + +# --------------------------------------------------------------------------- +# Pure helpers: the stamp, the addresses, the role +# --------------------------------------------------------------------------- + +def users_stamp(users: Any) -> str: + """A content id for an exported user list, so "behind" is answerable. + + The master compares this against what each peer last accepted to decide who + still needs the push, which is what makes the retry a comparison rather than + a queue: a peer that failed simply still disagrees. It is a hash of the list, + so it changes on a password change (the hashes are salted) and not on a + re-export of the same accounts. + """ + try: + blob = json.dumps(users, sort_keys=True, separators=(",", ":"), default=str) + except (TypeError, ValueError): # pragma: no cover - defensive + blob = repr(users) + return hashlib.sha256(blob.encode("utf-8")).hexdigest()[:16] + + +def node_base_url(host: str, port: int) -> str: + """``http://host:port`` for one node, or "" with no usable host.""" + host = str(host or "").strip() + if not host: + return "" + bracketed = f"[{host}]" if ":" in host and not host.startswith("[") else host + return f"http://{bracketed}:{int(port)}" + + +def peer_address(node) -> str: + """The address to reach a peer on. + + One derivation for the whole product (``api/server_routes.peer_host``: the UDP + source address, then the fabric IP, then the name), imported here rather than + at module scope so this module stays importable from the CLI without dragging + the API package in, and so a test can replace this seam with a loopback + address the real one deliberately refuses to publish. + """ + from ainode.api.server_routes import peer_host + + return peer_host(node) + + +def _local_node_ids(app, config) -> set: + """Every id that means THIS node, so it is never pushed to as a peer. + + Both spellings, because a node whose ``config.json`` has no ``node_id`` yet + (one that has never run ``ainode start``) announces itself as ``"unknown"`` + and would otherwise appear in its own peer list. + """ + ids = {getattr(config, "node_id", None)} + getter = getattr(app, "get", None) if app is not None else None + if callable(getter): + ids.add(getattr(getter("announcement"), "node_id", None)) + ids.discard(None) + return ids + + +def peer_targets(app) -> list[tuple[str, str]]: + """``(label, base url)`` for every other node of this cluster, from discovery. + + Offline members are left out: this is a list of nodes to push to now, not a + roster. A member with no usable address is left out too, and named in the + debug log rather than pushed to a guess. + """ + cluster = None + getter = getattr(app, "get", None) + if callable(getter): + cluster = getter("cluster_state") + if cluster is None: + return [] + config = getter("config") if callable(getter) else None + local_ids = _local_node_ids(app, config) + try: + from ainode.discovery.broadcast import NodeStatus + + members = [n for n in cluster.members() if n.status != NodeStatus.OFFLINE] + except Exception: + logger.exception("could not read this cluster's members") + return [] + + targets: list[tuple[str, str]] = [] + for node in members: + node_id = getattr(node, "node_id", None) + if node_id in local_ids: + continue + port = int(getattr(node, "web_port", 3000) or 3000) + base = node_base_url(peer_address(node), port) + if not base: + logger.debug("no usable address for %s, so accounts are not pushed to it", + node_id) + continue + targets.append((str(getattr(node, "node_name", "") or node_id or "?"), base)) + return targets + + +def master_target(app=None, config=None, info: Optional[dict] = None) -> str: + """The master's base URL from this node's point of view, or "". + + Discovery first (the elected master is the live answer), then the + ``master_address`` a join wrote into config.json, which is the whole answer on + a node whose discovery has not heard anybody yet. + """ + getter = getattr(app, "get", None) if app is not None else None + if config is None and callable(getter): + config = getter("config") + local_ids = _local_node_ids(app, config) + + cluster = getter("cluster_state") if callable(getter) else None + if cluster is not None: + try: + master = cluster.get_master() + except Exception: # pragma: no cover - defensive + master = None + if master is not None and getattr(master, "node_id", None) not in local_ids: + base = node_base_url(peer_address(master), + int(getattr(master, "web_port", 3000) or 3000)) + if base: + return base + + if isinstance(info, dict): + for node in info.get("members") or []: + if not isinstance(node, dict): + continue + if node.get("effective_role") != "master": + continue + if node.get("node_id") in local_ids: + continue + host = str(node.get("host") or node.get("node_name") or "").strip() + base = node_base_url(host, int(node.get("web_port") or 3000)) + if base: + return base + address = str(info.get("master_address") or "").strip() + if address: + return _base_from_address(address, config) + + address = str(getattr(config, "master_address", "") or "").strip() + if address: + return _base_from_address(address, config) + return "" + + +def _base_from_address(address: str, config=None) -> str: + """``http://host:port`` from whatever ``master_address`` holds.""" + from ainode.cluster.join import parse_host_port + + default_port = int(getattr(config, "web_port", 3000) or 3000) + try: + host, port = parse_host_port(address, default_port=default_port) + except ValueError: + return "" + return node_base_url(host, port) + + +def local_role(config, info: Optional[dict] = None, cluster=None) -> str: + """Is this node the account authority: ``"master"``, ``"worker"`` or ``"solo"``. + + ``"solo"`` is a node with nobody to disagree with: it is its own authority, so + an account added there is simply the node's own and no warning is owed. The + three sources are asked in order of how much they know: the running + ``ClusterState`` (what the server has), a ``GET /api/cluster/info`` payload + (what the CLI has while the service is up), then config.json alone (all a CLI + on a stopped node has). + + A configured role WINS over an election in one direction only: an operator who + wrote ``cluster_role: "worker"`` has said this node is not the authority, and + a node that cannot see its master must not decide it has been promoted. + """ + configured = str(getattr(config, "cluster_role", "") or "auto").strip().lower() + mode = str(getattr(config, "distributed_mode", "") or "").strip().lower() + if configured == "worker" or mode == "member": + return "worker" + + local_id = getattr(config, "node_id", None) + if cluster is not None: + try: + master = cluster.get_master() + peers = len([n for n in cluster.members() + if getattr(n, "node_id", None) != local_id]) + # An election that HAS an answer is the authority, even against a + # config that pinned this node master: two nodes both configured + # master must not both push, and the election already picks one. + if master is not None: + if getattr(master, "node_id", None) == local_id: + return "master" if peers else "solo" + return "worker" + except Exception: # pragma: no cover - defensive + logger.exception("could not read the cluster's master") + + if isinstance(info, dict): + members = [m for m in (info.get("members") or []) if isinstance(m, dict)] + info_id = info.get("my_node_id") or local_id + peers = len([m for m in members if m.get("node_id") != info_id]) + role = str(info.get("my_role") or "").strip().lower() + if role == "master": + return "master" if peers else "solo" + if role: + return role + + # Config.json alone, which is all a CLI on a stopped node has. + peer_ips = [p for p in (getattr(config, "peer_ips", []) or []) if p] + if configured == "master": + return "master" if peer_ips else "solo" + if str(getattr(config, "master_address", "") or "").strip(): + return "worker" + if peer_ips: + return "master" + return "solo" + + +# --------------------------------------------------------------------------- +# What a worker remembers about its last pull +# --------------------------------------------------------------------------- + +def ainode_home(home=None) -> Path: + """AINODE_HOME, resolved at call time so a test's redirect is honoured.""" + if home is not None: + return Path(home) + from ainode.core import config as core_config + + return Path(os.environ.get("AINODE_HOME") or core_config.AINODE_HOME) + + +def sync_state_path(home=None) -> Path: + return ainode_home(home) / SYNC_STATE_NAME + + +def read_sync_state(home=None) -> dict: + """The last pull this node recorded, or ``{}``. Never raises.""" + path = sync_state_path(home) + try: + loaded = json.loads(path.read_text()) + except (OSError, ValueError): + return {} + return loaded if isinstance(loaded, dict) else {} + + +def write_sync_state(stamp: str, master: str = "", users: int = 0, home=None) -> None: + """Record a successful pull, temp-then-replace. Never raises.""" + path = sync_state_path(home) + payload = { + "stamp": str(stamp or ""), + "master": str(master or ""), + "users": int(users or 0), + "at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), + } + try: + path.parent.mkdir(parents=True, exist_ok=True) + tmp = path.with_name(path.name + ".tmp") + tmp.write_text(json.dumps(payload, indent=2)) + tmp.replace(path) + except OSError as exc: + logger.warning("could not record the account sync state in %s: %s", path, exc) + + +# --------------------------------------------------------------------------- +# The loop: a master pushes, a worker pulls +# --------------------------------------------------------------------------- + +class AccountReplicator: + """One task per node that keeps accounts equal to the master's. + + Holds no accounts of its own: the store on the app is the only copy, and this + reads it per push so a change the CLI wrote on the box (which the store picks + up through ``reload_if_changed``) reaches the peers on the next tick even when + nothing called :meth:`notify_changed`. + """ + + def __init__(self, app, pull_interval: float = PULL_INTERVAL_SECONDS, + retry_interval: float = RETRY_INTERVAL_SECONDS): + self.app = app + self.pull_interval = float(pull_interval) + self.retry_interval = float(retry_interval) + #: peer label -> the stamp that peer last ACCEPTED. A peer missing from + #: here, or holding another stamp, is behind and gets the next push. + self.accepted: dict[str, str] = {} + self._wake = asyncio.Event() + self._task: Optional[asyncio.Task] = None + self._master_unreachable_logged = False + self._next_pull = 0.0 + + # -- state ------------------------------------------------------------ + + @property + def config(self): + getter = getattr(self.app, "get", None) + return getter("config") if callable(getter) else None + + def role(self) -> str: + getter = getattr(self.app, "get", None) + cluster = getter("cluster_state") if callable(getter) else None + return local_role(self.config, cluster=cluster) + + def store(self): + return open_store(self.app) + + def behind(self, targets: list[tuple[str, str]], stamp: str) -> list[tuple[str, str]]: + """The peers whose last accepted stamp is not *stamp*.""" + return [(label, base) for label, base in targets + if self.accepted.get(label) != stamp] + + # -- the two directions ------------------------------------------------ + + async def broadcast(self) -> dict: + """Push this node's accounts to every peer that is behind. + + Returns what happened, for the caller that wants to assert on it. A peer + that refuses or cannot be reached is logged and left behind, which is + exactly what makes the next tick retry it. + """ + store = self.store() + if store is None: + return {"pushed": [], "failed": [], "reason": "no account store"} + try: + reload_if_changed = getattr(store, "reload_if_changed", None) + if callable(reload_if_changed): + reload_if_changed() + users = store.export_users() + except Exception: + logger.exception("could not export this node's accounts") + return {"pushed": [], "failed": [], "reason": "export failed"} + + stamp = users_stamp(users) + targets = peer_targets(self.app) + # Forget peers that are no longer in the cluster view, so a node that left + # does not hold the loop at a 60 second tick forever. + known = {label for label, _ in targets} + for label in list(self.accepted): + if label not in known: + self.accepted.pop(label, None) + todo = self.behind(targets, stamp) + if not todo: + return {"pushed": [], "failed": [], "stamp": stamp, + "peers": len(targets)} + + session = self._session() + if session is None: + return {"pushed": [], "failed": [label for label, _ in todo], + "reason": "no HTTP session", "stamp": stamp} + + import aiohttp + + pushed: list[str] = [] + failed: list[str] = [] + for label, base in todo: + url = base + SYNC_PATH + try: + async with session.post( + url, json={"users": users}, + headers=fleet_headers(self.app), + timeout=aiohttp.ClientTimeout(total=REQUEST_TIMEOUT_SECONDS), + ) as resp: + body = await resp.json(content_type=None) + if resp.status != 200: + failed.append(label) + logger.warning("account sync to %s answered %s", label, + resp.status) + continue + except (aiohttp.ClientError, asyncio.TimeoutError, ValueError) as exc: + failed.append(label) + logger.warning("account sync to %s failed: %s", label, exc) + continue + self.accepted[label] = stamp + pushed.append(label) + changed = bool(isinstance(body, dict) and body.get("changed")) + if changed: + logger.info("accounts replicated to %s (%d account(s))", label, + len(users)) + return {"pushed": pushed, "failed": failed, "stamp": stamp, + "peers": len(targets)} + + async def pull(self) -> dict: + """Import the master's accounts onto this node. + + The master is the authority, so this REPLACES what is here. The first + failure to reach it is a WARNING and the rest are debug lines: a master + that is down for an hour must not write sixty identical warnings into the + log of a node that is working perfectly well otherwise. + """ + store = self.store() + if store is None: + return {"imported": False, "reason": "no account store"} + base = master_target(self.app) + if not base: + return {"imported": False, "reason": "no master known"} + session = self._session() + if session is None: + return {"imported": False, "reason": "no HTTP session"} + + import aiohttp + + url = base + EXPORT_PATH + try: + async with session.get( + url, headers=fleet_headers(self.app), + timeout=aiohttp.ClientTimeout(total=REQUEST_TIMEOUT_SECONDS), + ) as resp: + if resp.status != 200: + return self._master_unreachable(base, f"answered {resp.status}") + body = await resp.json(content_type=None) + except (aiohttp.ClientError, asyncio.TimeoutError, ValueError) as exc: + return self._master_unreachable(base, str(exc)) + + if not isinstance(body, dict) or not isinstance(body.get("users"), list): + return self._master_unreachable(base, "answered a body with no user list") + + users = body["users"] + try: + changed = bool(store.import_users(users)) + except Exception: + logger.exception("could not import the master's accounts") + return {"imported": False, "reason": "import failed"} + + self._master_unreachable_logged = False + stamp = str(body.get("stamp") or "") or users_stamp(users) + write_sync_state(stamp, master=base, users=len(users)) + if changed: + logger.info("accounts updated from the master at %s: %d account(s)", + base, len(users)) + return {"imported": True, "changed": changed, "count": len(users), + "stamp": stamp, "master": base} + + def _master_unreachable(self, base: str, reason: str) -> dict: + if not self._master_unreachable_logged: + self._master_unreachable_logged = True + logger.warning( + "cannot read accounts from the master at %s (%s); this node keeps " + "the accounts it has and will try again every %.0fs", + base, reason, self.pull_interval) + else: + logger.debug("master at %s still unreachable: %s", base, reason) + return {"imported": False, "reason": reason, "master": base} + + def _session(self): + getter = getattr(self.app, "get", None) + session = getter("client_session") if callable(getter) else None + if session is None or getattr(session, "closed", False): + return None + return session + + # -- one pass, and the loop around it --------------------------------- + + async def tick(self) -> dict: + """One pass, which is the unit the tests drive. + + A master pushes what is behind (nothing on the wire when every peer + agrees). A worker pulls, but only when its five minutes are up: the 60 + second wake exists for the master's retry, and it must not turn into a + poll of the master twelve times more often than intended. + """ + role = self.role() + if role == "master": + return await self.broadcast() + if role == "worker": + now = time.monotonic() + if now < self._next_pull: + return {"imported": False, "reason": "not due"} + result = await self.pull() + # A pull that FAILED waits the retry interval, not the full five + # minutes: a worker that came up while its master was still booting + # would otherwise sit without accounts for the rest of the interval. + self._next_pull = now + (self.pull_interval if result.get("imported") + else self.retry_interval) + return result + return {"reason": "solo node, nothing to replicate", "role": role} + + def notify_changed(self) -> None: + """Registered as ``app["users_changed"]``: wake the loop, never block it. + + The account routes call this inside a request, so it does exactly one + thing. On a worker the wake is harmless: the next tick pulls the master's + list back over the local change, which is the behaviour the CLI warns + about before it makes one. + """ + self._wake.set() + + async def _run(self) -> None: + while True: + try: + await self.tick() + except asyncio.CancelledError: + raise + except Exception: # pragma: no cover - a tick must never kill the loop + logger.exception("account replication tick failed") + try: + await asyncio.wait_for(self._wake.wait(), timeout=self._sleep_for()) + except asyncio.TimeoutError: + pass + self._wake.clear() + + def _sleep_for(self) -> float: + """Sixty seconds while anything may be behind, else the pull interval.""" + if self.role() == "master": + return self.retry_interval + return min(self.retry_interval, self.pull_interval) + + async def start(self) -> None: + if self._task is None or self._task.done(): + self._task = asyncio.get_event_loop().create_task(self._run()) + + async def stop(self) -> None: + task, self._task = self._task, None + if task is None: + return + task.cancel() + try: + await task + except asyncio.CancelledError: + pass + except Exception: # pragma: no cover - a dying task must not block cleanup + pass + + +def start_replication(app) -> Optional[AccountReplicator]: + """Wire account replication onto a running app, or answer why it did not. + + Called from the server's startup beside the other background tasks. Two gates, + both of them "this node has nobody to replicate with": no ``cluster_secret`` + means no fleet key, so every request this would make answers 401 and the + honest thing is to make none; clustering switched off means there is no + discovery to name a peer with. Returns None in both cases, and the node keeps + whatever accounts it has. + + The broadcaster is registered whatever this node's role is today, because the + role is an election and can change under a running process. It asks the role + per push instead, so a node that becomes master starts pushing and one that + stops being master stops. + """ + getter = getattr(app, "get", None) + if not callable(getter): + return None + config = getter("config") + if not str(cluster_secret_of(app) or "").strip(): + logger.info("account replication off: this node has no cluster_secret, so it " + "cannot authenticate to a peer") + return None + if not bool(getattr(config, "cluster_enabled", True)) and not str( + getattr(config, "master_address", "") or "").strip(): + logger.info("account replication off: clustering is disabled and no " + "master_address is set") + return None + + replicator = AccountReplicator(app) + app["users_replicator"] = replicator + app["users_changed"] = replicator.notify_changed + return replicator + + +async def stop_replication(app) -> None: + """Stop the loop, if one was started. Never raises.""" + getter = getattr(app, "get", None) + replicator = getter("users_replicator") if callable(getter) else None + if replicator is None: + return + await replicator.stop() + + +# --------------------------------------------------------------------------- +# The same push, from the CLI, over the stdlib +# --------------------------------------------------------------------------- + +def http_json(url: str, payload: Optional[dict] = None, headers: Optional[dict] = None, + timeout: float = REQUEST_TIMEOUT_SECONDS) -> tuple[int, Any]: + """One stdlib request: GET with no *payload*, POST with one. + + ``(status, body)``, where a status of 0 means the request never got an answer. + urllib rather than aiohttp: this runs from ``ainode auth user add`` on the + box, and the CLI's import cost is paid by every other subcommand too. It is a + module-level seam, so a test replaces this one name. + """ + import urllib.error + import urllib.request + + data = None + request_headers = dict(headers or {}) + if payload is not None: + data = json.dumps(payload).encode("utf-8") + request_headers["Content-Type"] = "application/json" + req = urllib.request.Request(url, data=data, + method="POST" if data is not None else "GET", + headers=request_headers) + try: + with urllib.request.urlopen(req, timeout=timeout) as resp: + raw = resp.read() + try: + return resp.status, json.loads(raw) + except ValueError: + return resp.status, None + except urllib.error.HTTPError as exc: + raw = exc.read() + try: + return exc.code, json.loads(raw) + except ValueError: + return exc.code, None + except (urllib.error.URLError, OSError, TimeoutError, ValueError) as exc: + return 0, {"error": str(exc)} + + +def cluster_info(config) -> Optional[dict]: + """``GET /api/cluster/info`` on THIS node, or None when it does not answer. + + The CLI runs inside the container the server runs in (the installer's wrapper + is ``docker exec``), so this is a loopback call, and it is the only way a + process that is not the server can know who the elected master is and which + peers discovery has heard from. Carries the fleet key, like every other + node-to-node read: the route needs a caller id once auth is on. + """ + port = int(getattr(config, "web_port", 3000) or 3000) + url = f"http://127.0.0.1:{port}{CLUSTER_INFO_PATH}" + status, body = http_json(url, headers=fleet_key_headers( + getattr(config, "cluster_secret", ""))) + if status != 200 or not isinstance(body, dict): + return None + return body + + +def peer_targets_from_info(info: Optional[dict], config) -> list[tuple[str, str]]: + """``(label, base url)`` for every other node in a ``/api/cluster/info`` body. + + ``/api/cluster/info`` names members but not their addresses, so the address + comes from the same three places it does everywhere else: the ``host`` field + when a future release adds one, then the member's name, which on this fleet is + a resolvable hostname. A member with neither is skipped rather than guessed at. + """ + if not isinstance(info, dict): + return [] + local_id = info.get("my_node_id") or getattr(config, "node_id", None) + targets: list[tuple[str, str]] = [] + for member in info.get("members") or []: + if not isinstance(member, dict): + continue + if member.get("node_id") and member.get("node_id") == local_id: + continue + status = str(member.get("status") or "").strip().lower() + if status == "offline": + continue + host = str(member.get("host") or member.get("fabric_ip") + or member.get("node_name") or "").strip() + base = node_base_url(host, int(member.get("web_port") or 3000)) + if not base: + continue + targets.append((str(member.get("node_name") or member.get("node_id") or "?"), + base)) + return targets + + +def peer_targets_from_config(config) -> list[tuple[str, str]]: + """``(label, base url)`` for the peers config.json names. + + The fallback for a CLI whose own node is not answering, so there is no cluster + view to read: a ``--job master`` install records its peers in ``peer_ips``, and + pushing to those is better than telling an operator the change reached nobody. + Their web port is this node's, which is true of every node the installer + touches, and a peer that disagrees is caught by the master's own loop later. + """ + port = int(getattr(config, "web_port", 3000) or 3000) + targets: list[tuple[str, str]] = [] + for peer in getattr(config, "peer_ips", []) or []: + host = str(peer or "").strip() + base = node_base_url(host, port) + if base: + targets.append((host, base)) + return targets + + +def replicate_from_cli(config, users: list, info: Optional[dict] = None) -> dict: + """Push *users* to every peer this node knows, with the fleet key. + + The CLI's half of the master's push: ``ainode auth user add`` writes the store + on the box, the running server picks the file up through ``reload_if_changed``, + and this is what puts the same accounts on the peers without waiting for the + server's own 60 second retry. A peer that does not answer is REPORTED and not + retried here, because the CLI exits: the master's loop pushes to it on the next + tick, which is the mechanism that actually guarantees delivery. + """ + targets = peer_targets_from_info(info, config) or peer_targets_from_config(config) + headers = fleet_key_headers(getattr(config, "cluster_secret", "")) + if not headers: + return {"pushed": [], "failed": [], "peers": len(targets), + "reason": "no cluster_secret on this node"} + pushed: list[str] = [] + failed: list[tuple[str, str]] = [] + for label, base in targets: + status, body = http_json(base + SYNC_PATH, payload={"users": users}, + headers=headers) + if status == 200: + pushed.append(label) + continue + detail = f"HTTP {status}" if status else "no answer" + if isinstance(body, dict) and body.get("error"): + detail = f"{detail}: {body['error']}" + failed.append((label, detail)) + return {"pushed": pushed, "failed": failed, "peers": len(targets)} diff --git a/ainode/cli/doctor.py b/ainode/cli/doctor.py index 16aad2eb..e8e2e87f 100644 --- a/ainode/cli/doctor.py +++ b/ainode/cli/doctor.py @@ -1259,6 +1259,206 @@ def check_auth(config: NodeConfig, home, peers_seen: int = 0) -> list[Check]: return [Check("auth.state", OK, detail, data=data)] +def users_file_records(raw) -> list[dict]: + """The account records out of whatever ``users.json`` holds. + + The file belongs to the account store (``ainode/auth/accounts.py``), and the + doctor reads it rather than importing the store so that a node whose store + cannot be imported still gets an answer. Tolerant about the shape for the same + reason: a list under ``users``, a map keyed by name, or a bare list all read as + the same set of accounts. Never returns a password hash to a caller: only the + name and the role are kept, because this check counts admins and nothing else. + """ + if isinstance(raw, dict): + holder = raw.get("users", raw.get("accounts")) + else: + holder = raw + rows: list = [] + if isinstance(holder, dict): + for name, value in holder.items(): + row = dict(value) if isinstance(value, dict) else {} + row.setdefault("name", name) + rows.append(row) + elif isinstance(holder, list): + for value in holder: + rows.append(dict(value) if isinstance(value, dict) else {"name": str(value)}) + return [{"name": str(row.get("name") or ""), + "role": str(row.get("role") or "member"), + "disabled": row.get("disabled") is True or row.get("enabled") is False} + for row in rows] + + +def _auth_required(home) -> Optional[bool]: + """Does this node require an API key? None when ``auth.json`` cannot be read. + + ``check_auth`` is the check that judges that file; this is the one fact the + Login check needs from it, because whether a missing account is a problem + depends entirely on whether anything is refused without one. + """ + path = Path(home) / "auth.json" + if not _exists(path): + return False + try: + raw = json.loads(path.read_text()) + except (OSError, ValueError): + return None + return bool(raw.get("enabled")) if isinstance(raw, dict) else None + + +def check_login(config: NodeConfig, home, peers_seen: int = 0) -> list[Check]: + """Can a human sign in to the dashboard, and is this node's list the fleet's? + + Three facts, and each one is a different way the login (#261) goes wrong: + + ``login.state``. A node with auth ON and no admin account has a dashboard + nobody can open without pasting an API key, which is the state the login exists + to replace: a FAIL, because the node cannot do its job for a human. With auth + OFF nothing is refused without a credential, so having no account is not a + finding, and accounts that exist are reported as the fact they are. + + ``login.store``. ``users.json`` holds password hashes, so it is 0600 like + ``auth.json`` and the secrets store, and ``--fix`` tightens it. + + ``login.sync``. Accounts come from the master (``auth/replication.py``), so a + WORKER holding a list older than the master's is about to surprise somebody: + the password they just set on the master does not work here yet. The comparison + is between two of the MASTER's own stamps, the one this node recorded when it + last imported and the one the master reports now, so it cannot drift with how + either side hashes a list. + """ + from ainode.auth.replication import local_role, read_sync_state + + path = Path(home) / "users.json" + auth_on = _auth_required(home) + role = local_role(config) + checks: list[Check] = [] + + records: Optional[list[dict]] = None + unreadable = "" + if _exists(path): + try: + records = users_file_records(json.loads(path.read_text())) + except (OSError, ValueError) as exc: + unreadable = str(exc) + else: + records = [] + + admins = [r for r in (records or []) if r["role"] == "admin" and not r["disabled"]] + data = {"path": str(path), "exists": _exists(path), "auth_enabled": auth_on, + "users": None if records is None else len(records), + "admins": None if records is None else len(admins), + "role": role, "peers_seen": int(peers_seen or 0)} + + if unreadable: + checks.append(Check("login.state", WARN, + f"cannot read {path}: {unreadable}; whether anybody can " + "sign in to this dashboard is unknown", + fix="ainode auth user list", data=data)) + elif auth_on is None: + checks.append(Check("login.state", WARN, + "cannot tell whether this node requires a credential, so " + "whether it needs a dashboard account is unknown", + fix="ainode auth status", data=data)) + elif not auth_on: + detail = ("no credential is required on this node, so the dashboard opens " + "without a login") + if records: + detail += f" ({len(records)} account(s) on the node, unused until auth is on)" + checks.append(Check("login.state", OK, detail, data=data)) + elif not records: + checks.append(Check("login.state", FAIL, + "a key is required and there is no dashboard account, so " + "the UI can only be opened by pasting an API key", + fix="ainode auth user add --admin (it prompts for " + "a password; add --password-stdin with no TTY)", + data=data)) + elif not admins: + checks.append(Check("login.state", WARN, + f"{len(records)} account(s) and no enabled admin, so " + "nobody can manage users or sessions from the dashboard", + fix="ainode auth user add --admin", + data=data)) + else: + checks.append(Check("login.state", OK, + f"{len(records)} dashboard account(s), {len(admins)} of " + "them admin, and a credential is required", + data=data)) + + if _exists(path): + try: + mode = stat.S_IMODE(path.stat().st_mode) + except OSError as exc: + checks.append(Check("login.store", WARN, f"cannot stat {path}: {exc}", + data={"path": str(path)})) + else: + store_data = {"path": str(path), "mode": oct(mode), + "fix_action": "chmod600"} + if mode == 0o600: + checks.append(Check("login.store", OK, f"{path} is mode 0600", + data=store_data)) + else: + checks.append(Check("login.store", WARN, + f"{path} is mode {oct(mode)} and holds password " + "hashes, so it is readable beyond its owner", + fix="ainode doctor --fix chmods it to 0600", + data=store_data)) + + if role == "worker": + checks += _check_account_sync(config, read_sync_state(home)) + return checks + + +def _check_account_sync(config: NodeConfig, recorded: dict) -> list[Check]: + """Is this worker's account list the master's current one? + + Split out because it is the one part of the Login check that leaves the box: + it asks the master for the stamp of its list, with the fleet key, the way every + other node-to-node read in the product does. + """ + from ainode.auth.replication import EXPORT_PATH, master_target + + base = master_target(config=config) + data = {"role": "worker", "master": base, + "recorded_stamp": str(recorded.get("stamp") or ""), + "recorded_at": str(recorded.get("at") or "")} + if not base: + return [Check("login.sync", WARN, + "this node is a cluster worker and nothing names its master, so " + "the master's accounts cannot reach it", + fix="ainode cluster token on the master, then the ainode join " + "line it prints", + data=data)] + + headers = fleet_key_headers(getattr(config, "cluster_secret", "")) + payload = http_json(f"{base}{EXPORT_PATH}", headers=headers) + if not isinstance(payload, dict): + return [Check("login.sync", WARN, + f"the master at {base} did not answer for its account list, so " + "this node cannot tell whether its logins are current", + fix="check the master's service, and that both nodes share one " + "cluster_secret", + data=data)] + + theirs = str(payload.get("stamp") or "") + data["master_stamp"] = theirs + if theirs and data["recorded_stamp"] == theirs: + return [Check("login.sync", OK, + f"accounts match the master at {base}", data=data)] + if not data["recorded_stamp"]: + return [Check("login.sync", WARN, + f"this node has never imported the master's accounts ({base} " + "reports a list), so a login made there does not work here yet", + fix="restart the service to pull now, or wait for the 5 minute " + "pull", + data=data)] + return [Check("login.sync", WARN, + f"this node's account list is older than the master's at {base} " + f"(imported {data['recorded_at'] or 'at an unknown time'}), so a " + "password changed there does not work here yet", + fix="restart the service to pull now, or wait for the 5 minute pull", + data=data)] + + def check_tls(config: NodeConfig) -> list[Check]: """Does anything here speak HTTPS, and is the certificate still good? @@ -1433,6 +1633,7 @@ def run_checks(home=None, config_path=None) -> list[Check]: checks += check_secrets(home) checks += check_cluster_secret(config, seen) checks += check_auth(config, home, seen) + checks += check_login(config, home, seen) checks += check_tls(config) checks += check_rate_limit(config) checks += check_hf_token(config, home) diff --git a/ainode/cli/main.py b/ainode/cli/main.py index 2484b7eb..c3dc31ba 100644 --- a/ainode/cli/main.py +++ b/ainode/cli/main.py @@ -874,6 +874,483 @@ def cmd_auth_key(args): console.print(" Usage: ainode auth key {create [--name NAME]|list|revoke }") +# ============================================================================= +# Dashboard accounts: `ainode auth user ...` and `ainode auth session ...` +# ============================================================================= +# +# The dashboard login (#261) is a second credential on the node: a named account +# with a password, in `/users.json`. These commands are how an +# operator creates the first one, because there is no sign-up page and there must +# not be: a route that mints the first admin over HTTP is a route anybody on the +# LAN can call before the operator does. +# +# The store is written HERE, on the box, and the running server picks the file up +# through `reload_if_changed` exactly as `ainode auth enable` is live. On the +# cluster's MASTER the change is then pushed to every peer +# (ainode/auth/replication.py); on a worker it is written and the operator is told +# it will be overwritten, because the master is the authority for accounts. +# +# Sessions are per node and are never replicated, so `ainode auth session ...` +# answers about this node only. That is the honest shape: a session is one +# browser's cookie against one node, and copying them would hand every node in the +# fleet a credential it never issued. + +#: Same promise AUTH_LIVE_NOTE makes about auth.json, for the account store. +USERS_LIVE_NOTE = (" In effect now: the running node re-reads users.json, " + "no restart needed.") + +#: Printed on a node that is not the account authority, once, and then the change +#: is made anyway: refusing would leave an operator who typed the command on the +#: wrong box with no accounts and no explanation. +NOT_THE_MASTER_NOTE = ( + " [yellow]Accounts are managed on the master[/yellow], and this node will be " + "overwritten by the next sync from it. Run the same command there to keep it.") + + +def _users_store(): + """The account store on this node, or None with a printed reason. + + One guarded import for the whole CLI (``auth/replication.py::open_store``), so + a release without the account store says so instead of raising an ImportError + at the operator. + """ + from ainode.auth.replication import open_store + + store = open_store() + if store is None: + console.print(" [yellow]This build has no account store[/yellow], so there " + "are no dashboard logins to manage on it.") + console.print(" API keys still work: ainode auth key create --name ") + return store + + +def _save_store(store) -> None: + """Persist the store, if persisting is a separate step on it. + + The store's mutators are modelled on ``AuthConfig``'s, which save themselves, + so this is normally a second write of identical bytes (temp-then-replace, so + it costs nothing and cannot half-apply). It is here because the alternative + failure is an account an operator was told about that is not on disk. + """ + save = getattr(store, "save", None) + if not callable(save): + return + try: + save() + except Exception as exc: # pragma: no cover - a store that cannot write + console.print(f" [red]Could not write the account store: {exc}[/red]") + raise SystemExit(1) + + +def _read_new_password(args, label: str = "Password") -> str: + """The new password: one read of stdin, or two prompts that must agree. + + ``--password-stdin`` exists because ``getpass`` needs a TTY and the installer's + host wrapper runs ``docker exec -it``: interactive from a terminal, but not + from a script, a unit file or ``ssh host ainode ...``. Only the newline the + shell added is stripped, so a password may contain spaces. + """ + from ainode.auth.replication import min_password_length + + minimum = min_password_length() + if getattr(args, "password_stdin", False): + password = sys.stdin.read().rstrip("\r\n") + else: + import getpass + + try: + password = getpass.getpass(f" {label}: ") + again = getpass.getpass(" Again: ") + except (EOFError, OSError): + console.print(" [red]No terminal to ask for a password on.[/red]") + console.print(" Pipe it instead: echo 'the password' | ainode auth user " + "add --password-stdin") + raise SystemExit(2) + if password != again: + console.print(" [red]The two passwords do not match.[/red] " + "Nothing was changed.") + raise SystemExit(2) + if len(password) < minimum: + console.print(f" [red]That password is too short.[/red] It needs at least " + f"{minimum} characters.") + raise SystemExit(2) + return password + + +def _account_rows(store) -> list: + """``list_users()`` as a list of dicts, whatever it hands back.""" + try: + rows = store.list_users() + except Exception as exc: + console.print(f" [red]Could not read the accounts: {exc}[/red]") + raise SystemExit(1) + out = [] + for row in rows or []: + if isinstance(row, dict): + out.append(row) + else: # a store that lists names only + out.append({"name": str(row)}) + return out + + +def _find_account(rows: list, name: str): + for row in rows: + if str(row.get("name") or "") == name: + return row + return None + + +def _account_enabled(row: dict) -> bool: + """Is this account allowed to sign in? Both spellings are read.""" + if row.get("disabled") is True: + return False + return row.get("enabled") is not False + + +def _admin_count(store, rows: list) -> int: + try: + return int(store.admin_count()) + except Exception: + return sum(1 for row in rows if str(row.get("role") or "") == "admin") + + +def _refuse_if_last_admin(store, rows: list, name: str, action: str) -> None: + """Refuse an action that would leave the dashboard with no admin. + + With auth on and no admin account, the dashboard can only be opened with an + API key, which is the state ``ainode doctor``'s Login check FAILs on. Refusing + here is cheaper than explaining it there. + """ + row = _find_account(rows, name) + if row is None or str(row.get("role") or "") != "admin": + return + if _admin_count(store, rows) > 1: + return + console.print(f" [red]{name} is the only admin account[/red], so it cannot be " + f"{action}.") + console.print(" Add another admin first: ainode auth user add --admin") + raise SystemExit(2) + + +def _set_account_enabled(store, name: str, enabled: bool) -> bool: + """Disable or enable one account, however this store spells it. + + The account contract names the two ROUTES (``/disable`` and ``/enable``) and + leaves the store's own method name open, so the three spellings a store of + this shape would use are tried in order. Raises ``NotImplementedError`` when + it has none: the CLI says so and points at the route, rather than inventing a + field name inside somebody else's file. + """ + setter = getattr(store, "set_enabled", None) + if callable(setter): + return setter(name, enabled) is not False + setter = getattr(store, "set_disabled", None) + if callable(setter): + return setter(name, not enabled) is not False + setter = getattr(store, "enable_user" if enabled else "disable_user", None) + if callable(setter): + return setter(name) is not False + raise NotImplementedError("this account store cannot disable an account") + + +def _session_rows(store, name: str) -> list: + """``sessions_for(name)`` as a list of dicts, whatever it hands back.""" + try: + rows = store.sessions_for(name) + except Exception as exc: + console.print(f" [yellow]Could not read {name}'s sessions: {exc}[/yellow]") + return [] + out = [] + for row in rows or []: + out.append(dict(row) if isinstance(row, dict) else {"id": str(row)}) + return out + + +def _account_counts(store) -> dict: + """Users, admins and sessions on this node, for ``ainode auth status``.""" + rows = _account_rows(store) + sessions = 0 + for row in rows: + sessions += len(_session_rows(store, str(row.get("name") or ""))) + return {"users": len(rows), "admins": _admin_count(store, rows), + "sessions": sessions} + + +def _replicate_after_change(config, store) -> None: + """Push a just-written account change to the peers, or say why it was not. + + Three answers, and each of them is a line an operator needs: a solo node has + nobody to tell, a worker has just written a change the master will overwrite, + and a master pushes now instead of leaving the peers to catch up on the + server's own 60 second retry. + """ + from ainode.auth.replication import ( + cluster_info, + local_role, + replicate_from_cli, + ) + + info = cluster_info(config) + role = local_role(config, info=info) + if role == "solo": + return + if role == "worker": + console.print(NOT_THE_MASTER_NOTE) + return + + try: + users = store.export_users() + except Exception as exc: + console.print(f" [yellow]Could not read the accounts to replicate: {exc}" + f"[/yellow]") + return + result = replicate_from_cli(config, users, info=info) + pushed, failed = result.get("pushed") or [], result.get("failed") or [] + if result.get("reason"): + console.print(f" [yellow]Not replicated: {result['reason']}[/yellow]") + return + if pushed: + console.print(f" Replicated to {len(pushed)} peer(s): {', '.join(pushed)}") + for label, detail in failed: + console.print(f" [yellow]{label} did not take the update ({detail})[/yellow]; " + "this node retries it every 60s while it is behind.") + if not pushed and not failed: + console.print(" No peers to replicate to yet.") + + +def cmd_auth_user(args): + """``ainode auth user {add|list|remove|passwd|disable|enable}``.""" + action = getattr(args, "user_action", None) + if action is None: + console.print(" Usage: ainode auth user {add [--admin] " + "[--password-stdin]|list|remove |passwd |" + "disable |enable }") + return + + store = _users_store() + if store is None: + raise SystemExit(2) + config = NodeConfig.load() + # Account names are lowercase by contract ([a-z0-9._-]), so what an operator + # typed with a capital is folded rather than refused. The name is echoed back + # in every line below, so the fold is visible. + name = str(getattr(args, "name", "") or "").strip().lower() + + if action == "list": + rows = _account_rows(store) + if not rows: + console.print(" No dashboard accounts on this node.") + console.print(" Add the first one: ainode auth user add --admin") + console.print(" Made in Texas") + return + table = Table(show_header=True, header_style="bold", box=None, pad_edge=False) + table.add_column("Name") + table.add_column("Role") + table.add_column("State") + table.add_column("Sessions") + for row in sorted(rows, key=lambda r: str(r.get("name") or "")): + account = str(row.get("name") or "") + table.add_row( + account, + str(row.get("role") or "member"), + "enabled" if _account_enabled(row) else "[yellow]disabled[/yellow]", + str(len(_session_rows(store, account))), + ) + console.print(table) + console.print() + console.print(f" {_admin_count(store, rows)} admin(s) of {len(rows)} " + "account(s). Sessions are per node and are not replicated.") + console.print(" Made in Texas") + return + + if not name: + console.print(f" [red]Which account?[/red] ainode auth user {action} ") + raise SystemExit(2) + + if action == "add": + role = "admin" if getattr(args, "admin", False) else "member" + password = _read_new_password(args) + try: + store.add_user(name, password, role) + except ValueError as exc: + console.print(f" [red]{exc}[/red]") + raise SystemExit(2) + except Exception as exc: + console.print(f" [red]Could not add {name}: {exc}[/red]") + raise SystemExit(1) + _save_store(store) + console.print(f" [green]Account {name} added[/green] as {role}.") + console.print(" Sign in at the dashboard with that name and password.") + console.print(USERS_LIVE_NOTE) + _replicate_after_change(config, store) + console.print(" Made in Texas") + return + + rows = _account_rows(store) + if _find_account(rows, name) is None: + console.print(f" [yellow]No account named '{name}' on this node.[/yellow]") + console.print(" List them: ainode auth user list") + raise SystemExit(2) + + if action == "remove": + _refuse_if_last_admin(store, rows, name, "removed") + try: + removed = store.remove_user(name) + except Exception as exc: + console.print(f" [red]Could not remove {name}: {exc}[/red]") + raise SystemExit(1) + if removed is False: + console.print(f" [yellow]No account named '{name}' on this node.[/yellow]") + raise SystemExit(2) + _save_store(store) + console.print(f" [green]Account {name} removed.[/green] Its sessions on this " + "node are gone with it.") + console.print(USERS_LIVE_NOTE) + _replicate_after_change(config, store) + console.print(" Made in Texas") + return + + if action == "passwd": + password = _read_new_password(args, label=f"New password for {name}") + try: + store.set_password(name, password) + except ValueError as exc: + console.print(f" [red]{exc}[/red]") + raise SystemExit(2) + except Exception as exc: + console.print(f" [red]Could not set {name}'s password: {exc}[/red]") + raise SystemExit(1) + _save_store(store) + console.print(f" [green]Password changed for {name}.[/green]") + console.print(" Existing sessions are untouched: sign them out with " + f"ainode auth session clear --user {name}") + console.print(USERS_LIVE_NOTE) + _replicate_after_change(config, store) + console.print(" Made in Texas") + return + + if action in ("disable", "enable"): + enabled = action == "enable" + if not enabled: + _refuse_if_last_admin(store, rows, name, "disabled") + try: + _set_account_enabled(store, name, enabled) + except NotImplementedError: + console.print(" [yellow]This account store has no disable of its own." + "[/yellow] Use the API route instead, on this node's own " + "port, with an operator key:") + console.print(f" POST /api/auth/users/{name}/{action}") + raise SystemExit(2) + except Exception as exc: + console.print(f" [red]Could not {action} {name}: {exc}[/red]") + raise SystemExit(1) + _save_store(store) + if enabled: + console.print(f" [green]Account {name} enabled.[/green]") + else: + console.print(f" [yellow]Account {name} disabled.[/yellow] It cannot " + "sign in, and its sessions are no longer accepted.") + console.print(USERS_LIVE_NOTE) + _replicate_after_change(config, store) + console.print(" Made in Texas") + return + + console.print(f" [red]Unknown account command '{action}'.[/red]") + raise SystemExit(2) + + +def cmd_auth_session(args): + """``ainode auth session {list|revoke|clear}``: this node's sign-ins. + + Per node on purpose (see the block comment above): a session is one browser's + cookie against one node, so there is nothing here to fan out and nothing to + replicate. + """ + action = getattr(args, "session_action", None) + if action is None: + console.print(" Usage: ainode auth session {list [--user NAME]|" + "revoke |clear [--user NAME]}") + return + + store = _users_store() + if store is None: + raise SystemExit(2) + only = str(getattr(args, "user", "") or "").strip().lower() + rows = _account_rows(store) + if only and _find_account(rows, only) is None: + console.print(f" [yellow]No account named '{only}' on this node.[/yellow]") + raise SystemExit(2) + names = [only] if only else [str(r.get("name") or "") for r in rows] + + if action == "list": + table = Table(show_header=True, header_style="bold", box=None, pad_edge=False) + table.add_column("Session") + table.add_column("User") + table.add_column("Created") + table.add_column("Last seen") + total = 0 + for account in sorted(names): + for session in _session_rows(store, account): + total += 1 + table.add_row( + str(session.get("id") or ""), + account, + str(session.get("created_at") or "[dim]unknown[/dim]"), + str(session.get("last_seen") + or session.get("last_used") or "[dim]unknown[/dim]"), + ) + if not total: + console.print(" No sessions on this node" + + (f" for {only}." if only else ".")) + console.print(" Made in Texas") + return + console.print(table) + console.print() + console.print(f" {total} session(s) on this node. Sessions are per node: " + "signing out here does not sign out anywhere else.") + console.print(" Made in Texas") + return + + if action == "revoke": + session_id = str(getattr(args, "session_id", "") or "").strip() + if not session_id: + console.print(" [red]Which session?[/red] ainode auth session revoke ") + raise SystemExit(2) + try: + gone = store.revoke_session(session_id) + except Exception as exc: + console.print(f" [red]Could not revoke that session: {exc}[/red]") + raise SystemExit(1) + if gone is False: + console.print(f" [yellow]No session '{session_id}' on this node." + "[/yellow] List them: ainode auth session list") + raise SystemExit(2) + console.print(f" [green]Session {session_id} revoked.[/green]") + console.print(" Made in Texas") + return + + if action == "clear": + revoked = 0 + for account in names: + for session in _session_rows(store, account): + session_id = str(session.get("id") or "") + if not session_id: + continue + try: + if store.revoke_session(session_id, user=account) is not False: + revoked += 1 + except Exception as exc: + console.print(f" [yellow]{account}: {exc}[/yellow]") + who = f"{only}" if only else "everybody" + console.print(f" [green]Signed out {who}[/green]: {revoked} session(s) " + "revoked on this node.") + console.print(" Made in Texas") + return + + console.print(f" [red]Unknown session command '{action}'.[/red]") + raise SystemExit(2) + + def cmd_auth(args): """Manage API key authentication.""" from ainode.auth.middleware import AuthConfig @@ -884,6 +1361,12 @@ def cmd_auth(args): if action == "key": return cmd_auth_key(args) + if action == "user": + return cmd_auth_user(args) + + if action == "session": + return cmd_auth_session(args) + if action == "enable": entry = auth_cfg.enable(getattr(args, "name", "") or "first key") console.print(" [green]Auth enabled.[/green]") @@ -916,6 +1399,27 @@ def cmd_auth(args): state = "[green]enabled[/green]" if auth_cfg.enabled else "[dim]disabled[/dim]" console.print(f" Auth: {state}") console.print(f" Keys: {len(auth_cfg.api_keys)}") + # The accounts half of the same question. Auth on with no admin account is + # a dashboard only a key can open, which is what ainode doctor FAILs on, so + # the number belongs beside the key count rather than behind another + # command. + from ainode.auth.replication import open_store + + store = open_store() + if store is not None: + counts = _account_counts(store) + console.print(f" Users: {counts['users']} " + f"({counts['admins']} admin)") + console.print(f" Sessions: {counts['sessions']} (this node only, never " + "replicated)") + if auth_cfg.enabled and not counts["users"]: + console.print(" [yellow]No dashboard account[/yellow], so the UI can " + "only be opened with an API key.") + console.print(" Fix: ainode auth user add --admin") + elif auth_cfg.enabled and not counts["admins"]: + console.print(" [yellow]No admin account[/yellow]: nobody can manage " + "users from the dashboard.") + console.print(" Fix: ainode auth user add --admin") if not auth_cfg.enabled: console.print(" API open, no key set" if not auth_cfg.api_keys else " API open, key set but not required") @@ -943,8 +1447,14 @@ def cmd_auth(args): console.print(" Made in Texas") else: - console.print(" Usage: ainode auth {enable|disable|status|key|new-key}") + console.print(" Usage: ainode auth {enable|disable|status|key|user|session" + "|new-key}") console.print(" ainode auth key {create [--name NAME]|list|revoke }") + console.print(" ainode auth user {add [--admin] " + "[--password-stdin]|list|remove |passwd |" + "disable |enable }") + console.print(" ainode auth session {list [--user NAME]|revoke |" + "clear [--user NAME]}") def cmd_prune_images(args): @@ -1625,6 +2135,61 @@ def main(): auth_key_sub.add_parser("list", help="The keys on this node, by id and name") auth_key_revoke = auth_key_sub.add_parser("revoke", help="Revoke a key by id") auth_key_revoke.add_argument("key_id", help="The key id from: ainode auth key list") + + # `auth user ...` is the DASHBOARD login (#261): a name and a password, which + # is a different credential from an API key and managed on the cluster's + # master. --password-stdin is the answer for anything with no terminal: getpass + # needs a TTY and the installer's wrapper runs `docker exec -it`, so the prompt + # works from a shell and not from a script, a unit or `ssh host ainode ...`. + auth_user = auth_sub.add_parser( + "user", help="Dashboard accounts: add, list, remove, change a password") + auth_user_sub = auth_user.add_subparsers(dest="user_action") + auth_user_add = auth_user_sub.add_parser( + "add", help="Create a dashboard account (prompts for the password twice)") + auth_user_add.add_argument("name", help="Account name (a-z, 0-9, dot, dash, " + "underscore)") + auth_user_add.add_argument( + "--admin", action="store_true", + help="Make it an admin: it may manage users and sessions") + auth_user_add.add_argument( + "--password-stdin", action="store_true", dest="password_stdin", + help="Read the password from stdin instead of prompting. Use this with no " + "TTY: echo 'the password' | ainode auth user add jason --admin " + "--password-stdin") + auth_user_sub.add_parser("list", help="The accounts on this node and their roles") + auth_user_remove = auth_user_sub.add_parser( + "remove", help="Delete an account (refused for the last admin)") + auth_user_remove.add_argument("name", help="The account to delete") + auth_user_passwd = auth_user_sub.add_parser( + "passwd", help="Change an account's password") + auth_user_passwd.add_argument("name", help="The account to change") + auth_user_passwd.add_argument( + "--password-stdin", action="store_true", dest="password_stdin", + help="Read the new password from stdin instead of prompting (no TTY needed)") + auth_user_disable = auth_user_sub.add_parser( + "disable", help="Stop an account signing in, keeping it (refused for the " + "last admin)") + auth_user_disable.add_argument("name", help="The account to disable") + auth_user_enable = auth_user_sub.add_parser( + "enable", help="Let an account sign in again") + auth_user_enable.add_argument("name", help="The account to enable") + + # `auth session ...` is this node's sign-ins. Never a fan-out: a session is one + # browser's cookie against one node, so there is nothing to replicate. + auth_session = auth_sub.add_parser( + "session", help="Sign-ins on THIS node: list, revoke one, sign everybody out") + auth_session_sub = auth_session.add_subparsers(dest="session_action") + auth_session_list = auth_session_sub.add_parser( + "list", help="The sessions on this node") + auth_session_list.add_argument("--user", default="", help="Only this account's") + auth_session_revoke = auth_session_sub.add_parser( + "revoke", help="Revoke one session by id") + auth_session_revoke.add_argument( + "session_id", help="The session id from: ainode auth session list") + auth_session_clear = auth_session_sub.add_parser( + "clear", help="Sign everybody out of this node") + auth_session_clear.add_argument( + "--user", default="", help="Sign out only this account") auth_parser.set_defaults(func=cmd_auth) # prune-images: reclaim the images an update replaced (#184) diff --git a/scripts/install.sh b/scripts/install.sh index 68e3cdbd..61d222d9 100755 --- a/scripts/install.sh +++ b/scripts/install.sh @@ -1346,6 +1346,19 @@ else ACCESS_LINE="API open, no key set. Require one in Config > API access." fi +# The two commands a human needs to be able to open the dashboard with a name and +# a password instead of pasting an API key (#261). Printed only when this node +# requires a credential, because with auth off the dashboard opens without a login +# and these lines would be advice about a problem nobody has. There is no route +# that mints the first admin: the first account is made on the box, by the operator. +print_login_lines() { + [ -n "$INSTALL_API_KEY" ] || auth_enabled_on_disk || return 0 + printf ' Login: ainode auth enable\n' + printf ' ainode auth user add --admin\n' + printf ' [the account is how a human signs in instead of pasting\n' + printf ' the key; --password-stdin where there is no terminal]\n' +} + # -- Banner ----------------------------------------------------------------- if [ "$DRY_RUN" = "true" ]; then printf '\n' @@ -1361,6 +1374,7 @@ if [ "$DRY_RUN" = "true" ]; then log " the daily tailnet certificate renewal (not installed)" log " Access: $ACCESS_LINE" print_api_key_box + print_login_lines printf '\n' exit 0 fi @@ -1377,6 +1391,7 @@ printf ' Status: ainode status\n' printf ' Logs: ainode logs -f\n' printf ' Update: ainode update\n' print_api_key_box +print_login_lines printf '\n' printf ' Made in Texas\n' printf '\n' diff --git a/tests/test_auth_replication.py b/tests/test_auth_replication.py new file mode 100644 index 00000000..4091f136 --- /dev/null +++ b/tests/test_auth_replication.py @@ -0,0 +1,809 @@ +"""Accounts replicate from the master, sessions never do (#261). + +A cluster cannot hold one account list per node: an operator who adds a login on +the master and then opens the dashboard of whichever node their bookmark points at +would be told their password is wrong. So ``ainode/auth/replication.py`` makes the +master the one authority, and this file pins the five things that has to mean: + +1. **The master pushes to its peers, with the FLEET KEY.** The export carries + password hashes, so the route takes that key and nothing else; a push with no + ``Authorization`` header would be refused by every node in a fleet with auth on, + which is the bug ``tests/test_fleet_auth.py`` exists to prevent in general. +2. **A peer that refused is retried.** Not queued: the master remembers the stamp + each peer accepted, so a peer that failed simply still disagrees and the next + tick pushes to it again. The tick is 60 seconds while anything is behind. +3. **A peer that agrees is not pushed to again**, so the idle cost of this is one + in-memory comparison rather than a request per minute per node. +4. **A worker pulls the master's list and imports it**, records the master's own + stamp so ``ainode doctor`` can compare like with like, and an unreachable master + is one WARNING and a node that keeps working, not a crash and not a log line + every tick. +5. **Sessions are never on the wire.** The push body carries ``users`` and nothing + else, and a pull does not touch the sessions this node issued itself. + +The account store lands on its own branch (``ainode/auth/accounts.py``), so these +tests drive the contract with :class:`StubStore` and replace the one guarded import +the package makes (``replication.users_store_class``). +""" + +from __future__ import annotations + +import asyncio +import logging +import time + +import pytest +import pytest_asyncio +from aiohttp import web +from aiohttp.test_utils import TestServer + +from ainode.auth import replication as rep +from ainode.auth.fleet import fleet_key +from ainode.core.config import NodeConfig +from ainode.discovery.broadcast import NodeStatus +from ainode.discovery.cluster import ClusterNode, ClusterState + +SECRET = "0123456789abcdef0123456789abcdef" +OTHER_SECRET = "fedcba9876543210fedcba9876543210" + +USERS = [ + {"name": "jason", "role": "admin", "password_hash": "hash-1"}, + {"name": "ops", "role": "member", "password_hash": "hash-2"}, +] + + +# ============================================================================= +# The store's contract, in memory +# ============================================================================= + +class StubStore: + """``UsersStore`` as this module uses it. The real one is another branch's.""" + + MIN_PASSWORD_LENGTH = 8 + + def __init__(self, users=None, sessions=None): + self.users = [dict(u) for u in (users or [])] + self.sessions = dict(sessions or {}) + self.reloads = 0 + self.imports: list[list] = [] + + # -- what replication calls ----------------------------------------- + def load(self): + return self + + def reload_if_changed(self) -> bool: + self.reloads += 1 + return False + + def export_users(self) -> list: + return [dict(u) for u in self.users] + + def import_users(self, users) -> bool: + self.imports.append([dict(u) for u in users]) + changed = [dict(u) for u in users] != self.users + self.users = [dict(u) for u in users] + return changed + + def sessions_for(self, name) -> list: + return list(self.sessions.get(name, [])) + + +@pytest.fixture(autouse=True) +def home(tmp_path, monkeypatch): + """Keep users-sync.json out of the operator's own ~/.ainode.""" + monkeypatch.setenv("AINODE_HOME", str(tmp_path)) + return tmp_path + + +@pytest.fixture(autouse=True) +def loopback_peers(monkeypatch): + """Reach a test server over loopback. + + ``api/server_routes.peer_host`` deliberately refuses to publish a 127.x + address (it is the one answer guaranteed wrong on another machine), so the + seam it hides behind is what a test replaces. + """ + monkeypatch.setattr(rep, "peer_address", lambda node: "127.0.0.1") + + +def _node(node_id, web_port=3000, role="auto"): + return ClusterNode(node_id=node_id, node_name=node_id, gpu_name="NVIDIA GB10", + gpu_memory_gb=128.0, unified_memory=True, model="", + status=NodeStatus.ONLINE, api_port=8000, web_port=web_port, + last_seen=0.0, role=role) + + +class FakePeer: + """A node that answers the two replication routes and records what it got.""" + + def __init__(self, users=None, stamp="master-stamp-1"): + self.app = web.Application() + self.app.router.add_post(rep.SYNC_PATH, self._sync) + self.app.router.add_get(rep.EXPORT_PATH, self._export) + self.posts: list[dict] = [] + self.gets: list[dict] = [] + self.status = 200 + self.users = [dict(u) for u in (users or USERS)] + self.stamp = stamp + + async def _sync(self, request): + body = await request.json() + self.posts.append({"body": body, + "authorization": request.headers.get("Authorization", "")}) + if self.status != 200: + return web.json_response({"error": "nope"}, status=self.status) + self.users = list(body.get("users") or []) + return web.json_response({"changed": True, "count": len(self.users)}) + + async def _export(self, request): + self.gets.append({"authorization": request.headers.get("Authorization", "")}) + if self.status != 200: + return web.json_response({"error": "nope"}, status=self.status) + return web.json_response({"users": self.users, "stamp": self.stamp}) + + +@pytest_asyncio.fixture +async def peer(): + served = FakePeer() + server = TestServer(served.app) + await server.start_server() + served.port = server.port + try: + yield served + finally: + await server.close() + + +@pytest_asyncio.fixture +async def session(): + import aiohttp + + async with aiohttp.ClientSession() as client: + yield client + + +def _master_app(session, peer_port, store, secret=SECRET): + """A master's app as replication reads it: a plain mapping is all it needs.""" + cluster = ClusterState() + cluster.add_node(_node("spark1", role="master")) + cluster.add_node(_node("spark2", web_port=peer_port)) + return { + "config": NodeConfig(node_id="spark1", node_name="spark1", + cluster_secret=secret, cluster_role="master"), + "cluster_state": cluster, + "client_session": session, + "users_store": store, + } + + +def _worker_app(session, master_port, store, secret=SECRET): + return { + "config": NodeConfig(node_id="spark3", node_name="spark3", + cluster_secret=secret, cluster_role="worker", + distributed_mode="member", + master_address=f"127.0.0.1:{master_port}"), + "client_session": session, + "users_store": store, + } + + +# ============================================================================= +# 1. The master pushes, with the fleet key +# ============================================================================= + +@pytest.mark.asyncio +async def test_the_master_pushes_its_accounts_to_every_peer_with_the_fleet_key( + peer, session): + store = StubStore(USERS) + app = _master_app(session, peer.port, store) + + result = await rep.AccountReplicator(app).broadcast() + + assert result["pushed"] == ["spark2"] + assert result["failed"] == [] + assert len(peer.posts) == 1 + assert peer.posts[0]["authorization"] == f"Bearer {fleet_key(SECRET)}" + assert peer.posts[0]["body"]["users"] == USERS + + +@pytest.mark.asyncio +async def test_the_key_on_the_push_follows_this_nodes_own_secret(peer, session): + """Rotation is a config edit, not a fleet restart: the header is derived per + request from whatever ``cluster_secret`` says now.""" + app = _master_app(session, peer.port, StubStore(USERS), secret=OTHER_SECRET) + + await rep.AccountReplicator(app).broadcast() + + assert peer.posts[0]["authorization"] == f"Bearer {fleet_key(OTHER_SECRET)}" + assert peer.posts[0]["authorization"] != f"Bearer {fleet_key(SECRET)}" + + +@pytest.mark.asyncio +async def test_the_push_carries_users_and_nothing_else(peer, session): + """Sessions are per node: a body that carried them would hand every node in + the fleet a credential it never issued.""" + store = StubStore(USERS, sessions={"jason": [{"id": "sess-1"}]}) + await rep.AccountReplicator(_master_app(session, peer.port, store)).broadcast() + + assert list(peer.posts[0]["body"]) == ["users"] + + +@pytest.mark.asyncio +async def test_a_peer_that_already_agrees_is_not_pushed_to_again(peer, session): + store = StubStore(USERS) + replicator = rep.AccountReplicator(_master_app(session, peer.port, store)) + + first = await replicator.broadcast() + second = await replicator.broadcast() + + assert first["pushed"] == ["spark2"] + assert second["pushed"] == [] + assert len(peer.posts) == 1, "an unchanged list must not be pushed every tick" + + +@pytest.mark.asyncio +async def test_a_changed_list_is_pushed_again(peer, session): + store = StubStore(USERS) + replicator = rep.AccountReplicator(_master_app(session, peer.port, store)) + await replicator.broadcast() + + store.users.append({"name": "second", "role": "member", "password_hash": "h3"}) + result = await replicator.broadcast() + + assert result["pushed"] == ["spark2"] + assert len(peer.posts) == 2 + assert [u["name"] for u in peer.posts[1]["body"]["users"]] == \ + ["jason", "ops", "second"] + + +@pytest.mark.asyncio +async def test_the_master_reads_the_store_off_disk_before_it_pushes(peer, session): + """``ainode auth user add`` writes the file on the box, so the push has to ask + the store whether the file moved rather than trusting what it holds.""" + store = StubStore(USERS) + await rep.AccountReplicator(_master_app(session, peer.port, store)).broadcast() + + assert store.reloads == 1 + + +# ============================================================================= +# 2. A peer that refused is retried +# ============================================================================= + +@pytest.mark.asyncio +async def test_a_peer_that_refuses_is_retried_on_the_next_tick(peer, session): + store = StubStore(USERS) + replicator = rep.AccountReplicator(_master_app(session, peer.port, store)) + + peer.status = 503 + first = await replicator.broadcast() + assert first["failed"] == ["spark2"] and first["pushed"] == [] + + peer.status = 200 + second = await replicator.broadcast() + assert second["pushed"] == ["spark2"] + assert peer.posts[-1]["body"]["users"] == USERS + + +@pytest.mark.asyncio +async def test_a_peer_that_cannot_be_reached_at_all_is_a_failure_not_a_crash(session): + store = StubStore(USERS) + # Port 1 on loopback: nothing listens there, so this is the "node rebooting" + # case, and it must leave the peer behind rather than raise. + app = _master_app(session, 1, store) + + result = await rep.AccountReplicator(app).broadcast() + + assert result["failed"] == ["spark2"] + assert result["pushed"] == [] + + +@pytest.mark.asyncio +async def test_the_retry_timer_is_sixty_seconds_on_a_master(peer, session): + """The loop's wake, which is what makes "retried while behind" true without a + queue. Pinned because the number is the promise in the PR body.""" + app = _master_app(session, peer.port, StubStore(USERS)) + replicator = rep.AccountReplicator(app) + + assert rep.RETRY_INTERVAL_SECONDS == 60.0 + assert replicator._sleep_for() == 60.0 + + +@pytest.mark.asyncio +async def test_a_peer_that_left_the_cluster_stops_being_behind(peer, session): + store = StubStore(USERS) + app = _master_app(session, peer.port, store) + replicator = rep.AccountReplicator(app) + peer.status = 503 + await replicator.broadcast() + assert replicator.accepted == {} + + app["cluster_state"].remove_node("spark2") + result = await replicator.broadcast() + + assert result["peers"] == 0 + assert result["failed"] == [] + + +@pytest.mark.asyncio +async def test_a_master_with_no_peers_makes_no_requests(session): + store = StubStore(USERS) + cluster = ClusterState() + cluster.add_node(_node("spark1", role="master")) + app = {"config": NodeConfig(node_id="spark1", cluster_secret=SECRET, + cluster_role="master"), + "cluster_state": cluster, "client_session": session, + "users_store": store} + + result = await rep.AccountReplicator(app).broadcast() + + assert result["peers"] == 0 and result["pushed"] == [] + + +# ============================================================================= +# 3. A worker pulls +# ============================================================================= + +@pytest.mark.asyncio +async def test_a_worker_imports_the_masters_accounts(peer, session, home): + store = StubStore([]) + app = _worker_app(session, peer.port, store) + + result = await rep.AccountReplicator(app).pull() + + assert result["imported"] is True and result["changed"] is True + assert store.imports == [USERS] + assert peer.gets[0]["authorization"] == f"Bearer {fleet_key(SECRET)}" + # The MASTER's own stamp is recorded, so the doctor compares two values of the + # same kind rather than two hashes computed by different code. + assert rep.read_sync_state(home)["stamp"] == "master-stamp-1" + assert rep.read_sync_state(home)["users"] == 2 + + +@pytest.mark.asyncio +async def test_a_worker_pull_leaves_this_nodes_own_sessions_alone(peer, session): + store = StubStore([], sessions={"jason": [{"id": "sess-1"}]}) + await rep.AccountReplicator(_worker_app(session, peer.port, store)).pull() + + assert store.sessions_for("jason") == [{"id": "sess-1"}] + + +@pytest.mark.asyncio +async def test_an_unchanged_pull_is_reported_as_unchanged(peer, session): + store = StubStore(USERS) + result = await rep.AccountReplicator(_worker_app(session, peer.port, store)).pull() + + assert result["imported"] is True and result["changed"] is False + + +@pytest.mark.asyncio +async def test_an_unreachable_master_warns_once_and_does_not_crash(session, caplog): + store = StubStore(USERS) + app = _worker_app(session, 1, store) # nothing listens on port 1 + replicator = rep.AccountReplicator(app) + + with caplog.at_level(logging.WARNING, logger=rep.logger.name): + first = await replicator.pull() + second = await replicator.pull() + + assert first["imported"] is False and second["imported"] is False + assert store.imports == [], "nothing may be imported from a master that is down" + warnings = [r for r in caplog.records if r.levelno == logging.WARNING] + assert len(warnings) == 1, "a master that is down must not warn on every tick" + assert "keeps the accounts it has" in warnings[0].getMessage() + + +@pytest.mark.asyncio +async def test_a_master_that_answers_rubbish_is_not_imported(session, home): + async def handler(request): + return web.json_response({"stamp": "s", "users": "not a list"}) + + app = web.Application() + app.router.add_get(rep.EXPORT_PATH, handler) + server = TestServer(app) + await server.start_server() + try: + store = StubStore(USERS) + result = await rep.AccountReplicator( + _worker_app(session, server.port, store)).pull() + finally: + await server.close() + + assert result["imported"] is False + assert store.imports == [] + assert rep.read_sync_state(home) == {} + + +@pytest.mark.asyncio +async def test_a_worker_with_no_master_makes_no_requests(session): + store = StubStore(USERS) + app = {"config": NodeConfig(node_id="spark3", cluster_secret=SECRET, + cluster_role="worker"), + "client_session": session, "users_store": store} + + result = await rep.AccountReplicator(app).pull() + + assert result == {"imported": False, "reason": "no master known"} + + +@pytest.mark.asyncio +async def test_a_worker_pulls_at_startup_and_then_on_its_own_interval(peer, session): + """The 60 second wake is the master's retry, not a poll of the master: a + worker that pulled on every one of them would ask twelve times too often.""" + store = StubStore([]) + replicator = rep.AccountReplicator(_worker_app(session, peer.port, store)) + + first = await replicator.tick() + second = await replicator.tick() + + assert first["imported"] is True + assert second == {"imported": False, "reason": "not due"} + assert len(peer.gets) == 1 + assert rep.PULL_INTERVAL_SECONDS == 300.0 + + +# ============================================================================= +# 4. Roles, and the nodes that do nothing +# ============================================================================= + +def test_a_node_with_no_cluster_secret_starts_nothing(): + app = {"config": NodeConfig(node_id="n1", cluster_secret=None)} + assert rep.start_replication(app) is None + assert "users_changed" not in app + + +def test_start_registers_the_broadcaster_the_routes_call(): + app = {"config": NodeConfig(node_id="n1", cluster_secret=SECRET)} + replicator = rep.start_replication(app) + + assert replicator is not None + assert app["users_changed"] == replicator.notify_changed + # It must never block a request: the routes call it inside a handler. + app["users_changed"]() + assert replicator._wake.is_set() + + +def test_a_solo_node_replicates_nothing(): + cluster = ClusterState() + cluster.add_node(_node("only", role="auto")) + app = {"config": NodeConfig(node_id="only", cluster_secret=SECRET), + "cluster_state": cluster} + + assert rep.AccountReplicator(app).role() == "solo" + + +@pytest.mark.asyncio +async def test_a_solo_tick_touches_nothing(session): + cluster = ClusterState() + cluster.add_node(_node("only", role="auto")) + store = StubStore(USERS) + app = {"config": NodeConfig(node_id="only", cluster_secret=SECRET), + "cluster_state": cluster, "client_session": session, + "users_store": store} + + result = await rep.AccountReplicator(app).tick() + + assert result["role"] == "solo" + assert store.reloads == 0 + + +def test_a_configured_worker_is_never_the_authority_whatever_discovery_says(): + """An operator who wrote cluster_role: worker has said this node is not it, + and a node that cannot see its master must not decide it was promoted.""" + cluster = ClusterState() + cluster.add_node(_node("me", role="worker")) + config = NodeConfig(node_id="me", cluster_role="worker") + + assert rep.local_role(config, cluster=cluster) == "worker" + assert rep.local_role(NodeConfig(node_id="me", distributed_mode="member")) \ + == "worker" + + +def test_the_election_outranks_a_config_that_pinned_two_masters(): + cluster = ClusterState() + cluster.add_node(_node("aaa", role="master")) + cluster.add_node(_node("zzz", role="master")) + + assert rep.local_role(NodeConfig(node_id="aaa", cluster_role="master"), + cluster=cluster) == "master" + assert rep.local_role(NodeConfig(node_id="zzz", cluster_role="master"), + cluster=cluster) == "worker" + + +def test_the_role_falls_back_to_the_api_then_to_config_json(): + """A CLI has no ClusterState. With the service up it has /api/cluster/info; + with the service down it has config.json and nothing else.""" + info = {"my_role": "master", "my_node_id": "m1", + "members": [{"node_id": "m1"}, {"node_id": "w1"}]} + assert rep.local_role(NodeConfig(node_id="m1"), info=info) == "master" + assert rep.local_role(NodeConfig(node_id="w1"), + info={"my_role": "worker", "my_node_id": "w1", + "members": [{"node_id": "m1"}]}) == "worker" + assert rep.local_role(NodeConfig(node_id="m1", cluster_role="master", + peer_ips=["10.0.0.2"])) == "master" + assert rep.local_role(NodeConfig(node_id="w1", + master_address="10.0.0.1:3000")) == "worker" + assert rep.local_role(NodeConfig(node_id="solo")) == "solo" + + +def test_the_master_url_comes_from_discovery_first_then_config(): + cluster = ClusterState() + cluster.add_node(_node("me", role="worker")) + cluster.add_node(_node("boss", web_port=3100, role="master")) + app = {"config": NodeConfig(node_id="me", cluster_role="worker", + master_address="10.9.9.9:3000"), + "cluster_state": cluster} + + assert rep.master_target(app) == "http://127.0.0.1:3100" + assert rep.master_target(config=NodeConfig(master_address="10.9.9.9")) \ + == "http://10.9.9.9:3000" + assert rep.master_target(config=NodeConfig()) == "" + + +# ============================================================================= +# 5. The CLI's push, over the stdlib +# ============================================================================= + +def test_the_cli_push_carries_the_fleet_key_to_every_peer(monkeypatch): + seen = [] + + def fake_http_json(url, payload=None, headers=None, timeout=10.0): + seen.append({"url": url, "payload": payload, "headers": headers}) + return 200, {"changed": True, "count": 2} + + monkeypatch.setattr(rep, "http_json", fake_http_json) + config = NodeConfig(node_id="m1", cluster_secret=SECRET, cluster_role="master", + peer_ips=["10.0.0.2", "10.0.0.3"]) + + result = rep.replicate_from_cli(config, USERS) + + assert result["pushed"] == ["10.0.0.2", "10.0.0.3"] + assert [call["url"] for call in seen] == [ + "http://10.0.0.2:3000/api/auth/users/sync", + "http://10.0.0.3:3000/api/auth/users/sync", + ] + for call in seen: + assert call["headers"]["Authorization"] == f"Bearer {fleet_key(SECRET)}" + assert list(call["payload"]) == ["users"] + + +def test_the_cli_push_reports_a_peer_that_refused_rather_than_retrying(monkeypatch): + """The CLI exits; the master's own loop is what guarantees delivery.""" + monkeypatch.setattr(rep, "http_json", + lambda url, payload=None, headers=None, timeout=10.0: + (0, {"error": "connection refused"})) + config = NodeConfig(cluster_secret=SECRET, cluster_role="master", + peer_ips=["10.0.0.2"]) + + result = rep.replicate_from_cli(config, USERS) + + assert result["pushed"] == [] + assert result["failed"][0][0] == "10.0.0.2" + assert "no answer" in result["failed"][0][1] + + +def test_the_cli_push_prefers_the_live_cluster_view_over_config(monkeypatch): + seen = [] + monkeypatch.setattr(rep, "http_json", + lambda url, payload=None, headers=None, timeout=10.0: + (seen.append(url), (200, {}))[1]) + config = NodeConfig(node_id="m1", cluster_secret=SECRET, cluster_role="master", + peer_ips=["10.0.0.9"]) + info = {"my_node_id": "m1", "members": [ + {"node_id": "m1", "node_name": "m1", "web_port": 3000}, + {"node_id": "w1", "node_name": "10.0.0.2", "web_port": 3000}, + {"node_id": "w2", "node_name": "10.0.0.3", "web_port": 3000, + "status": "offline"}, + ]} + + rep.replicate_from_cli(config, USERS, info=info) + + assert seen == ["http://10.0.0.2:3000/api/auth/users/sync"], \ + "an offline member and this node itself are not push targets" + + +def test_a_cli_push_with_no_cluster_secret_says_so_and_sends_nothing(monkeypatch): + monkeypatch.setattr(rep, "http_json", + lambda *a, **kw: pytest.fail("no key, so no request")) + config = NodeConfig(cluster_role="master", peer_ips=["10.0.0.2"]) + + result = rep.replicate_from_cli(config, USERS) + + assert result["pushed"] == [] and "cluster_secret" in result["reason"] + + +# ============================================================================= +# 6. The stamp, and the store seam +# ============================================================================= + +def test_the_stamp_follows_the_content_and_not_a_records_key_order(): + """What "behind" means. A store that writes its keys in another order must not + read as a changed list, or every tick would push to every peer forever.""" + reordered = [{"role": u["role"], "password_hash": u["password_hash"], + "name": u["name"]} for u in USERS] + assert rep.users_stamp(USERS) == rep.users_stamp(reordered) + assert rep.users_stamp(USERS) != rep.users_stamp(USERS[:1]) + changed = [dict(USERS[0], password_hash="other"), USERS[1]] + assert rep.users_stamp(USERS) != rep.users_stamp(changed) + + +def test_open_store_prefers_the_one_on_the_app(monkeypatch): + mine = StubStore(USERS) + assert rep.open_store({"users_store": mine}) is mine + + +def test_open_store_survives_a_release_with_no_account_store(monkeypatch): + monkeypatch.setattr(rep, "users_store_class", lambda: None) + assert rep.open_store() is None + assert rep.min_password_length() == rep.DEFAULT_MIN_PASSWORD + + +def test_open_store_keeps_a_load_that_returns_a_new_instance(monkeypatch): + """``AuthConfig.load()`` is a classmethod that returns a fresh object, and the + account store is modelled on it, so both spellings have to work.""" + class ClassmethodStore(StubStore): + made: list = [] + + @classmethod + def load(cls): # type: ignore[override] + fresh = ClassmethodStore(USERS) + cls.made.append(fresh) + return fresh + + monkeypatch.setattr(rep, "users_store_class", lambda: ClassmethodStore) + store = rep.open_store() + + assert ClassmethodStore.made and store is ClassmethodStore.made[-1] + assert store.export_users() == USERS + + +def test_the_sync_state_is_written_and_read_back(home): + rep.write_sync_state("stamp-9", master="http://10.0.0.1:3000", users=3) + state = rep.read_sync_state() + + assert state["stamp"] == "stamp-9" + assert state["master"] == "http://10.0.0.1:3000" + assert state["users"] == 3 + assert state["at"].endswith("Z") + # No credential goes in this file, so nothing here needs to be 0600. + assert "hash" not in rep.sync_state_path().read_text() + + +def test_an_unreadable_sync_state_is_an_empty_one(home): + rep.sync_state_path().write_text("{not json") + assert rep.read_sync_state() == {} + + +# ============================================================================= +# 7. The loop, started and stopped +# ============================================================================= + +@pytest.mark.asyncio +async def test_the_loop_pushes_when_it_is_woken_and_stops_cleanly(peer, session): + store = StubStore(USERS) + app = _master_app(session, peer.port, store) + replicator = rep.AccountReplicator(app, retry_interval=30.0) + await replicator.start() + try: + for _ in range(50): + if peer.posts: + break + await asyncio.sleep(0.02) + assert peer.posts, "the first tick pushes the current list" + + store.users = [dict(USERS[0])] + replicator.notify_changed() + for _ in range(50): + if len(peer.posts) > 1: + break + await asyncio.sleep(0.02) + assert len(peer.posts) == 2, "a change wakes the loop rather than waiting" + finally: + await rep.stop_replication(app | {"users_replicator": replicator}) + assert replicator._task is None + + +@pytest.mark.asyncio +async def test_a_tick_that_raises_does_not_kill_the_loop(session, monkeypatch): + app = {"config": NodeConfig(node_id="n1", cluster_secret=SECRET, + cluster_role="master", peer_ips=["10.0.0.2"]), + "client_session": session, "users_store": StubStore(USERS)} + replicator = rep.AccountReplicator(app, retry_interval=0.01) + calls = [] + + async def boom(): + calls.append(1) + if len(calls) == 1: + raise RuntimeError("the first tick explodes") + return {} + + monkeypatch.setattr(replicator, "tick", boom) + await replicator.start() + try: + for _ in range(100): + if len(calls) > 1: + break + await asyncio.sleep(0.01) + finally: + await replicator.stop() + assert len(calls) > 1, "the loop must survive a tick that raised" + + +@pytest.mark.asyncio +async def test_a_failed_pull_is_retried_on_the_retry_interval_not_the_full_five_minutes( + peer, session): + """A worker that came up while its master was still booting must not sit + without accounts for five minutes.""" + store = StubStore([]) + replicator = rep.AccountReplicator(_worker_app(session, peer.port, store)) + peer.status = 503 + + failed = await replicator.tick() + assert failed["imported"] is False + assert replicator._next_pull - time.monotonic() <= rep.RETRY_INTERVAL_SECONDS + + peer.status = 200 + replicator._next_pull = 0.0 + ok = await replicator.tick() + + assert ok["imported"] is True + assert replicator._next_pull - time.monotonic() > rep.RETRY_INTERVAL_SECONDS, \ + "a successful pull goes back to the five minute interval" + + +@pytest.mark.asyncio +async def test_stop_replication_is_safe_on_a_node_that_started_none(): + await rep.stop_replication({"config": NodeConfig()}) + + +@pytest.mark.asyncio +async def test_the_servers_startup_wires_the_broadcaster_and_its_cleanup_stops_it( + tmp_path, monkeypatch): + """The real ``create_app`` startup, because a loop nobody starts replicates + nothing and the account routes would find no ``users_changed`` to call. + + Clustering is off here with a ``master_address`` set, which is a real shape (a + member node whose discovery is disabled) and keeps the test off the UDP wire. + """ + from aiohttp.test_utils import TestClient + + import ainode.api.server as server + + monkeypatch.setenv("AINODE_HOME", str(tmp_path)) + monkeypatch.setattr("ainode.core.config.AINODE_HOME", tmp_path) + monkeypatch.setattr("ainode.core.config.CONFIG_FILE", tmp_path / "config.json") + monkeypatch.setattr("ainode.auth.middleware.AINODE_HOME", tmp_path) + monkeypatch.setattr("ainode.auth.middleware.AUTH_FILE", tmp_path / "auth.json") + config = NodeConfig(node_id="n1", node_name="n1", cluster_secret=SECRET, + cluster_enabled=False, cluster_role="worker", + master_address="127.0.0.1:1", onboarded=True) + app = server.create_app(config=config, engine=None) + + async with TestClient(TestServer(app)) as client: + assert client is not None + assert callable(app.get("users_changed")), \ + "the account routes call app['users_changed'] after every mutation" + replicator = app.get("users_replicator") + assert replicator is not None and replicator._task is not None + + assert app.get("users_replicator")._task is None, "cleanup cancels the loop" + + +@pytest.mark.asyncio +async def test_a_node_with_no_cluster_secret_comes_up_with_no_loop(tmp_path, + monkeypatch): + from aiohttp.test_utils import TestClient + + import ainode.api.server as server + + monkeypatch.setenv("AINODE_HOME", str(tmp_path)) + monkeypatch.setattr("ainode.core.config.AINODE_HOME", tmp_path) + monkeypatch.setattr("ainode.core.config.CONFIG_FILE", tmp_path / "config.json") + monkeypatch.setattr("ainode.auth.middleware.AINODE_HOME", tmp_path) + monkeypatch.setattr("ainode.auth.middleware.AUTH_FILE", tmp_path / "auth.json") + app = server.create_app(config=NodeConfig(node_id="n1", cluster_enabled=False, + onboarded=True), engine=None) + + async with TestClient(TestServer(app)): + assert app.get("users_replicator") is None + assert app.get("users_changed") is None diff --git a/tests/test_cli_auth_users.py b/tests/test_cli_auth_users.py new file mode 100644 index 00000000..432785f5 --- /dev/null +++ b/tests/test_cli_auth_users.py @@ -0,0 +1,716 @@ +"""``ainode auth user`` and ``ainode auth session``: the operator's side of the login. + +The dashboard login (#261) has no sign-up page, and must not: a route that mints the +first admin over HTTP is a route anybody who can reach port 3000 calls before the +operator does. So the first account is made on the box, and these commands are the +whole path to having one. What is pinned here: + +1. **The argparse wiring**, because a subcommand that is not wired is a typo an + operator discovers instead of a test. +2. **Both ways of supplying a password.** Interactive is two ``getpass`` prompts + that have to agree; ``--password-stdin`` is the answer for anything with no + terminal, which includes ``ssh host ainode ...`` and every script, because the + installer's host wrapper runs ``docker exec -it`` and getpass needs a TTY. +3. **A short password is refused** before anything is written. +4. **The last admin cannot be removed or disabled.** With auth on and no admin, the + dashboard can only be opened by pasting an API key, which is the state + ``ainode doctor``'s Login check FAILs on. +5. **Replication is triggered on a master and warned about on a worker**, and it + carries the fleet key. A change made on the wrong node is still made, because + refusing would leave an operator with neither an account nor an explanation. +6. **Sessions are never replicated**, and ``ainode auth status`` counts them beside + the keys. + +The account store lands on its own branch (``ainode/auth/accounts.py``), so the +store here is :class:`StubStore` and the one guarded import the package makes +(``replication.open_store``) is what the tests replace. +""" + +from __future__ import annotations + +import io +import sys +from unittest.mock import patch + +import pytest + +from ainode.auth import replication as rep +from ainode.auth.fleet import fleet_key +from ainode.cli import main as cli + +SECRET = "0123456789abcdef0123456789abcdef" + + +class StubStore: + """``UsersStore`` as the CLI uses it, in memory.""" + + MIN_PASSWORD_LENGTH = 8 + + def __init__(self, users=None, sessions=None, can_disable=True): + self.users = [dict(u) for u in (users or [])] + self.sessions = {k: [dict(s) for s in v] for k, v in (sessions or {}).items()} + self.saved = 0 + self.can_disable = can_disable + + # -- opening --------------------------------------------------------- + def load(self): + return self + + def save(self): + self.saved += 1 + + # -- accounts -------------------------------------------------------- + def add_user(self, name, password, role="member"): + if any(u["name"] == name for u in self.users): + raise ValueError(f"an account named {name} already exists") + if len(password) < self.MIN_PASSWORD_LENGTH: + raise ValueError("password too short") + self.users.append({"name": name, "role": role, "disabled": False, + "password_hash": f"hash-of-{password}"}) + return True + + def remove_user(self, name): + before = len(self.users) + self.users = [u for u in self.users if u["name"] != name] + self.sessions.pop(name, None) + return len(self.users) != before + + def set_password(self, name, password): + for user in self.users: + if user["name"] == name: + user["password_hash"] = f"hash-of-{password}" + return True + return False + + def set_enabled(self, name, enabled): + if not self.can_disable: + raise AttributeError("no such method") + for user in self.users: + if user["name"] == name: + user["disabled"] = not enabled + return True + return False + + def list_users(self): + return [{"name": u["name"], "role": u["role"], + "disabled": u.get("disabled", False)} for u in self.users] + + def admin_count(self): + return sum(1 for u in self.users + if u["role"] == "admin" and not u.get("disabled")) + + def has_users(self): + return bool(self.users) + + def export_users(self): + return [dict(u) for u in self.users] + + # -- sessions -------------------------------------------------------- + def sessions_for(self, name): + return [dict(s) for s in self.sessions.get(name, [])] + + def revoke_session(self, session_id, user=None): + for name, rows in self.sessions.items(): + if user and name != user: + continue + kept = [s for s in rows if s.get("id") != session_id] + if len(kept) != len(rows): + self.sessions[name] = kept + return True + return False + + +class NoDisableStore(StubStore): + """A store with no disable of its own: the one method the contract left open.""" + + def __init__(self, *args, **kwargs): + super().__init__(*args, **kwargs) + + set_enabled = None # type: ignore[assignment] + + +@pytest.fixture +def home(tmp_path, monkeypatch): + """Every file these commands touch, inside tmp_path. + + ``AUTH_FILE`` and ``CONFIG_FILE`` are computed at import, so the redirect has to + name them as well as the environment variable (the same shape + ``tests/test_fleet_auth.py`` uses). + """ + monkeypatch.setenv("AINODE_HOME", str(tmp_path)) + monkeypatch.setattr("ainode.core.config.AINODE_HOME", tmp_path) + monkeypatch.setattr("ainode.core.config.CONFIG_FILE", tmp_path / "config.json") + monkeypatch.setattr("ainode.auth.middleware.AINODE_HOME", tmp_path) + monkeypatch.setattr("ainode.auth.middleware.AUTH_FILE", tmp_path / "auth.json") + return tmp_path + + +@pytest.fixture +def store(monkeypatch): + """One store for the run, in place of the guarded import.""" + stub = StubStore() + monkeypatch.setattr(rep, "open_store", lambda app=None: stub) + return stub + + +@pytest.fixture +def no_peer_calls(monkeypatch): + """No replication request leaves the test unless a test asks for one. + + ``/api/cluster/info`` on 127.0.0.1 is a real request otherwise, and on the + operator's own machine it would reach their own node. + """ + calls: list[dict] = [] + + def fake_http_json(url, payload=None, headers=None, timeout=10.0): + calls.append({"url": url, "payload": payload, "headers": headers or {}}) + return 0, None + + monkeypatch.setattr(rep, "http_json", fake_http_json) + return calls + + +def run(*argv, stdin: str = ""): + """``ainode ``, with stdin fed from a string. Returns (exit code, output).""" + from rich.console import Console + + buffer = io.StringIO() + with patch.object(cli, "console", Console(file=buffer, width=100, + force_terminal=False)): + with patch.object(sys, "argv", ["ainode", *argv]): + with patch.object(sys, "stdin", io.StringIO(stdin)): + code = 0 + try: + cli.main() + except SystemExit as exc: + code = exc.code or 0 + return code, buffer.getvalue() + + +def _config(home, **keys): + import json + + payload = {"node_id": "n1", "node_name": "n1", "cluster_secret": SECRET, + "onboarded": True} + payload.update(keys) + (home / "config.json").write_text(json.dumps(payload)) + + +# ============================================================================= +# 1. The wiring +# ============================================================================= + +def test_every_account_subcommand_is_wired(home, store, no_peer_calls): + """A subcommand argparse does not know about exits 2 with a usage message, and + nothing in the product would notice.""" + _config(home) + for argv in (("auth", "user", "list"), + ("auth", "session", "list"), + ("auth", "status")): + code, out = run(*argv) + assert code == 0, f"{argv} exited {code}: {out}" + + +def test_the_usage_line_names_the_new_commands(home, store, no_peer_calls): + _config(home) + code, out = run("auth") + assert "user" in out and "session" in out + assert "--password-stdin" in out + + +def test_password_stdin_is_documented_in_the_help(capsys): + """The host wrapper runs `docker exec -it`, so the prompt needs a terminal and + the flag is the documented way to work without one.""" + with patch.object(sys, "argv", ["ainode", "auth", "user", "add", "--help"]): + with pytest.raises(SystemExit): + cli.main() + out = capsys.readouterr().out + assert "--password-stdin" in out + assert "TTY" in out or "tty" in out + + +def test_the_key_commands_still_work(home, monkeypatch): + """`ainode auth key ...` and `auth status|enable|disable` are not disturbed.""" + _config(home) + code, out = run("auth", "key", "create", "--name", "laptop") + assert code == 0 and "New API key created" in out + code, out = run("auth", "key", "list") + assert code == 0 and "laptop" in out + code, out = run("auth", "enable") + assert code == 0 and "Auth enabled" in out + code, out = run("auth", "disable") + assert code == 0 and "Auth disabled" in out + + +# ============================================================================= +# 2. Adding an account +# ============================================================================= + +def test_add_prompts_twice_and_writes_the_account(home, store, no_peer_calls, + monkeypatch): + _config(home) + asked = [] + + def fake_getpass(prompt=""): + asked.append(prompt) + return "a-good-password" + + monkeypatch.setattr("getpass.getpass", fake_getpass) + + code, out = run("auth", "user", "add", "jason", "--admin") + + assert code == 0, out + assert len(asked) == 2, "a new password is typed twice or it is a typo" + assert store.users == [{"name": "jason", "role": "admin", "disabled": False, + "password_hash": "hash-of-a-good-password"}] + assert store.saved >= 1, "the account has to be on disk, not just in memory" + assert "added" in out and "admin" in out + assert "re-reads users.json" in out, "the running node picks the file up live" + assert "Made in Texas" in out + + +def test_add_refuses_when_the_two_prompts_disagree(home, store, no_peer_calls, + monkeypatch): + _config(home) + answers = iter(["first-password", "second-password"]) + monkeypatch.setattr("getpass.getpass", lambda prompt="": next(answers)) + + code, out = run("auth", "user", "add", "jason") + + assert code == 2 + assert "do not match" in out + assert store.users == [] + + +def test_add_reads_the_password_from_stdin_with_password_stdin(home, store, + no_peer_calls, + monkeypatch): + _config(home) + monkeypatch.setattr("getpass.getpass", + lambda prompt="": pytest.fail("must not prompt with a pipe")) + + code, out = run("auth", "user", "add", "jason", "--admin", "--password-stdin", + stdin="piped-password\n") + + assert code == 0, out + assert store.users[0]["password_hash"] == "hash-of-piped-password", \ + "only the newline the shell added is stripped" + + +def test_add_refuses_a_short_password(home, store, no_peer_calls): + _config(home) + code, out = run("auth", "user", "add", "jason", "--password-stdin", stdin="short\n") + + assert code == 2 + assert "too short" in out and "8" in out + assert store.users == [] + + +def test_add_refuses_a_duplicate_name_with_the_stores_own_message(home, store, + no_peer_calls): + _config(home) + store.users = [{"name": "jason", "role": "admin", "disabled": False, + "password_hash": "x"}] + + code, out = run("auth", "user", "add", "jason", "--password-stdin", + stdin="a-good-password\n") + + assert code == 2 + assert "already exists" in out + + +def test_a_name_typed_with_capitals_is_folded_to_the_contracts_alphabet( + home, store, no_peer_calls): + _config(home) + code, out = run("auth", "user", "add", "Jason", "--password-stdin", + stdin="a-good-password\n") + + assert code == 0, out + assert store.users[0]["name"] == "jason" + assert "jason" in out + + +def test_a_node_with_no_account_store_says_so_instead_of_raising(home, monkeypatch, + no_peer_calls): + _config(home) + monkeypatch.setattr(rep, "open_store", lambda app=None: None) + + code, out = run("auth", "user", "list") + + assert code == 2 + assert "no account store" in out + assert "auth key create" in out, "the key path still works, so it is named" + + +# ============================================================================= +# 3. The last admin +# ============================================================================= + +def test_removing_the_last_admin_is_refused(home, store, no_peer_calls): + _config(home) + store.users = [{"name": "jason", "role": "admin", "disabled": False, + "password_hash": "x"}, + {"name": "ops", "role": "member", "disabled": False, + "password_hash": "y"}] + + code, out = run("auth", "user", "remove", "jason") + + assert code == 2 + assert "only admin" in out + assert [u["name"] for u in store.users] == ["jason", "ops"] + + +def test_disabling_the_last_admin_is_refused_too(home, store, no_peer_calls): + """A disabled admin cannot sign in, so it locks the dashboard exactly as hard + as a deleted one.""" + _config(home) + store.users = [{"name": "jason", "role": "admin", "disabled": False, + "password_hash": "x"}] + + code, out = run("auth", "user", "disable", "jason") + + assert code == 2 + assert "only admin" in out + assert store.users[0]["disabled"] is False + + +def test_a_second_admin_makes_the_first_removable(home, store, no_peer_calls): + _config(home) + store.users = [{"name": "jason", "role": "admin", "disabled": False, + "password_hash": "x"}, + {"name": "sem", "role": "admin", "disabled": False, + "password_hash": "y"}] + + code, out = run("auth", "user", "remove", "jason") + + assert code == 0, out + assert [u["name"] for u in store.users] == ["sem"] + assert "removed" in out + + +def test_removing_a_member_is_never_refused(home, store, no_peer_calls): + _config(home) + store.users = [{"name": "jason", "role": "admin", "disabled": False, + "password_hash": "x"}, + {"name": "ops", "role": "member", "disabled": False, + "password_hash": "y"}] + + code, out = run("auth", "user", "remove", "ops") + + assert code == 0, out + assert [u["name"] for u in store.users] == ["jason"] + + +def test_an_unknown_account_is_a_refusal_and_not_a_traceback(home, store, + no_peer_calls): + _config(home) + for action in ("remove", "passwd", "disable", "enable"): + code, out = run("auth", "user", action, "nobody") + assert code == 2, action + assert "No account named 'nobody'" in out + + +# ============================================================================= +# 4. The rest of the account commands +# ============================================================================= + +def test_list_shows_the_role_the_state_and_the_session_count(home, store, + no_peer_calls): + _config(home) + store.users = [{"name": "jason", "role": "admin", "disabled": False, + "password_hash": "THE-STORED-HASH"}, + {"name": "ops", "role": "member", "disabled": True, + "password_hash": "y"}] + store.sessions = {"jason": [{"id": "s1"}, {"id": "s2"}]} + + code, out = run("auth", "user", "list") + + assert code == 0, out + assert "jason" in out and "admin" in out and "disabled" in out + assert "not replicated" in out, "sessions are per node and the list says so" + assert "THE-STORED-HASH" not in out, "no password hash is ever printed" + + +def test_passwd_changes_the_hash_and_names_the_sign_out_command(home, store, + no_peer_calls): + _config(home) + store.users = [{"name": "jason", "role": "admin", "disabled": False, + "password_hash": "old"}] + + code, out = run("auth", "user", "passwd", "jason", "--password-stdin", + stdin="a-new-password\n") + + assert code == 0, out + assert store.users[0]["password_hash"] == "hash-of-a-new-password" + assert "session clear --user jason" in out + + +def test_disable_then_enable_round_trips(home, store, no_peer_calls): + _config(home) + store.users = [{"name": "jason", "role": "admin", "disabled": False, + "password_hash": "x"}, + {"name": "ops", "role": "member", "disabled": False, + "password_hash": "y"}] + + code, out = run("auth", "user", "disable", "ops") + assert code == 0, out + assert store.users[1]["disabled"] is True + assert "cannot sign in" in out + + code, out = run("auth", "user", "enable", "ops") + assert code == 0, out + assert store.users[1]["disabled"] is False + + +def test_a_store_with_no_disable_of_its_own_points_at_the_route(home, monkeypatch, + no_peer_calls): + """The one method the account contract did not name. The CLI asks for the three + spellings a store of this shape would use, and says so plainly when it has none + rather than writing a field name into somebody else's file.""" + _config(home) + stub = NoDisableStore([{"name": "jason", "role": "admin", "disabled": False, + "password_hash": "x"}, + {"name": "ops", "role": "member", "disabled": False, + "password_hash": "y"}]) + monkeypatch.setattr(rep, "open_store", lambda app=None: stub) + + code, out = run("auth", "user", "disable", "ops") + + assert code == 2 + assert "/api/auth/users/ops/disable" in out + + +# ============================================================================= +# 5. Sessions, which are per node +# ============================================================================= + +def test_session_list_shows_every_session_on_this_node(home, store, no_peer_calls): + _config(home) + store.users = [{"name": "jason", "role": "admin", "disabled": False, + "password_hash": "x"}, + {"name": "ops", "role": "member", "disabled": False, + "password_hash": "y"}] + store.sessions = {"jason": [{"id": "s1", "created_at": "2026-09-21T10:00:00Z"}], + "ops": [{"id": "s2"}]} + + code, out = run("auth", "session", "list") + + assert code == 0, out + assert "s1" in out and "s2" in out + assert "per node" in out + + +def test_session_list_can_be_narrowed_to_one_user(home, store, no_peer_calls): + _config(home) + store.users = [{"name": "jason", "role": "admin", "disabled": False, + "password_hash": "x"}, + {"name": "ops", "role": "member", "disabled": False, + "password_hash": "y"}] + store.sessions = {"jason": [{"id": "s1"}], "ops": [{"id": "s2"}]} + + code, out = run("auth", "session", "list", "--user", "jason") + + assert code == 0, out + assert "s1" in out and "s2" not in out + + +def test_session_revoke_takes_one_session_away(home, store, no_peer_calls): + _config(home) + store.users = [{"name": "jason", "role": "admin", "disabled": False, + "password_hash": "x"}] + store.sessions = {"jason": [{"id": "s1"}, {"id": "s2"}]} + + code, out = run("auth", "session", "revoke", "s1") + + assert code == 0, out + assert [s["id"] for s in store.sessions["jason"]] == ["s2"] + + +def test_session_revoke_of_an_unknown_id_is_a_refusal(home, store, no_peer_calls): + _config(home) + code, out = run("auth", "session", "revoke", "nope") + + assert code == 2 + assert "No session 'nope'" in out + + +def test_session_clear_signs_everybody_out(home, store, no_peer_calls): + _config(home) + store.users = [{"name": "jason", "role": "admin", "disabled": False, + "password_hash": "x"}, + {"name": "ops", "role": "member", "disabled": False, + "password_hash": "y"}] + store.sessions = {"jason": [{"id": "s1"}, {"id": "s2"}], "ops": [{"id": "s3"}]} + + code, out = run("auth", "session", "clear") + + assert code == 0, out + assert store.sessions == {"jason": [], "ops": []} + assert "3 session(s)" in out + + +def test_session_clear_can_be_narrowed_to_one_user(home, store, no_peer_calls): + _config(home) + store.users = [{"name": "jason", "role": "admin", "disabled": False, + "password_hash": "x"}, + {"name": "ops", "role": "member", "disabled": False, + "password_hash": "y"}] + store.sessions = {"jason": [{"id": "s1"}], "ops": [{"id": "s3"}]} + + code, out = run("auth", "session", "clear", "--user", "ops") + + assert code == 0, out + assert [s["id"] for s in store.sessions["jason"]] == ["s1"] + assert store.sessions["ops"] == [] + + +def test_a_session_command_never_replicates_anything(home, store, no_peer_calls): + """Sessions are per node: there is nothing here to fan out.""" + _config(home, cluster_role="master", peer_ips=["10.0.0.2"]) + store.users = [{"name": "jason", "role": "admin", "disabled": False, + "password_hash": "x"}] + store.sessions = {"jason": [{"id": "s1"}]} + + run("auth", "session", "clear") + + assert no_peer_calls == [] + + +# ============================================================================= +# 6. Replication, from the CLI +# ============================================================================= + +def test_a_master_pushes_the_change_to_its_peers_with_the_fleet_key( + home, store, no_peer_calls): + _config(home, cluster_role="master", peer_ips=["10.0.0.2"]) + + code, out = run("auth", "user", "add", "jason", "--admin", "--password-stdin", + stdin="a-good-password\n") + + assert code == 0, out + posts = [c for c in no_peer_calls if c["payload"] is not None] + assert [c["url"] for c in posts] == ["http://10.0.0.2:3000/api/auth/users/sync"] + assert posts[0]["headers"]["Authorization"] == f"Bearer {fleet_key(SECRET)}" + assert list(posts[0]["payload"]) == ["users"], "sessions are never replicated" + assert "10.0.0.2 did not take the update" in out, \ + "the stub answers nothing, so the operator is told and the loop retries" + + +def test_a_worker_writes_the_change_and_warns_that_it_will_be_overwritten( + home, store, no_peer_calls): + _config(home, cluster_role="worker", distributed_mode="member", + master_address="10.0.0.1:3000") + + code, out = run("auth", "user", "add", "jason", "--admin", "--password-stdin", + stdin="a-good-password\n") + + assert code == 0, out + assert store.users[0]["name"] == "jason", "the change is made, not refused" + assert "managed on the master" in out + assert "overwritten by the next sync" in out + assert [c for c in no_peer_calls if c["payload"] is not None] == [], \ + "a worker does not push its own accounts at the fleet" + + +def test_a_solo_node_says_nothing_about_replication(home, store, no_peer_calls): + _config(home) + + code, out = run("auth", "user", "add", "jason", "--admin", "--password-stdin", + stdin="a-good-password\n") + + assert code == 0, out + assert "master" not in out + assert [c for c in no_peer_calls if c["payload"] is not None] == [] + + +def test_the_role_comes_from_the_running_node_when_it_answers(home, store, + monkeypatch): + """`/api/cluster/info` is the live answer: a node whose config says nothing is + still a worker if the cluster elected somebody else.""" + _config(home) + calls: list[dict] = [] + + def fake_http_json(url, payload=None, headers=None, timeout=10.0): + calls.append({"url": url, "payload": payload}) + if rep.CLUSTER_INFO_PATH in url: + return 200, {"my_role": "worker", "my_node_id": "n1", + "master_address": "10.0.0.1:3000", + "members": [{"node_id": "m1"}, {"node_id": "n1"}]} + return 200, {"changed": True} + + monkeypatch.setattr(rep, "http_json", fake_http_json) + + code, out = run("auth", "user", "add", "jason", "--admin", "--password-stdin", + stdin="a-good-password\n") + + assert code == 0, out + assert "managed on the master" in out + assert [c["url"] for c in calls if c["payload"] is not None] == [] + + +def test_every_account_mutation_replicates_and_no_read_does(home, store, + no_peer_calls): + _config(home, cluster_role="master", peer_ips=["10.0.0.2"]) + store.users = [{"name": "jason", "role": "admin", "disabled": False, + "password_hash": "x"}, + {"name": "ops", "role": "member", "disabled": False, + "password_hash": "y"}] + + for argv, stdin in ((("auth", "user", "passwd", "ops"), "a-good-password\n"), + (("auth", "user", "disable", "ops"), ""), + (("auth", "user", "enable", "ops"), ""), + (("auth", "user", "remove", "ops"), "")): + no_peer_calls.clear() + argv = argv + ("--password-stdin",) if stdin else argv + code, out = run(*argv, stdin=stdin) + assert code == 0, out + assert [c for c in no_peer_calls if c["payload"] is not None], \ + f"{argv} did not replicate" + + no_peer_calls.clear() + run("auth", "user", "list") + assert no_peer_calls == [], "reading the list changes nothing, so it pushes nothing" + + +# ============================================================================= +# 7. `ainode auth status` +# ============================================================================= + +def test_status_counts_users_admins_and_sessions(home, store, no_peer_calls): + _config(home) + store.users = [{"name": "jason", "role": "admin", "disabled": False, + "password_hash": "x"}, + {"name": "ops", "role": "member", "disabled": False, + "password_hash": "y"}] + store.sessions = {"jason": [{"id": "s1"}, {"id": "s2"}], "ops": [{"id": "s3"}]} + + code, out = run("auth", "status") + + assert code == 0, out + assert "Users: 2 (1 admin)" in out + assert "Sessions: 3" in out + assert "never" in out and "replicated" in out + + +def test_status_says_a_protected_node_with_no_account_can_only_take_a_key( + home, store, no_peer_calls): + _config(home) + run("auth", "enable") + + code, out = run("auth", "status") + + assert code == 0, out + assert "Users: 0" in out + assert "only be opened with an API key" in out + assert "auth user add" in out + + +def test_status_still_works_on_a_build_with_no_account_store(home, monkeypatch, + no_peer_calls): + _config(home) + monkeypatch.setattr(rep, "open_store", lambda app=None: None) + + code, out = run("auth", "status") + + assert code == 0, out + assert "Auth:" in out and "Users:" not in out diff --git a/tests/test_doctor.py b/tests/test_doctor.py index 7168c85a..54ca9c80 100644 --- a/tests/test_doctor.py +++ b/tests/test_doctor.py @@ -675,6 +675,220 @@ def test_a_world_readable_secrets_store_warns_and_is_fixable(tmp_path): assert check.data["fix_action"] == "chmod600" +# -------------------------------------------------------------------- login + +def _write_users(home, records, mode=0o600): + """``users.json`` as the account store writes it: hashes included, 0600.""" + path = home / "users.json" + path.write_text(json.dumps({"users": records})) + os.chmod(path, mode) + return path + + +def _admin(name="jason", disabled=False): + return {"name": name, "role": "admin", "disabled": disabled, + "password_hash": "hash"} + + +def _member(name="ops"): + return {"name": name, "role": "member", "disabled": False, + "password_hash": "hash"} + + +def _auth_on(home, enabled=True): + (home / "auth.json").write_text(json.dumps( + {"enabled": enabled, "api_keys": [{"id": "k1", "key_hash": "h"}]})) + + +def test_auth_on_with_an_admin_account_is_the_pass(tmp_path): + _auth_on(tmp_path) + _write_users(tmp_path, [_admin(), _member()]) + checks = doc.check_login(NodeConfig(), tmp_path) + + state = _by_name(checks, "login.state") + assert state.status == OK + assert state.data["users"] == 2 and state.data["admins"] == 1 + assert _by_name(checks, "login.store").status == OK + + +def test_auth_off_is_a_pass_whether_or_not_anybody_has_an_account(tmp_path): + """Nothing is refused without a credential, so a missing login is not a finding.""" + check = _by_name(doc.check_login(NodeConfig(), tmp_path), "login.state") + assert check.status == OK + assert "without a login" in check.detail + + _write_users(tmp_path, [_admin()]) + check = _by_name(doc.check_login(NodeConfig(), tmp_path), "login.state") + assert check.status == OK + assert "unused until auth is on" in check.detail + + +def test_auth_on_with_no_account_at_all_is_the_fail(tmp_path): + """The dashboard is then reachable only by pasting an API key, which is the + thing the login exists to replace.""" + _auth_on(tmp_path) + check = _by_name(doc.check_login(NodeConfig(), tmp_path), "login.state") + + assert check.status == FAIL + assert "only be opened by pasting an API key" in check.detail + assert "ainode auth user add" in check.fix + assert doc.exit_code([check]) == 1 + + +def test_auth_on_with_accounts_but_no_enabled_admin_warns(tmp_path): + _auth_on(tmp_path) + _write_users(tmp_path, [_admin(disabled=True), _member()]) + check = _by_name(doc.check_login(NodeConfig(), tmp_path), "login.state") + + assert check.status == WARN + assert "no enabled admin" in check.detail + + +def test_a_world_readable_users_file_warns_and_is_fixable(tmp_path): + """It holds password hashes, so it is 0600 like auth.json and secrets.json.""" + _auth_on(tmp_path) + _write_users(tmp_path, [_admin()], mode=0o644) + check = _by_name(doc.check_login(NodeConfig(), tmp_path), "login.store") + + assert check.status == WARN + assert check.data["fix_action"] == "chmod600" + assert "password hashes" in check.detail + + +def test_the_login_fix_action_is_applied_by_fix(tmp_path): + _auth_on(tmp_path) + path = _write_users(tmp_path, [_admin()], mode=0o644) + checks = doc.check_login(NodeConfig(), tmp_path) + + done = doc.apply_fixes(checks, tmp_path / "config.json") + + assert any("chmod 0600" in line for line in done) + assert stat.S_IMODE(path.stat().st_mode) == 0o600 + + +def test_an_unreadable_users_file_warns_rather_than_crashing(tmp_path): + _auth_on(tmp_path) + (tmp_path / "users.json").write_text("{not json") + check = _by_name(doc.check_login(NodeConfig(), tmp_path), "login.state") + + assert check.status == WARN + assert "cannot read" in check.detail + + +def test_a_users_file_in_any_of_the_stores_shapes_is_counted(tmp_path): + """The file belongs to the account store, so the doctor reads it tolerantly + rather than importing a module that may not be there.""" + for raw in ({"users": [_admin()]}, + {"users": {"jason": {"role": "admin"}}}, + [_admin()]): + assert [r["role"] for r in doc.users_file_records(raw)] == ["admin"] + assert doc.users_file_records({"users": []}) == [] + assert doc.users_file_records("nonsense") == [] + + +def test_a_worker_whose_accounts_match_the_master_is_ok(tmp_path, monkeypatch): + from ainode.auth import replication as rep + + monkeypatch.setenv("AINODE_HOME", str(tmp_path)) + _auth_on(tmp_path) + _write_users(tmp_path, [_admin()]) + rep.write_sync_state("master-stamp-1", master="http://10.0.0.1:3000", users=1, + home=tmp_path) + monkeypatch.setattr(doc, "http_json", + lambda url, timeout=3.0, headers=None: + {"users": [], "stamp": "master-stamp-1"}) + config = NodeConfig(cluster_role="worker", master_address="10.0.0.1:3000", + cluster_secret="s" * 32) + + check = _by_name(doc.check_login(config, tmp_path), "login.sync") + + assert check.status == OK + + +def test_a_worker_behind_the_masters_stamp_warns(tmp_path, monkeypatch): + """The password somebody just set on the master does not work here yet, and the + two stamps compared are both the MASTER's own, so they cannot drift.""" + from ainode.auth import replication as rep + from ainode.auth.fleet import fleet_key + + monkeypatch.setenv("AINODE_HOME", str(tmp_path)) + _auth_on(tmp_path) + _write_users(tmp_path, [_admin()]) + rep.write_sync_state("master-stamp-1", master="http://10.0.0.1:3000", users=1, + home=tmp_path) + seen = [] + + def fake_http_json(url, timeout=3.0, headers=None): + seen.append({"url": url, "headers": headers or {}}) + return {"users": [], "stamp": "master-stamp-2"} + + monkeypatch.setattr(doc, "http_json", fake_http_json) + config = NodeConfig(cluster_role="worker", master_address="10.0.0.1:3000", + cluster_secret="s" * 32) + + check = _by_name(doc.check_login(config, tmp_path), "login.sync") + + assert check.status == WARN + assert "older than the master's" in check.detail + # Asked of the master, with the fleet key, like every other node-to-node read. + assert seen[0]["url"] == "http://10.0.0.1:3000/api/auth/users/export" + assert seen[0]["headers"]["Authorization"] == f"Bearer {fleet_key('s' * 32)}" + + +def test_a_worker_that_has_never_pulled_warns(tmp_path, monkeypatch): + monkeypatch.setenv("AINODE_HOME", str(tmp_path)) + _auth_on(tmp_path) + monkeypatch.setattr(doc, "http_json", + lambda url, timeout=3.0, headers=None: + {"users": [_admin()], "stamp": "master-stamp-1"}) + config = NodeConfig(cluster_role="worker", master_address="10.0.0.1:3000") + + check = _by_name(doc.check_login(config, tmp_path), "login.sync") + + assert check.status == WARN + assert "never imported" in check.detail + + +def test_a_worker_whose_master_does_not_answer_warns(tmp_path, monkeypatch): + monkeypatch.setenv("AINODE_HOME", str(tmp_path)) + _auth_on(tmp_path) + _write_users(tmp_path, [_admin()]) + monkeypatch.setattr(doc, "http_json", lambda url, timeout=3.0, headers=None: None) + config = NodeConfig(cluster_role="worker", master_address="10.0.0.1:3000") + + check = _by_name(doc.check_login(config, tmp_path), "login.sync") + + assert check.status == WARN + assert "did not answer" in check.detail + + +def test_a_worker_with_no_master_named_anywhere_warns(tmp_path, monkeypatch): + monkeypatch.setenv("AINODE_HOME", str(tmp_path)) + monkeypatch.setattr(doc, "http_json", + lambda url, timeout=3.0, headers=None: + pytest.fail("there is nobody to ask")) + config = NodeConfig(cluster_role="worker") + + check = _by_name(doc.check_login(config, tmp_path), "login.sync") + + assert check.status == WARN + assert "nothing names its master" in check.detail + assert "ainode join" in check.fix + + +def test_a_master_and_a_solo_node_ask_nobody_anything(tmp_path, monkeypatch): + """The sync check is the one part of this that leaves the box, so it runs on a + worker and nowhere else.""" + monkeypatch.setenv("AINODE_HOME", str(tmp_path)) + monkeypatch.setattr(doc, "http_json", + lambda url, timeout=3.0, headers=None: + pytest.fail("a master pulls from nobody")) + for config in (NodeConfig(cluster_role="master", peer_ips=["10.0.0.2"]), + NodeConfig()): + names = [c.name for c in doc.check_login(config, tmp_path)] + assert "login.sync" not in names + + # ------------------------------------------------------------------ hf token def test_an_hf_token_in_config_is_reported_without_its_value(tmp_path): From b4a23ff5fd42faa37d43f52254c656ad6b436136 Mon Sep 17 00:00:00 2001 From: webdevtodayjason Date: Mon, 21 Sep 2026 18:52:22 -0500 Subject: [PATCH 2/4] auth status: the key state before the account counts Reading order: what the API does without a key, then who can sign in. Co-Authored-By: Claude Fable 5.1 --- ainode/cli/main.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/ainode/cli/main.py b/ainode/cli/main.py index c3dc31ba..7dfd91aa 100644 --- a/ainode/cli/main.py +++ b/ainode/cli/main.py @@ -1399,6 +1399,9 @@ def cmd_auth(args): state = "[green]enabled[/green]" if auth_cfg.enabled else "[dim]disabled[/dim]" console.print(f" Auth: {state}") console.print(f" Keys: {len(auth_cfg.api_keys)}") + if not auth_cfg.enabled: + console.print(" API open, no key set" if not auth_cfg.api_keys + else " API open, key set but not required") # The accounts half of the same question. Auth on with no admin account is # a dashboard only a key can open, which is what ainode doctor FAILs on, so # the number belongs beside the key count rather than behind another @@ -1420,9 +1423,6 @@ def cmd_auth(args): console.print(" [yellow]No admin account[/yellow]: nobody can manage " "users from the dashboard.") console.print(" Fix: ainode auth user add --admin") - if not auth_cfg.enabled: - console.print(" API open, no key set" if not auth_cfg.api_keys - else " API open, key set but not required") # Whether this node can still talk to its own cluster with auth on. The # doctor says the same thing as a FAIL (cli/doctor.py::check_auth); this # is the version an operator sees while turning auth on. From 8fd696e359b2dcc67fc92ec1c4095418edfb249b Mon Sep 17 00:00:00 2001 From: webdevtodayjason Date: Mon, 21 Sep 2026 18:55:55 -0500 Subject: [PATCH 3/4] The host wrapper asks for a TTY only when it has one `--password-stdin` exists for the case with no terminal, and the wrapper the installer writes ran every forwarded command with `docker exec -it`, so `echo pw | ainode auth user add x --password-stdin` died on "the input device is not a TTY" before the CLI in the container ran at all. That is the same failure `ainode doctor --peer` already routes around by calling `docker exec` itself without -it (found against Spark-3). The wrapper now picks -i or -it from whether stdin and stdout are terminals, which fixes the pipe and leaves an interactive `ainode` exactly as it was. Co-Authored-By: Claude Fable 5.1 --- scripts/install.sh | 12 +++++-- tests/test_cli_auth_users.py | 67 ++++++++++++++++++++++++++++++++++++ 2 files changed, 77 insertions(+), 2 deletions(-) diff --git a/scripts/install.sh b/scripts/install.sh index 61d222d9..5bcd537d 100755 --- a/scripts/install.sh +++ b/scripts/install.sh @@ -806,14 +806,22 @@ restart_service() { # binary and no path to the tailnet daemon, so this variable is the cheapest way # for the CLI in there to know which tailnet node it is running on. forward_to_container() { + # A TTY is allocated only when there is one to pass on. \`docker exec -t\` with + # a pipe or an \`ssh host ainode ...\` command line dies with "the input device + # is not a TTY" before the CLI in the container runs at all, so a hardcoded + # -it is what would stop + # \`echo pw | ainode auth user add x --password-stdin\` working and what + # \`ainode doctor --peer\` already has to route around (cli/doctor.py). + local exec_tty="-i" + if [ -t 0 ] && [ -t 1 ]; then exec_tty="-it"; fi if docker exec ainode true 2>/dev/null; then - exec docker exec -it -e AINODE_TAILNET_NAME -e AINODE_HOST_SERVICE_STATE ainode ainode "\$@" + exec docker exec \$exec_tty -e AINODE_TAILNET_NAME -e AINODE_HOST_SERVICE_STATE ainode ainode "\$@" fi # Same sudo trap as update: mount the .ainode the unit uses. local conf_home conf_home="\$(resolve_ainode_home || true)" [ -n "\$conf_home" ] || conf_home="\$HOME/.ainode" - exec docker run --rm -it \\ + exec docker run --rm \$exec_tty \\ --entrypoint ainode \\ -e AINODE_TAILNET_NAME \\ -e AINODE_HOST_SERVICE_STATE \\ diff --git a/tests/test_cli_auth_users.py b/tests/test_cli_auth_users.py index 432785f5..b3fcb864 100644 --- a/tests/test_cli_auth_users.py +++ b/tests/test_cli_auth_users.py @@ -29,7 +29,11 @@ from __future__ import annotations import io +import os +import shutil +import subprocess import sys +from pathlib import Path from unittest.mock import patch import pytest @@ -714,3 +718,66 @@ def test_status_still_works_on_a_build_with_no_account_store(home, monkeypatch, assert code == 0, out assert "Auth:" in out and "Users:" not in out + + +# ============================================================================= +# 8. The installer, and the wrapper the flag has to survive +# ============================================================================= + +INSTALL_SH = Path(__file__).resolve().parent.parent / "scripts" / "install.sh" + + +def _render_install(tmp_path, env_extra=None): + """The real installer in --dry-run against a throwaway HOME. + + The same helper ``tests/test_fresh_install.py`` and ``tests/test_fleet_auth.py`` + use, kept here rather than imported so the three files cannot break each other. + """ + home = tmp_path / "install-home" + ainode_home = home / ".ainode" + home.mkdir(parents=True, exist_ok=True) + sysfs = tmp_path / "sys-class-net" + sysfs.mkdir(parents=True, exist_ok=True) + env = dict(os.environ) + env.update(HOME=str(home), AINODE_HOME=str(ainode_home), + AINODE_IMAGE="ghcr.io/getainode/ainode:9.9.9", + SYS_CLASS_NET=str(sysfs)) + for key in ("AINODE_PEERS", "HF_TOKEN", "AINODE_AUTH"): + env.pop(key, None) + env.update(env_extra or {}) + proc = subprocess.run(["bash", str(INSTALL_SH), "--dry-run"], + capture_output=True, text=True, timeout=180, env=env) + assert proc.returncode == 0, proc.stdout + proc.stderr + return ainode_home, proc + + +@pytest.mark.skipif(shutil.which("bash") is None, reason="needs bash") +def test_a_protected_install_prints_the_two_lines_an_operator_needs(tmp_path): + _, proc = _render_install(tmp_path) + + assert "ainode auth enable" in proc.stdout + assert "ainode auth user add --admin" in proc.stdout + assert "--password-stdin" in proc.stdout + + +@pytest.mark.skipif(shutil.which("bash") is None, reason="needs bash") +def test_an_open_install_says_nothing_about_a_login(tmp_path): + """With auth off the dashboard opens without one, so these lines would be + advice about a problem nobody has.""" + _, proc = _render_install(tmp_path, env_extra={"AINODE_AUTH": "off"}) + + assert "auth user add" not in proc.stdout + + +@pytest.mark.skipif(shutil.which("bash") is None, reason="needs bash") +def test_the_wrapper_asks_for_a_tty_only_when_it_has_one(tmp_path): + """``--password-stdin`` is the flag for a pipe, and a hardcoded ``docker exec + -it`` would kill the pipe before the CLI ran: docker answers "the input device + is not a TTY" and exits. Same failure ``ainode doctor --peer`` routes around.""" + ainode_home, _ = _render_install(tmp_path) + wrapper = (ainode_home / "ainode-wrapper").read_text() + + assert "docker exec -it" not in wrapper + assert '[ -t 0 ]' in wrapper + forward = wrapper[wrapper.index("forward_to_container()"):] + assert 'exec docker exec $exec_tty' in forward From 353e7fa0dc76486503369826cef7d179b082a8ca Mon Sep 17 00:00:00 2001 From: webdevtodayjason Date: Mon, 21 Sep 2026 18:56:52 -0500 Subject: [PATCH 4/4] Say why --password-stdin exists without naming a flag the wrapper no longer has Co-Authored-By: Claude Fable 5.1 --- ainode/cli/main.py | 14 ++++++++------ tests/test_cli_auth_users.py | 9 +++++---- 2 files changed, 13 insertions(+), 10 deletions(-) diff --git a/ainode/cli/main.py b/ainode/cli/main.py index 7dfd91aa..c8f0d131 100644 --- a/ainode/cli/main.py +++ b/ainode/cli/main.py @@ -945,10 +945,11 @@ def _save_store(store) -> None: def _read_new_password(args, label: str = "Password") -> str: """The new password: one read of stdin, or two prompts that must agree. - ``--password-stdin`` exists because ``getpass`` needs a TTY and the installer's - host wrapper runs ``docker exec -it``: interactive from a terminal, but not - from a script, a unit file or ``ssh host ainode ...``. Only the newline the - shell added is stripped, so a password may contain spaces. + ``--password-stdin`` exists because ``getpass`` needs a TTY, and a script, a + unit file or ``ssh host ainode ...`` has none. The installer's host wrapper + forwards a piped command without ``docker exec -t`` for the same reason, so + the pipe survives the container boundary. Only the newline the shell added is + stripped, so a password may contain spaces. """ from ainode.auth.replication import min_password_length @@ -2139,8 +2140,9 @@ def main(): # `auth user ...` is the DASHBOARD login (#261): a name and a password, which # is a different credential from an API key and managed on the cluster's # master. --password-stdin is the answer for anything with no terminal: getpass - # needs a TTY and the installer's wrapper runs `docker exec -it`, so the prompt - # works from a shell and not from a script, a unit or `ssh host ainode ...`. + # needs a TTY, so the prompt works from a shell and not from a script, a unit + # or `ssh host ainode ...`. The installer's wrapper allocates a TTY only when + # it has one, so a piped password reaches the container. auth_user = auth_sub.add_parser( "user", help="Dashboard accounts: add, list, remove, change a password") auth_user_sub = auth_user.add_subparsers(dest="user_action") diff --git a/tests/test_cli_auth_users.py b/tests/test_cli_auth_users.py index b3fcb864..7717f987 100644 --- a/tests/test_cli_auth_users.py +++ b/tests/test_cli_auth_users.py @@ -9,8 +9,9 @@ operator discovers instead of a test. 2. **Both ways of supplying a password.** Interactive is two ``getpass`` prompts that have to agree; ``--password-stdin`` is the answer for anything with no - terminal, which includes ``ssh host ainode ...`` and every script, because the - installer's host wrapper runs ``docker exec -it`` and getpass needs a TTY. + terminal, which includes ``ssh host ainode ...`` and every script, because + getpass needs a TTY. The installer's host wrapper had to stop hardcoding + ``docker exec -it`` for that flag to work at all, which is pinned at the bottom. 3. **A short password is refused** before anything is written. 4. **The last admin cannot be removed or disabled.** With auth on and no admin, the dashboard can only be opened by pasting an API key, which is the state @@ -223,8 +224,8 @@ def test_the_usage_line_names_the_new_commands(home, store, no_peer_calls): def test_password_stdin_is_documented_in_the_help(capsys): - """The host wrapper runs `docker exec -it`, so the prompt needs a terminal and - the flag is the documented way to work without one.""" + """getpass needs a terminal, and the flag is the documented way to work without + one, so the help text has to say so where an operator will read it.""" with patch.object(sys, "argv", ["ainode", "auth", "user", "add", "--help"]): with pytest.raises(SystemExit): cli.main()