diff --git a/README.md b/README.md index 0f839d1..a947c1c 100644 --- a/README.md +++ b/README.md @@ -33,3 +33,8 @@ skills/ Skills are thin trigger and workflow layers; the rule files are the source of truth. + +## Guides + +- [`docs/agent-fleet`](docs/agent-fleet/README.md): how to run many coding + agents on several machines from one orchestrator session. diff --git a/docs/agent-fleet/README.md b/docs/agent-fleet/README.md new file mode 100644 index 0000000..ca5c839 --- /dev/null +++ b/docs/agent-fleet/README.md @@ -0,0 +1,533 @@ +# Running a Two-Level Agent Fleet + +This page describes how the maintainer runs many coding agents at once, and +how to reproduce the setup. It is a guide, not a rule. Adapt it to your own +machines and budget. + +![Two-level agent fleet](agent-fleet.svg) + +## Architecture + +The fleet is a human plus two levels of agents. + +- **Level 0: the human.** You decide, approve releases and give instructions + in plain language. You talk only to the orchestrator. With Claude Code + [Remote Control](#remote-control-from-a-phone) you can also steer the same + orchestrator session from a phone. It is the same session, not a second + one. +- **Level 1: the orchestrator.** One Claude Code session on a laptop. It runs + rounds on a schedule, reads each agent's screen, dispatches tasks by + prompt, reviews and merges pull requests, and escalates decisions it should + not take. +- **Level 2: the machine agents.** Each remote machine runs + [herdr](https://herdr.dev/docs/), a terminal multiplexer for coding agents, + with several agents in its panes. Agents are Claude Code sessions (Opus or + Sonnet) or [pi](https://pi.dev) agents on cheaper models such as DeepSeek. + +Two shared services tie the levels together. + +- **GitHub.** Agents push branches and open pull requests. When work on an + issue starts, the issue is assigned to yourself (the person running the + fleet). Only the orchestrator merges. See + [GitHub access](#5-github-access) for how to enforce this. +- **A private state repository.** A small git repository holds the recovery + state, so a new orchestrator session can take over at any time. See + [Recovery state](#recovery-state). + +Each machine has a role. In the maintainer's fleet these are a quiet benchmark +host, an implementation host with a GPU, and a many-core scaling host. Adding +a machine adds a whole team of agents. + +## Reproduce the setup + +### 1. Network + +Install [Tailscale](https://tailscale.com/) on the laptop and on every +machine. The machines then reach each other on a private network and need no +open ports on the public internet. + +### 2. SSH key + +The orchestrator connects without a human at the keyboard. A passphrase does +not prevent this. Protect the key with a passphrase and load it into +ssh-agent once. After that, every connection uses the loaded key without a +prompt. herdr's own docs give the same advice for its background +connections (see +[Connecting machines](https://herdr.dev/docs/connecting-machines/)). + +```sh +ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519_fleet -C fleet +``` + +On macOS, let the keychain hold the passphrase, so the key is loaded again +after a reboot. Give each machine a Host alias in `~/.ssh/config` on the +laptop: + +```sshconfig +Host host-a + HostName host-a.your-tailnet.ts.net + User you + IdentityFile ~/.ssh/id_ed25519_fleet + IdentitiesOnly yes + AddKeysToAgent yes + UseKeychain yes +``` + +Then load the key once with `ssh-add --apple-use-keychain +~/.ssh/id_ed25519_fleet`. `UseKeychain` exists only on macOS (see Apple's +[TN2449][tn2449]); +on Linux, drop that line and run `ssh-add` once per login. +[`AddKeysToAgent`](https://man.openbsd.org/ssh_config.5) is standard +OpenSSH. + +Fallback: a dedicated key without a passphrase (`ssh-keygen -N ""`). It +needs no agent, but anyone who copies the file can log in to every machine. +Use it only if the agent setup does not work for you, and restrict it as +below. + +Restrict the key to the tailnet. Tailscale gives every device an address in +[`100.64.0.0/10`](https://tailscale.com/kb/1015/100.x-addresses). In +`~/.ssh/authorized_keys` on each machine, put the +[`from=`](https://man.openbsd.org/sshd.8#from=) option before the key: + +```text +from="100.64.0.0/10" ssh-ed25519 AAAA... fleet +``` + +Also turn off password login in `/etc/ssh/sshd_config` +(`PasswordAuthentication no` and `KbdInteractiveAuthentication no`, see +[sshd_config](https://man.openbsd.org/sshd_config.5)). Optionally, use +[Tailscale SSH](https://tailscale.com/kb/1193/tailscale-ssh) and +[access rules](https://tailscale.com/kb/1018/acls) to decide which devices +may connect to which machines. + +Check that `ssh host-a true` works without any prompt. + +### 3. herdr + +Install herdr on the laptop +([install docs](https://herdr.dev/docs/install/)): + +```sh +curl -fsSL https://herdr.dev/install.sh | sh +``` + +Then register each machine once, in an interactive terminal on the laptop: + +```sh +herdr machine add host-a +herdr machine list +``` + +`machine add` takes an SSH target, normally the Host alias from +`~/.ssh/config`. It must run in an interactive terminal, because it may ask +before it installs or replaces anything. If herdr is missing on the machine, +it offers to install it there. It then starts the remote background server +and saves a profile. The profile's label defaults to the SSH host without +any `user@` prefix, here `host-a`. That label (or the profile id) is what +`--machine` accepts. An arbitrary hostname does not work. Add +`--label ` to choose another label. See +[Connecting machines](https://herdr.dev/docs/connecting-machines/). + +The remote server ignores SIGHUP, so it keeps running after your SSH session +ends ([Persistence](https://herdr.dev/docs/persistence-remote/)). Nothing +restarts it after a reboot. A reboot also ends the agents in its panes; +herdr restores the saved layout, not the running processes +([Session state](https://herdr.dev/docs/session-state/)). If a machine +reboots often, run the server under systemd. The CLI reference provides +`herdr server` for this: it "runs the headless server explicitly" for +supervised setups. A minimal user unit, `~/.config/systemd/user/herdr.service` +on the machine: + +```ini +[Unit] +Description=herdr server + +[Service] +ExecStart=%h/.local/bin/herdr server +Restart=on-failure + +[Install] +WantedBy=default.target +``` + +Adjust the path to the output of `command -v herdr`. Enable it with +`systemctl --user enable --now herdr.service`, and run +[`loginctl enable-linger`][loginctl] +once so the user manager starts at boot. Panes inherit the service +environment, so add `Environment=` lines if tools are missing from `PATH`. +Enable the unit before `machine add`, or stop the server that `machine add` +started (`herdr server stop` on the machine) before you enable it. herdr's +docs do not ship such a unit. Treat it as a starting point. + +Adding and removing machines is the human's job. The orchestrator only uses +machines that are already registered. + +### 4. Orchestrator commands + +The orchestrator drives every machine with `herdr --machine