feat(backend): add remote-machine ssh supervision backend - #1
Open
farkas wants to merge 2 commits into
Open
Conversation
Add an experimental `ssh` runtime backend that runs the reference tmux adapter's operations against a REMOTE host's tmux over ssh, so the first mate can supervise a crewmate on a second machine (a separate account, or Codex/Grok/OpenCode/Kimi) in parallel with the local fleet. It is a thin adapter, not a new backend: every remote operation reuses the exact tmux command sequences in bin/backends/tmux.sh and the shared composer/verify-and-retry submit core in bin/fm-tmux-lib.sh, run against the remote's tmux through an ssh shim (a subshell-local `tmux` function that shell-quotes each argument so boundaries survive ssh's remote-shell re-parse). The shim never leaks to a caller that also drives a local tmux. - bin/backends/ssh.sh: config parsing (config/ssh-backend), the ssh shim, the backend contract ops (capture/send/kill/current-path/current-command/ composer-state/agent-state/target-exists), remote container/create-task primitives, and reverse-ssh state-signaling command builders. - bin/fm-backend.sh: register ssh as a KNOWN backend (not spawn-capable via fm-spawn, never auto-detected) with source, dispatch, and required-tools arms; supervision then flows through fm-peek/fm-send/fm-watch/fm-teardown with no change to those scripts. - State signaling uses reverse-ssh write-back into the local state/ files, so the watcher wakes with no change; graceful degradation to forward-ssh polling when the reverse key is not yet authorized. - config/ssh-backend is LOCAL/gitignored; docs/ssh-backend.md owns config, signaling rationale, provisioning/auth, and the Qwen-provider note; configuration.md/AGENTS.md/README point to it. - tests/fm-backend-ssh.test.sh: fake-ssh unit/conformance (CI) plus an opt-in live smoke; verified live against a real remote (tmux 3.7b over Tailscale), recorded in docs/verification/runtime-backends.md. Remote-worktree spawn through fm-spawn.sh is a deliberate, documented boundary (fm-spawn's local-worktree and turn-end-hook assumptions do not hold for a remote task); the adapter owns supervision plus the create primitives today.
The remote-machine ssh backend signaled state with a reverse-ssh write-back: the remote worker ran `ssh <local_host> ...` to append status and touch turn-end directly into the master's state/ over the tailnet. That gives the slave write access to the master and requires the master to accept an inbound connection, which breaks the trust model. Replace it with master-pull, master -> slave only: - The remote worker writes its own <id>.status / <id>.turn-ended to files LOCAL to the remote under remote_state_dir (pure mkdir/cat/printf, no ssh). - A master-side poller (bin/fm-ssh-pull.sh -> fm_backend_ssh_pull_state) SSHes OUT in one connection, reads those files, and reflects them into the local state/ files the watcher already scans - appending only not-yet-seen status lines and touching turn-end only when a monotonic remote counter advances, so no duplicate wakes. No watcher change is needed. The remote never connects to the master: config drops local_host/local_ssh_opts and adds remote_state_dir; provisioning no longer enables Remote Login or authorizes a remote key on the master. Re-verified live against a real remote (tmux + a trivial process) over Tailscale: the remote-write commands produced the remote's signal files, one outbound pull wrote them byte-for-byte into local state/, a re-pull added no duplicate line, and the remote initiated zero connections back.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds an experimental
sshruntime backend so the first mate can spawn and supervise a crewmate on another machine over SSH - a second Claude account, or Codex/Grok/OpenCode/Kimi - in parallel with the local fleet. Qwen and other open-source models work by pointing an existing harness (OpenCode/Kimi) at a Qwen provider; no harness code is added.Small and targeted: a new adapter + minimal
fm-backend.shregistration + one config file + docs + tests + the state-signaling piece. No changes tofm-spawn.shor the other backends.Design: reuse tmux-on-the-remote, don't reinvent
The adapter is thin. Every remote operation reuses the exact tmux command sequences in
bin/backends/tmux.shand the shared composer / verify-and-retry submit core inbin/fm-tmux-lib.sh, run against the remote's tmux through an SSH shim:fm_backend_ssh_via_tmuxruns a tmux-adapter function inside a subshell that defines a localtmuxshell function. That function shadows the tmux binary for every call the tmux adapter makes and forwards it tossh <host> <remote-tmux> ...with each argument shell-quoted, so argument boundaries (e.g.send-keys -l "text with spaces") survive SSH's remote-shell re-parse.Result: capture, send (keys + verified text submit), liveness (
agent_state), and kill all work over SSH, and flow throughfm-peek/fm-send/fm-watch/fm-teardownwith zero changes to those scripts (they already dispatch throughfm-backend.sh).State signaling: MASTER-PULL only (master -> slave)
Trust direction is master -> slave only. The slave (the remote machine) has NO write access to the master and NEVER initiates a connection back to it. This is the hard security requirement, and it is enforced in code and tests.
The remote worker's
state/<id>.statusand.turn-endedstill have to reach the local watcher, which reads local files. The mechanism:<id>.status/<id>.turn-endedto files LOCAL to the remote, underremote_state_dir- puremkdir/cat/printf, nosshon the remote's side at all.bin/fm-ssh-pull.sh->fm_backend_ssh_pull_state) SSHes OUT in one outbound connection, reads those remote files, and reflects them into the exact localstate/files the watcher already scans - appending only not-yet-seen status lines and touching turn-end only when a monotonic remote counter advances, so there are no duplicate wakes. No watcher change is needed.Every SSH connection is initiated by the master, outbound to the slave. The master enables no Remote Login, authorizes no remote key in its
authorized_keys, and exposes no inbound path. Config dropslocal_host/local_ssh_optsand addsremote_state_dir; provisioning is entirely on the remote.An earlier revision of this branch used a reverse-SSH write-back (the remote ran
ssh <local_host> "cat >> ..."into the master'sstate/). That gave the slave write access to the master and required the master to accept an inbound connection, which breaks the trust model. It was removed entirely and replaced with the master-pull design above.Registration + boundaries
bin/backends/ssh.sh: config parsing (config/ssh-backend), the SSH shim, the backend contract ops (capture / send / kill / current-path / current-command / composer-state / agent-state / target-exists), remote container/create-task primitives, and the remote-local state-write builders.bin/fm-backend.sh: registerssshas a KNOWN backend - explicit-only, never auto-detected, and not spawn-capable throughfm-spawn.sh. Remote-worktree spawn is a documented boundary (fm-spawn's local-worktree and turn-end-hook assumptions do not hold for a remote task), so the adapter owns supervision plus the create/container primitives.config/ssh-backendis LOCAL / gitignored and not inherited by secondmate homes;docs/ssh-backend.mdowns config, the master-pull rationale, provisioning/auth, and the Qwen-provider note.Tests
tests/fm-backend-ssh.test.sh: fake-ssh unit/conformance (CI) plus an opt-in live smoke.test_ssh_master_pull_state: one outbound pull reflects remote status/turn-end into localstate/byte-for-byte; a re-pull adds no duplicate line (idempotent).test_ssh_master_pull_quoting_safety: aremote_state_dirwith a space and single quote, and a status line with shell metacharacters, land verbatim (no injection).test_ssh_no_reverse_channel: asserts the reverse-SSH builders are removed and the remote-write commands are local-only (nossh).state/, a re-pull added no duplicate, and the remote initiated zero connections back.