Skip to content

feat(backend): add remote-machine ssh supervision backend - #1

Open
farkas wants to merge 2 commits into
mainfrom
fm/fm-ssh-backend
Open

feat(backend): add remote-machine ssh supervision backend#1
farkas wants to merge 2 commits into
mainfrom
fm/fm-ssh-backend

Conversation

@farkas

@farkas farkas commented Jul 27, 2026

Copy link
Copy Markdown
Owner

What

Adds an experimental ssh runtime backend so the first mate can spawn and supervise a crewmate on another machine over SSH - a second Claude account, or Codex/Grok/OpenCode/Kimi - in parallel with the local fleet. Qwen and other open-source models work by pointing an existing harness (OpenCode/Kimi) at a Qwen provider; no harness code is added.

Small and targeted: a new adapter + minimal fm-backend.sh registration + one config file + docs + tests + the state-signaling piece. No changes to fm-spawn.sh or the other backends.

Design: reuse tmux-on-the-remote, don't reinvent

The adapter is thin. Every remote operation reuses the exact tmux command sequences in bin/backends/tmux.sh and the shared composer / verify-and-retry submit core in bin/fm-tmux-lib.sh, run against the remote's tmux through an SSH shim:

  • fm_backend_ssh_via_tmux runs a tmux-adapter function inside a subshell that defines a local tmux shell function. That function shadows the tmux binary for every call the tmux adapter makes and forwards it to ssh <host> <remote-tmux> ... with each argument shell-quoted, so argument boundaries (e.g. send-keys -l "text with spaces") survive SSH's remote-shell re-parse.
  • The shim lives only inside the subshell, so it never affects a caller that also drives a local tmux.

Result: capture, send (keys + verified text submit), liveness (agent_state), and kill all work over SSH, and flow through fm-peek / fm-send / fm-watch / fm-teardown with zero changes to those scripts (they already dispatch through fm-backend.sh).

State signaling: MASTER-PULL only (master -> slave)

Trust direction is master -> slave only. The slave (the remote machine) has NO write access to the master and NEVER initiates a connection back to it. This is the hard security requirement, and it is enforced in code and tests.

The remote worker's state/<id>.status and .turn-ended still have to reach the local watcher, which reads local files. The mechanism:

  • The remote worker writes its own <id>.status / <id>.turn-ended to files LOCAL to the remote, under remote_state_dir - pure mkdir / cat / printf, no ssh on the remote's side at all.
  • A master-side poller (bin/fm-ssh-pull.sh -> fm_backend_ssh_pull_state) SSHes OUT in one outbound connection, reads those remote files, and reflects them into the exact local state/ files the watcher already scans - appending only not-yet-seen status lines and touching turn-end only when a monotonic remote counter advances, so there are no duplicate wakes. No watcher change is needed.

Every SSH connection is initiated by the master, outbound to the slave. The master enables no Remote Login, authorizes no remote key in its authorized_keys, and exposes no inbound path. Config drops local_host / local_ssh_opts and adds remote_state_dir; provisioning is entirely on the remote.

An earlier revision of this branch used a reverse-SSH write-back (the remote ran ssh <local_host> "cat >> ..." into the master's state/). That gave the slave write access to the master and required the master to accept an inbound connection, which breaks the trust model. It was removed entirely and replaced with the master-pull design above.

Registration + boundaries

  • bin/backends/ssh.sh: config parsing (config/ssh-backend), the SSH shim, the backend contract ops (capture / send / kill / current-path / current-command / composer-state / agent-state / target-exists), remote container/create-task primitives, and the remote-local state-write builders.
  • bin/fm-backend.sh: registers ssh as a KNOWN backend - explicit-only, never auto-detected, and not spawn-capable through fm-spawn.sh. Remote-worktree spawn is a documented boundary (fm-spawn's local-worktree and turn-end-hook assumptions do not hold for a remote task), so the adapter owns supervision plus the create/container primitives.
  • config/ssh-backend is LOCAL / gitignored and not inherited by secondmate homes; docs/ssh-backend.md owns config, the master-pull rationale, provisioning/auth, and the Qwen-provider note.

Tests

  • tests/fm-backend-ssh.test.sh: fake-ssh unit/conformance (CI) plus an opt-in live smoke.
    • test_ssh_master_pull_state: one outbound pull reflects remote status/turn-end into local state/ byte-for-byte; a re-pull adds no duplicate line (idempotent).
    • test_ssh_master_pull_quoting_safety: a remote_state_dir with a space and single quote, and a status line with shell metacharacters, land verbatim (no injection).
    • test_ssh_no_reverse_channel: asserts the reverse-SSH builders are removed and the remote-write commands are local-only (no ssh).
  • Verified live against a real remote (tmux 3.7b over Tailscale): the remote-write commands produced the remote's signal files, one outbound pull wrote them into local state/, a re-pull added no duplicate, and the remote initiated zero connections back.

farkas added 2 commits July 27, 2026 22:09
Add an experimental `ssh` runtime backend that runs the reference tmux
adapter's operations against a REMOTE host's tmux over ssh, so the first
mate can supervise a crewmate on a second machine (a separate account, or
Codex/Grok/OpenCode/Kimi) in parallel with the local fleet.

It is a thin adapter, not a new backend: every remote operation reuses the
exact tmux command sequences in bin/backends/tmux.sh and the shared
composer/verify-and-retry submit core in bin/fm-tmux-lib.sh, run against the
remote's tmux through an ssh shim (a subshell-local `tmux` function that
shell-quotes each argument so boundaries survive ssh's remote-shell
re-parse). The shim never leaks to a caller that also drives a local tmux.

- bin/backends/ssh.sh: config parsing (config/ssh-backend), the ssh shim,
  the backend contract ops (capture/send/kill/current-path/current-command/
  composer-state/agent-state/target-exists), remote container/create-task
  primitives, and reverse-ssh state-signaling command builders.
- bin/fm-backend.sh: register ssh as a KNOWN backend (not spawn-capable via
  fm-spawn, never auto-detected) with source, dispatch, and required-tools
  arms; supervision then flows through fm-peek/fm-send/fm-watch/fm-teardown
  with no change to those scripts.
- State signaling uses reverse-ssh write-back into the local state/ files, so
  the watcher wakes with no change; graceful degradation to forward-ssh
  polling when the reverse key is not yet authorized.
- config/ssh-backend is LOCAL/gitignored; docs/ssh-backend.md owns config,
  signaling rationale, provisioning/auth, and the Qwen-provider note;
  configuration.md/AGENTS.md/README point to it.
- tests/fm-backend-ssh.test.sh: fake-ssh unit/conformance (CI) plus an
  opt-in live smoke; verified live against a real remote (tmux 3.7b over
  Tailscale), recorded in docs/verification/runtime-backends.md.

Remote-worktree spawn through fm-spawn.sh is a deliberate, documented
boundary (fm-spawn's local-worktree and turn-end-hook assumptions do not
hold for a remote task); the adapter owns supervision plus the create
primitives today.
The remote-machine ssh backend signaled state with a reverse-ssh write-back:
the remote worker ran `ssh <local_host> ...` to append status and touch
turn-end directly into the master's state/ over the tailnet.
That gives the slave write access to the master and requires the master to
accept an inbound connection, which breaks the trust model.

Replace it with master-pull, master -> slave only:
- The remote worker writes its own <id>.status / <id>.turn-ended to files
  LOCAL to the remote under remote_state_dir (pure mkdir/cat/printf, no ssh).
- A master-side poller (bin/fm-ssh-pull.sh -> fm_backend_ssh_pull_state)
  SSHes OUT in one connection, reads those files, and reflects them into the
  local state/ files the watcher already scans - appending only not-yet-seen
  status lines and touching turn-end only when a monotonic remote counter
  advances, so no duplicate wakes. No watcher change is needed.

The remote never connects to the master: config drops local_host/local_ssh_opts
and adds remote_state_dir; provisioning no longer enables Remote Login or
authorizes a remote key on the master.

Re-verified live against a real remote (tmux + a trivial process) over
Tailscale: the remote-write commands produced the remote's signal files, one
outbound pull wrote them byte-for-byte into local state/, a re-pull added no
duplicate line, and the remote initiated zero connections back.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant