Agent CA rotation: per-handshake certificate reload, in-band renewal, bundle-ready trust store

CLAUDE.md

@@ -32,11 +32,13 @@ cmd/network-agent/    # CLI entry point (main.go)
 internal/
   protocol/           # Binary wire format (encode/decode) — no external deps
   acl/                # IP/CIDR access list parser
-  config/             # File paths, TLS config, env vars
-  caroot/             # Embedded CA certificate (ca.crt)
+  config/             # File paths, TLS config, CA bundle, env vars
+  caroot/             # Embedded CA bundle (ca.crt) + bundle parsing
+  certrenew/          # Validate + atomically install a renewed client cert
   tunnel/             # mTLS connection + goroutine lifecycle
   setup/              # Interactive setup wizard
   daemon/             # PID file, background process management
+  testca/             # Test-only: DeployHQ-shaped certificate fixtures
 ```
 
 ## Protocol (Wire Format)
@@ -50,7 +52,23 @@ Frame layout — must not change without coordinating with the server side:
 `total_frame_size` includes the 2-byte length field itself.
 
 Commands: `CREATE_REQUEST(1)`, `CREATE_RESPONSE(2)`, `DESTROY(3)`, `DATA(4)`,
-`REJECT(5)`, `RECONNECT(6)`, `KEEPALIVE(7)`.
+`REJECT(5)`, `RECONNECT(6)`, `KEEPALIVE(7)`, `RENEW_REQUEST(8)`,
+`RENEW_RESPONSE(9)`.
+
+`RENEW_REQUEST` / `RENEW_RESPONSE` carry in-band certificate renewal and are
+implemented identically in the DeployHQ backend and the Ruby `deploy-agent`
+gem — the three must not drift:
+
+- `RENEW_REQUEST` (agent → server), sent once per connection immediately after
+  the handshake. Payload: UTF-8 `"go/<version>"` (the gem sends `"ruby/<version>"`).
+  Nothing else on the wire identifies the implementation or version.
+- `RENEW_RESPONSE` (server → agent, never unsolicited). Payload
+  `[status:1][body]`: `0` RENEWED (body = new client certificate PEM), `1`
+  CURRENT (no body), `2` ERROR (body = UTF-8 message, logged and ignored).
+
+Unknown command bytes must stay tolerated — the `default:` branch in
+`server_conn.go` only logs. Deployed agents rely on that to survive new
+commands.
 
 Reference implementations:
 - Agent side: `../deploy-agent/lib/deploy_agent/server_connection.rb`
@@ -73,6 +91,11 @@ Reference implementations:
 | `DEPLOY_AGENT_PROXY_IP`        | `agent.deployhq.com`                               |
 | `DEPLOY_AGENT_CERTIFICATE_URL` | `https://api.deployhq.com/api/v1/agents/create`   |
 | `DEPLOY_AGENT_NOVERIFY`        | unset (set to skip TLS verification in dev)        |
+| `DEPLOY_AGENT_CA_FILE`         | unset (uses `internal/caroot`'s embedded bundle)   |
+
+`DEPLOY_AGENT_CA_FILE` replaces the embedded CA bundle with a PEM file on disk.
+Read through `config.CACert(caroot.CACert)`, which both `run` and `check` go
+through. An unreadable or empty file is an error, never a silent fall back.
 
 ## Testing
 
@@ -100,5 +123,11 @@ GOOS=windows GOARCH=amd64 CGO_ENABLED=0 go build ./cmd/network-agent
 
 - Keep the protocol package dependency-free — it is the ground truth for wire format
 - The access list is reloaded on each reconnect (see `tunnel/agent.go`) — no restart needed for ACL changes
+- The client certificate is reloaded from disk on **every handshake**
+  (`config.NewTLSConfig`'s `GetClientCertificate`), not cached in
+  `tls.Config.Certificates`. One `*tls.Config` is built at start-up and reused
+  for every reconnect, so caching it there would mean a renewed `agent.crt`
+  never took effect without a process restart — the thing certificate renewal
+  exists to avoid. Do not "optimise" this back into `Certificates`.
 - Daemon backgrounding uses re-exec (not fork) — see `internal/daemon/`
 - Build tags `!windows` / `windows` separate Unix/Windows daemon code

README.md

@@ -70,6 +70,33 @@ Add `-v` / `--verbose` before the command for debug logging.
 10.0.0.0/8
 ```
 
+## Upgrading
+
+```bash
+network-agent update && network-agent restart
+```
+
+`update` replaces the binary in place; `restart` is what makes the running
+agent pick it up. Both steps are needed — there is no background update check.
+
+## Certificate renewal
+
+DeployHQ is rotating the certificate authority behind the agent connection. The
+agent handles its side automatically: on each connection it tells DeployHQ which
+version it is running, and if DeployHQ has re-issued its client certificate the
+agent validates the new one, replaces `~/.deploy/agent.crt`, and reconnects to
+start using it. Nothing is written unless the new certificate pairs with the
+existing private key, keeps the same identity, and is signed by a trusted CA —
+and the replacement is atomic, so a failure never leaves a half-written
+certificate behind.
+
+Nothing is required of you beyond running a recent version. There is no
+interruption to deployments, no new claim code, and no change to
+`~/.deploy/agent.key`, which never leaves the machine.
+
+`network-agent check` prints the certificate authorities the binary trusts and
+when they expire.
+
 ## Migration from Ruby gem
 
 Users already running the Ruby gem can migrate in place — the Go binary uses the
@@ -88,6 +115,7 @@ To roll back: `network-agent stop`, then `gem install deploy-agent` and `deploy-
 | `DEPLOY_AGENT_PROXY_IP`      | `agent.deployhq.com`                             | Agent server hostname/IP            |
 | `DEPLOY_AGENT_CERTIFICATE_URL` | `https://api.deployhq.com/api/v1/agents/create` | Certificate provisioning endpoint   |
 | `DEPLOY_AGENT_NOVERIFY`      | unset                                            | Set to skip TLS server verification |
+| `DEPLOY_AGENT_CA_FILE`       | unset (uses the bundled CAs)                     | Path to a PEM bundle of certificate authorities to trust instead of the ones built into the binary. Unreadable or empty is an error, not a fall back. |
 
 ## Building from source
 

cmd/network-agent/main.go

@@ -153,12 +153,27 @@ func cmdRun(paths config.Paths, verbose bool) {
 	}
 	log := slog.New(slog.NewTextHandler(os.Stdout, &slog.HandlerOptions{Level: level}))
 
-	tlsCfg, err := config.NewTLSConfig(paths, caroot.CACert, config.VerifyTLS())
+	caCert, err := config.CACert(caroot.CACert)
+	if err != nil {
+		log.Error("CA certificate error", "err", err)
+		os.Exit(1)
+	}
+
+	tlsCfg, err := config.NewTLSConfig(paths, caCert, config.VerifyTLS())
 	if err != nil {
 		log.Error("TLS configuration error", "err", err)
 		os.Exit(1)
 	}
 
+	// The same trust anchors gate in-band certificate renewal: a certificate
+	// the agent would not accept from the server is not one it will install
+	// for itself.
+	caRoots, err := config.NewCertPool(caCert)
+	if err != nil {
+		log.Error("CA certificate error", "err", err)
+		os.Exit(1)
+	}
+
 	// Handle SIGTERM / SIGINT for clean shutdown
 	sigc := make(chan os.Signal, 1)
 	signal.Notify(sigc, syscall.SIGTERM, syscall.SIGINT)
@@ -173,7 +188,13 @@ func cmdRun(paths config.Paths, verbose bool) {
 	// If PID already written by the parent (start), this is a no-op for our PID.
 	_ = daemon.WritePID(paths.PID)
 
-	if err := tunnel.RunAgent(tlsCfg, paths, log); err != nil {
+	opts := tunnel.Options{
+		Version: Version,
+		Paths:   paths,
+		CARoots: caRoots,
+	}
+
+	if err := tunnel.RunAgent(tlsCfg, opts, log); err != nil {
 		log.Error("agent stopped with error", "err", err)
 		daemon.RemovePID(paths.PID)
 		os.Exit(1)
@@ -260,11 +281,33 @@ func cmdCheck(paths config.Paths) {
 		fmt.Println(")")
 	}
 
+	// --- CA bundle ---------------------------------------------------------
+	// Which CAs this binary trusts is the thing a CA rotation turns on, so
+	// surface it rather than leaving operators to inspect the binary.
+	caCert, caErr := config.CACert(caroot.CACert)
+	if caErr != nil {
+		fmt.Printf("  CA bundle     FAIL  %v\n", caErr)
+		ok = false
+	} else if cas, err := caroot.ParseBundle(caCert); err != nil {
+		fmt.Printf("  CA bundle     FAIL  %v\n", err)
+		ok = false
+	} else {
+		source := "embedded"
+		if override := os.Getenv(config.CAFileEnv); override != "" {
+			source = override
+		}
+		fmt.Printf("  CA bundle     OK    %d CA(s) from %s\n", len(cas), source)
+		for _, c := range cas {
+			fmt.Printf("                      %s (expires %s)\n",
+				c.Subject.CommonName, c.NotAfter.UTC().Format("2006-01-02"))
+		}
+	}
+
 	// --- Connectivity ------------------------------------------------------
 	serverAddr := config.ServerHost() + ":" + config.ServerPort
 	fmt.Printf("  Connectivity  ")
 
-	tlsCfg, err := config.NewTLSConfig(paths, caroot.CACert, config.VerifyTLS())
+	tlsCfg, err := config.NewTLSConfig(paths, caCert, config.VerifyTLS())
 	if err != nil {
 		fmt.Printf("FAIL  TLS config error: %v\n", err)
 		ok = false

internal/caroot/ca.crt

@@ -31,3 +31,33 @@ l4CPcGbB0L8yyIhGwiEfrZpjx6hOelX1daG8QPTvSSYpB6ODtQeb3zpDf8vU8M7T
 oAwG8/0g1Owh/a970vIKu4TBa4D2IiCfA3KPWlsIUSoeu9uBTKmUQ0Raa0AhZWPv
 JI4XgcL63KznYzLm0BOxvTYMxDfn7fs=
 -----END CERTIFICATE-----
+-----BEGIN CERTIFICATE-----
+MIIFGzCCAwOgAwIBAgIBATANBgkqhkiG9w0BAQsFADAvMRowGAYDVQQDDBFEZXBs
+b3lIUSBBZ2VudCBDQTERMA8GA1UECgwIRGVwbG95SFEwHhcNMjYwOTE1MDczODI4
+WhcNMzYwOTE1MDczODI4WjAvMRowGAYDVQQDDBFEZXBsb3lIUSBBZ2VudCBDQTER
+MA8GA1UECgwIRGVwbG95SFEwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoIC
+AQC4bMAQgGZC7ZUO9Wq2+rWodOXL4IZ6Ae+Dfce+SlTsQUx5HrYrmMFTE9NJTxcE
+dGYjAVVwpdvgT8flFKRiQa7j1xVOLr73ra7Uro0ZGHw1TOBFFAhXLZ25oXT8SsB7
+1aaRZksZAPGR1ZkWRvKqUkRge9LERtcmZDOSeKyf3i3PQp3AaODA4tRwsQnyKWoA
+6h9Ruk9jnHqNRapiEjjdkubKKRjgU28IO0Abom5dRsDvvFYVquaQmud745kdaxDr
+fQA9Kyz8lzNSglmTHACjzxtX1AGRpn5yLO7PqaV0M/QqnKZhaKW+tzP3pWdsDZ8v
+RBGDhyVYeyHYSPAEOgsRdTqQYpnXkR1FLLS/FCFQ/UgyWFStFVv3l9mzq/fQLr/O
+sdPzG9KNNTz4OqMrGOeRMcBXsKQJjIRS6TSzggPagcxT2iPRHCGQknko8/O3IyM1
+9YIsLm7kxw2oEIzhHx8Wv3s+SafJr4cuTu1n9RfyP3LmU/C6GbfrH/jDvVuLvFdH
+9AY55RNCfCgCKWjqDeYrzGxIcuoZ57H6DJlGouw0MiSg4hUUqNnM1EQG410bJouA
+itprB7FIGcZ9uHQvpEgs8NP/DTRsmUP+51Vdp6SLbFSNrt5v/APDrd3ya8R3f7g0
+XVLFl/jxNM4Ki+L6ZGwRz4SYThew0OZXsezbnYiGEro1JwIDAQABo0IwQDAdBgNV
+HQ4EFgQUgnCdcTQCEbmADry95bq7UAoX71AwDwYDVR0TAQH/BAUwAwEB/zAOBgNV
+HQ8BAf8EBAMCAQYwDQYJKoZIhvcNAQELBQADggIBAHaN63Zqs4W5FTTqQPYnkMH0
+/hMzRVVcQ8tZng6rGy4HdONixQ2XOwFOCMbl8uqRB/ihyLbwiLIVgsqG3wMEIiqV
+lxNDYerUtdBolam/lqA371d18lEqKFiFfncCRrd+AzTMcYnYZZK026smx1JCxDCP
+t1rB/T9Lv0g2aRyWTKsKBeI5GFOIEIHScP1k/R7549kNEwerFKX5USWszDfnouiV
+hSz9UjlY1vQQ//qYgimevJxlGudt2b+73+BkT5THdNKBxeyn02/DbhaH/1zafE/7
+lxAKmN94HkAOM2TD27ZPRDIK4jmKK/Hx03X0kYD+KL/9Mr/4PIjlObRFF8BUawp1
+Kpn/Caw4gNz76IujGXGvJp9FxsZUscRRRdikIZ3AtR0/6Wyf8fFaAd89DoBGSzLM
+mRynJbnK5ROlVzvxX43ow7sBLtoh1+/RCsWcCVgOC2EMCW2n4b5M0USv7FBrhPie
+F50NkMqsTtEoxw8kcKvLd4FbPvWuIxPsXzxlnFaHTbVgTevgL6O8IVeWLqdqwFGK
+owB7z1tEQTN6sV4+fDDtaVQFCalEV9QPAolVLYdBUApc866heDHmzW85tkq2zoDG
+2cj80hCcI42DPvrb9nEcIcZiFZDCcIlDjRf6kDmqBW4oXr5V1yAxrC+i0UA2+LWj
+NVgD1tDYCLJlNY3GA6j8
+-----END CERTIFICATE-----

internal/caroot/caroot.go

@@ -1,7 +1,53 @@
 // Package caroot embeds the DeployHQ CA certificate used to verify the agent server.
+//
+// ca.crt is a PEM bundle, not necessarily a single certificate: during a CA
+// rotation it holds both the outgoing and the incoming CA so that one binary
+// verifies an agent server presenting a leaf from either.
 package caroot
 
-import _ "embed"
+import (
+	"crypto/x509"
+	"encoding/pem"
+	"fmt"
+
+	_ "embed"
+)
 
 //go:embed ca.crt
 var CACert []byte
+
+// Certificates parses every certificate in the embedded bundle, in file order.
+//
+// It exists so the number of trusted CAs is observable — by `network-agent
+// check`, by tests, and by anyone verifying that a release actually shipped the
+// bundle it was supposed to.
+func Certificates() ([]*x509.Certificate, error) {
+	return ParseBundle(CACert)
+}
+
+// ParseBundle parses every CERTIFICATE block in a PEM bundle. Non-certificate
+// blocks are skipped; any unparsable certificate block, or a bundle holding no
+// certificates at all, is an error.
+func ParseBundle(pemData []byte) ([]*x509.Certificate, error) {
+	var certs []*x509.Certificate
+	rest := pemData
+	for {
+		var block *pem.Block
+		block, rest = pem.Decode(rest)
+		if block == nil {
+			break
+		}
+		if block.Type != "CERTIFICATE" {
+			continue
+		}
+		cert, err := x509.ParseCertificate(block.Bytes)
+		if err != nil {
+			return nil, fmt.Errorf("parsing certificate %d in bundle: %w", len(certs)+1, err)
+		}
+		certs = append(certs, cert)
+	}
+	if len(certs) == 0 {
+		return nil, fmt.Errorf("no certificates found in bundle")
+	}
+	return certs, nil
+}

internal/caroot/caroot_test.go

@@ -0,0 +1,56 @@
+package caroot_test
+
+import (
+	"testing"
+	"time"
+
+	"github.com/deployhq/network-agent/internal/caroot"
+)
+
+// TestEmbeddedBundleParses is the release-time guard for the CA bundle: it
+// proves the embedded PEM is well formed and reports how many CAs it holds, so
+// appending the new DeployHQ CA before a release is a visible, verifiable
+// change rather than an unchecked assumption.
+func TestEmbeddedBundleParses(t *testing.T) {
+	certs, err := caroot.Certificates()
+	if err != nil {
+		t.Fatalf("parsing embedded CA bundle: %v", err)
+	}
+	if len(certs) < 1 {
+		t.Fatal("embedded CA bundle holds no certificates")
+	}
+
+	t.Logf("embedded CA bundle holds %d certificate(s)", len(certs))
+	for i, c := range certs {
+		t.Logf("  [%d] subject=%q issuer=%q serial=%s notAfter=%s isCA=%v",
+			i, c.Subject.String(), c.Issuer.String(), c.SerialNumber,
+			c.NotAfter.UTC().Format(time.RFC3339), c.IsCA)
+
+		if !c.IsCA {
+			t.Errorf("[%d] %q is not a CA certificate", i, c.Subject.CommonName)
+		}
+	}
+}
+
+// TestParseBundleRejectsGarbage keeps ParseBundle honest: a bundle that does
+// not parse must be an error, never an empty-but-successful result that would
+// ship a binary trusting nothing.
+func TestParseBundleRejectsGarbage(t *testing.T) {
+	tests := []struct {
+		name string
+		data []byte
+	}{
+		{"empty", nil},
+		{"not pem", []byte("hello, world")},
+		{"pem but not a certificate", []byte("-----BEGIN RSA PRIVATE KEY-----\nZm9v\n-----END RSA PRIVATE KEY-----\n")},
+		{"corrupt certificate block", []byte("-----BEGIN CERTIFICATE-----\nZm9v\n-----END CERTIFICATE-----\n")},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			if _, err := caroot.ParseBundle(tt.data); err == nil {
+				t.Error("expected an error, got none")
+			}
+		})
+	}
+}