feat: pre-warm tailscale path to k3s server on game node start

[Flegma]

Problem

When a game/GPU node reboots, the Tailscale direct path to the k3s server can come up one-way (tx>0, rx 0). tailscaled self-reports healthy (BackendState=Running, empty Health), so the existing 5stack-tailscale-state-check passes every cycle, yet the node cannot actually reach the control plane. k3s-agent then loops on failed to get CA certs ... context deadline exceeded, never joins the cluster, no pods (including game-server-node-connector) get scheduled, and the node shows Offline in the panel (the connector's 30s Redis heartbeat never fires, so the API's 90s TTL flips it offline).

Observed on a live GPU node: stuck ~27 minutes after a reboot. A manual tailscale ping <server> re-established the path and k3s-agent recovered on its own within one retry.

Fix

Add a best-effort ExecStartPre drop-in (tailscale-prewarm.conf) that runs before k3s starts: if K3S_URL is set, it does a bounded tailscale ping to the server, forcing Tailscale's NAT traversal to (re)establish the direct path bidirectionally, so k3s-agent never validates against a dead one-way path.

• Written to whichever k3s unit dir the node actually uses (k3s-agent.service.d on agents), via a one-line check for k3s-agent.service.env.
• No-op on server nodes (no K3S_URL, nothing upstream to reach), so it is safe regardless of role.
• Prefixed with - and wrapped in || true so a failed pre-warm can never block k3s from starting.
• Does not touch the existing 5stack-tailscale-state-check logic or the other drop-ins.

Testing

• Validated the exact parse + ping on a live agent node: K3S_URL=https://100.90.141.113:6443 gives host=100.90.141.113, then pong ... 21ms.
• Installed the same drop-in on that live node: systemd registered the ExecStartPre, systemd-analyze verify was clean, and k3s-agent stayed active (not restarted).
• bash -n clean; shellcheck reports no new findings (only pre-existing ones).

Scope / follow-ups

Deliberately minimal. Two related things intentionally left out:

1. The periodic 5stack-tailscale-state-check still only inspects tailscaled's self-reported health, so it will not detect a one-way path during steady state (only the pre-warm at start guards it). A functional reachability probe (TCP to the k3s server) in that check would let a mid-run degradation self-heal too.
2. The existing resilience drop-ins are still written to k3s.service.d only; aligning those with the node's real unit dir is a separate change.

[Flegma]

Code review

No issues found in the diff. Verified:

• Host parsing (${K3S_URL#*://} / ${H%%[:/]*}) is correct for https://host:port, no-port, and trailing-slash forms; the only value that would break it is a bracketed IPv6 literal, which can't occur here since node IPs come from tailscale ip -4.
• Genuine no-op on server nodes (no K3S_URL, so the [ -n "$H" ] guard skips the ping).
• Non-blocking: leading - + timeout 15 on the ping + trailing || true mean a failed or hung pre-warm can never block k3s from starting.
• Confirmed empirically via a throwaway systemd unit that ${K3S_URL#*://} reaches bash intact (systemd does not expand it inside the single-quoted bash -c), and the ping succeeds.

Pre-existing, out of scope (already noted in the PR description): the sibling update-tailscale-ip.conf and tailscale-state-check.conf are written to k3s.service.d, which is inert on agent nodes (they only have k3s-agent.service), and systemctl restart k3s at the end errors on agent provisioning. So on agent nodes the existing self-heal state-check never actually fires. Worth a dedicated follow-up in the agent-provisioning path.

5stack-panel/game-node-server-setup.sh

Lines 197 to 226 in 0be3beb

	cat <<-'DROPIN' >/etc/systemd/system/k3s.service.d/update-tailscale-ip.conf
	[Service]
	ExecStartPre=/bin/bash -c 'TSIP=$(tailscale ip -4 2>/dev/null | head -n 1); if [ -n "$TSIP" ] && [ -f /etc/rancher/k3s/config.yaml ]; then sed -i "s/^node-ip:.*/node-ip: $TSIP/" /etc/rancher/k3s/config.yaml; echo "[5stack] Updated k3s node-ip to $TSIP"; fi'
	DROPIN
	cat <<-'DROPIN' >/etc/systemd/system/k3s.service.d/tailscale-state-check.conf
	[Unit]
	After=tailscaled.service
	Wants=tailscaled.service
	[Service]
	ExecStartPre=/usr/local/bin/5stack-tailscale-state-check.sh
	DROPIN
	# Pre-warm the Tailscale path to the k3s server on (re)start so a one-way path
	# after a reboot cannot leave the node unable to reach the control plane. Lands
	# in whichever k3s unit dir this node uses; harmless no-op on server nodes.
	PREWARM_DIR=/etc/systemd/system/k3s.service.d
	[ -f /etc/systemd/system/k3s-agent.service.env ] && PREWARM_DIR=/etc/systemd/system/k3s-agent.service.d
	mkdir -p "$PREWARM_DIR"
	rm -f "$PREWARM_DIR/tailscale-prewarm.conf"
	cat <<-'DROPIN' >"$PREWARM_DIR/tailscale-prewarm.conf"
	[Unit]
	After=tailscaled.service
	Wants=tailscaled.service
	[Service]
	ExecStartPre=-/bin/bash -c '. /etc/systemd/system/k3s-agent.service.env 2>/dev/null || true; H=${K3S_URL#*://}; H=${H%%[:/]*}; [ -n "$H" ] && echo "[5stack] pre-warming tailscale path to $H" && timeout 15 tailscale ping -c 3 "$H" >/dev/null 2>&1 || true'
	DROPIN

[Flegma]

Amended to b42391c: replaced the ${H%%[:/]*} host parse with a %-free cut pipeline. systemd collapses %% to a literal % inside ExecStartPre= (confirmed on systemd 255), so the %% longest-match silently ran as % shortest-match. It was harmless for the real single-colon K3S_URL, but the operator was misleading. Verified the new parse through a transient systemd unit (a:b:c to a, host/p:9 to host, https://<ipv4>:6443 to <ipv4>). Same fix applied to the canonical agent path in 5stackgg/api#362.

5stack-panel/game-node-server-setup.sh

Lines 211 to 226 in b42391c

	# Pre-warm the Tailscale path to the k3s server on (re)start so a one-way path
	# after a reboot cannot leave the node unable to reach the control plane. Lands
	# in whichever k3s unit dir this node uses; harmless no-op on server nodes.
	PREWARM_DIR=/etc/systemd/system/k3s.service.d
	[ -f /etc/systemd/system/k3s-agent.service.env ] && PREWARM_DIR=/etc/systemd/system/k3s-agent.service.d
	mkdir -p "$PREWARM_DIR"
	rm -f "$PREWARM_DIR/tailscale-prewarm.conf"
	cat <<-'DROPIN' >"$PREWARM_DIR/tailscale-prewarm.conf"
	[Unit]
	After=tailscaled.service
	Wants=tailscaled.service
	[Service]
	ExecStartPre=-/bin/bash -c '. /etc/systemd/system/k3s-agent.service.env 2>/dev/null || true; H=$(echo "${K3S_URL#*://}" | cut -d: -f1 | cut -d/ -f1); [ -n "$H" ] && echo "[5stack] pre-warming tailscale path to $H" && timeout 15 tailscale ping -c 3 "$H" >/dev/null 2>&1 || true'
	DROPIN