[xiaods]

Follow-up to #659. That PR fixed four E2B-SDK 2.56.0 compatibility defects; running the same guide (levels 1–5) against a live 4C8G cluster surfaced eight more, and they turned out to explain a class of failures the earlier PR could not have found — including one that made the cluster refuse every new sandbox.

Verified with the official SDKs pinned to 2.56.0: Python 44/44 (sync + async) and JavaScript 17/17, each run repeatedly against a live deployment.

The eight defects fall into three groups: a panic in the sandbox daemon that killed the whole sandbox, a leak in the warm pool that permanently burned the node's capacity, and protocol-shape bugs that made the SDK answer wrongly.

Part 1 — the sandbox daemon panicked, taking the sandbox with it

sandboxd is the Pod's PID 1. A panic there kills the container, and with RestartPolicy: Never the Pod is Failed for good.

1. A failed syscall's errno was decoded with an overflowing cast — sandboxd/src/files.zig

A failing Linux syscall returns the negated errno as a usize (ENOENT is 0xFFFF_FFFF_FFFF_FFFE). Recovering it must reinterpret the whole word, never narrow it:

const err = std.posix.errno(@as(usize, @bitCast(@as(isize, @intCast(rc)))));

@intCast narrows that value to isize and overflows — and in ReleaseSafe an overflow is a panic. So every errno path in the daemon was a landmine:

thread 18 panic: integer does not fit in destination type
sandboxd/src/files.zig:491:64: in handleMkdir

files.make_dir() on a directory whose parent does not exist hit it, which is an ordinary thing for an agent to do. The same idiom was in handleMove, handleSignal, every getdents64 call site, and — worse — the open() error checks, where a failed open leaves fd as a huge usize so fd < 0 never fires and the code went on to use a garbage descriptor.

The new sandboxd/src/syscall.zig decodes the errno without narrowing (std.os.linux.errno bitcasts), and dirReadCount does the same for a directory read whose error must read as "end of directory".

2. stdout and stderr shared one pipe — sandboxd/src/exec.zig

The child did dup2(stdout_pipe[1], 1) and dup2(stdout_pipe[1], 2), so the stream carried no channel information at all. The e2b layer could only broadcast every frame as stdout, and the SDK's stderr was permanently empty while its on_stderr callback never fired:

r = sbx.commands.run("echo out; echo err >&2")
r.stdout   # 'out\ndata: err'   ← stderr mixed into stdout
r.stderr   # ''                 ← always empty

The pipes are now separate, drained together with poll() (reading one to EOF before the other deadlocks as soon as a child fills the unread pipe), and each frame names its channel. The payload is base64 for two reasons: it is JSON-safe, and it keeps the SSE frame boundary intact — a raw payload containing a blank line used to split the frame and leak the data: prefix into the client's stdout.

3. Six hard-coded write lengths did not match their string literals — sandboxd/src/{exec,execctl,pty}.zig

write(client_fd, "data: {\"done\":true}\n\n".ptr, 24) writes 24 bytes of a 21-byte literal, so three bytes of whatever follows it in the image were appended to every stream. The lengths are now derived from the literal.

Part 2 — dead Pods were never reclaimed, and the node refused every new sandbox

4. detectResetting() only promoted Running Pods — pkg/sandboxmatrix/controller.go

A crashed sandboxd leaves a Failed Pod. DestroySession relabels it resetting, and detectResetting — which runs every 10 s — skipped anything that was not Running:

if pod.Status.Phase != corev1.PodRunning {
    continue
}

The only recycler for dead Pods (recycleUnhealthyWarmPods) is scoped to state=warm, so a resetting Pod was nobody's business. It sat there forever.

That is not untidy, it is a capacity leak: CheckCapacity counts every Pod carrying the state label against the node's memory budget, so each dead Pod permanently burned one of a 4C8G node's ~13 sandbox slots. Nine crashes filled the node, and it stayed full — 30 consecutive retries over 25 minutes, all refused:

409: warm pool full: 13 pods, 6656Mi/8002340Ki used.

detectResetting now reclaims dead Pods and orphaned Pending ones.

5. DestroySession recycled a terminated Pod — pkg/sandboxmatrix/grpc/orchestrator.go

A Pod whose container exited can never serve a session again, and the workspace reset that releasePod assumes could not have happened (the daemon was dead) — so handing it back to the pool handed a dirty workspace to the next session. It is now deleted, claim and all.

6. CheckCapacity counted terminated Pods, and ignored CPU — pkg/sandboxmatrix/grpc/orchestrator.go

Two problems in one check:

• It summed the memory limits of every Pod with the state label, including the dead ones. The nine dead Pods above summed to 4.5 GiB of limits, so the node looked full while it was using 34% of its memory.
• It never looked at CPU at all. On a small node CPU is what binds first: 13 sandbox Pods × 500m is 6.5 CPU on a 4-CPU node, and the Pods the scheduler could not place sat Pending forever ("0/1 nodes are available: 1 Insufficient cpu"), each holding a slot and starving every other sandbox.

Terminated Pods are now excluded, and CPU requests are checked the way the scheduler counts them.

7. The Pod IP cache could hand back the address of a Pod being replaced — pkg/sandboxmatrix/grpc/orchestrator.go

A resume re-creates the Pod while the old one is still terminating, and both carry the session-id label. The IP lookup listed by label and returned the dying Pod's address, so every later call went to a Pod that no longer existed — the SDK saw "file not found" after a resume that had actually worked, and then "connection timed out" to a stale address. The session's own Pod is now preferred, and a terminating Pod is never used.

Part 3 — the E2B protocol surface

8. Six protocol-shape bugs — pkg/sandbox/e2b, pkg/sandboxmatrix/grpc

• The read path could not carry binary files. WriteFileRequest.Content is a proto string, so fix(sandbox): four E2B SDK 2.56.0 compatibility fixes #659 fixed the write path by riding base64 — but the read path still returned the file as escaped JSON text, which cannot hold non-UTF-8 bytes. The SDK received an empty file for every binary payload. The read now asks sandboxd for base64.
• commands.run(envs=...) was parsed and then dropped before reaching the gateway, so the SDK's env had no effect on the command. ExecRequest had no env field at all; it does now, and the per-call env is merged over the session's own env.
• Process/List and Process/Start disagreed on the pid space. List reported sandboxd's pids while Start handed out the e2b layer's, so the documented flow — "take the pid from commands.list() and pass it to commands.connect()" — always failed with "process not found". Both now use the same space.
• Process/Connect never delivered an end event, so CommandHandle.wait() raised "Command ended without an end event" and the attach reported neither output nor an exit code.
• The streaming WatchDir was an honest 501 on the assumption that "no SDK uses it". That is true of the Python SDK (which uses the polling trio) but not of the JS SDK: its files.watchDir() calls the streaming RPC, so the JS SDK's watch failed with "501: WatchDir is not supported: K8E has no watch". It is now implemented on top of the polling trio, with the frame shape the JS client parses.
• RemoveWatcher answered 200 with no content type, which the Connect client rejects ("invalid content-type: ''").

One more, found while measuring the above: files.list walked the entire workspace (a populated .venv is thousands of files) and then stat'ed every entry, so listing one directory took seconds. sandboxd's list now takes a path and a depth, and the walk is bounded by both.

Tests

Go:

• TestDetectResetting_ReclaimsDeadPods — a Failed Pod is reclaimed instead of lingering
• TestDetectResetting_ReclaimsOrphanedPending — a Pod stuck Pending past the grace period is reclaimed; a recent one is left alone
• TestDetectResetting_PromotesRunning — the healthy path still promotes to warm
• TestCheckCapacity_IgnoresTerminatedPods — dead Pods do not consume the budget
• TestCheckCapacity_RejectsWhenActuallyFull — the check still refuses a genuinely full node
• TestCheckCapacity_RejectsWhenCPUFull — CPU is now part of the check
• TestCheckCapacity_AcceptsWhenCPUHasRoom — a node with CPU to spare is accepted
• TestDestroySession_DeletesTerminatedPod — a terminated Pod is deleted, claim and all
• TestDestroySession_KeepsClaimForPendingPod — a Pending Pod keeps its claim (deleting it left the Pod referencing a claim that no longer existed)
• TestEnvdProcessConnectDeliversEndEvent — the attach stream carries an end event
• TestEnvdFSWatchDirStreaming — the streaming WatchDir emits a start frame and a filesystem event frame with the numeric type
• TestParseChannelFrame, TestSSEFramerChannelFramesKeepsBoundaries — the channel-tagged frame format
• The fake gateway now decodes encoding="base64" the way real sandboxd does

sandboxd (Zig):

• errnoFromSyscall: negated errno decodes without overflow
• errnoFromSyscall: real mkdir failure decodes to the right errno
• dirReadCount: a failed read decodes as end-of-directory, not a panic
• urlDecode: percent-encoded path decodes to a filesystem path

go test ./pkg/... — all pass. cd sandboxd && zig build test — 59/59 on Linux.

Notes

• The CPU half of the capacity check degrades gracefully. A node that does
  not report an allocatable CPU (some test fixtures, some managed clusters) is
  not CPU-constrained as far as the scheduler is concerned either, so the CPU
  comparison is skipped rather than failing the request — a node with memory to
  spare must not refuse a session. The first version of this check turned a
  missing CPU into an error, which broke sandbox_create on such a node; the
  MCP interop job caught it.
• The imagePullPolicy change is part of this PR on purpose: sandbox Pods defaulted to Always, so every Pod re-pulled the image from the registry. That defeats an air-gapped or locally-patched deployment, and it made verifying a patched sandbox image impossible (the pull overwrote it). A digest-pinned image is immutable, so re-pulling it could only ever fetch the same bytes.
• hack/reclaim-sandbox-pods.sh is a stopgap for operators who cannot upgrade yet: it reclaims the dead and orphaned Pods that update README to delight the focus #4 leaves behind. It is not a substitute for the fix — the panic still happens, and each crash still costs a slot until the sweep runs.
• The JS SDK's read(path, 'bytes') returns the body as a string, so a payload that is not valid UTF-8 comes back lossy (invalid bytes decode to U+FFFD). That is the SDK's transport, not the server: the raw HTTP response carries the exact bytes. The Python SDK returns real bytes.
• The guide's Level 4 example is Python, which is why the streaming WatchDir gap went unnoticed; the JS SDK has always been affected.

[github-actions]

Test Results

1 021 tests  +13   994 ✅ +13   7m 10s ⏱️ +22s
  120 suites ± 0    27 💤 ± 0 
    1 files   ± 0     0 ❌ ± 0 

Results for commit 787e03b. ± Comparison against base commit 2e8998c.

♻️ This comment has been updated with latest results.

[sonarqubecloud]

Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud