Repository navigation
Subprocess Pipe Deadlock in shells.py on Non-UTF-8 Output #13
Description
Summary
subprocess.Popen is instantiated with text=True but omits errors="replace". When a command outputs non-UTF-8 bytes, _reader encounters a UnicodeDecodeError, exits its reading loop, and blocks in wait(). The unconsumed stdout fills the 64KB OS pipe buffer, causing a permanent deadlock between the child process and the reader thread.
Root Cause
In openhack/shells.py:91-122:
proc = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
...
def _reader(self, sh: BackgroundShell) -> None:
try:
for line in sh.proc.stdout:
sh.append(line.rstrip("\n"))
except Exception:
pass
finally:
sh.returncode = sh.proc.wait() # DEADLOCK: child write() is blockedReproduction
manager.spawn("python3 -c 'import sys; sys.stdout.buffer.write(b\"\\xff\" * 100000)'")
# The reader crashes on decode, stops draining the pipe, and hangs in sh.proc.wait() forever.Impact
Background shell tasks and security scanners (strings, fuzzers, compiled binary tests) freeze the CLI indefinitely.
Proposed Fix
Add errors="replace" to text-mode Popen calls:
proc = subprocess.Popen(..., text=True, errors="replace")
Activity
[-]Read Only Plan Mode Dispatches Dangerous Tools (`run_command`, `write_file`)[/-][+]Subprocess Pipe Deadlock in `shells.py` on Non-UTF-8 Output[/+]kayareyouokay commentedon Oct 3, 2026
Yeah, this looks like a real deadlock in the background shell path.
errors="replace"on the text-modePopenshould be the minimal fix, since it lets the reader keep draining stdout even when a tool emits non-UTF-8 bytes. I suppose maybe we can also add a regression test using a large stream ofb"\xff"bytes, to make sure the shell exits instead of hanging once the pipe fills.Also, the foreground command helper could use the same decoding policy too. It shouldn’t deadlock there, but invalid output can still cause an avoidable command failure.
aijadugar commentedon Oct 10, 2026
I’d like to take this. I have a minimal fix ready: errors="replace" on the background shell Popen (plus the same policy in run_killable, text mode only) and a regression test. I noticed fix/windows-subprocess-decoding touches the same area. Should I open a standalone PR, or is that branch landing soon?
Yuren0312 commentedon Oct 10, 2026
Thanks for the clear write-up. I've opened PR #19 with the
errors="replace"fix: background-shell output is now decoded with replacement characters, so non-UTF-8 bytes no longer crash the reader thread and deadlock the stdout pipe. Added a regression test (test_non_utf8_output_does_not_deadlock) that spawns a command writing 200000 invalid bytes and asserts the shell exits cleanly with the bytes decoded as U+FFFD.