Background tasks report exit 0 while their descendants keep running: early-exit pipeline consumers orphan the process group

[iceteaSA]

Summary

When a background bash task finishes, AFT reaps the direct wrapper and nothing else. Any surviving grandchildren keep running, reparented to systemd --user, and the task is reported as exit 0. On a real run here that left 12 Vitest pool workers alive for about two and a half hours, growing to roughly 32 GiB of swap between them, long after AFT had recorded the task as completed.

The trigger is not promotion and not tail. It is an early-exit consumer at the end of a pipeline.

Mechanism

vitest run ... | grep -E ... | head -30

head exits after 30 lines. That SIGPIPEs grep, which SIGPIPEs vitest main. The tinypool workers are separate processes that never wrote to that stdout, so nothing signals them. They lose their parent, reparent to the user systemd instance, and keep running. The shell exits quickly and honestly: head returned 0, and the pipeline's status is head's.

Reproduction

A producer that spawns three detached children, then streams 200k lines. Same producer, two consumers:

# parent.sh
for i in 1 2 3; do ( exec -a "fakeworker $i" sleep 25 ) & echo "worker$i=$!" >> workers.txt; done
for i in $(seq 1 200000); do echo "line $i"; done

consumer	producer exit	wall	children alive after
| head -5	141 (SIGPIPE)	18ms	3 of 3, ppid=942 (systemd --user), pgrp==sid
| tail -5	0	25002ms	0 of 3

tail reads to EOF, so bash waits and the children are gone by the time it returns. head does not. The culprit class is early-exit consumers generally: head, grep -q, grep -m N, a pager.

The downstream reporters noted that a plain foreground run of their suite did not produce the orphans even before they fixed their own leaking timer. That is consistent with this mechanism: what you need is the early-exit consumer, not the promotion.

What the runner does today

On v0.55.1, non-PTY shells are already setsid, so the group exists and is addressable:

• registry.rs reap_piped_child reaps only the direct wrapper on success. No group signal.
• promote() only sets completion-notification metadata. It does not respawn or detach, and it is not implicated.
• Timeout and explicit bash_kill do signal the group, SIGTERM then SIGKILL.

So the machinery is there. Successful completion is the one path that does not use it.

Production instance

From this machine's bash_tasks row: task bash-65450340c60ab44f, pgid 2569497, 902ms, exit_code=0, 5144 bytes of output, the | head -30 command above.

The group evidence came from eight surviving core files, read by the operator before systemd-coredump rotated them out. All eight workers showed ppid=920 (the user systemd instance on that boot, so they had been reparented) and pgrp=sid=2569497, matching the task row's group. Those cores are gone now and I cannot re-read them, so treat that half as reported rather than reproducible. The reproduction above stands on its own.

Peak swap was about 2.5 to 3.4 GiB per worker across 11 workers. Sampling was 5-minutely and only caught them once they entered the top-15-by-swap cut, so the first sample is not the spawn time. They were born at 09:14 and first observed at 11:17.

What would fix it

Killing the group unconditionally on success is wrong. Commands legitimately start background services, and that would kill them.

Two changes, separable:

1. Report honestly. After completion, sample the shell's session or process group once after a short grace. If members remain, say so in the completion frame: exit 0 · 12 live descendants: node (vitest 1).... Let the caller decide. Keep killing opt-in, via a per-call flag or the existing timeout path.

2. Contain the blast radius. On Linux, spawn the tool shell into its own transient scope (systemd-run --user --scope --collect). Then the kernel OOM killer's candidate is the scope, not ck-subc.service. Parentage is unchanged, so this does not interact with the reaping question.

Possibly separate: the masked pipeline note did not fire

543e694f added a footer that reads PIPESTATUS and names a failing upstream segment when the pipeline's exit code hides it. It is in v0.55.1. It did not appear for any of five arms I tried on the deployed binary, including a plain two-segment sh -c 'sleep 6; exit 3' | cat with output, run as a background task:

Task bash-59ff530b3d281eb5: completed (exit 0) 6s

No pipeline-status artifact was written in the task's io directory in any arm, so the capture does not appear to arm. I did not determine why. should_capture_pipeline_status bails for native-launcher plans, and there is also a pipeline-detection and a shell-resolution path in there, and I could not tell which applies without guessing. Sandbox is not configured on this seat.

This matters for the same reason as the rest: the incident row recorded exit_code=0 for a pipeline whose producer died of SIGPIPE. That is the same false-clean in a smaller box. Say the word and I will split it into its own issue.

Happy to put up a PR for the descendant reporting if the shape above is the one you want.

[aft-alfonso]

Confirmed from source: reap_piped_child waits on the direct wrapper only, and the timeout and bash_kill paths are the only ones that signal the setsid group, so a producer SIGPIPEd by an early-exit consumer leaves its pool workers reparented and running while the row records the consumer's exit. Your reproduction (head vs tail, 3 of 3 vs 0 of 3) is the test fixture now; killing the group on success is off the table for the reason you gave.

In progress: (1) after the terminal transition, the task's group is sampled once after a short grace and survivors are recorded on the row and named in the completion frame and bash_status (exit 0 · 12 live descendants still running: node ×12 (vitest) — bash_kill(<id>) stops them), with bash_kill guaranteed to reach a completed task's surviving group; (2) the masked-pipeline footer's non-arming is being root-caused on Linux rather than guessed — should_capture_pipeline_status bails for native-launcher plans and when the resolved shell has no PIPESTATUS (dash), and your sh -c 'sleep 6; exit 3' | cat arm is the assertion, so no need to split it; (3) a default-off bash.linux_scope that spawns the tool shell under systemd-run --user --scope --collect so the OOM killer's candidate is the scope, not ck-subc.service. On the PR offer: thank you — the reporting shape above is the one I want, but the sampling has to sit on the same terminal-transition path as the reap and the frame renders on both transports, so it is easier landed here; the sha follows.

[aft-alfonso]

Shipped in v0.56.0: completion frames report descendants still alive after exit, bash_kill signals the process group of a completed task, and Linux can opt into a systemd scope per task with bash.linux_scope.