From ff285db15b0003cbf9c4c02021d63265eda96f10 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 19 Aug 2026 21:30:29 +0000 Subject: [PATCH 1/4] Kill what a chain started, and stream output as it arrives MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two things a build wants that this did not do: cancelling a chain left the compiler that chain had started still running, and nothing was printed until a chain had finished. Cancelling now kills the chain's process group rather than its shell alone, which takes down what the chain started however deep it goes. Job control is what puts a job in a group of its own, and a non-interactive shell is allowed not to have any: `set -m` is accepted and ignored by dash and busybox ash, and zsh refuses it outright, and fatally, without a terminal to hand a group the foreground with. So the library asks rather than assumes. At the first `chain` it starts one job under `set -m` and asks whether a process group by that pid exists: a job that was given its own group leads it, a job that was not is in the shell's group, and nothing else can hold that id while the job itself holds the pid, so `kill -0` is the whole of the test and `ps` is not needed for it. bash, ksh93 and mksh pass it, zsh passes it when it has a terminal, and dash and busybox ash do not and get exactly what they had before. The probe runs in a subshell for zsh's sake, since a special built-in that fails takes a non-interactive shell down with it and would end the build script rather than report that job control is not available. Monitor mode is only on across the fork itself, because that is when the group is decided and because this is the calling script's shell; the chain turns it off in itself, so that the background commands it runs stay in the group that cancelling kills. The kill goes to the chain first and to the group once the chain has gone: a chain still there to see its own command killed reports it, which put a `Terminated` in the middle of a cancelled chain's output under bash and mksh and a line about the job from ksh93. The exit trap skips that wait, because nothing it kills will be printed and a chain that made itself deaf to the signal must not be able to hold the trap open. STREAM=1 prints every line as it arrives instead, under the label of the chain that wrote it, right aligned so the bars line up: npm │ added 214 packages in 8s composer │ Installing dependencies from lock file The poll that watches for finished chains reads the logs on the same trip round, so the one process already doing the printing is the only one that prints and two chains cannot mix into one line. A chain's position is a count of lines: `tail` starts from the line after it, a line with no newline yet is left where it is until it has one, and the last line of a chain that ended without one goes out at the end. The report is then one line a chain, since the output itself has already gone by. STREAM_SEP is the bar, and follows the locale: a box drawing character where it says UTF-8 and an ASCII pipe where it does not. The README's `POLL=0.5 . ./parallel.sh` was wrong in two of the shells this supports. An assignment in front of a special built-in persists in a POSIX shell and is undone in bash and zsh, so the setting was quietly dropped there; it now says to set it before sourcing, or in the environment. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01WWgsYpG4sTSefc6FQURumD --- README.md | 75 ++++++++++-- parallel.sh | 264 ++++++++++++++++++++++++++++++++++++++---- spec/parallel_spec.sh | 156 +++++++++++++++++++++++++ spec/spec_helper.sh | 39 ++++++- 4 files changed, 494 insertions(+), 40 deletions(-) diff --git a/README.md b/README.md index 068317a..5fda78a 100644 --- a/README.md +++ b/README.md @@ -4,8 +4,8 @@ `parallel.sh` runs chains of commands at the same time, stops everything at the first failure, and prints each chain's output in one block instead of -interleaved. It is POSIX sh, one file, and nothing else, so it runs in a -build container as it is. +interleaved — or labelled, line by line, as it arrives. It is POSIX sh, one +file, and nothing else, so it runs in a build container as it is. Before, where `npm` waits on `composer` for no reason: @@ -128,9 +128,54 @@ first poll, after which polling falls back to whole seconds. Set `POLL` for a different interval and `POLL_WHOLE` for a different fallback: ```sh -POLL=0.5 . ./parallel.sh +POLL=0.5 +. ./parallel.sh +``` + +Set it before sourcing rather than in front of the `.`, which is a special +built-in: an assignment in front of one persists in a POSIX shell and does +not in bash or zsh, so `POLL=0.5 . ./parallel.sh` is two different things +depending on where it runs. The environment works everywhere too, so +`POLL=0.5 ./build.sh` on a script that sources the library is the other way +to do it. + +## STREAM + +`run` holds each chain's output and prints it grouped, which means nothing +is printed until a chain has finished. Set `STREAM` and it prints every line +as it arrives instead, under the label of the chain it came from: + +```sh +STREAM=1 ./build.sh +``` + +``` + npm │ added 214 packages in 8s +composer │ Installing dependencies from lock file + npm │ vite v5.4.2 building for production... +composer │ Generating optimized autoload files + npm │ built in 4.21s + +--- composer +--- npm +``` + +The labels are right aligned to the longest of them, so the bars line up. A +line is printed once it is whole, so a chain that writes one in two writes +still gets one line, and the report at the end is one line a chain — how +each one ended, in declaration order — because the output itself has already +gone by. + +`STREAM_SEP` is the bar. It defaults to `│` where the locale says the +terminal is UTF-8, and to `|` where it does not: + +```sh +STREAM=1 STREAM_SEP='|' ./build.sh ``` +Streamed output is read on the same poll that watches for finished chains, +so a line can be up to `POLL` behind the command that wrote it. + ## Benchmarks `bench/bench.sh` runs four builds twice each, once in declaration order the @@ -162,8 +207,8 @@ best of three runs: cannot finish before its longest chain does, so the most chains can do is hide the rest behind it. -Eight chains that do nothing at all finish in 0.15s: one `mktemp`, eight -forks, and a poll or two. That is about what the library costs a build with +Eight chains that do nothing at all finish in 0.15s: one `mktemp`, one job +control probe, eight forks, and a poll or two. That is about what the library costs a build with nothing to gain. The table is a report on shapes, not a measurement of any real build. A @@ -185,10 +230,18 @@ too noisy to be held to one. ## Caveats -`kill` stops a cancelled chain's shell, not its grandchildren. A command -that spawned its own children can leave one behind that outlives the chain -it belonged to. In a build container that exits anyway this does not matter; -in a long-lived shell it will. +Cancelling a chain kills its process group where the shell can give a chain +one of its own, which takes down what the chain started however deep it +goes. Job control is what puts a chain in a group, and a non-interactive +shell is not obliged to have any, so the library starts one job under +`set -m` at the first `chain` and asks whether that job got a group to +itself. bash, ksh93 and mksh give it one, and zsh does when it has a +terminal. dash and busybox ash accept `set -m` and start the job in the +shell's own group regardless, and there cancelling still kills the chain's +shell and not its grandchildren: a command that spawned its own children can +leave one behind that outlives the chain it belonged to. In a build +container that exits anyway this does not matter; in a long-lived shell it +will. Fractional `sleep` is not in POSIX, though both GNU coreutils and BSD accept it. Whole seconds are the fallback, not the default, so a build on a shell @@ -201,8 +254,8 @@ under zsh, removes it. The library has no `local`, because POSIX sh has none. It keeps its own state in names starting with `_`, but its loops use `i`, `n`, `cmd`, `code`, -`label` and `mark`, and sourcing it will clobber those in the calling -script. +`label`, `mark`, `seen`, `line` and `pending`, and sourcing it will clobber +those in the calling script. ## Prior art diff --git a/parallel.sh b/parallel.sh index 7a0921f..a51b938 100644 --- a/parallel.sh +++ b/parallel.sh @@ -9,15 +9,29 @@ # chains run at the same time. The first failure anywhere cancels the rest. # `run` blocks, prints each chain's output in declaration order, and returns # the exit code of the chain that failed. +# +# With STREAM set, output is labelled and printed as it arrives instead. POLL=${POLL:-0.1} # seconds between checks POLL_WHOLE=${POLL_WHOLE:-1} # used instead if this sleep rejects fractions +STREAM=${STREAM:-0} # 1 to label and print output as it arrives + +# The bar between a label and its line. A box drawing character reads as +# three bytes of noise in a terminal that is not expecting UTF-8, and the +# locale is what says whether it is, so the default follows the locale and +# an ASCII pipe stands in everywhere else. +case ${LC_ALL:-${LC_CTYPE:-${LANG:-}}} in +*[Uu][Tt][Ff]8* | *[Uu][Tt][Ff]-8*) STREAM_SEP=${STREAM_SEP:-'│'} ;; +*) STREAM_SEP=${STREAM_SEP:-'|'} ;; +esac _work=$(mktemp -d) _count=0 _failed=0 _signal=0 # what to exit with if a signal arrives; 0 until one does _fractional='' # whether sleep takes POLL; unknown until the first nap +_groups='' # whether a chain can have its own process group; unknown +_monitor='' # whether the calling shell had job control on already _cleanup() { rm -rf "$_work" || :; } @@ -28,6 +42,10 @@ _cleanup() { rm -rf "$_work" || :; } # for a signal at all, so the signal traps clean up and exit themselves; and # a trap whose test fails is abandoned under set -e, so _finish decides with # a case, which cannot fail, rather than a test, which can. +# +# Nothing here has anything to say, and one shell has: ksh93 reports the +# chains this trap killed, on the way out, on the stderr of a script that is +# already leaving. That is what the redirection is for. _finish() { _ec=$1 case $_signal in @@ -37,7 +55,7 @@ _finish() { _stop_chains _cleanup exit "$_ec" -} +} 2>/dev/null # job control's last word on the chains, and not the script's # Take the chains down on the way out. Nothing else does: a chain is its own # process, so an interrupted build that only cleaned up after itself would @@ -50,12 +68,85 @@ _stop_chains() { i=1 while [ "$i" -le "$_count" ]; do if [ ! -f "$_work/$i.code" ] && [ -f "$_work/$i.pid" ]; then - kill "$(cat "$_work/$i.pid")" 2>/dev/null || : + _kill_chain "$(cat "$_work/$i.pid")" now fi i=$((i + 1)) done } +# Stop one chain, and with it everything the chain started where the shell +# was able to give the chain a process group of its own. +# +# The chain's own shell goes first, on its pid. Killing the group outright +# would work as well, but the chain would still be there to see the command +# it was running killed, and shells report that: what it came to was a +# `Terminated` in the middle of a cancelled chain's output under bash and +# mksh, and a line about the job from ksh93. So `quietly` waits for the +# chain's shell to go before taking the group, and once it has gone there is +# nobody left to report anything. What is left in the group is what the +# chain started, however deep it goes: a group outlives its leader for +# exactly as long as one of them is still running, which is as long as there +# is anything in it worth killing. +# +# `now` skips that wait, and the exit trap uses it. Nothing that trap kills +# will be printed, so it has nothing to keep quiet for, and a chain that has +# made itself deaf to the signal must not be able to hold the trap — and the +# script — open while it waits for a shell that is not going to go. +# +# Neither kill can fail: cancelling races the chain finishing on its own, +# and this is called from the exit trap, which a failure would abandon. +_kill_chain() { + kill "$1" 2>/dev/null || : + case $_groups in + yes) + case $2 in + quietly) wait "$1" 2>/dev/null || : ;; + esac + kill -- "-$1" 2>/dev/null || : + ;; + esac +} + +# Whether this shell will put a background job in a process group of its own, +# which is what makes it possible to cancel a chain's children along with the +# chain. Job control is POSIX, but a non-interactive shell is allowed to +# leave it out: dash and busybox ash take `set -m` and start the job in the +# shell's own group anyway, and zsh refuses the option outright unless it has +# a terminal to hand the group the foreground with. +_probe_groups() { + case $- in + *m*) _monitor=yes ;; + *) _monitor=no ;; + esac + # The probe's own kill is job control's business to announce, and some + # shells do announce it, so the probe reports through its status alone + # and anything it has to say goes nowhere. + if _group_probe 2>/dev/null; then _groups=yes; else _groups=no; fi +} + +# The probe runs in a subshell because `set` is a special builtin, and a +# special builtin that fails takes a non-interactive shell down with it: +# zsh, which refuses -m without a terminal, would end the build script here +# rather than report that it cannot do it. +# +# A job that was given its own group leads that group, so a group with its +# pid for an id exists. A job that was not is in the shell's group, and no +# other group can have that id while the job itself holds the pid, so asking +# after the group is the whole of the test and `ps` is not needed for it. +_group_probe() ( + set -m 2>/dev/null || exit 1 + sleep 1 & + _p=$! + kill -0 -- "-$_p" 2>/dev/null + _ok=$? + # SIGKILL, because the probe must not be able to hang or to leave the + # nap behind: mksh takes the group kill in a subshell without the job + # dying of it, and would then wait out the second it asked for. + kill -9 -- "-$_p" 2>/dev/null || kill -9 "$_p" 2>/dev/null + wait "$_p" 2>/dev/null # zsh complains about a job still running at exit + exit "$_ok" +) + trap '_finish $?' EXIT trap '_signal=130; _finish 130' INT trap '_signal=143; _finish 143' TERM @@ -66,30 +157,59 @@ chain() { printf '%s' "$1" >"$_work/$n.label" shift - ( - trap - INT TERM # don't inherit the parent's handlers - code=0 - # Always leave a status behind, even if a command calls exit. - # Write, then rename, so the reader never sees a half-written file. - # shellcheck disable=SC2154 # ec is assigned in the same trap - trap 'ec=$?; [ "$code" -ne 0 ] || code=$ec - printf "%s" "$code" >"$_work/$n.code.part" - mv "$_work/$n.code.part" "$_work/$n.code"' EXIT - - for cmd in "$@"; do - eval "$cmd" || { - code=$? - break - } - done - ) >"$_work/$n.log" 2>&1 & + # Job control decides which process group a job starts in, and decides it + # when the job starts, so the option only has to be on across the fork + # below. It goes back afterwards, because this is the calling script's + # shell: under job control that script would find its own background jobs + # taken out of its process group too. + case $_groups in '') _probe_groups ;; esac + case $_groups in yes) set -m ;; esac + + # The braces are for zsh, which announces every job it starts once it has + # job control, on the calling script's stdout, in the middle of the report + # the script is there to print. The announcement is this shell's, not the + # chain's, so it is this shell's stdout that has to point elsewhere while + # the chain starts. The chain's own output goes to its log either way, and + # a fork that fails still has stderr to say so on. + { + ( + trap - INT TERM # don't inherit the parent's handlers + + # The chain has the group; it does not need job control of + # its own, and is worse off with it. A chain that inherited + # it would give a group of its own to every background + # command it ran, putting them outside the group that + # cancelling kills. + case $- in *m*) set +m ;; esac + + code=0 + # Always leave a status behind, even if a command calls + # exit. Write, then rename, so the reader never sees a + # half-written file. + # shellcheck disable=SC2154 # ec is assigned in the same trap + trap 'ec=$?; [ "$code" -ne 0 ] || code=$ec + printf "%s" "$code" >"$_work/$n.code.part" + mv "$_work/$n.code.part" "$_work/$n.code"' EXIT + + for cmd in "$@"; do + eval "$cmd" || { + code=$? + break + } + done + ) >"$_work/$n.log" 2>&1 & + } >/dev/null printf '%s' "$!" >"$_work/$n.pid" + + if [ "$_groups" = yes ] && [ "$_monitor" = no ]; then set +m; fi } run() { + _streaming && _prefixes _await _cancel + _streaming && _flush _report return "$_failed" } @@ -118,6 +238,7 @@ _nap() { # Wait for everything, or return early as soon as one chain fails _await() { while :; do + _streaming && _pump pending=0 i=1 while [ "$i" -le "$_count" ]; do @@ -142,14 +263,98 @@ _cancel() { while [ "$i" -le "$_count" ]; do if [ ! -f "$_work/$i.code" ]; then : >"$_work/$i.cancelled" - kill "$(cat "$_work/$i.pid")" 2>/dev/null + _kill_chain "$(cat "$_work/$i.pid")" quietly fi i=$((i + 1)) done wait 2>/dev/null || : # a killed job must not abort a script that set -e } +# Whether output is streamed as it arrives rather than held and grouped. +_streaming() { + case ${STREAM:-0} in + '' | 0 | no | off | false) return 1 ;; + *) return 0 ;; + esac +} + +# The label a streamed line carries, one per chain, right aligned to the +# longest of them so that the bars line up under each other. Labels are +# known by the time `run` is called, which is the first moment a width can +# be worked out, and they are kept in variables because the pump wants them +# on every poll. +_prefixes() { + _pad=0 + i=1 + while [ "$i" -le "$_count" ]; do + label=$(cat "$_work/$i.label") + [ "${#label}" -gt "$_pad" ] && _pad=${#label} + i=$((i + 1)) + done + + i=1 + while [ "$i" -le "$_count" ]; do + label=$(cat "$_work/$i.label") + while [ "${#label}" -lt "$_pad" ]; do label=" $label"; done + eval "_label_$i=\$label" + i=$((i + 1)) + done +} + +# What every chain has written since the last look round. The poll is the +# only thing printing, so two chains that write at the same moment come out +# as whole lines one after the other rather than mixed into each other. +_pump() { + i=1 + while [ "$i" -le "$_count" ]; do + _emit "$i" now + i=$((i + 1)) + done +} + +# The rest of it, once the chains are done: what the last poll did not get +# to, and the final line of a chain that ended without a newline. +_flush() { + i=1 + while [ "$i" -le "$_count" ]; do + _emit "$i" last + i=$((i + 1)) + done +} + +# One chain's new output. The count of lines already printed is the whole of +# the reader's position: `tail` starts from the line after it, and a line +# that has no newline yet is left where it is, so a line written in two goes +# out in one piece instead of as two labelled halves. `read` fails on that +# partial line without counting it, which is what leaves it to be read again +# next time round — until `last`, when there is no next time and what is +# there is all there will be. +# +# The loop reads a file rather than a pipe on purpose: a pipe would put it +# in a subshell in most shells, and the count it keeps would go with it. +_emit() { + n=$1 + eval "seen=\${_seen_$n:-0} label=\$_label_$n" + tail -n "+$((seen + 1))" "$_work/$n.log" \ + >"$_work/$n.chunk" 2>/dev/null || return 0 + + line='' + while IFS= read -r line; do + printf '%s %s %s\n' "$label" "$STREAM_SEP" "$line" + seen=$((seen + 1)) + line='' + done <"$_work/$n.chunk" + + if [ -n "$line" ] && [ "$2" = last ]; then + printf '%s %s %s\n' "$label" "$STREAM_SEP" "$line" + seen=$((seen + 1)) + fi + + eval "_seen_$n=\$seen" +} + _report() { + _streaming && printf '\n' i=1 while [ "$i" -le "$_count" ]; do label=$(cat "$_work/$i.label") @@ -165,11 +370,20 @@ _report() { mark="[!]" fi - printf '\n%s %s\n' "$mark" "$label" - cat "$_work/$i.log" - [ -n "$code" ] && [ "$code" -ne 0 ] && - printf '[!] %s exited %s\n' "$label" "$code" - [ -f "$_work/$i.cancelled" ] && printf '[.] %s cancelled\n' "$label" + if _streaming; then + # The output has already gone by, in the order it happened, so + # what is left to report is a line a chain saying how it ended. + printf '%s %s' "$mark" "$label" + [ -n "$code" ] && [ "$code" -ne 0 ] && printf ' exited %s' "$code" + [ -f "$_work/$i.cancelled" ] && printf ' cancelled' + printf '\n' + else + printf '\n%s %s\n' "$mark" "$label" + cat "$_work/$i.log" + [ -n "$code" ] && [ "$code" -ne 0 ] && + printf '[!] %s exited %s\n' "$label" "$code" + [ -f "$_work/$i.cancelled" ] && printf '[.] %s cancelled\n' "$label" + fi i=$((i + 1)) done } diff --git a/spec/parallel_spec.sh b/spec/parallel_spec.sh index 408b192..ae738f2 100644 --- a/spec/parallel_spec.sh +++ b/spec/parallel_spec.sh @@ -139,6 +139,31 @@ Describe 'parallel.sh' The output should include "... slow" The output should include "[.] slow cancelled" End + + # The chain is a shell of its own, so killing it on its pid alone + # leaves whatever it was running still running: the compiler + # outlives the build that gave up on it. Where the shell can put a + # chain in a process group of its own, the whole group goes, and + # what the chain started goes with it however deep it is. + It "kills what a cancelled chain started, not only the chain" + Skip if "this shell leaves a background job in its own process group" no_process_groups + + Data + #|set -eu + #|. "$LIB" + #|chain "slow" \ + #| "sh -c 'sleep 30 & printf %s \$! >\"$WORK/childpid\"; sleep 30'" + #|chain "quick" \ + #| "until [ -f '$WORK/childpid' ]; do sleep 1; done" \ + #| "exit 4" + #|run + End + + When call driver + The status should equal 4 + The output should include "[.] slow cancelled" + The value "$(process_state "$WORK/childpid")" should equal "stopped" + End End Describe 'output grouping' @@ -283,4 +308,135 @@ Describe 'parallel.sh' The value "$(chain_state)" should equal "stopped" End End + + # STREAM=1 prints each line as it arrives, under the label of the chain + # it came from, instead of holding the output and grouping it at the end. + Describe 'streaming' + It 'labels every line with the chain it came from' + Data + #|set -eu + #|STREAM=1 STREAM_SEP='|' + #|. "$LIB" + #|chain "npm" "printf 'one\n'; printf 'two\n'" + #|chain "composer" "printf 'three\n'" + #|run + End + + When call driver + The status should equal 0 + The output should include " npm | one" + The output should include " npm | two" + The output should include "composer | three" + End + + # The labels are right aligned to the longest of them, so the bars + # line up whatever the chains are called. + It 'pads the labels to the same width' + Data + #|set -eu + #|STREAM=1 STREAM_SEP='|' + #|. "$LIB" + #|chain "a" "printf 'short\n'" + #|chain "a-much-longer-one" "printf 'long\n'" + #|run + End + + When call driver + The status should equal 0 + The line 1 of output should equal " a | short" + The line 2 of output should equal "a-much-longer-one | long" + End + + # The point of it: the parent prints what a chain wrote while the + # other chains are still going. The second chain here waits to see + # the first chain's line in what `run` has already printed, which it + # can only do if the line was printed before either chain finished. + It 'prints a line while the chains are still running' + Data + #|set -eu + #|STREAM=1 STREAM_SEP='|' + #|. "$LIB" + #|chain "first" "printf 'the first line\n'" + #|chain "second" \ + #| "until grep -q 'the first line' '$WORK/out' 2>/dev/null || + #| [ ! -d '$WORK' ]; do sleep 1; done" \ + #| "printf 'saw it\n'" + #|run >"$WORK/out" + #|cat "$WORK/out" + End + + When call driver + The status should equal 0 + The output should include "second | saw it" + End + + # A line is a line once it has its newline. A chain that writes one + # in two goes gets one labelled line, not two. + It 'waits for a half-written line to be finished' + Data + #|set -eu + #|STREAM=1 STREAM_SEP='|' + #|. "$LIB" + #|chain "half" "printf 'a line'; sleep 1; printf ' in two writes\n'" + #|run + End + + When call driver + The status should equal 0 + The line 1 of output should equal "half | a line in two writes" + End + + It 'prints a last line that never got its newline' + Data + #|set -eu + #|STREAM=1 STREAM_SEP='|' + #|. "$LIB" + #|chain "abrupt" "printf 'no newline here'" + #|run + End + + When call driver + The status should equal 0 + The line 1 of output should equal "abrupt | no newline here" + End + + # The output has already gone by, so the report at the end is one + # line a chain, in declaration order, saying how each one ended. + It 'reports how each chain ended, a line each' + Data + #|set -eu + #|STREAM=1 STREAM_SEP='|' + #|. "$LIB" + #|chain "fine" "printf 'all good\n'" + #|chain "slow" \ + #| "touch '$WORK/slow-started'" \ + #| "until [ -f '$WORK/done' ] || [ ! -d '$WORK' ]; do sleep 1; done" + #|chain "broken" \ + #| "until [ -f '$WORK/slow-started' ]; do sleep 1; done" \ + #| "exit 3" + #|run + End + + When call driver + The status should equal 3 + The output should include "fine | all good" + The output should include "--- fine" + The output should include "... slow cancelled" + The output should include "[!] broken exited 3" + End + + It 'takes the bar between label and line from STREAM_SEP' + Data + #|set -eu + #|STREAM=1 STREAM_SEP='>>' + #|. "$LIB" + #|chain "one" "printf 'a line\n'" + #|run + End + + When call driver + The status should equal 0 + The line 1 of output should equal "one >> a line" + End + End End diff --git a/spec/spec_helper.sh b/spec/spec_helper.sh index 3d6660d..79d776b 100644 --- a/spec/spec_helper.sh +++ b/spec/spec_helper.sh @@ -178,13 +178,25 @@ reported_workdir() { } # Whether the chain a finished driver reported through $WORK/chainpid is -# still running. A process that has just been killed can sit as a zombie -# until the shell that started it goes and init reaps it, so this waits a -# few seconds for an answer rather than believing the first one. +# still running. chain_state() { - _pid=$(cat "$WORK/chainpid") + process_state "$WORK/chainpid" +} + +# The same for any pid a driver left in a file, which is how a chain's own +# children are asked after. A process that has just been killed can sit as a +# zombie until whatever adopted it reaps it, so this waits a few seconds for +# an answer rather than believing the first one, and reads a pid that is +# still answering as stopped once it has become a zombie: a zombie is a +# process that has died, whatever `kill -0` makes of it. Not every `ps` here +# takes -p, and the ones that do not simply leave the wait to decide. +process_state() { + _pid=$(cat "$1") _waited=0 while kill -0 "$_pid" 2>/dev/null; do + case $(ps -o stat= -p "$_pid" 2>/dev/null) in + *Z*) break ;; + esac if [ "$_waited" -ge 5 ]; then printf 'running' return 0 @@ -195,6 +207,25 @@ chain_state() { printf 'stopped' } +# Whether the shell under test leaves a background job in the shell's own +# process group, which is the library's own test for whether it can take a +# chain's children down with the chain. Job control is optional in a +# non-interactive shell: dash and busybox ash take `set -m` and ignore it, +# and zsh refuses it outright without a terminal. +no_process_groups() { + # shellcheck disable=SC2016 # the probe is the other shell's to expand + ! "$SHELLSPEC_SHELL" -c ' + set -m 2>/dev/null || exit 1 + sleep 1 & + p=$! + kill -0 -- "-$p" 2>/dev/null + ok=$? + kill -9 -- "-$p" 2>/dev/null || kill -9 "$p" 2>/dev/null + wait "$p" 2>/dev/null + exit "$ok" + ' 2>/dev/null +} + lines_in() { wc -l <"$1" | tr -d ' ' } From f7692eb600fede63bd37a899fc10ac155f50cdf5 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 19 Aug 2026 21:50:27 +0000 Subject: [PATCH 2/4] Probe for process groups where the chains are started MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The FreeBSD runner failed the example that asks whether cancelling a chain takes what the chain started with it. The kill was not the problem: the probe was answering for a subshell. `set` is a special built-in, and a special built-in that fails takes a non-interactive shell down with it, so the probe ran in a subshell to keep zsh's refusal of `set -m` from ending the build script. But a subshell is a different place to ask from. mksh already showed that much — it reports the group is there and then will not kill it — and FreeBSD's sh answered for the subshell what was not true of the script, so the library believed it had process groups, did not, and left the chain's children running. The example ran because the spec asked the same question its own way and got the other answer. Only the question of whether the option can be set at all needs a subshell now, and the rest of the probe runs where the chains will be started, which is the only place whose answer matters. The spec no longer keeps a copy of the probe either: it asks the library, so the two cannot disagree about which shells the example is for. Two shells had something to say about all this on the way past — dash announces that it has no terminal for job control, and job control announces the job the probe starts — and neither is the build script's news. The chain drops job control on what the library turned on rather than on what `$-` reports, since dash leaves `m` out of `$-` with monitor mode set and the ash-derived shells it is one of are exactly the ones that would be wrong about. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01WWgsYpG4sTSefc6FQURumD --- parallel.sh | 68 ++++++++++++++++++++++++--------------------- spec/spec_helper.sh | 25 ++++++++--------- 2 files changed, 48 insertions(+), 45 deletions(-) diff --git a/parallel.sh b/parallel.sh index a51b938..c5d7341 100644 --- a/parallel.sh +++ b/parallel.sh @@ -113,39 +113,42 @@ _kill_chain() { # leave it out: dash and busybox ash take `set -m` and start the job in the # shell's own group anyway, and zsh refuses the option outright unless it has # a terminal to hand the group the foreground with. -_probe_groups() { - case $- in - *m*) _monitor=yes ;; - *) _monitor=no ;; - esac - # The probe's own kill is job control's business to announce, and some - # shells do announce it, so the probe reports through its status alone - # and anything it has to say goes nowhere. - if _group_probe 2>/dev/null; then _groups=yes; else _groups=no; fi -} - -# The probe runs in a subshell because `set` is a special builtin, and a -# special builtin that fails takes a non-interactive shell down with it: -# zsh, which refuses -m without a terminal, would end the build script here -# rather than report that it cannot do it. +# +# The probe starts its job here, in the shell that will be starting the +# chains, and not in a subshell: a subshell is a different place to ask from +# and gives different answers. mksh says the group is there and then will not +# kill it, and FreeBSD's sh answered for a subshell what was not true of the +# script — which is a probe reporting on itself rather than on the chains. # # A job that was given its own group leads that group, so a group with its # pid for an id exists. A job that was not is in the shell's group, and no # other group can have that id while the job itself holds the pid, so asking # after the group is the whole of the test and `ps` is not needed for it. -_group_probe() ( - set -m 2>/dev/null || exit 1 - sleep 1 & +_probe_groups() { + case $- in + *m*) _monitor=yes ;; + *) _monitor=no ;; + esac + _groups=no + + # Whether the option can be set at all is a question for a subshell, + # because `set` is a special builtin and a special builtin that fails + # takes a non-interactive shell down with it: zsh, which refuses -m + # without a terminal, would end the build script rather than answer. + (set -m) 2>/dev/null || return 0 + set -m 2>/dev/null # dash says out loud that it has no terminal for it + + # Braces and a redirection for the same reason `chain` has them: a shell + # with job control announces the jobs it starts, and this one is not the + # script's news. SIGKILL because the probe must not be able to hang. + { sleep 1 & } >/dev/null 2>&1 _p=$! - kill -0 -- "-$_p" 2>/dev/null - _ok=$? - # SIGKILL, because the probe must not be able to hang or to leave the - # nap behind: mksh takes the group kill in a subshell without the job - # dying of it, and would then wait out the second it asked for. + if kill -0 -- "-$_p" 2>/dev/null; then _groups=yes; fi kill -9 -- "-$_p" 2>/dev/null || kill -9 "$_p" 2>/dev/null - wait "$_p" 2>/dev/null # zsh complains about a job still running at exit - exit "$_ok" -) + wait "$_p" 2>/dev/null || : + + case $_monitor in no) set +m 2>/dev/null ;; esac +} trap '_finish $?' EXIT trap '_signal=130; _finish 130' INT @@ -176,11 +179,14 @@ chain() { trap - INT TERM # don't inherit the parent's handlers # The chain has the group; it does not need job control of - # its own, and is worse off with it. A chain that inherited - # it would give a group of its own to every background - # command it ran, putting them outside the group that - # cancelling kills. - case $- in *m*) set +m ;; esac + # its own, and is worse off with it. A chain that kept it + # would give a group of its own to what it started, putting + # it outside the group that cancelling kills. Off on what + # the library turned on, not on what `$-` reports: dash + # leaves `m` out of `$-` with monitor mode set, and the + # ash-derived shells it is one of are exactly the ones this + # would be wrong about. + case $_groups in yes) set +m ;; esac code=0 # Always leave a status behind, even if a command calls diff --git a/spec/spec_helper.sh b/spec/spec_helper.sh index 79d776b..ae782cd 100644 --- a/spec/spec_helper.sh +++ b/spec/spec_helper.sh @@ -208,22 +208,19 @@ process_state() { } # Whether the shell under test leaves a background job in the shell's own -# process group, which is the library's own test for whether it can take a -# chain's children down with the chain. Job control is optional in a -# non-interactive shell: dash and busybox ash take `set -m` and ignore it, -# and zsh refuses it outright without a terminal. +# process group, in which case cancelling a chain cannot take the chain's +# children with it. Job control is optional in a non-interactive shell: dash +# and busybox ash take `set -m` and start the job in the shell's group +# anyway, and zsh refuses it outright without a terminal. +# +# The library is asked rather than a copy of its probe run here: a copy can +# answer differently from the real one — the FreeBSD runner had them +# disagree — and then an example is skipped that should have run, or run +# that should have been skipped, and the failure reads as a broken kill. no_process_groups() { # shellcheck disable=SC2016 # the probe is the other shell's to expand - ! "$SHELLSPEC_SHELL" -c ' - set -m 2>/dev/null || exit 1 - sleep 1 & - p=$! - kill -0 -- "-$p" 2>/dev/null - ok=$? - kill -9 -- "-$p" 2>/dev/null || kill -9 "$p" 2>/dev/null - wait "$p" 2>/dev/null - exit "$ok" - ' 2>/dev/null + [ "$("$SHELLSPEC_SHELL" -c '. "$LIB"; _probe_groups; printf %s "$_groups"' \ + 2>/dev/null)" != yes ] } lines_in() { From b449bf1cfe83d7815ce20333a24f5ea99361d23f Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 19 Aug 2026 21:51:09 +0000 Subject: [PATCH 3/4] Stream by default, and group on request MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A build is easier to follow while it is running than after it has finished, so labelled output is what `run` does now and grouping is what a build asks for. `STREAM=0` gets the old behaviour: nothing printed until a chain has finished, and nothing a chain wrote anywhere but under its own heading. The report a streamed run ends with says how each chain ended, and now says it in the lines the grouped report ends a chain on: `--- label` for a chain that finished, `[!] label exited 3` for one that failed, `[.] label cancelled` for one that was cancelled. A build that greps its output for one of them finds it whichever way the output was printed. The specs follow the default: the two opening examples are what a build sees without asking for anything, and grouped output has a Describe of its own that asks for it. Two of them assert on lines rather than on a pattern of several parts, which is a better assertion here anyway — grouped output is in declaration order all the way down, so every line of it is known — and avoids ksh93, which would not match such a pattern against a subject the length of a report. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01WWgsYpG4sTSefc6FQURumD --- README.md | 80 ++++++++++++++++---------------- parallel.sh | 24 ++++++---- spec/parallel_spec.sh | 105 +++++++++++++++++++++++++++++++++++------- 3 files changed, 145 insertions(+), 64 deletions(-) diff --git a/README.md b/README.md index 5fda78a..4092fc6 100644 --- a/README.md +++ b/README.md @@ -3,9 +3,9 @@ [![CI](https://github.com/glhd/parallel-build/actions/workflows/ci.yml/badge.svg)](https://github.com/glhd/parallel-build/actions/workflows/ci.yml) `parallel.sh` runs chains of commands at the same time, stops everything at -the first failure, and prints each chain's output in one block instead of -interleaved — or labelled, line by line, as it arrives. It is POSIX sh, one -file, and nothing else, so it runs in a build container as it is. +the first failure, and labels every line it prints with the chain it came +from, as that chain writes it. It is POSIX sh, one file, and nothing else, +so it runs in a build container as it is. Before, where `npm` waits on `composer` for no reason: @@ -69,23 +69,26 @@ gantt lint, exits 1 :crit, 00:00:00, 00:00:10 ``` -When something fails, the failure is at the bottom and the rest is grouped -above it: +Every line says which chain wrote it, the labels are right aligned so the +bars line up, and the last word is a line a chain saying how it ended: ``` ---- composer install -Installing dependencies -Generating autoload files +composer install │ Installing dependencies + npm build │ added 214 packages +composer install │ Generating optimized autoload files + npm build │ building for production... + npm build │ ERR! Build failed in 4.21s -[!] npm build -added 214 packages -building for production... +--- composer install [!] npm build exited 1 - -... assets [.] assets cancelled ``` +A line is printed once it is whole, so a command that writes a line in two +writes still gets one line, and the chain it belongs to is the only thing +that decides where it goes. `STREAM=0` holds the output and groups it +instead, which is [below](#stream). + ## Install It is one file. Vendor it: @@ -108,10 +111,11 @@ order, each one only if the one before it succeeded, and the chain stops at the first failure. Each command is a string, evaluated by the shell, so pipes, redirects and `&&` work inside one. -`run` waits for every chain, prints their output in the order they were -declared, and returns the exit code of the chain that failed, or 0. The -first failure cancels the chains still running. Because `run` returns that -code, it can be the last line of a build script: +`run` waits for every chain, prints what they write under the label of the +chain that wrote it, ends with a line a chain saying how each one ended, and +returns the exit code of the chain that failed, or 0. The first failure +cancels the chains still running. Because `run` returns that code, it can be +the last line of a build script: ```sh run @@ -141,39 +145,37 @@ to do it. ## STREAM -`run` holds each chain's output and prints it grouped, which means nothing -is printed until a chain has finished. Set `STREAM` and it prints every line -as it arrives instead, under the label of the chain it came from: +Output is labelled and printed as it arrives. `STREAM=0` holds each chain's +output instead and prints it in one block a chain, in declaration order, +with the failure at the bottom — nothing is printed until a chain has +finished, and nothing a chain wrote is anywhere but under its own heading: ```sh -STREAM=1 ./build.sh +STREAM=0 ./build.sh ``` ``` - npm │ added 214 packages in 8s -composer │ Installing dependencies from lock file - npm │ vite v5.4.2 building for production... -composer │ Generating optimized autoload files - npm │ built in 4.21s - ---- composer ---- npm -``` +--- composer install +Installing dependencies +Generating optimized autoload files + +[!] npm build +added 214 packages +building for production... +[!] npm build exited 1 -The labels are right aligned to the longest of them, so the bars line up. A -line is printed once it is whole, so a chain that writes one in two writes -still gets one line, and the report at the end is one line a chain — how -each one ended, in declaration order — because the output itself has already -gone by. +... assets +[.] assets cancelled +``` -`STREAM_SEP` is the bar. It defaults to `│` where the locale says the -terminal is UTF-8, and to `|` where it does not: +`STREAM_SEP` is the bar between a label and its line. It defaults to `│` +where the locale says the terminal is UTF-8, and to `|` where it does not: ```sh -STREAM=1 STREAM_SEP='|' ./build.sh +STREAM_SEP='|' ./build.sh ``` -Streamed output is read on the same poll that watches for finished chains, +Labelled output is read on the same poll that watches for finished chains, so a line can be up to `POLL` behind the command that wrote it. ## Benchmarks diff --git a/parallel.sh b/parallel.sh index c5d7341..cc94b42 100644 --- a/parallel.sh +++ b/parallel.sh @@ -7,14 +7,15 @@ # # Commands in a chain run in order and stop at the first failure. Separate # chains run at the same time. The first failure anywhere cancels the rest. -# `run` blocks, prints each chain's output in declaration order, and returns -# the exit code of the chain that failed. +# `run` blocks, prints what the chains write under the label of the chain +# that wrote it, and returns the exit code of the chain that failed. # -# With STREAM set, output is labelled and printed as it arrives instead. +# With STREAM=0 the output is held instead, and printed in one block a chain +# in declaration order. POLL=${POLL:-0.1} # seconds between checks POLL_WHOLE=${POLL_WHOLE:-1} # used instead if this sleep rejects fractions -STREAM=${STREAM:-0} # 1 to label and print output as it arrives +STREAM=${STREAM:-1} # 0 to hold the output and print it grouped # The bar between a label and its line. A box drawing character reads as # three bytes of noise in a terminal that is not expecting UTF-8, and the @@ -278,7 +279,7 @@ _cancel() { # Whether output is streamed as it arrives rather than held and grouped. _streaming() { - case ${STREAM:-0} in + case ${STREAM:-1} in '' | 0 | no | off | false) return 1 ;; *) return 0 ;; esac @@ -379,10 +380,15 @@ _report() { if _streaming; then # The output has already gone by, in the order it happened, so # what is left to report is a line a chain saying how it ended. - printf '%s %s' "$mark" "$label" - [ -n "$code" ] && [ "$code" -ne 0 ] && printf ' exited %s' "$code" - [ -f "$_work/$i.cancelled" ] && printf ' cancelled' - printf '\n' + # They are the lines the grouped report ends a chain on, so a + # build that greps for one finds it either way. + if [ -f "$_work/$i.cancelled" ]; then + printf '[.] %s cancelled\n' "$label" + elif [ -n "$code" ] && [ "$code" -ne 0 ]; then + printf '[!] %s exited %s\n' "$label" "$code" + else + printf '%s %s\n' "$mark" "$label" + fi else printf '\n%s %s\n' "$mark" "$label" cat "$_work/$i.log" diff --git a/spec/parallel_spec.sh b/spec/parallel_spec.sh index ae738f2..656274b 100644 --- a/spec/parallel_spec.sh +++ b/spec/parallel_spec.sh @@ -12,9 +12,12 @@ Describe 'parallel.sh' Describe 'chains that succeed' - It 'runs two chains and reports both, in declaration order' + # Output is labelled and printed as it arrives unless a build asks + # for it grouped, so this is what a build sees by default. + It 'labels every line with the chain it came from' Data #|set -eu + #|STREAM_SEP='|' #|. "$LIB" #|chain "first" "printf 'one output\n'" #|chain "second" "printf 'two output\n'" @@ -23,12 +26,19 @@ Describe 'parallel.sh' When call driver The status should equal 0 - The output should match pattern "*--- first*one output*--- second*two output*" + The output should include " first | one output" + The output should include "second | two output" + # Then a line a chain, in declaration order. Which of the two + # labelled lines lands first is up to the chains, but the + # report is always the last two. + The line 4 of output should equal "--- first" + The line 5 of output should equal "--- second" End It 'handles a single chain with a single command' Data #|set -eu + #|STREAM_SEP='|' #|. "$LIB" #|chain "only" "printf 'the one output\n'" #|run @@ -36,8 +46,8 @@ Describe 'parallel.sh' When call driver The status should equal 0 - The line 2 of output should equal "--- only" - The line 3 of output should equal "the one output" + The line 1 of output should equal "only | the one output" + The line 3 of output should equal "--- only" End End @@ -136,7 +146,6 @@ Describe 'parallel.sh' The status should equal 4 The file "$WORK/slow-started" should be exist The file "$WORK/slow-finished" should not be exist - The output should include "... slow" The output should include "[.] slow cancelled" End @@ -166,10 +175,57 @@ Describe 'parallel.sh' End End - Describe 'output grouping' + # STREAM=0 holds each chain's output and prints it in one block instead, + # in declaration order, which is what a build that would rather read the + # whole of a chain at once asks for. + Describe 'grouped output' + It "prints each chain's output in one block" + Data + #|set -eu + #|STREAM=0 + #|. "$LIB" + #|chain "first" "printf 'one output\n'" + #|chain "second" "printf 'two output\n'" + #|run + End + + When call driver + The status should equal 0 + The output should match pattern "*--- first*one output*--- second*two output*" + End + + It 'marks a chain that failed and one that was cancelled' + Data + #|set -eu + #|STREAM=0 + #|. "$LIB" + #|chain "slow" \ + #| "touch '$WORK/slow-started'" \ + #| "until [ -f '$WORK/done' ] || [ ! -d '$WORK' ]; do sleep 1; done" + #|chain "broken" \ + #| "until [ -f '$WORK/slow-started' ]; do sleep 1; done" \ + #| "printf 'the reason\n'; exit 2" + #|run + End + + When call driver + The status should equal 2 + # Grouped output is in declaration order all the way down, so + # every line of it is known. Lines rather than a pattern: a + # bracket is a set in a glob, `[!]` a set that opens by + # negating itself, and ksh93 will not match a pattern of + # several parts against a subject this long anyway. + The line 2 of output should equal "... slow" + The line 3 of output should equal "[.] slow cancelled" + The line 5 of output should equal "[!] broken" + The line 6 of output should equal "the reason" + The line 7 of output should equal "[!] broken exited 2" + End + It "collects a command's stdout and stderr into its own group" Data #|set -eu + #|STREAM=0 #|. "$LIB" #|chain "noisy" "printf 'went to stdout\n'; printf 'went to stderr\n' >&2" #|chain "quiet" "printf 'other chain\n'" @@ -185,6 +241,7 @@ Describe 'parallel.sh' It 'passes command strings through untouched' Data #|set -eu + #|STREAM=0 #|. "$LIB" #|cd "$WORK" #|touch alpha beta @@ -309,13 +366,13 @@ Describe 'parallel.sh' End End - # STREAM=1 prints each line as it arrives, under the label of the chain - # it came from, instead of holding the output and grouping it at the end. + # What `run` does unless a build asks for grouping: each line as it + # arrives, under the label of the chain it came from. Describe 'streaming' It 'labels every line with the chain it came from' Data #|set -eu - #|STREAM=1 STREAM_SEP='|' + #|STREAM_SEP='|' #|. "$LIB" #|chain "npm" "printf 'one\n'; printf 'two\n'" #|chain "composer" "printf 'three\n'" @@ -334,7 +391,7 @@ Describe 'parallel.sh' It 'pads the labels to the same width' Data #|set -eu - #|STREAM=1 STREAM_SEP='|' + #|STREAM_SEP='|' #|. "$LIB" #|chain "a" "printf 'short\n'" #|chain "a-much-longer-one" "printf 'long\n'" @@ -354,7 +411,7 @@ Describe 'parallel.sh' It 'prints a line while the chains are still running' Data #|set -eu - #|STREAM=1 STREAM_SEP='|' + #|STREAM_SEP='|' #|. "$LIB" #|chain "first" "printf 'the first line\n'" #|chain "second" \ @@ -375,7 +432,7 @@ Describe 'parallel.sh' It 'waits for a half-written line to be finished' Data #|set -eu - #|STREAM=1 STREAM_SEP='|' + #|STREAM_SEP='|' #|. "$LIB" #|chain "half" "printf 'a line'; sleep 1; printf ' in two writes\n'" #|run @@ -389,7 +446,7 @@ Describe 'parallel.sh' It 'prints a last line that never got its newline' Data #|set -eu - #|STREAM=1 STREAM_SEP='|' + #|STREAM_SEP='|' #|. "$LIB" #|chain "abrupt" "printf 'no newline here'" #|run @@ -405,7 +462,7 @@ Describe 'parallel.sh' It 'reports how each chain ended, a line each' Data #|set -eu - #|STREAM=1 STREAM_SEP='|' + #|STREAM_SEP='|' #|. "$LIB" #|chain "fine" "printf 'all good\n'" #|chain "slow" \ @@ -421,14 +478,30 @@ Describe 'parallel.sh' The status should equal 3 The output should include "fine | all good" The output should include "--- fine" - The output should include "... slow cancelled" + The output should include "[.] slow cancelled" The output should include "[!] broken exited 3" End + It "labels what a command writes to stderr as well" + Data + #|set -eu + #|STREAM_SEP='|' + #|. "$LIB" + #|chain "noisy" "printf 'to stdout\n'; printf 'to stderr\n' >&2" + #|run + End + + When call driver + The status should equal 0 + The output should include "noisy | to stdout" + The output should include "noisy | to stderr" + The stderr should equal "" + End + It 'takes the bar between label and line from STREAM_SEP' Data #|set -eu - #|STREAM=1 STREAM_SEP='>>' + #|STREAM_SEP='>>' #|. "$LIB" #|chain "one" "printf 'a line\n'" #|run From f815e4af31eb51b36a7b89a71d6c9bc85b69a99f Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 20 Aug 2026 01:14:21 +0000 Subject: [PATCH 4/4] Say nothing at the end about a chain that finished A streamed run ended on a line a chain, and the line for a chain that finished was a heading with nothing under it: `--- npm` reads as the start of a block in grouped output, and in streamed output the block is already above it, spread through everything else that was running at the time. What is worth saying at the end is what the output does not already say: which chain failed, with what status, and which chains were cancelled with it. A run where nothing failed now ends on its last line of output, blank line included, since there is nothing to put under it. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01WWgsYpG4sTSefc6FQURumD --- README.md | 14 +++++++------- parallel.sh | 18 +++++++++++------- spec/parallel_spec.sh | 20 ++++++++++---------- 3 files changed, 28 insertions(+), 24 deletions(-) diff --git a/README.md b/README.md index 4092fc6..6e18f84 100644 --- a/README.md +++ b/README.md @@ -69,8 +69,9 @@ gantt lint, exits 1 :crit, 00:00:00, 00:00:10 ``` -Every line says which chain wrote it, the labels are right aligned so the -bars line up, and the last word is a line a chain saying how it ended: +Every line says which chain wrote it, and the labels are right aligned so +the bars line up. A build that works prints its output and stops there; when +one fails, the last word is what failed and what was cancelled with it: ``` composer install │ Installing dependencies @@ -79,7 +80,6 @@ composer install │ Generating optimized autoload files npm build │ building for production... npm build │ ERR! Build failed in 4.21s ---- composer install [!] npm build exited 1 [.] assets cancelled ``` @@ -112,10 +112,10 @@ the first failure. Each command is a string, evaluated by the shell, so pipes, redirects and `&&` work inside one. `run` waits for every chain, prints what they write under the label of the -chain that wrote it, ends with a line a chain saying how each one ended, and -returns the exit code of the chain that failed, or 0. The first failure -cancels the chains still running. Because `run` returns that code, it can be -the last line of a build script: +chain that wrote it, ends by naming the chain that failed and the chains +cancelled with it, and returns the exit code of the chain that failed, or 0. +The first failure cancels the chains still running. Because `run` returns +that code, it can be the last line of a build script: ```sh run diff --git a/parallel.sh b/parallel.sh index cc94b42..300cd00 100644 --- a/parallel.sh +++ b/parallel.sh @@ -361,7 +361,11 @@ _emit() { } _report() { - _streaming && printf '\n' + # The blank line only when there is a line to put under it. A streamed + # run says nothing at the end about a chain that finished, and chains + # are cancelled by a failure and by nothing else, so a run that failed + # is exactly a run with something left to say. + _streaming && [ "$_failed" -ne 0 ] && printf '\n' i=1 while [ "$i" -le "$_count" ]; do label=$(cat "$_work/$i.label") @@ -378,16 +382,16 @@ _report() { fi if _streaming; then - # The output has already gone by, in the order it happened, so - # what is left to report is a line a chain saying how it ended. - # They are the lines the grouped report ends a chain on, so a - # build that greps for one finds it either way. + # Only what the output did not already say. A chain that + # finished said so line by line as it went, and a heading with + # nothing under it is a heading for nothing; what is left is + # the chain that failed and the chains that went down with it. + # The lines are the ones the grouped report ends a chain on, so + # a build that greps for one finds it either way. if [ -f "$_work/$i.cancelled" ]; then printf '[.] %s cancelled\n' "$label" elif [ -n "$code" ] && [ "$code" -ne 0 ]; then printf '[!] %s exited %s\n' "$label" "$code" - else - printf '%s %s\n' "$mark" "$label" fi else printf '\n%s %s\n' "$mark" "$label" diff --git a/spec/parallel_spec.sh b/spec/parallel_spec.sh index 656274b..ad3382c 100644 --- a/spec/parallel_spec.sh +++ b/spec/parallel_spec.sh @@ -28,11 +28,10 @@ Describe 'parallel.sh' The status should equal 0 The output should include " first | one output" The output should include "second | two output" - # Then a line a chain, in declaration order. Which of the two - # labelled lines lands first is up to the chains, but the - # report is always the last two. - The line 4 of output should equal "--- first" - The line 5 of output should equal "--- second" + # And that is the whole of it. A chain that finished said so as + # it went, so there is nothing to add under its name at the end. + The output should not include "--- first" + The output should not include "--- second" End It 'handles a single chain with a single command' @@ -47,7 +46,7 @@ Describe 'parallel.sh' When call driver The status should equal 0 The line 1 of output should equal "only | the one output" - The line 3 of output should equal "--- only" + The output should not include "--- only" End End @@ -457,9 +456,10 @@ Describe 'parallel.sh' The line 1 of output should equal "abrupt | no newline here" End - # The output has already gone by, so the report at the end is one - # line a chain, in declaration order, saying how each one ended. - It 'reports how each chain ended, a line each' + # What is left to say when the output has gone by: the chain that + # failed, and the chains that were cancelled with it. Nothing about + # the chain that finished, which said so as it went. + It 'ends on what failed and what was cancelled with it' Data #|set -eu #|STREAM_SEP='|' @@ -477,7 +477,7 @@ Describe 'parallel.sh' When call driver The status should equal 3 The output should include "fine | all good" - The output should include "--- fine" + The output should not include "--- fine" The output should include "[.] slow cancelled" The output should include "[!] broken exited 3" End