diff --git a/README.md b/README.md index 068317a..6e18f84 100644 --- a/README.md +++ b/README.md @@ -3,9 +3,9 @@ [![CI](https://github.com/glhd/parallel-build/actions/workflows/ci.yml/badge.svg)](https://github.com/glhd/parallel-build/actions/workflows/ci.yml) `parallel.sh` runs chains of commands at the same time, stops everything at -the first failure, and prints each chain's output in one block instead of -interleaved. It is POSIX sh, one file, and nothing else, so it runs in a -build container as it is. +the first failure, and labels every line it prints with the chain it came +from, as that chain writes it. It is POSIX sh, one file, and nothing else, +so it runs in a build container as it is. Before, where `npm` waits on `composer` for no reason: @@ -69,23 +69,26 @@ gantt lint, exits 1 :crit, 00:00:00, 00:00:10 ``` -When something fails, the failure is at the bottom and the rest is grouped -above it: +Every line says which chain wrote it, and the labels are right aligned so +the bars line up. A build that works prints its output and stops there; when +one fails, the last word is what failed and what was cancelled with it: ``` ---- composer install -Installing dependencies -Generating autoload files +composer install │ Installing dependencies + npm build │ added 214 packages +composer install │ Generating optimized autoload files + npm build │ building for production... + npm build │ ERR! Build failed in 4.21s -[!] npm build -added 214 packages -building for production... [!] npm build exited 1 - -... assets [.] assets cancelled ``` +A line is printed once it is whole, so a command that writes a line in two +writes still gets one line, and the chain it belongs to is the only thing +that decides where it goes. `STREAM=0` holds the output and groups it +instead, which is [below](#stream). + ## Install It is one file. Vendor it: @@ -108,10 +111,11 @@ order, each one only if the one before it succeeded, and the chain stops at the first failure. Each command is a string, evaluated by the shell, so pipes, redirects and `&&` work inside one. -`run` waits for every chain, prints their output in the order they were -declared, and returns the exit code of the chain that failed, or 0. The -first failure cancels the chains still running. Because `run` returns that -code, it can be the last line of a build script: +`run` waits for every chain, prints what they write under the label of the +chain that wrote it, ends by naming the chain that failed and the chains +cancelled with it, and returns the exit code of the chain that failed, or 0. +The first failure cancels the chains still running. Because `run` returns +that code, it can be the last line of a build script: ```sh run @@ -128,8 +132,51 @@ first poll, after which polling falls back to whole seconds. Set `POLL` for a different interval and `POLL_WHOLE` for a different fallback: ```sh -POLL=0.5 . ./parallel.sh +POLL=0.5 +. ./parallel.sh +``` + +Set it before sourcing rather than in front of the `.`, which is a special +built-in: an assignment in front of one persists in a POSIX shell and does +not in bash or zsh, so `POLL=0.5 . ./parallel.sh` is two different things +depending on where it runs. The environment works everywhere too, so +`POLL=0.5 ./build.sh` on a script that sources the library is the other way +to do it. + +## STREAM + +Output is labelled and printed as it arrives. `STREAM=0` holds each chain's +output instead and prints it in one block a chain, in declaration order, +with the failure at the bottom — nothing is printed until a chain has +finished, and nothing a chain wrote is anywhere but under its own heading: + +```sh +STREAM=0 ./build.sh +``` + ``` +--- composer install +Installing dependencies +Generating optimized autoload files + +[!] npm build +added 214 packages +building for production... +[!] npm build exited 1 + +... assets +[.] assets cancelled +``` + +`STREAM_SEP` is the bar between a label and its line. It defaults to `│` +where the locale says the terminal is UTF-8, and to `|` where it does not: + +```sh +STREAM_SEP='|' ./build.sh +``` + +Labelled output is read on the same poll that watches for finished chains, +so a line can be up to `POLL` behind the command that wrote it. ## Benchmarks @@ -162,8 +209,8 @@ best of three runs: cannot finish before its longest chain does, so the most chains can do is hide the rest behind it. -Eight chains that do nothing at all finish in 0.15s: one `mktemp`, eight -forks, and a poll or two. That is about what the library costs a build with +Eight chains that do nothing at all finish in 0.15s: one `mktemp`, one job +control probe, eight forks, and a poll or two. That is about what the library costs a build with nothing to gain. The table is a report on shapes, not a measurement of any real build. A @@ -185,10 +232,18 @@ too noisy to be held to one. ## Caveats -`kill` stops a cancelled chain's shell, not its grandchildren. A command -that spawned its own children can leave one behind that outlives the chain -it belonged to. In a build container that exits anyway this does not matter; -in a long-lived shell it will. +Cancelling a chain kills its process group where the shell can give a chain +one of its own, which takes down what the chain started however deep it +goes. Job control is what puts a chain in a group, and a non-interactive +shell is not obliged to have any, so the library starts one job under +`set -m` at the first `chain` and asks whether that job got a group to +itself. bash, ksh93 and mksh give it one, and zsh does when it has a +terminal. dash and busybox ash accept `set -m` and start the job in the +shell's own group regardless, and there cancelling still kills the chain's +shell and not its grandchildren: a command that spawned its own children can +leave one behind that outlives the chain it belonged to. In a build +container that exits anyway this does not matter; in a long-lived shell it +will. Fractional `sleep` is not in POSIX, though both GNU coreutils and BSD accept it. Whole seconds are the fallback, not the default, so a build on a shell @@ -201,8 +256,8 @@ under zsh, removes it. The library has no `local`, because POSIX sh has none. It keeps its own state in names starting with `_`, but its loops use `i`, `n`, `cmd`, `code`, -`label` and `mark`, and sourcing it will clobber those in the calling -script. +`label`, `mark`, `seen`, `line` and `pending`, and sourcing it will clobber +those in the calling script. ## Prior art diff --git a/parallel.sh b/parallel.sh index 7a0921f..300cd00 100644 --- a/parallel.sh +++ b/parallel.sh @@ -7,17 +7,32 @@ # # Commands in a chain run in order and stop at the first failure. Separate # chains run at the same time. The first failure anywhere cancels the rest. -# `run` blocks, prints each chain's output in declaration order, and returns -# the exit code of the chain that failed. +# `run` blocks, prints what the chains write under the label of the chain +# that wrote it, and returns the exit code of the chain that failed. +# +# With STREAM=0 the output is held instead, and printed in one block a chain +# in declaration order. POLL=${POLL:-0.1} # seconds between checks POLL_WHOLE=${POLL_WHOLE:-1} # used instead if this sleep rejects fractions +STREAM=${STREAM:-1} # 0 to hold the output and print it grouped + +# The bar between a label and its line. A box drawing character reads as +# three bytes of noise in a terminal that is not expecting UTF-8, and the +# locale is what says whether it is, so the default follows the locale and +# an ASCII pipe stands in everywhere else. +case ${LC_ALL:-${LC_CTYPE:-${LANG:-}}} in +*[Uu][Tt][Ff]8* | *[Uu][Tt][Ff]-8*) STREAM_SEP=${STREAM_SEP:-'│'} ;; +*) STREAM_SEP=${STREAM_SEP:-'|'} ;; +esac _work=$(mktemp -d) _count=0 _failed=0 _signal=0 # what to exit with if a signal arrives; 0 until one does _fractional='' # whether sleep takes POLL; unknown until the first nap +_groups='' # whether a chain can have its own process group; unknown +_monitor='' # whether the calling shell had job control on already _cleanup() { rm -rf "$_work" || :; } @@ -28,6 +43,10 @@ _cleanup() { rm -rf "$_work" || :; } # for a signal at all, so the signal traps clean up and exit themselves; and # a trap whose test fails is abandoned under set -e, so _finish decides with # a case, which cannot fail, rather than a test, which can. +# +# Nothing here has anything to say, and one shell has: ksh93 reports the +# chains this trap killed, on the way out, on the stderr of a script that is +# already leaving. That is what the redirection is for. _finish() { _ec=$1 case $_signal in @@ -37,7 +56,7 @@ _finish() { _stop_chains _cleanup exit "$_ec" -} +} 2>/dev/null # job control's last word on the chains, and not the script's # Take the chains down on the way out. Nothing else does: a chain is its own # process, so an interrupted build that only cleaned up after itself would @@ -50,12 +69,88 @@ _stop_chains() { i=1 while [ "$i" -le "$_count" ]; do if [ ! -f "$_work/$i.code" ] && [ -f "$_work/$i.pid" ]; then - kill "$(cat "$_work/$i.pid")" 2>/dev/null || : + _kill_chain "$(cat "$_work/$i.pid")" now fi i=$((i + 1)) done } +# Stop one chain, and with it everything the chain started where the shell +# was able to give the chain a process group of its own. +# +# The chain's own shell goes first, on its pid. Killing the group outright +# would work as well, but the chain would still be there to see the command +# it was running killed, and shells report that: what it came to was a +# `Terminated` in the middle of a cancelled chain's output under bash and +# mksh, and a line about the job from ksh93. So `quietly` waits for the +# chain's shell to go before taking the group, and once it has gone there is +# nobody left to report anything. What is left in the group is what the +# chain started, however deep it goes: a group outlives its leader for +# exactly as long as one of them is still running, which is as long as there +# is anything in it worth killing. +# +# `now` skips that wait, and the exit trap uses it. Nothing that trap kills +# will be printed, so it has nothing to keep quiet for, and a chain that has +# made itself deaf to the signal must not be able to hold the trap — and the +# script — open while it waits for a shell that is not going to go. +# +# Neither kill can fail: cancelling races the chain finishing on its own, +# and this is called from the exit trap, which a failure would abandon. +_kill_chain() { + kill "$1" 2>/dev/null || : + case $_groups in + yes) + case $2 in + quietly) wait "$1" 2>/dev/null || : ;; + esac + kill -- "-$1" 2>/dev/null || : + ;; + esac +} + +# Whether this shell will put a background job in a process group of its own, +# which is what makes it possible to cancel a chain's children along with the +# chain. Job control is POSIX, but a non-interactive shell is allowed to +# leave it out: dash and busybox ash take `set -m` and start the job in the +# shell's own group anyway, and zsh refuses the option outright unless it has +# a terminal to hand the group the foreground with. +# +# The probe starts its job here, in the shell that will be starting the +# chains, and not in a subshell: a subshell is a different place to ask from +# and gives different answers. mksh says the group is there and then will not +# kill it, and FreeBSD's sh answered for a subshell what was not true of the +# script — which is a probe reporting on itself rather than on the chains. +# +# A job that was given its own group leads that group, so a group with its +# pid for an id exists. A job that was not is in the shell's group, and no +# other group can have that id while the job itself holds the pid, so asking +# after the group is the whole of the test and `ps` is not needed for it. +_probe_groups() { + case $- in + *m*) _monitor=yes ;; + *) _monitor=no ;; + esac + _groups=no + + # Whether the option can be set at all is a question for a subshell, + # because `set` is a special builtin and a special builtin that fails + # takes a non-interactive shell down with it: zsh, which refuses -m + # without a terminal, would end the build script rather than answer. + (set -m) 2>/dev/null || return 0 + set -m 2>/dev/null # dash says out loud that it has no terminal for it + + # Braces and a redirection for the same reason `chain` has them: a shell + # with job control announces the jobs it starts, and this one is not the + # script's news. SIGKILL because the probe must not be able to hang. + { sleep 1 & } >/dev/null 2>&1 + _p=$! + if kill -0 -- "-$_p" 2>/dev/null; then _groups=yes; fi + kill -9 -- "-$_p" 2>/dev/null || kill -9 "$_p" 2>/dev/null + wait "$_p" 2>/dev/null || : + + case $_monitor in no) set +m 2>/dev/null ;; esac +} + trap '_finish $?' EXIT trap '_signal=130; _finish 130' INT trap '_signal=143; _finish 143' TERM @@ -66,30 +161,62 @@ chain() { printf '%s' "$1" >"$_work/$n.label" shift - ( - trap - INT TERM # don't inherit the parent's handlers - code=0 - # Always leave a status behind, even if a command calls exit. - # Write, then rename, so the reader never sees a half-written file. - # shellcheck disable=SC2154 # ec is assigned in the same trap - trap 'ec=$?; [ "$code" -ne 0 ] || code=$ec - printf "%s" "$code" >"$_work/$n.code.part" - mv "$_work/$n.code.part" "$_work/$n.code"' EXIT - - for cmd in "$@"; do - eval "$cmd" || { - code=$? - break - } - done - ) >"$_work/$n.log" 2>&1 & + # Job control decides which process group a job starts in, and decides it + # when the job starts, so the option only has to be on across the fork + # below. It goes back afterwards, because this is the calling script's + # shell: under job control that script would find its own background jobs + # taken out of its process group too. + case $_groups in '') _probe_groups ;; esac + case $_groups in yes) set -m ;; esac + + # The braces are for zsh, which announces every job it starts once it has + # job control, on the calling script's stdout, in the middle of the report + # the script is there to print. The announcement is this shell's, not the + # chain's, so it is this shell's stdout that has to point elsewhere while + # the chain starts. The chain's own output goes to its log either way, and + # a fork that fails still has stderr to say so on. + { + ( + trap - INT TERM # don't inherit the parent's handlers + + # The chain has the group; it does not need job control of + # its own, and is worse off with it. A chain that kept it + # would give a group of its own to what it started, putting + # it outside the group that cancelling kills. Off on what + # the library turned on, not on what `$-` reports: dash + # leaves `m` out of `$-` with monitor mode set, and the + # ash-derived shells it is one of are exactly the ones this + # would be wrong about. + case $_groups in yes) set +m ;; esac + + code=0 + # Always leave a status behind, even if a command calls + # exit. Write, then rename, so the reader never sees a + # half-written file. + # shellcheck disable=SC2154 # ec is assigned in the same trap + trap 'ec=$?; [ "$code" -ne 0 ] || code=$ec + printf "%s" "$code" >"$_work/$n.code.part" + mv "$_work/$n.code.part" "$_work/$n.code"' EXIT + + for cmd in "$@"; do + eval "$cmd" || { + code=$? + break + } + done + ) >"$_work/$n.log" 2>&1 & + } >/dev/null printf '%s' "$!" >"$_work/$n.pid" + + if [ "$_groups" = yes ] && [ "$_monitor" = no ]; then set +m; fi } run() { + _streaming && _prefixes _await _cancel + _streaming && _flush _report return "$_failed" } @@ -118,6 +245,7 @@ _nap() { # Wait for everything, or return early as soon as one chain fails _await() { while :; do + _streaming && _pump pending=0 i=1 while [ "$i" -le "$_count" ]; do @@ -142,14 +270,102 @@ _cancel() { while [ "$i" -le "$_count" ]; do if [ ! -f "$_work/$i.code" ]; then : >"$_work/$i.cancelled" - kill "$(cat "$_work/$i.pid")" 2>/dev/null + _kill_chain "$(cat "$_work/$i.pid")" quietly fi i=$((i + 1)) done wait 2>/dev/null || : # a killed job must not abort a script that set -e } +# Whether output is streamed as it arrives rather than held and grouped. +_streaming() { + case ${STREAM:-1} in + '' | 0 | no | off | false) return 1 ;; + *) return 0 ;; + esac +} + +# The label a streamed line carries, one per chain, right aligned to the +# longest of them so that the bars line up under each other. Labels are +# known by the time `run` is called, which is the first moment a width can +# be worked out, and they are kept in variables because the pump wants them +# on every poll. +_prefixes() { + _pad=0 + i=1 + while [ "$i" -le "$_count" ]; do + label=$(cat "$_work/$i.label") + [ "${#label}" -gt "$_pad" ] && _pad=${#label} + i=$((i + 1)) + done + + i=1 + while [ "$i" -le "$_count" ]; do + label=$(cat "$_work/$i.label") + while [ "${#label}" -lt "$_pad" ]; do label=" $label"; done + eval "_label_$i=\$label" + i=$((i + 1)) + done +} + +# What every chain has written since the last look round. The poll is the +# only thing printing, so two chains that write at the same moment come out +# as whole lines one after the other rather than mixed into each other. +_pump() { + i=1 + while [ "$i" -le "$_count" ]; do + _emit "$i" now + i=$((i + 1)) + done +} + +# The rest of it, once the chains are done: what the last poll did not get +# to, and the final line of a chain that ended without a newline. +_flush() { + i=1 + while [ "$i" -le "$_count" ]; do + _emit "$i" last + i=$((i + 1)) + done +} + +# One chain's new output. The count of lines already printed is the whole of +# the reader's position: `tail` starts from the line after it, and a line +# that has no newline yet is left where it is, so a line written in two goes +# out in one piece instead of as two labelled halves. `read` fails on that +# partial line without counting it, which is what leaves it to be read again +# next time round — until `last`, when there is no next time and what is +# there is all there will be. +# +# The loop reads a file rather than a pipe on purpose: a pipe would put it +# in a subshell in most shells, and the count it keeps would go with it. +_emit() { + n=$1 + eval "seen=\${_seen_$n:-0} label=\$_label_$n" + tail -n "+$((seen + 1))" "$_work/$n.log" \ + >"$_work/$n.chunk" 2>/dev/null || return 0 + + line='' + while IFS= read -r line; do + printf '%s %s %s\n' "$label" "$STREAM_SEP" "$line" + seen=$((seen + 1)) + line='' + done <"$_work/$n.chunk" + + if [ -n "$line" ] && [ "$2" = last ]; then + printf '%s %s %s\n' "$label" "$STREAM_SEP" "$line" + seen=$((seen + 1)) + fi + + eval "_seen_$n=\$seen" +} + _report() { + # The blank line only when there is a line to put under it. A streamed + # run says nothing at the end about a chain that finished, and chains + # are cancelled by a failure and by nothing else, so a run that failed + # is exactly a run with something left to say. + _streaming && [ "$_failed" -ne 0 ] && printf '\n' i=1 while [ "$i" -le "$_count" ]; do label=$(cat "$_work/$i.label") @@ -165,11 +381,25 @@ _report() { mark="[!]" fi - printf '\n%s %s\n' "$mark" "$label" - cat "$_work/$i.log" - [ -n "$code" ] && [ "$code" -ne 0 ] && - printf '[!] %s exited %s\n' "$label" "$code" - [ -f "$_work/$i.cancelled" ] && printf '[.] %s cancelled\n' "$label" + if _streaming; then + # Only what the output did not already say. A chain that + # finished said so line by line as it went, and a heading with + # nothing under it is a heading for nothing; what is left is + # the chain that failed and the chains that went down with it. + # The lines are the ones the grouped report ends a chain on, so + # a build that greps for one finds it either way. + if [ -f "$_work/$i.cancelled" ]; then + printf '[.] %s cancelled\n' "$label" + elif [ -n "$code" ] && [ "$code" -ne 0 ]; then + printf '[!] %s exited %s\n' "$label" "$code" + fi + else + printf '\n%s %s\n' "$mark" "$label" + cat "$_work/$i.log" + [ -n "$code" ] && [ "$code" -ne 0 ] && + printf '[!] %s exited %s\n' "$label" "$code" + [ -f "$_work/$i.cancelled" ] && printf '[.] %s cancelled\n' "$label" + fi i=$((i + 1)) done } diff --git a/spec/parallel_spec.sh b/spec/parallel_spec.sh index 408b192..ad3382c 100644 --- a/spec/parallel_spec.sh +++ b/spec/parallel_spec.sh @@ -12,9 +12,12 @@ Describe 'parallel.sh' Describe 'chains that succeed' - It 'runs two chains and reports both, in declaration order' + # Output is labelled and printed as it arrives unless a build asks + # for it grouped, so this is what a build sees by default. + It 'labels every line with the chain it came from' Data #|set -eu + #|STREAM_SEP='|' #|. "$LIB" #|chain "first" "printf 'one output\n'" #|chain "second" "printf 'two output\n'" @@ -23,12 +26,18 @@ Describe 'parallel.sh' When call driver The status should equal 0 - The output should match pattern "*--- first*one output*--- second*two output*" + The output should include " first | one output" + The output should include "second | two output" + # And that is the whole of it. A chain that finished said so as + # it went, so there is nothing to add under its name at the end. + The output should not include "--- first" + The output should not include "--- second" End It 'handles a single chain with a single command' Data #|set -eu + #|STREAM_SEP='|' #|. "$LIB" #|chain "only" "printf 'the one output\n'" #|run @@ -36,8 +45,8 @@ Describe 'parallel.sh' When call driver The status should equal 0 - The line 2 of output should equal "--- only" - The line 3 of output should equal "the one output" + The line 1 of output should equal "only | the one output" + The output should not include "--- only" End End @@ -136,15 +145,86 @@ Describe 'parallel.sh' The status should equal 4 The file "$WORK/slow-started" should be exist The file "$WORK/slow-finished" should not be exist - The output should include "... slow" The output should include "[.] slow cancelled" End + + # The chain is a shell of its own, so killing it on its pid alone + # leaves whatever it was running still running: the compiler + # outlives the build that gave up on it. Where the shell can put a + # chain in a process group of its own, the whole group goes, and + # what the chain started goes with it however deep it is. + It "kills what a cancelled chain started, not only the chain" + Skip if "this shell leaves a background job in its own process group" no_process_groups + + Data + #|set -eu + #|. "$LIB" + #|chain "slow" \ + #| "sh -c 'sleep 30 & printf %s \$! >\"$WORK/childpid\"; sleep 30'" + #|chain "quick" \ + #| "until [ -f '$WORK/childpid' ]; do sleep 1; done" \ + #| "exit 4" + #|run + End + + When call driver + The status should equal 4 + The output should include "[.] slow cancelled" + The value "$(process_state "$WORK/childpid")" should equal "stopped" + End End - Describe 'output grouping' + # STREAM=0 holds each chain's output and prints it in one block instead, + # in declaration order, which is what a build that would rather read the + # whole of a chain at once asks for. + Describe 'grouped output' + It "prints each chain's output in one block" + Data + #|set -eu + #|STREAM=0 + #|. "$LIB" + #|chain "first" "printf 'one output\n'" + #|chain "second" "printf 'two output\n'" + #|run + End + + When call driver + The status should equal 0 + The output should match pattern "*--- first*one output*--- second*two output*" + End + + It 'marks a chain that failed and one that was cancelled' + Data + #|set -eu + #|STREAM=0 + #|. "$LIB" + #|chain "slow" \ + #| "touch '$WORK/slow-started'" \ + #| "until [ -f '$WORK/done' ] || [ ! -d '$WORK' ]; do sleep 1; done" + #|chain "broken" \ + #| "until [ -f '$WORK/slow-started' ]; do sleep 1; done" \ + #| "printf 'the reason\n'; exit 2" + #|run + End + + When call driver + The status should equal 2 + # Grouped output is in declaration order all the way down, so + # every line of it is known. Lines rather than a pattern: a + # bracket is a set in a glob, `[!]` a set that opens by + # negating itself, and ksh93 will not match a pattern of + # several parts against a subject this long anyway. + The line 2 of output should equal "... slow" + The line 3 of output should equal "[.] slow cancelled" + The line 5 of output should equal "[!] broken" + The line 6 of output should equal "the reason" + The line 7 of output should equal "[!] broken exited 2" + End + It "collects a command's stdout and stderr into its own group" Data #|set -eu + #|STREAM=0 #|. "$LIB" #|chain "noisy" "printf 'went to stdout\n'; printf 'went to stderr\n' >&2" #|chain "quiet" "printf 'other chain\n'" @@ -160,6 +240,7 @@ Describe 'parallel.sh' It 'passes command strings through untouched' Data #|set -eu + #|STREAM=0 #|. "$LIB" #|cd "$WORK" #|touch alpha beta @@ -283,4 +364,152 @@ Describe 'parallel.sh' The value "$(chain_state)" should equal "stopped" End End + + # What `run` does unless a build asks for grouping: each line as it + # arrives, under the label of the chain it came from. + Describe 'streaming' + It 'labels every line with the chain it came from' + Data + #|set -eu + #|STREAM_SEP='|' + #|. "$LIB" + #|chain "npm" "printf 'one\n'; printf 'two\n'" + #|chain "composer" "printf 'three\n'" + #|run + End + + When call driver + The status should equal 0 + The output should include " npm | one" + The output should include " npm | two" + The output should include "composer | three" + End + + # The labels are right aligned to the longest of them, so the bars + # line up whatever the chains are called. + It 'pads the labels to the same width' + Data + #|set -eu + #|STREAM_SEP='|' + #|. "$LIB" + #|chain "a" "printf 'short\n'" + #|chain "a-much-longer-one" "printf 'long\n'" + #|run + End + + When call driver + The status should equal 0 + The line 1 of output should equal " a | short" + The line 2 of output should equal "a-much-longer-one | long" + End + + # The point of it: the parent prints what a chain wrote while the + # other chains are still going. The second chain here waits to see + # the first chain's line in what `run` has already printed, which it + # can only do if the line was printed before either chain finished. + It 'prints a line while the chains are still running' + Data + #|set -eu + #|STREAM_SEP='|' + #|. "$LIB" + #|chain "first" "printf 'the first line\n'" + #|chain "second" \ + #| "until grep -q 'the first line' '$WORK/out' 2>/dev/null || + #| [ ! -d '$WORK' ]; do sleep 1; done" \ + #| "printf 'saw it\n'" + #|run >"$WORK/out" + #|cat "$WORK/out" + End + + When call driver + The status should equal 0 + The output should include "second | saw it" + End + + # A line is a line once it has its newline. A chain that writes one + # in two goes gets one labelled line, not two. + It 'waits for a half-written line to be finished' + Data + #|set -eu + #|STREAM_SEP='|' + #|. "$LIB" + #|chain "half" "printf 'a line'; sleep 1; printf ' in two writes\n'" + #|run + End + + When call driver + The status should equal 0 + The line 1 of output should equal "half | a line in two writes" + End + + It 'prints a last line that never got its newline' + Data + #|set -eu + #|STREAM_SEP='|' + #|. "$LIB" + #|chain "abrupt" "printf 'no newline here'" + #|run + End + + When call driver + The status should equal 0 + The line 1 of output should equal "abrupt | no newline here" + End + + # What is left to say when the output has gone by: the chain that + # failed, and the chains that were cancelled with it. Nothing about + # the chain that finished, which said so as it went. + It 'ends on what failed and what was cancelled with it' + Data + #|set -eu + #|STREAM_SEP='|' + #|. "$LIB" + #|chain "fine" "printf 'all good\n'" + #|chain "slow" \ + #| "touch '$WORK/slow-started'" \ + #| "until [ -f '$WORK/done' ] || [ ! -d '$WORK' ]; do sleep 1; done" + #|chain "broken" \ + #| "until [ -f '$WORK/slow-started' ]; do sleep 1; done" \ + #| "exit 3" + #|run + End + + When call driver + The status should equal 3 + The output should include "fine | all good" + The output should not include "--- fine" + The output should include "[.] slow cancelled" + The output should include "[!] broken exited 3" + End + + It "labels what a command writes to stderr as well" + Data + #|set -eu + #|STREAM_SEP='|' + #|. "$LIB" + #|chain "noisy" "printf 'to stdout\n'; printf 'to stderr\n' >&2" + #|run + End + + When call driver + The status should equal 0 + The output should include "noisy | to stdout" + The output should include "noisy | to stderr" + The stderr should equal "" + End + + It 'takes the bar between label and line from STREAM_SEP' + Data + #|set -eu + #|STREAM_SEP='>>' + #|. "$LIB" + #|chain "one" "printf 'a line\n'" + #|run + End + + When call driver + The status should equal 0 + The line 1 of output should equal "one >> a line" + End + End End diff --git a/spec/spec_helper.sh b/spec/spec_helper.sh index 3d6660d..ae782cd 100644 --- a/spec/spec_helper.sh +++ b/spec/spec_helper.sh @@ -178,13 +178,25 @@ reported_workdir() { } # Whether the chain a finished driver reported through $WORK/chainpid is -# still running. A process that has just been killed can sit as a zombie -# until the shell that started it goes and init reaps it, so this waits a -# few seconds for an answer rather than believing the first one. +# still running. chain_state() { - _pid=$(cat "$WORK/chainpid") + process_state "$WORK/chainpid" +} + +# The same for any pid a driver left in a file, which is how a chain's own +# children are asked after. A process that has just been killed can sit as a +# zombie until whatever adopted it reaps it, so this waits a few seconds for +# an answer rather than believing the first one, and reads a pid that is +# still answering as stopped once it has become a zombie: a zombie is a +# process that has died, whatever `kill -0` makes of it. Not every `ps` here +# takes -p, and the ones that do not simply leave the wait to decide. +process_state() { + _pid=$(cat "$1") _waited=0 while kill -0 "$_pid" 2>/dev/null; do + case $(ps -o stat= -p "$_pid" 2>/dev/null) in + *Z*) break ;; + esac if [ "$_waited" -ge 5 ]; then printf 'running' return 0 @@ -195,6 +207,22 @@ chain_state() { printf 'stopped' } +# Whether the shell under test leaves a background job in the shell's own +# process group, in which case cancelling a chain cannot take the chain's +# children with it. Job control is optional in a non-interactive shell: dash +# and busybox ash take `set -m` and start the job in the shell's group +# anyway, and zsh refuses it outright without a terminal. +# +# The library is asked rather than a copy of its probe run here: a copy can +# answer differently from the real one — the FreeBSD runner had them +# disagree — and then an example is skipped that should have run, or run +# that should have been skipped, and the failure reads as a broken kill. +no_process_groups() { + # shellcheck disable=SC2016 # the probe is the other shell's to expand + [ "$("$SHELLSPEC_SHELL" -c '. "$LIB"; _probe_groups; printf %s "$_groups"' \ + 2>/dev/null)" != yes ] +} + lines_in() { wc -l <"$1" | tr -d ' ' }