Alternate signal stack: --use_alt_stack=no cannot prevent the install, and SIGSTKSZ sizing aborts the test binary on musl

[jamescowens]

Summary

On musl-based systems (Alpine), a Boost.Test binary can die before running a single test:

Test setup error: system_error produced by: exp: Out of memory

The cause is sigaltstack(2) returning ENOMEM for an undersized buffer.
BOOST_TEST_ALT_STACK_SIZE is SIGSTKSZ, which glibc 2.34+ made dynamic but musl hardcodes at
8192. The kernel's actual requirement comes from the CPU's XSAVE area and is published as auxv
AT_MINSIGSTKSZ. On CPUs with AMX register state that value exceeds 8192, so the install fails.

The documented workaround, --use_alt_stack=no, does not prevent it.

1. --use_alt_stack=no is applied too late to matter

p_use_alt_stack is set from the command line in exactly one place,
unit_test_monitor_t::execute_and_translate (impl/unit_test_monitor.ipp), which runs test cases
and fixtures.

The failing install happens earlier. Captured with strace -k with the switch set:

sigaltstack({ss_sp=0x…, ss_flags=0, ss_size=14528}, NULL) = 0
  > libboost_unit_test_framework … execution_monitor::catch_signals
  > libboost_unit_test_framework … execution_monitor::execute
  > libboost_unit_test_framework … execution_monitor::vexecute
  > libboost_unit_test_framework … boost::unit_test::framework::init(bool(*)(), int, char**)
  > libboost_unit_test_framework … boost::unit_test::unit_test_main

framework::init() is where the command line is parsed, so the monitor it uses still holds the
constructor default. The switch is applied strictly after the call it is meant to suppress.

It is not inert — it does suppress the per-test-case installs (21 sigaltstack calls become 9 in
our suite) — it simply cannot reach the one that aborts the process.

The option's own help text says the feature "can be disabled using this command line switch",
which is not true for the init phase.

2. p_use_alt_stack's documented default contradicts the code

execution_monitor.hpp:

The @em p_use_alt_stack property is a boolean flag (default value is false) …

impl/execution_monitor.ipp:

, p_use_alt_stack( true )

The true default is what makes item 1 reachable at all.

3. BOOST_TEST_SYS_ASSERT stringizes the wrong token

#define BOOST_TEST_SYS_ASSERT( cond ) \
    BOOST_TEST_I_ASSRT( cond, ::boost::system_error( BOOST_STRINGIZE( exp ) ) )

The parameter is cond; it stringizes the literal token exp. Every one of the six call sites in
execution_monitor.ipp therefore reports the same useless name, which is why the message above
says exp: and identifies nothing. One-word fix: BOOST_STRINGIZE( cond ).

This one is minor in isolation and disproportionately expensive in practice — it was the main
reason diagnosing the above took as long as it did.

4. ~signal_handler disables an alternate stack it may not have installed

signal_handler::~signal_handler()
{
    ...
#ifdef BOOST_TEST_USE_ALT_STACK
    sigstk.ss_size  = MINSIGSTKSZ;
    sigstk.ss_flags = SS_DISABLE;
    if( ::sigaltstack( &sigstk, 0 ) == -1 ) { ...report... }
#endif

The install is guarded — it happens only if( sigstk.ss_flags & SS_DISABLE ), i.e. only when no
alternate stack was already active, and only when alt_stack is non-null. The teardown is not
guarded at all. It runs whenever BOOST_TEST_USE_ALT_STACK is compiled in, regardless of whether
this handler installed anything, and tears down whatever the caller had.

So a program that installs its own alternate signal stack before invoking Boost.Test has it
silently removed at the end of the first monitored scope, and never restored. That is caller state
the library did not create and does not own.

It also defeats the natural workaround for items 1-3. Installing an adequately sized stack before
main() does make framework::init() skip its install — but the matching destructor then disables
it, so every subsequent test case finds no stack active and gets Boost's own SIGSTKSZ-sized one.
Measured with strace, counting installs (ss_flags == 0) in one suite:

configuration	caller's install	Boost's installs
--use_alt_stack=no only	0	1 (from framework::init)
pre-installed stack only	1	6 (one per test case)
both	1	0

Suggested fix: remember whether this handler installed the stack, and on teardown restore the
previous stack_t rather than unconditionally disabling.

Reproduction

Needs musl and a CPU whose AT_MINSIGSTKSZ exceeds 8192. Measured relationship, from two
machines here — AT_MINSIGSTKSZ ≈ CPUID.0xD:EBX + ~940 bytes of signal-frame overhead:

CPU	XSAVE	AT_MINSIGSTKSZ
Xeon E5-2687W v2 (AVX)	832	1776
Core i9-13900K (AVX2)	2696	3632
CI runner, observed	≈11016	11952

~11 KB of XSAVE state is only reached with AMX (tile data is 8192 bytes above AVX-512's ~2.7 KB),
i.e. Sapphire Rapids and later. On such a host, under musl, every Boost.Test binary fails at
startup.

Suggested fixes (summary)

• Size from the kernel, not from SIGSTKSZ: use AT_MINSIGSTKSZ (or
  sysconf(_SC_SIGSTKSZ) where available) and take the maximum with SIGSTKSZ/MINSIGSTKSZ.
  This is the substantive fix; the rest is cosmetic by comparison.
• Apply p_use_alt_stack to the monitor used by framework::init, or document that the switch
  does not cover initialisation.
• Correct the documented default in execution_monitor.hpp.
• BOOST_STRINGIZE( exp ) → BOOST_STRINGIZE( cond ).

Workaround for others hitting this

Boost only installs when no alternate stack is active:

BOOST_TEST_SYS_ASSERT( ::sigaltstack( 0, &sigstk ) != -1 );
if( sigstk.ss_flags & SS_DISABLE ) { …install SIGSTKSZ bytes… }

So installing an adequately sized stack during static initialisation — before main(), therefore
before framework::init() — makes Boost skip its own install. That needs no Boost recompilation,
so it works under BOOST_TEST_DYN_LINK where -DBOOST_TEST_DISABLE_ALT_STACK is inert, and it
keeps the alternate stack rather than giving up stack-overflow reporting.

Versions

Observed on 1.84.0 (Alpine Edge) and 1.86.0. All three items verified present on develop as of
2026-08-25.

[mborland]

Hi, since you're clearly using an LLM to generate this bug report, can you please use it to generate the patch set since it already has the context needed? Thanks.