Fix issues around glibc mis-calculating IE TLS sizes when spindle is running - #213
Conversation
This adds a large library as a dependency to the testsuite, which triggers the spindle issues around too-large TLS causing application exits.
GLIBC has a bug where IE TLS is allocated at audit library startup rather than application startup, which happens before application libraries are identified and loaded. This can leave the app with insufficient IE TLS. This works around the issue by manually doing the IE TLS calculation at exec() time, then using the GLIBC_TUNABLES environment variable to set the TLS size to a sufficient level for the application. The IE TLS calculation is done on a single node and communicated to clients via the spindle network. Co-Authored-By: Claude <noreply@anthropic.com>
| $CC $CFLAGS -o $BINDIR/libminaudit.so -fPIC -shared $SRCDIR/minaudit.c | ||
| fi | ||
| LD_AUDIT=$BINDIR/libminaudit.so $BINDIR/statictls >& /dev/null | ||
| RESULT=$? |
There was a problem hiding this comment.
This line is producing a syntax error when run by configure:
/home/spindleuser/Spindle/scripts/static_tls_test/run_test.sh: 16: Syntax error: Bad fd number
See, for example, the build of the serial image: https://github.com/llnl/Spindle/actions/runs/35164411510/job/105022155537#step:4:1050
I think the problem is the >& /dev/null in the previous line. The script specifies #!/bin/sh but that form of redirect is a bash extension.
|
I've run the E4S testsuite against this PR and found that there are five tests that are failing with this PR that pass without it: Fail:
Timeout:
Working on collecting more information on these failures |
|
The # $LONG_PATHS is "/foo:" repeated 1024 times
spindleuser@spindlenode:~$ ./hello
Hello world!
spindleuser@spindlenode:~$ LD_LIBRARY_PATH=$LONG_PATHS ./hello
Hello world!
spindleuser@spindlenode:~$ LD_LIBRARY_PATH=$LONG_PATHS spindle --calc-tls=false ./hello
Hello world!
spindleuser@spindlenode:~$ LD_LIBRARY_PATH=$LONG_PATHS spindle --calc-tls=true ./hello
spindleuser@spindlenode:~$ ls core.*
core.spindle_be.71Backtrace: #0 library_hash_lookup (path=0x626d7f759770 "./hello")
at /home/spindleuser/Spindle/src/server/auditserver/filemngt_calc_static_tls.c:712
#1 static_tls_size_for_dso (path=path@entry=0x626d7f759770 "./hello", interp=interp@entry=0x7fff9f04b360 "",
interp_str_size=interp_str_size@entry=4097, is_script_fd=is_script_fd@entry=0x7fff9f04a2bc,
tls_size_result=tls_size_result@entry=0x7fff9f04a2c0, tls_alignment_result=tls_alignment_result@entry=0x7fff9f04a2d8)
at /home/spindleuser/Spindle/src/server/auditserver/filemngt_calc_static_tls.c:304
#2 0x0000626d53e0cee1 in calc_static_tls_for_executable_nocache (info=info@entry=0x626d7f7352e0,
use_interp=use_interp@entry=0x0, tls_size_result_p=tls_size_result_p@entry=0x7fff9f04d3d8,
tls_alignment_result_p=tls_alignment_result_p@entry=0x7fff9f04d3e0)
at /home/spindleuser/Spindle/src/server/auditserver/filemngt_calc_static_tls.c:174
#3 0x0000626d53e0e17c in calc_static_tls_for_executable (info=info@entry=0x626d7f7352e0,
tls_size=tls_size@entry=0x7fff9f04d458, tls_alignment=tls_alignment@entry=0x7fff9f04d460)
at /home/spindleuser/Spindle/src/server/auditserver/filemngt_calc_static_tls.c:130
#4 0x0000626d53dff9ef in handle_calculate_total_static_tls_needed (tls_alignment=0x7fff9f04d450, tls_size=0x7fff9f04d448,
tlsinfo=0x626d7f7352e0) at /home/spindleuser/Spindle/src/server/auditserver/ldcs_audit_server_handlers.c:3200
#5 handle_tls_request (procdata=procdata@entry=0x626d53e35b00 <ldcs_process_data>, tlsinfo=0x626d7f7352e0,
from=from@entry=0x1) at /home/spindleuser/Spindle/src/server/auditserver/ldcs_audit_server_handlers.c:3167
#6 0x0000626d53e04b8f in handle_client_static_tls_req (msg=0x0, nc=<optimized out>,
procdata=0x626d53e35b00 <ldcs_process_data>)
at /home/spindleuser/Spindle/src/server/auditserver/ldcs_audit_server_handlers.c:3130
#7 handle_client_message (procdata=procdata@entry=0x626d53e35b00 <ldcs_process_data>, nc=nc@entry=0,
msg=msg@entry=0x7fff9f04d570) at /home/spindleuser/Spindle/src/server/auditserver/ldcs_audit_server_handlers.c:1917
#8 0x0000626d53df55f3 in _ldcs_client_CB (fd=<optimized out>, id=<optimized out>, data=0x626d53e35b00 <ldcs_process_data>)
at /home/spindleuser/Spindle/src/server/auditserver/ldcs_audit_server_client_cb.c:75
#9 0x0000626d53e0f5a4 in ldcs_listen () at /home/spindleuser/Spindle/src/server/comlib/ldcs_api_listen.c:176
#10 0x0000626d53de47db in ldcs_audit_server_run ()
at /home/spindleuser/Spindle/src/server/auditserver/ldcs_audit_server_process.c:261
#11 0x0000626d53ddb24a in spindleRunBE (port=21940, num_ports=<optimized out>, unique_id=121727925622538310,
security_type=<optimized out>, post_setup=0x0) at /home/spindleuser/Spindle/src/server/startup/spindle_be.cc:177
#12 0x0000626d53dd9542 in main (argc=4, argv=0x7fff9f04dab8)
at /home/spindleuser/Spindle/src/server/startup/spindle_be_main.cc:82 |
|
The Server backtrace: tau_core_flux-shell.1438_bt_full.txt |
|
Debug logs from timeout case: spindle_output.node-3.11337.txt |
|
This is a simple script that will usually reproduce the TAU fault without TAU or E4S on Tuolumne: #!/bin/bash
ORIGINAL_LDLP="$LD_LIBRARY_PATH"
for i in $(seq 200); do
# Interleave unique LD_LIBRARY_PATH values with non-unique
# values while launching subprocesses
for j in $(seq 10); do
LD_LIBRARY_PATH="/nonexistent/iter${i}:${ORIGINAL_LDLP}" /usr/bin/env true &
done
for j in $(seq 10); do
/usr/bin/env true &
done
wait
doneWith the system-installed Spindle on Tuolumne, on two nodes with 8 ranks total, this completes in around 30sec: [chaimov1@tuolumne1009:tls-test]$ time flux run -o spindle -N2 --tasks-per-node=4 ./tls-test.sh
real 0m27.061s
user 0m0.116s
sys 0m0.046sWith the PR#213 version of Spindle, this hangs and produces a coredump with a backtrace matching the TAU failure: [chaimov1@tuolumne1009:tls-test]$ flux run -o userrc=$HOME/spindle-pr213-inst/etc/spindle/spindle.rc -o spindle -N2 --tasks-per-node=4 ./tls-test.sh
^Cflux-job: one more ctrl-C within 2s to cancel or ctrl-Z to detach
^C16.855s: job.exception f8TNqeVEw type=cancel severity=0 interrupted by ctrl-C
flux-job: task(s) Terminated
[chaimov1@tuolumne1009:tls-test]$ ls
tls-test.sh tuolumne1010-flux-shell-269387.coreBacktrace: #0 0x00001555534d71e4 in __strcmp_evex () from /lib64/libc.so.6
#1 0x0000155542151a4a in strcmp_w_null (b=<optimized out>, a=<optimized out>) at ../../utils/static_tls.c:26
#2 static_tls_info_equal (info_a=0x1746790, info_b=info_b@entry=0x1747770) at ../../utils/static_tls.c:30
#3 0x000015554213f6d7 in handle_tls_process_result (procdata=procdata@entry=0x155542204be0 <ldcs_process_data>, tlsinfo=tlsinfo@entry=0x1747770,
tlskey=tlskey@entry=0xfdcaf0 "/usr/bin/env|/opt/cray/pe/lib64:/opt/cray/lib64:/opt/cray/pe/papi/7.2.0.2/lib64|NONE|/g/g19/chaimov1/tls-test|NONE|12|66|0|24|0", tls_size=338, tls_alignment=8) at ldcs_audit_server_handlers.c:3356
#4 0x000015554213fb6a in handle_tls_request (procdata=procdata@entry=0x155542204be0 <ldcs_process_data>, tlsinfo=0x1747770, from=from@entry=0x1) at ldcs_audit_server_handlers.c:3173
#5 0x00001555421449bf in handle_client_static_tls_req (msg=0x155542204be0 <ldcs_process_data>, nc=<optimized out>, procdata=0x155542204be0 <ldcs_process_data>) at ldcs_audit_server_handlers.c:3130
#6 handle_client_message (procdata=procdata@entry=0x155542204be0 <ldcs_process_data>, nc=nc@entry=26, msg=msg@entry=0x7fffffff7980) at ldcs_audit_server_handlers.c:1917
#7 0x0000155542136835 in _ldcs_client_CB (fd=<optimized out>, id=<optimized out>, data=0x155542204be0 <ldcs_process_data>) at ldcs_audit_server_client_cb.c:75
#8 0x0000155542153034 in ldcs_listen () at ldcs_api_listen.c:176
#9 0x0000155542148270 in ldcs_audit_server_run () at ldcs_audit_server_process.c:261
#10 0x000015554212311b in spindleRunBE (port=21940, num_ports=<optimized out>, unique_id=<optimized out>, security_type=security_type@entry=0, post_setup=post_setup@entry=0x0) at spindle_be.cc:177
#11 0x0000155542121c99 in run_spindle_backend (ctx=0x7fd9d0) at flux-spindle.c:287
#12 wait_for_shell_init (f=<optimized out>, arg=0x7fd9d0) at flux-spindle.c:354
#13 0x00001555550f5043 in ev_invoke_pending (loop=0x6c4960) at ev.c:3770
#14 0x00001555550f90e8 in ev_run (flags=0, loop=0x6c4960) at ev.c:4190
#15 ev_run (loop=0x6c4960, flags=flags@entry=0) at ev.c:4021
#16 0x00001555550b8f5f in flux_reactor_run (r=0x6c2ff0, flags=flags@entry=0) at reactor.c:91
#17 0x0000000000416e39 in main (argc=<optimized out>, argv=<optimized out>) at shell.c:2198 |
Adds dynamic sizing to message recieves from clients. The server will now maintain one buffer of size equal to the largest message received, dynamically growing it if needed for larger messages. Since we don't process multiple messages at once, this should remain safe and keep us from needing to manage message buffer cleanup. This feature was only added to ldcs_api_pipe.c under the assumption we need to cleanup/delete the biter and socket mechanisms.
Fixes an issue where STATIC_TLS_ALLOC_BUG was being treated as either defined or undefined, when it should be treated as 1 or 0. Also fix issue in the static_tls/runtest.sh script that detects whether we have the GLIBC bug present. On Ubuntu, the libraries that the test adds need to be specified after the source file when building. Also fix issue with CFLAGS being mis-passed to the static_tls/runtest.sh script.
|
@nchaimov, I've fixed the issues. The tau and xyce issues were also fixed by the same client size buffer fix that took care of the hpctoolkit, libpressio, and openfoam fix. I think they were just two manifestations of the same memory corruption. |
|
I've run the E4S testsuite again on the new changes. The bt_full.txt |
|
I could use some help reproducing this. If I log into frank should I be able to find the container this is running in somewhere? And if so, what command would run the tau test? |
|
Nevermind. Claude helped me spot a use-after free based on the stack trace that is likely the source of this bug. I'll fix that and try it before we do too much work reproducing the issue. |
|
As mentioned, I (aka Claude) found a use-after-free that would have caused this crash. Though I never actually reproduced the crash. Can you retest? |
|
I've run the TAU test 10 times without any failures. I'm now running the full E4S testsuite. |
|
I've confirmed that all five of the failures seen on this PR in the E4S testsuite are now fixed, and I've seen no new failures. |
This branch was successfully deployed
2 participants
GLIBC has a bug where any LD_AUDITer disables the normal calculation for an application's IE TLS (also called static TLS in this commit). Applications that need more than the default amount of IE TLS will fail with a message about insufficient TLS under spindle. Based on conversations with GLIBC folks, this may not be fixed for a while (or ever).
To work around this, this commit manually calculates the amount of IE TLS needed before each exec() operation, then if more than the default it uses the GLIBC_TUNABLES environment variable to set the IE TLS amount. Spindle does this scalably by:
This PR also adds several libraries (libbigtls.*) to the testsuite's set of library loads. These libraries have enough TLS to overrun the default IE TLS provided by glibc and require the above algorithm to kick in.