Attempting to StackTrace a mod_perl process fails

[codon]

When I try to StacTrace a mod_perl process, I get an error:

$ perl bin/perl-stacktrace 17962
[Thread debugging using libthread_db enabled]
[New Thread 0x7fbe88efb750 (LWP 17962)]
0x00007fbe87eb05b5 in waitpid () from /lib/libpthread.so.0
[Switching to Thread 0x7fbe88efb750 (LWP 17962)]
/tmp/_eeCXzm74I.gdb:588: Error in sourced command file:
Cannot access memory at address 0xffffffff83873d18

I get a similar message when I try to do the same thing executing the gdb script generated by the -m option.

[mgruberman]

Please re-run the program using the verbose flag or change $DEBUG in the gdb script to 1.

Additionally, since I'm likely to still be in the dark and if you're feeling able, add debugging statements to show the values of the stuff being dereferenced:

printf "foo_bar=%d\n", $some_var

[mgruberman]

Actually, I wrote some not-yet working commits to the tmp/20110727 branch. You could look at them and see that it's useful to run x/128xw $some_variable to dump memory. Also, having command tracing on for the gdb would be useful.

[codon]

I set $DEBUG to 1 in the gdb script.

# !!!!!!!   DO NOT EDIT THIS FILE   !!!!!!!
# This file is built by /site/perl/perl-5.10.1-1/lib/site_perl/5.10.1/x86_64-linux/App/Stacktrace.pm from its data.
# Any changes made here will be lost!
#
set $CONTEXT_cop = 8
set $CONTEXT_sizeof = 112
set $CONTEXT_type = 0
set $COP_gv = 56
set $COP_line = 36
set $CXTYPEMASK = 255
set $CXt_EVAL = 2
set $CXt_FORMAT = 6
set $CXt_SUB = 1
set $GP_sv = 0
set $GV_gp = 16
set $STACKINFO_cxix = 32
set $STACKINFO_cxstack = 8
set $STACKINFO_prev = 16
set $SV_any = 0
set $SV_pv = 16
set $SV_uv = 24
# Provides the commands:
#
#   perl_backtrace_5_10_x
#   perl_backtrace_5_12_x
#   perl_backtrace_5_14_x
#
#   perl_backtrace_5_14_x_x86_64
#   perl_backtrace_5_14_x_thread_x86_64
#   perl_backtrace_5_12_x_thread_x86_64
#   perl_backtrace_5_10_1_thread_x86_64
#
#   perl_backtrace_5_14_x_thread_i686
#
# Example usage:
#
#     gdb -p 7107
#     (gdb) source gdbinit.txt
#     ... set lots of constants
#     (gdb) perl_backtrace_5_14_x
#     (gdb) detach
#     (gdb) quit

#perl_backtrace_5_14_x -> perl_backtrace_5_12_threads
#perl_backtrace_5_14_x -> perl_backtrace_nothreads
#perl_backtrace_5_12_x -> perl_backtrace_5_12_threads
#perl_backtrace_5_12_x -> perl_backtrace_nothreads
#perl_backtrace_5_12_threads -> perl_backtrace_a_thread
#perl_backtrace_5_12_threads -> perl_backtrace_an_interp
#perl_backtrace_5_10_x -> perl_backtrace_5_10_threads
#perl_backtrace_5_10_x -> perl_backtrace_nothreads
#perl_backtrace_5_10_threads -> perl_backtrace_a_thread
#perl_backtrace_5_10_threads -> perl_backtrace_an_interp
#perl_backtrace_5_8_9 -> perl_backtrace_5_8_9_threads
#perl_backtrace_5_8_9 -> perl_backtrace_5_8_nothreads
#perl_backtrace_5_8_9_threads -> perl_backtrace_5_8_9_a_thread
#perl_backtrace_5_8_9_a_thread -> perl_backtrace_5_8_9_an_interp
#perl_backtrace_5_8_9_an_interp
#perl_backtrace_5_8_x -> perl_backtrace_5_8_threads
#perl_backtrace_5_8_x -> perl_backtrace_5_8_nothreads
#perl_backtrace_5_8_threads -> perl_backtrace_5_8_a_thread
#perl_backtrace_5_8_a_thread -> perl_backtrace_5_8_an_interp
#perl_backtrace_5_8_an_interp
#perl_backtrace_5_8_nothreads
#perl_backtrace_an_interp
#perl_backtrace_a_thread
#perl_backtrace_nothreads

set $DEBUG = 1
set $PERL_ITHR_JOINABLE           =  0
set $PERL_ITHR_DETACHED           =  1
set $PERL_ITHR_JOINED             =  2
set $PERL_ITHR_FINISHED           =  4
set $PERL_ITHR_THREAD_EXIT_ONLY   =  8
set $PERL_ITHR_NONVIABLE          = 16
set $PERL_ITHR_DIED               = 32

set $PERL_ITHR_UNCALLABLE  = $PERL_ITHR_DETACHED | $PERL_ITHR_JOINED

define perl_backtrace_an_interp
    set $stackinfo = (int) *((int*) ($INTERPRETER_curstackinfo + (int) $interpreter))
    while $stackinfo != 0
        set $cxstack = (int) *((int*) ($STACKINFO_cxstack + (int) $stackinfo))
        set $cxix = (int) *((int*) ($STACKINFO_cxix + (int) $stackinfo))
        set $i = 0
        while $i <= $cxix
            set $context = (int) (($CONTEXT_sizeof * $i) + (int) $cxstack)
            set $type = (int) (((int) *((int*) ($CONTEXT_type + (int) $context))) & $CXTYPEMASK)
            if $type == $CXt_SUB || $type == $CXt_EVAL || $type == $CXt_FORMAT
                set $file = 0
                set $cop = (int) *((int*) ($CONTEXT_cop + (int) $context))
                set $file = (char*) *(int*) ($COP_file + (int) $cop)
                if $file == 0
                    set $file = "undef"
                end
                set $line = (int) *((int*) ($COP_line + (int) $cop))
                printf "%s:%d\n", $file, $line
            else
                if $DEBUG
                    printf "%d\t... # (context*){cx_type=%d}\n", $i, $type
                end
            end
            set $i = $i + 1
        end
        set $stackinfo = (int) *((int*) ($STACKINFO_prev + (int) $stackinfo))
    end
end
define perl_backtrace_a_thread
    set $tid = (int) *((int*) ($THREAD_tid + (int) $thread))
    set $statei = (int) *((int*) ($THREAD_state + (int) $thread))
    if $statei == $PERL_ITHR_DETACHED
        set $state = "detached"
    else
        if $statei == $PERL_ITHR_JOINED
            set $state = "joined"
        else
            if $statei = $PERL_ITHR_FINISHED
                set $state = "finished"
            else
                if $statei == $PERL_ITHR_THREAD_EXIT_ONLY
                    set $state = "exit()"
                else
                    if $statei == $PERL_ITHR_NONVIABLE
                        set $state = "thread creation failed"
                    else
                        if $statei == $PERL_ITHR_DIED
                            set $state = "died"
                        else
                            if $statei == $PERL_ITHR_UNCALLABLE
                                set $state = "uncallable"
                            else
                                set $state = "???"
                            end
                        end
                    end
                end
            end
        end
    end
    printf "thread %d %s:\n", $tid, $state
    set $interpreter = (int) *((int*) ($THREAD_interpreter + (int) $thread))
    perl_backtrace_an_interp
end
define perl_backtrace_nothreads
    set $stackinfo = (int) PL_curstackinfo
    while $stackinfo
        set $cxstack = (int) *((int*) ($STACKINFO_cxstack + (int) $stackinfo))
        set $cxix = (int) *((int*) ($STACKINFO_cxix + (int) $stackinfo))
        set $i = 0
        while $i <= $cxix
            set $context = (int) (($CONTEXT_sizeof * $i) + (int) $cxstack)
            set $type = (int) (((int) *((int*) ($CONTEXT_type + (int) $context))) & $CXTYPEMASK)
            if $type == $CXt_SUB || $type == $CXt_EVAL || $type == $CXt_FORMAT
                set $file = 0
                set $cop = (int) *((int*) ($CONTEXT_cop + (int) $context))
                set $gv = (int) *((int*) ($COP_gv + (int) $cop))
                if $gv
                    set $sv = (int) *((int*) ($GP_sv + *((int*) ($GV_gp + (int) $gv))))
                    if $sv
                        set $file = (char*) *(int*) ($SV_pv + (int) $sv)
                    end
                end
                if ! $file
                    set $file = "undef"
                end
                set $line = (int) *((int*) ($COP_line + (int) $cop))
                printf "%s:%d\n", $file, $line
            end
            set $i = $i + 1
        end
        set $stackinfo = (int) *((int*) ($STACKINFO_prev + (int) $stackinfo))
    end
end

define perl_backtrace_5_8_an_interp
    set $stackinfo = (int) *((int*) ($INTERPRETER_curstackinfo + (int) $interpreter))
    while $stackinfo
        set $cxstack = (int) *((int*) ($STACKINFO_cxstack + (int) $stackinfo))
        set $cxix = (int) *((int*) ($STACKINFO_cxix + (int) $stackinfo))
        set $i = 0
        while $i <= $cxix
            set $context = (int) (($CONTEXT_sizeof * $i) + (int) $cxstack)
            set $type = (int) (((int) *($CONTEXT_type + (int) $context)) & $CXTYPEMASK)
            if $type == $CXt_SUB || $type == $CXt_EVAL || $type == $CXt_FORMAT
                set $file = 0
                set $cop = (int) *((int*) ($CONTEXT_cop + (int) $context))
                set $file = (char*) *(int*) ($COP_file + (int) $cop)
                if ! $file
                    set $file = "undef"
                end
                set $line = (int) *((int*) ($COP_line + (int) $cop))
                printf "%s:%d\n", $file, $line
            else
                if $DEBUG
                    printf "%d\t... # (context*){cx_type=%d}\n", $i, $type
                end
            end
            set $i = $i + 1
        end
        set $stackinfo = (int) *((int*) ($STACKINFO_prev + (int) $stackinfo))
    end
end
define perl_backtrace_5_8_nothreads
    set $stackinfo = (int) PL_curstackinfo
    while $stackinfo
        set $cxstack = (int) *((int*) ($STACKINFO_cxstack + (int) $stackinfo))
        set $cxix = (int) *((int*) ($STACKINFO_cxix + (int) $stackinfo))
        set $i = 0
        while $i <= $cxix
            set $context = (int) (($CONTEXT_sizeof * $i) + (int) $cxstack)
            set $type = (int) (((int) *($CONTEXT_type + (int) $context)) & $CXTYPEMASK)
            if $type == $CXt_SUB || $type == $CXt_EVAL || $type == $CXt_FORMAT
                set $cop = (int) *((int*) ($CONTEXT_cop + (int) $context))
                set $gv = (int) *((int*) ($COP_gv + (int) $cop))
                if $gv
                    set $sv = (int) *((int*) ($GP_sv + *((int*) ($GV_gp + *((int*) ($SV_any + (int) $gv))))))
                    if $sv
                        set $file =  (char*) ($XPV_pv + (int) *((int*) ($SV_any + (int) $sv)))
                    else
                        set $file = "undef"
                    end
                else
                    set $file = "undef"
                end
                set $line = (int) *((int*) ($COP_line + (int) $cop))
                printf "%s:%d\n", $file, $line
            end
            set $i = $i + 1
        end
        set $stackinfo = (int) *((int*) ($STACKINFO_prev + (int) $stackinfo))
    end
end
define perl_backtrace_5_8_a_thread
    if $thread
        set $tid = (int) *((int*) ($THREAD_tid + (int) $thread))
        set $statei = (int) *((int*) ($THREAD_state + (int) $thread))
        if $statei == $PERL_ITHR_DETACHED
            set $state = "detached"
        else
            if $statei == $PERL_ITHR_JOINED
                set $state = "joined"
            else
                if $statei = $PERL_ITHR_FINISHED
                    set $state = "finished"
                else
                    if $statei == $PERL_ITHR_THREAD_EXIT_ONLY
                        set $state = "exit()"
                    else
                        if $statei == $PERL_ITHR_NONVIABLE
                            set $state = "thread creation failed"
                        else
                            if $statei == $PERL_ITHR_DIED
                                set $state = "died"
                            else
                                if $statei == $PERL_ITHR_UNCALLABLE
                                    set $state = "uncallable"
                                else
                                    set $state = "???"
                                end
                            end
                        end
                    end
                end
            end
        end
        printf "thread %d %s:\n", $tid, $state
        set $interpreter = (int) *((int*) ($THREAD_interpreter + (int) $thread))
    end
    perl_backtrace_5_8_an_interp
end
define perl_backtrace_5_8_threads
    set $main_thread = (int) threads
    set $thread = $main_thread
    perl_backtrace_5_8_a_thread
    if $thread
        set $thread = (int) *((int*) ($THREAD_next + (int) $thread))
        while $thread && $thread != $main_thread
            perl_backtrace_5_8_a_thread
            set $thread = (int) *((int*) ($THREAD_next + (int) $thread))
        end
    end
end
define perl_backtrace_5_8_x
    set $interpreter = (int) Perl_get_context()
    if $interpreter
        perl_backtrace_5_8_threads
    else
        perl_backtrace_5_8_nothreads
    end
end

define perl_backtrace_5_8_9_an_interp
    set $stackinfo = (int) *((int*) ($INTERPRETER_curstackinfo + (int) $interpreter))
    while $stackinfo
        set $cxstack = (int) *((int*) ($STACKINFO_cxstack + (int) $stackinfo))
        set $cxix = (int) *((int*) ($STACKINFO_cxix + (int) $stackinfo))
        set $i = 0
        while $i <= $cxix
            set $context = (int) (($CONTEXT_sizeof * $i) + (int) $cxstack)
            set $type = (int) (((int) *((int*) ($CONTEXT_type + (int) $context))) & $CXTYPEMASK)
            if $type == $CXt_SUB || $type == $CXt_EVAL || $type == $CXt_FORMAT
                set $file = 0
                set $cop = (int) *((int*) ($CONTEXT_cop + (int) $context))
                set $file = (char*) *(int*) ($COP_file + (int) $cop)
                if ! $file
                    set $file = "undef"
                end
                set $line = (int) *((int*) ($COP_line + (int) $cop))
                printf "%s:%d\n", $file, $line
            else
                if $DEBUG
                    printf "%d\t... # (context*){cx_type=%d}\n", $i, $type
                end
            end
            set $i = $i + 1
        end
        set $stackinfo = (int) *((int*) ($STACKINFO_prev + (int) $stackinfo))
    end
end
define perl_backtrace_5_8_9_a_thread
    set $tid = (int) *((int*) ($THREAD_tid + (int) $thread))
    set $statei = (int) *((int*) ($THREAD_state + (int) $thread))
    if $statei == $PERL_ITHR_DETACHED
        set $state = "detached"
    else
        if $statei == $PERL_ITHR_JOINED
            set $state = "joined"
        else
            if $statei = $PERL_ITHR_FINISHED
                set $state = "finished"
            else
                if $statei == $PERL_ITHR_THREAD_EXIT_ONLY
                    set $state = "exit()"
                else
                    if $statei == $PERL_ITHR_NONVIABLE
                        set $state = "thread creation failed"
                    else
                        if $statei == $PERL_ITHR_DIED
                            set $state = "died"
                        else
                            if $statei == $PERL_ITHR_UNCALLABLE
                                set $state = "uncallable"
                            else
                                set $state = "???"
                            end
                        end
                    end
                end
            end
        end
    end
    printf "thread %d %s:\n", $tid, $state
    set $interpreter = (int) *((int*) ($THREAD_interpreter + (int) $thread))
    perl_backtrace_5_8_9_an_interp
end
define perl_backtrace_5_8_9_threads
    set $main_thread = (int) threads
    set $thread = $main_thread
    perl_backtrace_5_8_9_a_thread
    if $thread
        set $thread = (int) *((int*) ($THREAD_next + (int) $thread))
        while $thread && $thread != $main_thread
            perl_backtrace_5_8_9_a_thread
            set $thread = (int) *((int*) ($THREAD_next + (int) $thread))
        end
    end
end
define perl_backtrace_5_8_9
    set $interpreter = (int) Perl_get_context()
    if $interpreter
        perl_backtrace_5_8_9_threads
    else
        perl_backtrace_5_8_nothreads
    end
end
define perl_backtrace_5_10_threads
    set $modglobal = (int) *((int*) ($INTERPRETER_modglobal + (int) $interpreter))
    set $my_pool_svp = (int) Perl_hv_fetch((int) $interpreter, (int) $modglobal, $POOL_KEY, $POOL_KEY_LEN, 0)
    if $my_pool_svp
        set $my_pool_sv = (int) *((int*) (int) $my_pool_svp)
        set $my_pool_svval = (int) *((int*) ($SV_any + (int) $my_pool_sv))
        set $my_poolp = (int) *((int*) ($SV_iv + (int) $my_pool_svval))
        set $main_thread = $POOLP_main_thread + (int) $my_poolp
        set $thread = $main_thread
        perl_backtrace_a_thread
        set $thread = (int) *((int*) ($THREAD_next + (int) $main_thread))
        while $thread != $main_thread
            perl_backtrace_a_thread
            set $thread = (int) *((int*) ($THREAD_next + (int) $thread))
        end
    else
        perl_backtrace_an_interp
    end
end
define perl_backtrace_5_10_x
    set $interpreter = (int) Perl_get_context()
    if $interpreter
        perl_backtrace_5_10_threads
    else
        perl_backtrace_nothreads
    end
end
define perl_backtrace_5_12_threads
    set $modglobal = (int) *((int*) ($INTERPRETER_modglobal + (int) $interpreter))
    set $my_pool_svp = (int) Perl_hv_fetch((int) $interpreter, (int) $modglobal, $POOL_KEY, $POOL_KEY_LEN, 0)
    if $my_pool_svp
        set $my_pool_sv = (int) *((int*) (int) $my_pool_svp)
        set $my_pool_svval = (int) *((int*) ($SV_any + (int) $my_pool_sv))
        set $my_poolp = (int) *((int*) ($SV_uv + (int) $my_pool_svval))
        set $main_thread = $POOLP_main_thread + (int) $my_poolp
        set $thread = $main_thread
        perl_backtrace_a_thread
        set $thread = (int) *((int*) ($THREAD_next + (int) $main_thread))
        while $thread != $main_thread
            perl_backtrace_a_thread
            set $thread = (int) *((int*) ($THREAD_next + (int) $thread))
        end
    else
        perl_backtrace_an_interp
    end
end
define perl_backtrace_5_12_x
    set $interpreter = (int) Perl_get_context()
    if $interpreter
        perl_backtrace_5_12_threads
    else
        perl_backtrace_nothreads
    end
end
define perl_backtrace_5_14_x
    perl_backtrace_5_12_x
end
define perl_backtrace_5_14_x_x86_64
    # 5.14.0-linux-x86_64-linux: 3
    # 5.13.11-linux-x86_64-linux: 3
    # 5.13.3-linux-x86_64-linux: 3
    # 5.13.2-linux-x86_64-linux: 3
    # 5.13.4-linux-x86_64-linux: 3
    # 5.14.0-dragonfly-x86_64-dragonfly: 2
    # 5.13.6-linux-x86_64-linux: 2
    # 5.13.5-linux-x86_64-linux: 2
    # 5.13.10-linux-x86_64-linux: 2
    # 5.13.8-linux-x86_64-linux: 2
    # 5.13.9-linux-x86_64-linux: 2
    # 5.13.7-linux-x86_64-linux: 2
    # 5.14.1-linux-x86_64-linux: 1
    set $CONTEXT_cop = 8
    set $CONTEXT_sizeof = 80
    set $CONTEXT_type = 0
    set $COP_gv = 48
    set $COP_line = 36
    set $CXTYPEMASK = 15
    set $CXt_EVAL = 10
    set $CXt_FORMAT = 9
    set $CXt_SUB = 8
    set $GP_sv = 0
    set $GV_gp = 16
    set $STACKINFO_cxix = 32
    set $STACKINFO_cxstack = 8
    set $STACKINFO_prev = 16
    set $SV_any = 0
    set $SV_iv = 32
    set $SV_pv = 16
    perl_backtrace_5_14_x
end
define perl_backtrace_5_14_x_thread_x86_64
    # 5.14.0-linux-x86_64-linux-thread-multi: 8
    # 5.13.11-linux-x86_64-linux-thread-multi: 4
    # 5.15.0-linux-x86_64-linux-thread-multi: 4
    # 5.13.8-linux-x86_64-linux-thread-multi: 3
    # 5.13.10-linux-x86_64-linux-thread-multi: 3
    # 5.14.1 RC1-linux-x86_64-linux-thread-multi: 2
    # 5.13.7-linux-x86_64-linux-thread-multi: 2
    # 5.13.9-linux-x86_64-linux-thread-multi: 2
    # 5.14.0-linux-x86_64-linux-thread-multi-ld: 1
    set $CONTEXT_cop = 8
    set $CONTEXT_sizeof = 80
    set $CONTEXT_type = 0
    set $COP_file = 48
    set $COP_line = 36
    set $CXTYPEMASK = 15
    set $CXt_EVAL = 10
    set $CXt_FORMAT = 9
    set $CXt_SUB = 8
    set $GP_sv = 0
    set $GV_gp = 16
    set $INTERPRETER_curstackinfo = 608
    set $INTERPRETER_modglobal = 1768
    set $POOLP_main_thread = 0
    set $STACKINFO_cxix = 32
    set $STACKINFO_cxstack = 8
    set $STACKINFO_prev = 16
    set $SV_any = 0
    set $SV_iv = 32
    set $SV_pv = 16
    set $THREAD_interpreter = 16
    set $THREAD_next = 0
    set $THREAD_state = 76
    set $THREAD_tid = 24
    perl_backtrace_5_14_x
end
define perl_backtrace_5_12_x_thread_x86_64
    # 5.12.3-linux-x86_64-linux-thread-multi: 4
    # 5.13.0-linux-x86_64-linux-thread-multi: 2
    # 5.12.4 RC1-linux-x86_64-linux-thread-multi: 2
    set $CONTEXT_cop = 8
    set $CONTEXT_sizeof = 80
    set $CONTEXT_type = 0
    set $COP_file = 48
    set $COP_line = 36
    set $CXTYPEMASK = 15
    set $CXt_EVAL = 10
    set $CXt_FORMAT = 9
    set $CXt_SUB = 8
    set $GP_sv = 0
    set $GV_gp = 16
    set $INTERPRETER_curstackinfo = 600
    set $INTERPRETER_modglobal = 1736
    set $POOLP_main_thread = 0
    set $STACKINFO_cxix = 32
    set $STACKINFO_cxstack = 8
    set $STACKINFO_prev = 16
    set $SV_any = 0
    set $SV_iv = 24
    set $SV_pv = 16
    set $THREAD_interpreter = 16
    set $THREAD_next = 0
    set $THREAD_state = 76
    set $THREAD_tid = 24
    perl_backtrace_5_12_x
end
define perl_backtrace_5_10_1_thread_x86_64
    # 5.10.1-linux-x86_64-linux-thread-multi: 4
    set $CONTEXT_cop = 8
    set $CONTEXT_sizeof = 112
    set $CONTEXT_type = 0
    set $COP_file = 56
    set $COP_line = 36
    set $CXTYPEMASK = 255
    set $CXt_EVAL = 2
    set $CXt_FORMAT = 6
    set $CXt_SUB = 1
    set $GP_sv = 0
    set $GV_gp = 16
    set $INTERPRETER_curstackinfo = 592
    set $INTERPRETER_modglobal = 1720
    set $POOLP_main_thread = 0
    set $STACKINFO_cxix = 32
    set $STACKINFO_cxstack = 8
    set $STACKINFO_prev = 16
    set $SV_any = 0
    set $SV_pv = 16
    set $SV_uv = 24
    set $THREAD_interpreter = 16
    set $THREAD_next = 0
    set $THREAD_state = 76
    set $THREAD_tid = 24
    perl_backtrace_5_10_x
end
define perl_backtrace_5_14_x_i686
    # 5.14.1 RC1-linux-i686-linux-thread-multi: 1
    set $CONTEXT_cop = 8
    set $CONTEXT_sizeof = 48
    set $CONTEXT_type = 0
    set $COP_file = 28
    set $COP_line = 20
    set $CXTYPEMASK = 15
    set $CXt_EVAL = 10
    set $CXt_FORMAT = 9
    set $CXt_SUB = 8
    set $GP_sv = 0
    set $GV_gp = 12
    set $INTERPRETER_curstackinfo = 368
    set $INTERPRETER_modglobal = 1080
    set $POOLP_main_thread = 0
    set $STACKINFO_cxix = 16
    set $STACKINFO_cxstack = 4
    set $STACKINFO_prev = 8
    set $SV_any = 0
    set $SV_iv = 16
    set $SV_pv = 12
    set $THREAD_interpreter = 8
    set $THREAD_next = 0
    set $THREAD_state = 44
    set $THREAD_tid = 12
    perl_backtrace_5_14_x
end
perl_backtrace_5_10_x
detach
quit

I can't tell that it made any difference in output. I'm including the entire output (including set-up) here. Initially, I have one Apache child tied up handling a "request". I issue an "httpd graceful" (wrapped in some in-house overhead). Then I have several defunct Apache children and the parent waiting on the busy child to finish. Technically, it's waiting on a cronolog process to finish, but the child evidently has a handle open on that pipe, keeping cronolog alive.

iheffner@hardtop:~/git/next/call_processor$ pstree -clap 17962
httpd,17962 -f /home/iheffner/git/next/call_processor/conf/httpd.conf
  ├─cronolog,9817 /home/iheffner/git/next/call_processor/logs/error.log.%Y-%m-%d
  ├─cronolog,9829 /home/iheffner/git/next/call_processor/logs/nx-call-processor.log.%Y-%m-%d
  ├─httpd,9830 -f /home/iheffner/git/next/call_processor/conf/httpd.conf
  ├─httpd,9831 -f /home/iheffner/git/next/call_processor/conf/httpd.conf
  ├─httpd,9834 -f /home/iheffner/git/next/call_processor/conf/httpd.conf
  ├─httpd,9835 -f /home/iheffner/git/next/call_processor/conf/httpd.conf
  │   └─cronolog,11381 /home/iheffner/git/next/call_processor/logs/cdr.log.%Y-%m-%d
  └─httpd,9836 -f /home/iheffner/git/next/call_processor/conf/httpd.conf
iheffner@hardtop:~/git/next/call_processor$ bin/httpd-graceful graceful
iheffner@hardtop:~/git/next/call_processor$ pstree -clap 17962
httpd,17962 -f /home/iheffner/git/next/call_processor/conf/httpd.conf
  ├─cronolog,9817 /home/iheffner/git/next/call_processor/logs/error.log.%Y-%m-%d
  ├─cronolog,9829 /home/iheffner/git/next/call_processor/logs/nx-call-processor.log.%Y-%m-%d
  ├─(httpd,9830)
  ├─(httpd,9831)
  ├─(httpd,9834)
  ├─httpd,9835 -f /home/iheffner/git/next/call_processor/conf/httpd.conf
  │   └─cronolog,11381 /home/iheffner/git/next/call_processor/logs/cdr.log.%Y-%m-%d
  └─(httpd,9836)
iheffner@hardtop:~/git/next/call_processor$ gdb -x ~/perl-stacktrace.gdb -p 17962
GNU gdb 6.8-debian
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Attaching to process 17962
Reading symbols from /site/httpd/httpd-2.2.14-1/bin/httpd...done.
Reading symbols from /lib/libm.so.6...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /usr/lib/libaprutil-1.so.0...done.
Loaded symbols for /usr/lib/libaprutil-1.so.0
Reading symbols from /usr/lib/libexpat.so.1...done.
Loaded symbols for /usr/lib/libexpat.so.1
Reading symbols from /usr/lib/libapr-1.so.0...done.
Loaded symbols for /usr/lib/libapr-1.so.0
Reading symbols from /lib/librt.so.1...done.
Loaded symbols for /lib/librt.so.1
Reading symbols from /lib/libcrypt.so.1...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /lib/libpthread.so.0...done.
[Thread debugging using libthread_db enabled]
[New Thread 0x7fbe88efb750 (LWP 17962)]
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libdl.so.2...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /usr/lib/libldap_r-2.4.so.2...done.
Loaded symbols for /usr/lib/libldap_r-2.4.so.2
Reading symbols from /usr/lib/liblber-2.4.so.2...done.
Loaded symbols for /usr/lib/liblber-2.4.so.2
Reading symbols from /usr/lib/libdb-4.6.so...done.
Loaded symbols for /usr/lib/libdb-4.6.so
Reading symbols from /usr/lib/libpq.so.5...done.
Loaded symbols for /usr/lib/libpq.so.5
Reading symbols from /usr/lib/libmysqlclient_r.so.15...done.
Loaded symbols for /usr/lib/libmysqlclient_r.so.15
Reading symbols from /usr/lib/libsqlite3.so.0...done.
Loaded symbols for /usr/lib/libsqlite3.so.0
Reading symbols from /lib/libuuid.so.1...done.
Loaded symbols for /lib/libuuid.so.1
Reading symbols from /lib/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib/libresolv.so.2...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /usr/lib/libsasl2.so.2...done.
Loaded symbols for /usr/lib/libsasl2.so.2
Reading symbols from /usr/lib/libgnutls.so.26...done.
Loaded symbols for /usr/lib/libgnutls.so.26
Reading symbols from /usr/lib/libssl.so.0.9.8...done.
Loaded symbols for /usr/lib/libssl.so.0.9.8
Reading symbols from /usr/lib/libcrypto.so.0.9.8...done.
Loaded symbols for /usr/lib/libcrypto.so.0.9.8
Reading symbols from /usr/lib/libkrb5.so.3...done.
Loaded symbols for /usr/lib/libkrb5.so.3
Reading symbols from /lib/libcom_err.so.2...done.
Loaded symbols for /lib/libcom_err.so.2
Reading symbols from /usr/lib/libgssapi_krb5.so.2...done.
Loaded symbols for /usr/lib/libgssapi_krb5.so.2
Reading symbols from /lib/libnsl.so.1...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /usr/lib/libz.so.1...done.
Loaded symbols for /usr/lib/libz.so.1
Reading symbols from /usr/lib/libtasn1.so.3...done.
Loaded symbols for /usr/lib/libtasn1.so.3
Reading symbols from /usr/lib/libgpg-error.so.0...done.
Loaded symbols for /usr/lib/libgpg-error.so.0
Reading symbols from /usr/lib/libgcrypt.so.11...done.
Loaded symbols for /usr/lib/libgcrypt.so.11
Reading symbols from /usr/lib/libk5crypto.so.3...done.
Loaded symbols for /usr/lib/libk5crypto.so.3
Reading symbols from /usr/lib/libkrb5support.so.0...done.
Loaded symbols for /usr/lib/libkrb5support.so.0
Reading symbols from /lib/libkeyutils.so.1...done.
Loaded symbols for /lib/libkeyutils.so.1
Reading symbols from /lib/libnss_files.so.2...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /site/httpd/httpd-2.2.14-1/modules/mod_authz_host.so...done.
Loaded symbols for /site/httpd/httpd-2.2.14-1/modules/mod_authz_host.so
Reading symbols from /site/httpd/httpd-2.2.14-1/modules/mod_log_config.so...done.
Loaded symbols for /site/httpd/httpd-2.2.14-1/modules/mod_log_config.so
Reading symbols from /site/httpd/httpd-2.2.14-1/modules/mod_env.so...done.
Loaded symbols for /site/httpd/httpd-2.2.14-1/modules/mod_env.so
Reading symbols from /site/httpd/httpd-2.2.14-1/modules/mod_mime.so...done.
Loaded symbols for /site/httpd/httpd-2.2.14-1/modules/mod_mime.so
Reading symbols from /site/httpd/httpd-2.2.14-1/modules/mod_status.so...done.
Loaded symbols for /site/httpd/httpd-2.2.14-1/modules/mod_status.so
Reading symbols from /site/httpd/httpd-2.2.14-1/modules/mod_dir.so...done.
Loaded symbols for /site/httpd/httpd-2.2.14-1/modules/mod_dir.so
Reading symbols from /site/httpd/httpd-2.2.14-1/modules/mod_perl.so...done.
Loaded symbols for /site/httpd/httpd-2.2.14-1/modules/mod_perl.so
Reading symbols from /lib/libutil.so.1...done.
Loaded symbols for /lib/libutil.so.1
Reading symbols from /site/httpd/httpd-2.2.13-1/lib/perl/lib/site_perl/5.10.1/x86_64-linux/x86_64-linux/auto/Apache2/Connection/Connection.so...done.
Loaded symbols for /site/httpd/httpd-2.2.13-1/lib/perl/lib/site_perl/5.10.1/x86_64-linux/x86_64-linux/auto/Apache2/Connection/Connection.so
Reading symbols from /site/httpd/httpd-2.2.13-1/lib/perl/lib/site_perl/5.10.1/x86_64-linux/x86_64-linux/auto/Apache2/Const/Const.so...done.
Loaded symbols for /site/httpd/httpd-2.2.13-1/lib/perl/lib/site_perl/5.10.1/x86_64-linux/x86_64-linux/auto/Apache2/Const/Const.so
Reading symbols from /site/httpd/httpd-2.2.13-1/lib/perl/lib/site_perl/5.10.1/x86_64-linux/x86_64-linux/auto/APR/Socket/Socket.so...done.
Loaded symbols for /site/httpd/httpd-2.2.13-1/lib/perl/lib/site_perl/5.10.1/x86_64-linux/x86_64-linux/auto/APR/Socket/Socket.so
Reading symbols from /site/httpd/httpd-2.2.13-1/lib/perl/lib/site_perl/5.10.1/x86_64-linux/x86_64-linux/auto/APR/Const/Const.so...done.
Loaded symbols for /site/httpd/httpd-2.2.13-1/lib/perl/lib/site_perl/5.10.1/x86_64-linux/x86_64-linux/auto/APR/Const/Const.so
Reading symbols from /site/perl/perl-5.10.1-1/lib/5.10.1/x86_64-linux/auto/Time/HiRes/HiRes.so...done.
Loaded symbols for /site/perl/perl-5.10.1-1/lib/5.10.1/x86_64-linux/auto/Time/HiRes/HiRes.so
Reading symbols from /site/perl/perl-5.10.1-1/lib/5.10.1/x86_64-linux/auto/Sys/Hostname/Hostname.so...done.
Loaded symbols for /site/perl/perl-5.10.1-1/lib/5.10.1/x86_64-linux/auto/Sys/Hostname/Hostname.so
Reading symbols from /site/perl/perl-5.10.1-1/lib/5.10.1/x86_64-linux/auto/Data/Dumper/Dumper.so...done.
Loaded symbols for /site/perl/perl-5.10.1-1/lib/5.10.1/x86_64-linux/auto/Data/Dumper/Dumper.so
Reading symbols from /site/perl/perl-5.10.1-1/lib/5.10.1/x86_64-linux/auto/MIME/Base64/Base64.so...done.
Loaded symbols for /site/perl/perl-5.10.1-1/lib/5.10.1/x86_64-linux/auto/MIME/Base64/Base64.so
Reading symbols from /site/perl/perl-5.10.1-1/lib/5.10.1/x86_64-linux/auto/Digest/SHA/SHA.so...done.
Loaded symbols for /site/perl/perl-5.10.1-1/lib/5.10.1/x86_64-linux/auto/Digest/SHA/SHA.so
Reading symbols from /site/perl/perl-5.10.1-1/lib/5.10.1/x86_64-linux/auto/Cwd/Cwd.so...done.
Loaded symbols for /site/perl/perl-5.10.1-1/lib/5.10.1/x86_64-linux/auto/Cwd/Cwd.so
Reading symbols from /site/perl/perl-5.10.1-1/lib/5.10.1/x86_64-linux/auto/Fcntl/Fcntl.so...done.
Loaded symbols for /site/perl/perl-5.10.1-1/lib/5.10.1/x86_64-linux/auto/Fcntl/Fcntl.so
Reading symbols from /site/perl/perl-5.10.1-1/lib/5.10.1/x86_64-linux/auto/IO/IO.so...done.
Loaded symbols for /site/perl/perl-5.10.1-1/lib/5.10.1/x86_64-linux/auto/IO/IO.so
Reading symbols from /site/perl/perl-5.10.1-1/lib/5.10.1/x86_64-linux/auto/Socket/Socket.so...done.
Loaded symbols for /site/perl/perl-5.10.1-1/lib/5.10.1/x86_64-linux/auto/Socket/Socket.so
Reading symbols from /site/perl/perl-5.10.1-1/lib/5.10.1/x86_64-linux/auto/Storable/Storable.so...done.
Loaded symbols for /site/perl/perl-5.10.1-1/lib/5.10.1/x86_64-linux/auto/Storable/Storable.so
Reading symbols from /site/perl/perl-5.10.1-1/lib/site_perl/5.10.1/x86_64-linux/auto/JSON/XS/XS.so...done.
Loaded symbols for /site/perl/perl-5.10.1-1/lib/site_perl/5.10.1/x86_64-linux/auto/JSON/XS/XS.so
Reading symbols from /site/perl/perl-5.10.1-1/lib/5.10.1/x86_64-linux/auto/B/B.so...done.
Loaded symbols for /site/perl/perl-5.10.1-1/lib/5.10.1/x86_64-linux/auto/B/B.so
Reading symbols from /site/perl/perl-5.10.1-1/lib/5.10.1/x86_64-linux/auto/POSIX/POSIX.so...done.
Loaded symbols for /site/perl/perl-5.10.1-1/lib/5.10.1/x86_64-linux/auto/POSIX/POSIX.so
Reading symbols from /site/perl/perl-5.10.1-1/lib/5.10.1/x86_64-linux/auto/File/Glob/Glob.so...done.
Loaded symbols for /site/perl/perl-5.10.1-1/lib/5.10.1/x86_64-linux/auto/File/Glob/Glob.so
0x00007fbe87eb05b5 in waitpid () from /lib/libpthread.so.0
[Switching to Thread 0x7fbe88efb750 (LWP 17962)]
/home/iheffner/perl-stacktrace.gdb:588: Error in sourced command file:
Cannot access memory at address 0xffffffff83873d18
(gdb) quit
The program is running.  Quit anyway (and detach it)? (y or n) y
Detaching from program: /site/httpd/httpd-2.2.14-1/bin/httpd, process 17962
iheffner@hardtop:~/git/next/call_processor$

[mgruberman]

Could you try with set trace on? That'll at least show us what lines are being executed.

[mgruberman]

Also, if you substitute set $each_var = ... for print $each_var = ... then gdb will also print the value of each expression as it is evaluated. That would also be helpful. Sorry for not having an updated script for you yet. :-(

[mgruberman]

I just released v0.09 which integrates the above suggestions and has oodles of additional debugging and fail-over code. Your case will probably still fail but may usefully report more diagnostics if you run it with perl-stacktrace -v ....