I have a pretty straightforward LAMP stack, with Apache 2.4.23 running a handful of SSL-enabled sites, with PHP 7.1.8 running in a typical mod_php setup. I compiled Apache against OpenSSL 1.0.2j, and everything works great 99% of the time. I've had a handful of segfaults over the past few months, and this morning there was something that just went horribly awry and basically brought the server to its knees. When I rebooted, there was a 2.8 GB core file from Apache, and another 0.9 GB one from the day before. The backtrace seems to both indicate they had issues in the same area. Both have this line:
ssleay_rand_add.part.0 () from /usr/local/openssl/lib/libcrypto.so.1.0.0
...followed by a failed assertion from libc, and then the dump:
(gdb) thread apply all bt
Thread 27 (Thread 0x7f99e4ff9700 (LWP 8493)):
#0 0x00007f99fc6693d1 in do_lookup_x () from /lib64/ld-linux-x86-64.so.2
#1 0x00007f99fc669caf in _dl_lookup_symbol_x () from /lib64/ld-linux-x86-64.so.2
#2 0x00007f99fc66e5b6 in _dl_fixup () from /lib64/ld-linux-x86-64.so.2
#3 0x00007f99fc6750c0 in _dl_runtime_resolve () from /lib64/ld-linux-x86-64.so.2
#4 0x0000000000459148 in sig_coredump (sig=6) at mpm_unix.c:959
#5 <signal handler called>
#6 0x00007f99fb1921d7 in raise () from /lib64/libc.so.6
#7 0x00007f99fb1938c8 in abort () from /lib64/libc.so.6
#8 0x00007f99fb18b146 in __assert_fail_base () from /lib64/libc.so.6
#9 0x00007f99fb18b1f2 in __assert_fail () from /lib64/libc.so.6
#10 0x00007f99f8114617 in ssleay_rand_add.part.0 () from /usr/local/openssl/lib/libcrypto.so.1.0.0
#11 0x00007f99f86d5df0 in ssl_rand_seed (s=s#entry=0x2605c58, p=0x7f99e8045438, nCtx=nCtx#entry=SSL_RSCTX_CONNECT, prefix=prefix#entry=0x7f99f86e11dd "") at ssl_engine_rand.c:112
#12 0x00007f99f86c3a0b in ssl_init_ssl_connection (c=0x7f99e8045748, r=0x0) at mod_ssl.c:468
#13 0x0000000000456edb in ap_run_pre_connection (c=c#entry=0x7f99e8045748, csd=csd#entry=0x7f99e80454b0) at connection.c:43
#14 0x000000000046a11a in process_socket (my_thread_num=8, my_child_num=0, cs=0x7f99e80456b8, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1052
#15 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#16 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#17 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7f99e47f8700 (LWP 8494)):
#0 ap_method_number_of (method=0x7f99ac0ce478 "POST") at http_protocol.c:940
#1 0x0000000000438ad5 in read_request_line (bb=0x7f99ac0ce3f0, r=0x7f99ac0ccee0) at protocol.c:634
#2 ap_read_request (conn=conn#entry=0x7f99e803f4a8) at protocol.c:973
#3 0x000000000045eb5d in ap_process_http_async_connection (c=0x7f99e803f4a8) at http_core.c:146
#4 ap_process_http_connection (c=0x7f99e803f4a8) at http_core.c:248
#5 0x0000000000456da0 in ap_run_process_connection (c=c#entry=0x7f99e803f4a8) at connection.c:42
#6 0x000000000046a1b2 in process_socket (my_thread_num=9, my_child_num=0, cs=0x7f99e803f418, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1102
#7 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#8 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7f99dcfe9700 (LWP 8509)):
#0 ap_method_number_of (method=0x7f99780c9b10 "GET") at http_protocol.c:940
#1 0x0000000000438ad5 in read_request_line (bb=0x7f99780c9a80, r=0x7f99780c8570) at protocol.c:634
#2 ap_read_request (conn=conn#entry=0x7f99e8053d68) at protocol.c:973
#3 0x000000000045eb5d in ap_process_http_async_connection (c=0x7f99e8053d68) at http_core.c:146
#4 ap_process_http_connection (c=0x7f99e8053d68) at http_core.c:248
#5 0x0000000000456da0 in ap_run_process_connection (c=c#entry=0x7f99e8053d68) at connection.c:42
#6 0x000000000046a1b2 in process_socket (my_thread_num=24, my_child_num=0, cs=0x7f99e8053cd8, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1102
#7 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#8 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7f99e77fe700 (LWP 8488)):
#0 apr_md5_init (context=context#entry=0x7f99e77fdd80) at crypto/apr_md5.c:147
#1 0x00000000004561ae in ap_md5_binary (p=0x7f99e80495f8, buf=0x2585820 "(DOMAIN NAME REMOVED):443", length=25) at util_md5.c:63
#2 0x00007f99f86c3a72 in ssl_init_ssl_connection (c=0x7f99e8049908, r=0x0) at mod_ssl.c:493
#3 0x0000000000456edb in ap_run_pre_connection (c=c#entry=0x7f99e8049908, csd=csd#entry=0x7f99e8049670) at connection.c:43
#4 0x000000000046a11a in process_socket (my_thread_num=3, my_child_num=0, cs=0x7f99e8049878, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1052
#5 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#6 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7f99e5ffb700 (LWP 8491)):
#0 apr_md5_init (context=context#entry=0x7f99e5ffad80) at crypto/apr_md5.c:147
#1 0x00000000004561ae in ap_md5_binary (p=0x7f99e804d7b8, buf=0x2585820 "(DOMAIN NAME REMOVED):443", length=25) at util_md5.c:63
#2 0x00007f99f86c3a72 in ssl_init_ssl_connection (c=0x7f99e804dac8, r=0x0) at mod_ssl.c:493
#3 0x0000000000456edb in ap_run_pre_connection (c=c#entry=0x7f99e804dac8, csd=csd#entry=0x7f99e804d830) at connection.c:43
#4 0x000000000046a11a in process_socket (my_thread_num=6, my_child_num=0, cs=0x7f99e804da38, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1052
#5 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#6 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7f99df7ee700 (LWP 8504)):
#0 ap_method_number_of (method=0x7f998411d5d8 "GET") at http_protocol.c:940
#1 0x0000000000438ad5 in read_request_line (bb=0x7f998411d550, r=0x7f998411c040) at protocol.c:634
#2 ap_read_request (conn=conn#entry=0x7f99e804fba8) at protocol.c:973
#3 0x000000000045eb5d in ap_process_http_async_connection (c=0x7f99e804fba8) at http_core.c:146
#4 ap_process_http_connection (c=0x7f99e804fba8) at http_core.c:248
#5 0x0000000000456da0 in ap_run_process_connection (c=c#entry=0x7f99e804fba8) at connection.c:42
#6 0x000000000046a1b2 in process_socket (my_thread_num=19, my_child_num=0, cs=0x7f99e804fb18, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1102
#7 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#8 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7f99e17f2700 (LWP 8500)):
#0 apr_md5_init (context=context#entry=0x7f99e17f1d80) at crypto/apr_md5.c:147
#1 0x00000000004561ae in ap_md5_binary (p=0x7f99e8059cf8, buf=0x2585820 "(DOMAIN NAME REMOVED):443", length=25) at util_md5.c:63
#2 0x00007f99f86c3a72 in ssl_init_ssl_connection (c=0x7f99e805a008, r=0x0) at mod_ssl.c:493
#3 0x0000000000456edb in ap_run_pre_connection (c=c#entry=0x7f99e805a008, csd=csd#entry=0x7f99e8059d70) at connection.c:43
#4 0x000000000046a11a in process_socket (my_thread_num=15, my_child_num=0, cs=0x7f99e8059f78, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1052
#5 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#6 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7f99e7fff700 (LWP 8487)):
#0 0x00007f99fb24ee27 in munmap () from /lib64/libc.so.6
#1 0x00007f99fb18b137 in __assert_fail_base () from /lib64/libc.so.6
#2 0x00007f99fb18b1f2 in __assert_fail () from /lib64/libc.so.6
#3 0x00007f99f8114617 in ssleay_rand_add.part.0 () from /usr/local/openssl/lib/libcrypto.so.1.0.0
#4 0x00007f99f86d5df0 in ssl_rand_seed (s=s#entry=0x258ca08, p=0x7f99e8038ef8, nCtx=nCtx#entry=SSL_RSCTX_CONNECT, prefix=prefix#entry=0x7f99f86e11dd "") at ssl_engine_rand.c:112
#5 0x00007f99f86c3a0b in ssl_init_ssl_connection (c=0x7f99e8039218, r=0x0) at mod_ssl.c:468
#6 0x0000000000456edb in ap_run_pre_connection (c=c#entry=0x7f99e8039218, csd=csd#entry=0x7f99e8038f80) at connection.c:43
#7 0x000000000046a11a in process_socket (my_thread_num=2, my_child_num=0, cs=0x7f99e8039188, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1052
#8 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#9 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7f99e0ff1700 (LWP 8501)):
#0 ap_method_number_of (method=0x7f99a4068468 "POST") at http_protocol.c:940
#1 0x0000000000438ad5 in read_request_line (bb=0x7f99a40683e0, r=0x7f99a4066ed0) at protocol.c:634
#2 ap_read_request (conn=conn#entry=0x7f99e803b2f8) at protocol.c:973
#3 0x000000000045eb5d in ap_process_http_async_connection (c=0x7f99e803b2f8) at http_core.c:146
#4 ap_process_http_connection (c=0x7f99e803b2f8) at http_core.c:248
#5 0x0000000000456da0 in ap_run_process_connection (c=c#entry=0x7f99e803b2f8) at connection.c:42
#6 0x000000000046a1b2 in process_socket (my_thread_num=16, my_child_num=0, cs=0x7f99e803b268, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1102
#7 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#8 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7f99e2ff5700 (LWP 8497)):
#0 ap_method_number_of (method=0x7f997acaead8 "POST") at http_protocol.c:940
#1 0x0000000000438ad5 in read_request_line (bb=0x7f997acaea50, r=0x7f997acad540) at protocol.c:634
#2 ap_read_request (conn=conn#entry=0x7f99e8037118) at protocol.c:973
#3 0x000000000045eb5d in ap_process_http_async_connection (c=0x7f99e8037118) at http_core.c:146
#4 ap_process_http_connection (c=0x7f99e8037118) at http_core.c:248
#5 0x0000000000456da0 in ap_run_process_connection (c=c#entry=0x7f99e8037118) at connection.c:42
#6 0x000000000046a1b2 in process_socket (my_thread_num=12, my_child_num=0, cs=0x7f99e8037088, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1102
#7 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#8 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7f99e37f6700 (LWP 8496)):
#0 0x00007f99fb24ee27 in munmap () from /lib64/libc.so.6
#1 0x00007f99fb18b137 in __assert_fail_base () from /lib64/libc.so.6
#2 0x00007f99fb18b1f2 in __assert_fail () from /lib64/libc.so.6
#3 0x00007f99f8114617 in ssleay_rand_add.part.0 () from /usr/local/openssl/lib/libcrypto.so.1.0.0
#4 0x00007f99f86d5df0 in ssl_rand_seed (s=s#entry=0x258ca08, p=0x7f99e8055b38, nCtx=nCtx#entry=SSL_RSCTX_CONNECT, prefix=prefix#entry=0x7f99f86e11dd "") at ssl_engine_rand.c:112
#5 0x00007f99f86c3a0b in ssl_init_ssl_connection (c=0x7f99e8055e48, r=0x0) at mod_ssl.c:468
#6 0x0000000000456edb in ap_run_pre_connection (c=c#entry=0x7f99e8055e48, csd=csd#entry=0x7f99e8055bb0) at connection.c:43
#7 0x000000000046a11a in process_socket (my_thread_num=11, my_child_num=0, cs=0x7f99e8055db8, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1052
#8 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#9 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7f99defed700 (LWP 8505)):
#0 ap_method_number_of (method=0x7f998812ff18 "GET") at http_protocol.c:940
#1 0x0000000000438ad5 in read_request_line (bb=0x7f998812fe90, r=0x7f998812e980) at protocol.c:634
#2 ap_read_request (conn=conn#entry=0x7f99e8062388) at protocol.c:973
#3 0x000000000045eb5d in ap_process_http_async_connection (c=0x7f99e8062388) at http_core.c:146
#4 ap_process_http_connection (c=0x7f99e8062388) at http_core.c:248
#5 0x0000000000456da0 in ap_run_process_connection (c=c#entry=0x7f99e8062388) at connection.c:42
#6 0x000000000046a1b2 in process_socket (my_thread_num=20, my_child_num=0, cs=0x7f99e80622f8, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1102
#7 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#8 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7f99e3ff7700 (LWP 8495)):
#0 ap_method_number_of (method=0x7f99b005d778 "GET") at http_protocol.c:940
#1 0x0000000000438ad5 in read_request_line (bb=0x7f99b005d6f0, r=0x7f99b005c1e0) at protocol.c:634
#2 ap_read_request (conn=conn#entry=0x7f99e8068628) at protocol.c:973
#3 0x000000000045eb5d in ap_process_http_async_connection (c=0x7f99e8068628) at http_core.c:146
#4 ap_process_http_connection (c=0x7f99e8068628) at http_core.c:248
#5 0x0000000000456da0 in ap_run_process_connection (c=c#entry=0x7f99e8068628) at connection.c:42
#6 0x000000000046a1b2 in process_socket (my_thread_num=10, my_child_num=0, cs=0x7f99e8068598, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1102
#7 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#8 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7f99e07f0700 (LWP 8502)):
#0 ap_method_number_of (method=0x7f998c004218 "GET") at http_protocol.c:940
#1 0x0000000000438ad5 in read_request_line (bb=0x7f998c004190, r=0x7f998c002c80) at protocol.c:634
#2 ap_read_request (conn=conn#entry=0x7f99e8047828) at protocol.c:973
#3 0x000000000045eb5d in ap_process_http_async_connection (c=0x7f99e8047828) at http_core.c:146
#4 ap_process_http_connection (c=0x7f99e8047828) at http_core.c:248
#5 0x0000000000456da0 in ap_run_process_connection (c=c#entry=0x7f99e8047828) at connection.c:42
#6 0x000000000046a1b2 in process_socket (my_thread_num=17, my_child_num=0, cs=0x7f99e8047798, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1102
#7 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#8 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7f99ddfeb700 (LWP 8507)):
#0 ap_method_number_of (method=0x7f99800da758 "GET") at http_protocol.c:940
#1 0x0000000000438ad5 in read_request_line (bb=0x7f99800da6d0, r=0x7f99800d91c0) at protocol.c:634
#2 ap_read_request (conn=conn#entry=0x7f99e8066548) at protocol.c:973
#3 0x000000000045eb5d in ap_process_http_async_connection (c=0x7f99e8066548) at http_core.c:146
#4 ap_process_http_connection (c=0x7f99e8066548) at http_core.c:248
#5 0x0000000000456da0 in ap_run_process_connection (c=c#entry=0x7f99e8066548) at connection.c:42
#6 0x000000000046a1b2 in process_socket (my_thread_num=22, my_child_num=0, cs=0x7f99e80664b8, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1102
#7 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#8 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7f99e27f4700 (LWP 8498)):
#0 ap_method_number_of (method=0x7f999c0e1ba0 "GET") at http_protocol.c:940
#1 0x0000000000438ad5 in read_request_line (bb=0x7f999c0e1b10, r=0x7f999c0e0600) at protocol.c:634
#2 ap_read_request (conn=conn#entry=0x7f99e805c0e8) at protocol.c:973
#3 0x000000000045eb5d in ap_process_http_async_connection (c=0x7f99e805c0e8) at http_core.c:146
#4 ap_process_http_connection (c=0x7f99e805c0e8) at http_core.c:248
#5 0x0000000000456da0 in ap_run_process_connection (c=c#entry=0x7f99e805c0e8) at connection.c:42
#6 0x000000000046a1b2 in process_socket (my_thread_num=13, my_child_num=0, cs=0x7f99e805c058, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1102
#7 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#8 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7f99e57fa700 (LWP 8492)):
#0 apr_md5_init (context=context#entry=0x7f99e57f9d80) at crypto/apr_md5.c:147
#1 0x00000000004561ae in ap_md5_binary (p=0x7f99e8051978, buf=0x2585820 "(DOMAIN NAME REMOVED):443", length=25) at util_md5.c:63
#2 0x00007f99f86c3a72 in ssl_init_ssl_connection (c=0x7f99e8051c88, r=0x0) at mod_ssl.c:493
#3 0x0000000000456edb in ap_run_pre_connection (c=c#entry=0x7f99e8051c88, csd=csd#entry=0x7f99e80519f0) at connection.c:43
#4 0x000000000046a11a in process_socket (my_thread_num=7, my_child_num=0, cs=0x7f99e8051bf8, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1052
#5 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#6 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7f99dc7e8700 (LWP 8510)):
#0 0x00007f99fb72d6d5 in pthread_cond_wait##GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000046d3cc in ap_queue_info_wait_for_idler (queue_info=0x257f008, had_to_block=had_to_block#entry=0x7f99dc7e7ebc) at fdqueue.c:180
#2 0x000000000046a523 in get_worker (have_idle_worker_p=0x7f99dc7e7eb0, blocking=1, all_busy=0x7f99dc7e7ebc) at event.c:1328
#3 0x000000000046b2c8 in get_worker (all_busy=0x7f99dc7e7ebc, blocking=1, have_idle_worker_p=0x7f99dc7e7eb0) at event.c:1203
#4 listener_thread (thd=0x2580028, dummy=<optimized out>) at event.c:1762
#5 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f99ed7fe700 (LWP 8485)):
#0 ap_method_number_of (method=0x7f99d40e6ce8 "GET") at http_protocol.c:940
#1 0x0000000000438ad5 in read_request_line (bb=0x7f99d40e6c60, r=0x7f99d40e5750) at protocol.c:634
#2 ap_read_request (conn=conn#entry=0x7f99e8043668) at protocol.c:973
#3 0x000000000045eb5d in ap_process_http_async_connection (c=0x7f99e8043668) at http_core.c:146
#4 ap_process_http_connection (c=0x7f99e8043668) at http_core.c:248
#5 0x0000000000456da0 in ap_run_process_connection (c=c#entry=0x7f99e8043668) at connection.c:42
#6 0x000000000046a1b2 in process_socket (my_thread_num=0, my_child_num=0, cs=0x7f99e80435d8, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1102
#7 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#8 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f99de7ec700 (LWP 8506)):
#0 ap_method_number_of (method=0x7f997c06aa08 "GET") at http_protocol.c:940
#1 0x0000000000438ad5 in read_request_line (bb=0x7f997c06a980, r=0x7f997c069470) at protocol.c:634
#2 ap_read_request (conn=conn#entry=0x7f99e8064468) at protocol.c:973
#3 0x000000000045eb5d in ap_process_http_async_connection (c=0x7f99e8064468) at http_core.c:146
#4 ap_process_http_connection (c=0x7f99e8064468) at http_core.c:248
#5 0x0000000000456da0 in ap_run_process_connection (c=c#entry=0x7f99e8064468) at connection.c:42
#6 0x000000000046a1b2 in process_socket (my_thread_num=21, my_child_num=0, cs=0x7f99e80643d8, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1102
#7 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#8 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7f99fc871740 (LWP 8483)):
#0 0x00007f99fb73049d in read () from /lib64/libpthread.so.0
#1 0x000000000045a034 in ap_mpm_podx_check (pod=<optimized out>) at mpm_unix.c:535
#2 0x000000000042b3f7 in child_main (child_num_arg=child_num_arg#entry=0, child_bucket=child_bucket#entry=0) at event.c:2368
#3 0x000000000046c2a7 in make_child (s=0x25b8d58, slot=0, bucket=bucket#entry=0) at event.c:2461
#4 0x000000000046cb45 in perform_idle_server_maintenance (num_buckets=<optimized out>, child_bucket=<optimized out>) at event.c:2664
#5 server_main_loop (num_buckets=1, remaining_children_to_start=<optimized out>) at event.c:2798
#6 event_run (_pconf=<optimized out>, plog=<optimized out>, s=<optimized out>) at event.c:2876
#7 0x000000000043271e in ap_run_mpm (pconf=pconf#entry=0x2555138, plog=0x2582378, s=0x25b8d58) at mpm_common.c:94
#8 0x000000000042be71 in main (argc=3, argv=0x7fff71da6a28) at main.c:783
Thread 6 (Thread 0x7f99ecffd700 (LWP 8486)):
#0 ap_method_number_of (method=0x7f99cc006ea8 "GET") at http_protocol.c:940
#1 0x0000000000438ad5 in read_request_line (bb=0x7f99cc006e20, r=0x7f99cc005910) at protocol.c:634
#2 ap_read_request (conn=conn#entry=0x7f99e8057f28) at protocol.c:973
#3 0x000000000045eb5d in ap_process_http_async_connection (c=0x7f99e8057f28) at http_core.c:146
#4 ap_process_http_connection (c=0x7f99e8057f28) at http_core.c:248
#5 0x0000000000456da0 in ap_run_process_connection (c=c#entry=0x7f99e8057f28) at connection.c:42
#6 0x000000000046a1b2 in process_socket (my_thread_num=1, my_child_num=0, cs=0x7f99e8057e98, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1102
#7 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#8 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f99e6ffd700 (LWP 8489)):
#0 0x00007f99fb24ee27 in munmap () from /lib64/libc.so.6
#1 0x00007f99fb18b137 in __assert_fail_base () from /lib64/libc.so.6
#2 0x00007f99fb18b1f2 in __assert_fail () from /lib64/libc.so.6
#3 0x00007f99f8114617 in ssleay_rand_add.part.0 () from /usr/local/openssl/lib/libcrypto.so.1.0.0
#4 0x00007f99f86d5e88 in ssl_rand_seed (s=s#entry=0x2590578, p=0x7f99e8041278, nCtx=nCtx#entry=SSL_RSCTX_CONNECT, prefix=prefix#entry=0x7f99f86e11dd "") at ssl_engine_rand.c:119
#5 0x00007f99f86c3a0b in ssl_init_ssl_connection (c=0x7f99e8041588, r=0x0) at mod_ssl.c:468
#6 0x0000000000456edb in ap_run_pre_connection (c=c#entry=0x7f99e8041588, csd=csd#entry=0x7f99e80412f0) at connection.c:43
#7 0x000000000046a11a in process_socket (my_thread_num=4, my_child_num=0, cs=0x7f99e80414f8, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1052
#8 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#9 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f99dffef700 (LWP 8503)):
#0 ap_method_number_of (method=0x7f99900da7d8 "GET") at http_protocol.c:940
#1 0x0000000000438ad5 in read_request_line (bb=0x7f99900da750, r=0x7f99900d9240) at protocol.c:634
#2 ap_read_request (conn=conn#entry=0x7f99e804b9e8) at protocol.c:973
#3 0x000000000045eb5d in ap_process_http_async_connection (c=0x7f99e804b9e8) at http_core.c:146
#4 ap_process_http_connection (c=0x7f99e804b9e8) at http_core.c:248
#5 0x0000000000456da0 in ap_run_process_connection (c=c#entry=0x7f99e804b9e8) at connection.c:42
#6 0x000000000046a1b2 in process_socket (my_thread_num=18, my_child_num=0, cs=0x7f99e804b958, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1102
#7 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#8 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f99e67fc700 (LWP 8490)):
#0 apr_md5_init (context=context#entry=0x7f99e67fbd80) at crypto/apr_md5.c:147
#1 0x00000000004561ae in ap_md5_binary (p=0x7f99e805deb8, buf=0x2585780 "(DOMAIN NAME #2 REMOVED):443", length=20) at util_md5.c:63
#2 0x00007f99f86c3a72 in ssl_init_ssl_connection (c=0x7f99e805e1c8, r=0x0) at mod_ssl.c:493
#3 0x0000000000456edb in ap_run_pre_connection (c=c#entry=0x7f99e805e1c8, csd=csd#entry=0x7f99e805df30) at connection.c:43
#4 0x000000000046a11a in process_socket (my_thread_num=5, my_child_num=0, cs=0x7f99e805e138, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1052
#5 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#6 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f99e1ff3700 (LWP 8499)):
#0 ap_method_number_of (method=0x7f99a0004218 "GET") at http_protocol.c:940
#1 0x0000000000438ad5 in read_request_line (bb=0x7f99a0004190, r=0x7f99a0002c80) at protocol.c:634
#2 ap_read_request (conn=conn#entry=0x7f99e80602a8) at protocol.c:973
#3 0x000000000045eb5d in ap_process_http_async_connection (c=0x7f99e80602a8) at http_core.c:146
#4 ap_process_http_connection (c=0x7f99e80602a8) at http_core.c:248
#5 0x0000000000456da0 in ap_run_process_connection (c=c#entry=0x7f99e80602a8) at connection.c:42
#6 0x000000000046a1b2 in process_socket (my_thread_num=14, my_child_num=0, cs=0x7f99e8060218, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1102
#7 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#8 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f99fb25473d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f99dd7ea700 (LWP 8508)):
#0 0x00007f99fb1921d7 in raise () from /lib64/libc.so.6
#1 0x00007f99fb1938c8 in abort () from /lib64/libc.so.6
#2 0x00007f99fb18b146 in __assert_fail_base () from /lib64/libc.so.6
#3 0x00007f99fb18b1f2 in __assert_fail () from /lib64/libc.so.6
#4 0x00007f99f8114617 in ssleay_rand_add.part.0 () from /usr/local/openssl/lib/libcrypto.so.1.0.0
#5 0x00007f99f86d5df0 in ssl_rand_seed (s=s#entry=0x2605c58, p=0x7f99e803d0b8, nCtx=nCtx#entry=SSL_RSCTX_CONNECT, prefix=prefix#entry=0x7f99f86e11dd "") at ssl_engine_rand.c:112
#6 0x00007f99f86c3a0b in ssl_init_ssl_connection (c=0x7f99e803d3c8, r=0x0) at mod_ssl.c:468
#7 0x0000000000456edb in ap_run_pre_connection (c=c#entry=0x7f99e803d3c8, csd=csd#entry=0x7f99e803d130) at connection.c:43
#8 0x000000000046a11a in process_socket (my_thread_num=23, my_child_num=0, cs=0x7f99e803d338, sock=<optimized out>, p=<optimized out>, thd=<optimized out>) at event.c:1052
#9 worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:1963
#10 0x00007f99fb729dc5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f99fb25473d in clone () from /lib64/libc.so.6
However, I'm no expert at reading core dumps - I've read through a few posts like How to analyze a program's core dump file with gdb? and https://httpd.apache.org/dev/debugging.html, but I'm not quite sure where to go next.
Should I be assuming that there's something wrong in my OpenSSL library and I should recompile it? Or am I missing something else? All feedback welcome!
Should I be assuming that there's something wrong in my OpenSSL library and I should recompile it?
Most likely.
You are using /usr/local/openssl/lib/libcrypto.so.1.0.0, which I assume is not the one that came with your distribution.
It appears that you didn't compile this library for use with threads, but your Apache is using that library from multiple threads, triggering assertion failure.
This is the only assert I can find inside ssleay_rand_add, and it is only compiled in when !defined(OPENSSL_THREADS), which I assume is true for your build, and is the result of ./Configure no-threads ....
You should rebuild libcrypto.so with ./Configure threads .... Documentation.
I'm getting a very infrequent ( once every 10-15 days ) crash of my application. Based on the core dump backtrace, its crashing while redrawing the UI during the function cairo_pattern_destroy.
wxWidgets version 2.95
cairo version 1.8.0
Fedora core 10
LXDE windows manager
Here is the full backtrace for one of the crashes. I have 3 of them and they all end at __libc_free in cairo_pattern_destroy. If anyone has any suggestions it would be greatly appreciated.
#0 __libc_free (mem=0x7) at malloc.c:3599
3599 if (chunk_is_mmapped(p)) /* release mmapped memory. */
(gdb) bt
#0 __libc_free (mem=0x7) at malloc.c:3599
#1 0x00ff76d6 in cairo_pattern_destroy (pattern=0xc7b5088)
at cairo-pattern.c:738
#2 0x00fe7a5d in _cairo_gstate_fini (gstate=0xd20bc00) at cairo-gstate.c:204
#3 0x00fe7a98 in _cairo_gstate_restore (gstate=0x0, freelist=0xdb2e1f4)
at cairo-gstate.c:260
#4 0x00fe1ffd in cairo_restore (cr=0xdb2e070) at cairo.c:363
#5 0x07881fe2 in gdk_pango_renderer_draw_glyphs (renderer=0xc50d800,
font=0xa29d808, glyphs=0xceed960, x=129024, y=91136) at gdkpango.c:247
#6 0x07d345ea in pango_renderer_draw_glyphs (renderer=0xc50d800,
font=0xa29d808, glyphs=0xceed960, x=129024, y=91136)
at pango-renderer.c:639
#7 0x07d3466e in pango_renderer_default_draw_glyph_item (renderer=0xc50d800,
text=0xd411b00 "Pause", glyph_item=0xb2e11d80, x=129024, y=91136)
at pango-renderer.c:715
#8 0x07d3455a in pango_renderer_draw_glyph_item (renderer=0xc50d800,
text=0xd411b00 "Pause", glyph_item=0xb2e11d80, x=129024, y=91136)
at pango-renderer.c:703
#9 0x07d34dd3 in pango_renderer_draw_layout_line (renderer=0xc50d800,
line=0xd242648, x=129024, y=91136) at pango-renderer.c:568
#10 0x07d350e1 in pango_renderer_draw_layout (renderer=0xc50d800,
layout=0xd5a5e10, x=129024, y=77824) at pango-renderer.c:192
#11 0x07880f11 in IA__gdk_draw_layout_with_colors (drawable=0xc690580,
---Type <return> to continue, or q <return> to quit---
gc=0xa26f050, x=126, y=76, layout=0xd5a5e10, foreground=0x0,
background=0x0) at gdkpango.c:951
#12 0x078810c1 in IA__gdk_draw_layout (drawable=0xc690580, gc=0xa26f050,
x=126, y=76, layout=0xd5a5e10) at gdkpango.c:1013
#13 0x07a96e92 in gtk_default_draw_layout (style=0xa360500, window=0xc690580,
state_type=GTK_STATE_INSENSITIVE, use_text=0, area=0xcdf165c,
widget=0xd7afa60, detail=0x7c42081 "label", x=126, y=76, layout=0xda63110)
at gtkstyle.c:5084
#14 0x07a92fa2 in IA__gtk_paint_layout (style=0xa360500, window=0xc690580,
state_type=GTK_STATE_INSENSITIVE, use_text=0, area=0xcdf165c,
widget=0xd7afa60, detail=0x7c42081 "label", x=126, y=76, layout=0xda63110)
at gtkstyle.c:6401
#15 0x07a0ea9c in gtk_label_expose (widget=0xd7afa60, event=0xcdf1650)
at gtklabel.c:2848
#16 0x07a1e116 in _gtk_marshal_BOOLEAN__BOXED (closure=0xa1daae8,
return_value=0xbfae5910, n_param_values=2, param_values=0xdb00740,
invocation_hint=0xbfae58fc, marshal_data=0x7a0e990) at gtkmarshalers.c:84
#17 0x05a80959 in g_type_class_meta_marshal (closure=0xa1daae8,
return_value=0xbfae5910, n_param_values=2, param_values=0xdb00740,
invocation_hint=0xbfae58fc, marshal_data=0xc8) at gclosure.c:878
#18 0x05a82108 in IA__g_closure_invoke (closure=0xa1daae8,
return_value=0xbfae5910, n_param_values=2, param_values=0xdb00740,
invocation_hint=0xbfae58fc) at gclosure.c:767
---Type <return> to continue, or q <return> to quit---
#19 0x05a982cd in signal_emit_unlocked_R (node=0xa1dabe0, detail=0,
instance=0xd7afa60, emission_return=0xbfae5a48,
instance_and_params=0xdb00740) at gsignal.c:3282
#20 0x05a99bbb in IA__g_signal_emit_valist (instance=0xd7afa60, signal_id=38,
detail=0, var_args=0xbfae5aa0 "�Z��P\026�\f`�z\r�\225�\a`�z\rho\035\n")
at gsignal.c:2987
#21 0x05a9a1b6 in IA__g_signal_emit (instance=0xd7afa60, signal_id=38,
detail=0) at gsignal.c:3034
#22 0x07b333ae in gtk_widget_event_internal (widget=0xd7afa60, event=0xcdf1650)
at gtkwidget.c:4745
#23 0x079941a3 in IA__gtk_container_propagate_expose (container=0xc050930,
child=0xd7afa60, event=0xda6fdf8) at gtkcontainer.c:2687
#24 0x079941d1 in gtk_container_expose_child (child=0xd7afa60,
client_data=0xbfae5b68) at gtkcontainer.c:2575
#25 0x0795e61d in gtk_bin_forall (container=0xc050930, include_internals=1,
callback=0x79941b0 <gtk_container_expose_child>, callback_data=0xbfae5b68)
at gtkbin.c:128
#26 0x07994d66 in IA__gtk_container_forall (container=0xc050930,
callback=0x79941b0 <gtk_container_expose_child>, callback_data=0xbfae5b68)
at gtkcontainer.c:1455
#27 0x07996450 in gtk_container_expose (widget=0xc050930, event=0xda6fdf8)
at gtkcontainer.c:2598
#28 0x079689b1 in gtk_button_expose (widget=0xc050930, event=0xda6fdf8)
---Type <return> to continue, or q <return> to quit---
at gtkbutton.c:1348
#29 0x07a1e116 in _gtk_marshal_BOOLEAN__BOXED (closure=0xa1daae8,
return_value=0xbfae5d70, n_param_values=2, param_values=0xd6b2140,
invocation_hint=0xbfae5d5c, marshal_data=0x7968930) at gtkmarshalers.c:84
#30 0x05a80959 in g_type_class_meta_marshal (closure=0xa1daae8,
return_value=0xbfae5d70, n_param_values=2, param_values=0xd6b2140,
invocation_hint=0xbfae5d5c, marshal_data=0xc8) at gclosure.c:878
#31 0x05a82108 in IA__g_closure_invoke (closure=0xa1daae8,
return_value=0xbfae5d70, n_param_values=2, param_values=0xd6b2140,
invocation_hint=0xbfae5d5c) at gclosure.c:767
#32 0x05a982cd in signal_emit_unlocked_R (node=0xa1dabe0, detail=0,
instance=0xc050930, emission_return=0xbfae5ea8,
instance_and_params=0xd6b2140) at gsignal.c:3282
#33 0x05a99bbb in IA__g_signal_emit_valist (instance=0xc050930, signal_id=38,
detail=0,
var_args=0xbfae5f00 "\030_�����\r0\t\005\f�\225�\a0\t\005\fho\035\n")
at gsignal.c:2987
#34 0x05a9a1b6 in IA__g_signal_emit (instance=0xc050930, signal_id=38,
detail=0) at gsignal.c:3034
#35 0x07b333ae in gtk_widget_event_internal (widget=0xc050930, event=0xda6fdf8)
at gtkwidget.c:4745
#36 0x079941a3 in IA__gtk_container_propagate_expose (container=0xc010228,
child=0xc050930, event=0xbfae63f4) at gtkcontainer.c:2687
---Type <return> to continue, or q <return> to quit---
#37 0x079941d1 in gtk_container_expose_child (child=0xc050930,
client_data=0xbfae5fd8) at gtkcontainer.c:2575
#38 0x079d2f79 in gtk_fixed_forall (container=0xc010228, include_internals=1,
callback=0x79941b0 <gtk_container_expose_child>, callback_data=0xbfae5fd8)
at gtkfixed.c:449
#39 0x07994d66 in IA__gtk_container_forall (container=0xc010228,
callback=0x79941b0 <gtk_container_expose_child>, callback_data=0xbfae5fd8)
at gtkcontainer.c:1455
#40 0x07996450 in gtk_container_expose (widget=0xc010228, event=0xbfae63f4)
at gtkcontainer.c:2598
#41 0x07a1e116 in _gtk_marshal_BOOLEAN__BOXED (closure=0xa1daae8,
return_value=0xbfae61b0, n_param_values=2, param_values=0xd1e88c8,
invocation_hint=0xbfae619c, marshal_data=0x79963b0) at gtkmarshalers.c:84
#42 0x05a80959 in g_type_class_meta_marshal (closure=0xa1daae8,
return_value=0xbfae61b0, n_param_values=2, param_values=0xd1e88c8,
invocation_hint=0xbfae619c, marshal_data=0xc8) at gclosure.c:878
#43 0x05a821db in IA__g_closure_invoke (closure=0xa1daae8,
return_value=0xbfae61b0, n_param_values=2, param_values=0xd1e88c8,
invocation_hint=0xbfae619c) at gclosure.c:767
#44 0x05a982cd in signal_emit_unlocked_R (node=0xa1dabe0, detail=0,
instance=0xc010228, emission_return=0xbfae62e8,
instance_and_params=0xd1e88c8) at gsignal.c:3282
#45 0x05a99bbb in IA__g_signal_emit_valist (instance=0xc010228, signal_id=38,
---Type <return> to continue, or q <return> to quit---
detail=0,
var_args=0xbfae6340 "Xc���c��(\002\001\f�\225�\a(\002\001\fho\035\n")
at gsignal.c:2987
#46 0x05a9a1b6 in IA__g_signal_emit (instance=0xc010228, signal_id=38,
detail=0) at gsignal.c:3034
#47 0x07b333ae in gtk_widget_event_internal (widget=0xc010228,
event=0xbfae63f4) at gtkwidget.c:4745
#48 0x07a1821d in IA__gtk_main_do_event (event=0xbfae63f4) at gtkmain.c:1553
#49 0x07892625 in gdk_window_process_updates_internal (window=0xc690580)
at gdkwindow.c:2598
#50 0x07892c3f in IA__gdk_window_process_all_updates () at gdkwindow.c:2664
#51 0x07994eff in gtk_container_idle_sizer (data=0x0) at gtkcontainer.c:1309
#52 0x078760cb in gdk_threads_dispatch (data=0xc935530) at gdk.c:473
#53 0x066122d1 in g_idle_dispatch (source=0xd663908, callback=0xffffffff,
user_data=0xc935530) at gmain.c:4235
#54 0x06614208 in g_main_dispatch () at gmain.c:2144
#55 IA__g_main_context_dispatch (context=0xa1d5508) at gmain.c:2697
#56 0x066178b3 in g_main_context_iterate (context=0xa1d5508, block=1,
dispatch=1, self=0xa1ba5e0) at gmain.c:2778
#57 0x06617dd2 in IA__g_main_loop_run (loop=0xbb03cc0) at gmain.c:2986
#58 0x07a18489 in IA__gtk_main () at gtkmain.c:1200
#59 0x006b333d in wxGUIEventLoop::DoRun ()
from /usr/local/lib/libwx_gtk2u_core-2.9.so.5
This is an old question, so no one may actually be interested in the answer. I ran into this on an older version of Fedora 10 LXDE, with a very similar stack trace, and a crash in the same place. This turns out to be a design problem with Cairo 1.8.0.
Looking a frame 1 in my case, I see the following code:
734 i = solid_pattern_cache.size++ %
735 ARRAY_LENGTH (solid_pattern_cache.patterns);
736 /* swap an old pattern for this 'cache-hot' pattern */
737 if (solid_pattern_cache.patterns[i])
738 free (solid_pattern_cache.patterns[i]);
739
740 solid_pattern_cache.patterns[i] = (cairo_solid_pattern_t *) pattern;
They are using a counter with modular arithmetic to form the index into a fixed size cache. The counter is declared as a signed integer:
(gdb) ptype solid_pattern_cache
type = struct {
cairo_solid_pattern_t *patterns[4];
int size;
}
In my case, the counter has overflowed, resulting in a negative index into the cache. This results in attempting to free a random address.
(gdb) p solid_pattern_cache
$2 = {patterns = {0xb441b5e0, 0xb4983688, 0xb495bf58, 0xb3978388},
size = -2147483644}
(gdb) p /x solid_pattern_cache.size
$4 = 0x80000004
It's not clear to me why an unsigned counter was not used instead.