Hi!
This might be a long mail, so I'll start quickly:
Case: working on a PHP5/postgres system that segfaults ( apparently
randomly ) on different spots in the program.
- I have managed to track it down to our custom class loader ( static
object ) that manage all loading and initializing of objects. - In this class there is a function that has the responsibility to
require files, this is where the segfaults appears, all it does is to
check if a class is already loaded, then goes over some directories (
6-7 ) and finds a file based on the class name. This function have
just two function calls: "class_exists()" and "require_once()" - Now: with the minimal case ( still too large to post here ) the
program segfaults ( see further down ) when trying to include a file.
If I cheat and include all the files before the class loader kicks in
the program works fine.
If the segfault was the same everytime it should be easy to track
down, but I've had a large amount of different segfaults running
apache through gdb. The other thing is that one screen can be loaded
different times before the segfault comes as well, sometimes the
segfault comes straight away, other times after 5-6 reloads of the
same screen.
Configure:
./configure --prefix=/usr
--sysconfdir=/etc/php
--localstatedir=/var/lib/php
--with-apxs=/usr/sbin/apxs
--enable-ftp
--with-pear
--enable-soap
--with-ldap
--with-pgsql=/usr
--with-mysql=/usr
--with-ldap-ssl
--enable-calender
--with-gd
--with-png
--with-gmp
--with-dba
--with-xml
--with-gettext
--enable-trans-sid
--enable-sysvshm
--with-iconv
--enable-ctype
--with-regexp=php
--with-jpeg
--with-zlib
--with-xsl
--enable-debug
--enable-memory-limit
System:
- Poweredge 1750
- Dual 2.4Ghz Xeon
- Slackware 10 w/2.6.7 kernel
- GCC 3.3.4
- EEC memory 2GB
- Apache 1.3.31
- PHP5.0.0 release ( and latest snap )
Memory limit: 50MB
Memory tested with memtest86 for 17 hours
Segfaults has also been reproduced on a Dell D800 laptop. Slackware
10, PHP5.0.0, so there are no problems with the main machine we are
working on.
The segfaults:
- The one that comes up most often:
Program received signal SIGSEGV, Segmentation fault.
0x40c64e63 in _mem_block_check () from /usr/libexec/apache/libphp5.so
(gdb) bt
#0 0x40c64e63 in _mem_block_check () from /usr/libexec/apache/libphp5.so
#1 0x40c64f8d in _mem_block_check () from /usr/libexec/apache/libphp5.so
#2 0x40c64200 in _efree () from /usr/libexec/apache/libphp5.so
#3 0x40c4ddf0 in _php_stream_stat_path () from /usr/libexec/apache/libphp5.so
#4 0x40bd92e2 in php_stat () from /usr/libexec/apache/libphp5.so
#5 0x00000002 in ?? ()
#6 0xbfffd1d0 in ?? ()
#7 0x00000000 in ?? ()
#8 0xbfffd100 in ?? ()
#9 0x081a45cc in ?? ()
#10 0x0845a6a0 in ?? ()
#11 0x40c709d4 in _zval_ptr_dtor () from /usr/libexec/apache/libphp5.so
Previous frame inner to this frame (corrupt stack?)
As I said earlier: there are several segfaults, but this is the one
that shows up most of the times, it would have been "ok" if I had one
point where I could start to investigate, but here are some of the
other that I've encountered:
=======
Program received signal SIGSEGV, Segmentation fault.
0x40be22a5 in _efree () from /usr/libexec/apache/libphp5.so
(gdb) bt
#0 0x40be22a5 in _efree () from /usr/libexec/apache/libphp5.so
#1 0x40bf0804 in destroy_op_array () from /usr/libexec/apache/libphp5.so
#2 0x40bf0478 in destroy_zend_function () from /usr/libexec/apache/libphp5.so
#3 0x40bf049f in zend_function_dtor () from /usr/libexec/apache/libphp5.so
#4 0x40bfe798 in zend_hash_destroy () from /usr/libexec/apache/libphp5.so
#5 0x40bf05cc in destroy_zend_class () from /usr/libexec/apache/libphp5.so
#6 0x40bfe664 in zend_hash_del_key_or_index () from
/usr/libexec/apache/libphp5.so
#7 0x40bfebc0 in zend_hash_reverse_apply () from /usr/libexec/apache/libphp5.so
#8 0x40bed2dc in shutdown_executor () from /usr/libexec/apache/libphp5.so
#9 0x40bf77ee in zend_deactivate () from /usr/libexec/apache/libphp5.so
#10 0x40bbc9a2 in php_request_shutdown () from /usr/libexec/apache/libphp5.so
#11 0x40c2243f in apache_php_module_main () from /usr/libexec/apache/libphp5.so
#12 0x40c23269 in apache_php_module_main () from /usr/libexec/apache/libphp5.so
#13 0x40c2270d in apache_php_module_main () from /usr/libexec/apache/libphp5.so
#14 0x08054a17 in ap_invoke_handler ()
#15 0x08069f69 in ap_some_auth_required ()
#16 0x08069fc8 in ap_process_request ()
#17 0x08060f24 in ap_child_terminate ()
#18 0x080610f2 in ap_child_terminate ()
#19 0x08061258 in ap_child_terminate ()
#20 0x08061918 in ap_child_terminate ()
#21 0x08062150 in main ()
========
Program received signal SIGSEGV, Segmentation fault.
0x40bdfc29 in lex_scan () from /usr/libexec/apache/libphp5.so
(gdb) bt
#0 0x40bdfc29 in lex_scan () from /usr/libexec/apache/libphp5.so
#1 0x40bdf98d in lex_scan () from /usr/libexec/apache/libphp5.so
#2 0x40bea9f8 in zendlex () from /usr/libexec/apache/libphp5.so
#3 0x40bdc140 in zendparse () from /usr/libexec/apache/libphp5.so
#4 0x40bdc5c1 in compile_file () from /usr/libexec/apache/libphp5.so
#5 0x40c1bde3 in zend_include_or_eval_handler () from
/usr/libexec/apache/libphp5.so
#6 0x40c165c8 in execute () from /usr/libexec/apache/libphp5.so
#7 0x40c19e97 in zend_do_fcall_common_helper () from
/usr/libexec/apache/libphp5.so
#8 0x40c1a1a3 in zend_do_fcall_by_name_handler () from
/usr/libexec/apache/libphp5.so
#9 0x40c165c8 in execute () from /usr/libexec/apache/libphp5.so
#10 0x40bee17e in zend_call_function () from /usr/libexec/apache/libphp5.so
#11 0x40c0635c in zend_call_method () from /usr/libexec/apache/libphp5.so
#12 0x40c08e74 in zend_objects_destroy_object () from
/usr/libexec/apache/libphp5.so
#13 0x40c0b0ec in zend_objects_store_call_destructors () from
/usr/libexec/apache/libphp5.so
#14 0x40bed361 in shutdown_executor () from /usr/libexec/apache/libphp5.so
#15 0x40bf77ee in zend_deactivate () from /usr/libexec/apache/libphp5.so
#16 0x40bbc9a2 in php_request_shutdown () from /usr/libexec/apache/libphp5.so
#17 0x40c2243f in apache_php_module_main () from /usr/libexec/apache/libphp5.so
#18 0x40c23269 in apache_php_module_main () from /usr/libexec/apache/libphp5.so
#19 0x40c2270d in apache_php_module_main () from /usr/libexec/apache/libphp5.so
#20 0x08054a17 in ap_invoke_handler ()
#21 0x08069f69 in ap_some_auth_required ()
#22 0x08069fc8 in ap_process_request ()
#23 0x08060f24 in ap_child_terminate ()
#24 0x080610f2 in ap_child_terminate ()
#25 0x08061258 in ap_child_terminate ()
#26 0x08061918 in ap_child_terminate ()
#27 0x08062150 in main ()
=========
Program received signal SIGSEGV, Segmentation fault.
0x40bfeef3 in zend_hash_find () from /usr/libexec/apache/libphp5.so
(gdb) bt
#0 0x40bfeef3 in zend_hash_find () from /usr/libexec/apache/libphp5.so
#1 0x40c21cf1 in zend_init_opcodes_handlers () from
/usr/libexec/apache/libphp5.so
#2 0x084e5e3c in ?? ()
#3 0x0845b064 in ?? ()
#4 0x00000003 in ?? ()
#5 0xbfffc128 in ?? ()
#6 0x4025e60c in ?? () from /lib/libc.so.6
#7 0x4025f0a0 in __after_morecore_hook () from /lib/libc.so.6
#8 0xbfffc108 in ?? ()
#9 0x40c9bed0 in ?? () from /usr/libexec/apache/libphp5.so
#10 0x084e5e3c in ?? ()
#11 0x00000000 in ?? ()
#12 0xbfffc138 in ?? ()
#13 0x40c21a7f in zend_init_opcodes_handlers () from
/usr/libexec/apache/libphp5.so
#14 0x084e5e3c in ?? ()
#15 0x0845b064 in ?? ()
#16 0x00000003 in ?? ()
#17 0xbfffc128 in ?? ()
#18 0x00000324 in ?? ()
#19 0x00000044 in ?? ()
#20 0x03ffc138 in ?? ()
#21 0x00000000 in ?? ()
#22 0x00000000 in ?? ()
#23 0x00000318 in ?? ()
#24 0x00000000 in ?? ()
#25 0x40c9bed0 in ?? () from /usr/libexec/apache/libphp5.so
#26 0x082b18c0 in ?? ()
#27 0xbfffc190 in ?? ()
#28 0xbfffc158 in ?? ()
#29 0x40c1cfd7 in zend_isset_isempty_dim_obj_handler () from
/usr/libexec/apache/libphp5.so
Previous frame inner to this frame (corrupt stack?)
We also tried to compile with "-O2 -march=i486 -mcpu=i686" (
previously: "-O3 -march=i486 -mcpu=i686" ) and the first thing that
appeard was:
=========
Program received signal SIGSEGV, Segmentation fault.
0x40c85255 in zend_hash_destroy () from /usr/libexec/apache/libphp5.so
(gdb) bt
#0 0x40c85255 in zend_hash_destroy () from /usr/libexec/apache/libphp5.so
#1 0x40cf66e0 in php_tiff_bytes_per_format () from
/usr/libexec/apache/libphp5.so
#2 0x0000020c in ?? ()
#3 0x00000000 in ?? ()
#4 0x00000000 in ?? ()
#5 0xbfffdfe0 in ?? ()
#6 0x40cf66e0 in php_tiff_bytes_per_format () from
/usr/libexec/apache/libphp5.so
#7 0x40d32970 in ?? () from /usr/libexec/apache/libphp5.so
#8 0x40d4f49c in executor_globals () from /usr/libexec/apache/libphp5.so
#9 0xbfffe07c in ?? ()
#10 0xbfffe318 in ?? ()
#11 0x40c704fc in shutdown_executor () from /usr/libexec/apache/libphp5.so
#12 0x00000000 in ?? ()
#13 0x0000010d in ?? ()
#14 0x00000000 in ?? ()
#15 0x00000000 in ?? ()
#16 0xbfffe080 in ?? ()
#17 0x00000026 in ?? ()
#18 0x00000000 in ?? ()
#19 0x00000000 in ?? ()
#20 0x00000000 in ?? ()
#21 0x01000000 in ?? ()
#22 0x40d4f49c in executor_globals () from /usr/libexec/apache/libphp5.so
#23 0x40d4f2a0 in zend_compile_file () from /usr/libexec/apache/libphp5.so
#24 0x01ffdfd8 in ?? ()
#25 0x401852f4 in snprintf () from /lib/libc.so.6
Previous frame inner to this frame (corrupt stack?)
So: we are having some problems with this right now, hope I have given
enough information about the system/settings ( I have more backtraces
if someone wants that ).
Can someone enlighten us on what is going on here? Anyone that has
experienced something like this before?
Cheers
Håvard Eide
If the segfault was the same everytime it should be easy to track
down, but I've had a large amount of different segfaults running
apache through gdb. The other thing is that one screen can be loaded
different times before the segfault comes as well, sometimes the
segfault comes straight away, other times after 5-6 reloads of the
same screen.
This tends to be the sort of problem where Valgrind can be very useful.
It sounds like memory corruption somewhere.
Run it like this:
valgrind --leak-check=yes --show-reachable=yes --num-callers=16 --error-limit=no --tool=memcheck --logfile=out /usr/bin/apache -X
Then hit the server and have a look at the out.<pid> file.
-Rasmus
Trying again with a link to the file:
Just ran it through valgrind with the options that Rasmus suggested
and got a output file: http://www.ikomm.no/valgrind/out.pid27848 I
can see what'
happening, but have no idea where to go from here.
Can someone have a look at it?
Cheers
If the segfault was the same everytime it should be easy to track
down, but I've had a large amount of different segfaults running
apache through gdb. The other thing is that one screen can be loaded
different times before the segfault comes as well, sometimes the
segfault comes straight away, other times after 5-6 reloads of the
same screen.This tends to be the sort of problem where Valgrind can be very useful.
It sounds like memory corruption somewhere.Run it like this:
valgrind --leak-check=yes --show-reachable=yes --num-callers=16 --error-limit=no --tool=memcheck --logfile=out /usr/bin/apache -X
Then hit the server and have a look at the out.<pid> file.
-Rasmus