problem with apache segfault

16 years ago by Rathnakar Konda — view source — reply

unread

Hi Guys,

We are have a problem with apache segfault on our production server. Please read bellow for description.

Its
a web application written in php5 and implemented most of the oop
concepts and lot of regular expressions, curl, mcrypt, simplexml,
mssql, exceptions and user defined error handlers. When we run this app
on our test server, we had no problems, but when we moved it on to the
production(here we used have big amount of traffic), initially we saw
no problems from our end user testing but from system log, we saw lots
of 'segfaults' and thus requests were being dropped(difference in
traffic).

Weird thing is that, on the same apache httpd, there
is an another application running successfully which is having lesser
oop concepts but with same libraries. We are running these two
applications with virtual host concept. We see 'segfaults' only when
the traffic is very high on the first application.

We have upgraded our php module from 5.2.6 to 5.2.9 but with no result. We have the core dump of the apache below:

#0 0x000055555557dfee in ap_merge_per_dir_configs () from /usr/sbin/httpd
No symbol table info available.
#1 0x000055555557b121 in ap_directory_walk () from /usr/sbin/httpd
No symbol table info available.
#2 0x00005555555765b9 in ap_is_recursion_limit_exceeded () from /usr/sbin/httpd
No symbol table info available.
#3 0x0000555555578b42 in ap_run_map_to_storage () from /usr/sbin/httpd
No symbol table info available.
#4 0x0000555555579cbc in ap_process_request_internal () from /usr/sbin/httpd
No symbol table info available.
#5 0x000055555558b668 in ap_process_request () from /usr/sbin/httpd
No symbol table info available.
#6 0x0000555555588900 in ap_register_input_filter () from /usr/sbin/httpd
No symbol table info available.
#7 0x0000555555584a92 in ap_run_process_connection () from /usr/sbin/httpd
No symbol table info available.
#8 0x000055555558f27b in ap_graceful_stop_signalled () from /usr/sbin/httpd
No symbol table info available.
#9 0x000055555558f50a in ap_graceful_stop_signalled () from /usr/sbin/httpd
No symbol table info available.
#10 0x000055555558fd7b in ap_mpm_run () from /usr/sbin/httpd
No symbol table info available.
#11 0x000055555556ade4 in main () from /usr/sbin/httpd
No symbol table info available.

We
have tried modifying most of the curl implementation but with no use.
Also now we have no clues of the origin of the bug. Any kind of help
regarding this is mostly appreciated.

THANK YOU
--rats

  Check out the all-new Messenger 9.0! Go to http://in.messenger.yahoo.com/

16 years ago by Rasmus Lerdorf — view source — reply

unread

Note that your backtrace doesn't touch PHP at all. It is entirely in
Apache land and it is a crash on a graceful shutdown.

-Rasmus

Rathnakar Konda wrote:

Hi Guys,

We are have a problem with apache segfault on our production server. Please read bellow for description.

Its
a web application written in php5 and implemented most of the oop
concepts and lot of regular expressions, curl, mcrypt, simplexml,
mssql, exceptions and user defined error handlers. When we run this app
on our test server, we had no problems, but when we moved it on to the
production(here we used have big amount of traffic), initially we saw
no problems from our end user testing but from system log, we saw lots
of 'segfaults' and thus requests were being dropped(difference in
traffic).

Weird thing is that, on the same apache httpd, there
is an another application running successfully which is having lesser
oop concepts but with same libraries. We are running these two
applications with virtual host concept. We see 'segfaults' only when
the traffic is very high on the first application.

We have upgraded our php module from 5.2.6 to 5.2.9 but with no result. We have the core dump of the apache below:

#0 0x000055555557dfee in ap_merge_per_dir_configs () from /usr/sbin/httpd
No symbol table info available.
#1 0x000055555557b121 in ap_directory_walk () from /usr/sbin/httpd
No symbol table info available.
#2 0x00005555555765b9 in ap_is_recursion_limit_exceeded () from /usr/sbin/httpd
No symbol table info available.
#3 0x0000555555578b42 in ap_run_map_to_storage () from /usr/sbin/httpd
No symbol table info available.
#4 0x0000555555579cbc in ap_process_request_internal () from /usr/sbin/httpd
No symbol table info available.
#5 0x000055555558b668 in ap_process_request () from /usr/sbin/httpd
No symbol table info available.
#6 0x0000555555588900 in ap_register_input_filter () from /usr/sbin/httpd
No symbol table info available.
#7 0x0000555555584a92 in ap_run_process_connection () from /usr/sbin/httpd
No symbol table info available.
#8 0x000055555558f27b in ap_graceful_stop_signalled () from /usr/sbin/httpd
No symbol table info available.
#9 0x000055555558f50a in ap_graceful_stop_signalled () from /usr/sbin/httpd
No symbol table info available.
#10 0x000055555558fd7b in ap_mpm_run () from /usr/sbin/httpd
No symbol table info available.
#11 0x000055555556ade4 in main () from /usr/sbin/httpd
No symbol table info available.

We
have tried modifying most of the curl implementation but with no use.
Also now we have no clues of the origin of the bug. Any kind of help
regarding this is mostly appreciated.

THANK YOU
--rats
  Check out the all-new Messenger 9.0! Go to http://in.messenger.yahoo.com/

16 years ago by Rathnakar Konda — view source — reply

unread

pleasure hearing from you Rasmus.

after observing lots of core dumps from apache, we got a different segfault and its back trace is given bellow

#0 0x00002aaab1c46688 in ZEND_FETCH_DIM_RW_SPEC_VAR_UNUSED_HANDLER (
execute_data=0x5555714ea6c8)
at /usr/local/src/php/php-5.2.9/Zend/zend_vm_execute.h:13204
opline = (zend_op *) 0x5555714e8798
free_op1 = {var = 0x2aaaac1454fd}
#1 0x00005555714e86b8 in ?? ()
No symbol table info available.
#2 0x00002aaaac145afb in apr_pool_destroy () from /usr/lib64/libapr-1.so.0
No symbol table info available.
#3 0x000055555556a27b in suck_in_APR () from /usr/sbin/httpd
No symbol table info available.
#4 0x000055555556adf6 in main () from /usr/sbin/httpd
No symbol table info available.

we are still observing core dumps for any other issues.

--rats

16 years ago by William A. Rowe — view source — reply

unread

Rathnakar Konda wrote:

after observing lots of core dumps from apache, we got a different segfault and its back trace is given bellow

#0 0x00002aaab1c46688 in ZEND_FETCH_DIM_RW_SPEC_VAR_UNUSED_HANDLER (
execute_data=0x5555714ea6c8)
at /usr/local/src/php/php-5.2.9/Zend/zend_vm_execute.h:13204
opline = (zend_op *) 0x5555714e8798
free_op1 = {var = 0x2aaaac1454fd}

you appear to have .pdb symbols for your php. Now you need them for
apache...

#2 0x00002aaaac145afb in apr_pool_destroy () from /usr/lib64/libapr-1.so.0
No symbol table info available.

See http://httpd.apache.org/dev/debugging.html on how to grab the
-win32-symbols.zip package. Unpack it over httpd and you will probably
have more legible backtraces. Ensure you have all the debugging symbols
for php you'll have something completely legible.

16 years ago by jvlad — view source — reply

unread

""William A. Rowe, Jr."" wrowe@rowe-clan.net wrote in message
news:49CB613B.4070004@rowe-clan.net...

Rathnakar Konda wrote:

after observing lots of core dumps from apache, we got a different
segfault and its back trace is given bellow

#0 0x00002aaab1c46688 in ZEND_FETCH_DIM_RW_SPEC_VAR_UNUSED_HANDLER (
execute_data=0x5555714ea6c8)
at /usr/local/src/php/php-5.2.9/Zend/zend_vm_execute.h:13204
opline = (zend_op *) 0x5555714e8798
free_op1 = {var = 0x2aaaac1454fd}

you appear to have .pdb symbols for your php. Now you need them for
apache...

#2 0x00002aaaac145afb in apr_pool_destroy () from
/usr/lib64/libapr-1.so.0
No symbol table info available.

See http://httpd.apache.org/dev/debugging.html on how to grab the
-win32-symbols.zip package. Unpack it over httpd and you will probably
have more legible backtraces. Ensure you have all the debugging symbols
for php you'll have something completely legible.

aha, and /usr/local/src/php/php-5.2.9/Zend/zend_vm_execute.h is a very
common path under Win32 :)

@Rathnakar Konda, is it possible that php was compiled against wrong Apache
headers? Did you install something like OS-supplied httpd-devel package to
compile php or grabbed apache sources from their web site?

-jv

16 years ago by William A. Rowe — view source — reply

unread

jvlad wrote:

#2 0x00002aaaac145afb in apr_pool_destroy () from
/usr/lib64/libapr-1.so.0
No symbol table info available.
See http://httpd.apache.org/dev/debugging.html on how to grab the
-win32-symbols.zip package. Unpack it over httpd and you will probably
have more legible backtraces. Ensure you have all the debugging symbols
for php you'll have something completely legible.

aha, and /usr/local/src/php/php-5.2.9/Zend/zend_vm_execute.h is a very
common path under Win32 :)

Here? Yea, it is. But looking at the libapr-1.so.0 module, agreed that
I've misread this :)

@Rathnakar Konda, is it possible that php was compiled against wrong Apache
headers? Did you install something like OS-supplied httpd-devel package to
compile php or grabbed apache sources from their web site?

I'm not used to seeing bad stack unwinds on unix, most people don't optimize
that aggressively.

If this is system-httpd, that's fine; if you installed an httpd package,
there is likely an httpd-devel package that also includes the debugging
symbols you need (unstripped).

If not and you've built this, you can simply try -O0, or -g, or for httpd
--enable-maintainer-mode, but offhand I can't think of a trivial way to
simply avoid -strip, which is what you want for your build when things
go wonky in your gdb where output.

Bill