thank you so much all of you guys, but still we are facing the same problem.
we recompiled new php source, upgraded httpd to 2.2.11 but still getting same
backtrace. nothing more. we removed .htaccess rules and tested but no use.
can you guys suggest me any other ways of debugging pls? actually i am not a
system administration guy. I am a php developer with little knowledge on httpd.
But we have system administration group who are working on this problem.
just pls check out again the segfault statement that we are getting:
httpd[5250]: segfault at 00007fff84114ff8 rip 00002aaab1bd47ac rsp 00007fff84115000 error
6
that bold line is repeating many times randomly on our log file and always it is same on
specific server. we have 6 load balanced servers and all are giving the same error randomly.
address locations are different from each server but for a specific server it is same always
and every day.
you guys think any kind of attack thats happening over our systems? actually we do not accept
much input from users other than a keyword(for search) which can be an unicode and we have some
POST requests which are purely browser generated data(no user data here).
im again pasting our gdb backtraces of two different core dumps for reference:
#0 0x000055555557dfee in ap_merge_per_dir_configs () from /usr/sbin/httpd
No symbol table info available.
#1 0x000055555557b121 in ap_directory_walk () from /usr/sbin/httpd
No symbol table info
available.
#2 0x00005555555765b9 in ap_is_recursion_limit_exceeded () from /usr/sbin/httpd
No symbol table info available.
#3 0x0000555555578b42 in ap_run_map_to_storage () from /usr/sbin/httpd
No symbol table info available.
#4 0x0000555555579cbc in ap_process_request_internal () from /usr/sbin/httpd
No symbol table info available.
#5 0x000055555558b668 in ap_process_request () from /usr/sbin/httpd
No symbol table info available.
#6 0x0000555555588900 in ap_register_input_filter () from /usr/sbin/httpd
No symbol table info available.
above thing is happening many times say 10 to 15 times a minute(actualy
depends on the traffic at that time)
#0 0x00002aaab1c46688 in ZEND_FETCH_DIM_RW_SPEC_VAR_UNUSED_HANDLER (
execute_data=0x5555714ea6c8)
at
/usr/local/src/php/php-5.2.9/Zend/zend_vm_execute.h:13204
opline = (zend_op *) 0x5555714e8798
free_op1 = {var = 0x2aaaac1454fd}
#1 0x00005555714e86b8 in ?? ()
No symbol table info available.
#2 0x00002aaaac145afb in apr_pool_destroy () from /usr/lib64/libapr-1.so.0
No symbol table info available.
#3 0x000055555556a27b in suck_in_APR () from /usr/sbin/httpd
No
symbol table info available.
#4 0x000055555556adf6 in main () from /usr/sbin/httpd
No symbol table info available.
above thing is happening in very lesser frequency compared to the above.
--thank you so much guys, rats.
jvlad wrote:
#2 0x00002aaaac145afb in apr_pool_destroy () from
/usr/lib64/libapr-1.so.0
No symbol table info available.
See http://httpd.apache.org/dev/debugging.html on how to grab the
-win32-symbols.zip package. Unpack it over httpd and you will probably
have more legible backtraces. Ensure you have all the debugging symbols
for php you'll have something completely legible.aha, and /usr/local/src/php/php-5.2.9/Zend/zend_vm_execute.h is a very
common path under Win32 :)
Here? Yea, it is.
But looking at the libapr-1.so.0 module, agreed that
I've misread this :)
@Rathnakar Konda, is it possible that php was compiled against wrong Apache
headers? Did you install something like OS-supplied httpd-devel package to
compile php or grabbed apache sources from their web site?
I'm not used to seeing bad stack unwinds on unix, most people don't optimize
that aggressively.
If this is system-httpd, that's fine; if you installed an httpd package,
there is likely an httpd-devel package that also includes the debugging
symbols you need (unstripped).
If not and you've built this, you can simply try -O0, or -g, or for httpd
--enable-maintainer-mode, but offhand I can't think of a trivial way to
simply avoid -strip, which is what you want for your build when things
go wonky in your gdb where output.
Bill
Original E-Mail:
Hi Guys,
We are have a problem with apache segfault on our production server. Please read bellow for description.
Its
a web application written in php5 and implemented most of the oop
concepts and lot of regular expressions, curl, mcrypt, simplexml,
mssql, exceptions and user defined error handlers. When we run this app
on our test server, we had no problems, but when we moved it on to the
production(here we used have big amount of traffic), initially we saw
no problems from our end user testing but from system log, we saw lots
of 'segfaults' and thus requests were being dropped(difference in
traffic).Weird thing is that, on the
same apache httpd, there
is an another application running successfully which is having lesser
oop concepts but with same libraries. We are running these two
applications with virtual host concept. We see 'segfaults' only when
the traffic is very high on the first application.We have upgraded our php module from 5.2.6 to 5.2.9 but with no result. We have the core dump of the apache below:
#0 0x000055555557dfee in ap_merge_per_dir_configs () from /usr/sbin/httpd
No symbol table info available.
#1 0x000055555557b121 in ap_directory_walk () from /usr/sbin/httpd
No symbol table info available.
#2 0x00005555555765b9 in ap_is_recursion_limit_exceeded () from /usr/sbin/httpd
No symbol table info available.
#3 0x0000555555578b42 in ap_run_map_to_storage () from /usr/sbin/httpd
No symbol table info available.
#4 0x0000555555579cbc in ap_process_request_internal () from /usr/sbin/httpd
No symbol table info available.
#5 0x000055555558b668 in ap_process_request () from /usr/sbin/httpd
No symbol table info available.
#6 0x0000555555588900 in ap_register_input_filter () from /usr/sbin/httpd
No symbol table info available.
Add more friends to your messenger and enjoy! Invite them now.
Add more friends to your messenger and enjoy! Go to http://messenger.yahoo.com/invite/