zend memory managment null pointer dereference

14 years ago by Chester — view source — reply

unread

Hi,

I have recently seen httpd core dump on my system. After looking into this issue it is the php module which is actually core dumping more specifically the zend memory management part of php.

#0 0x0115d224 in zend_mm_remove_from_free_list (heap=0x99596e8,
p=<value optimized out>)
at /scratchpad/workdirs/ssilva/build/RPM/BUILD/php-5.2.17/Zend/zend_alloc.c:837

The following line is failing in zend_mm_remove_from_fee_list which is in zend_alloc.c

#if ZEND_MM_SAFE_UNLINKING
if (UNEXPECTED(prev->next_free_block != mm_block) || UNEXPECTED(next->prev_free_block != mm_block)) {
zend_mm_panic("zend_mm_heap corrupted");
}
#endif

The core file tells me that next pointer is null so this is crashing when trying to dereference a null pointer. My first question is next ever expected to be null? After looking through the code, it seems that if there is only one element in the list that next & prev will be both equal to mm_block, is this a true assumption?

I am not sure what has gotten my system into this state the only hints I have is that there were a fair amount of objects created and an init 6 was sent and during the shut down of the php module this issue was seen.

I have done some investigation into this issue and came across a fairly long bug report https://bugs.php.net/bug.php?id=40479. However I am not sure if this is the same issue since I never see the "zend_mm_heap corrupted" message since the code fails due to the null pointer dereference before outputting this error message however this may be related in the fact that my heap is being corrupted.

Since I do not have a solid reproducer or know why my heap is in this state. I was wondering if I could get some information to help me debug this. Do you have any hunches on what could be setting next to null? What does it mean when next is null is this because the list is at the end or is this a dangling pointer and next should never be null? Do you have any hints on a possible reproducer using just php which would put me into the state where next is null? Also during my research on similar issues I have read that cache may have a role in causing the heap to be corrupted. How could the cache be corrupting the heap? Is that possible?

My current version of php is:
PHP 5.2.17 (cli) (built: Aug 2 2011 13:33:29)
Copyright (c) 1997-2010 The PHP Group
Zend Engine v2.2.0, Copyright (c) 1998-2010 Zend Technologies
with Xdebug v2.0.3, Copyright (c) 2002-2007, by Derick Rethans

I am running centos 5.6 and this issue has been seen on both 32 bit and 64 bit.

Also this issue has been seen only after upgrading from php 5.2.6 to 5.2.17. Do you know of a change made between these releases that could have an affect on the memory management?

Thread 1 (Thread 30656):
#0 0x0115d224 in zend_mm_remove_from_free_list (heap=0x99596e8,
p=<value optimized out>)
at /scratchpad/workdirs/ssilva/build/RPM/BUILD/php-5.2.17/Zend/zend_alloc.c:837
#1 _zend_mm_free_int (heap=0x99596e8, p=<value optimized out>)
at /scratchpad/workdirs/ssilva/build/RPM/BUILD/php-5.2.17/Zend/zend_alloc.c:1979
#2 0x0117965a in _zval_dtor_func (zvalue=0x9a2e300)
at /scratchpad/workdirs/ssilva/build/RPM/BUILD/php-5.2.17/Zend/zend_variables.c:43
#3 0x0116c388 in _zval_dtor (zval_ptr=0x9d60ae0)
at /scratchpad/workdirs/ssilva/build/RPM/BUILD/php-5.2.17/Zend/zend_variables.h:35
#4 _zval_ptr_dtor (zval_ptr=0x9d60ae0)
at /scratchpad/workdirs/ssilva/build/RPM/BUILD/php-5.2.17/Zend/zend_execute_API.c:414
#5 0x01184d77 in zend_hash_destroy (ht=0x9d5ecfc)
at /scratchpad/workdirs/ssilva/build/RPM/BUILD/php-5.2.17/Zend/zend_hash.c:526
#6 0x0117050d in destroy_zend_class (pce=0x9eaf7b4)
at /scratchpad/workdirs/ssilva/build/RPM/BUILD/php-5.2.17/Zend/zend_opcode.c:185
#7 0x01184d77 in zend_hash_destroy (ht=0x99599b0)
at /scratchpad/workdirs/ssilva/build/RPM/BUILD/php-5.2.17/Zend/zend_hash.c:526
#8 0x0117a09e in zend_shutdown ()
at /scratchpad/workdirs/ssilva/build/RPM/BUILD/php-5.2.17/Zend/zend.c:736
#9 0x01131f35 in php_module_shutdown ()
at /scratchpad/workdirs/ssilva/build/RPM/BUILD/php-5.2.17/main/main.c:1918
#10 0x01131ff7 in php_module_shutdown_wrapper (sapi_globals=0x12849e0)
at /scratchpad/workdirs/ssilva/build/RPM/BUILD/php-5.2.17/main/main.c:1889
#11 0x01202211 in php_apache_child_shutdown (tmp=0x0)
at /scratchpad/workdirs/ssilva/build/RPM/BUILD/php-5.2.17/sapi/apache2handler/sapi_apache2.c:369
#12 0x00cdbe8d in ?? () from /usr/lib/libapr-1.so.0
#13 0x00cdc63d in apr_pool_destroy () from /usr/lib/libapr-1.so.0
#14 0x003909e5 in ?? ()
#15 0x00390a0d in ?? ()
#16 <signal handler called>
#17 zend_hash_destroy (ht=0xa42b9a8)
at /scratchpad/workdirs/ssilva/build/RPM/BUILD/php-5.2.17/Zend/zend_hash.c:522

Thanks,
Alan

14 years ago by Antony Dovgal — view source — reply

unread

Hi,

I have recently seen httpd core dump on my system. After looking into this issue it is the php module which is actually core dumping more specifically the zend memory management part of php.

#0 0x0115d224 in zend_mm_remove_from_free_list (heap=0x99596e8,
p=<value optimized out>)
at /scratchpad/workdirs/ssilva/build/RPM/BUILD/php-5.2.17/Zend/zend_alloc.c:837

This is the worst kind of segfault since GDB backtrace doesn't carry any useful information in this case,
it just tells us that the crash has happened at the request shutdown.
But the real problem that caused the crash has occurred somewhere during the request and there's no way to know when & where basing on the backtrace only.

I'd suggest to try reproducing this problem with Valgrind: https://bugs.php.net/bugs-getting-valgrind-log.php, this would give us a lot more info.

--
Wbr,
Antony Dovgal

http://pinba.org - realtime profiling for PHP

14 years ago by Antony Dovgal — view source — reply

unread

Antony,

Thank you for the quick response. I will work towards finding a reproducer and using valgrind to gather more information.

Since I have been trying for to find a reproducer but have had no luck so far, do you think reducing the cache size
would help in finding a reproducer? If so how is this done? Or does cache/cache size have nothing to do with this area of code?

Reducing the cache size will not help for sure, but disabling the cache entirely will
help us to understand whether this problem is caused by your cache or not.

Btw, you didn't specify which kind of cache is that.

Also would you like to me open a bug ticket to track this issue?

It depends on the kind of cache you're using (there are several caches on the market
that aren't part of PHP project) and whether the cache is guilty or not.

--
Wbr,
Antony Dovgal

http://pinba.org - realtime profiling for PHP

14 years ago by Chester — view source — reply

unread

Antony,

I am fairly confident we are not using any third party caching (ie apc, eAccelerator,etc). However other then running php --version is there any other way to verify what caching system is being used?

[root@cab4en2b7 ~]# php --version
PHP 5.2.17 (cli) (built: Aug 2 2011 13:33:29)
Copyright (c) 1997-2010 The PHP Group
Zend Engine v2.2.0, Copyright (c) 1998-2010 Zend Technologies
with Xdebug v2.0.3, Copyright (c) 2002-2007, by Derick Rethans

Also trying to find a reproducer I have wrote a test php script which creates multiple objects and another script which sends signals to the script which is creating objects to try to get into the faulty cleanup. I have had little success with this and was wondering if this is not the best way to try to reproduce this or if there is a better way which may cause the memory to get corrupted? Also will this type of issue be seen with stand alone php or would I need this to be running with http?

Thanks,
Alan

-----Original Message-----
From: Antony Dovgal [mailto:tony@daylessday.org]
Sent: Monday, October 03, 2011 4:21 PM
To: Chester, Alan; php-dev
Subject: Re: [PHP-DEV] zend memory managment null pointer dereference

Antony,

Thank you for the quick response. I will work towards finding a reproducer and using valgrind to gather more information.

Since I have been trying for to find a reproducer but have had no luck so far, do you think reducing the cache size
would help in finding a reproducer? If so how is this done? Or does cache/cache size have nothing to do with this area of code?

Reducing the cache size will not help for sure, but disabling the cache entirely will
help us to understand whether this problem is caused by your cache or not.

Btw, you didn't specify which kind of cache is that.

Also would you like to me open a bug ticket to track this issue?

It depends on the kind of cache you're using (there are several caches on the market
that aren't part of PHP project) and whether the cache is guilty or not.

--
Wbr,
Antony Dovgal

http://pinba.org - realtime profiling for PHP

zend memory managment null pointer dereference

-- Wbr, Antony Dovgal

-- Wbr, Antony Dovgal

-- Wbr, Antony Dovgal

--
Wbr,
Antony Dovgal

--
Wbr,
Antony Dovgal

--
Wbr,
Antony Dovgal