Hi all
In this example (using php-5.4.11 on Linux) the memory will grow non-stop:
for ( $fp = fopen('/dev/urandom', 'rb'); true;) {
eval ('$ano_fnc = function() {$x = "'.bin2hex(fread($fp, mt_rand(1,
10000))).'";};');
echo "Mem usage: ".memory_get_usage()."\n";
}
But in this example not:
for ( $fp = fopen('/dev/urandom', 'rb'); true;) {
eval ('$ano_fnc = function() {$x = "'.bin2hex(fread($fp,
10000)).'";};');
echo "Mem usage: ".memory_get_usage()."\n";
}
The only different in the second example is the fixed body length of the
eval()-created anoymous function.
I wondering why the memory in the second code-example will be freed at
some point while in the first example not.
I don't think it's a PHP-Bug but i also not understand this behavior.
Is there one who can explain this?
Thank you very much.
Greetings
Hans-Juergen Petrich
The only different in the second example is the fixed body length of
the eval()-created anoymous function.
I wondering why the memory in the second code-example will be freed at
some point while in the first example not.I don't think it's a PHP-Bug but i also not understand this behavior.
Is there one who can explain this?
Thank you very much.Greetings
Hans-Juergen Petrich
It can be simplified to:
<?php
while (true) {
$value = mt_rand(1, 10000);
$a = '$ano_fnc = function() {' . str_repeat(" ", $value) . '};';
eval ($a);
echo "Mem usage: ".memory_get_usage()."\n";
}
The function is being removed, $ano_fnc alternates between
object(Closure)#1 and object(Closure)#2, it's not leaking function objects.
I suspect eval is checking a hashtable which is being forced to resize
continuously, but it shouldn't grow unbounded either.
Hi Terry and all
thank you very much for your response.
The only thing that confused me about what you say that the second doesn't grow
Yes, about that i was [and am still :-)] also confused... why the 2nd one won't grow non-stop
so I checked and it does -- just the same as the first.
Right, it grows, but not non-stop as in the 1st one.
The memory will stop growing (on my machine) at ~2491584 bytes and the loop is able to run forever,
creating each eval() furthermore uniqe ano-function's but not endless-filling Zend-internal tables.
but this still leaves the function record itself in the function_table hash so with a non-zero reference count and this doesn't get DTORed until request shutdown
Not familar with the Zend-internals but just about so i was imaging and expecting it.
That why i [still] also confused/wondering why in the 2nd example the memory will not grow endless.
It seems that the function records in the function_table will be DTORed (or similar cleaned up) before request-shutdown at some point...
Could this be the case?
Hans-Jürgen Petrich wrote:
Hi Terry and all
thank you very much for your response.The only thing that confused me about what you say that the second
doesn't grow
Yes, about that i was [and am still :-)] also confused... why the 2nd
one won't grow non-stopso I checked and it does -- just the same as the first.
Right, it grows, but not non-stop as in the 1st one.The memory will stop growing (on my machine) at ~2491584 bytes and the
loop is able to run forever,
creating each eval() furthermore uniqe ano-function's but not
endless-filling Zend-internal tables.but this still leaves the function record itself in the
function_table hash so with a non-zero reference count and this
doesn't get DTORed until request shutdown
Not familar with the Zend-internals but just about so i was imaging
and expecting it.That why i [still] also confused/wondering why in the 2nd example the
memory will not grow endless.
It seems that the function records in the function_table will be
DTORed (or similar cleaned up) before request-shutdown at some point...Could this be the case?
As you are reassigning $ano_fnc, the old closure is being destructed.
Had you used create_function(), it wouldn't happen.
Now the question is, if it is correctly freeing the functions (and it is
good that it does so), why is it not doing it when they have different
lengths?
Am 04.02.2013, 17:57 Uhr, schrieb "Ángel González" keisial@gmail.com:
Hans-Jürgen Petrich wrote:
Hi Terry and all
thank you very much for your response.The only thing that confused me about what you say that the second
doesn't grow
Yes, about that i was [and am still :-)] also confused... why the 2nd
one won't grow non-stopso I checked and it does -- just the same as the first.
Right, it grows, but not non-stop as in the 1st one.The memory will stop growing (on my machine) at ~2491584 bytes and the
loop is able to run forever,
creating each eval() furthermore uniqe ano-function's but not
endless-filling Zend-internal tables.but this still leaves the function record itself in the
function_table hash so with a non-zero reference count and this
doesn't get DTORed until request shutdown
Not familar with the Zend-internals but just about so i was imaging
and expecting it.That why i [still] also confused/wondering why in the 2nd example the
memory will not grow endless.
It seems that the function records in the function_table will be
DTORed (or similar cleaned up) before request-shutdown at some point...Could this be the case?
As you are reassigning $ano_fnc, the old closure is being destructed.
Had you usedcreate_function(), it wouldn't happen.
Now the question is, if it is correctly freeing the functions (and it is
good that it does so), why is it not doing it when they have different
lengths?
Now the question is, if it is correctly freeing the functions (and it is
good that it does so), why is it not doing it when they have different
lengths?
Yes, you bring it to the point. Thats my question.
Hoping of course that maybe it could be possible that
freeing the memory also on different length.
Thank you.
If you did the eval at line 20 in say /tmp/xx.php then the
INCLUDE_OR_EVAL instruction calls the Zend compiler with the args:
(1) the source to be compiled and
(2) the title "/tmp/x.php(20) : eval()'d code"
The compiler than gives the closure function a magic name:
"\0{closure}/tmp/x.php(20) : eval()'d code0xHHHHHHHH"
where 0xHHHHHHHH is the hex address of the "function" substring in the
evaluated string. The compiler uses a zend_hash_update to insert this
into the CG(function_table).
What happens if you use a fixed length string replacing another string
of the same length dropping its refcount to 0 is that the allocator is
clever and will tend to reallocate the old one and hence the address of
the string is the same and the address of the offset of the "function"
substring is the same so it regenerates the same magic name -- pretty
much as an accidental side-effect. When this happens, it's this hash
update function that calls the DTOR on any pre-existing function with
this name.
I simply put a breakpoint on the relevant line in the
zend_do_begin_function_declaration() code and if you used a fixed offset
into the same string you only got one {closure} entry. If the
allocation ended up "randomizing" the address, then the {closure}
entries grew until memory exhaustion.
As I said -- interesting. Need to think about the consequences before I
submit a bugrep.
Regards
Terry
Am 05.02.2013, 20:53 Uhr, schrieb Terry Ellison Terry@ellisons.org.uk:
On 04/02/13 10:57, Ángel González
wrote:
<snip>The memory will stop growing (on my machine) at ~2491584 bytes and theloop is able to run forever,
creating each eval() furthermore uniqe ano-function's but not
endless-filling Zend-internal tables.but this still leaves the function record itself in thefunction_table hash so with a non-zero reference count and this
doesn't get DTORed until request shutdownNot familar with the Zend-internals but just about so i was imagingand expecting it.
That why i [still] also confused/wondering why in the 2nd example the
memory will not grow endless.
It seems that the function records in the function_table will be
DTORed (or similar cleaned up) before request-shutdown at some point...Could this be the case?
As you are reassigning $ano_fnc, the old closure is being destructed.Had you used
create_function(), it wouldn't happen.
Now the question is, if it is correctly freeing the functions (and it is
good that it does so), why is it not doing it when they have different
lengths?It's a bug. The Closure class DTOR does not delete the derefenced
function from the CG(function_table).If you did the eval at line 20 in say /tmp/xx.php then the
INCLUDE_OR_EVAL instruction calls the Zend compiler with the args:(1) the source to be compiled and (2) the title "/tmp/x.php(20) : eval()'d code"The compiler than gives the closure function a magic name:
"\0{closure}/tmp/x.php(20) : eval()'d code0xHHHHHHHH"where 0xHHHHHHHH is the hex address of the "function" substring in
the evaluated string. The compiler uses a zend_hash_update to
insert this into the CG(function_table).What happens if you use a fixed length string replacing another
string of the same length dropping its refcount to 0 is that the
allocator is clever and will tend to reallocate the old one and
hence the address of the string is the same and the address of the
offset of the "function" substring is the same so it regenerates the
same magic name -- pretty much as an accidental side-effect. When
this happens, it's this hash update function that calls the DTOR on
any pre-existing function with this name.I simply put a breakpoint on the relevant line in the
zend_do_begin_function_declaration() code and if you used a fixed
offset into the same string you only got one {closure} entry. If
the allocation ended up "randomizing" the address, then the
{closure} entries grew until memory exhaustion.As I said -- interesting. Need to think about the consequences
before I submit a bugrep.Regards
Terry
Very interessting Terry. Would be great if the bug could be fixed.
If you post a bug report would you post the bug-report-url here also
that i (and/or other readers) can furthermore trace the discussion of this issue?
Thank you again
Greetings
Hi Terry and all
thank you very much for your response.The only thing that confused me about what you say that the second
doesn't grow
Yes, about that i was [and am still :-)] also confused... why the 2nd
one won't grow non-stopso I checked and it does -- just the same as the first.
Right, it grows, but not non-stop as in the 1st one.The memory will stop growing (on my machine) at ~2491584 bytes and the
loop is able to run forever,
creating each eval() furthermore uniqe ano-function's but not
endless-filling Zend-internal tables.but this still leaves the function record itself in the
function_table hash so with a non-zero reference count and this
doesn't get DTORed until request shutdown
Not familar with the Zend-internals but just about so i was imaging
and expecting it.That why i [still] also confused/wondering why in the 2nd example the
memory will not grow endless.
It seems that the function records in the function_table will be
DTORed (or similar cleaned up) before request-shutdown at some point...Could this be the case?
OK, Hans-Jürgen, this one has got me interested. I am developing a fork
of APC optimized for cgi and cli use -- but that's a different topic --
though understanding the DTOR processes for compiler objects interests
me because of this. I'll go through the code and especially for Closure
objects to understand why. However thinking through this logically:
-
The fact that the second does stop growing means that that the
reassignment of the global $ano sets the RC of the previous closure
object to zero triggering DTOR of the lamba function. -
There is something pathological about the first case which is
frustrating garbage collection on the lambda function DTOR.
I replaced your inner loop by:
$len = $argv[1] & 1 ? $argv[2] : mt_rand(1, $argv[2]);
$str = "'." . bin2hex(fread($fp, $len)) . "'";
if ($argv[1] & 2) $str = "function() {\$y = $str; };";
eval ("\$x = $str;");
echo "Mem usage: ".memory_get_usage()."\n";
to allow me to use arg1 to select one of the four test cases 0..3 and
arg2 is the (max) string size, n say. This clearly shows the fact that
the memory explosion only occurs if the string is allocated inside the
lambda function.
-
If you substitute n<15 then memory growth rapidly stabilises for PHP
5.3.17 0, but still explodes for n>14 -
In the case of PHP 5.4.6 a similar effect occurs except that
explosion occurs at n>11. -
The fact that 5.3 and 5.4 are different is notable -- however, the
fact that 5.4 is still (eventually) stable for n<12 means that this
isn't a string interning issue.
Interesting. Merits more research :-)
Am 04.02.2013, 19:11 Uhr, schrieb Terry Ellison Terry@ellisons.org.uk:
Hi Terry and all
thank you very much for your response.The only thing that confused me about what you say that the second
doesn't grow
Yes, about that i was [and am still :-)] also confused... why the 2nd
one won't grow non-stopso I checked and it does -- just the same as the first.
Right, it grows, but not non-stop as in the 1st one.The memory will stop growing (on my machine) at ~2491584 bytes and the
loop is able to run forever,
creating each eval() furthermore uniqe ano-function's but not
endless-filling Zend-internal tables.but this still leaves the function record itself in the
function_table hash so with a non-zero reference count and this
doesn't get DTORed until request shutdown
Not familar with the Zend-internals but just about so i was imaging
and expecting it.That why i [still] also confused/wondering why in the 2nd example the
memory will not grow endless.
It seems that the function records in the function_table will be
DTORed (or similar cleaned up) before request-shutdown at some point...Could this be the case?
OK, Hans-Jürgen, this one has got me interested. I am developing a fork
of APC optimized for cgi and cli use -- but that's a different topic --
though understanding the DTOR processes for compiler objects interests
me because of this. I'll go through the code and especially for Closure
objects to understand why. However thinking through this logically:
The fact that the second does stop growing means that that the
reassignment of the global $ano sets the RC of the previous closure
object to zero triggering DTOR of the lamba function.There is something pathological about the first case which is
frustrating garbage collection on the lambda function DTOR.I replaced your inner loop by:
$len = $argv[1] & 1 ? $argv[2] : mt_rand(1, $argv[2]); $str = "'." . bin2hex(fread($fp, $len)) . "'"; if ($argv[1] & 2) $str = "function() {\$y = $str; };"; eval ("\$x = $str;"); echo "Mem usage: ".memory_get_usage()."\n";to allow me to use arg1 to select one of the four test cases 0..3 and
arg2 is the (max) string size, n say. This clearly shows the fact that
the memory explosion only occurs if the string is allocated inside the
lambda function.
If you substitute n<15 then memory growth rapidly stabilises for PHP
5.3.17 0, but still explodes for n>14In the case of PHP 5.4.6 a similar effect occurs except that
explosion occurs at n>11.The fact that 5.3 and 5.4 are different is notable -- however, the
fact that 5.4 is still (eventually) stable for n<12 means that this
isn't a string interning issue.Interesting. Merits more research :-)
Hi Terry
i made similar tests with similar results.
Currently, in my script, i'm just padding the function-body with additional spaces enough to ensure having always a fixed length to avoid endless-memory growing.
But with non-good feeling because i'm still not sure what is going on internaly... and my knowledge about the php/zend internals is poor :-)
So i'm very glad getting your interest in this issue.
Logical thinking... since the memory-growing depends "only" on variable length of the function-body (and not of the content it self) i could imaging that
it is possible to get the php/zend internals to also with variable lengths of the function-body... seems not far away.
But as i said... my knowledge ends at this point but i'm very interessted about your results, if you research it further more.
Thank you again
Greetings
Am 03.02.2013 16:27, schrieb Hans-Juergen Petrich:
Is there one who can explain this?
Can you explain why you are using eval() instead of a real anonymous
function? Thanks!
--
Sebastian Bergmann Co-Founder and Principal Consultant
http://sebastian-bergmann.de/ http://thePHP.cc/
Am 03.02.2013 18:07, schrieb Sebastian Bergmann:
Can you explain why you are using eval() instead of a real anonymous
function? Thanks!
Please ignore my email; I mixed up create_function() and eval(). Then
again, your code still does not make sense to me ...
--
Sebastian Bergmann Co-Founder and Principal Consultant
http://sebastian-bergmann.de/ http://thePHP.cc/
2013/2/3 Sebastian Bergmann sebastian@php.net
Am 03.02.2013 18:07, schrieb Sebastian Bergmann:
Can you explain why you are using eval() instead of a real anonymous
function? Thanks!Please ignore my email; I mixed up
create_function()and eval(). Then
again, your code still does not make sense to me ...
Don't know, how create_function() works internally, but it seems, that they
at least act quite similar
<?php
create_function('','}echo "Foo";{'); // "Foo"
http://codepad.viper-7.com/h6NQfO
--
Sebastian Bergmann Co-Founder and Principal Consultant
http://sebastian-bergmann.de/ http://thePHP.cc/--
Hello
Then again, your code still does not make sense to me ...
Fully understand :-)
Not want wasting your time with my specific situation... but there are
situation where it make sense :-)
For example... in my situation, i have time intensive (mathematically)
operations to do in pure php
and was searching a way to speed this operations up.
With PHP's eval() feature, creating php code at run time, this would be,
in my situation, a possible way.
Very simplified, i have to deal with the following:
function calculate($ints)
{
$result = 0;
for ($i=0; $i<10000000; ++$i) {
$result += $ints[0] + $ints[1] + $ints[2] + $ints[3] + $ints[4] +
$ints[5];
}
return $result;
}
$ints = array(1,2,3,4,5,6);
$result = calculate($ints);
A faster way would be:
$ints = array(1,2,3,4,5,6);
eval('$calculate = function () {
$result = 0;
for ($i=0; $i<10000000; ++$i) {
$result += '.$ints[0].' + '.$ints[1].' + '.$ints[2].' + '.$ints[3].' +
'.$ints[4].' + '.$ints[5].';
}
return $result;
};');
$result = $calculate();
The function code it self is in reality more complex and variable in the
function-body length, but the princip is the same.
The code-architecture, in my specific situation, requires also that the
operation encapsulated in a function.
(And for security: It is ensured that $int is an 1-dim array filled only
with integers)
$calculate() works very fine and, of course, much faster than the
hardcoded calculate().
Each time $ints is changing i also have, of course, re-define/creater
$calculate() via eval.
In such situations, for performance reasons, it would makes sense using
eval() for re/creating ano-functions, doesn't it?
And as long as the function-body has a fixed length each time re/creating
the ano-function via eval() it would also not let grow the memory endless.
In this example (using php-5.4.11 on Linux) the memory will grow
non-stop:for ( $fp = fopen('/dev/urandom', 'rb'); true;) {
eval ('$ano_fnc = function() {$x = "'.bin2hex(fread($fp,
mt_rand(1, 10000))).'";};');
echo "Mem usage: ".memory_get_usage()."\n";
}But in this example not:
for ( $fp = fopen('/dev/urandom', 'rb'); true;) {
eval ('$ano_fnc = function() {$x = "'.bin2hex(fread($fp,
10000)).'";};');
echo "Mem usage: ".memory_get_usage()."\n";
}
Hans-Juergen, I've raised a bugrep https://bugs.php.net/bug.php?id=64291
which you might want to review and add any appropriate comments. I had
to think about this one. It's worthwhile observing that this second
example is the only occasion, as far as I know, that PHP does any
garbage collection of code objects before request shutdown. For example
create_function() objects are given the name "\0LambdaN" where N is the
count of the number of created functions so far in this request. They
are registered in the function table and persist until request
shutdown. That's the way PHP handles them by design.
As I said in the bugrep, the normal behaviour of persistence is what you
want because if you think about the normal use of the anonymous
function, say
while (!feof($fp)) {
$line = preg_replace_callback(
'|<p>\s*\w|',
function($matches) {return strtolower($matches[0]);},
gets($fp)
);
echo $line;
}
Then the anonymous function is compiled once and rebound to the closure
object which is passed as the
second argument for the callback each time through the loop. OK, doing
the closure CTOR/DTOR once per loop. is not the cleverest of ideas and
this is the sort of thing that would be hoisted out of the loop in a
language which did such optimization, but PHP doesn't. It's a LOT better
that compiling a new function each loop (which is how Example #1 on
http://php.net/manual/en/function.preg-replace-callback.php does it!)
This is what you want to happen.
It's just too complicated for PHP to work out if the function might or
not be rebound to. I suspect the bug here is really the implicit
assumption that the magic function name generated by the
eval ('$ano_fnc = function() { ... }');
is unique, but as your examples shows, thanks to garbage collection and
reuse of memory, sometimes it isn't. In these circumstances thank to
the use of a hash update and the table DTOR the old one is deleted.
So assume that what you are doing is exploiting a bug, so my advice is
not to do this. It might be fixed in a future release.
Regards Terry
>
> On 03/02/13 15:27, Hans-Juergen Petrich
> wrote:
>
>
>
>> In this
>> example (using php-5.4.11 on Linux) the memory will grow non-stop:
>>
>>
>> for ( $fp = fopen('/dev/urandom', 'rb'); true;) {
>>
>> eval ('$ano_fnc = function() {$x = "'.bin2hex(fread($fp,
>> mt_rand(1, 10000))).'";};');
>>
>> echo "Mem usage: ".memory_get_usage()."\n";
>>
>> }
>>
>>
>>
>> But in this example not:
>>
>>
>> for ( $fp = fopen('/dev/urandom', 'rb'); true;) {
>>
>> eval ('$ano_fnc = function() {$x = "'.bin2hex(fread($fp,
>> 10000)).'";};');
>>
>> echo "Mem usage: ".memory_get_usage()."\n";
>>
>> }
>>
>>
>>
>
> Hans-Juergen, I've raised a bugrep
> https://bugs.php.net/bug.php?id=64291 which you might want to review
> and add any appropriate comments. I had to think about this one. It's worthwhile observing that this second example is the only
> occasion, as far as I know, that PHP does any garbage collection of
> code objects before request shutdown. For example `create_function()`
> objects are given the name "\0LambdaN" where N is the count of the
> number of created functions so far in this request. They are
> registered in the function table and persist until request
> shutdown. That's the way PHP handles them by design.
>
>
> As I said in the bugrep, the normal behaviour of persistence is what
> you want because if you think about the normal use of the anonymous
> function, say
>
>
> while (!feof($fp)) {
>
> $line = preg_replace_callback(
>
> '|
\s*\w|',
>
> function($matches) {return strtolower($matches[0]);},
>
> gets($fp)
>
> );
>
> echo $line;
>
> }
>
>
> Then the anonymous function is compiled once and rebound to the
> closure object which is passed as the
> second argument for the callback each time through the loop. OK,
> doing the closure CTOR/DTOR once per loop. is not the cleverest of
> ideas and this is the sort of thing that would be hoisted out of the
> loop in a language which did such optimization, but PHP doesn't. It's a LOT better that compiling a new function each loop (which is
> how Example #1 on
> http://php.net/manual/en/function.preg-replace-callback.php does
> it!) This is what you want to happen.
>
> It's just too complicated for PHP to work out if the function might
> or not be rebound to. I suspect the bug here is really the implicit
> assumption that the magic function name generated by the
>
> eval ('$ano_fnc = function() { ... }');
>
>
> is unique, but as your examples shows, thanks to garbage collection
> and reuse of memory, sometimes it isn't. In these circumstances
> thank to the use of a hash update and the table DTOR the old one is
> deleted.
>
>
> So assume that what you are doing is exploiting a bug, so my advice
> is not to do this. It might be fixed in a future release.
>
>
> Regards Terry
>
>
Thank you Terry for this information. Yes, i avoid using it. I'll use `create_function()` for a limited count... which will do the work also.
Again thank you.
Greetings
Hans-Jürgen