(non)growing memory while creating anoymous functions via eval()

Am 04.02.2013, 17:57 Uhr, schrieb "Ángel González" keisial@gmail.com:

Hans-Jürgen Petrich wrote:

Hi Terry and all
thank you very much for your response.

The only thing that confused me about what you say that the second
doesn't grow
Yes, about that i was [and am still :-)] also confused... why the 2nd
one won't grow non-stop

so I checked and it does -- just the same as the first.
Right, it grows, but not non-stop as in the 1st one.

The memory will stop growing (on my machine) at ~2491584 bytes and the
loop is able to run forever,
creating each eval() furthermore uniqe ano-function's but not
endless-filling Zend-internal tables.

but this still leaves the function record itself in the
function_table hash so with a non-zero reference count and this
doesn't get DTORed until request shutdown
Not familar with the Zend-internals but just about so i was imaging
and expecting it.

That why i [still] also confused/wondering why in the 2nd example the
memory will not grow endless.
It seems that the function records in the function_table will be
DTORed (or similar cleaned up) before request-shutdown at some point...

Could this be the case?

As you are reassigning $ano_fnc, the old closure is being destructed.
Had you used create_function(), it wouldn't happen.
Now the question is, if it is correctly freeing the functions (and it is
good that it does so), why is it not doing it when they have different
lengths?

Now the question is, if it is correctly freeing the functions (and it is
good that it does so), why is it not doing it when they have different
lengths?

Yes, you bring it to the point. Thats my question.
Hoping of course that maybe it could be possible that
freeing the memory also on different length.

Thank you.

<snip> >> The memory will stop growing (on my machine) at ~2491584 bytes and the >> loop is able to run forever, >> creating each eval() furthermore uniqe ano-function's but not >> endless-filling Zend-internal tables. >> >>> but this still leaves the function record itself in the >>> function_table hash so with a non-zero reference count and this >>> doesn't get DTORed until request shutdown >> Not familar with the Zend-internals but just about so i was imaging >> and expecting it. >> >> That why i [still] also confused/wondering why in the 2nd example the >> memory will not grow *endless*. >> It seems that the function records in the function_table will be >> DTORed (or similar cleaned up) before request-shutdown at some point... >> >> Could this be the case? > As you are reassigning $ano_fnc, the old closure is being destructed. > Had you used `create_function()`, it wouldn't happen. > Now the question is, if it is correctly freeing the functions (and it is > good that it does so), why is it not doing it when they have different > lengths? It's a bug. The Closure class DTOR does not delete the derefenced function from the CG(function_table).

If you did the eval at line 20 in say /tmp/xx.php then the
INCLUDE_OR_EVAL instruction calls the Zend compiler with the args:
(1) the source to be compiled and
(2) the title "/tmp/x.php(20) : eval()'d code"

The compiler than gives the closure function a magic name:

 "\0{closure}/tmp/x.php(20) : eval()'d code0xHHHHHHHH"

where 0xHHHHHHHH is the hex address of the "function" substring in the
evaluated string. The compiler uses a zend_hash_update to insert this
into the CG(function_table).

What happens if you use a fixed length string replacing another string
of the same length dropping its refcount to 0 is that the allocator is
clever and will tend to reallocate the old one and hence the address of
the string is the same and the address of the offset of the "function"
substring is the same so it regenerates the same magic name -- pretty
much as an accidental side-effect. When this happens, it's this hash
update function that calls the DTOR on any pre-existing function with
this name.

I simply put a breakpoint on the relevant line in the
zend_do_begin_function_declaration() code and if you used a fixed offset
into the same string you only got one {closure} entry. If the
allocation ended up "randomizing" the address, then the {closure}
entries grew until memory exhaustion.

As I said -- interesting. Need to think about the consequences before I
submit a bugrep.
Regards
Terry

Am 04.02.2013, 19:11 Uhr, schrieb Terry Ellison Terry@ellisons.org.uk:

Hi Terry and all
thank you very much for your response.

The only thing that confused me about what you say that the second
doesn't grow
Yes, about that i was [and am still :-)] also confused... why the 2nd
one won't grow non-stop

so I checked and it does -- just the same as the first.
Right, it grows, but not non-stop as in the 1st one.

The memory will stop growing (on my machine) at ~2491584 bytes and the
loop is able to run forever,
creating each eval() furthermore uniqe ano-function's but not
endless-filling Zend-internal tables.

but this still leaves the function record itself in the
function_table hash so with a non-zero reference count and this
doesn't get DTORed until request shutdown
Not familar with the Zend-internals but just about so i was imaging
and expecting it.

That why i [still] also confused/wondering why in the 2nd example the
memory will not grow endless.
It seems that the function records in the function_table will be
DTORed (or similar cleaned up) before request-shutdown at some point...

Could this be the case?

OK, Hans-Jürgen, this one has got me interested. I am developing a fork
of APC optimized for cgi and cli use -- but that's a different topic --
though understanding the DTOR processes for compiler objects interests
me because of this. I'll go through the code and especially for Closure
objects to understand why. However thinking through this logically:

1. The fact that the second does stop growing means that that the
   reassignment of the global $ano sets the RC of the previous closure
   object to zero triggering DTOR of the lamba function.

2. There is something pathological about the first case which is
   frustrating garbage collection on the lambda function DTOR.

I replaced your inner loop by:

     $len = $argv[1] & 1 ? $argv[2] : mt_rand(1, $argv[2]);
     $str = "'." . bin2hex(fread($fp, $len)) . "'";
     if ($argv[1] & 2) $str = "function() {\$y = $str; };";
     eval ("\$x = $str;");
     echo "Mem usage: ".memory_get_usage()."\n";

to allow me to use arg1 to select one of the four test cases 0..3 and
arg2 is the (max) string size, n say. This clearly shows the fact that
the memory explosion only occurs if the string is allocated inside the
lambda function.

4. If you substitute n<15 then memory growth rapidly stabilises for PHP
   5.3.17 0, but still explodes for n>14

5. In the case of PHP 5.4.6 a similar effect occurs except that
   explosion occurs at n>11.

6. The fact that 5.3 and 5.4 are different is notable -- however, the
   fact that 5.4 is still (eventually) stable for n<12 means that this
   isn't a string interning issue.

Interesting. Merits more research :-)

Hi Terry
i made similar tests with similar results.

Currently, in my script, i'm just padding the function-body with additional spaces enough to ensure having always a fixed length to avoid endless-memory growing.
But with non-good feeling because i'm still not sure what is going on internaly... and my knowledge about the php/zend internals is poor :-)

So i'm very glad getting your interest in this issue.

Logical thinking... since the memory-growing depends "only" on variable length of the function-body (and not of the content it self) i could imaging that
it is possible to get the php/zend internals to also with variable lengths of the function-body... seems not far away.

But as i said... my knowledge ends at this point but i'm very interessted about your results, if you research it further more.

Thank you again
Greetings