# (non)growing memory while creating anoymous functions via eval() - Thread: https://externals.io/message/65572 - Original archive: https://news-web.php.net/php.internals/65572 - Started by Hans-Juergen Petrich on 2013-02-03T15:27:16+00:00 - Messages: 15 - Reply by email to internals@lists.php.net - More for agents: search with https://externals.io/search.md?q={query} · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt --- ## #65572 — Hans-Juergen Petrich — 2013-02-03T15:27:16+00:00 URL: https://externals.io/message/65572#65572 Hi all In this example (using php-5.4.11 on Linux) the memory will grow non-stop: for ( $fp = fopen('/dev/urandom', 'rb'); true;) { eval ('$ano_fnc = function() {$x = "'.bin2hex(fread($fp, mt_rand(1, 10000))).'";};'); echo "Mem usage: ".memory_get_usage()."\n"; } But in this example not: for ( $fp = fopen('/dev/urandom', 'rb'); true;) { eval ('$ano_fnc = function() {$x = "'.bin2hex(fread($fp, 10000)).'";};'); echo "Mem usage: ".memory_get_usage()."\n"; } The only different in the second example is the fixed body length of the eval()-created anoymous function. I wondering why the memory in the second code-example will be freed at some point while in the first example not. I don't think it's a PHP-Bug but i also not understand this behavior. Is there one who can explain this? Thank you very much. Greetings Hans-Juergen Petrich --- ## #65574 — keisial@gmail.com — 2013-02-03T16:28:33+00:00 URL: https://externals.io/message/65572#65574 · In reply to: #65572 > The only different in the second example is the fixed body length of > the eval()-created anoymous function. > I wondering why the memory in the second code-example will be freed at > some point while in the first example not. > > I don't think it's a PHP-Bug but i also not understand this behavior. > > Is there one who can explain this? > Thank you very much. > > Greetings > Hans-Juergen Petrich It can be simplified to: > while (true) { > $value = mt_rand(1, 10000); > $a = '$ano_fnc = function() {' . str_repeat(" ", $value) . '};'; > eval ($a); > echo "Mem usage: ".memory_get_usage()."\n"; > } The function *is* being removed, $ano_fnc alternates between object(Closure)#1 and object(Closure)#2, it's not leaking function objects. I suspect eval is checking a hashtable which is being forced to resize continuously, but it shouldn't grow unbounded either. --- ## #65575 — Sebastian Bergmann — 2013-02-03T17:07:40+00:00 URL: https://externals.io/message/65572#65575 · In reply to: #65572 Am 03.02.2013 16:27, schrieb Hans-Juergen Petrich: > Is there one who can explain this? Can you explain why you are using eval() instead of a real anonymous function? Thanks! -- Sebastian Bergmann Co-Founder and Principal Consultant http://sebastian-bergmann.de/ http://thePHP.cc/ --- ## #65576 — Sebastian Bergmann — 2013-02-03T17:33:05+00:00 URL: https://externals.io/message/65572#65576 · In reply to: #65575 Am 03.02.2013 18:07, schrieb Sebastian Bergmann: > Can you explain why you are using eval() instead of a real anonymous > function? Thanks! Please ignore my email; I mixed up create_function() and eval(). Then again, your code still does not make sense to me ... -- Sebastian Bergmann Co-Founder and Principal Consultant http://sebastian-bergmann.de/ http://thePHP.cc/ --- ## #65578 — Sebastian Krebs — 2013-02-03T18:31:15+00:00 URL: https://externals.io/message/65572#65578 · In reply to: #65576 2013/2/3 Sebastian Bergmann > Am 03.02.2013 18:07, schrieb Sebastian Bergmann: > > […] > > Please ignore my email; I mixed up create_function() and eval(). Then > again, your code still does not make sense to me ... > Don't know, how create_function() works internally, but it seems, that they at least act quite similar > -- > Sebastian Bergmann Co-Founder and Principal Consultant > http://sebastian-bergmann.de/ http://thePHP.cc/ > > -- > > > > -- github.com/KingCrunch --- ## #65602 — petrich@tronic-media.com — 2013-02-04T01:18:22+00:00 URL: https://externals.io/message/65572#65602 · In reply to: #65574 Hi Terry and all thank you very much for your response. > The only thing that confused me about what you say that the second *doesn't* grow Yes, about that i was [and am still :-)] also confused... why the 2nd one won't grow *non-stop* > so I checked and it does -- just the same as the first. Right, it grows, but not non-stop as in the 1st one. The memory will stop growing (on my machine) at ~2491584 bytes and the loop is able to run forever, creating each eval() furthermore uniqe ano-function's but not endless-filling Zend-internal tables. > but this still leaves the function record itself in the function_table hash so with a non-zero reference count and this doesn't get DTORed until request shutdown Not familar with the Zend-internals but just about so i was imaging and expecting it. That why i [still] also confused/wondering why in the 2nd example the memory will not grow *endless*. It seems that the function records in the function_table will be DTORed (or similar cleaned up) before request-shutdown at some point... Could this be the case? --- ## #65604 — Hans-Juergen Petrich — 2013-02-04T04:36:59+00:00 URL: https://externals.io/message/65572#65604 · In reply to: #65576 Hello > Then again, your code still does not make sense to me ... Fully understand :-) Not want wasting your time with my specific situation... but there are situation where it make sense :-) For example... in my situation, i have time intensive (mathematically) operations to do in pure php and was searching a way to speed this operations up. With PHP's eval() feature, creating php code at run time, this would be, in my situation, a possible way. Very simplified, i have to deal with the following: function calculate($ints) { $result = 0; for ($i=0; $i<10000000; ++$i) { $result += $ints[0] + $ints[1] + $ints[2] + $ints[3] + $ints[4] + $ints[5]; } return $result; } $ints = array(1,2,3,4,5,6); $result = calculate($ints); A faster way would be: $ints = array(1,2,3,4,5,6); eval('$calculate = function () { $result = 0; for ($i=0; $i<10000000; ++$i) { $result += '.$ints[0].' + '.$ints[1].' + '.$ints[2].' + '.$ints[3].' + '.$ints[4].' + '.$ints[5].'; } return $result; };'); $result = $calculate(); The function code it self is in reality more complex and variable in the function-body length, but the princip is the same. The code-architecture, in my specific situation, requires also that the operation encapsulated in a function. (And for security: It is ensured that $int is an 1-dim array filled only with integers) $calculate() works very fine and, of course, much faster than the hardcoded calculate(). Each time $ints is changing i also have, of course, re-define/creater $calculate() via eval. In such situations, for performance reasons, it would makes sense using eval() for re/creating ano-functions, doesn't it? And as long as the function-body has a fixed length each time re/creating the ano-function via eval() it would also not let grow the memory endless. --- ## #65627 — keisial@gmail.com — 2013-02-04T10:57:35+00:00 URL: https://externals.io/message/65572#65627 · In reply to: #65602 Hans-Jürgen Petrich wrote: > Hi Terry and all > thank you very much for your response. > > > […] > Yes, about that i was [and am still :-)] also confused... why the 2nd > one won't grow *non-stop* > > > […] > Right, it grows, but not non-stop as in the 1st one. > > The memory will stop growing (on my machine) at ~2491584 bytes and the > loop is able to run forever, > creating each eval() furthermore uniqe ano-function's but not > endless-filling Zend-internal tables. > > > […] > Not familar with the Zend-internals but just about so i was imaging > and expecting it. > > That why i [still] also confused/wondering why in the 2nd example the > memory will not grow *endless*. > It seems that the function records in the function_table will be > DTORed (or similar cleaned up) before request-shutdown at some point... > > Could this be the case? As you are reassigning $ano_fnc, the old closure is being destructed. Had you used create_function(), it wouldn't happen. Now the question is, if it is correctly freeing the functions (and it is good that it does so), why is it not doing it when they have different lengths? --- ## #65632 — Terry Ellison — 2013-02-04T12:11:31+00:00 URL: https://externals.io/message/65572#65632 · In reply to: #65602 > Hi Terry and all > thank you very much for your response. > > > […] > Yes, about that i was [and am still :-)] also confused... why the 2nd > one won't grow *non-stop* > > > […] > Right, it grows, but not non-stop as in the 1st one. > > The memory will stop growing (on my machine) at ~2491584 bytes and the > loop is able to run forever, > creating each eval() furthermore uniqe ano-function's but not > endless-filling Zend-internal tables. > > > […] > Not familar with the Zend-internals but just about so i was imaging > and expecting it. > > That why i [still] also confused/wondering why in the 2nd example the > memory will not grow *endless*. > It seems that the function records in the function_table will be > DTORed (or similar cleaned up) before request-shutdown at some point... > > Could this be the case? > OK, Hans-Jürgen, this one has got me interested. I am developing a fork of APC optimized for cgi and cli use -- but that's a different topic -- though understanding the DTOR processes for compiler objects interests me because of this. I'll go through the code and especially for Closure objects to understand why. However thinking through this logically: 1. The fact that the second does stop growing means that that the reassignment of the global $ano sets the RC of the previous closure object to zero triggering DTOR of the lamba function. 2. There is something pathological about the first case which is frustrating garbage collection on the lambda function DTOR. I replaced your inner loop by: $len = $argv[1] & 1 ? $argv[2] : mt_rand(1, $argv[2]); $str = "'." . bin2hex(fread($fp, $len)) . "'"; if ($argv[1] & 2) $str = "function() {\$y = $str; };"; eval ("\$x = $str;"); echo "Mem usage: ".memory_get_usage()."\n"; to allow me to use arg1 to select one of the four test cases 0..3 and arg2 is the (max) string size, n say. This clearly shows the fact that the memory explosion only occurs if the string is allocated *inside* the lambda function. 4. If you substitute n<15 then memory growth rapidly stabilises for PHP 5.3.17 0, but still explodes for n>14 5. In the case of PHP 5.4.6 a similar effect occurs except that explosion occurs at n>11. 6. The fact that 5.3 and 5.4 are different is notable -- however, the fact that 5.4 is still (eventually) stable for n<12 means that this isn't a string interning issue. Interesting. Merits more research :-) --- ## #65637 — petrich@tronic-media.com — 2013-02-04T15:27:57+00:00 URL: https://externals.io/message/65572#65637 · In reply to: #65632 Am 04.02.2013, 19:11 Uhr, schrieb Terry Ellison : > > […] > OK, Hans-Jürgen, this one has got me interested. I am developing a fork > of APC optimized for cgi and cli use -- but that's a different topic -- > though understanding the DTOR processes for compiler objects interests > me because of this. I'll go through the code and especially for Closure > objects to understand why. However thinking through this logically: > > 1. The fact that the second does stop growing means that that the > reassignment of the global $ano sets the RC of the previous closure > object to zero triggering DTOR of the lamba function. > > 2. There is something pathological about the first case which is > frustrating garbage collection on the lambda function DTOR. > > I replaced your inner loop by: > > $len = $argv[1] & 1 ? $argv[2] : mt_rand(1, $argv[2]); > $str = "'." . bin2hex(fread($fp, $len)) . "'"; > if ($argv[1] & 2) $str = "function() {\$y = $str; };"; > eval ("\$x = $str;"); > echo "Mem usage: ".memory_get_usage()."\n"; > > to allow me to use arg1 to select one of the four test cases 0..3 and > arg2 is the (max) string size, n say. This clearly shows the fact that > the memory explosion only occurs if the string is allocated *inside* the > lambda function. > > 4. If you substitute n<15 then memory growth rapidly stabilises for PHP > 5.3.17 0, but still explodes for n>14 > > 5. In the case of PHP 5.4.6 a similar effect occurs except that > explosion occurs at n>11. > > 6. The fact that 5.3 and 5.4 are different is notable -- however, the > fact that 5.4 is still (eventually) stable for n<12 means that this > isn't a string interning issue. > > Interesting. Merits more research :-) > Hi Terry i made similar tests with similar results. Currently, in my script, i'm just padding the function-body with additional spaces enough to ensure having always a fixed length to avoid endless-memory growing. But with non-good feeling because i'm still not sure what is going on internaly... and my knowledge about the php/zend internals is poor :-) So i'm very glad getting your interest in this issue. Logical thinking... since the memory-growing depends "only" on variable length of the function-body (and not of the content it self) i could imaging that it is possible to get the php/zend internals to also with variable lengths of the function-body... seems not far away. But as i said... my knowledge ends at this point but i'm very interessted about your results, if you research it further more. Thank you again Greetings --- ## #65638 — petrich@tronic-media.com — 2013-02-04T15:37:09+00:00 URL: https://externals.io/message/65572#65638 · In reply to: #65627 Am 04.02.2013, 17:57 Uhr, schrieb "Ángel González" : > Hans-Jürgen Petrich wrote: > > […] > > As you are reassigning $ano_fnc, the old closure is being destructed. > Had you used create_function(), it wouldn't happen. > Now the question is, if it is correctly freeing the functions (and it is > good that it does so), why is it not doing it when they have different > lengths? > > > Now the question is, if it is correctly freeing the functions (and it is > good that it does so), why is it not doing it when they have different > lengths? Yes, you bring it to the point. Thats my question. Hoping of course that maybe it could be possible that freeing the memory also on different length. Thank you. --- ## #65671 — Terry Ellison — 2013-02-05T13:53:24+00:00 URL: https://externals.io/message/65572#65671 · In reply to: #65627 > > […] > As you are reassigning $ano_fnc, the old closure is being destructed. > Had you used create_function(), it wouldn't happen. > Now the question is, if it is correctly freeing the functions (and it is > good that it does so), why is it not doing it when they have different > lengths? It's a bug. The Closure class DTOR does not delete the derefenced function from the CG(function_table). If you did the eval at line 20 in say /tmp/xx.php then the INCLUDE_OR_EVAL instruction calls the Zend compiler with the args: (1) the source to be compiled and (2) the title "/tmp/x.php(20) : eval()'d code" The compiler than gives the closure function a magic name: "\0{closure}/tmp/x.php(20) : eval()'d code0xHHHHHHHH" where 0xHHHHHHHH is the hex address of the "function" substring in the evaluated string. The compiler uses a zend_hash_update to insert this into the CG(function_table). What happens if you use a fixed length string replacing another string of the same length dropping its refcount to 0 is that the allocator is clever and will tend to reallocate the old one and hence the address of the string is the same and the address of the offset of the "function" substring is the same so it regenerates the same magic name -- pretty much as an accidental side-effect. When this happens, it's this hash update function that calls the DTOR on any pre-existing function with this name. I simply put a breakpoint on the relevant line in the zend_do_begin_function_declaration() code and if you used a fixed offset into the same string you only got one {closure} entry. If the allocation ended up "randomizing" the address, then the {closure} entries grew until memory exhaustion. As I said -- interesting. Need to think about the consequences before I submit a bugrep. Regards Terry --- ## #65702 — petrich@tronic-media.com — 2013-02-06T17:59:30+00:00 URL: https://externals.io/message/65572#65702 · In reply to: #65671 Am 05.02.2013, 20:53 Uhr, schrieb Terry Ellison : > > On 04/02/13 10:57, Ángel González > wrote: > > > The memory will stop growing (on my machine) at ~2491584 bytes and the > loop is able to run forever, > creating each eval() furthermore uniqe ano-function's but not > endless-filling Zend-internal tables. > > > but this still leaves the function record itself in the > function_table hash so with a non-zero reference count and this > doesn't get DTORed until request shutdown > > Not familar with the Zend-internals but just about so i was imaging > and expecting it. > > That why i [still] also confused/wondering why in the 2nd example the > memory will not grow *endless*. > It seems that the function records in the function_table will be > DTORed (or similar cleaned up) before request-shutdown at some point... > > Could this be the case? > > > > […] > > It's a bug. The Closure class DTOR does not delete the derefenced > function from the CG(function_table). > > If you did the eval at line 20 in say /tmp/xx.php then the > INCLUDE_OR_EVAL instruction calls the Zend compiler with the args: > > (1) the source to be compiled and > > (2) the title "/tmp/x.php(20) : eval()'d code" > > > The compiler than gives the closure function a magic name: > > > "\0{closure}/tmp/x.php(20) : eval()'d code0xHHHHHHHH" > > > where 0xHHHHHHHH is the hex address of the "function" substring in > the evaluated string. The compiler uses a zend_hash_update to > insert this into the CG(function_table). > > > What happens if you use a fixed length string replacing another > string of the same length dropping its refcount to 0 is that the > allocator is clever and will tend to reallocate the old one and > hence the address of the string is the same and the address of the > offset of the "function" substring is the same so it regenerates the > same magic name -- pretty much as an accidental side-effect. When > this happens, it's this hash update function that calls the DTOR on > any pre-existing function with this name. > > > I simply put a breakpoint on the relevant line in the > zend_do_begin_function_declaration() code and if you used a fixed > offset into the same string you only got one {closure} entry. If > the allocation ended up "randomizing" the address, then the > {closure} entries grew until memory exhaustion. > > > As I said -- interesting. Need to think about the consequences > before I submit a bugrep. > > Regards > > Terry > Very interessting Terry. Would be great if the bug could be fixed. If you post a bug report would you post the bug-report-url here also that i (and/or other readers) can furthermore trace the discussion of this issue? Thank you again Greetings --- ## #66185 — Terry Ellison — 2013-02-25T13:03:12+00:00 URL: https://externals.io/message/65572#66185 · In reply to: #65572 > In this example (using php-5.4.11 on Linux) the memory will grow > non-stop: > > for ( $fp = fopen('/dev/urandom', 'rb'); true;) { > eval ('$ano_fnc = function() {$x = "'.bin2hex(fread($fp, > mt_rand(1, 10000))).'";};'); > echo "Mem usage: ".memory_get_usage()."\n"; > } > > > But in this example not: > > for ( $fp = fopen('/dev/urandom', 'rb'); true;) { > eval ('$ano_fnc = function() {$x = "'.bin2hex(fread($fp, > 10000)).'";};'); > echo "Mem usage: ".memory_get_usage()."\n"; > } > Hans-Juergen, I've raised a bugrep https://bugs.php.net/bug.php?id=64291 which you might want to review and add any appropriate comments. I had to think about this one. It's worthwhile observing that this second example is the only occasion, as far as I know, that PHP does any garbage collection of code objects before request shutdown. For example create_function() objects are given the name "\0LambdaN" where N is the count of the number of created functions so far in this request. They are registered in the function table and persist until request shutdown. That's the way PHP handles them by design. As I said in the bugrep, the normal behaviour of persistence is what you want because if you think about the normal use of the anonymous function, say while (!feof($fp)) { $line = preg_replace_callback( '|

\s*\w|', function($matches) {return strtolower($matches[0]);}, gets($fp) ); echo $line; } Then the anonymous function is compiled once and rebound to the closure object which is passed as the second argument for the callback each time through the loop. OK, doing the closure CTOR/DTOR once per loop. is not the cleverest of ideas and this is the sort of thing that would be hoisted out of the loop in a language which did such optimization, but PHP doesn't. It's a LOT better that compiling a new function each loop (which is how Example #1 on http://php.net/manual/en/function.preg-replace-callback.php does it!) This is what you want to happen. It's just too complicated for PHP to work out if the function might or not be rebound to. I suspect the bug here is really the implicit assumption that the magic function name generated by the eval ('$ano_fnc = function() { ... }'); is unique, but as your examples shows, thanks to garbage collection and reuse of memory, sometimes it isn't. In these circumstances thank to the use of a hash update and the table DTOR the old one is deleted. So assume that what you are doing is exploiting a bug, so my advice is not to do this. It might be fixed in a future release. Regards Terry --- ## #66315 — petrich@tronic-media.com — 2013-02-28T07:19:13+00:00 URL: https://externals.io/message/65572#66315 · In reply to: #66185 Am 25.02.2013, 20:03 Uhr, schrieb Terry Ellison : > > On 03/02/13 15:27, Hans-Juergen Petrich > wrote: > > > > > […] > > Hans-Juergen, I've raised a bugrep > https://bugs.php.net/bug.php?id=64291 which you might want to review > and add any appropriate comments. I had to think about this one. It's worthwhile observing that this second example is the only > occasion, as far as I know, that PHP does any garbage collection of > code objects before request shutdown. For example create_function() > objects are given the name "\0LambdaN" where N is the count of the > number of created functions so far in this request. They are > registered in the function table and persist until request > shutdown. That's the way PHP handles them by design. > > > As I said in the bugrep, the normal behaviour of persistence is what > you want because if you think about the normal use of the anonymous > function, say > > > while (!feof($fp)) { > > $line = preg_replace_callback( > > '|

\s*\w|', > > function($matches) {return strtolower($matches[0]);}, > > gets($fp) > > ); > > echo $line; > > } > > > Then the anonymous function is compiled once and rebound to the > closure object which is passed as the > second argument for the callback each time through the loop. OK, > doing the closure CTOR/DTOR once per loop. is not the cleverest of > ideas and this is the sort of thing that would be hoisted out of the > loop in a language which did such optimization, but PHP doesn't. It's a LOT better that compiling a new function each loop (which is > how Example #1 on > http://php.net/manual/en/function.preg-replace-callback.php does > it!) This is what you want to happen. > > It's just too complicated for PHP to work out if the function might > or not be rebound to. I suspect the bug here is really the implicit > assumption that the magic function name generated by the > > eval ('$ano_fnc = function() { ... }'); > > > is unique, but as your examples shows, thanks to garbage collection > and reuse of memory, sometimes it isn't. In these circumstances > thank to the use of a hash update and the table DTOR the old one is > deleted. > > > So assume that what you are doing is exploiting a bug, so my advice > is not to do this. It might be fixed in a future release. > > > Regards Terry > > Thank you Terry for this information. Yes, i avoid using it. I'll use create_function() for a limited count... which will do the work also. Again thank you. Greetings Hans-Jürgen