# Memory leak in eval()'d code - Thread: https://externals.io/message/106084 - Original archive: https://news-web.php.net/php.internals/106084 - Started by Benjamin Morel on 2019-06-28T00:41:43+00:00 - Messages: 19 - Reply by email to internals@lists.php.net - More for agents: search with https://externals.io/search.md?q={query} · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt --- ## #106084 — Benjamin Morel — 2019-06-28T00:41:43+00:00 URL: https://externals.io/message/106084#106084 Hi internals, I've tracked down a memory leak to an anonymous class created within eval(): ``` for ($i = 0; $i < 1000000; $i++) { $object = eval('return new class {};'); if ($i % 1000 == 0) { echo `memory_get_usage()` . "\n"; } } ``` The memory usage quickly ramps up and memory_limit is reached in seconds. Without eval(), the memory usage stays flat. I'm on 7.3.6. *Is this a bug?* Or is this some inherent limitation of eval() that cannot be fixed? Thanks for any insight. Ben *In case you're wondering (I'm sure you are!), I need this to dynamically create an object that implements an arbitrary number of interfaces , for a schema.org structured data reader library.* --- ## #106092 — johannes@schlueters.de — 2019-06-28T15:59:08+00:00 URL: https://externals.io/message/106084#106092 > Hi internals, > > I've tracked down a memory leak to an anonymous class created within > eval(): > > ``` > for ($i = 0; $i < 1000000; $i++) { > $object = eval('return new class {};'); > > if ($i % 1000 == 0) { > echo `memory_get_usage()` . "\n"; > } > } > > ``` > > The memory usage quickly ramps up and memory_limit is reached in > seconds. > Without eval(), the memory usage stays flat. I'm on 7.3.6. > > *Is this a bug?* Or is this some inherent limitation of eval() that > cannot > be fixed? I case this is non-trivial to fix. Each invocation recompile a new piece of code, which creates a new class. Similar to for ($i = 0; $i < 1000000; $i++) { eval("function f$i(){}"); } f1(); f2(); ... or for ($i = 0; $i < 1000000; $i++) { file_put_contents("c$i.php", 'return new class {};'); $object = include "c$i.php"; } Theoretical it is thinkable we could either detect equal classes (but unlikely to happen ... if it's the same why do it in eval, zthus cost likely is higher than benefit in most cases?) or unload the class once it's not needed anymore (which is hard to detect with reflection and other mechanisms which aren't bound to an instance's zval and also not cheap) johannes --- ## #106093 — Benjamin Morel — 2019-06-28T16:49:49+00:00 URL: https://externals.io/message/106084#106093 > Theoretical it is thinkable we could either detect equal classes (but > unlikely to happen ... if it's the same why do it in eval, zthus cost > likely is higher than benefit in most cases?) Thanks for your reply. I do it in eval() because schema.org has many types (more than 600 at the moment), and each object ("Thing") can be an instance of one or more of these types. This creates an unmanageable number of possible combinations that I cannot pre-compute, so I need to dynamically create a definition for an object that implements an arbitrary number of interfaces. > or unload the class once > it's not needed anymore (which is hard to detect with reflection and > other mechanisms which aren't bound to an instance's zval and also not > cheap) AFAIK, unloading a class is not possible from userland, I guess you're talking about detecting this in PHP itself? Anyway, that's an edge case, I found a workaround which is to keep a cache of freshly created objects, indexed by interface name(s). Anytime I request a combination of types that's already been handled in the past, I return a clone of the cached object. Even though the possible combinations of types are huge and impossible to predict in advance, there are never more than a handful of such combinations used in a single document, and I'd be surprised if there are actually that many combinations actually used in the wild. So this should hardly be a problem. Thank you, Ben On Fri, 28 Jun 2019 at 17:59, Johannes Schlüter wrote: > > > […] > > I case this is non-trivial to fix. Each invocation recompile a new > piece of code, which creates a new class. > > Similar to > > for ($i = 0; $i < 1000000; $i++) { > eval("function f$i(){}"); > } > f1(); > f2(); > ... > > or > > for ($i = 0; $i < 1000000; $i++) { > file_put_contents("c$i.php", 'return new class {};'); > $object = include "c$i.php"; > } > > Theoretical it is thinkable we could either detect equal classes (but > unlikely to happen ... if it's the same why do it in eval, zthus cost > likely is higher than benefit in most cases?) or unload the class once > it's not needed anymore (which is hard to detect with reflection and > other mechanisms which aren't bound to an instance's zval and also not > cheap) > > johannes --- ## #106094 — johannes@schlueters.de — 2019-06-28T17:48:48+00:00 URL: https://externals.io/message/106084#106094 > > > […] >not > > […] > >AFAIK, unloading a class is not possible from userland, I guess you're >talking about detecting this in PHP itself? Correct. >Anyway, that's an edge case, I found a workaround which is to keep a >cache >of freshly created objects, indexed by interface name(s). >Anytime I request a combination of types that's already been handled in >the >past, I return a clone of the cached object. > >Even though the possible combinations of types are huge and impossible >to >predict in advance, there are never more than a handful of such >combinations used in a single document, and I'd be surprised if there >are >actually that many combinations actually used in the wild. So this >should >hardly be a problem. This is going to be off-topic for his list: you could rethink your approach. Maybe you can generate the classes only on demand (if you can derive the information from a class name an evil way is `function __autoload($classname) { /* figure out what is needed */ eval("class $classname { ...}")}` ... absolutely evil, but hides the machinery) or you could rethink whether you really need distinct classes or whether a single type with accessor routines (either some custom or __call/__set/__get) isn't sufficient. Or something else ... johannes --- ## #106102 — Stanislav Malyshev — 2019-06-28T22:17:14+00:00 URL: https://externals.io/message/106084#106102 Hi! > I case this is non-trivial to fix. Each invocation recompile a new > piece of code, which creates a new class. That's not a "leak". You create new objects (in this case, classes), they take memory. -- Stas Malyshev smalyshev@gmail.com --- ## #106103 — Benjamin Morel — 2019-06-28T22:37:06+00:00 URL: https://externals.io/message/106084#106103 > > That's not a "leak". You create new objects (in this case, classes), > they take memory. Why do they not "leak" memory without eval() then? Replace with `$object = new class {};` and memory usage stays flat. There has do be some kind of garbage collection for these anonymous classes. Ben --- ## #106104 — Stanislav Malyshev — 2019-06-28T22:52:27+00:00 URL: https://externals.io/message/106084#106104 > That's not a "leak". You create new objects (in this case, classes), > they take memory. > > > Why do they not "leak" memory without eval() then? Replace with `$object > = new class {};` and memory usage stays flat. > There has do be some kind of garbage collection for these anonymous classes. AFAIR this does not create new classes, since it's the same code, and same code means same class. But eval() has new code every time, thus new class. Generally I don't think PHP has any operation that can destroy an existing class. It won't be easy too since you don't know whether there are any objects of this class around (unless you're in shutdown). -- Stas Malyshev smalyshev@gmail.com --- ## #106105 — me@jhdxr.com — 2019-06-29T07:07:24+00:00 URL: https://externals.io/message/106084#106105 I think we missed the point here. Clousre, or anonymous class, should not be considered as normal class. We expect normal class existing once we declare it till end of script. However, for anonymous class, it's usually used within certain scope, so not only the instances, the class itself should be included in the GC as well. I guess the problem here is we didn't GC the space for definition of anonymous classes. Regards, CHU Zhaowei > -----Original Message----- > From: Stanislav Malyshev > Sent: Saturday, June 29, 2019 6:52 AM > To: Benjamin Morel > Cc: PHP Internals > Subject: Re: [PHP-DEV] Memory leak in eval()'d code > > > > > > […] > > AFAIR this does not create new classes, since it's the same code, and same code > means same class. But eval() has new code every time, thus new class. Generally > I don't think PHP has any operation that can destroy an existing class. It won't be > easy too since you don't know whether there are any objects of this class > around (unless you're in shutdown). > > -- > Stas Malyshev > smalyshev@gmail.com > > -- > To unsubscribe, visit: > http://www.php.net/unsub.php --- ## #106106 — Nikita Popov — 2019-06-29T07:57:44+00:00 URL: https://externals.io/message/106084#106106 > I think we missed the point here. Clousre, or anonymous class, should not > be considered as normal class. We expect normal class existing once we > declare it till end of script. However, for anonymous class, it's usually > used within certain scope, so not only the instances, the class itself > should be included in the GC as well. > I guess the problem here is we didn't GC the space for definition of > anonymous classes. > > Regards, > CHU Zhaowei > As Johannes already pointed out, we cannot garbage collect anonymous class definitions due to the existence of opaque references. A simple example of code that currently works: $obj = eval('return new class {}'); $class = get_class($obj); // create opaque reference unset($obj); // drop last direct reference $obj = new $class; In the end, an anonymous class is essentially a class with a random name and some inline construction sugar, but apart from that does not differ from ordinary classes. (I've also seen some calls to allow syntax like $class = class {}; that would make that even more obvious.) The situation here would be different if we had first-class classes and did not refer to classes by name. But as-is, I don't think garbage collecting anonymous classes is a possibility. Nikita > -----Original Message----- > > […] > classes), > > […] > classes. > > […] > same code > > […] > Generally > > […] > It won't be > > […] > visit: > > […] --- ## #106107 — Benjamin Morel — 2019-06-29T09:48:49+00:00 URL: https://externals.io/message/106084#106107 > As Johannes already pointed out, we cannot garbage collect anonymous class definitions due to the existence of opaque references. I still don't understand *why*, then, the memory stays flat when the new class does not happen in eval(): ``` for ($i = 0; $i < 1000000; $i++) { $object = new class {}; if ($i % 1000 == 0) { echo `memory_get_usage()` . "\n"; } } ``` I checked `get_class()` for all instances returned, **it's always the same class name whether it's called from eval() or not**, so this code should only ever create a single class in memory, right? Why the leak with eval()? > $class = get_class($obj); > $obj = new $class; Interesting. Can this behaviour be relied upon (is it documented?), or is this an implementation detail that may change at any time? Thank you, Ben --- ## #106108 — me@jhdxr.com — 2019-06-29T15:30:50+00:00 URL: https://externals.io/message/106084#106108 Thanks for the example. Now I understand the root cause. There is a similar bug report(https://bugs.php.net/bug.php?id=74938) for anonymous function. I thought it leaked for the same reason, but now I think I’m wrong. We don’t have to create new classes for them since they are all instances of Closure class, and it’s not possible to create opaque reference for functions. __FUNCTION__ returns “{closure}”. Could you take a look? Back to the anonymous class, I’m thinking if we can have any workaround for this issue, otherwise it will be a huge problem if someone wants to write applications running for a long time. It will be a memory leak that cannot be fixed in userland code unless he decides to drop anonymous class at all. Since most of the use cases should not have opaque references, can we add a flag in zval to indicate if an object has this kind of reference? The number of methods to create such opaque references should be limited. __CLASS__, `get_class()`, reflections, clone. Did I miss any other way? With the help of this flag, we are able to know if it’s safe to remove the class definition during destruction. Regards, CHU Zhaowei From: Nikita Popov Sent: Saturday, June 29, 2019 3:58 PM To: CHU Zhaowei Cc: Stanislav Malyshev ; Benjamin Morel ; PHP Internals Subject: Re: [PHP-DEV] Memory leak in eval()'d code I think we missed the point here. Clousre, or anonymous class, should not be considered as normal class. We expect normal class existing once we declare it till end of script. However, for anonymous class, it's usually used within certain scope, so not only the instances, the class itself should be included in the GC as well. I guess the problem here is we didn't GC the space for definition of anonymous classes. Regards, CHU Zhaowei As Johannes already pointed out, we cannot garbage collect anonymous class definitions due to the existence of opaque references. A simple example of code that currently works: $obj = eval('return new class {}'); $class = get_class($obj); // create opaque reference unset($obj); // drop last direct reference $obj = new $class; In the end, an anonymous class is essentially a class with a random name and some inline construction sugar, but apart from that does not differ from ordinary classes. (I've also seen some calls to allow syntax like $class = class {}; that would make that even more obvious.) The situation here would be different if we had first-class classes and did not refer to classes by name. But as-is, I don't think garbage collecting anonymous classes is a possibility. Nikita > -----Original Message----- > From: Stanislav Malyshev > > Sent: Saturday, June 29, 2019 6:52 AM > To: Benjamin Morel > > Cc: PHP Internals > > Subject: Re: [PHP-DEV] Memory leak in eval()'d code > > > > > > […] > > AFAIR this does not create new classes, since it's the same code, and same code > means same class. But eval() has new code every time, thus new class. Generally > I don't think PHP has any operation that can destroy an existing class. It won't be > easy too since you don't know whether there are any objects of this class > around (unless you're in shutdown). > > -- > Stas Malyshev > smalyshev@gmail.com > > -- > To unsubscribe, visit: > http://www.php.net/unsub.php --- ## #106111 — Nikita Popov — 2019-06-29T16:20:43+00:00 URL: https://externals.io/message/106084#106111 > Thanks for the example. Now I understand the root cause. There is a > similar bug report(https://bugs.php.net/bug.php?id=74938) for anonymous > function. I thought it leaked for the same reason, but now I think I’m > wrong. We don’t have to create new classes for them since they are all > instances of Closure class, and it’s not possible to create opaque > reference for functions. __FUNCTION__ returns “{closure}”. Could you take > a look? > This leaks for the same reason (function definition is not garbage collected), but in this case we could indeed implement the garbage collection without any observable changes to semantics. I don't think it will be simple to do this though. Back to the anonymous class, I’m thinking if we can have any workaround > for this issue, otherwise it will be a huge problem if someone wants to > write applications running for a long time. It will be a memory leak that > cannot be fixed in userland code unless he decides to drop anonymous class > at all. > I think you got caught up in a wrong premise here: Just using anonymous classes does not cause leaks. The problem here is that OP is *creating* many classes (the fact that they are anonymous ultimately doesn't matter) by eval'ing code. That is what causes the leak. Unless you are doing something like this (which you obviously shouldn't, for more reasons than this), there is nothing to worry about. As such, I don't really think there is a problem worth solving here. Nikita Since most of the use cases should not have opaque references, can we add a > flag in zval to indicate if an object has this kind of reference? The > number of methods to create such opaque references should be limited. > __CLASS__, `get_class()`, reflections, clone. Did I miss any other way? With > the help of this flag, we are able to know if it’s safe to remove the class > definition during destruction. > Regards, > > CHU Zhaowei > > > > *From:* Nikita Popov > *Sent:* Saturday, June 29, 2019 3:58 PM > *To:* CHU Zhaowei > *Cc:* Stanislav Malyshev ; Benjamin Morel < > benjamin.morel@gmail.com>; PHP Internals > *Subject:* Re: [PHP-DEV] Memory leak in eval()'d code > > > > > > I think we missed the point here. Clousre, or anonymous class, should not > be considered as normal class. We expect normal class existing once we > declare it till end of script. However, for anonymous class, it's usually > used within certain scope, so not only the instances, the class itself > should be included in the GC as well. > I guess the problem here is we didn't GC the space for definition of > anonymous classes. > > Regards, > CHU Zhaowei > > > > As Johannes already pointed out, we cannot garbage collect anonymous class > definitions due to the existence of opaque references. A simple example of > code that currently works: > > > > $obj = eval('return new class {}'); > > $class = get_class($obj); // create opaque reference > > unset($obj); // drop last direct reference > > $obj = new $class; > > > > In the end, an anonymous class is essentially a class with a random name > and some inline construction sugar, but apart from that does not differ > from ordinary classes. (I've also seen some calls to allow syntax like > $class = class {}; that would make that even more obvious.) > > > > The situation here would be different if we had first-class classes and > did not refer to classes by name. But as-is, I don't think garbage > collecting anonymous classes is a possibility. > > > > Nikita > > > > > […] > classes), > > […] > classes. > > […] > same code > > […] > Generally > > […] > It won't be > > […] > visit: > > […] --- ## #106115 — Benjamin Morel — 2019-06-29T21:25:41+00:00 URL: https://externals.io/message/106084#106115 > The problem here is that OP is *creating* many classes (the fact that they are anonymous ultimately doesn't matter) by eval'ing code. That is what causes the leak. I still don't understand why there is no leak when *not* using eval(), though. Ben --- ## #106116 — Nikita Popov — 2019-06-29T21:51:57+00:00 URL: https://externals.io/message/106084#106116 On Sat, Jun 29, 2019 at 11:25 PM Benjamin Morel wrote: > > […] > they are anonymous ultimately doesn't matter) by eval'ing code. That is > what causes the leak. > > I still don't understand why there is no leak when *not* using eval(), > though. > > Ben > Without eval you are creating many objects of the same anonymous class. With eval you are creating a single object each for many anonymous classes. In the first case there is only a single class, in the second there are as many classes as there are eval()s. That's the theory. In practice we have a long-standing bug with RTD key collisions -- this means that the names generated for anonymous classes (and closures for that matter) are not actually unique. For anonymous classes the current workaround is to reuse the existing class definition if an RTD key collision occurs. For your original code (and I should say that this is very specific to the code because it is susceptible to memory layout details) it happens that the RTD key for all those anonymous classes is the same, so in the end you end up with just one class definition also in the eval() case. The leaked memory is an allocated but unused class entry. We could plug that particular leak -- but I should emphasize that the current behavior is incorrect and once the key collision issue is resolved this will create a new class for each eval. Nikita --- ## #106117 — David Walker — 2019-06-29T21:56:23+00:00 URL: https://externals.io/message/106084#106117 > On Sat, Jun 29, 2019 at 11:25 PM Benjamin Morel > wrote: > > The leaked memory is an allocated but unused class > entry. We could plug that particular leak -- but I should emphasize that > the current behavior is incorrect and once the key collision issue is > resolved this will create a new class for each eval. To clarify; you’re saying that the behavior seen in the non scaled case is kind of “wrong” in the sense that it’s exposing the big in naming of the anon class. Once the bug is resolved, the two behaviors shown should mimic each other, wherein the memory footprint would grow as in the example eval() case. — Dave --- ## #106118 — Nikita Popov — 2019-06-29T22:04:52+00:00 URL: https://externals.io/message/106084#106118 > > > > > > […] > > > To clarify; you’re saying that the behavior seen in the non scaled case is > kind of “wrong” in the sense that it’s exposing the big in naming of the > anon class. Once the bug is resolved, the two behaviors shown should mimic > each other, wherein the memory footprint would grow as in the example > eval() case. > What is the "non scaled case"? To repeat what I said earlier: Without eval there is no and will not be a leak. The bug is that eval() currently reuses an existing class instead of creating a new one each time. Calling get_class(eval("return new class {};")) should be returning a new string every time, while for OPs script it currently does not. tl;dr: while (1) { get_class(new class {}); // should always return the same get_class(eval("return new class {};")); // should always return different, currently may return same due to bug } Nikita --- ## #106119 — Benjamin Morel — 2019-06-29T22:19:53+00:00 URL: https://externals.io/message/106084#106119 > > What is the "non scaled case"? To repeat what I said earlier: Without eval > there is no and will not be a leak. > The bug is that eval() currently reuses an existing class instead of > creating a new one each time. Calling get_class(eval("return new class > {};")) should be returning a new string every time, while for OPs script it > currently does not. > tl;dr: > while (1) { > get_class(new class {}); // should always return the same > get_class(eval("return new class {};")); // should always return > different, currently may return same due to bug > } Ok thank you, anyway the good news is that I found a workaround for now (creating classes of a given type once then cloning). And if I understand correctly, once the issue you mention is resolved (and `get_class()` returns a distinct class every time), I should have another alternative: ``` $object = eval('return new class (...) {};'); $class = get_class($object); $anotherObjectOfSameClass = new $class(); ``` And I do understand that my use case is border line, and does not deserve too much attention. It's very useful to be able to perform instanceof on objects that dynamically implement an arbritary set of interfaces, though. Ben --- ## #106120 — Stanislav Malyshev — 2019-06-30T02:43:22+00:00 URL: https://externals.io/message/106084#106120 Hi! > I still don't understand *why*, then, the memory stays flat when the new > class does not happen in eval(): Because that code creates only one class. > I checked `get_class()` for all instances returned, **it's always the same > class name whether it's called from eval() or not**, so this code should > only ever create a single class in memory, right? Why the leak with eval()? There's no leak, eval() just creates new class each time. -- Stas Malyshev smalyshev@gmail.com --- ## #106121 — me@jhdxr.com — 2019-06-30T12:48:32+00:00 URL: https://externals.io/message/106084#106121 Eval is not the only case that will cause memory usage growing actually. https://3v4l.org/fPhlU is an example using include inside a loop, which is used to simulate including same file several times. I agree it might be trivial to improve for eval since it's not recommended to use due to security concern nowadays. But I think we should pay attention to this issue since include/require behave the same. In theory, I understand how it happens. You won't be surprised if you copy the code and paste it several times and more memory is used. It's reasonable that more memory is needed to compile longer codes. Include several times just work like copy&paste several times. But since we are including the same file, is it possible to detect and reuse the space? Regards, CHU Zhaowei > -----Original Message----- > From: Stanislav Malyshev > Sent: Sunday, June 30, 2019 10:43 AM > To: Benjamin Morel ; Nikita Popov > > Cc: CHU Zhaowei ; PHP Internals > Subject: Re: [PHP-DEV] Memory leak in eval()'d code > > Hi! > > > […] > > Because that code creates only one class. > > > […] > eval()? > > There's no leak, eval() just creates new class each time. > -- > Stas Malyshev > smalyshev@gmail.com > > -- > To unsubscribe, visit: > http://www.php.net/unsub.php