# Trapping "memory exhausted" error - Thread: https://externals.io/message/28857 - Original archive: https://news-web.php.net/php.internals/28857 - Started by David Sklar on 2007-04-16T10:19:07+00:00 - Messages: 10 - Reply by email to internals@lists.php.net - More for agents: search with https://externals.io/search.md?q={query} · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt --- ## #28857 — David Sklar — 2007-04-16T10:19:07+00:00 URL: https://externals.io/message/28857#28857 I am interested in being able to trap the (currently) fatal error that results when memory usage exceeds the defined memory limit. I was thinking it could work as follows: - in addition to a memory_limit configuration directive, there could be a "memory_limit_grace" configuration directive. This gets stored in the struct _zend_mm_heap, along with the limit. - Also added to struct _zend_mm_heap is a "initial limit reached" flag - When zend_mm_safe_error() in Zend/zend_alloc.c is invoked under the current conditions (memory_limit exceeded), it sets the "initial limit reached" flag, adjusts the heap limit to the "memory_limit_grace" value and throws some non-fatal error (or an exception if it is feasible from here.) - When zend_mm_safe_error() is invoked and the "initial limit reached" flag is already set, it throws the fatal error exactly as it does now. This "grace period" would provide a way to gracefully exit when a memory limit is reached, but also has the hard limit still enforced so that code which is supposed to be gracefully exiting doesn't chew up too much additional memory. Is it feasible to adjust the heap limit and throw a non-fatal error from within zend_mm_safe_error()? David --- ## #28864 — Stanislav Malyshev — 2007-04-16T18:18:28+00:00 URL: https://externals.io/message/28857#28864 · In reply to: #28857 > - in addition to a memory_limit configuration directive, there could > be a "memory_limit_grace" configuration directive. This gets stored in > the struct _zend_mm_heap, along with the limit. That could be a problem because it's very hard to know exact memory requirements for PHP code for most people not intimately knowing the engine internals, and also recovering from failed allocation usually is not simple since management structures could be in half-done state. So writing useful handler for such an error would be rather hard to do. Having said that, what exactly would you plan to do in this error handler? -- Stanislav Malyshev, Zend Products Engineer stas@zend.com http://www.zend.com/ --- ## #28868 — Richard Lynch — 2007-04-16T20:23:15+00:00 URL: https://externals.io/message/28857#28868 · In reply to: #28857 You might also come at it from the other direction and detect/notify at some number smaller than the current hard limit, configurable in php.ini... This might play better with anything relying on the current behaviour. > I am interested in being able to trap the (currently) fatal error that > results when memory usage exceeds the defined memory limit. > > I was thinking it could work as follows: > > - in addition to a memory_limit configuration directive, there could > be a "memory_limit_grace" configuration directive. This gets stored in > the struct _zend_mm_heap, along with the limit. > > - Also added to struct _zend_mm_heap is a "initial limit reached" flag > > - When zend_mm_safe_error() in Zend/zend_alloc.c is invoked under the > current conditions (memory_limit exceeded), it sets the "initial limit > reached" flag, adjusts the heap limit to the "memory_limit_grace" > value and throws some non-fatal error (or an exception if it is > feasible from here.) > > - When zend_mm_safe_error() is invoked and the "initial limit reached" > flag is already set, it throws the fatal error exactly as it does now. > > This "grace period" would provide a way to gracefully exit when a > memory limit is reached, but also has the hard limit still enforced so > that code which is supposed to be gracefully exiting doesn't chew up > too much additional memory. > > Is it feasible to adjust the heap limit and throw a non-fatal error > from within zend_mm_safe_error()? > > David > > -- > > > > -- Some people have a "gift" link here. Know what I want? I want you to buy a CD from some indie artist. http://cdbaby.com/browse/from/lynch Yeah, I get a buck. So? --- ## #28876 — David Sklar — 2007-04-17T08:16:06+00:00 URL: https://externals.io/message/28857#28876 · In reply to: #28864 > > […] > > That could be a problem because it's very hard to know exact memory > requirements for PHP code for most people not intimately knowing the > engine internals, and also recovering from failed allocation usually is > not simple since management structures could be in half-done state. So > writing useful handler for such an error would be rather hard to do. > > Having said that, what exactly would you plan to do in this error handler? In my current situation, bail out, but gracefully -- perhaps just send a 302 to an error page and exit; maybe instead throw away the output buffer and then directly output some friendly error text. The idea would be not to recover from the failure and proceed with a separate script execution path that requires a lot less memory, but to have things fail more gracefully than just splatting a Fatal Error. So being able to calculate with great accuracy the memory requirements of particular bits of PHP code shouldn't be a prerequisite to making this sort of thing useful. That said, given the history of just about every other feature in PHP, it wouldn't be surprising for people to attempt to use this feature to do all sorts of crazy things after the soft memory limit has been reached, and in that case they'd want to be able to calculate the appropriate size of their grace period, but if they're going to go to such contortions, then that's what they're stuck with. David Richard Lynch wrote: > You might also come at it from the other direction and detect/notify > at some number smaller than the current hard limit, configurable in > php.ini... > > This might play better with anything relying on the current behaviour This is interesting and might actually take care of a lot of cases. I don't know enough about the existing memory manager to know if it would handle the case where a single new allocation would blow through both the (lower) soft limit and the hard limit at the same time. David --- ## #28891 — Stanislav Malyshev — 2007-04-17T17:29:44+00:00 URL: https://externals.io/message/28857#28891 · In reply to: #28876 > In my current situation, bail out, but gracefully -- perhaps just send > a 302 to an error page and exit; maybe instead throw away the output > buffer and then directly output some friendly error text. I think you can do it with shutdown handler, shouldn't it work? I.e. you can do it this way: 1. Install shutdown handler 2. Do something that takes a lot of memory 3. Memory is exhausted, script is terminated 4. The memory allocated by "something" is cleaned up, part by the engine, part manually 5. Shutdown handler does something like output "sorry, better luck next time" requiring minimal amount of memory. -- Stanislav Malyshev, Zend Products Engineer stas@zend.com http://www.zend.com/ --- ## #28908 — Brian Shire — 2007-04-17T19:57:47+00:00 URL: https://externals.io/message/28857#28908 · In reply to: #28857 I have code at Facebook that currently does something similar by catching fatal errors but there's a lot of gotcha's obviously that need to be handled properly when the engine is in this state. When possible we enable the printing of backtraces and other debug information, however you run the risk that instead of a nice backtrace you get a nice "Segmentation fault" error instead. This can be wrapped up in an extension and is all C code functionality, not exactly a soft memory limit so it's a little different than what's being discussed, but it might be another option to what you're looking for. I'm happy to share with a few initially who would find it useful/give feedback. -shire > I am interested in being able to trap the (currently) fatal error that > results when memory usage exceeds the defined memory limit. > > I was thinking it could work as follows: > > - in addition to a memory_limit configuration directive, there could > be a "memory_limit_grace" configuration directive. This gets stored in > the struct _zend_mm_heap, along with the limit. > > - Also added to struct _zend_mm_heap is a "initial limit reached" flag > > - When zend_mm_safe_error() in Zend/zend_alloc.c is invoked under the > current conditions (memory_limit exceeded), it sets the "initial limit > reached" flag, adjusts the heap limit to the "memory_limit_grace" > value and throws some non-fatal error (or an exception if it is > feasible from here.) > > - When zend_mm_safe_error() is invoked and the "initial limit reached" > flag is already set, it throws the fatal error exactly as it does now. > > This "grace period" would provide a way to gracefully exit when a > memory limit is reached, but also has the hard limit still enforced so > that code which is supposed to be gracefully exiting doesn't chew up > too much additional memory. > > Is it feasible to adjust the heap limit and throw a non-fatal error > from within zend_mm_safe_error()? > > David --- ## #28911 — Richard Lynch — 2007-04-17T21:56:19+00:00 URL: https://externals.io/message/28857#28911 · In reply to: #28876 > Richard Lynch wrote: > > […] > > This is interesting and might actually take care of a lot of cases. I > don't know enough about the existing memory manager to know if it > would handle the case where a single new allocation would blow through > both the (lower) soft limit and the hard limit at the same time. If an allocation blows through the hard limit, it should behave exactly as it does now, for BC, imho. If you let the developer decide on a "soft limit" or a percentage of the "hard limit" where some callback function happens, it's on their heads if they do something silly that rips right past their selected soft/hard limits. -- Some people have a "gift" link here. Know what I want? I want you to buy a CD from some indie artist. http://cdbaby.com/browse/from/lynch Yeah, I get a buck. So? --- ## #28914 — David Sklar — 2007-04-18T07:37:55+00:00 URL: https://externals.io/message/28857#28914 · In reply to: #28911 > > > […] > > If an allocation blows through the hard limit, it should behave > exactly as it does now, for BC, imho. > > If you let the developer decide on a "soft limit" or a percentage of > the "hard limit" where some callback function happens, it's on their > heads if they do something silly that rips right past their selected > soft/hard limits. Well, the BC case could be handled by doing what you suggest if the "grace limit" config option isn't defined. I think it's important, in the case that the grace limit is set up, to trigger its behavior even if a single allocation blows away both so that the recovery can be graceful. For example, if you're retrieving a feed to parse or the contents of a file and that feed or file is unexpectedly large so it blows through both memory limits. David --- ## #28915 — David Sklar — 2007-04-18T07:50:52+00:00 URL: https://externals.io/message/28857#28915 · In reply to: #28891 > > […] > > I think you can do it with shutdown handler, shouldn't it work? I.e. you > can do it this way: > 1. Install shutdown handler > 2. Do something that takes a lot of memory > 3. Memory is exhausted, script is terminated > 4. The memory allocated by "something" is cleaned up, part by the > engine, part manually > 5. Shutdown handler does something like output "sorry, better luck next > time" requiring minimal amount of memory. Ah, excellent, thanks. I'll take a look. This looks like what http://ez.no/doc/components/view/latest/(file)/Execution/ezcExecution.html does (Derick mentions this in the comments at http://php100.wordpress.com/2007/04/16/graceful-recovery/) David --- ## #28920 — Stanislav Malyshev — 2007-04-18T16:28:24+00:00 URL: https://externals.io/message/28857#28920 · In reply to: #28915 > Ah, excellent, thanks. I'll take a look. This looks like what > http://ez.no/doc/components/view/latest/(file)/Execution/ezcExecution.html > does (Derick mentions this in the comments at > http://php100.wordpress.com/2007/04/16/graceful-recovery/) Yes, that's basically where I've got the idea. Looks like that's the best one could do in pure PHP. -- Stanislav Malyshev, Zend Products Engineer stas@zend.com http://www.zend.com/