I am interested in being able to trap the (currently) fatal error that
results when memory usage exceeds the defined memory limit.
I was thinking it could work as follows:
-
in addition to a memory_limit configuration directive, there could
be a "memory_limit_grace" configuration directive. This gets stored in
the struct _zend_mm_heap, along with the limit. -
Also added to struct _zend_mm_heap is a "initial limit reached" flag
-
When zend_mm_safe_error() in Zend/zend_alloc.c is invoked under the
current conditions (memory_limit exceeded), it sets the "initial limit
reached" flag, adjusts the heap limit to the "memory_limit_grace"
value and throws some non-fatal error (or an exception if it is
feasible from here.) -
When zend_mm_safe_error() is invoked and the "initial limit reached"
flag is already set, it throws the fatal error exactly as it does now.
This "grace period" would provide a way to gracefully exit when a
memory limit is reached, but also has the hard limit still enforced so
that code which is supposed to be gracefully exiting doesn't chew up
too much additional memory.
Is it feasible to adjust the heap limit and throw a non-fatal error
from within zend_mm_safe_error()?
David
- in addition to a memory_limit configuration directive, there could
be a "memory_limit_grace" configuration directive. This gets stored in
the struct _zend_mm_heap, along with the limit.
That could be a problem because it's very hard to know exact memory
requirements for PHP code for most people not intimately knowing the
engine internals, and also recovering from failed allocation usually is
not simple since management structures could be in half-done state. So
writing useful handler for such an error would be rather hard to do.
Having said that, what exactly would you plan to do in this error handler?
Stanislav Malyshev, Zend Products Engineer
stas@zend.com http://www.zend.com/
- in addition to a memory_limit configuration directive, there could
be a "memory_limit_grace" configuration directive. This gets stored in
the struct _zend_mm_heap, along with the limit.That could be a problem because it's very hard to know exact memory
requirements for PHP code for most people not intimately knowing the
engine internals, and also recovering from failed allocation usually is
not simple since management structures could be in half-done state. So
writing useful handler for such an error would be rather hard to do.Having said that, what exactly would you plan to do in this error handler?
In my current situation, bail out, but gracefully -- perhaps just send
a 302 to an error page and exit; maybe instead throw away the output
buffer and then directly output some friendly error text.
The idea would be not to recover from the failure and proceed with a
separate script execution path that requires a lot less memory, but to
have things fail more gracefully than just splatting a Fatal Error. So
being able to calculate with great accuracy the memory requirements of
particular bits of PHP code shouldn't be a prerequisite to making this
sort of thing useful.
That said, given the history of just about every other feature in PHP,
it wouldn't be surprising for people to attempt to use this feature to
do all sorts of crazy things after the soft memory limit has been
reached, and in that case they'd want to be able to calculate the
appropriate size of their grace period, but if they're going to go to
such contortions, then that's what they're stuck with.
David
Richard Lynch ceo@l-i-e.com wrote:
You might also come at it from the other direction and detect/notify
at some number smaller than the current hard limit, configurable in
php.ini...This might play better with anything relying on the current behaviour
This is interesting and might actually take care of a lot of cases. I
don't know enough about the existing memory manager to know if it
would handle the case where a single new allocation would blow through
both the (lower) soft limit and the hard limit at the same time.
David
In my current situation, bail out, but gracefully -- perhaps just send
a 302 to an error page and exit; maybe instead throw away the output
buffer and then directly output some friendly error text.
I think you can do it with shutdown handler, shouldn't it work? I.e. you
can do it this way:
- Install shutdown handler
- Do something that takes a lot of memory
- Memory is exhausted, script is terminated
- The memory allocated by "something" is cleaned up, part by the
engine, part manually - Shutdown handler does something like output "sorry, better luck next
time" requiring minimal amount of memory.
--
Stanislav Malyshev, Zend Products Engineer
stas@zend.com http://www.zend.com/
In my current situation, bail out, but gracefully -- perhaps just send
a 302 to an error page and exit; maybe instead throw away the output
buffer and then directly output some friendly error text.I think you can do it with shutdown handler, shouldn't it work? I.e. you
can do it this way:
- Install shutdown handler
- Do something that takes a lot of memory
- Memory is exhausted, script is terminated
- The memory allocated by "something" is cleaned up, part by the
engine, part manually- Shutdown handler does something like output "sorry, better luck next
time" requiring minimal amount of memory.
Ah, excellent, thanks. I'll take a look. This looks like what
http://ez.no/doc/components/view/latest/(file)/Execution/ezcExecution.html
does (Derick mentions this in the comments at
http://php100.wordpress.com/2007/04/16/graceful-recovery/)
David
Ah, excellent, thanks. I'll take a look. This looks like what
http://ez.no/doc/components/view/latest/(file)/Execution/ezcExecution.html
does (Derick mentions this in the comments at
http://php100.wordpress.com/2007/04/16/graceful-recovery/)
Yes, that's basically where I've got the idea. Looks like that's the
best one could do in pure PHP.
Stanislav Malyshev, Zend Products Engineer
stas@zend.com http://www.zend.com/
Richard Lynch ceo@l-i-e.com wrote:
You might also come at it from the other direction and detect/notify
at some number smaller than the current hard limit, configurable in
php.ini...This might play better with anything relying on the current
behaviourThis is interesting and might actually take care of a lot of cases. I
don't know enough about the existing memory manager to know if it
would handle the case where a single new allocation would blow through
both the (lower) soft limit and the hard limit at the same time.
If an allocation blows through the hard limit, it should behave
exactly as it does now, for BC, imho.
If you let the developer decide on a "soft limit" or a percentage of
the "hard limit" where some callback function happens, it's on their
heads if they do something silly that rips right past their selected
soft/hard limits.
--
Some people have a "gift" link here.
Know what I want?
I want you to buy a CD from some indie artist.
http://cdbaby.com/browse/from/lynch
Yeah, I get a buck. So?
Richard Lynch ceo@l-i-e.com wrote:
You might also come at it from the other direction and detect/notify
at some number smaller than the current hard limit, configurable in
php.ini...This might play better with anything relying on the current
behaviourThis is interesting and might actually take care of a lot of cases. I
don't know enough about the existing memory manager to know if it
would handle the case where a single new allocation would blow through
both the (lower) soft limit and the hard limit at the same time.If an allocation blows through the hard limit, it should behave
exactly as it does now, for BC, imho.If you let the developer decide on a "soft limit" or a percentage of
the "hard limit" where some callback function happens, it's on their
heads if they do something silly that rips right past their selected
soft/hard limits.
Well, the BC case could be handled by doing what you suggest if the
"grace limit" config option isn't defined.
I think it's important, in the case that the grace limit is set up, to
trigger its behavior even if a single allocation blows away both so
that the recovery can be graceful. For example, if you're retrieving a
feed to parse or the contents of a file and that feed or file is
unexpectedly large so it blows through both memory limits.
David
You might also come at it from the other direction and detect/notify
at some number smaller than the current hard limit, configurable in
php.ini...
This might play better with anything relying on the current behaviour.
I am interested in being able to trap the (currently) fatal error that
results when memory usage exceeds the defined memory limit.I was thinking it could work as follows:
in addition to a memory_limit configuration directive, there could
be a "memory_limit_grace" configuration directive. This gets stored in
the struct _zend_mm_heap, along with the limit.Also added to struct _zend_mm_heap is a "initial limit reached" flag
When zend_mm_safe_error() in Zend/zend_alloc.c is invoked under the
current conditions (memory_limit exceeded), it sets the "initial limit
reached" flag, adjusts the heap limit to the "memory_limit_grace"
value and throws some non-fatal error (or an exception if it is
feasible from here.)When zend_mm_safe_error() is invoked and the "initial limit reached"
flag is already set, it throws the fatal error exactly as it does now.This "grace period" would provide a way to gracefully exit when a
memory limit is reached, but also has the hard limit still enforced so
that code which is supposed to be gracefully exiting doesn't chew up
too much additional memory.Is it feasible to adjust the heap limit and throw a non-fatal error
from within zend_mm_safe_error()?David
--
--
Some people have a "gift" link here.
Know what I want?
I want you to buy a CD from some indie artist.
http://cdbaby.com/browse/from/lynch
Yeah, I get a buck. So?
I have code at Facebook that currently does something similar by
catching fatal errors but there's a lot of gotcha's obviously that
need to be handled properly when the engine is in this state. When
possible we enable the printing of backtraces and other debug
information, however you run the risk that instead of a nice
backtrace you get a nice "Segmentation fault" error instead. This
can be wrapped up in an extension and is all C code functionality,
not exactly a soft memory limit so it's a little different than
what's being discussed, but it might be another option to what you're
looking for. I'm happy to share with a few initially who would find
it useful/give feedback.
-shire
I am interested in being able to trap the (currently) fatal error that
results when memory usage exceeds the defined memory limit.I was thinking it could work as follows:
in addition to a memory_limit configuration directive, there could
be a "memory_limit_grace" configuration directive. This gets stored in
the struct _zend_mm_heap, along with the limit.Also added to struct _zend_mm_heap is a "initial limit reached" flag
When zend_mm_safe_error() in Zend/zend_alloc.c is invoked under the
current conditions (memory_limit exceeded), it sets the "initial limit
reached" flag, adjusts the heap limit to the "memory_limit_grace"
value and throws some non-fatal error (or an exception if it is
feasible from here.)When zend_mm_safe_error() is invoked and the "initial limit reached"
flag is already set, it throws the fatal error exactly as it does now.This "grace period" would provide a way to gracefully exit when a
memory limit is reached, but also has the hard limit still enforced so
that code which is supposed to be gracefully exiting doesn't chew up
too much additional memory.Is it feasible to adjust the heap limit and throw a non-fatal error
from within zend_mm_safe_error()?David