Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:107310
MIME-Version: 1.0
References: <CAPrKfG5pbretCasiA7zaoRVDwq3gbqqyYZ5uP7YfLAHNQ36Y7A@mail.gmail.com>
 <CAEYWF=40YHyQnA7v=mSb1zDdniQ1yfseXS6gPGH5npV8r19PmA@mail.gmail.com>
In-Reply-To: <CAEYWF=40YHyQnA7v=mSb1zDdniQ1yfseXS6gPGH5npV8r19PmA@mail.gmail.com>
Date: Tue, 24 Sep 2019 02:39:45 -0700
Message-ID: <CAPrKfG7oOmWGwotufgbj0kbp6hHmLmtGdaBSS8M_-1R32CskGA@mail.gmail.com>
To: bishop@php.net
Cc: "internals@lists.php.net" <internals@lists.php.net>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Subject: Re: [PHP-DEV] Shutdown Memory Allowance (aka Soft Memory Limit)
From: sarkedev@gmail.com (Peter Stalman)

On Tue, Sep 24, 2019 at 12:01 AM Bishop Bettini <bishop@php.net> wrote:
>
> On Tue, Sep 24, 2019 at 2:26 AM Peter Stalman <sarkedev@gmail.com> wrote:
>>
>> When PHP runs out of memory, a fatal error is triggered and whatever shu=
tdown
>> functions or error handlers take over.
>>
>> However, in the case of error logging, or just logging in general, there=
 often
>> needs to be additional memory used to accommodate the final logging proc=
ess.
>>
>> This can sort of be accomplished in userland a few ways:
>>
>> 1. Pre-allocating memory in a variable, such as the Yii2 error handler
>> (http://bit.ly/2kLnpd2), but this requires wasting memory on every reque=
st.
>>
>> 2. Continuously checking the memory usage, but this increases code compl=
exity
>> needlessly and also wastes resources with constant checking.
>>
>> 3. A second process with its own memory allowance, but this also increas=
es
>> complexity and transferring the required data for logging would require
>> serialization without using additional memory.  I'm not sure how this wo=
uld be
>> accomplished.
>>
>> So I would like to suggest an option for setting a shutdown memory allow=
ance,
>> which would be the amount of additional memory allowed to be used by any
>> registered error handlers or shutdown functions.
>>
>> I think a C implementation of this in PHP would be far more efficient th=
an the
>> userland implementionations I mentioned.
>
>
> Memory parachutes help when against a hard limit, but I'm not sure they'r=
e applicable in the soft limit scenario you describe. When the OOM conditio=
n fires, PHP resets its soft tracking of memory. You are free to allocate a=
s much on the stack of the shutdown function, up to the soft limit again.
>
> Here[1] we see the shutdown function called because of OOM, then we alloc=
ate a local variable that consumes less than the soft limit (which is ok), =
while here[2] we see the shutdown function itself is constrained to the sof=
t limit (it's killed b/c of OOM).
>
> Perhaps I am misunderstanding the scenario. Could you elaborate further, =
perhaps provide a concrete example demonstrating where a parachute would be=
 needed?
>
> [1]:https://3v4l.org/bXMlX
> [2]:https://3v4l.org/HDkRc

Hi Bishop,

Thanks for your reply.  However, I do not think your statement that the mem=
ory
limit resets is correct.  At least, not in a practical real-world sense, as=
 the
logger (or perhaps other shutdown functions as well) need to be able to acc=
ess

Your examples do not really work for a few reasons.  I don't think 2048 is =
a
valid value for the memory limit, as the output in your examples show it is
2 mb (which I believe is the minimum).  The docs also state the number
is in bytes.

Even so, this is not a real-world number, and the way PHP handles it's memo=
ry I
don't think using only one variable will give us a real-world example eithe=
r.
As you can see from your first example, PHP tries to assign an additional 2=
 mb
just to add another element to the array.  I believe there is dome doubling
going on with the memory allocation, when increasing the array size, so it
doesn't have to be done on every append.  Since that additional assignment
failed, the actual memory usage goes back to what it was before the assignm=
ent,
which means any subsequent memory usage can be up to the 2 mb that you trie=
d to
assign.  That's why the smaller loop in the shutdown function works, as the
original variable is probably only about 1 mb in size.

The fact that your second example tries to assign only 516 kb when it fails=
 at
the memory limit supports this, and also suggests that that array was much
smaller when it hit the limit.  If it could assign as much as the original,=
 then
it would have tried to increase the memory allocation by about the same amo=
unt.

I've put together my own example as you suggested, using an array of string=
s
instead.  This would reduce the incremental memory allocations as there is =
no
longer a single giant variable being doubled.  It fails when trying to assi=
gn
416 kb.

https://3v4l.org/BqfvU

If you are correct then the shutdown function should be able to recreate th=
e
original array and not run out of memory.  However, as you can see, it fail=
s
after only assigning 2 of the 100 elements of the original array, which sug=
gests
the shutdown function is using the same memory and it's limit.

Thanks,
Peter