# Trying to find out where the memory went

- Thread: https://externals.io/message/52894
- Original archive: https://news-web.php.net/php.internals/52894
- Started by Mike van Riel on 2011-06-04T21:38:00+00:00
- Messages: 18
- Reply by email to internals@lists.php.net
- More for agents: search with https://externals.io/search.md?q={query} · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt

---

## #52894 — Mike van Riel — 2011-06-04T21:38:00+00:00

URL: https://externals.io/message/52894#52894

Dear Internals,

During development of DocBlox I encountered a (for me) unusual 
situation
with regards to memory usage.

I hope you can shed some light on this for me as I do not understand.

The situations is as follows:

I have a php file containing about 53 KLOC (including whitespace and
comments), which is about 2.1MB in size. When I execute the
memory_get_peak_usage after running the token_get_all method on its
content it reports that 232MB of RAM have been used in the process.

I am having trouble understanding how 244003984B (232MB) RAM could be
used.

The following is what I have calculated:
* 640.952B to start with (measured);
* 2.1MB to load the file contents into memory using file_get_contents
* 68 bytes for the resulting array
* 68 bytes for each child array representing a token
* 68 bytes for each element in a token array (which can be either 1 or
3, depending whether it is actually a token or literal)
* 2.1MB in total for the string contents of the token literals /
contents (equivalent to the byte size of the file)

I have used the count method to retrieve the number of tokens (276697)
and come to the following sum (everything is retrieved and calculated 
in
bytes):

640952+2165950+68+(276697*68)+(276697*3*68)+2165950=80234436 = 76M

This is a worst case formula where I assume that every token in the
array consists of 3 elements.

Based on this calculation I would be missing 156MB of memory; anybody
know where that went?

I used the following snippet of code for my tests:

     var_dump(memory_get_peak_usage());
     $tokens = token_get_all(file_get_contents('<PATH>'));
     var_dump(count($tokens));
     var_dump(memory_get_peak_usage());

I hope this mail did not scare anyone ;)

Kind regards,

Mike van Riel

---

## #52895 — david.zuelke@bitextender.com — 2011-06-04T21:51:50+00:00

URL: https://externals.io/message/52894#52895 · In reply to: #52894

What does

var_dump(memory_get_peak_usage());
token_get_all(file_get_contents('<PATH>'));
var_dump(memory_get_peak_usage());

get you?

David




> Dear Internals,
> 
> During development of DocBlox I encountered a (for me) unusual situation
> with regards to memory usage.
> 
> I hope you can shed some light on this for me as I do not understand.
> 
> The situations is as follows:
> 
> I have a php file containing about 53 KLOC (including whitespace and
> comments), which is about 2.1MB in size. When I execute the
> memory_get_peak_usage after running the token_get_all method on its
> content it reports that 232MB of RAM have been used in the process.
> 
> I am having trouble understanding how 244003984B (232MB) RAM could be
> used.
> 
> The following is what I have calculated:
> * 640.952B to start with (measured);
> * 2.1MB to load the file contents into memory using file_get_contents
> * 68 bytes for the resulting array
> * 68 bytes for each child array representing a token
> * 68 bytes for each element in a token array (which can be either 1 or
> 3, depending whether it is actually a token or literal)
> * 2.1MB in total for the string contents of the token literals /
> contents (equivalent to the byte size of the file)
> 
> I have used the count method to retrieve the number of tokens (276697)
> and come to the following sum (everything is retrieved and calculated in
> bytes):
> 
> 640952+2165950+68+(276697*68)+(276697*3*68)+2165950=80234436 = 76M
> 
> This is a worst case formula where I assume that every token in the
> array consists of 3 elements.
> 
> Based on this calculation I would be missing 156MB of memory; anybody
> know where that went?
> 
> I used the following snippet of code for my tests:
> 
>    var_dump(memory_get_peak_usage());
>    $tokens = token_get_all(file_get_contents('<PATH>'));
>    var_dump(count($tokens));
>    var_dump(memory_get_peak_usage());
> 
> I hope this mail did not scare anyone ;)
> 
> Kind regards,
> 
> Mike van Riel

---

## #52896 — Mike van Riel — 2011-06-04T22:01:18+00:00

URL: https://externals.io/message/52894#52896 · In reply to: #52895

Hey David,

That gives me the following output:

     int(640720)
     int(244001144)

Mike


> What does
>
> var_dump(memory_get_peak_usage());
> token_get_all(file_get_contents('<PATH>'));
> var_dump(memory_get_peak_usage());
>
> get you?
>
> David

---

## #52927 — david.zuelke@bitextender.com — 2011-06-05T12:01:52+00:00

URL: https://externals.io/message/52894#52927 · In reply to: #52896

Smells like a memory leak if `gc_collect_cycles()` doesn't fix it.

David




> Hey David,
> 
> That gives me the following output:
> 
>    int(640720)
>    int(244001144)
> 
> Mike
> 
> 
> > […]

---

## #52933 — Julien Pauli — 2011-06-05T13:36:43+00:00

URL: https://externals.io/message/52894#52933 · In reply to: #52927

Seems like leak.

Try disabling ZendMM to see if something noticeable happens (memory
peak should be lower).
USE_ZEND_ALLOC=0

Cheers,
Julien

On Sun, Jun 5, 2011 at 2:01 PM, David Zülke
<david.zuelke@bitextender.com> wrote:
> Smells like a memory leak if `gc_collect_cycles()` doesn't fix it.
>
> David
>
>
> 
>
> > […]

---

## #53076 — Mike van Riel — 2011-06-06T20:30:51+00:00

URL: https://externals.io/message/52894#53076 · In reply to: #52933

David and Pauli,

When I change the test script to:

     var_dump(memory_get_peak_usage());
     `gc_collect_cycles()`;
     token_get_all(file_get_contents(<FILE>));
     `gc_collect_cycles()`;
     var_dump(memory_get_peak_usage());

And execute the following bash line preceding:

     export USE_ZEND_ALLOC=0

I get the following output:

     int(8240)
     int(8240)

When I remove the gc_collect_cycles I get the same result.
Even assigning the results to a variable do not increase the peak 
memory.

FYI: When I change the argument of memory_get_peak_usage to 'true', I 
get the following results:

     int(262144)
     int(262144)

This amount is astoundingly less than the previous conclusions and less 
than my own calculations would show.
Of course this leads me to the following questions:

1. Does it hurt to disable the Zend MM?
2. Can it be done from inside a PHP Script?
3. Why is the memory consumption so much lower, even lower than my 
calculations?

I assume it is a good thing to at least try to create an easy way to 
reproduce the issue (cannot include my test file) and create a bug 
report about this :)

Thank you for your assistance thus far.

Mike


> Seems like leak.
>
> Try disabling ZendMM to see if something noticeable happens (memory
> peak should be lower).
> USE_ZEND_ALLOC=0
>
> Cheers,
> Julien
>
> On Sun, Jun 5, 2011 at 2:01 PM, David Zülke
> <david.zuelke@bitextender.com> wrote:
> > […]

---

## #53102 — johannes@schlueters.de — 2011-06-06T23:05:12+00:00

URL: https://externals.io/message/52894#53102 · In reply to: #53076

> 1. Does it hurt to disable the Zend MM?
> 2. Can it be done from inside a PHP Script?
> 3. Why is the memory consumption so much lower, even lower than my 
> calculations?

When you disable Zend MM PHP will not use it but directly the system's
allocator, which won't be counted by ZendMM therefore. you therefore
loose all PHP-specific allocation improvements and the memory leak
protections. Nothing you'd actually want ;-)

Disabling the ZendMM is mostly useful for using memory debuggers (incl.
valgrind) to do further checks.

> I assume it is a good thing to at least try to create an easy way to 
> reproduce the issue (cannot include my test file) and create a bug 
> report about this :)

Reproducible bug reports are always a good thing.

johannes

---

## #53145 — david.zuelke@bitextender.com — 2011-06-07T14:28:25+00:00

URL: https://externals.io/message/52894#53145 · In reply to: #53076

Please test the exact thing I suggested :)

var_dump(memory_get_usage());
token_get_all(file_get_contents(<FILE>));
 `gc_collect_cycles()`;
var_dump(memory_get_usage());

 `memory_get_peak_usage()` is irrelevant, and USE_ZEND_ALLOC won't give accurate results anymore when looking at memory usage.

If the above gives the same numbers you got initially, then there's a memleak in `token_get_all()`.

David




> David and Pauli,
> 
> When I change the test script to:
> 
>    var_dump(memory_get_peak_usage());
>    `gc_collect_cycles()`;
>    token_get_all(file_get_contents(<FILE>));
>    `gc_collect_cycles()`;
>    var_dump(memory_get_peak_usage());
> 
> And execute the following bash line preceding:
> 
>    export USE_ZEND_ALLOC=0
> 
> I get the following output:
> 
>    int(8240)
>    int(8240)
> 
> When I remove the gc_collect_cycles I get the same result.
> Even assigning the results to a variable do not increase the peak memory.
> 
> FYI: When I change the argument of memory_get_peak_usage to 'true', I get the following results:
> 
>    int(262144)
>    int(262144)
> 
> This amount is astoundingly less than the previous conclusions and less than my own calculations would show.
> Of course this leads me to the following questions:
> 
> 1. Does it hurt to disable the Zend MM?
> 2. Can it be done from inside a PHP Script?
> 3. Why is the memory consumption so much lower, even lower than my calculations?
> 
> I assume it is a good thing to at least try to create an easy way to reproduce the issue (cannot include my test file) and create a bug report about this :)
> 
> Thank you for your assistance thus far.
> 
> Mike
> 
> 
> > […]

---

## #53146 — Ferenc Kovacs — 2011-06-07T14:34:21+00:00

URL: https://externals.io/message/52894#53146 · In reply to: #53145

On Tue, Jun 7, 2011 at 4:28 PM, David Zülke <david.zuelke@bitextender.com>wrote:

> Please test the exact thing I suggested :)
>

AFAIK he did.
"   int(640720)
   int(244001144)"
except if you suggested something else off-list.

Tyrael

---

## #53147 — david.zuelke@bitextender.com — 2011-06-07T14:38:28+00:00

URL: https://externals.io/message/52894#53147 · In reply to: #53146

Damn I'm an idiot. I meant `memory_get_usage()` all along. Sorry Mike. Then it'll make sense... `memory_get_usage()`, but a `gc_collect_cycles()` before the second call.

So, my first email should have had this code in it:

var_dump(memory_get_usage());
token_get_all(file_get_contents('<PATH>'));
var_dump(memory_get_usage());

And then, a comparison to this would be useful:

var_dump(memory_get_usage());
token_get_all(file_get_contents('<PATH>'));
 `gc_collect_cycles()`;
var_dump(memory_get_usage());

David





> On Tue, Jun 7, 2011 at 4:28 PM, David Zülke <david.zuelke@bitextender.com>wrote:
> 
> > […]
> 
> AFAIK he did.
> "   int(640720)
>   int(244001144)"
> except if you suggested something else off-list.
> 
> Tyrael

---

## #53148 — david.zuelke@bitextender.com — 2011-06-07T14:41:59+00:00

URL: https://externals.io/message/52894#53148 · In reply to: #52894

One thing to keep in mind of course is that each zval incurs an overhead. $x = 1; requires 144 bytes of memory in total IIRC.

David




> Dear Internals,
> 
> During development of DocBlox I encountered a (for me) unusual situation
> with regards to memory usage.
> 
> I hope you can shed some light on this for me as I do not understand.
> 
> The situations is as follows:
> 
> I have a php file containing about 53 KLOC (including whitespace and
> comments), which is about 2.1MB in size. When I execute the
> memory_get_peak_usage after running the token_get_all method on its
> content it reports that 232MB of RAM have been used in the process.
> 
> I am having trouble understanding how 244003984B (232MB) RAM could be
> used.
> 
> The following is what I have calculated:
> * 640.952B to start with (measured);
> * 2.1MB to load the file contents into memory using file_get_contents
> * 68 bytes for the resulting array
> * 68 bytes for each child array representing a token
> * 68 bytes for each element in a token array (which can be either 1 or
> 3, depending whether it is actually a token or literal)
> * 2.1MB in total for the string contents of the token literals /
> contents (equivalent to the byte size of the file)
> 
> I have used the count method to retrieve the number of tokens (276697)
> and come to the following sum (everything is retrieved and calculated in
> bytes):
> 
> 640952+2165950+68+(276697*68)+(276697*3*68)+2165950=80234436 = 76M
> 
> This is a worst case formula where I assume that every token in the
> array consists of 3 elements.
> 
> Based on this calculation I would be missing 156MB of memory; anybody
> know where that went?
> 
> I used the following snippet of code for my tests:
> 
>    var_dump(memory_get_peak_usage());
>    $tokens = token_get_all(file_get_contents('<PATH>'));
>    var_dump(count($tokens));
>    var_dump(memory_get_peak_usage());
> 
> I hope this mail did not scare anyone ;)
> 
> Kind regards,
> 
> Mike van Riel

---

## #53166 — Mike van Riel — 2011-06-07T17:44:55+00:00

URL: https://externals.io/message/52894#53166 · In reply to: #53145

I have ran the script that you provided and got the following results:

    int(635192)
    int(635944)

Which is far less than the peak memory result.

I use memory_get_peak_usage to measure what the worst case memory output
is in my application. I expect this to be the actual memory used (and
thus when the server starts swapping if this number exceeds the physical
memory).

Is my assertion about the meaning of memory_get_peak_usage incorrect?

Mike  


> Please test the exact thing I suggested :)
> 
> var_dump(memory_get_usage());
> token_get_all(file_get_contents(<FILE>));
> `gc_collect_cycles()`;
> var_dump(memory_get_usage());
> 
> `memory_get_peak_usage()` is irrelevant, and USE_ZEND_ALLOC won't give accurate results anymore when looking at memory usage.
> 
> If the above gives the same numbers you got initially, then there's a memleak in `token_get_all()`.
> 
> David
> 
> 
> 
> 
> > […]

---

## #53167 — Mike van Riel — 2011-06-07T17:49:08+00:00

URL: https://externals.io/message/52894#53167 · In reply to: #53166

Before I forget; without gc_collect_cycles I get the following output
using memory_get_usage instead of memory_get_peak_usage:

    int(634640)
    int(635392)

Mike


> I have ran the script that you provided and got the following results:
> 
>     int(635192)
>     int(635944)
> 
> Which is far less than the peak memory result.
> 
> I use memory_get_peak_usage to measure what the worst case memory output
> is in my application. I expect this to be the actual memory used (and
> thus when the server starts swapping if this number exceeds the physical
> memory).
> 
> Is my assertion about the meaning of memory_get_peak_usage incorrect?
> 
> Mike  
> 
> 
> > […]

---

## #53168 — david.zuelke@bitextender.com — 2011-06-07T17:50:28+00:00

URL: https://externals.io/message/52894#53168 · In reply to: #53166

memory_get_peak_usage() is the maximum amount of memory used by the VM of PHP (but not by some extensions for instance) up until the point where that function is called. So the actual memory usage may be even higher IIRC. But yeah, you're basically right. I've explained in another message why it might be so much more than you expected (zval overhead, basically)

David




> I have ran the script that you provided and got the following results:
> 
>    int(635192)
>    int(635944)
> 
> Which is far less than the peak memory result.
> 
> I use memory_get_peak_usage to measure what the worst case memory output
> is in my application. I expect this to be the actual memory used (and
> thus when the server starts swapping if this number exceeds the physical
> memory).
> 
> Is my assertion about the meaning of memory_get_peak_usage incorrect?
> 
> Mike  
> 
> 
> > […]

---

## #53172 — Mike van Riel — 2011-06-07T18:26:50+00:00

URL: https://externals.io/message/52894#53172 · In reply to: #53168

Am i then also correct to assume that the output of
memory_get_peak_usage is used for determining the memory_limit?

Also: after correcting with your new information (zval = 114 bytes
instead of 68) I still have a rather large offset:

    640952+2165950+114+(276697*114)+(276697*3*114)+2165950 = 131146798 =
125M

(not trying to be picky here; I just don't understand)

_If_ my calculations are correct then a zval should be approx 216 bytes
(excluding string contents):

    ((244000000-640952-2165950-2165950) / 4) / 276697 = 215.9647B

Mike


> `memory_get_peak_usage()` is the maximum amount of memory used by the VM of PHP (but not by some extensions for instance) up until the point where that function is called. So the actual memory usage may be even higher IIRC. But yeah, you're basically right. I've explained in another message why it might be so much more than you expected (zval overhead, basically)
> 
> David

---

## #53174 — david.zuelke@bitextender.com — 2011-06-07T19:03:07+00:00

URL: https://externals.io/message/52894#53174 · In reply to: #53172

144 (not 114!) bytes is for an integer; I'm not quite sure what the overheads are for arrays, which `token_get_all()` produces in abundance :) An empty array seems to occupy 312 bytes of memory.

Also, strings have memory allocated in 8 byte increments as far as I know, so "1" eats up 8 bytes, and "12345678901234567" will consume 24 bytes for the raw text, not 17.

David




> Am i then also correct to assume that the output of
> memory_get_peak_usage is used for determining the memory_limit?
> 
> Also: after correcting with your new information (zval = 114 bytes
> instead of 68) I still have a rather large offset:
> 
>    640952+2165950+114+(276697*114)+(276697*3*114)+2165950 = 131146798 =
> 125M
> 
> (not trying to be picky here; I just don't understand)
> 
> _If_ my calculations are correct then a zval should be approx 216 bytes
> (excluding string contents):
> 
>    ((244000000-640952-2165950-2165950) / 4) / 276697 = 215.9647B
> 
> Mike
> 
> 
> > […]

---

## #53191 — johannes@schlueters.de — 2011-06-08T00:00:19+00:00

URL: https://externals.io/message/52894#53191 · In reply to: #53174

> 144 (not 114!) bytes is for an integer; I'm not quite sure what the
> overheads are for arrays, which `token_get_all()` produces in
> abundance :) An empty array seems to occupy 312 bytes of memory.
> 
> Also, strings have memory allocated in 8 byte increments as far as I
> know, so "1" eats up 8 bytes, and "12345678901234567" will consume 24
> bytes for the raw text, not 17.

I'm too lazy to do the actual math (well best would be to do
sizeof(zval), sizeof(HashTable), sizeof(Bucket) on your system) and
there are few things to consider:

      * The sizes are different from 32 bit and 64bit; with 64bit
        there's a difference between Windows and Unix/Linux (on Win a
        long will still be 32 bit, but pointers 64 bit, on Linux/Unix
        both are 64bit)
      * On some architectures memory segments have to be aligned in some
        way which might waste memory
      * As David mentioned HashTables (Arrays) are more complex.
      * `token_get_all()` returns an array of (string | array of (long,
        string, long) )
      * A long takes sizeof(zval)
      * A string takes sizeof(zval)+strlen()+1
      * and array is a HashTable + space for buckets, this includes
        place for some not used elements
      * Each element inside the HT needs additional space for a Bucket
        with some meta data
      * While running your script you also keep the complete script file
        in memory. You also keep some temporary parser data in memory
        while the resulting array is being filled.

In the end it's not fully trivial to gather the size needed. And I'm
sure my list is missing loooots of things.

http://schlueters.de/blog/archives/142-HashTables.html has an short
introduction to HashTables. Skipping many of the details.

johannes

> David
> 
> 
> 
> 
> > […]

---

## #53194 — Julien Pauli — 2011-06-08T08:23:24+00:00

URL: https://externals.io/message/52894#53194 · In reply to: #53191

I wrote about ZendMM some time ago
(http://julien-pauli.developpez.com/tutoriels/php/internals/zend-memory-manager/)
, that's in french language ;-)

To shorten the conversation a little bit, I would suggest to trace the
memory with valgrind/massif. That's not too hard if you know what you
do, if not, then it can take some time.

Basically, Johannes gave some good hints; but memory management is a
hard task to compute and deal with, I suggest you dont try to figure
out how many memory it "would" take, as the computation is really too
hard to be accurate. Only memory debuggers will show you exactly what
happens.

BTW, there might be a little leak inside `token_get_all()` as it doesn't
seem to free memory it allocated. Not very easy to find as it plays
with lex scanner.

Julien.P

2011/6/8 Johannes Schlüter <johannes@schlueters.de>:
> 
> > […]
>
> I'm too lazy to do the actual math (well best would be to do
> sizeof(zval), sizeof(HashTable), sizeof(Bucket) on your system) and
> there are few things to consider:
>
>      * The sizes are different from 32 bit and 64bit; with 64bit
>        there's a difference between Windows and Unix/Linux (on Win a
>        long will still be 32 bit, but pointers 64 bit, on Linux/Unix
>        both are 64bit)
>      * On some architectures memory segments have to be aligned in some
>        way which might waste memory
>      * As David mentioned HashTables (Arrays) are more complex.
>      * `token_get_all()` returns an array of (string | array of (long,
>        string, long) )
>      * A long takes sizeof(zval)
>      * A string takes sizeof(zval)+strlen()+1
>      * and array is a HashTable + space for buckets, this includes
>        place for some not used elements
>      * Each element inside the HT needs additional space for a Bucket
>        with some meta data
>      * While running your script you also keep the complete script file
>        in memory. You also keep some temporary parser data in memory
>        while the resulting array is being filled.
>
> In the end it's not fully trivial to gather the size needed. And I'm
> sure my list is missing loooots of things.
>
> http://schlueters.de/blog/archives/142-HashTables.html has an short
> introduction to HashTables. Skipping many of the details.
>
> johannes
>
> > […]