Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:53194
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: pass (pb1.pair.com: domain gmail.com designates 209.85.220.170 as permitted sender)
DomainKey-Status: bad
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:sender:in-reply-to:references:from:date
         :x-google-sender-auth:message-id:subject:to:cc:content-type
         :content-transfer-encoding;
        b=LNVJNjknCysYxIAoe58AX291NHWSdpJBxnUdd4rT1aAI2l7ZkdJsZqR8imjWwspNC7
         x0KrlLGluLF5B6K5cKSoaZ2cTrQc/6GPgucACEr0xtknX57VaIXRyf36PM4C2TIZnfpY
         a0WlPQeuSGMU1M2u+/F0eKZwNg+XLItLspMr8=
MIME-Version: 1.0
Sender: doctorrock83@gmail.com
In-Reply-To: <1307491219.23373.91.camel@guybrush>
References: <ff98f4d70b6306710582b9debdbb667f@naenius.com> <73A0E06E-55E0-4C48-B8F8-BCAFA3216892@bitextender.com>
 <f2ca38728993c1c1c581566c2dcc2b6b@naenius.com> <DCF7D272-575F-4463-97BD-6EDF3B823055@bitextender.com>
 <BANLkTikHceMABMur_Ais7A02JwjZ2Rvnmw@mail.gmail.com> <33db85fd3ebaa4f330120da5fa6445e1@naenius.com>
 <1C32E9BA-4858-4E24-89DA-0F36DFED352B@bitextender.com> <1307468695.2671.7.camel@chronos>
 <2B395C3E-A66B-4A0E-8F14-31B1154ECDC9@bitextender.com> <1307471210.2671.13.camel@chronos>
 <CC220629-F73D-4B88-BA44-06695B91B81C@bitextender.com> <1307491219.23373.91.camel@guybrush>
Date: Wed, 8 Jun 2011 10:23:24 +0200
Message-ID: <BANLkTi=jrZ=EM-CkHDZY6BZbLeuMnh6_Yw@mail.gmail.com>
To: =?ISO-8859-1?Q?Johannes_Schl=FCter?= <johannes@schlueters.de>
Cc: =?ISO-8859-1?Q?David_Z=FClke?= <david.zuelke@bitextender.com>, 
	Mike van Riel <mike.vanriel@naenius.com>, internals@lists.php.net
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Subject: Re: [PHP-DEV] Trying to find out where the memory went
From: jpauli@php.net (Julien Pauli)

I wrote about ZendMM some time ago
(http://julien-pauli.developpez.com/tutoriels/php/internals/zend-memory-man=
ager/)
, that's in french language ;-)

To shorten the conversation a little bit, I would suggest to trace the
memory with valgrind/massif. That's not too hard if you know what you
do, if not, then it can take some time.

Basically, Johannes gave some good hints; but memory management is a
hard task to compute and deal with, I suggest you dont try to figure
out how many memory it "would" take, as the computation is really too
hard to be accurate. Only memory debuggers will show you exactly what
happens.

BTW, there might be a little leak inside token_get_all() as it doesn't
seem to free memory it allocated. Not very easy to find as it plays
with lex scanner.

Julien.P

2011/6/8 Johannes Schl=FCter <johannes@schlueters.de>:
> On Tue, 2011-06-07 at 21:03 +0200, David Z=FClke wrote:
>> 144 (not 114!) bytes is for an integer; I'm not quite sure what the
>> overheads are for arrays, which token_get_all() produces in
>> abundance :) An empty array seems to occupy 312 bytes of memory.
>>
>> Also, strings have memory allocated in 8 byte increments as far as I
>> know, so "1" eats up 8 bytes, and "12345678901234567" will consume 24
>> bytes for the raw text, not 17.
>
> I'm too lazy to do the actual math (well best would be to do
> sizeof(zval), sizeof(HashTable), sizeof(Bucket) on your system) and
> there are few things to consider:
>
> =A0 =A0 =A0* The sizes are different from 32 bit and 64bit; with 64bit
> =A0 =A0 =A0 =A0there's a difference between Windows and Unix/Linux (on Wi=
n a
> =A0 =A0 =A0 =A0long will still be 32 bit, but pointers 64 bit, on Linux/U=
nix
> =A0 =A0 =A0 =A0both are 64bit)
> =A0 =A0 =A0* On some architectures memory segments have to be aligned in =
some
> =A0 =A0 =A0 =A0way which might waste memory
> =A0 =A0 =A0* As David mentioned HashTables (Arrays) are more complex.
> =A0 =A0 =A0* token_get_all() returns an array of (string | array of (long=
,
> =A0 =A0 =A0 =A0string, long) )
> =A0 =A0 =A0* A long takes sizeof(zval)
> =A0 =A0 =A0* A string takes sizeof(zval)+strlen()+1
> =A0 =A0 =A0* and array is a HashTable + space for buckets, this includes
> =A0 =A0 =A0 =A0place for some not used elements
> =A0 =A0 =A0* Each element inside the HT needs additional space for a Buck=
et
> =A0 =A0 =A0 =A0with some meta data
> =A0 =A0 =A0* While running your script you also keep the complete script =
file
> =A0 =A0 =A0 =A0in memory. You also keep some temporary parser data in mem=
ory
> =A0 =A0 =A0 =A0while the resulting array is being filled.
>
> In the end it's not fully trivial to gather the size needed. And I'm
> sure my list is missing loooots of things.
>
> http://schlueters.de/blog/archives/142-HashTables.html has an short
> introduction to HashTables. Skipping many of the details.
>
> johannes
>
>> David
>>
>>
>> On 07.06.2011, at 20:26, Mike van Riel wrote:
>>
>> > Am i then also correct to assume that the output of
>> > memory_get_peak_usage is used for determining the memory_limit?
>> >
>> > Also: after correcting with your new information (zval =3D 114 bytes
>> > instead of 68) I still have a rather large offset:
>> >
>> > =A0 =A0640952+2165950+114+(276697*114)+(276697*3*114)+2165950 =3D 1311=
46798 =3D
>> > 125M
>> >
>> > (not trying to be picky here; I just don't understand)
>> >
>> > _If_ my calculations are correct then a zval should be approx 216 byte=
s
>> > (excluding string contents):
>> >
>> > =A0 =A0((244000000-640952-2165950-2165950) / 4) / 276697 =3D 215.9647B
>> >
>> > Mike
>> >
>> > On Tue, 2011-06-07 at 19:50 +0200, David Z=FClke wrote:
>> >> memory_get_peak_usage() is the maximum amount of memory used by the V=
M of PHP (but not by some extensions for instance) up until the point where=
 that function is called. So the actual memory usage may be even higher IIR=
C. But yeah, you're basically right. I've explained in another message why =
it might be so much more than you expected (zval overhead, basically)
>> >>
>> >> David
>> >
>> >
>> >
>>
>
>
>