Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:14061
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Message-ID: <5.1.0.14.2.20041213142006.019fca70@localhost>
Date: Mon, 13 Dec 2004 14:22:17 -0800
To: "Matt W" <php_lists@realplain.com>,"Derick Rethans" <derick@php.net>,
 <internals@lists.php.net>
In-Reply-To: <00c501c4dea7$b66ae520$0100a8c0@pc1>
References: <002e01c4dde1$b69fe9d0$0100a8c0@pc1>
 <Pine.LNX.4.58.0412091252210.12298@localhost>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"; format=flowed
Subject: Re: [PHP-DEV] Memory needed for arrays? And major performance
  problems filling an array
From: andi@zend.com (Andi Gutmans)

Hi Matt,

The problem is most probably because the Zend array hash table reallocs 
everytime it starts to hold too many elements. The larger it gets, the less 
likely it manages to reallocate in the same memory area and then it has to 
allocate a new big memory chunk and memcpy() all the old data to it. This 
is probably where you are feeling the hit.

Andi

At 05:02 AM 12/10/2004 -0600, Matt W wrote:
>----- Original Message -----
>From: "Derick Rethans"
>Sent: Thursday, December 09, 2004
>
>
> > On Thu, 9 Dec 2004, Matt W wrote:
> >
> > > Before I describe my issue, I had been wondering how I can
>tell/estimate
> > > how much memory PHP will use for arrays?  Is there at least a rough
>formula
> > > someone can tell me?
> >
> > 42 * number of array elements + size of keys + data
>
>Thanks Derick. Is that exact or just an estimate?
>
>So PHP needs 42 bytes for each array element, OK. For a multi-dimensional
>array, I assume the "number of elements" is the total from all dimensions
>added up. e.g. $foo[] = array(123, 456); is 3 elements (1 + 2)?
>
>
> > > This is disappointing, especially being caused by an array that doesn't
> > > seem too outrageously large (and I'd like to go larger). :-( It's going
>to
> > > suck if I can't come up with a way to get it faster... Is there
>anything
> > > that would be different on a version newer than 4.3.6, as I didn't try
>any
> > > yet? Or if it's some Windows thing, then I don't care as much.
> >
> > > Comments, suggestions?
> >
> > Make a custom C extension.. I don't know why you needs arrays this
> > large, but it sounds like a application design flaw.
>
>I need arrays this large because it's building a search index, which could
>have 100s of millions of entries; and I want to buffer many entries in
>arrays before doing writes... So even these "large" arrays are just working
>on a small chunk at a time. :-)
>
>I was playing with it more today trying to figure out why it was
>dramatically slowing down while filling the second large array ($foo). Now
>I understand more what's going own (thanks in part to that memory usage
>formula), but there's still a couple odd things. (And I realize that having
>a 40k element array with an 8 element array for each value takes a lot of
>memory!)
>
>Somewhere in the early part of filling the 40k element array, PHP seems to
>hit a "bump" where something (mem allocation?) takes much more time, and
>then it fills the array quickly again (e.g. 40k doesn't take much longer to
>create than 10k, for example). That I can understand I guess (though I
>always like to know exactly what's happening? :-)).
>
>The most puzzling part to me still remains: when I unset() the first large
>array ($big_arr), why does filling the second array take more than twice as
>long (5+ seconds vs. 2+)? If a smaller second array is used, it *does* go
>faster with the first array unset, which is what I'd expect. So why, when
>it gets to a certain size does having unset the first array make things
>slower?? :-/
>
>
>Thanks if anyone has an explanation to help me better understand what's
>going on internally (especially the unset() thing).
>
>Matt
>
>--
>PHP Internals - PHP Runtime Development Mailing List
>To unsubscribe, visit: http://www.php.net/unsub.php