Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:14040
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Message-ID: <00c501c4dea7$b66ae520$0100a8c0@pc1>
To: "Derick Rethans" <derick@php.net>, <internals@lists.php.net>
References: <002e01c4dde1$b69fe9d0$0100a8c0@pc1> <Pine.LNX.4.58.0412091252210.12298@localhost>
Date: Fri, 10 Dec 2004 05:02:16 -0600
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
Subject: Re: [PHP-DEV] Memory needed for arrays? And major performance problems filling an array
From: php_lists@realplain.com ("Matt W")

----- Original Message -----
From: "Derick Rethans"
Sent: Thursday, December 09, 2004


> On Thu, 9 Dec 2004, Matt W wrote:
>
> > Before I describe my issue, I had been wondering how I can
tell/estimate
> > how much memory PHP will use for arrays?  Is there at least a rough
formula
> > someone can tell me?
>
> 42 * number of array elements + size of keys + data

Thanks Derick. Is that exact or just an estimate?

So PHP needs 42 bytes for each array element, OK. For a multi-dimensional
array, I assume the "number of elements" is the total from all dimensions
added up. e.g. $foo[] = array(123, 456); is 3 elements (1 + 2)?


> > This is disappointing, especially being caused by an array that doesn't
> > seem too outrageously large (and I'd like to go larger). :-( It's going
to
> > suck if I can't come up with a way to get it faster... Is there
anything
> > that would be different on a version newer than 4.3.6, as I didn't try
any
> > yet? Or if it's some Windows thing, then I don't care as much.
>
> > Comments, suggestions?
>
> Make a custom C extension.. I don't know why you needs arrays this
> large, but it sounds like a application design flaw.

I need arrays this large because it's building a search index, which could
have 100s of millions of entries; and I want to buffer many entries in
arrays before doing writes... So even these "large" arrays are just working
on a small chunk at a time. :-)

I was playing with it more today trying to figure out why it was
dramatically slowing down while filling the second large array ($foo). Now
I understand more what's going own (thanks in part to that memory usage
formula), but there's still a couple odd things. (And I realize that having
a 40k element array with an 8 element array for each value takes a lot of
memory!)

Somewhere in the early part of filling the 40k element array, PHP seems to
hit a "bump" where something (mem allocation?) takes much more time, and
then it fills the array quickly again (e.g. 40k doesn't take much longer to
create than 10k, for example). That I can understand I guess (though I
always like to know exactly what's happening? :-)).

The most puzzling part to me still remains: when I unset() the first large
array ($big_arr), why does filling the second array take more than twice as
long (5+ seconds vs. 2+)? If a smaller second array is used, it *does* go
faster with the first array unset, which is what I'd expect. So why, when
it gets to a certain size does having unset the first array make things
slower?? :-/


Thanks if anyone has an explanation to help me better understand what's
going on internally (especially the unset() thing).

Matt