Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:60381
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: unknown (pb1.pair.com: domain php.net does not designate 188.138.97.18 as permitted sender)
To: "'Larry Garfield'" <larry@garfieldtech.com>,
	<internals@lists.php.net>
References: <4F9E2173.50005@garfieldtech.com> <000e01cd269a$06f15230$14d3f690$@php.net> <CAEZPtU4kw1HbRJr=zoEY8nFiBYEdNRka0rp2O+fbmtEPM2xaNg@mail.gmail.com> <002701cd26b5$1cd0bb70$56723250$@thetaphi.de> <CAL+t6cGSHU7Q1VE2BkYJcTnYL1a0XJY3eUHOUgGnM7CFi1RzLw@mail.gmail.com> <4F9EC8B4.3040600@garfieldtech.com>
In-Reply-To: <4F9EC8B4.3040600@garfieldtech.com>
Date: Mon, 30 Apr 2012 23:00:57 +0200
Message-ID: <00ab01cd2714$57336e70$059a4b50$@php.net>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit
Thread-Index: AQHSGBh16tjoKJ9vPkoJpKVM1exOXAG6vPgfAvqifT4CST78sAKyKnrWArKfqW+WRtq+0A==
Content-Language: de
Subject: RE: [PHP-DEV] readfile() memory usage
From: thetaphi@php.net ("Uwe Schindler")

Hi Larry,

From my understanding this is correct, except the part about the code
history: I am not sure what PHP did in the past (before the new streams
API), so before saying that it was always this way, you should review
ext/std/file.c and main/streams.c in the GIT/SVN/CVS history! But the other
notes seem to be the reason for the "persistent urban legends" :-)

Uwe

P.S.: By the way, I will have a MMap blog post, too; focusing on the same
urban legends about to the use of Lucene's MMapDirectory in Apache Lucene
and Apache Solr. I am just a little bit overcrowded with work.
-----
Uwe Schindler
thetaphi@php.net - http://www.php.net
NSAPI SAPI developer
Bremen, Germany


> -----Original Message-----
> From: Larry Garfield [mailto:larry@garfieldtech.com]
> Sent: Monday, April 30, 2012 7:16 PM
> To: internals@lists.php.net
> Subject: Re: [PHP-DEV] readfile() memory usage
> 
> Thanks all.  So it sounds like the answer is:
> 
> 1) readfile() has always been memory-safe as far as PHP is concerned.
> 2) Because it uses mmap(), GFL trying to understand its memory usage from
> top.
> 3) Operating systems have gotten better at such things in the past decade.
> 4) So given #2 and #3, the "readfile() will kill your memory, don't use
it" line is
> a persistent urban legend that belongs on Snopes as debunked.  Looping on
> fread() for performance is a red herring.
> 
> Is that an accurate summary?  If so, I will blog my benchmark results and
this
> conversation.
> 
> --Larry Garfield
> 
> On 4/30/12 5:33 AM, Sherif Ramadan wrote:
> >> Mmapping of course uses memory, but the memory used here is not from
> >> PHP's memory manager, it's memory that's already used for the O/S
> >> cache. The memory mapping used here does not enforce loading the file
> >> contents into O/S cache; it just gets a virtual address into the O/S
> >> cache. If the actual file contents are not yet in O/S cache, the O/S
> >> will hit a page fault and load the pages into memory. Apache Server
uses
> the same mechanism to serve files.
> >>
> >> Maybe the "user confusion" about memory usage comes from that fact
> >> (they see lots of *virtual* memory used by PHP when viewed in top). I
> >> know this user confusion from my work in the Apache Lucene/Solr
> >> project, where one option (used on 64 bit operating systems) is to
> >> memory-map the whole Lucene full-text index. Users then see hundreds of
> Gigabytes of "virtual memory"
> >> usage in TOP / Windows Task Manager and are afraid of running their
> >> machine out of memory. This is always hard to explain to people that
> >> are not used to the term "virtual memory".
> >>
> >
> >
> > That's a very good point and a detailed look at the stack can show
> > some of the underlying mechanics coming from readfile and how its
> > pretty much is just implemented like fpassthru delegating a stream, as
> > you said.
> >
> > http://i.imgur.com/PWiTv.png
> >
> >
> >>
> >>> About php memory usage, one has to use an external tools to actually
> >>> see
> >> this
> >>> memory usage as it is not managed by the zend memory manager.
> >>
> >> Of course...
> >>
> >
> > Also, running valgrind and taking a closer look at what memory blocks
> > PHP is allocating here it can be better determined what's leaking and
> > what isn't, of course...
> >
> > http://pastebin.com/LSQcUsBL
> >
> >
> > I can certainly attest to users being deceived by memory readings in
> > top in the past. It can be very deceiving if you think look at what
> > free memory top or `free` shows you especially after some huge
> > allocation. There's a clear difference here though between the Zend
> > memory manager allocating these blocks in PHP and what's going on with
> > readfile. The memory the system knows is available to it isn't being
> > tied up in this case.
> >
> 
> --
> PHP Internals - PHP Runtime Development Mailing List To unsubscribe,
visit:
> http://www.php.net/unsub.php