Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:60379
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: error (pb1.pair.com: domain garfieldtech.com from 66.111.4.25 cause and error)
Message-ID: <4F9EC8B4.3040600@garfieldtech.com>
Date: Mon, 30 Apr 2012 12:15:32 -0500
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:11.0) Gecko/20120327 Thunderbird/11.0.1
MIME-Version: 1.0
To: internals@lists.php.net
References: <4F9E2173.50005@garfieldtech.com> <000e01cd269a$06f15230$14d3f690$@php.net> <CAEZPtU4kw1HbRJr=zoEY8nFiBYEdNRka0rp2O+fbmtEPM2xaNg@mail.gmail.com> <002701cd26b5$1cd0bb70$56723250$@thetaphi.de> <CAL+t6cGSHU7Q1VE2BkYJcTnYL1a0XJY3eUHOUgGnM7CFi1RzLw@mail.gmail.com>
In-Reply-To: <CAL+t6cGSHU7Q1VE2BkYJcTnYL1a0XJY3eUHOUgGnM7CFi1RzLw@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Subject: Re: [PHP-DEV] readfile() memory usage
From: larry@garfieldtech.com (Larry Garfield)

Thanks all.  So it sounds like the answer is:

1) readfile() has always been memory-safe as far as PHP is concerned.
2) Because it uses mmap(), GFL trying to understand its memory usage 
from top.
3) Operating systems have gotten better at such things in the past decade.
4) So given #2 and #3, the "readfile() will kill your memory, don't use 
it" line is a persistent urban legend that belongs on Snopes as 
debunked.  Looping on fread() for performance is a red herring.

Is that an accurate summary?  If so, I will blog my benchmark results 
and this conversation.

--Larry Garfield

On 4/30/12 5:33 AM, Sherif Ramadan wrote:
>> Mmapping of course uses memory, but the memory used here is not from PHP's
>> memory manager, it's memory that's already used for the O/S cache. The
>> memory mapping used here does not enforce loading the file contents into O/S
>> cache; it just gets a virtual address into the O/S cache. If the actual file
>> contents are not yet in O/S cache, the O/S will hit a page fault and load
>> the pages into memory. Apache Server uses the same mechanism to serve files.
>>
>> Maybe the "user confusion" about memory usage comes from that fact (they see
>> lots of *virtual* memory used by PHP when viewed in top). I know this user
>> confusion from my work in the Apache Lucene/Solr project, where one option
>> (used on 64 bit operating systems) is to memory-map the whole Lucene
>> full-text index. Users then see hundreds of Gigabytes of "virtual memory"
>> usage in TOP / Windows Task Manager and are afraid of running their machine
>> out of memory. This is always hard to explain to people that are not used to
>> the term "virtual memory".
>>
>
>
> That's a very good point and a detailed look at the stack can show
> some of the underlying mechanics coming from readfile and how its
> pretty much is just implemented like fpassthru delegating a stream, as
> you said.
>
> http://i.imgur.com/PWiTv.png
>
>
>>
>>> About php memory usage, one has to use an external tools to actually see
>> this
>>> memory usage as it is not managed by the zend memory manager.
>>
>> Of course...
>>
>
> Also, running valgrind and taking a closer look at what memory blocks
> PHP is allocating here it can be better determined what's leaking and
> what isn't, of course...
>
> http://pastebin.com/LSQcUsBL
>
>
> I can certainly attest to users being deceived by memory readings in
> top in the past. It can be very deceiving if you think look at what
> free memory top or `free` shows you especially after some huge
> allocation. There's a clear difference here though between the Zend
> memory manager allocating these blocks in PHP and what's going on with
> readfile. The memory the system knows is available to it isn't being
> tied up in this case.
>