Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:60370
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: unknown (pb1.pair.com: domain php.net does not designate 188.138.97.18 as permitted sender)
To: "'Larry Garfield'" <larry@garfieldtech.com>,
	<internals@lists.php.net>
References: <4F9E2173.50005@garfieldtech.com>
In-Reply-To: <4F9E2173.50005@garfieldtech.com>
Date: Mon, 30 Apr 2012 08:25:24 +0200
Message-ID: <000e01cd269a$06f15230$14d3f690$@php.net>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit
Thread-Index: AQHSGBh16tjoKJ9vPkoJpKVM1exOXJapAdcg
Content-Language: de
Subject: RE: [PHP-DEV] readfile() memory usage
From: thetaphi@php.net ("Uwe Schindler")

Hi,

Readfile() is internally implemented in the same way like fpassthru()
(actually the same backend function is called, readfile only opening a
stream before delegating to passthru). Both methods delegate from PHP
user-space to an internal streams API methods php_stream_passthru(). This
one has 2 implementations:

- If the underlying stream allows MMAP, it will use memory mapping (mapping
file to *virtual* memory) and copy the mapped buffer to output. Please note
memory mapping does *not* load the file into memory, it only *maps* the file
contents to virtual memory like a swap file
(http://en.wikipedia.org/wiki/Mmap).
- If this is not the case, it copies the whole file in blocks of 8192 bytes
using a conventional loop.

I verified, this code is at least in PHP 5.2 and 5.3, maybe earlier, too.

Uwe

-----
Uwe Schindler
thetaphi@php.net - http://www.php.net
NSAPI SAPI developer
Bremen, Germany

> -----Original Message-----
> From: Larry Garfield [mailto:larry@garfieldtech.com]
> Sent: Monday, April 30, 2012 7:22 AM
> To: internals@lists.php.net
> Subject: [PHP-DEV] readfile() memory usage
> 
> So, I've been reading articles for a decade now that say that readfile()
is great
> and wonderful except for memory usage.  Specifically, that it reads a file
into
> memory entirely, and then prints it to stdout from there.  So if you're
outputing
> a big file you will hit your memory limit and kill the server.  Thus, one
should
> always loop over fread() instead.  The most recent article I found saying
that
> was from 2007, with a StackExchange thread saying the same from 2011.
I've
> even found mention of it in old PHP Bugs.
> 
> However, I cannot replicate that in my own testing.  Earlier today I was
running
> some benchmarks of different file streaming techniques in PHP
> (5.3.6 specifically) and found that fread() looping, fpassthru(),
readfile(), and
> stream_copy_to_stream() perform almost identically on memory, and all are
> identical on CPU except for fread() which is slower, which makes sense
since
> you're looping in PHP space.
> 
> What's more, I cranked my memory limit down to 10 MB and then tried
> streaming a 20 MB file.  No change.  The PHP peak memory never left around
> a half-meg or so, most of which I presume is just the Apache/PHP overhead.
> But it's not actually possible for readfile() to be buffering the whole
file into
> memory before printing and not die if the file is bigger than the memory
limit.
> I verified that the data I'm getting downloaded from the script is
correct, and
> exactly matches the file that it should be streaming.
> 
> My first thought was that this is yet another case of PHP improving and
fixing a
> long-standing bug, but somehow the rest of the world not knowing about it
so
> "conventional wisdom" persists long after it's still wise.  However, I
found no
> mention of readfile() in the PHP 5 change log[1] at all aside from one
note
> from back in 5.0.0 Beta 1 about improving performance under Windows.  (I'm
> on Linux.)
> 
> So, what's going on here?  Has readfile() been memory-safe for that long
> without anyone noticing?  Is my test completely flawed (although I don't
see
> how since I can verify that the code works as expected)?  Something else?
> 
> Please un-confuse me!
> 
> (Note: Sending this to internals since this is an engine question, and I
am more
> likely to reach whoever it was that un-sucked readfile() sometime in the
silent
> past that way. <g>)
> 
> --Larry Garfield
> 
> --
> PHP Internals - PHP Runtime Development Mailing List To unsubscribe,
visit:
> http://www.php.net/unsub.php