Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:60369
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: error (pb1.pair.com: domain garfieldtech.com from 66.111.4.25 cause and error)
Message-ID: <4F9E2173.50005@garfieldtech.com>
Date: Mon, 30 Apr 2012 00:21:55 -0500
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0.2) Gecko/20120216 Thunderbird/10.0.2
MIME-Version: 1.0
To: internals@lists.php.net
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Subject: readfile() memory usage
From: larry@garfieldtech.com (Larry Garfield)

So, I've been reading articles for a decade now that say that readfile() 
is great and wonderful except for memory usage.  Specifically, that it 
reads a file into memory entirely, and then prints it to stdout from 
there.  So if you're outputing a big file you will hit your memory limit 
and kill the server.  Thus, one should always loop over fread() 
instead.  The most recent article I found saying that was from 2007, 
with a StackExchange thread saying the same from 2011.  I've even found 
mention of it in old PHP Bugs.

However, I cannot replicate that in my own testing.  Earlier today I was 
running some benchmarks of different file streaming techniques in PHP 
(5.3.6 specifically) and found that fread() looping, fpassthru(), 
readfile(), and stream_copy_to_stream() perform almost identically on 
memory, and all are identical on CPU except for fread() which is slower, 
which makes sense since you're looping in PHP space.

What's more, I cranked my memory limit down to 10 MB and then tried 
streaming a 20 MB file.  No change.  The PHP peak memory never left 
around a half-meg or so, most of which I presume is just the Apache/PHP 
overhead.  But it's not actually possible for readfile() to be buffering 
the whole file into memory before printing and not die if the file is 
bigger than the memory limit.  I verified that the data I'm getting 
downloaded from the script is correct, and exactly matches the file that 
it should be streaming.

My first thought was that this is yet another case of PHP improving and 
fixing a long-standing bug, but somehow the rest of the world not 
knowing about it so "conventional wisdom" persists long after it's still 
wise.  However, I found no mention of readfile() in the PHP 5 change 
log[1] at all aside from one note from back in 5.0.0 Beta 1 about 
improving performance under Windows.  (I'm on Linux.)

So, what's going on here?  Has readfile() been memory-safe for that long 
without anyone noticing?  Is my test completely flawed (although I don't 
see how since I can verify that the code works as expected)?  Something 
else?

Please un-confuse me!

(Note: Sending this to internals since this is an engine question, and I 
am more likely to reach whoever it was that un-sucked readfile() 
sometime in the silent past that way. <g>)

--Larry Garfield