Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:60371
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: pass (pb1.pair.com: domain gmail.com designates 209.85.215.42 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <4F9E2173.50005@garfieldtech.com>
References: <4F9E2173.50005@garfieldtech.com>
Date: Mon, 30 Apr 2012 02:27:07 -0400
Message-ID: <CAL+t6cE-+MLtw7L67EE3PTpmcm6RMOWmbA5SQw_1xnSHLBmYsA@mail.gmail.com>
To: Larry Garfield <larry@garfieldtech.com>
Cc: internals@lists.php.net
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Subject: Re: [PHP-DEV] readfile() memory usage
From: theanomaly.is@gmail.com (Sherif Ramadan)

> So, I've been reading articles for a decade now that say that readfile() =
is
> great and wonderful except for memory usage. =A0Specifically, that it rea=
ds a
> file into memory entirely, and then prints it to stdout from there. =A0So=
 if
> you're outputing a big file you will hit your memory limit and kill the
> server. =A0Thus, one should always loop over fread() instead. =A0The most=
 recent
> article I found saying that was from 2007, with a StackExchange thread
> saying the same from 2011. =A0I've even found mention of it in old PHP Bu=
gs.
>

Well, this is not true, but I haven't seen the StackExchange thread
you're referring to (I won't bother to comment on the specifics of a
discussion I wasn't privy to). I can say that readfile uses PHP's
streams, actually. So it's to the same effect. If you're just writing
more code to this yourself, you haven't done anything different than
what readfile() already does. Here's your proof:

http://lxr.php.net/opengrok/xref/PHP_5_4/ext/standard/file.c#1346


> However, I cannot replicate that in my own testing. =A0Earlier today I wa=
s
> running some benchmarks of different file streaming techniques in PHP (5.=
3.6
> specifically) and found that fread() looping, fpassthru(), readfile(), an=
d
> stream_copy_to_stream() perform almost identically on memory, and all are
> identical on CPU except for fread() which is slower, which makes sense si=
nce
> you're looping in PHP space.
>

That's because they're all using pretty much the same code in PHP :)

> What's more, I cranked my memory limit down to 10 MB and then tried
> streaming a 20 MB file. =A0No change. =A0The PHP peak memory never left a=
round a
> half-meg or so, most of which I presume is just the Apache/PHP overhead.
> =A0But it's not actually possible for readfile() to be buffering the whol=
e
> file into memory before printing and not die if the file is bigger than t=
he
> memory limit. =A0I verified that the data I'm getting downloaded from the
> script is correct, and exactly matches the file that it should be streami=
ng.
>

You're absolutely correct. readfile cleans up after itself. It's
basically just a stream that reads from the file and sends the output
directly for you. It doesn't load the entire file into memory first,
which is what file_get_contents() does, for example. Therefore you
aren't using any more memory in PHP with readfile then if you did a
simple fopen(...), while(!feof($fp)) echo fread(...); fclose(); in
PHP. Here's a small test you can use to demonstrate that point
effectively.

http://pastebin.com/7EEDGDSd

This code uses readfile() to read a 10MB file to output. Here's a
snapshot of the result of running this script on my local server in
Chrome.

http://i.imgur.com/ir7nO.png

What you can see here is that we transferred 10MB of data (albeit I
have compression with gzip) without ever peaking over roughly 270KB of
memory usage in PHP.

Now here's the result of using the same code except replace readfile()
with file_get_contents();

http://i.imgur.com/kxAAJ.png

(Keep in mind I didn't actually output the file here, but I was
demonstrating the clear difference in memory consumption). Now,
file_get_contents() is still actually using the same streaming
capabilities PHP offers, with the exception being it's calling
fopen(), fread(), and fclose() in one go and without cleaning up the
memory. The purpose is to store/return the file contents in memory as
a string so that you can do something with it in your code. The
purpose of readfile() is just to output the file (you wouldn't care
about doing anything with it in your code here). So the two have very
different use-cases.

> My first thought was that this is yet another case of PHP improving and
> fixing a long-standing bug, but somehow the rest of the world not knowing
> about it so "conventional wisdom" persists long after it's still wise.
> =A0However, I found no mention of readfile() in the PHP 5 change log[1] a=
t all
> aside from one note from back in 5.0.0 Beta 1 about improving performance
> under Windows. =A0(I'm on Linux.)
>

I believe you're talking about Wez's commit (that was way back during
the PHP5 beta days). I dug up another one that had to do with a minor
memory leak issue with readfile() around the same time. That was
basically an issue with how PHP handles memory clean up and shutdown
after the user has aborted the connection
(http://us.php.net/manual/en/function.ignore-user-abort.php)

> So, what's going on here? =A0Has readfile() been memory-safe for that lon=
g
> without anyone noticing? =A0Is my test completely flawed (although I don'=
t see
> how since I can verify that the code works as expected)? =A0Something els=
e?
>
> Please un-confuse me!
>

I _hope_ this may shed at least some light as to your confusion.
Please let me know if there's anything else I missed.

> (Note: Sending this to internals since this is an engine question, and I =
am
> more likely to reach whoever it was that un-sucked readfile() sometime in
> the silent past that way. <g>)
>
> --Larry Garfield
>
> --
> PHP Internals - PHP Runtime Development Mailing List
> To unsubscribe, visit: http://www.php.net/unsub.php
>