Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:60393
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: error (pb1.pair.com: domain lerdorf.com from 209.85.220.170 cause and error)
Message-ID: <4F9FF9EF.4040107@lerdorf.com>
Date: Tue, 01 May 2012 07:57:51 -0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:11.0) Gecko/20120410 Thunderbird/11.0.1
MIME-Version: 1.0
To: Paul Reinheimer <preinheimer@gmail.com>
CC: internals@lists.php.net, Larry Garfield <larry@garfieldtech.com>
References: <4F9E2173.50005@garfieldtech.com> <000e01cd269a$06f15230$14d3f690$@php.net> <CAEZPtU4kw1HbRJr=zoEY8nFiBYEdNRka0rp2O+fbmtEPM2xaNg@mail.gmail.com> <002701cd26b5$1cd0bb70$56723250$@thetaphi.de> <CAL+t6cGSHU7Q1VE2BkYJcTnYL1a0XJY3eUHOUgGnM7CFi1RzLw@mail.gmail.com> <4F9EC8B4.3040600@garfieldtech.com> <CABy5rUbnZmqFSHX1ee4mwWXyZMiUiik8Ak_BHnjKTbrixtRPXg@mail.gmail.com> <4F9F4208.7020106@garfieldtech.com> <CABy5rUYAr08ciTDntO=9P-c0Jm=_AN-xqUKv1FrQ-EtjaVSAYA@mail.gmail.com> <4F9FF2B9.7060301@lerdorf.com>
In-Reply-To: <4F9FF2B9.7060301@lerdorf.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Subject: Re: [PHP-DEV] readfile() memory usage
From: rasmus@lerdorf.com (Rasmus Lerdorf)

On 05/01/2012 07:27 AM, Rasmus Lerdorf wrote:
> On 05/01/2012 06:39 AM, Paul Reinheimer wrote:
>> Hi Larry,
>>
>>> Fascinating.  I even verified the md5sum of the file I got on the other end
>>> just to be sure.  I'll hold off on the blog post then. :-)  I look forward
>>> to your test setup.
>>
>> The server in question is still on PHP 5.2.13
>>
>> Script:
>> <?php
>> ini_set('memory_limit', '8M');
>> $name = uniqid() . ".zip";
>> header('Content-type: application/zip');
>> header("Content-Disposition: attachment; filename=\"$name\"");
>> readfile('../../filestorage/4f9e9e3b9bcff.zip');
>>
>> File Information:
>> [user@host public]$ ls -alh ../../filestorage/4f9e9e3b9bcff.zip
>> -rw-r--r-- 1 apache apache 27M Apr 30 10:14 ../../filestorage/4f9e9e3b9bcff.zip
>>
>> Error:
>> [Tue May 01 09:30:48 2012] [error] [client 198.136.162.2] PHP Fatal
>> error:  Allowed memory size of 8388608 bytes exhausted (tried to
>> allocate 27617281 bytes) in
>> /home/lots/of/path.org/stuff/public/rf822.php on line 6
>>
>>
>> I'll try something newer, but I wanted to prove myself not crazy and
>> do it on the server in question first.
> 
> That's odd, because PHP 5.2 has identical code in this respect. Wez
> committed these changes in 2002:
> 
> https://github.com/php/php-src/commit/a662f012bba5a6fdc50533673f3fff47bf9af219#diff-5
> 
> So it has been like this for quite a while. Does that server have
> implicit unlimited output buffering turned on in your ini file?

And actually, that patch just made it use streams. Even before the
internal streams API the function worked the same way. In PHP 4.2 it
used php_passthru_fd which looked like this:

https://github.com/php/php-src/blob/PHP-4.2.0/ext/standard/file.c#L1526

Sascha added the mmap implementation we still use today in 1999:

https://github.com/php/php-src/commit/dda0b783df7d849df01fa831febbc1e34b5b8dd3

But even prior to that readfile would still buffer in 8k chunks. The PHP
2.0.1 implementation (very scary to look at code I wrote 17 or 18 years
ago):

/*
 * Read a file and write the ouput to stdout
 */
void ReadFile(void) {
        Stack *s;
        char buf[8192],temp[8];
        FILE *fp;
        int b,i, size;

        s = Pop();
        if(!s) {
                Error("Stack error in ReadFile");
                return;
        }
        if(!*(s->strval)) {
                Push("-1",LNUMBER);
                return;
        }
#if DEBUG
        Debug("Opening [%s]\n",s->strval);
#endif
        StripSlashes(s->strval);

#if PHP_SAFE_MODE
        if(!CheckUid(s->strval,1)) {
                Error("SAFE MODE Restriction in effect.  Invalid owner
of file to be read.");
                Push("-1",LNUMBER);
                return;
        }
#endif
        fp = fopen(s->strval,"r");
        if(!fp) {
                Error("ReadFile(\"%s\") - %s",s->strval,strerror(errno));
                Push("-1",LNUMBER);
                return;
        }
        size= 0;
        php_header(0,NULL);
        while((b = fread(buf, 1, sizeof(buf), fp)) > 0) {
                for(i = 0; i < b; i++)
                        PUTC(buf [i]);
                size += b ;
        }
        fclose(fp);
        sprintf(temp,"%d",size);
        Push(temp,LNUMBER);
}

-Rasmus