Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:43237
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: pass (pb1.pair.com: domain zend.com designates 63.205.162.116 as permitted sender)
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Date: Mon, 2 Mar 2009 20:10:58 -0800
Message-ID: <698DE66518E7CA45812BD18E807866CE029F4805@us-ex1.zend.net>
In-Reply-To: <002a01c99b56$97b7fd00$c727f700$@com>
Thread-Topic: [PHP-DEV] Stream chunk size
Thread-Index: AcmbVpXRvcgBz3lWRwWZQb1SjgaG2wAXfqOA
References: <002a01c99b56$97b7fd00$c727f700$@com>
To: "Jonathan Bond-Caron" <jbondc@openmv.com>
Cc: "PHP Developers Mailing List" <internals@lists.php.net>
Subject: RE: [PHP-DEV] Stream chunk size
From: andi@zend.com ("Andi Gutmans")

I don't see a fundamental issue why it could not be arbitrary.
The only challenge which may be an issue is that this code clearly =
allocates the buffer on the stack for what are probably performance =
reasons. If you allow arbitrary chunk size and use alloca() =
(do_alloca()) for stack allocation you might kill the stack.
I suggest you do some performance tests and if you need to keep it on =
the stack then create some arbitrary limit like 8K and use stack below =
that and use heap above that (code will be uglier).

Andi

> -----Original Message-----
> From: Jonathan Bond-Caron [mailto:jbondc@openmv.com]
> Sent: Monday, March 02, 2009 8:48 AM
> To: 'PHP Developers Mailing List'
> Subject: [PHP-DEV] Stream chunk size
>=20
> Hi everyone, I have a question about streams and the maximum 'chunk =
size' of
> 8192.
>=20
>=20
>=20
> I've read README.STREAMS and found these slides by Wez:
>=20
> http://netevil.org/blog/2008/07/slides-php-streams
>=20
>=20
>=20
> While trying to write an Amazon S3 stream wrapper and I ran into an =
issue
> with large files:
>=20
>=20
>=20
> $fp =3D fopen('s3://mvtest/large.html', 'r'); // 30 mb
>=20
>=20
>=20
> // This is OK
>=20
> fseek($fp, 10);
>=20
> echo fread($fp, 100) . "\n"; // 100 bytes
>=20
> echo fread($fp, 100) . "\n"; // 100 bytes
>=20
>=20
>=20
> // This is OK (according to documentation, max 8192 bytes)
>=20
> echo fread($fp, 65536) . "\n"; // 8192 bytes
>=20
>=20
>=20
> My issue is I would like to request larger 'chunks', something like:
>=20
> stream_set_chunk_size($fp, 65536);
>=20
>=20
>=20
> echo fread($fp, 65536) . "\n"; // 65536 bytes
>=20
> echo fread($fp, 100000) . "\n"; // 65536 bytes
>=20
> echo fread($fp, 15) . "\n"; // 15 bytes
>=20
>=20
>=20
> Then copying to a file and avoiding memory issues:
>=20
>=20
>=20
> $wfp =3D fopen('/tmp/large.html');
>=20
> stream_copy_to_stream($fp, $wfp); // read 65536 byte chunks, write =
default
> 8192 byte chunks
>=20
>=20
>=20
> stream_set_chunk_size($wfp, 65536);
>=20
> stream_copy_to_stream($fp, $wfp); // read & write 65536 byte chunks
>=20
> copy('s3://mvtest/large.html', '/tmp/large.html'); // read & write =
default
> 8192 byte chunks
>=20
>=20
>=20
> Going through the PHP 5.2 source, it looks like there's support for it =
but
> at some places the 8192 'chunk' is hardcoded:
>=20
>=20
>=20
> #define CHUNK_SIZE     8192
>=20
>=20
>=20
> PHPAPI size_t _php_stream_copy_to_stream(php_stream *src, php_stream =
*dest,
> size_t maxlen STREAMS_DC TSRMLS_DC)
>=20
> {
>=20
>                 char buf[CHUNK_SIZE]; =DF  Is there any reason the =
php_stream
> *src->chunk_size isn't used?
>=20
>=20
>=20
> stream_set_chunk_size($fp, 65536); // Would mean src->chunk_size =3D =
65536;
>=20
>=20
>=20
> I'd like to try to write a patch for it, anything that I should know =
about
> streams and why the limit is there?