Stream chunk size View as Markdown

I don't see a fundamental issue why it could not be arbitrary.
The only challenge which may be an issue is that this code clearly allocates the buffer on the stack for what are probably performance reasons. If you allow arbitrary chunk size and use alloca() (do_alloca()) for stack allocation you might kill the stack.
I suggest you do some performance tests and if you need to keep it on the stack then create some arbitrary limit like 8K and use stack below that and use heap above that (code will be uglier).

Andi

-----Original Message-----
From: Jonathan Bond-Caron [mailto:jbondc@openmv.com]
Sent: Monday, March 02, 2009 8:48 AM
To: 'PHP Developers Mailing List'
Subject: [PHP-DEV] Stream chunk size

Hi everyone, I have a question about streams and the maximum 'chunk size' of

I've read README.STREAMS and found these slides by Wez:

http://netevil.org/blog/2008/07/slides-php-streams

While trying to write an Amazon S3 stream wrapper and I ran into an issue
with large files:

$fp = fopen('s3://mvtest/large.html', 'r'); // 30 mb

// This is OK

fseek($fp, 10);

echo fread($fp, 100) . "\n"; // 100 bytes

echo fread($fp, 100) . "\n"; // 100 bytes

// This is OK (according to documentation, max 8192 bytes)

echo fread($fp, 65536) . "\n"; // 8192 bytes

My issue is I would like to request larger 'chunks', something like:

stream_set_chunk_size($fp, 65536);

echo fread($fp, 65536) . "\n"; // 65536 bytes

echo fread($fp, 100000) . "\n"; // 65536 bytes

echo fread($fp, 15) . "\n"; // 15 bytes

Then copying to a file and avoiding memory issues:

$wfp = fopen('/tmp/large.html');

stream_copy_to_stream($fp, $wfp); // read 65536 byte chunks, write default
8192 byte chunks

stream_set_chunk_size($wfp, 65536);

stream_copy_to_stream($fp, $wfp); // read & write 65536 byte chunks

copy('s3://mvtest/large.html', '/tmp/large.html'); // read & write default
8192 byte chunks

Going through the PHP 5.2 source, it looks like there's support for it but
at some places the 8192 'chunk' is hardcoded:

#define CHUNK_SIZE 8192

PHPAPI size_t _php_stream_copy_to_stream(php_stream *src, php_stream *dest,
size_t maxlen STREAMS_DC TSRMLS_DC)

{

            char buf[CHUNK_SIZE]; ß  Is there any reason the php_stream

*src->chunk_size isn't used?

stream_set_chunk_size($fp, 65536); // Would mean src->chunk_size = 65536;

I'd like to try to write a patch for it, anything that I should know about
streams and why the limit is there?