# Large file support for PHP - Thread: https://externals.io/message/32767 - Original archive: https://news-web.php.net/php.internals/32767 - Started by Wez Furlong on 2007-10-14T22:13:50+00:00 - Messages: 16 - Reply by email to internals@lists.php.net - More for agents: search with https://externals.io/search.md?q={query} · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt --- ## #32767 — Wez Furlong — 2007-10-14T22:13:50+00:00 URL: https://externals.io/message/32767#32767 This bug has been open for a while: http://bugs.php.net/bug.php?id=27792 Having run into this issue recently, here's a patch (hopefully attached, mail.app and list filters willing) against PHP 5.3 to address it. This patch will promote to double the file sizes that overflow LONG_MAX, which works transparently for my script. Note that this only touches the obvious functions in the core; there may be other extensions that need to behave similarly. The defines that I jam into CFLAGS are known to work in our other projects on Linux, Solaris, FreeBSD and OSX. The other (weirder) platforms might need some other adjustments to work correctly. --Wez. --- ## #32769 — sean finney — 2007-10-15T06:27:12+00:00 URL: https://externals.io/message/32767#32769 · In reply to: #32767 hi, some non-php-dev comments in case they're helpful... On Monday 15 October 2007 12:13:50 am Wez Furlong wrote: > This bug has been open for a while: > http://bugs.php.net/bug.php?id=27792 > Having run into this issue recently, here's a patch (hopefully > attached, mail.app and list filters willing) against PHP 5.3 to > address it. using size_t/off_t is always good > This patch will promote to double the file sizes that overflow > LONG_MAX, which works transparently for my script. > Note that this only touches the obvious functions in the core; there > may be other extensions that need to behave similarly. why not just use SIZE_MAX, and not have to worry about anything bigger? I can't profess to fully understand what's going on, but it looks like the functions are being converted from returning a long to returning either a long or a double, and then the result is parsed as a string??? > The defines that I jam into CFLAGS are known to work in our other > projects on Linux, Solaris, FreeBSD and OSX. The other (weirder) > platforms might need some other adjustments to work correctly. "getconf LFS_CFLAGS", assuming "getconf" exists... sean --- ## #32775 — Joe Orton — 2007-10-15T10:43:55+00:00 URL: https://externals.io/message/32767#32775 > Hi, > > please keep in mind that compiling PHP with large file support breaks > binary compatibility... > One of the globals contain a "stat" struct that has different size for > LFS or no LFS. More harmfully, it also changes the ABI of any library/application linked against PHP where that ABI uses off_t. So PHP would see all off_t's in the libraries API as 64-bit, whereas they were built as 32-bit. This breaks structure offsets etc very badly. The way this is done for e.g APR avoids the issue; using only -D_LARGEFILE64_SOURCE, off64_t in place of off_t, and fstat64, open64/O_LARGEFILE, etc. joe --- ## #32782 — sean finney — 2007-10-15T17:10:45+00:00 URL: https://externals.io/message/32767#32782 hi, On Monday 15 October 2007 09:26:30 am Stefan Esser wrote: > please keep in mind that compiling PHP with large file support breaks > binary compatibility... > One of the globals contain a "stat" struct that has different size for > LFS or no LFS. yes, this is of course a big deal for some people, esp if you're using proprietary software that's built against the "original" abi. of course if you're only using OSS extensions, you can simply recompile them against the new api/abi and there's no drawback. btw - in our (debian's) case, the benefits outweighed the consequences, so we've been shipping with LFS turned on for somewhere around a year now. sean --- ## #32783 — Stanislav Malyshev — 2007-10-15T17:41:11+00:00 URL: https://externals.io/message/32767#32783 · In reply to: #32767 I didn't dive yet too deep into the patch, but shouldn't it be fixed on stream level and not function level? I.e. there are a lot of functions using streams (including files) - would they support bigger files too? I also think that while using size_t is good, changing binary structures might be rather dangerous, unless we can ensure all PHPs built on the certain platform would use the same setting. > Having run into this issue recently, here's a patch (hopefully attached, > mail.app and list filters willing) against PHP 5.3 to address it. > This patch will promote to double the file sizes that overflow LONG_MAX, > which works transparently for my script. > Note that this only touches the obvious functions in the core; there may > be other extensions that need to behave similarly. > > The defines that I jam into CFLAGS are known to work in our other > projects on Linux, Solaris, FreeBSD and OSX. The other (weirder) > platforms might need some other adjustments to work correctly. > > --Wez. > -- Stanislav Malyshev, Zend Software Architect stas@zend.com http://www.zend.com/ (408)253-8829 MSN: stas@zend.com --- ## #32786 — Wez Furlong — 2007-10-15T18:31:40+00:00 URL: https://externals.io/message/32767#32786 · In reply to: #32783 > I didn't dive yet too deep into the patch, but shouldn't it be > fixed on stream level and not function level? I.e. there are a lot > of functions using streams (including files) - would they support > bigger files too? Yes, the patch does that; it turns on LFS in the headers, which promotes the off_t and size_t types that are used by streams to the 64-bit versions. This is the one liner in configure.in. The other larger part of the patch is to make PHP functions capable of returning and accepting numbers that are too big to fit into a long. > I also think that while using size_t is good, changing binary > structures might be rather dangerous, unless we can ensure all PHPs > built on the certain platform would use the same setting. That's why 5.3 is a good point to apply this patch, rather than sneaking it into 5.2.x --Wez. --- ## #32804 — sean finney — 2007-10-16T06:44:02+00:00 URL: https://externals.io/message/32767#32804 · In reply to: #32783 hi, On Monday 15 October 2007 07:41:11 pm Stanislav Malyshev wrote: > I didn't dive yet too deep into the patch, but shouldn't it be fixed on > stream level and not function level? I.e. there are a lot of functions > using streams (including files) - would they support bigger files too? i would suggest that anywhere where one is doing something with a size or offset and not using the posix size_t/off_t types should get such changes. and like i said, i don't see the motivation behind this extra step of returning the size in double form if it's bigger than LONG_MAX. > I also think that while using size_t is good, changing binary structures > might be rather dangerous, unless we can ensure all PHPs built on the > certain platform would use the same setting. i don't think switching from long/int -> size_t is a problem in the scope of function internal variables. the only place where you need to worry about this is in headers/structs/function declarations that are exported to the API/ABI. sean --- ## #32816 — Wez Furlong — 2007-10-16T14:37:57+00:00 URL: https://externals.io/message/32767#32816 · In reply to: #32804 > > i would suggest that anywhere where one is doing something with a > size or > offset and not using the posix size_t/off_t types should get such > changes. > and like i said, i don't see the motivation behind this extra step of > returning the size in double form if it's bigger than LONG_MAX. PHP's native integer type is long, how else are you going to relay numbers longer than a long back to the script without rewriting the engine to add additional integer types, which is a massive changeset? > i don't think switching from long/int -> size_t is a problem in the > scope of > function internal variables. the only place where you need to > worry about > this is in headers/structs/function declarations that are exported > to the > API/ABI. Or just bump our API number(s) and "not worry about it", since the module loading code will refuse to load an incompatible module. PHP 5.3 is an ideal point to make this kind of change, as I've already stated. --Wez. --- ## #32818 — Stanislav Malyshev — 2007-10-16T22:16:15+00:00 URL: https://externals.io/message/32767#32818 · In reply to: #32786 > Yes, the patch does that; it turns on LFS in the headers, which promotes > the off_t and size_t types that are used by streams to the 64-bit > versions. This is the one liner in configure.in. > The other larger part of the patch is to make PHP functions capable of > returning and accepting numbers that are too big to fit into a long. I think it's better to do it in .h file and not CFLAGS (or both?) so that if somebody includes PHP headers they would always get the same result. This would prevent one from building incompatible module. Is it guaranteed that once you have this define you would always get the same result on all builds (regardless of libc difference, etc.)? -- Stanislav Malyshev, Zend Software Architect stas@zend.com http://www.zend.com/ (408)253-8829 MSN: stas@zend.com --- ## #32819 — Wez Furlong — 2007-10-16T22:20:25+00:00 URL: https://externals.io/message/32767#32819 · In reply to: #32818 You might have a point there; I'd assumed that CFLAGS made it through to php-config, but it doesn't look like they do. It should be a simple matter to define them in php_config.h instead. --Wez. > > […] > > I think it's better to do it in .h file and not CFLAGS (or both?) > so that if somebody includes PHP headers they would always get the > same result. This would prevent one from building incompatible module. > > Is it guaranteed that once you have this define you would always > get the same result on all builds (regardless of libc difference, > etc.)? > -- > Stanislav Malyshev, Zend Software Architect > stas@zend.com http://www.zend.com/ > (408)253-8829 MSN: stas@zend.com --- ## #32820 — Stanislav Malyshev — 2007-10-17T00:08:06+00:00 URL: https://externals.io/message/32767#32820 · In reply to: #32782 > yes, this is of course a big deal for some people, esp if you're using > proprietary software that's built against the "original" abi. of course if > you're only using OSS extensions, you can simply recompile them against the > new api/abi and there's no drawback. That's not only about OSS vs. proprietary - even if you use only OSS extensions, but you do not compile everything from the source on each install (and have many platforms to install on - such as big organization with dozens of servers), you might have trouble. -- Stanislav Malyshev, Zend Software Architect stas@zend.com http://www.zend.com/ (408)253-8829 MSN: stas@zend.com --- ## #32822 — sean finney — 2007-10-17T06:38:57+00:00 URL: https://externals.io/message/32767#32822 · In reply to: #32816 On Tuesday 16 October 2007 04:37:57 pm Wez Furlong wrote: > PHP's native integer type is long, how else are you going to relay > numbers longer than a long back to the script without rewriting the > engine to add additional integer types, which is a massive changeset? okay, chalk this up to my IANAPHPD ignorance. > > Or just bump our API number(s) and "not worry about it", since the > module loading code will refuse to load an incompatible module. > > PHP 5.3 is an ideal point to make this kind of change, as I've > already stated. the general impression that i got was that zend/php did not want to break their abi because then they'd have to support an additional set of platforms for the zend optimiser. but maybe that was just hearsay. sean --- ## #32823 — sean finney — 2007-10-17T06:48:22+00:00 URL: https://externals.io/message/32767#32823 · In reply to: #32820 hi stanislav, On Wednesday 17 October 2007 02:08:06 am Stanislav Malyshev wrote: > > […] > > That's not only about OSS vs. proprietary - even if you use only OSS > extensions, but you do not compile everything from the source on each > install (and have many platforms to install on - such as big > organization with dozens of servers), you might have trouble. but this wouldn't be any different from any other api/abi bump. furthermore, in the context of running debian (and by extension ubuntu) servers, you wouldn't need to do anything if you were only using the pre-packaged modules, as they'd be compiled with the right options for you, ideally. so you'd just need to do an "apt-get upgrade", etc. and if you were using other pecl extensions, i think the command to update that isn't so complicated either :) sean --- ## #32824 — Stanislav Malyshev — 2007-10-17T07:13:03+00:00 URL: https://externals.io/message/32767#32824 · In reply to: #32823 > but this wouldn't be any different from any other api/abi bump. furthermore, Of course. If there are two different PHP versions with different API numbers, it's OK. What's less OK is when there's two PHP builds with same API numbers which are binary incompatible. > in the context of running debian (and by extension ubuntu) servers, you > wouldn't need to do anything if you were only using the pre-packaged modules, You mean - pre-packaged by Debian? > need to do an "apt-get upgrade", etc. and if you were using other pecl > extensions, i think the command to update that isn't so complicated either :) Compatibility is not complicated if you deal with one machine where you control everything and can rebuild everything from source. It starts to get worse when you have a lot of machines on which you want to achieve some kind of common environment. Having indistinguishable binary incompatible PHP versions complicates things. -- Stanislav Malyshev, Zend Software Architect stas@zend.com http://www.zend.com/ (408)253-8829 MSN: stas@zend.com --- ## #32838 — sean finney — 2007-10-17T21:08:56+00:00 URL: https://externals.io/message/32767#32838 · In reply to: #32824 On Wednesday 17 October 2007 09:13:03 am Stanislav Malyshev wrote: > Of course. If there are two different PHP versions with different API > numbers, it's OK. What's less OK is when there's two PHP builds with > same API numbers which are binary incompatible. right. > > […] > > You mean - pre-packaged by Debian? yeah. there are a number of precompiled modules provided by the debian php package as well as other precompiled pecl packages available in the debian archive. all of them are compiled against the same api, which is provided by a php{4,5}-dev package, which you can install on your system if you want to build your own modules as well. > Compatibility is not complicated if you deal with one machine where you > control everything and can rebuild everything from source. It starts to > get worse when you have a lot of machines on which you want to achieve > some kind of common environment. Having indistinguishable binary > incompatible PHP versions complicates things. i'd say this has more to do with the administration habits of the individual sysadmin. i for one would do my utmost to avoid ever finding myself in a situation where i was compiling php (or any other moderately complex software for that matter) from source on more than a couple different OS/architecuture/distribution combinations, and instead try to consolidate things. for example, using the distributions' precompiled packages, backported packages, 3rd party packages, compiling once and nfs-mounting, rsyncing, etc. but anyway, this is maybe getting a little side tracked.... sean --- ## #33070 — Stanislav Malyshev — 2007-11-09T20:54:51+00:00 URL: https://externals.io/message/32767#33070 · In reply to: #32767 > Having run into this issue recently, here's a patch (hopefully attached, > mail.app and list filters willing) against PHP 5.3 to address it. > This patch will promote to double the file sizes that overflow LONG_MAX, > which works transparently for my script. > Note that this only touches the obvious functions in the core; there may > be other extensions that need to behave similarly. > > The defines that I jam into CFLAGS are known to work in our other > projects on Linux, Solaris, FreeBSD and OSX. The other (weirder) > platforms might need some other adjustments to work correctly. I wonder - if we enable this patch, and there's some library that PHP modules use, which is not compiled with 64-bit files support, will it work OK (including using PHP file handles, calling file functions, etc.)? I am worried about something like changing the structure of FILE* and then both modules using different settings would access it. So do you know how LF-enabled code interoperates with non-lf-enabled code? Also I'm not sure MAKE_DOUBLE_ZVAL_INCREF is necessary - as I see, it's used only in 2 places. -- Stanislav Malyshev, Zend Software Architect stas@zend.com http://www.zend.com/ (408)253-8829 MSN: stas@zend.com