# _FILE_OFFSET_BITS=64 considered harmful - Thread: https://externals.io/message/13440 - Original archive: https://news-web.php.net/php.internals/13440 - Started by Joe Orton on 2004-10-22T06:22:49+00:00 - Messages: 13 - Reply by email to internals@lists.php.net - More for agents: search with https://externals.io/search.md?q={query} · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt --- ## #13440 — Joe Orton — 2004-10-22T06:22:49+00:00 URL: https://externals.io/message/13440#13440 There are serious problems from enabling LFS support like this in a project like PHP. If I have some library which uses off_t in its API, e.g. zlib, and I happened to not compile it with LFS support, e.g. as in most Linux distributions, I now *cannot* call the zlib functions using off_t from PHP, because the size of off_t when you #include will be different from the size when zlib is compiled. Similar cases for any SAPI module if the server interface exposes off_t. The safe way of doing this which we use in APR 1.0 is to define a wrapper type (e.g php_off_t) which is typedef'd to off64_t when -D_LARGEFILE64_SOURCE is supported, and off_t otherwise, to open files using O_LARGEFILE, and to use the "transitional LFS" functions like lseek64 etc in place of lseek etc when php_off_t is off64_t. ----- Forwarded message from Edin Kadribasic ----- From: "Edin Kadribasic" To: php-cvs@lists.php.net Date: Thu, 21 Oct 2004 23:57:39 -0000 Subject: [PHP-CVS] cvs: php-src(PHP_5_0) / configure.in edink Thu Oct 21 19:57:39 2004 EDT Modified files: (Branch: PHP_5_0) /php-src configure.in Log: MFB: Linux LFS support, fixes 27792 http://cvs.php.net/diff.php/php-src/configure.in?r1=1.514.2.11&r2=1.514.2.12&ty=u Index: php-src/configure.in diff -u php-src/configure.in:1.514.2.11 php-src/configure.in:1.514.2.12 --- php-src/configure.in:1.514.2.11 Tue Sep 28 05:13:32 2004 +++ php-src/configure.in Thu Oct 21 19:57:37 2004 @@ -1,4 +1,4 @@ -dnl ## $Id: configure.in,v 1.514.2.11 2004/09/28 09:13:32 wez Exp $ -*- sh -*- +dnl ## $Id: configure.in,v 1.514.2.12 2004/10/21 23:57:37 edink Exp $ -*- sh -*- dnl ## Process this file with autoconf to produce a configure script. divert(1) @@ -196,6 +196,13 @@ LIBS="$LIBS -lbe -lroot";; *mips*) CPPFLAGS="$CPPFLAGS -D_XPG_IV";; +*linux*) + AC_MSG_CHECKING([for Linux LFS_CFLAGS]) + LFS_LINUX_CFLAGS=`getconf LFS_CFLAGS` + if test "x$LFS_LINUX_CFLAGS" != "x"; then + CFLAGS="$CFLAGS $LFS_LINUX_CFLAGS" + fi + AC_MSG_RESULT([$LFS_LINUX_CFLAGS]) esac -- PHP CVS Mailing List (http://www.php.net/) ----- End forwarded message --- ## #13442 — Wez Furlong — 2004-10-22T08:53:14+00:00 URL: https://externals.io/message/13440#13442 · In reply to: #13440 What I planned to do with the streams API for 5.1 was define php_stream_off_t to be a 64-bit type (regardless of LFS support), adjust the API where it is needed, and handle the LFS stuff centrally, using the transitional LFS functions you mentioned if they are present. I confess that I haven't delved in to LFS on *nix very deeply at this point, so if you have any tips, I'd welcome your opinion. --Wez. > The safe way of doing this which we use in APR 1.0 is to define a > wrapper type (e.g php_off_t) which is typedef'd to off64_t when > -D_LARGEFILE64_SOURCE is supported, and off_t otherwise, to open files > using O_LARGEFILE, and to use the "transitional LFS" functions like > lseek64 etc in place of lseek etc when php_off_t is off64_t. --- ## #13448 — Joe Orton — 2004-10-22T09:45:08+00:00 URL: https://externals.io/message/13440#13448 · In reply to: #13442 > What I planned to do with the streams API for 5.1 was define > php_stream_off_t to be a 64-bit type (regardless of LFS support), > adjust the API where it is needed, and handle the LFS stuff centrally, > using the transitional LFS functions you mentioned if they are > present. The difference in that approach is that you introduce a bunch of new error cases on platforms *without* LFS support, where you'd have to check for passing an out-of-range 64-bit php_stream_off_t value to a function taking a 32-bit off_t. I think using php_stream_off_t = off_t or off64_t as appropriate is simpler for that reason. But both approaches are feasible, the important thing is to avoid using -D_FILE_OFFSET_BITS=64, which just breaks so much. Regards, joe --- ## #13449 — Wez Furlong — 2004-10-22T09:53:25+00:00 URL: https://externals.io/message/13440#13449 · In reply to: #13448 > > > […] > > The difference in that approach is that you introduce a bunch of new > error cases on platforms *without* LFS support, where you'd have to > check for passing an out-of-range 64-bit php_stream_off_t value to a > function taking a 32-bit off_t. I think using php_stream_off_t = off_t > or off64_t as appropriate is simpler for that reason. I think it is important to force the size of the type, because we need to be aware of the size of the type when passing it to/from zvals. I'm happy with checking the range for non-LFS platforms, as this should have minimal impact compared to the cost of actually operating on the descriptor, and helps us to standardize on the API. > But both approaches are feasible, the important thing is to avoid using > -D_FILE_OFFSET_BITS=64, which just breaks so much. OK, so we should roll back Edins patch from all branches and wait for a safer fix. --Wez. --- ## #13450 — Joe Orton — 2004-10-22T10:08:42+00:00 URL: https://externals.io/message/13440#13450 · In reply to: #13449 > > > […] > > I think it is important to force the size of the type, because we need to > be aware of the size of the type when passing it to/from zvals. > I'm happy with checking the range for non-LFS platforms, > as this should have minimal impact compared to the cost of actually > operating on the descriptor, and helps us to standardize on the API. It's not the runtime cost which was my concern for APR so much as the cost of more complex code and the risk of having more if() conditions which aren't exercised in testing. But in httpd/APR we can ensure that file offsets are stored in apr_off_t's in 99% of the code so it's a different ball game if you need to pass these about with zvals I guess. > > […] > > OK, so we should roll back Edins patch from all branches and wait for > a safer fix. Gets my vote... joe --- ## #13451 — Edin Kadribasic — 2004-10-22T10:59:04+00:00 URL: https://externals.io/message/13440#13451 · In reply to: #13450 > > […] As far as I can tell we use off_t only once when calling external libraries. And yes it is in a call to zlib when seeking inside gzipped files. However I consider crashing apache children with signal 25 when doing simple `is_file()` or `fopen()` on large files really harmful. Apache flat out refuses to start if you have enabled php error log and that file happen to be 2GB or bigger. > > […] > > Gets my vote... OK. Reverted. Edin --- ## #13452 — Joe Orton — 2004-10-22T11:33:51+00:00 URL: https://externals.io/message/13440#13452 · In reply to: #13451 > However I consider crashing apache children with signal 25 when doing simple > `is_file()` or `fopen()` on large files really harmful. Apache flat out refuses > to start if you have enabled php error log and that file happen to be 2GB or > bigger. For Linux at least that was fixed since 2.0.50, or are you using 1.3 still? w.r.t `is_file()` or open(), can you give more details? open() on a >2Gb file will give EFBIG not die with SIGXFSZ. *writing* beyond 2Gb will die with SIGXFSZ, yes, is that what you mean? I had a patch to make sure `SIGXFSZ` was ignored but didn't bother with it since the large logfile stuff went in which solves the common problem. Regards, joe --- ## #13453 — Edin Kadribasic — 2004-10-22T12:26:50+00:00 URL: https://externals.io/message/13440#13453 · In reply to: #13452 > > > […] > > For Linux at least that was fixed since 2.0.50, or are you using 1.3 > still? I still use 1.3 (like, I guess, most of the PHP installations outthere). > w.r.t `is_file()` or open(), can you give more details? open() on a >2Gb > file will give EFBIG not die with SIGXFSZ. *writing* beyond 2Gb will > die with SIGXFSZ, yes, is that what you mean? I had a patch to make > sure `SIGXFSZ` was ignored but didn't bother with it since the large > logfile stuff went in which solves the common problem. Yes, `fwrite()` will die with SIGXFSZ. `is_file()` `file_exists()` and friends will fail returning bugus results. A number of bug reports mention these issues. Just a quick google search gave me #15260 #27765 #24411 #27792. Edin --- ## #13460 — Joe Orton — 2004-10-22T13:42:49+00:00 URL: https://externals.io/message/13440#13460 · In reply to: #13453 > > > […] > > I still use 1.3 (like, I guess, most of the PHP installations outthere). Then consider this your official wake-up call :) > > […] > > Yes, `fwrite()` will die with SIGXFSZ. I'll fix this in httpd, hopefully for 2.0.53. joe --- ## #13461 — Ilia Alshanetsky — 2004-10-22T13:49:55+00:00 URL: https://externals.io/message/13440#13461 · In reply to: #13460 Joe Orton wrote: > Then consider this your official wake-up call :) LOL. I use Apache 1.3 and found it to work way better then 2.0. 2.0 does not offer any useful improvements over the 1.3 base, so why switch? If anything in my personal experience 1.3 works way better then 2.0 with PHP. Ilia --- ## #13462 — Derick Rethans — 2004-10-22T13:51:44+00:00 URL: https://externals.io/message/13440#13462 · In reply to: #13460 > > […] > > Then consider this your official wake-up call :) Perhaps we can consider that when Apache 2 uses static modules like Apache 1.3. But for now Apache 1.3 works great, and that can not always be said about Apache 2, so I don't see a real reason why to switch over. Derick -- Derick Rethans http://derickrethans.nl | http://ez.no | http://xdebug.org --- ## #13463 — Edin Kadribasic — 2004-10-22T14:02:55+00:00 URL: https://externals.io/message/13440#13463 · In reply to: #13460 > > > […] > > Then consider this your official wake-up call :) This is unrealistic. I'd rather have PHP working with my Apache 1.3 without crashing it. So so far _FILE_OFFSET_BITS=64 fixes one crash, makes is_*() functions work, allows opening an writing large files. On the downside it creates binary compatibility problems with external libraries which use off_t in function prototypes (I found one such call in php4 source, but it could be that I have missed some). I know I will have it enabled in all my PHP builds... Edin --- ## #13467 — Joe Orton — 2004-10-22T15:16:09+00:00 URL: https://externals.io/message/13440#13467 · In reply to: #13461 > Joe Orton wrote: > > […] > > LOL. I use Apache 1.3 and found it to work way better then 2.0. 2.0 does > not offer any useful improvements over the 1.3 base, so why switch? If > anything in my personal experience 1.3 works way better then 2.0 with PHP. Silly old me for daring to point out that a nasty 1.3 bug which is fixed in 2.0 might make it interesting for someone to upgrade, eh? :) I'm glad you can choose 1.3, I'm glad others can choose 2.0. joe