# Build failure - Thread: https://externals.io/message/28680 - Original archive: https://news-web.php.net/php.internals/28680 - Started by Matt Wilmas on 2007-04-05T03:50:21+00:00 - Messages: 25 - Reply by email to internals@lists.php.net - More for agents: search with https://externals.io/search.md?q={query} · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt --- ## #28680 — Matt Wilmas — 2007-04-05T03:50:21+00:00 URL: https://externals.io/message/28680#28680 Hi, Maybe just a Windows problem if it wasn't noticed yet, but I was compiling the latest 5.2 snapshot and got: main.obj : error LNK2019: unresolved external symbol _php_flock referenced in function _php_log_err Release_TS\php5ts.dll : fatal error LNK1120: 1 unresolved externals Caused by this recent commit, http://news.php.net/php.cvs/43683, and I commented the php_flock line as a workaround. The Windows 5.2 snapshots haven't been updated because of this either, of course. Matt --- ## #28681 — Rasmus Lerdorf — 2007-04-05T05:29:23+00:00 URL: https://externals.io/message/28680#28681 · In reply to: #28680 Matt Wilmas wrote: > Hi, > > Maybe just a Windows problem if it wasn't noticed yet, but I was compiling > the latest 5.2 snapshot and got: > > main.obj : error LNK2019: unresolved external symbol _php_flock referenced > in function _php_log_err > Release_TS\php5ts.dll : fatal error LNK1120: 1 unresolved externals > > Caused by this recent commit, http://news.php.net/php.cvs/43683, and I > commented the php_flock line as a workaround. The Windows 5.2 snapshots > haven't been updated because of this either, of course. I see no reason for that lock at all as I commented when this was committed, but Ilia never replied. This is a single write operation now since those fprintf's are now one, so that part of the fix is good, but the lock call is not needed since single writes in append mode are atomic, even on Windows. So, your work around is fine and should actually be committed. -Rasmus --- ## #28682 — Richard Quadling — 2007-04-05T10:47:20+00:00 URL: https://externals.io/message/28680#28682 · In reply to: #28681 http://rquadling.php1h.com/main.c.diff.txt for HEAD. I don't know how to patch a specific version. > Matt Wilmas wrote: > > […] > > I see no reason for that lock at all as I commented when this was > committed, but Ilia never replied. This is a single write operation now > since those fprintf's are now one, so that part of the fix is good, but > the lock call is not needed since single writes in append mode are > atomic, even on Windows. > > So, your work around is fine and should actually be committed. > > -Rasmus > > -- > > > > -- ----- Richard Quadling Zend Certified Engineer : http://zend.com/zce.php?c=ZEND002498&r=213474731 "Standing on the shoulders of some very clever giants!" --- ## #28683 — Ilia Alshanetsky — 2007-04-05T12:28:16+00:00 URL: https://externals.io/message/28680#28683 · In reply to: #28681 Rasmus, Sorry for the delay in the reply. According to my tests on linux using the sample script provided by the original bug reporter having no lock causes a problem when the error message is >4k in length. In this case multiple buffers are used and corruption can happen (it did on a dual cpu machine with 10 error log writing threads running), which is why I feel the lock is needed. > Matt Wilmas wrote: > > […] > > I see no reason for that lock at all as I commented when this was > committed, but Ilia never replied. This is a single write > operation now > since those fprintf's are now one, so that part of the fix is good, > but > the lock call is not needed since single writes in append mode are > atomic, even on Windows. > > So, your work around is fine and should actually be committed. > > -Rasmus > > -- > > > Ilia Alshanetsky --- ## #28686 — Richard Quadling — 2007-04-05T14:42:49+00:00 URL: https://externals.io/message/28680#28686 · In reply to: #28683 Using PHP 5.2.2-dev (cli) (built: Mar 23 2007 07:02:57) I can replicate the problem on Windows. Using this single line command at the CMD prompt: for %x in (A B C D E F G H I J K L M N O P Q R S T U V W X Y Z) do start php -r "ini_set('error_log','/tmp/test.log');for($i=0;$i<100;$i++)error_log(str_repeat('%x',5000));" The log file has many broken lines. > Rasmus, > > Sorry for the delay in the reply. According to my tests on linux > using the sample script provided by the original bug reporter having > no lock causes a problem when the error message is >4k in length. In > this case multiple buffers are used and corruption can happen (it did > on a dual cpu machine with 10 error log writing threads running), > which is why I feel the lock is needed. > > > > > > […] > > Ilia Alshanetsky > > -- > > > > -- ----- Richard Quadling Zend Certified Engineer : http://zend.com/zce.php?c=ZEND002498&r=213474731 "Standing on the shoulders of some very clever giants!" --- ## #28687 — Richard Quadling — 2007-04-05T14:47:06+00:00 URL: https://externals.io/message/28680#28687 · In reply to: #28686 Using ProcMon on WinXP, you can see that the different processes interleave the writes but in blocks of 4096 bytes. The remaining 929 bytes will sometimes be written after another processes 4096/929 bytes. Locking IS required here. > Using PHP 5.2.2-dev (cli) (built: Mar 23 2007 07:02:57) I can > replicate the problem on Windows. > > Using this single line command at the CMD prompt: > > for %x in (A B C D E F G H I J K L M N O P Q R S T U V W X Y Z) do > start php -r "ini_set('error_log','/tmp/test.log');for($i=0;$i<100;$i++)error_log(str_repeat('%x',5000));" > > The log file has many broken lines. > > > > […] > > > -- > ----- > Richard Quadling > Zend Certified Engineer : http://zend.com/zce.php?c=ZEND002498&r=213474731 > "Standing on the shoulders of some very clever giants!" > -- ----- Richard Quadling Zend Certified Engineer : http://zend.com/zce.php?c=ZEND002498&r=213474731 "Standing on the shoulders of some very clever giants!" --- ## #28688 — Rasmus Lerdorf — 2007-04-05T15:23:45+00:00 URL: https://externals.io/message/28680#28688 · In reply to: #28683 Ilia Alshanetsky wrote: > Rasmus, > > Sorry for the delay in the reply. According to my tests on linux using > the sample script provided by the original bug reporter having no lock > causes a problem when the error message is >4k in length. In this case > multiple buffers are used and corruption can happen (it did on a dual > cpu machine with 10 error log writing threads running), which is why I > feel the lock is needed. Did you do this test before or after you condensed the fprintf calls into a single call? With multiple fprintf calls there would be a problem, but with a single one I don't see how it could possibly screw it up. -Rasmus --- ## #28689 — Rasmus Lerdorf — 2007-04-05T15:27:03+00:00 URL: https://externals.io/message/28680#28689 · In reply to: #28686 Yes, but again, is this test with the single fprintf call? That's the real fix for this problem, not the lock. -Rasmus Richard Quadling wrote: > Using PHP 5.2.2-dev (cli) (built: Mar 23 2007 07:02:57) I can > replicate the problem on Windows. > > Using this single line command at the CMD prompt: > > for %x in (A B C D E F G H I J K L M N O P Q R S T U V W X Y Z) do > start php -r > "ini_set('error_log','/tmp/test.log');for($i=0;$i<100;$i++)error_log(str_repeat('%x',5000));" > > > The log file has many broken lines. > > > > […] --- ## #28690 — Richard Quadling — 2007-04-05T15:28:04+00:00 URL: https://externals.io/message/28680#28690 · In reply to: #28688 Is it possible that at the lowest layer within the C library that 4K blocking is performed? > Ilia Alshanetsky wrote: > > […] > > Did you do this test before or after you condensed the fprintf calls > into a single call? With multiple fprintf calls there would be a > problem, but with a single one I don't see how it could possibly screw > it up. > > -Rasmus > > -- > > > > -- ----- Richard Quadling Zend Certified Engineer : http://zend.com/zce.php?c=ZEND002498&r=213474731 "Standing on the shoulders of some very clever giants!" --- ## #28691 — Richard Quadling — 2007-04-05T15:30:32+00:00 URL: https://externals.io/message/28680#28691 · In reply to: #28689 This test is with the multiline fprintf and no lock. Adding the php_flock() has stalled the build process for Win32 (I can't build on Win32 as I don't know how!). > Yes, but again, is this test with the single fprintf call? That's the > real fix for this problem, not the lock. > > -Rasmus > > Richard Quadling wrote: > > […] > > -- ----- Richard Quadling Zend Certified Engineer : http://zend.com/zce.php?c=ZEND002498&r=213474731 "Standing on the shoulders of some very clever giants!" --- ## #28693 — Rob Richards — 2007-04-05T18:05:27+00:00 URL: https://externals.io/message/28680#28693 · In reply to: #28691 I can confirm that it does fail with the single fprintf call without the lock on Win32. Using `flock()` does fix the problem in my tests. I was able to fix win build using `flock()` instead of php_flock() and defining HAVE_FLOCK in win32/flock.h Rob Richard Quadling wrote: > This test is with the multiline fprintf and no lock. > > Adding the php_flock() has stalled the build process for Win32 (I > can't build on Win32 as I don't know how!). > > > > > > […] --- ## #28694 — Rasmus Lerdorf — 2007-04-05T18:36:49+00:00 URL: https://externals.io/message/28680#28694 · In reply to: #28693 That still doesn't make sense to me. Could you try doing an `sprintf()` followed by an `fwrite()` then? Is `fprintf()` perhaps doing something stupid internally? I'd really like to avoid a lock here. -Rasmus Rob Richards wrote: > I can confirm that it does fail with the single fprintf call without the > lock on Win32. > Using `flock()` does fix the problem in my tests. > I was able to fix win build using `flock()` instead of php_flock() and > defining HAVE_FLOCK in win32/flock.h > > Rob > > Richard Quadling wrote: > > […] --- ## #28695 — Rob Richards — 2007-04-05T20:40:41+00:00 URL: https://externals.io/message/28680#28695 · In reply to: #28694 No difference using `sprintf()`/fwrite() instead of `fprintf()`. I did come across a similar issue from apache: http://mail-archives.apache.org/mod_mbox/httpd-dev/199503.mbox/%3C9502272049.aa02036@paris.ics.uci.edu%3E Changing to use VCWD_OPEN_MODE, write() and close() seems to work. Rob Rasmus Lerdorf wrote: > That still doesn't make sense to me. Could you try doing an `sprintf()` > followed by an `fwrite()` then? Is `fprintf()` perhaps doing something > stupid internally? I'd really like to avoid a lock here. > > -Rasmus > > Rob Richards wrote: > > > […] --- ## #28784 — Richard Quadling — 2007-04-13T10:33:22+00:00 URL: https://externals.io/message/28680#28784 · In reply to: #28695 > No difference using `sprintf()`/fwrite() instead of `fprintf()`. > > I did come across a similar issue from apache: > http://mail-archives.apache.org/mod_mbox/httpd-dev/199503.mbox/%3C9502272049.aa02036@paris.ics.uci.edu%3E > > Changing to use VCWD_OPEN_MODE, write() and close() seems to work. > > Rob > Using PHP 5.2.2RC2-dev (cli) (built: Apr 13 2007 04:03:02) on Windows with for %x in (A B C D E F G H I J K L M N O P Q R S T U V W X Y Z) do start php -r "ini_set('error_log','/tmp/test.log');for($i=0;$i<1000;$i++)error_log(str_repeat('%x',5000));"`; And then doing some analysis of the log file I'm finding that not all the data has been written. Array ( [A] => 4965993 [B] => 4590918 [C] => 4525905 [D] => 4695939 [E] => 4495899 [F] => 4710942 [G] => 4495899 [H] => 4645929 [I] => 4540908 [J] => 4580916 [K] => 4535907 [L] => 4470894 [M] => 4480896 [N] => 4550910 [O] => 4610922 [P] => 4500900 [Q] => 4630926 [R] => 4480896 [S] => 4500900 [T] => 4535907 [U] => 4630926 [V] => 4470894 [W] => 4645929 [X] => 4825965 [Y] => 4845969 [Z] => 4920984 ) Lines (Should be 26 * 1000) : 23973 Longest (Should be 5024) : 5024 Shortest (Should be 5024) : 5024 Out of the 26,000 lines expected we are missing 2027 lines. So no corruption of the lines, just missing ones. The only explanation I can think of is that the locking is working, but nothing is waiting for the lock to become available. Richard. -- ----- Richard Quadling Zend Certified Engineer : http://zend.com/zce.php?c=ZEND002498&r=213474731 "Standing on the shoulders of some very clever giants!" --- ## #28786 — Ilia Alshanetsky — 2007-04-13T12:55:47+00:00 URL: https://externals.io/message/28680#28786 · In reply to: #28784 The new implementation does not use any locks, instead it uses direct io, where locks are not necessary for append operations. > > > […] > > Using PHP 5.2.2RC2-dev (cli) (built: Apr 13 2007 04:03:02) on > Windows with > > for %x in (A B C D E F G H I J K L M N O P Q R S T U V W X Y Z) do > start php -r "ini_set('error_log','/tmp/test.log');for($i=0;$i<1000; > $i++)error_log(str_repeat('%x',5000));"`; > > And then doing some analysis of the log file I'm finding that not all > the data has been written. > > Array > ( > [A] => 4965993 > [B] => 4590918 > [C] => 4525905 > [D] => 4695939 > [E] => 4495899 > [F] => 4710942 > [G] => 4495899 > [H] => 4645929 > [I] => 4540908 > [J] => 4580916 > [K] => 4535907 > [L] => 4470894 > [M] => 4480896 > [N] => 4550910 > [O] => 4610922 > [P] => 4500900 > [Q] => 4630926 > [R] => 4480896 > [S] => 4500900 > [T] => 4535907 > [U] => 4630926 > [V] => 4470894 > [W] => 4645929 > [X] => 4825965 > [Y] => 4845969 > [Z] => 4920984 > ) > Lines (Should be 26 * 1000) : 23973 > Longest (Should be 5024) : 5024 > Shortest (Should be 5024) : 5024 > > Out of the 26,000 lines expected we are missing 2027 lines. > > So no corruption of the lines, just missing ones. > > The only explanation I can think of is that the locking is working, > but nothing is waiting for the lock to become available. > > Richard. > > > -- > ----- > Richard Quadling > Zend Certified Engineer : http://zend.com/zce.php? > c=ZEND002498&r=213474731 > "Standing on the shoulders of some very clever giants!" Ilia Alshanetsky --- ## #28787 — Richard Quadling — 2007-04-13T13:42:06+00:00 URL: https://externals.io/message/28680#28787 · In reply to: #28786 So why are there missing entries? I can even get the corruption back again if I use a shorter line (100 rather than 5000). > The new implementation does not use any locks, instead it uses direct > io, where locks are not necessary for append operations. > > > > > > […] > > Ilia Alshanetsky > > > > > -- ----- Richard Quadling Zend Certified Engineer : http://zend.com/zce.php?c=ZEND002498&r=213474731 "Standing on the shoulders of some very clever giants!" --- ## #28812 — Rob Richards — 2007-04-14T11:16:35+00:00 URL: https://externals.io/message/28680#28812 · In reply to: #28787 I can sometimes reproduce the missing entries though never get data corruption anymore. This might now be a Windows only issue with how it is caching writes. I did a little debugging and no errors occurred. The file was opened successfully every time, the data was written (correct number of bytes as well) every single time, but depending upon the load on my system at the time it was a crap shoot whether every single write actually made it into the physical file or not. Adding a lock here did resolve it so that it worked 100% of the time. Rob Richard Quadling wrote: > So why are there missing entries? > > I can even get the corruption back again if I use a shorter line (100 > rather than 5000). > > > > > […] --- ## #28853 — Richard Quadling — 2007-04-16T08:55:29+00:00 URL: https://externals.io/message/28680#28853 · In reply to: #28812 That's good news. Has a patch been submitted? Even if it is initially only a win32 compiler directive wrapping the lock? > I can sometimes reproduce the missing entries though never get data > corruption anymore. > This might now be a Windows only issue with how it is caching writes. I > did a little debugging and no errors occurred. > The file was opened successfully every time, the data was written > (correct number of bytes as well) every single time, but depending upon > the load on my system at the time it was a crap shoot whether every > single write actually made it into the physical file or not. Adding a > lock here did resolve it so that it worked 100% of the time. > > Rob > > Richard Quadling wrote: > > […] > -- ----- Richard Quadling Zend Certified Engineer : http://zend.com/zce.php?c=ZEND002498&r=213474731 "Standing on the shoulders of some very clever giants!" --- ## #28858 — William A. Rowe — 2007-04-16T12:02:01+00:00 URL: https://externals.io/message/28680#28858 · In reply to: #28853 On Windows, there is no such thing as a true open-write-for-append. Unlike Unix, write for append mode is not atomic, internally it is a seek + write. Mix that with multiple writers, and you have an inherent race condition built it. Richard Quadling wrote: > That's good news. Has a patch been submitted? Even if it is initially > only a win32 compiler directive wrapping the lock? > > > > […] --- ## #28859 — Richard Quadling — 2007-04-16T12:07:47+00:00 URL: https://externals.io/message/28680#28859 · In reply to: #28858 So isn't locking the solution for Windows? > On Windows, there is no such thing as a true open-write-for-append. > > Unlike Unix, write for append mode is not atomic, internally it is a > seek + write. Mix that with multiple writers, and you have an inherent > race condition built it. > > > Richard Quadling wrote: > > […] > > -- ----- Richard Quadling Zend Certified Engineer : http://zend.com/zce.php?c=ZEND002498&r=213474731 "Standing on the shoulders of some very clever giants!" --- ## #28860 — William A. Rowe — 2007-04-16T14:46:18+00:00 URL: https://externals.io/message/28680#28860 · In reply to: #28859 Richard Quadling wrote: > So isn't locking the solution for Windows? If this is single writer process, even with multithreads - a mutex is most efficient, otherwise with concurrent writer processes, file locking makes the most sense. Bill --- ## #28861 — Richard Quadling — 2007-04-16T14:53:44+00:00 URL: https://externals.io/message/28680#28861 · In reply to: #28860 The testing I'm using is to launch multiple copies of the same PHP script simultaneously using php.exe (CLI mode). This could easily be multiple FastCGI scripts or multiple ISAPI threads. > Richard Quadling wrote: > > […] > > If this is single writer process, even with multithreads - a mutex > is most efficient, otherwise with concurrent writer processes, file > locking makes the most sense. > > Bill > -- ----- Richard Quadling Zend Certified Engineer : http://zend.com/zce.php?c=ZEND002498&r=213474731 "Standing on the shoulders of some very clever giants!" --- ## #28889 — Rob Richards — 2007-04-17T14:59:51+00:00 URL: https://externals.io/message/28680#28889 · In reply to: #28861 Unless anyone sees a way to do this without adding the lock, I'd like to commit this patch. It adds a php_flock call only for windows. In doing so I got rid of the win32/flock files and now compile with flock_compat from main. Rob Richard Quadling wrote: > The testing I'm using is to launch multiple copies of the same PHP > script simultaneously using php.exe (CLI mode). This could easily be > multiple FastCGI scripts or multiple ISAPI threads. > > > > […] --- ## #28917 — Richard Quadling — 2007-04-18T10:46:24+00:00 URL: https://externals.io/message/28680#28917 · In reply to: #28889 This looks like it will be available in snaps.php.net in about 40 minutes. Looking forward to it! > Unless anyone sees a way to do this without adding the lock, I'd like to > commit this patch. > It adds a php_flock call only for windows. In doing so I got rid of the > win32/flock files and now compile with flock_compat from main. > > Rob > > > Richard Quadling wrote: > > […] > > Index: ext/standard/config.w32 > =================================================================== > RCS file: /repository/php-src/ext/standard/config.w32,v > retrieving revision 1.4.2.2 > diff -u -r1.4.2.2 config.w32 > --- ext/standard/config.w32 4 Jan 2006 21:31:29 -0000 1.4.2.2 > +++ ext/standard/config.w32 17 Apr 2007 11:34:16 -0000 > @@ -16,5 +16,5 @@ > url_scanner_ex.c ftp_fopen_wrapper.c http_fopen_wrapper.c \ > php_fopen_wrapper.c credits.c css.c var_unserializer.c ftok.c sha1.c \ > user_filters.c uuencode.c filters.c proc_open.c \ > - streamsfuncs.c http.c", false /* never shared */); > + streamsfuncs.c http.c flock_compat.c", false /* never shared */); > > Index: main/main.c > =================================================================== > RCS file: /repository/php-src/main/main.c,v > retrieving revision 1.640.2.23.2.34 > diff -u -r1.640.2.23.2.34 main.c > --- main/main.c 16 Apr 2007 08:09:56 -0000 1.640.2.23.2.34 > +++ main/main.c 17 Apr 2007 11:49:49 -0000 > @@ -63,6 +63,7 @@ > #ifdef PHP_WIN32 > #include > #include "win32/php_registry.h" > +#include "ext/standard/flock_compat.h" > #endif > #include "php_syslog.h" > #include "Zend/zend_exceptions.h" > @@ -362,8 +363,11 @@ > time(&error_time); > strftime(error_time_str, sizeof(error_time_str), "%d-%b-%Y %H:%M:%S", php_localtime_r(&error_time, &tmbuf)); > len = spprintf(&tmp, 0, "[%s] %s%s", error_time_str, log_message, PHP_EOL); > +#ifdef PHP_WIN32 > + php_flock(fd, 2); > +#endif > write(fd, tmp, len); > - efree(tmp); > + efree(tmp); > close(fd); > return; > } > Index: win32/build/config.w32 > =================================================================== > RCS file: /repository/php-src/win32/build/config.w32,v > retrieving revision 1.40.2.8.2.9 > diff -u -r1.40.2.8.2.9 config.w32 > --- win32/build/config.w32 16 Apr 2007 08:09:56 -0000 1.40.2.8.2.9 > +++ win32/build/config.w32 17 Apr 2007 11:39:05 -0000 > @@ -309,7 +309,7 @@ > ADD_SOURCES("main/streams", "streams.c cast.c memory.c filter.c plain_wrapper.c \ > userspace.c transports.c xp_socket.c mmap.c"); > > -ADD_SOURCES("win32", "crypt_win32.c flock.c glob.c md5crypt.c readdir.c \ > +ADD_SOURCES("win32", "crypt_win32.c glob.c md5crypt.c readdir.c \ > registry.c select.c sendmail.c time.c wfile.c winutil.c wsyslog.c globals.c"); > > ADD_SOURCES("regex", "regcomp.c regerror.c regexec.c regfree.c"); > > -- ----- Richard Quadling Zend Certified Engineer : http://zend.com/zce.php?c=ZEND002498&r=213474731 "Standing on the shoulders of some very clever giants!" --- ## #28923 — Richard Quadling — 2007-04-19T07:57:05+00:00 URL: https://externals.io/message/28680#28923 · In reply to: #28917 And lo, the snapshot was "A Good Thing". > This looks like it will be available in snaps.php.net in about 40 minutes. > > Looking forward to it! > > > > […] > > > -- > ----- > Richard Quadling > Zend Certified Engineer : http://zend.com/zce.php?c=ZEND002498&r=213474731 > "Standing on the shoulders of some very clever giants!" > -- ----- Richard Quadling Zend Certified Engineer : http://zend.com/zce.php?c=ZEND002498&r=213474731 "Standing on the shoulders of some very clever giants!"