Hi,
Maybe just a Windows problem if it wasn't noticed yet, but I was compiling
the latest 5.2 snapshot and got:
main.obj : error LNK2019: unresolved external symbol _php_flock referenced
in function _php_log_err
Release_TS\php5ts.dll : fatal error LNK1120: 1 unresolved externals
Caused by this recent commit, http://news.php.net/php.cvs/43683, and I
commented the php_flock line as a workaround. The Windows 5.2 snapshots
haven't been updated because of this either, of course.
Matt
Matt Wilmas wrote:
Hi,
Maybe just a Windows problem if it wasn't noticed yet, but I was compiling
the latest 5.2 snapshot and got:main.obj : error LNK2019: unresolved external symbol _php_flock referenced
in function _php_log_err
Release_TS\php5ts.dll : fatal error LNK1120: 1 unresolved externalsCaused by this recent commit, http://news.php.net/php.cvs/43683, and I
commented the php_flock line as a workaround. The Windows 5.2 snapshots
haven't been updated because of this either, of course.
I see no reason for that lock at all as I commented when this was
committed, but Ilia never replied. This is a single write operation now
since those fprintf's are now one, so that part of the fix is good, but
the lock call is not needed since single writes in append mode are
atomic, even on Windows.
So, your work around is fine and should actually be committed.
-Rasmus
http://rquadling.php1h.com/main.c.diff.txt for HEAD. I don't know how
to patch a specific version.
Matt Wilmas wrote:
Hi,
Maybe just a Windows problem if it wasn't noticed yet, but I was compiling
the latest 5.2 snapshot and got:main.obj : error LNK2019: unresolved external symbol _php_flock referenced
in function _php_log_err
Release_TS\php5ts.dll : fatal error LNK1120: 1 unresolved externalsCaused by this recent commit, http://news.php.net/php.cvs/43683, and I
commented the php_flock line as a workaround. The Windows 5.2 snapshots
haven't been updated because of this either, of course.I see no reason for that lock at all as I commented when this was
committed, but Ilia never replied. This is a single write operation now
since those fprintf's are now one, so that part of the fix is good, but
the lock call is not needed since single writes in append mode are
atomic, even on Windows.So, your work around is fine and should actually be committed.
-Rasmus
--
--
Richard Quadling
Zend Certified Engineer : http://zend.com/zce.php?c=ZEND002498&r=213474731
"Standing on the shoulders of some very clever giants!"
Rasmus,
Sorry for the delay in the reply. According to my tests on linux
using the sample script provided by the original bug reporter having
no lock causes a problem when the error message is >4k in length. In
this case multiple buffers are used and corruption can happen (it did
on a dual cpu machine with 10 error log writing threads running),
which is why I feel the lock is needed.
Matt Wilmas wrote:
Hi,
Maybe just a Windows problem if it wasn't noticed yet, but I was
compiling
the latest 5.2 snapshot and got:main.obj : error LNK2019: unresolved external symbol _php_flock
referenced
in function _php_log_err
Release_TS\php5ts.dll : fatal error LNK1120: 1 unresolved externalsCaused by this recent commit, http://news.php.net/php.cvs/43683,
and I
commented the php_flock line as a workaround. The Windows 5.2
snapshots
haven't been updated because of this either, of course.I see no reason for that lock at all as I commented when this was
committed, but Ilia never replied. This is a single write
operation now
since those fprintf's are now one, so that part of the fix is good,
but
the lock call is not needed since single writes in append mode are
atomic, even on Windows.So, your work around is fine and should actually be committed.
-Rasmus
--
Ilia Alshanetsky
Using PHP 5.2.2-dev (cli) (built: Mar 23 2007 07:02:57) I can
replicate the problem on Windows.
Using this single line command at the CMD prompt:
for %x in (A B C D E F G H I J K L M N O P Q R S T U V W X Y Z) do
start php -r "ini_set('error_log','/tmp/test.log');for($i=0;$i<100;$i++)error_log(str_repeat('%x',5000));"
The log file has many broken lines.
Rasmus,
Sorry for the delay in the reply. According to my tests on linux
using the sample script provided by the original bug reporter having
no lock causes a problem when the error message is >4k in length. In
this case multiple buffers are used and corruption can happen (it did
on a dual cpu machine with 10 error log writing threads running),
which is why I feel the lock is needed.Matt Wilmas wrote:
Hi,
Maybe just a Windows problem if it wasn't noticed yet, but I was
compiling
the latest 5.2 snapshot and got:main.obj : error LNK2019: unresolved external symbol _php_flock
referenced
in function _php_log_err
Release_TS\php5ts.dll : fatal error LNK1120: 1 unresolved externalsCaused by this recent commit, http://news.php.net/php.cvs/43683,
and I
commented the php_flock line as a workaround. The Windows 5.2
snapshots
haven't been updated because of this either, of course.I see no reason for that lock at all as I commented when this was
committed, but Ilia never replied. This is a single write
operation now
since those fprintf's are now one, so that part of the fix is good,
but
the lock call is not needed since single writes in append mode are
atomic, even on Windows.So, your work around is fine and should actually be committed.
-Rasmus
--
Ilia Alshanetsky
--
--
Richard Quadling
Zend Certified Engineer : http://zend.com/zce.php?c=ZEND002498&r=213474731
"Standing on the shoulders of some very clever giants!"
Using ProcMon on WinXP, you can see that the different processes
interleave the writes but in blocks of 4096 bytes. The remaining 929
bytes will sometimes be written after another processes 4096/929
bytes.
Locking IS required here.
Using PHP 5.2.2-dev (cli) (built: Mar 23 2007 07:02:57) I can
replicate the problem on Windows.Using this single line command at the CMD prompt:
for %x in (A B C D E F G H I J K L M N O P Q R S T U V W X Y Z) do
start php -r "ini_set('error_log','/tmp/test.log');for($i=0;$i<100;$i++)error_log(str_repeat('%x',5000));"The log file has many broken lines.
Rasmus,
Sorry for the delay in the reply. According to my tests on linux
using the sample script provided by the original bug reporter having
no lock causes a problem when the error message is >4k in length. In
this case multiple buffers are used and corruption can happen (it did
on a dual cpu machine with 10 error log writing threads running),
which is why I feel the lock is needed.Matt Wilmas wrote:
Hi,
Maybe just a Windows problem if it wasn't noticed yet, but I was
compiling
the latest 5.2 snapshot and got:main.obj : error LNK2019: unresolved external symbol _php_flock
referenced
in function _php_log_err
Release_TS\php5ts.dll : fatal error LNK1120: 1 unresolved externalsCaused by this recent commit, http://news.php.net/php.cvs/43683,
and I
commented the php_flock line as a workaround. The Windows 5.2
snapshots
haven't been updated because of this either, of course.I see no reason for that lock at all as I commented when this was
committed, but Ilia never replied. This is a single write
operation now
since those fprintf's are now one, so that part of the fix is good,
but
the lock call is not needed since single writes in append mode are
atomic, even on Windows.So, your work around is fine and should actually be committed.
-Rasmus
--
Ilia Alshanetsky
--
--
Richard Quadling
Zend Certified Engineer : http://zend.com/zce.php?c=ZEND002498&r=213474731
"Standing on the shoulders of some very clever giants!"
--
Richard Quadling
Zend Certified Engineer : http://zend.com/zce.php?c=ZEND002498&r=213474731
"Standing on the shoulders of some very clever giants!"
Yes, but again, is this test with the single fprintf call? That's the
real fix for this problem, not the lock.
-Rasmus
Richard Quadling wrote:
Using PHP 5.2.2-dev (cli) (built: Mar 23 2007 07:02:57) I can
replicate the problem on Windows.Using this single line command at the CMD prompt:
for %x in (A B C D E F G H I J K L M N O P Q R S T U V W X Y Z) do
start php -r
"ini_set('error_log','/tmp/test.log');for($i=0;$i<100;$i++)error_log(str_repeat('%x',5000));"The log file has many broken lines.
Rasmus,
Sorry for the delay in the reply. According to my tests on linux
using the sample script provided by the original bug reporter having
no lock causes a problem when the error message is >4k in length. In
this case multiple buffers are used and corruption can happen (it did
on a dual cpu machine with 10 error log writing threads running),
which is why I feel the lock is needed.Matt Wilmas wrote:
Hi,
Maybe just a Windows problem if it wasn't noticed yet, but I was
compiling
the latest 5.2 snapshot and got:main.obj : error LNK2019: unresolved external symbol _php_flock
referenced
in function _php_log_err
Release_TS\php5ts.dll : fatal error LNK1120: 1 unresolved externalsCaused by this recent commit, http://news.php.net/php.cvs/43683,
and I
commented the php_flock line as a workaround. The Windows 5.2
snapshots
haven't been updated because of this either, of course.I see no reason for that lock at all as I commented when this was
committed, but Ilia never replied. This is a single write
operation now
since those fprintf's are now one, so that part of the fix is good,
but
the lock call is not needed since single writes in append mode are
atomic, even on Windows.So, your work around is fine and should actually be committed.
-Rasmus
--
Ilia Alshanetsky
This test is with the multiline fprintf and no lock.
Adding the php_flock() has stalled the build process for Win32 (I
can't build on Win32 as I don't know how!).
Yes, but again, is this test with the single fprintf call? That's the
real fix for this problem, not the lock.-Rasmus
Richard Quadling wrote:
Using PHP 5.2.2-dev (cli) (built: Mar 23 2007 07:02:57) I can
replicate the problem on Windows.Using this single line command at the CMD prompt:
for %x in (A B C D E F G H I J K L M N O P Q R S T U V W X Y Z) do
start php -r
"ini_set('error_log','/tmp/test.log');for($i=0;$i<100;$i++)error_log(str_repeat('%x',5000));"The log file has many broken lines.
Rasmus,
Sorry for the delay in the reply. According to my tests on linux
using the sample script provided by the original bug reporter having
no lock causes a problem when the error message is >4k in length. In
this case multiple buffers are used and corruption can happen (it did
on a dual cpu machine with 10 error log writing threads running),
which is why I feel the lock is needed.Matt Wilmas wrote:
Hi,
Maybe just a Windows problem if it wasn't noticed yet, but I was
compiling
the latest 5.2 snapshot and got:main.obj : error LNK2019: unresolved external symbol _php_flock
referenced
in function _php_log_err
Release_TS\php5ts.dll : fatal error LNK1120: 1 unresolved externalsCaused by this recent commit, http://news.php.net/php.cvs/43683,
and I
commented the php_flock line as a workaround. The Windows 5.2
snapshots
haven't been updated because of this either, of course.I see no reason for that lock at all as I commented when this was
committed, but Ilia never replied. This is a single write
operation now
since those fprintf's are now one, so that part of the fix is good,
but
the lock call is not needed since single writes in append mode are
atomic, even on Windows.So, your work around is fine and should actually be committed.
-Rasmus
--
Ilia Alshanetsky
--
--
Richard Quadling
Zend Certified Engineer : http://zend.com/zce.php?c=ZEND002498&r=213474731
"Standing on the shoulders of some very clever giants!"
I can confirm that it does fail with the single fprintf call without the
lock on Win32.
Using flock() does fix the problem in my tests.
I was able to fix win build using flock() instead of php_flock() and
defining HAVE_FLOCK in win32/flock.h
Rob
Richard Quadling wrote:
This test is with the multiline fprintf and no lock.
Adding the php_flock() has stalled the build process for Win32 (I
can't build on Win32 as I don't know how!).Yes, but again, is this test with the single fprintf call? That's the
real fix for this problem, not the lock.-Rasmus
Richard Quadling wrote:
Using PHP 5.2.2-dev (cli) (built: Mar 23 2007 07:02:57) I can
replicate the problem on Windows.Using this single line command at the CMD prompt:
for %x in (A B C D E F G H I J K L M N O P Q R S T U V W X Y Z) do
start php -r"ini_set('error_log','/tmp/test.log');for($i=0;$i<100;$i++)error_log(str_repeat('%x',5000));"
The log file has many broken lines.
Rasmus,
Sorry for the delay in the reply. According to my tests on linux
using the sample script provided by the original bug reporter having
no lock causes a problem when the error message is >4k in length. In
this case multiple buffers are used and corruption can happen (it did
on a dual cpu machine with 10 error log writing threads running),
which is why I feel the lock is needed.Matt Wilmas wrote:
Hi,
Maybe just a Windows problem if it wasn't noticed yet, but I was
compiling
the latest 5.2 snapshot and got:main.obj : error LNK2019: unresolved external symbol _php_flock
referenced
in function _php_log_err
Release_TS\php5ts.dll : fatal error LNK1120: 1 unresolved
externalsCaused by this recent commit, http://news.php.net/php.cvs/43683,
and I
commented the php_flock line as a workaround. The Windows 5.2
snapshots
haven't been updated because of this either, of course.I see no reason for that lock at all as I commented when this was
committed, but Ilia never replied. This is a single write
operation now
since those fprintf's are now one, so that part of the fix is good,
but
the lock call is not needed since single writes in append mode are
atomic, even on Windows.So, your work around is fine and should actually be committed.
-Rasmus
--
Ilia Alshanetsky
That still doesn't make sense to me. Could you try doing an sprintf()
followed by an fwrite() then? Is fprintf() perhaps doing something
stupid internally? I'd really like to avoid a lock here.
-Rasmus
Rob Richards wrote:
I can confirm that it does fail with the single fprintf call without the
lock on Win32.
Usingflock()does fix the problem in my tests.
I was able to fix win build usingflock()instead of php_flock() and
defining HAVE_FLOCK in win32/flock.hRob
Richard Quadling wrote:
This test is with the multiline fprintf and no lock.
Adding the php_flock() has stalled the build process for Win32 (I
can't build on Win32 as I don't know how!).Yes, but again, is this test with the single fprintf call? That's the
real fix for this problem, not the lock.-Rasmus
Richard Quadling wrote:
Using PHP 5.2.2-dev (cli) (built: Mar 23 2007 07:02:57) I can
replicate the problem on Windows.Using this single line command at the CMD prompt:
for %x in (A B C D E F G H I J K L M N O P Q R S T U V W X Y Z) do
start php -r"ini_set('error_log','/tmp/test.log');for($i=0;$i<100;$i++)error_log(str_repeat('%x',5000));"
The log file has many broken lines.
Rasmus,
Sorry for the delay in the reply. According to my tests on linux
using the sample script provided by the original bug reporter having
no lock causes a problem when the error message is >4k in length. In
this case multiple buffers are used and corruption can happen (it did
on a dual cpu machine with 10 error log writing threads running),
which is why I feel the lock is needed.Matt Wilmas wrote:
Hi,
Maybe just a Windows problem if it wasn't noticed yet, but I was
compiling
the latest 5.2 snapshot and got:main.obj : error LNK2019: unresolved external symbol _php_flock
referenced
in function _php_log_err
Release_TS\php5ts.dll : fatal error LNK1120: 1 unresolved
externalsCaused by this recent commit, http://news.php.net/php.cvs/43683,
and I
commented the php_flock line as a workaround. The Windows 5.2
snapshots
haven't been updated because of this either, of course.I see no reason for that lock at all as I commented when this was
committed, but Ilia never replied. This is a single write
operation now
since those fprintf's are now one, so that part of the fix is good,
but
the lock call is not needed since single writes in append mode are
atomic, even on Windows.So, your work around is fine and should actually be committed.
-Rasmus
--
Ilia Alshanetsky
No difference using sprintf()/fwrite() instead of fprintf().
I did come across a similar issue from apache:
3C9502272049.aa02036@paris.ics.uci.edu%3E" rel="nofollow" target="_blank">http://mail-archives.apache.org/mod_mbox/httpd-dev/199503.mbox/%3C9502272049.aa02036@paris.ics.uci.edu%3E
Changing to use VCWD_OPEN_MODE, write() and close() seems to work.
Rob
Rasmus Lerdorf wrote:
That still doesn't make sense to me. Could you try doing an
sprintf()
followed by anfwrite()then? Isfprintf()perhaps doing something
stupid internally? I'd really like to avoid a lock here.-Rasmus
Rob Richards wrote:
I can confirm that it does fail with the single fprintf call without the
lock on Win32.
Usingflock()does fix the problem in my tests.
I was able to fix win build usingflock()instead of php_flock() and
defining HAVE_FLOCK in win32/flock.hRob
Richard Quadling wrote:
This test is with the multiline fprintf and no lock.
Adding the php_flock() has stalled the build process for Win32 (I
can't build on Win32 as I don't know how!).Yes, but again, is this test with the single fprintf call? That's the
real fix for this problem, not the lock.-Rasmus
Richard Quadling wrote:
Using PHP 5.2.2-dev (cli) (built: Mar 23 2007 07:02:57) I can
replicate the problem on Windows.Using this single line command at the CMD prompt:
for %x in (A B C D E F G H I J K L M N O P Q R S T U V W X Y Z) do
start php -r"ini_set('error_log','/tmp/test.log');for($i=0;$i<100;$i++)error_log(str_repeat('%x',5000));"
The log file has many broken lines.
Rasmus,
Sorry for the delay in the reply. According to my tests on linux
using the sample script provided by the original bug reporter having
no lock causes a problem when the error message is >4k in length. In
this case multiple buffers are used and corruption can happen (it did
on a dual cpu machine with 10 error log writing threads running),
which is why I feel the lock is needed.Matt Wilmas wrote:
Hi,
Maybe just a Windows problem if it wasn't noticed yet, but I was
compiling
the latest 5.2 snapshot and got:main.obj : error LNK2019: unresolved external symbol _php_flock
referenced
in function _php_log_err
Release_TS\php5ts.dll : fatal error LNK1120: 1 unresolvedexternals
Caused by this recent commit, http://news.php.net/php.cvs/43683,
and I
commented the php_flock line as a workaround. The Windows 5.2
snapshots
haven't been updated because of this either, of course.I see no reason for that lock at all as I commented when this was
committed, but Ilia never replied. This is a single write
operation now
since those fprintf's are now one, so that part of the fix is good,
but
the lock call is not needed since single writes in append mode are
atomic, even on Windows.So, your work around is fine and should actually be committed.
-Rasmus
--
Ilia Alshanetsky
No difference using
sprintf()/fwrite() instead offprintf().I did come across a similar issue from apache:
3C9502272049.aa02036@paris.ics.uci.edu%3E" rel="nofollow" target="_blank">http://mail-archives.apache.org/mod_mbox/httpd-dev/199503.mbox/%3C9502272049.aa02036@paris.ics.uci.edu%3EChanging to use VCWD_OPEN_MODE, write() and close() seems to work.
Rob
Using PHP 5.2.2RC2-dev (cli) (built: Apr 13 2007 04:03:02) on Windows with
for %x in (A B C D E F G H I J K L M N O P Q R S T U V W X Y Z) do
start php -r "ini_set('error_log','/tmp/test.log');for($i=0;$i<1000;$i++)error_log(str_repeat('%x',5000));"`;
And then doing some analysis of the log file I'm finding that not all
the data has been written.
Array
(
[A] => 4965993
[B] => 4590918
[C] => 4525905
[D] => 4695939
[E] => 4495899
[F] => 4710942
[G] => 4495899
[H] => 4645929
[I] => 4540908
[J] => 4580916
[K] => 4535907
[L] => 4470894
[M] => 4480896
[N] => 4550910
[O] => 4610922
[P] => 4500900
[Q] => 4630926
[R] => 4480896
[S] => 4500900
[T] => 4535907
[U] => 4630926
[V] => 4470894
[W] => 4645929
[X] => 4825965
[Y] => 4845969
[Z] => 4920984
)
Lines (Should be 26 * 1000) : 23973
Longest (Should be 5024) : 5024
Shortest (Should be 5024) : 5024
Out of the 26,000 lines expected we are missing 2027 lines.
So no corruption of the lines, just missing ones.
The only explanation I can think of is that the locking is working,
but nothing is waiting for the lock to become available.
Richard.
--
Richard Quadling
Zend Certified Engineer : http://zend.com/zce.php?c=ZEND002498&r=213474731
"Standing on the shoulders of some very clever giants!"
The new implementation does not use any locks, instead it uses direct
io, where locks are not necessary for append operations.
No difference using
sprintf()/fwrite() instead offprintf().I did come across a similar issue from apache:
http://mail-archives.apache.org/mod_mbox/httpd-dev/199503.mbox/%
3C9502272049.aa02036@paris.ics.uci.edu%3EChanging to use VCWD_OPEN_MODE, write() and close() seems to work.
Rob
Using PHP 5.2.2RC2-dev (cli) (built: Apr 13 2007 04:03:02) on
Windows withfor %x in (A B C D E F G H I J K L M N O P Q R S T U V W X Y Z) do
start php -r "ini_set('error_log','/tmp/test.log');for($i=0;$i<1000;
$i++)error_log(str_repeat('%x',5000));"`;And then doing some analysis of the log file I'm finding that not all
the data has been written.Array
(
[A] => 4965993
[B] => 4590918
[C] => 4525905
[D] => 4695939
[E] => 4495899
[F] => 4710942
[G] => 4495899
[H] => 4645929
[I] => 4540908
[J] => 4580916
[K] => 4535907
[L] => 4470894
[M] => 4480896
[N] => 4550910
[O] => 4610922
[P] => 4500900
[Q] => 4630926
[R] => 4480896
[S] => 4500900
[T] => 4535907
[U] => 4630926
[V] => 4470894
[W] => 4645929
[X] => 4825965
[Y] => 4845969
[Z] => 4920984
)
Lines (Should be 26 * 1000) : 23973
Longest (Should be 5024) : 5024
Shortest (Should be 5024) : 5024Out of the 26,000 lines expected we are missing 2027 lines.
So no corruption of the lines, just missing ones.
The only explanation I can think of is that the locking is working,
but nothing is waiting for the lock to become available.Richard.
--
Richard Quadling
Zend Certified Engineer : http://zend.com/zce.php?
c=ZEND002498&r=213474731
"Standing on the shoulders of some very clever giants!"
Ilia Alshanetsky
So why are there missing entries?
I can even get the corruption back again if I use a shorter line (100
rather than 5000).
The new implementation does not use any locks, instead it uses direct
io, where locks are not necessary for append operations.No difference using
sprintf()/fwrite() instead offprintf().I did come across a similar issue from apache:
http://mail-archives.apache.org/mod_mbox/httpd-dev/199503.mbox/%
3C9502272049.aa02036@paris.ics.uci.edu%3EChanging to use VCWD_OPEN_MODE, write() and close() seems to work.
Rob
Using PHP 5.2.2RC2-dev (cli) (built: Apr 13 2007 04:03:02) on
Windows withfor %x in (A B C D E F G H I J K L M N O P Q R S T U V W X Y Z) do
start php -r "ini_set('error_log','/tmp/test.log');for($i=0;$i<1000;
$i++)error_log(str_repeat('%x',5000));"`;And then doing some analysis of the log file I'm finding that not all
the data has been written.Array
(
[A] => 4965993
[B] => 4590918
[C] => 4525905
[D] => 4695939
[E] => 4495899
[F] => 4710942
[G] => 4495899
[H] => 4645929
[I] => 4540908
[J] => 4580916
[K] => 4535907
[L] => 4470894
[M] => 4480896
[N] => 4550910
[O] => 4610922
[P] => 4500900
[Q] => 4630926
[R] => 4480896
[S] => 4500900
[T] => 4535907
[U] => 4630926
[V] => 4470894
[W] => 4645929
[X] => 4825965
[Y] => 4845969
[Z] => 4920984
)
Lines (Should be 26 * 1000) : 23973
Longest (Should be 5024) : 5024
Shortest (Should be 5024) : 5024Out of the 26,000 lines expected we are missing 2027 lines.
So no corruption of the lines, just missing ones.
The only explanation I can think of is that the locking is working,
but nothing is waiting for the lock to become available.Richard.
--
Richard Quadling
Zend Certified Engineer : http://zend.com/zce.php?
c=ZEND002498&r=213474731
"Standing on the shoulders of some very clever giants!"Ilia Alshanetsky
--
Richard Quadling
Zend Certified Engineer : http://zend.com/zce.php?c=ZEND002498&r=213474731
"Standing on the shoulders of some very clever giants!"
I can sometimes reproduce the missing entries though never get data
corruption anymore.
This might now be a Windows only issue with how it is caching writes. I
did a little debugging and no errors occurred.
The file was opened successfully every time, the data was written
(correct number of bytes as well) every single time, but depending upon
the load on my system at the time it was a crap shoot whether every
single write actually made it into the physical file or not. Adding a
lock here did resolve it so that it worked 100% of the time.
Rob
Richard Quadling wrote:
So why are there missing entries?
I can even get the corruption back again if I use a shorter line (100
rather than 5000).The new implementation does not use any locks, instead it uses direct
io, where locks are not necessary for append operations.No difference using
sprintf()/fwrite() instead offprintf().I did come across a similar issue from apache:
http://mail-archives.apache.org/mod_mbox/httpd-dev/199503.mbox/%
3C9502272049.aa02036@paris.ics.uci.edu%3EChanging to use VCWD_OPEN_MODE, write() and close() seems to work.
Rob
Using PHP 5.2.2RC2-dev (cli) (built: Apr 13 2007 04:03:02) on
Windows withfor %x in (A B C D E F G H I J K L M N O P Q R S T U V W X Y Z) do
start php -r "ini_set('error_log','/tmp/test.log');for($i=0;$i<1000;
$i++)error_log(str_repeat('%x',5000));"`;And then doing some analysis of the log file I'm finding that not all
the data has been written.Array
(
[A] => 4965993
[B] => 4590918
[C] => 4525905
[D] => 4695939
[E] => 4495899
[F] => 4710942
[G] => 4495899
[H] => 4645929
[I] => 4540908
[J] => 4580916
[K] => 4535907
[L] => 4470894
[M] => 4480896
[N] => 4550910
[O] => 4610922
[P] => 4500900
[Q] => 4630926
[R] => 4480896
[S] => 4500900
[T] => 4535907
[U] => 4630926
[V] => 4470894
[W] => 4645929
[X] => 4825965
[Y] => 4845969
[Z] => 4920984
)
Lines (Should be 26 * 1000) : 23973
Longest (Should be 5024) : 5024
Shortest (Should be 5024) : 5024Out of the 26,000 lines expected we are missing 2027 lines.
So no corruption of the lines, just missing ones.
The only explanation I can think of is that the locking is working,
but nothing is waiting for the lock to become available.Richard.
--
Richard Quadling
Zend Certified Engineer : http://zend.com/zce.php?
c=ZEND002498&r=213474731
"Standing on the shoulders of some very clever giants!"Ilia Alshanetsky
That's good news. Has a patch been submitted? Even if it is initially
only a win32 compiler directive wrapping the lock?
I can sometimes reproduce the missing entries though never get data
corruption anymore.
This might now be a Windows only issue with how it is caching writes. I
did a little debugging and no errors occurred.
The file was opened successfully every time, the data was written
(correct number of bytes as well) every single time, but depending upon
the load on my system at the time it was a crap shoot whether every
single write actually made it into the physical file or not. Adding a
lock here did resolve it so that it worked 100% of the time.Rob
Richard Quadling wrote:
So why are there missing entries?
I can even get the corruption back again if I use a shorter line (100
rather than 5000).The new implementation does not use any locks, instead it uses direct
io, where locks are not necessary for append operations.No difference using
sprintf()/fwrite() instead offprintf().I did come across a similar issue from apache:
http://mail-archives.apache.org/mod_mbox/httpd-dev/199503.mbox/%
3C9502272049.aa02036@paris.ics.uci.edu%3EChanging to use VCWD_OPEN_MODE, write() and close() seems to work.
Rob
Using PHP 5.2.2RC2-dev (cli) (built: Apr 13 2007 04:03:02) on
Windows withfor %x in (A B C D E F G H I J K L M N O P Q R S T U V W X Y Z) do
start php -r "ini_set('error_log','/tmp/test.log');for($i=0;$i<1000;
$i++)error_log(str_repeat('%x',5000));"`;And then doing some analysis of the log file I'm finding that not all
the data has been written.Array
(
[A] => 4965993
[B] => 4590918
[C] => 4525905
[D] => 4695939
[E] => 4495899
[F] => 4710942
[G] => 4495899
[H] => 4645929
[I] => 4540908
[J] => 4580916
[K] => 4535907
[L] => 4470894
[M] => 4480896
[N] => 4550910
[O] => 4610922
[P] => 4500900
[Q] => 4630926
[R] => 4480896
[S] => 4500900
[T] => 4535907
[U] => 4630926
[V] => 4470894
[W] => 4645929
[X] => 4825965
[Y] => 4845969
[Z] => 4920984
)
Lines (Should be 26 * 1000) : 23973
Longest (Should be 5024) : 5024
Shortest (Should be 5024) : 5024Out of the 26,000 lines expected we are missing 2027 lines.
So no corruption of the lines, just missing ones.
The only explanation I can think of is that the locking is working,
but nothing is waiting for the lock to become available.Richard.
--
Richard Quadling
Zend Certified Engineer : http://zend.com/zce.php?
c=ZEND002498&r=213474731
"Standing on the shoulders of some very clever giants!"Ilia Alshanetsky
--
Richard Quadling
Zend Certified Engineer : http://zend.com/zce.php?c=ZEND002498&r=213474731
"Standing on the shoulders of some very clever giants!"
On Windows, there is no such thing as a true open-write-for-append.
Unlike Unix, write for append mode is not atomic, internally it is a
seek + write. Mix that with multiple writers, and you have an inherent
race condition built it.
Richard Quadling wrote:
That's good news. Has a patch been submitted? Even if it is initially
only a win32 compiler directive wrapping the lock?I can sometimes reproduce the missing entries though never get data
corruption anymore.
This might now be a Windows only issue with how it is caching writes. I
did a little debugging and no errors occurred.
The file was opened successfully every time, the data was written
(correct number of bytes as well) every single time, but depending upon
the load on my system at the time it was a crap shoot whether every
single write actually made it into the physical file or not. Adding a
lock here did resolve it so that it worked 100% of the time.Rob
Richard Quadling wrote:
So why are there missing entries?
I can even get the corruption back again if I use a shorter line (100
rather than 5000).The new implementation does not use any locks, instead it uses direct
io, where locks are not necessary for append operations.No difference using
sprintf()/fwrite() instead offprintf().I did come across a similar issue from apache:
http://mail-archives.apache.org/mod_mbox/httpd-dev/199503.mbox/%
3C9502272049.aa02036@paris.ics.uci.edu%3EChanging to use VCWD_OPEN_MODE, write() and close() seems to work.
Rob
Using PHP 5.2.2RC2-dev (cli) (built: Apr 13 2007 04:03:02) on
Windows withfor %x in (A B C D E F G H I J K L M N O P Q R S T U V W X Y Z) do
start php -r "ini_set('error_log','/tmp/test.log');for($i=0;$i<1000;
$i++)error_log(str_repeat('%x',5000));"`;And then doing some analysis of the log file I'm finding that not
all
the data has been written.Array
(
[A] => 4965993
[B] => 4590918
[C] => 4525905
[D] => 4695939
[E] => 4495899
[F] => 4710942
[G] => 4495899
[H] => 4645929
[I] => 4540908
[J] => 4580916
[K] => 4535907
[L] => 4470894
[M] => 4480896
[N] => 4550910
[O] => 4610922
[P] => 4500900
[Q] => 4630926
[R] => 4480896
[S] => 4500900
[T] => 4535907
[U] => 4630926
[V] => 4470894
[W] => 4645929
[X] => 4825965
[Y] => 4845969
[Z] => 4920984
)
Lines (Should be 26 * 1000) : 23973
Longest (Should be 5024) : 5024
Shortest (Should be 5024) : 5024Out of the 26,000 lines expected we are missing 2027 lines.
So no corruption of the lines, just missing ones.
The only explanation I can think of is that the locking is working,
but nothing is waiting for the lock to become available.Richard.
--
Richard Quadling
Zend Certified Engineer : http://zend.com/zce.php?
c=ZEND002498&r=213474731
"Standing on the shoulders of some very clever giants!"Ilia Alshanetsky
So isn't locking the solution for Windows?
On Windows, there is no such thing as a true open-write-for-append.
Unlike Unix, write for append mode is not atomic, internally it is a
seek + write. Mix that with multiple writers, and you have an inherent
race condition built it.Richard Quadling wrote:
That's good news. Has a patch been submitted? Even if it is initially
only a win32 compiler directive wrapping the lock?I can sometimes reproduce the missing entries though never get data
corruption anymore.
This might now be a Windows only issue with how it is caching writes. I
did a little debugging and no errors occurred.
The file was opened successfully every time, the data was written
(correct number of bytes as well) every single time, but depending upon
the load on my system at the time it was a crap shoot whether every
single write actually made it into the physical file or not. Adding a
lock here did resolve it so that it worked 100% of the time.Rob
Richard Quadling wrote:
So why are there missing entries?
I can even get the corruption back again if I use a shorter line (100
rather than 5000).The new implementation does not use any locks, instead it uses direct
io, where locks are not necessary for append operations.No difference using
sprintf()/fwrite() instead offprintf().I did come across a similar issue from apache:
http://mail-archives.apache.org/mod_mbox/httpd-dev/199503.mbox/%
3C9502272049.aa02036@paris.ics.uci.edu%3EChanging to use VCWD_OPEN_MODE, write() and close() seems to work.
Rob
Using PHP 5.2.2RC2-dev (cli) (built: Apr 13 2007 04:03:02) on
Windows withfor %x in (A B C D E F G H I J K L M N O P Q R S T U V W X Y Z) do
start php -r "ini_set('error_log','/tmp/test.log');for($i=0;$i<1000;
$i++)error_log(str_repeat('%x',5000));"`;And then doing some analysis of the log file I'm finding that not
all
the data has been written.Array
(
[A] => 4965993
[B] => 4590918
[C] => 4525905
[D] => 4695939
[E] => 4495899
[F] => 4710942
[G] => 4495899
[H] => 4645929
[I] => 4540908
[J] => 4580916
[K] => 4535907
[L] => 4470894
[M] => 4480896
[N] => 4550910
[O] => 4610922
[P] => 4500900
[Q] => 4630926
[R] => 4480896
[S] => 4500900
[T] => 4535907
[U] => 4630926
[V] => 4470894
[W] => 4645929
[X] => 4825965
[Y] => 4845969
[Z] => 4920984
)
Lines (Should be 26 * 1000) : 23973
Longest (Should be 5024) : 5024
Shortest (Should be 5024) : 5024Out of the 26,000 lines expected we are missing 2027 lines.
So no corruption of the lines, just missing ones.
The only explanation I can think of is that the locking is working,
but nothing is waiting for the lock to become available.Richard.
--
Richard Quadling
Zend Certified Engineer : http://zend.com/zce.php?
c=ZEND002498&r=213474731
"Standing on the shoulders of some very clever giants!"Ilia Alshanetsky
--
Richard Quadling
Zend Certified Engineer : http://zend.com/zce.php?c=ZEND002498&r=213474731
"Standing on the shoulders of some very clever giants!"
Richard Quadling wrote:
So isn't locking the solution for Windows?
If this is single writer process, even with multithreads - a mutex
is most efficient, otherwise with concurrent writer processes, file
locking makes the most sense.
Bill
The testing I'm using is to launch multiple copies of the same PHP
script simultaneously using php.exe (CLI mode). This could easily be
multiple FastCGI scripts or multiple ISAPI threads.
Richard Quadling wrote:
So isn't locking the solution for Windows?
If this is single writer process, even with multithreads - a mutex
is most efficient, otherwise with concurrent writer processes, file
locking makes the most sense.Bill
--
Richard Quadling
Zend Certified Engineer : http://zend.com/zce.php?c=ZEND002498&r=213474731
"Standing on the shoulders of some very clever giants!"
Unless anyone sees a way to do this without adding the lock, I'd like to
commit this patch.
It adds a php_flock call only for windows. In doing so I got rid of the
win32/flock files and now compile with flock_compat from main.
Rob
Richard Quadling wrote:
The testing I'm using is to launch multiple copies of the same PHP
script simultaneously using php.exe (CLI mode). This could easily be
multiple FastCGI scripts or multiple ISAPI threads.Richard Quadling wrote:
So isn't locking the solution for Windows?
If this is single writer process, even with multithreads - a mutex
is most efficient, otherwise with concurrent writer processes, file
locking makes the most sense.Bill
This looks like it will be available in snaps.php.net in about 40 minutes.
Looking forward to it!
Unless anyone sees a way to do this without adding the lock, I'd like to
commit this patch.
It adds a php_flock call only for windows. In doing so I got rid of the
win32/flock files and now compile with flock_compat from main.Rob
Richard Quadling wrote:
The testing I'm using is to launch multiple copies of the same PHP
script simultaneously using php.exe (CLI mode). This could easily be
multiple FastCGI scripts or multiple ISAPI threads.Richard Quadling wrote:
So isn't locking the solution for Windows?
If this is single writer process, even with multithreads - a mutex
is most efficient, otherwise with concurrent writer processes, file
locking makes the most sense.Bill
Index: ext/standard/config.w32
RCS file: /repository/php-src/ext/standard/config.w32,v
retrieving revision 1.4.2.2
diff -u -r1.4.2.2 config.w32
--- ext/standard/config.w32 4 Jan 2006 21:31:29 -0000 1.4.2.2
+++ ext/standard/config.w32 17 Apr 2007 11:34:16 -0000
@@ -16,5 +16,5 @@
url_scanner_ex.c ftp_fopen_wrapper.c http_fopen_wrapper.c
php_fopen_wrapper.c credits.c css.c var_unserializer.c ftok.c sha1.c
user_filters.c uuencode.c filters.c proc_open.c \
streamsfuncs.c http.c", false /* never shared */);
streamsfuncs.c http.c flock_compat.c", false /* never shared */);Index: main/main.c
RCS file: /repository/php-src/main/main.c,v
retrieving revision 1.640.2.23.2.34
diff -u -r1.640.2.23.2.34 main.c
--- main/main.c 16 Apr 2007 08:09:56 -0000 1.640.2.23.2.34
+++ main/main.c 17 Apr 2007 11:49:49 -0000
@@ -63,6 +63,7 @@
#ifdef PHP_WIN32
#include <io.h>
#include "win32/php_registry.h"
+#include "ext/standard/flock_compat.h"
#endif
#include "php_syslog.h"
#include "Zend/zend_exceptions.h"
@@ -362,8 +363,11 @@
time(&error_time);
strftime(error_time_str, sizeof(error_time_str), "%d-%b-%Y %H:%M:%S", php_localtime_r(&error_time, &tmbuf));
len = spprintf(&tmp, 0, "[%s] %s%s", error_time_str, log_message, PHP_EOL);
+#ifdef PHP_WIN32
php_flock(fd, 2);+#endif
write(fd, tmp, len);
efree(tmp);
efree(tmp); close(fd); return; }Index: win32/build/config.w32
RCS file: /repository/php-src/win32/build/config.w32,v
retrieving revision 1.40.2.8.2.9
diff -u -r1.40.2.8.2.9 config.w32
--- win32/build/config.w32 16 Apr 2007 08:09:56 -0000 1.40.2.8.2.9
+++ win32/build/config.w32 17 Apr 2007 11:39:05 -0000
@@ -309,7 +309,7 @@
ADD_SOURCES("main/streams", "streams.c cast.c memory.c filter.c plain_wrapper.c
userspace.c transports.c xp_socket.c mmap.c");-ADD_SOURCES("win32", "crypt_win32.c flock.c glob.c md5crypt.c readdir.c
+ADD_SOURCES("win32", "crypt_win32.c glob.c md5crypt.c readdir.c
registry.c select.c sendmail.c time.c wfile.c winutil.c wsyslog.c globals.c");ADD_SOURCES("regex", "regcomp.c regerror.c regexec.c regfree.c");
--
Richard Quadling
Zend Certified Engineer : http://zend.com/zce.php?c=ZEND002498&r=213474731
"Standing on the shoulders of some very clever giants!"
And lo, the snapshot was "A Good Thing".
This looks like it will be available in snaps.php.net in about 40 minutes.
Looking forward to it!
Unless anyone sees a way to do this without adding the lock, I'd like to
commit this patch.
It adds a php_flock call only for windows. In doing so I got rid of the
win32/flock files and now compile with flock_compat from main.Rob
Richard Quadling wrote:
The testing I'm using is to launch multiple copies of the same PHP
script simultaneously using php.exe (CLI mode). This could easily be
multiple FastCGI scripts or multiple ISAPI threads.Richard Quadling wrote:
So isn't locking the solution for Windows?
If this is single writer process, even with multithreads - a mutex
is most efficient, otherwise with concurrent writer processes, file
locking makes the most sense.Bill
Index: ext/standard/config.w32
RCS file: /repository/php-src/ext/standard/config.w32,v
retrieving revision 1.4.2.2
diff -u -r1.4.2.2 config.w32
--- ext/standard/config.w32 4 Jan 2006 21:31:29 -0000 1.4.2.2
+++ ext/standard/config.w32 17 Apr 2007 11:34:16 -0000
@@ -16,5 +16,5 @@
url_scanner_ex.c ftp_fopen_wrapper.c http_fopen_wrapper.c
php_fopen_wrapper.c credits.c css.c var_unserializer.c ftok.c sha1.c
user_filters.c uuencode.c filters.c proc_open.c \
streamsfuncs.c http.c", false /* never shared */);
streamsfuncs.c http.c flock_compat.c", false /* never shared */);Index: main/main.c
RCS file: /repository/php-src/main/main.c,v
retrieving revision 1.640.2.23.2.34
diff -u -r1.640.2.23.2.34 main.c
--- main/main.c 16 Apr 2007 08:09:56 -0000 1.640.2.23.2.34
+++ main/main.c 17 Apr 2007 11:49:49 -0000
@@ -63,6 +63,7 @@
#ifdef PHP_WIN32
#include <io.h>
#include "win32/php_registry.h"
+#include "ext/standard/flock_compat.h"
#endif
#include "php_syslog.h"
#include "Zend/zend_exceptions.h"
@@ -362,8 +363,11 @@
time(&error_time);
strftime(error_time_str, sizeof(error_time_str), "%d-%b-%Y %H:%M:%S", php_localtime_r(&error_time, &tmbuf));
len = spprintf(&tmp, 0, "[%s] %s%s", error_time_str, log_message, PHP_EOL);
+#ifdef PHP_WIN32
php_flock(fd, 2);+#endif
write(fd, tmp, len);
efree(tmp);
efree(tmp); close(fd); return; }Index: win32/build/config.w32
RCS file: /repository/php-src/win32/build/config.w32,v
retrieving revision 1.40.2.8.2.9
diff -u -r1.40.2.8.2.9 config.w32
--- win32/build/config.w32 16 Apr 2007 08:09:56 -0000 1.40.2.8.2.9
+++ win32/build/config.w32 17 Apr 2007 11:39:05 -0000
@@ -309,7 +309,7 @@
ADD_SOURCES("main/streams", "streams.c cast.c memory.c filter.c plain_wrapper.c
userspace.c transports.c xp_socket.c mmap.c");-ADD_SOURCES("win32", "crypt_win32.c flock.c glob.c md5crypt.c readdir.c
+ADD_SOURCES("win32", "crypt_win32.c glob.c md5crypt.c readdir.c
registry.c select.c sendmail.c time.c wfile.c winutil.c wsyslog.c globals.c");ADD_SOURCES("regex", "regcomp.c regerror.c regexec.c regfree.c");
--
Richard Quadling
Zend Certified Engineer : http://zend.com/zce.php?c=ZEND002498&r=213474731
"Standing on the shoulders of some very clever giants!"
--
Richard Quadling
Zend Certified Engineer : http://zend.com/zce.php?c=ZEND002498&r=213474731
"Standing on the shoulders of some very clever giants!"
Ilia Alshanetsky wrote:
Rasmus,
Sorry for the delay in the reply. According to my tests on linux using
the sample script provided by the original bug reporter having no lock
causes a problem when the error message is >4k in length. In this case
multiple buffers are used and corruption can happen (it did on a dual
cpu machine with 10 error log writing threads running), which is why I
feel the lock is needed.
Did you do this test before or after you condensed the fprintf calls
into a single call? With multiple fprintf calls there would be a
problem, but with a single one I don't see how it could possibly screw
it up.
-Rasmus
Is it possible that at the lowest layer within the C library that 4K
blocking is performed?
Ilia Alshanetsky wrote:
Rasmus,
Sorry for the delay in the reply. According to my tests on linux using
the sample script provided by the original bug reporter having no lock
causes a problem when the error message is >4k in length. In this case
multiple buffers are used and corruption can happen (it did on a dual
cpu machine with 10 error log writing threads running), which is why I
feel the lock is needed.Did you do this test before or after you condensed the fprintf calls
into a single call? With multiple fprintf calls there would be a
problem, but with a single one I don't see how it could possibly screw
it up.-Rasmus
--
--
Richard Quadling
Zend Certified Engineer : http://zend.com/zce.php?c=ZEND002498&r=213474731
"Standing on the shoulders of some very clever giants!"