# [PATCH] Microsecond resolution and accuracy on Windows - Thread: https://externals.io/message/77014 - Original archive: https://news-web.php.net/php.internals/77014 - Started by Matt Wilmas on 2014-09-01T18:36:30+00:00 - Messages: 17 - Reply by email to internals@lists.php.net - More for agents: search with https://externals.io/search.md?q={query} · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt --- ## #77014 — Matt Wilmas — 2014-09-01T18:36:30+00:00 URL: https://externals.io/message/77014#77014 Hi all! I'm back after several years, and will have a few more changes for Windows, at least. (It was CVS back then, so I still have to figure some things out... Just had to edit files on Github site. :-/) Anyway, this patch is for microtime, etc. on Windows XP-7. In March 2013, the fix for bug #64370 reduced the resolution to 1/64 or 1/100 of a second or so -- hardly "micro" time. :*( See also bugs #64633 and #65626. The highest resolution time function available before Windows 8 has simply been used by itself since then. High-resolution Performance Counters were always used until then, but in a way that may become inaccurate and error-prone on some Windows+(virtual) hardware combinations. One problem with the long-standing previous Performance Counter implementation is that it tried to be monotonic (always increasing). But `gettimeofday()` IS supposed to be affected by changes in system time. Another problem was assuming that Performance Counters would track at a nearly-perfect rate with the real clock even without system time changes. The solution to have high-resolution like the past, and accuracy like now (after #64370 fix) is fairly simple and straightforward: Use Performance Counters, but check that the value is within "range" of the real, current time! With this implementation, you get high-resolution at least over short periods (e.g. when it matters) AND accuracy over a longer time between calls (e.g. won't notice few milliseconds of lost resolution), with no buggy/incorrect times. (It seems there *could* be a Counter synchronization issue between multiple cores/processors on *some* hardware on *XP*. I don't think this is a concern, nor related to previous bugs? http://msdn.microsoft.com/en-us/library/windows/desktop/dn553408(v=vs.85).aspx#qpc_support_in_windows_versions ) It's much more optimized than what's there now, and slightly over the old implementation. Not sure if I should give the saved patch link, or the "live compare" (?) on Github, so I'll do both for now: http://realplain.com/php/microtime_5_4.diff https://github.com/matt-moo/php-src/compare/PHP-5.4.diff Against 5.4 since that's what I quickly worked on so it's ready for the next 5.4 release (Stas?). (Although I guess we're supposed to change the oldest branch usually?) What do you think? Questions or comments? Thanks, Matt --- ## #77018 — Stas Malyshev — 2014-09-01T20:49:21+00:00 URL: https://externals.io/message/77014#77018 · In reply to: #77014 Hi! > It's much more optimized than what's there now, and slightly over the old > implementation. Not sure if I should give the saved patch link, or the > "live compare" (?) on Github, so I'll do both for now: > http://realplain.com/php/microtime_5_4.diff > https://github.com/matt-moo/php-src/compare/PHP-5.4.diff > > Against 5.4 since that's what I quickly worked on so it's ready for the next > 5.4 release (Stas?). (Although I guess we're supposed to change the oldest > branch usually?) Looking at the patch, it looks like unfortunately it changes a global structure (_php_win32_core_globals) which breaks binary compatibility. I think if you move the additional value to the end of the structure it should be ok though, since other offsets should not change then. I'm also not sure how important it is how have it for 5.4. Does the problem that this patch fixes exist only in older versions of Windows or on all versions? What are the actual effects of this problem - is it just lower resolution of microtime or there can be something seriously wrong with the whole result? If it's just lower resolution, I'd prefer this to go into 5.5 as the change is pretty extensive. -- Stanislav Malyshev, Software Architect SugarCRM: http://www.sugarcrm.com/ --- ## #77019 — Stas Malyshev — 2014-09-01T20:52:39+00:00 URL: https://externals.io/message/77014#77019 · In reply to: #77018 Hi! > I'm also not sure how important it is how have it for 5.4. Does the > problem that this patch fixes exist only in older versions of Windows or > on all versions? What are the actual effects of this problem - is it > just lower resolution of microtime or there can be something seriously > wrong with the whole result? If it's just lower resolution, I'd prefer > this to go into 5.5 as the change is pretty extensive. I'd like also to hear more from Anatol and other windows devs on how important they think this is - I haven't done anything with PHP on windows for some time so I'd like more feedback. -- Stanislav Malyshev, Software Architect SugarCRM: http://www.sugarcrm.com/ --- ## #77020 — Matt Wilmas — 2014-09-01T21:45:45+00:00 URL: https://externals.io/message/77014#77020 · In reply to: #77018 Hi Stas, ----- Original Message ----- From: "Stas Malyshev" Sent: Monday, September 01, 2014 > Hi! > > > […] > > Looking at the patch, it looks like unfortunately it changes a global > structure (_php_win32_core_globals) which breaks binary compatibility. I > think if you move the additional value to the end of the structure it > should be ok though, since other offsets should not change then. I was wondering about that myself (back when it was changed last, March 2013) when a few members were removed from that structure, so I figured it was OK to put one back. :-) No problem with binary compatibility then...? http://git.php.net/?p=php-src.git;a=commitdiff;h=b903d2d6cdf9a9efac181a21e95ea93dc8a864dd#patch5 > I'm also not sure how important it is how have it for 5.4. Does the > problem that this patch fixes exist only in older versions of Windows or > on all versions? What are the actual effects of this problem - is it > just lower resolution of microtime or there can be something seriously > wrong with the whole result? If it's just lower resolution, I'd prefer > this to go into 5.5 as the change is pretty extensive. I don't think the change is that extensive (maybe if you're comparing to current version ;-)). As I said in my first "5.4 - last call" reply, it's similar to the previous old version. You can see what was removed from time.c last March in the above diff... My changes are like a simplified version of that (there for over a decade), restored, but without the possibility of "seriously wrong results" (not an issue now either, but we've also had very low resolution). The issues, and these changes, don't affect Windows 8/Server 2012 and later (they have a high-res time function like *nix `gettimeofday()`). So Win 7 and before. See referenced bugs: #64633, #65626 (uniqid()) Thanks, Matt --- ## #77021 — Matt Wilmas — 2014-09-01T22:00:44+00:00 URL: https://externals.io/message/77014#77021 · In reply to: #77019 Hi Stas, ----- Original Message ----- From: "Stas Malyshev" Sent: Monday, September 01, 2014 > Hi! > > > […] > > I'd like also to hear more from Anatol and other windows devs on how > important they think this is - I haven't done anything with PHP on > windows for some time so I'd like more feedback. From comments in bug reports and brief exchanges on Twitter in October, Pierre wanted to go back to using high-res Performance Counters like the previous 10+ years. While Anatol explains the potential issues, and the low-res, but accurate, plain time functions should be used. My changes provide the best of both, AFAICT. Didn't get to correct 5.3 somehow last year... :*( - Matt --- ## #77022 — Stas Malyshev — 2014-09-01T22:06:11+00:00 URL: https://externals.io/message/77014#77022 · In reply to: #77020 Hi! > I was wondering about that myself (back when it was changed last, March > 2013) when a few members were removed from that structure, so I figured it > was OK to put one back. :-) No problem with binary compatibility then...? > > http://git.php.net/?p=php-src.git;a=commitdiff;h=b903d2d6cdf9a9efac181a21e95ea93dc8a864dd#patch5 I didn't review that patch but if it was made in a stable version it's not a good thing too. Anatol, any comments on that? -- Stanislav Malyshev, Software Architect SugarCRM: http://www.sugarcrm.com/ --- ## #77029 — Anatol Belski — 2014-09-02T08:03:08+00:00 URL: https://externals.io/message/77014#77029 · In reply to: #77014 Hi Matt, > Hi all! > > > I'm back after several years, and will have a few more changes for > Windows, > at least. (It was CVS back then, so I still have to figure some things > out... Just had to edit files on Github site. :-/) > > Anyway, this patch is for microtime, etc. on Windows XP-7. In March > 2013, > > > the fix for bug #64370 reduced the resolution to 1/64 or 1/100 of a > second > > or so -- hardly "micro" time. :*( See also bugs #64633 and #65626. The > highest resolution time function available before Windows 8 has simply > been used by itself since then. High-resolution Performance Counters were > always used until then, but in a way that may become inaccurate and > error-prone on some Windows+(virtual) hardware combinations. > > One problem with the long-standing previous Performance Counter > implementation is that it tried to be monotonic (always increasing). But > `gettimeofday()` IS supposed to be affected by changes in system time. > Another problem was assuming that Performance Counters would track at a > nearly-perfect rate with the real clock even without system time changes. > > The solution to have high-resolution like the past, and accuracy like now > (after #64370 fix) is fairly simple and straightforward: Use Performance > Counters, but check that the value is within "range" of the real, > current time! > > With this implementation, you get high-resolution at least over short > periods (e.g. when it matters) AND accuracy over a longer time between > calls (e.g. won't notice few milliseconds of lost resolution), with no > buggy/incorrect times. (It seems there *could* be a Counter > synchronization issue between multiple cores/processors on *some* hardware > on *XP*. I don't think this is a concern, nor related to previous bugs? > http://msdn.microsoft.com/en-us/library/windows/desktop/dn553408(v=vs.85) > . > aspx#qpc_support_in_windows_versions ) > > It's much more optimized than what's there now, and slightly over the old > implementation. Not sure if I should give the saved patch link, or the > "live compare" (?) on Github, so I'll do both for now: > http://realplain.com/php/microtime_5_4.diff > https://github.com/matt-moo/php-src/compare/PHP-5.4.diff > > > Against 5.4 since that's what I quickly worked on so it's ready for the > next 5.4 release (Stas?). (Although I guess we're supposed to change the > oldest > > branch usually?) > > What do you think? Questions or comments? > > unfortunately the patch doesn't fix the both issues, a quick test shows just a return to the old behaviour: >php -n -r "$t0 = microtime(true); do { $t1 = microtime(true); echo $t1 - $t0, PHP_EOL; } while($t1 >= $t0 && ($t0 = $t1));" .......................................... 4.7922134399414E-5 4.6968460083008E-5 4.7922134399414E-5 4.8160552978516E-5 4.7922134399414E-5 4.7922134399414E-5 4.8160552978516E-5 4.887580871582E-5 0.00020909309387207 -0.015614986419678 I took an arbitrary win7 VM for that. The error happens after a couple of seconds of running. That's just before starting to argue. We probably would get WTFs from the users which had this before and had that fixed. IMHO, it's still better not to have $t1 < $t0 even with worse accuracy. I still think that this issue is unsolvable within PHP. Fixing this for one group of users will break it for another. There are too many factors affecting this. Besides the hardware we cannot affect, there are programs affecting the behaviour (for example some could use SetSystemTimeAdjustment()). And that is IMO the main mistake with this approach (and the one before removing QPC) - it's that QPC is based on an independent source, while GetSystemTimeAdjustment and similar time functions are synchronizable to an external source. Haven't tried, but just asking - would it work correct on day light saving days? QPC doc also states it is dedicated for time intervals measurements. The user would think that he gets a real correct microtime timestamp, while in reality it's a "computed" one, but the equation used for the computation is far incomplete. Inbetween i was developing pecl.php.net/hrtime which uses APIs similar to QPC on linux. While testing it even on linux we have seen a shift between gettimeofday and monotonic clocks. Well, there one needs much longer time to reproduce and in reality one doesn't know which of those APIs exactly has a bug. But just as an example. At the end - I would not include this patch that fast especially shortly before the PHP-5.4 doors are closing afterwards. While it were of course amazing to have a fix, the patch doesn't seem to fix the bad accuracy but brings the old behavior back. I have also a couple of notes to the code, i'd leave my comments on the patch repo. Matt, I have anyway to thank you for the effort and energy you spent on this. Regards Anatol --- ## #77030 — Anatol Belski — 2014-09-02T09:20:13+00:00 URL: https://externals.io/message/77014#77030 · In reply to: #77018 > Hi! > > > > […] > > Looking at the patch, it looks like unfortunately it changes a global > structure (_php_win32_core_globals) which breaks binary compatibility. I > think if you move the additional value to the end of the structure it > should be ok though, since other offsets should not change then. > AFAIR the patch with ABI break was at the time where 5.5 was alpha, so the change was backported. Was not intentional to break ABI and just based on the good user feedback, and good tested anyway. Still used and shows no issue in the phpt and app unit tests. > I'm also not sure how important it is how have it for 5.4. Does the > problem that this patch fixes exist only in older versions of Windows or on > all versions? What are the actual effects of this problem - is it just > lower resolution of microtime or there can be something seriously wrong > with the whole result? If it's just lower resolution, I'd prefer this to > go into 5.5 as the change is pretty extensive. -- That's the lower resolution on systems preceding win8. The previous issue was time_before > time_after which was caused by the same approach Matt has now in the patch. Also I've developed pecl.php.net/hrtime since then which provides the exact solution - a stopwatch for inteval measurements, no synchronized timestamps. The microtime in the core delivers the real timestamp, not a computed one. But well, with an accuracy which one could wish to be better. Currently, something like this can never be true $time0 = microtime(true); time1 = microtime(true); var_dump($time1 - $time0 < 0); For the practical case, i'm not sure if with the Matt's patch one will be able to make an ebay bid in the right time (assumed it's expected to have microsecond precision). However with a lower accuracy there might be an issue, too. Phony buggy timestamp vs. worse accuracy. Please consider also the other issues like hardware/software i mentioned in the other mail. Regards Anatol --- ## #77034 — Anatol Belski — 2014-09-02T11:43:55+00:00 URL: https://externals.io/message/77014#77034 · In reply to: #77020 > Hi Stas, > > > ----- Original Message ----- > From: "Stas Malyshev" > Sent: Monday, September 01, 2014 > > > > […] > old > > […] > > > […] > > I was wondering about that myself (back when it was changed last, March > 2013) when a few members were removed from that structure, so I figured it > > > was OK to put one back. :-) No problem with binary compatibility > then...? > > http://git.php.net/?p=php-src.git;a=commitdiff;h=b903d2d6cdf9a9efac181a21 > e 95ea93dc8a864dd#patch5 > > > > […] > > I don't think the change is that extensive (maybe if you're comparing to > current version ;-)). As I said in my first "5.4 - last call" reply, it's > > > similar to the previous old version. You can see what was removed from > time.c last March in the above diff... My changes are like a simplified > version of that (there for over a decade), restored, but without the > possibility of "seriously wrong results" (not an issue now either, but > we've also had very low resolution). > > The issues, and these changes, don't affect Windows 8/Server 2012 and > later (they have a high-res time function like *nix `gettimeofday()`). So > Win 7 > and before. See referenced bugs: #64633, #65626 (uniqid()) > > Unfortunately that's not a PR so I cannot comment there directly, so I'd leave a couple of the comments to the code here: - GetSystemTimeAdjustment can be disabled by some other app, so it should be called each time when `gettimeofday()` is called - GetSystemTimeAdjustment could fail, so return value should have be checked - global variables should be moved into the globals struct Matt, I really think it's too short in time to take this in. I would suggest to leave this as is for 5.4 and 5.5 and stabilize to suggest to 5.6.1 or master after the good testing. It were probably applicable to win7 then. Thanks Anatol --- ## #77075 — Matt Wilmas — 2014-09-05T17:20:20+00:00 URL: https://externals.io/message/77014#77075 · In reply to: #77029 Hi Anatol! ----- Original Message ----- From: "Anatol Belski" Sent: Tuesday, September 02, 2014 > Hi Matt, > > > > […] > unfortunately the patch doesn't fix the both issues, a quick test shows > just a return to the old behaviour: > > > […] > $t0, PHP_EOL; } while($t1 >= $t0 && ($t0 = $t1));" > .......................................... > 4.7922134399414E-5 > 4.6968460083008E-5 > 4.7922134399414E-5 > 4.8160552978516E-5 > 4.7922134399414E-5 > 4.7922134399414E-5 > 4.8160552978516E-5 > 4.887580871582E-5 > 0.00020909309387207 > -0.015614986419678 > > I took an arbitrary win7 VM for that. The error happens after a couple of > seconds of running. That's just before starting to argue. We probably > would get WTFs from the users which had this before and had that fixed. > IMHO, it's still better not to have $t1 < $t0 even with worse accuracy. Thanks for all the feedback! I was just waiting to get something "solid" before replying. :-) I was like, "Well, hell," when I saw this first reply. :*( BUT, I think it's fixed now! Patch updated (at links above for anyone). I just quickly implemented what I had in my mind for a while Sat. morning (half the time was trying to consider what would happen in different scenarios), so I didn't spend much time on it at all. :-) Brief checking after, but yep, I was able to reproduce your result (XP, real system) after a few minutes (15-20 max) and realized what was happening. I thought about and tried a few things in the following couple days, before settling (more "obviously" now) on what's there now ("computed" out-of-valid-range logic was too simple before, oops). Maybe THIS is actually the correct logic I had in mind last October! I haven't seen any problem with it after hours and hours (which seems right). The only time there could be t1 < t0 is if the clock is set backwards (which obviously happens now with the current low-res version). > I still think that this issue is unsolvable within PHP. Fixing this for > one group of users will break it for another. There are too many factors > affecting this. Besides the hardware we cannot affect, there are programs > affecting the behaviour (for example some could use > SetSystemTimeAdjustment()). See if you can break/find a problem with it now, please. :-) > And that is IMO the main mistake with this approach (and the one before > removing QPC) - it's that QPC is based on an independent source, while > GetSystemTimeAdjustment and similar time functions are synchronizable to > an external source. Haven't tried, but just asking - would it work correct > on day light saving days? QPC doc also states it is dedicated for time > intervals measurements. The user would think that he gets a real correct > microtime timestamp, while in reality it's a "computed" one, but the > equation used for the computation is far incomplete. By "it," you mean my/any QPC implementation with DST changes? Yes, GetSystemTime... is like `time()` I assume and doesn't change with DST -- otherwise PHP's date stuff would be affected. And my version of course always respects the currently-reported time, anyway, unlike the OLD version. > Inbetween i was developing pecl.php.net/hrtime which uses APIs similar to > QPC on linux. While testing it even on linux we have seen a shift between > gettimeofday and monotonic clocks. Well, there one needs much longer time > to reproduce and in reality one doesn't know which of those APIs exactly > has a bug. But just as an example. Yeah, I saw about your hrtime package in the one bug comment (interesting), but haven't looked at it yet... > At the end - I would not include this patch that fast especially shortly > before the PHP-5.4 doors are closing afterwards. While it were of course > amazing to have a fix, the patch doesn't seem to fix the bad accuracy but > brings the old behavior back. I have also a couple of notes to the code, > i'd leave my comments on the patch repo. Matt, I have anyway to thank you > for the effort and energy you spent on this. Still hoping for 5.4 for at least the "security" related view with `uniqid()`. :-) BTW, I also removed the useless modulo from `microtime()`. (The GitHub patch is breaking the uniqid.c header... ignore.) > Regards > > Anatol Thanks again, Matt --- ## #77076 — Stas Malyshev — 2014-09-05T17:28:30+00:00 URL: https://externals.io/message/77014#77076 · In reply to: #77075 Hi! > Still hoping for 5.4 for at least the "security" related view with `uniqid()`. > :-) > > BTW, I also removed the useless modulo from `microtime()`. (The GitHub patch > is breaking the uniqid.c header... ignore.) Let's ensure it works fine first, when we're sure it's good for 5.5 we can see what are the implications for 5.4 :) -- Stanislav Malyshev, Software Architect SugarCRM: http://www.sugarcrm.com/ --- ## #77077 — Matt Wilmas — 2014-09-05T17:40:20+00:00 URL: https://externals.io/message/77014#77077 · In reply to: #77030 Hi Anatol, Stas, ----- Original Message ----- From: "Anatol Belski" Sent: Tuesday, September 02, 2014 > > > […] > AFAIR the patch with ABI break was at the time where 5.5 was alpha, so the > change was backported. Was not intentional to break ABI and just based on > the good user feedback, and good tested anyway. Still used and shows no > issue in the phpt and app unit tests. OK before, so should be OK now, right? I just figured the Windows globals were "special"/not important and could be changed after seeing that... Is anyone/anything actually using _php_win32_core_globals besides the core? Nothing interesting there...? > > […] > That's the lower resolution on systems preceding win8. The previous issue > was time_before > time_after which was caused by the same approach Matt > has now in the patch. Hopefully fixed now! > Also I've developed pecl.php.net/hrtime since then which provides the > exact solution - a stopwatch for inteval measurements, no synchronized > timestamps. The microtime in the core delivers the real timestamp, not a > computed one. But well, with an accuracy which one could wish to be > better. Currently, something like this can never be true > > $time0 = microtime(true); > time1 = microtime(true); > > var_dump($time1 - $time0 < 0); Of course it *can* be true, currently. IF the system time is set backwards! Unlike time itself, computer clocks aren't monotonic. > For the practical case, i'm not sure if with the Matt's patch one will be > able to make an ebay bid in the right time (assumed it's expected to have > microsecond precision). However with a lower accuracy there might be an > issue, too. Phony buggy timestamp vs. worse accuracy. Please consider also > the other issues like hardware/software i mentioned in the other mail. eBay bid?!? :-O You'd have bigger problems than timestamps with that. Like waiting for Windows' timers, before the unknowns like network latency. :-) > Regards > > Anatol - Matt --- ## #77079 — Matt Wilmas — 2014-09-05T18:05:40+00:00 URL: https://externals.io/message/77014#77079 · In reply to: #77034 Hi Anatol, ----- Original Message ----- From: "Anatol Belski" Sent: Tuesday, September 02, 2014 > Unfortunately that's not a PR so I cannot comment there directly, so I'd > leave a couple of the comments to the code here: It looks like there can be "commit comments?" *shrug* In the future, should a Pull Request usually be created? I didn't know if that sends some sort of "Hey, add this!" notification, or simply makes it show up as a PR. > - GetSystemTimeAdjustment can be disabled by some other app, so it should > be called each time when `gettimeofday()` is called I don't know what you mean by "disabled." You mean the adjustment changing? I sort of already alluded to not thinking it's necessary to check it each time `gettimeofday()` is called -- I was already thinking of a much more complicated method last year involving that stuff, before finding this spring that Vista/7 aren't very finely adjustable like XP regarding *setting* SystemTimeAdjustment... > - GetSystemTimeAdjustment could fail, so return value should have be > checked I know it could fail (though not sure why), technically, but I don't know a reason for checking. time_update_rate is already initialized in time.c. > - global variables should be moved into the globals struct Why? Just because? I thought it's fine to use true globals when it's safe to (I've seen it in another place), like function pointers. No threading issues when they're only modified during startup (1 thread). > Matt, I really think it's too short in time to take this in. I would > suggest to leave this as is for 5.4 and 5.5 and stabilize to suggest to > 5.6.1 or master after the good testing. It were probably applicable to > win7 then. I was glad to see Stas' reply to "let's see for 5.4" to, well, see if there's any other problems found. :-) There's that `uniqid()` bug/issue like I said. 5.4 is also the last "official" release (with PGO) for Windows XP. > Thanks > > Anatol Thanks again for feedback, Matt --- ## #77114 — Anatol Belski — 2014-09-09T17:59:50+00:00 URL: https://externals.io/message/77014#77114 · In reply to: #77079 Hi Matt, > Hi Anatol, > > > ----- Original Message ----- > From: "Anatol Belski" > Sent: Tuesday, September 02, 2014 > > > > […] > > It looks like there can be "commit comments?" *shrug* In the future, > should a Pull Request usually be created? I didn't know if that sends > some sort of "Hey, add this!" notification, or simply makes it show up as > a PR. > > > […] > > I don't know what you mean by "disabled." You mean the adjustment > changing? I sort of already alluded to not thinking it's necessary to > check it each time `gettimeofday()` is called -- I was already thinking of a > much more complicated method last year involving that stuff, before > finding this spring that Vista/7 aren't very finely adjustable like XP > regarding *setting* SystemTimeAdjustment... > > > > […] > > I know it could fail (though not sure why), technically, but I don't know > a reason for checking. time_update_rate is already initialized in time.c. > > > > […] > > Why? Just because? I thought it's fine to use true globals when it's > safe to (I've seen it in another place), like function pointers. No > threading issues when they're only modified during startup (1 thread). > > > […] > > I was glad to see Stas' reply to "let's see for 5.4" to, well, see if > there's any other problems found. :-) There's that `uniqid()` bug/issue > like I said. > > > 5.4 is also the last "official" release (with PGO) for Windows XP. > coming to this right now. I'm just responding here as otherwise we'd have three parallel threads to follow, not handy. I've tested your new variant and it doesn't show $t1 < $t0 on the same VM. However I still see an issue. The main issue that it tries to solve a hardware bug with math, that can work or not. Furthermore, looking closer at the math in the patch, I don't think it's correct. Obviously we neither should care about winxp at this times, nor about vista. Regarding win7, there are still too many issues to have a stable synchronizable timing in high resolution. You can agree or not, here are just the facts: - time-of-day clock accuracy depends on tick frequency (RTC interrupt) and/or additionally on HPET - QPC can be based on HPET, RDTSC, ACPI, etc. - the interrupt frequency depends on HW used - QPC is not synchronizable to an external source, while time-of-day clock is - QPC and partly time-of-day clock additionally depends whether it's a real HW or a VM, number of processors, thread safety, function call latency, etc. - while retrieving QPC is cheap, time-of-day clock will lead to loosing some microseconds - there are also some new HW which provides multimedia timing chips For more I'd just mention the site you already was linking to (did read that also a lot before), and especially to this doc http://www.windowstimestamp.com/PartIIAdjustmentofSystemTime.pdf starting at 4. As well as the MSDN pages for the corresponding functions. The consequence of all this is that it's completely unpredictable to mix QPC and time-of-day clock. There's no guarantee that time update interval will match the interrupt interval. Back to the math in the patch, lpTimeIncrement from GetSystemTimeAdjustment() will be constant, but lpTimeAdjustment can change. You currently don't take it in account. lpTimeAdjustment can be changed by a SetSystemTimeAdjustment() call, from an arbitrary source like some program or NTP update. Time adjustment can be disabled (and effectively that's not handled atm). That means from the patch ts = target - time_update_rate doesn't bring you to the time at the previous update interval. Say when lpTimeAdjustment == lpTimeIncrement, there were no gain at all (same as if time adjustment were disabled). That's why I meant GetSystemTimeAdjustment() should be called as frequently as `gettimeofday()`. From http://msdn.microsoft.com/en-us/library/windows/desktop/ms724394%28v=vs.85%29.aspx I read the following: ============= For each lpTimeIncrement period of time that actually passes, lpTimeAdjustment will be added to the time of day. If the lpTimeAdjustment value is smaller than lpTimeIncrement, the system time-of-day clock will advance at a rate slower than normal. If the lpTimeAdjustment value is larger than lpTimeIncrement, the time-of-day clock will advance at a rate faster than normal. If lpTimeAdjustment equals lpTimeIncrement, the time-of-day clock will advance at its normal speed. ============= The conclusion - while QPC is constant (or actually should be), the time-of-day clock might shift at an arbitrary rate. Taking in account, that the update interval can most likely be not the same (for instance RDTS vs PM clock), the result of the math is arbitrary. For more, GetSystemTimeAsFileTime() and GetSystemTimeAdjustment() will surely take several microseconds to complete, which add to the uncertainty. So even when used, they should be feched right after QPC. At the end, we have "some" value which passes into the range between the current QPC and the time-of-day clock values, but it's obviously not the value one could call even approximately correct. The uncertainty might be low enough to fit into the max resolution (say 300us were the max resolution and 30ms the uncertainty), however that's just a hope. For an application which really matters about the microsecond resolution it'll be not suitable. The timestamp will be just phony. That's what i meant with the example of the ebay bid, apart from net latency of course. But ebay uses the millisecond resolution anyway, like actually almost any real world app nowadays, still. It's more like example when you have PHP with this patch and try to communicate with another machine which is synchronizing time say per NTP and as in that case exact timing would matter, how much discrepancy will it have? After one month of running an FCGI process? And so on. IMHO how it looks with pre win8 - the time-of-day API is not suitable for performance measurements. I actually always presumed that it was your goal. For this goal the hrtime stuff can be used where you even can measure with pure QPC ticks when on a fast machine (say when you're out of even ns range). As if we cannot provide a valid synchronizable timestamp with QPC, the qualitative performance measurement is easy done using relative time intervals. May I suggest you to pack your work into a PECL extension, or maybe even merge with hrtime. As there are several aspects anyway where we have an issue, more or less. For example `uniqid()` you've mentioned. Or actually, `usleep()` IMHO - much more of issue than microtime currently as the resolution used there can be much worse. Anyway, it can come back to core after it was good tested, is well thought and solid. That's what I would suggest to do. Best regards Anatol --- ## #77141 — Matt Wilmas — 2014-09-10T20:46:48+00:00 URL: https://externals.io/message/77014#77141 · In reply to: #77114 Hi Anatol, all, Sorry for top post... Stas must be getting confused with all these details! :-) Is Pierre going to weigh in? Are there any other Windows people? While I don't think I'd disagree with much of what you've said, I don't see that using my patch is *causing* or *creating* new issues that you're referring to. Whatever is broken with Windows is still the *same* with what's there now since last year. All I wanted to do, that I think Windows users want, is make `microtime()` at least resemble *micro* time (very well, resolution-wise) so that it *is* suitable for performance measurements. It's useless for that now. And it happens to fix the `uniqid()` problem to boot. I don't think there were [m]any issues over 10+ years using `microtime()` for that? It was that implementation's errors over time (which mine doesn't have) that were revealed when compared to another value, like REQUEST_TIME[_FLOAT]. Maybe I used the wrong subject wording or NEWS file update -- I'm not trying to improve actual "accuracy" over what's available now -- just insure that it stays accurate as it can be (within range) with respect to the current time, unlike the old implementation. So things are MUCH better for `microtime()` useable resolution and `uniqid()`. What are some actual usage scenarios that my implementation would make WORSE than they are now? I don't see how... It seems you're against the idea, in principle, of it working at all like the old way (in a good way)? :-/ The patch doesn't make all these new problems like you are suggesting. More below... ----- Original Message ----- From: "Anatol Belski" Sent: Tuesday, September 09, 2014 > Hi Matt, > > [...] > > coming to this right now. I'm just responding here as otherwise we'd have > three parallel threads to follow, not handy. > > I've tested your new variant and it doesn't show $t1 < $t0 on the same VM. > However I still see an issue. The main issue that it tries to solve a > hardware bug with math, that can work or not. I'm not trying to solve a hardware bug at all with math. The idea is only to indicate the *certain* passage of time between calls while the returned time, otherwise, remains the same (as one example). > Furthermore, looking closer > at the math in the patch, I don't think it's correct. Obviously we neither > should care about winxp at this times, nor about vista. Regarding win7, > there are still too many issues to have a stable synchronizable timing in > high resolution. You can agree or not, here are just the facts: You are talking about "synchronizable," etc. but I'm not trying to do anything like that. Just a time that usually moves forward (except window of x amount of time it could stay the same if clock is advancing *very* slowly (QPC too far ahead and "clamped" to max); still better than now). > - time-of-day clock accuracy depends on tick frequency (RTC interrupt) > and/or additionally on HPET > - QPC can be based on HPET, RDTSC, ACPI, etc. > - the interrupt frequency depends on HW used And on timer resolution, right? AFAIK the default timer resolution (lowest) is always equal to the time_update_rate (lpTimeIncrement). Again, not that this matters much anyway, since currently results are same or worse than my implementation. Mine doesn't make anything worse. > - QPC is not synchronizable to an external source, while time-of-day clock > is See above again. :-) Only moving the micro time forward when it doesn't otherwise change. > - QPC and partly time-of-day clock additionally depends whether it's a > real HW or a VM, number of processors, thread safety, function call > latency, etc. > - while retrieving QPC is cheap, time-of-day clock will lead to loosing > some microseconds I was actually wondering about the QPC latency (but not an issue raised before) on diff systems... I don't see why there'd be a delay getting system time. Regardless, I can call `microtime()` on my system (older Q9400) 2 million times/sec. (inc. loop overhead), so not losing microseconds when half the calls return the same value. :-) > - there are also some new HW which provides multimedia timing chips > > For more I'd just mention the site you already was linking to (did read > that also a lot before), and especially to this doc > http://www.windowstimestamp.com/PartIIAdjustmentofSystemTime.pdf starting > at 4. As well as the MSDN pages for the corresponding functions. > > The consequence of all this is that it's completely unpredictable to mix > QPC and time-of-day clock. There's no guarantee that time update interval > will match the interrupt interval. Back to the math in the patch, > lpTimeIncrement from GetSystemTimeAdjustment() will be constant, but > lpTimeAdjustment can change. You currently don't take it in account. > lpTimeAdjustment can be changed by a SetSystemTimeAdjustment() call, from > an arbitrary source like some program or NTP update. Time adjustment can > be disabled (and effectively that's not handled atm). That means from the > patch I already said how I'm purposely not taking any SetSystemTimeAdjustment() calls into account (besides the fact that it's probably never used). Complicates things much more than is needed to simply estimate time passage between actual system time updates. The clock changing +/- 1 sec/hour (that's probably getting extreme as far as errors) is less than 5 microseconds per 1/64 second update interval. Doesn't matter when it's less than any fluctuation anyway. Certainly not an issue compared to now! > ts = target - time_update_rate > > doesn't bring you to the time at the previous update interval. If the estimate is behind (either over time or because clock jumped forward), that simply brings it to the minimum amount within range (so it "jumps" as little as acceptable). The only way that could *not* be at/within the previous update interval is if the clock was updating *slower* than normal (TimeAdjustment < TimeIncrement) in which case the estimate would be *ahead*, not behind. > Say when > lpTimeAdjustment == lpTimeIncrement, there were no gain at all (same as if > time adjustment were disabled). There's always some gain -- can't stop the clock. :-) But if could, still no worse than now! > That's why I meant > GetSystemTimeAdjustment() should be called as frequently as > `gettimeofday()`. From > http://msdn.microsoft.com/en-us/library/windows/desktop/ms724394%28v=vs.85%29.aspx > I read the following: > ============= > For each lpTimeIncrement period of time that actually passes, > lpTimeAdjustment will be added to the time of day. If the lpTimeAdjustment > value is smaller than lpTimeIncrement, the system time-of-day clock will > advance at a rate slower than normal. If the lpTimeAdjustment value is > larger than lpTimeIncrement, the time-of-day clock will advance at a rate > faster than normal. If lpTimeAdjustment equals lpTimeIncrement, the > time-of-day clock will advance at its normal speed. > ============= Yes, I know all that... > The conclusion - while QPC is constant (or actually should be), the > time-of-day clock might shift at an arbitrary rate. Taking in account, > that the update interval can most likely be not the same (for instance > RDTS vs PM clock), the result of the math is arbitrary. For more, > GetSystemTimeAsFileTime() and GetSystemTimeAdjustment() will surely take > several microseconds to complete, which add to the uncertainty. So even > when used, they should be feched right after QPC. At the end, we have > "some" value which passes into the range between the current QPC and the > time-of-day clock values, but it's obviously not the value one could call > even approximately correct. The uncertainty might be low enough to fit > into the max resolution (say 300us were the max resolution and 30ms the > uncertainty), however that's just a hope. > > For an application which really matters about the microsecond resolution > it'll be not suitable. The timestamp will be just phony. That's what i > meant with the example of the ebay bid, apart from net latency of course. > But ebay uses the millisecond resolution anyway, like actually almost any > real world app nowadays, still. It's more like example when you have PHP > with this patch and try to communicate with another machine which is > synchronizing time say per NTP and as in that case exact timing would > matter, how much discrepancy will it have? After one month of running an > FCGI process? And so on. Discrepancy? No more than now. After running for any amount of time? No more than now. It would be very hard to synchronize a Windows system's time anyway (I guess on windowstimestamp.com it's trying to help with that, but who's using that?), so the bidding resolution stuff is almost moot anyway -- problem is same now, my patch doesn't hurt anything like I'm saying. You can't do any synchronized communication now either anyway. How would you? Even IF a clock was truely synchronized, there's no way in PHP (or anything) to wait for/until a certain time to the millisecond... But of course this is all theory stuff that applies now/regardless and nothing to do with my patch implementation. > IMHO how it looks with pre win8 - the time-of-day API is not suitable for > performance measurements. I actually always presumed that it was your > goal. For this goal the hrtime stuff can be used where you even can > measure with pure QPC ticks when on a fast machine (say when you're out of > even ns range). As if we cannot provide a valid synchronizable timestamp > with QPC, the qualitative performance measurement is easy done using > relative time intervals. Most would not be using something not commonly available... And regarding relative time intervals, couldn't QPC counter roll over on some system (like VM)? Or it handles that somehow? Or be out-of-sync between cores on some hardware with XP? > May I suggest you to pack your work into a PECL extension, or maybe even > merge with hrtime. As there are several aspects anyway where we have an > issue, more or less. For example `uniqid()` you've mentioned. Or actually, > `usleep()` IMHO - much more of issue than microtime currently as the > resolution used there can be much worse. Anyway, it can come back to core > after it was good tested, is well thought and solid. That's what I would > suggest to do. Yeah `usleep()` is way more coarse on Windows because of timer res., right? Nothing can be done about that. > Best regards > > Anatol Thanks, Matt --- ## #77142 — Pierre Joye — 2014-09-10T23:21:19+00:00 URL: https://externals.io/message/77014#77142 · In reply to: #77141 Hi Matt, I do not think we should change anything in a stable branch now. We may consider it for 5.6.1 once we are 200% sure we can back port it to 5.5. Btw your patch is not windows only but also change the uuid function. Not a bad thing as u never got why we have this modulo anyway :) Cheers, Pierre --- ## #77146 — Anatol Belski — 2014-09-11T02:20:55+00:00 URL: https://externals.io/message/77014#77146 · In reply to: #77141 Hi Matt, > Hi Anatol, all, > > > Sorry for top post... Stas must be getting confused with all these > details! :-) Is Pierre going to weigh in? Are there any other Windows > people? > > While I don't think I'd disagree with much of what you've said, I don't > see that using my patch is *causing* or *creating* new issues that you're > referring to. Whatever is broken with Windows is still the *same* with > what's there now since last year. > > All I wanted to do, that I think Windows users want, is make `microtime()` > at least resemble *micro* time (very well, resolution-wise) so that it *is* > suitable for performance measurements. It's useless for that now. And > it > > happens to fix the `uniqid()` problem to boot. > > I don't think there were [m]any issues over 10+ years using `microtime()` > for that? It was that implementation's errors over time (which mine > doesn't have) that were revealed when compared to another value, like > REQUEST_TIME[_FLOAT]. > > > Maybe I used the wrong subject wording or NEWS file update -- I'm not > trying to improve actual "accuracy" over what's available now -- just > insure that > > it stays accurate as it can be (within range) with respect to the current > time, unlike the old implementation. > > So things are MUCH better for `microtime()` useable resolution and > `uniqid()`. > > What are some actual usage scenarios that my implementation would make > WORSE > than they are now? I don't see how... It seems you're against the idea, > in principle, of it working at all like the old way (in a good way)? :-/ > The > > > patch doesn't make all these new problems like you are suggesting. > > More below... > > > > ----- Original Message ----- > From: "Anatol Belski" > Sent: Tuesday, September 09, 2014 > > > > […] > have > > […] > VM. > > > […] > > I'm not trying to solve a hardware bug at all with math. The idea is > only > > to indicate the *certain* passage of time between calls while the > returned > > time, otherwise, remains the same (as one example). > > > > […] > neither > > […] > in > > […] > > You are talking about "synchronizable," etc. but I'm not trying to do > anything like that. Just a time that usually moves forward (except window > > > of x amount of time it could stay the same if clock is advancing *very* > slowly (QPC too far ahead and "clamped" to max); still better than now). > > > […] > > And on timer resolution, right? AFAIK the default timer resolution > (lowest) > is always equal to the time_update_rate (lpTimeIncrement). Again, not that > this matters much anyway, since currently results are same or worse than > my implementation. Mine doesn't make anything worse. > > > […] > clock > > […] > > See above again. :-) Only moving the micro time forward when it doesn't > otherwise change. > > > […] > > I was actually wondering about the QPC latency (but not an issue raised > before) on diff systems... I don't see why there'd be a delay getting > system time. Regardless, I can call `microtime()` on my system (older > Q9400) > 2 million times/sec. (inc. loop overhead), so not losing microseconds when > > > half the calls return the same value. :-) > > > […] > starting > > […] > interval > > […] > from > > […] > the > > […] > > I already said how I'm purposely not taking any SetSystemTimeAdjustment() > calls into account (besides the fact that it's probably never used). > Complicates things much more than is needed to simply estimate time > passage between actual system time updates. The clock changing +/- 1 > sec/hour (that's probably getting extreme as far as errors) is less than 5 > microseconds per 1/64 second update interval. Doesn't matter when it's > less than any fluctuation anyway. Certainly not an issue compared to now! > > > > […] > > If the estimate is behind (either over time or because clock jumped > forward), that simply brings it to the minimum amount within range (so it > "jumps" as little as acceptable). The only way that could *not* be > at/within the previous update interval is if the clock was updating > *slower* > than normal (TimeAdjustment < TimeIncrement) in which case the estimate > would be *ahead*, not behind. > > > […] > if > > […] > > There's always some gain -- can't stop the clock. :-) But if could, > still > > no worse than now! > > > […] > http://msdn.microsoft.com/en-us/library/windows/desktop/ms724394%28v=vs.8 > 5 > %29.aspx > > > […] > lpTimeAdjustment > > […] > rate > > […] > > Yes, I know all that... > > > > […] > call > > […] > course. > > […] > any > > […] > > Discrepancy? No more than now. After running for any amount of time? > No > > > more than now. > > It would be very hard to synchronize a Windows system's time anyway (I > guess on windowstimestamp.com it's trying to help with that, but who's > using that?), so the bidding resolution stuff is almost moot anyway -- > problem is same now, my patch doesn't hurt anything like I'm saying. > > You can't do any synchronized communication now either anyway. How would > you? Even IF a clock was truely synchronized, there's no way in PHP (or > anything) to wait for/until a certain time to the millisecond... > > But of course this is all theory stuff that applies now/regardless and > nothing to do with my patch implementation. > > > […] > for > > […] > of > > […] > > Most would not be using something not commonly available... And > regarding > > relative time intervals, couldn't QPC counter roll over on some system > (like > VM)? Or it handles that somehow? Or be out-of-sync between cores on some > > > hardware with XP? > > > […] > core > > […] > > Yeah `usleep()` is way more coarse on Windows because of timer res., right? > Nothing can be done about that. > > > > […] > > Thanks, > Matt > > > > -- > > > > > I'm not against the principle in general, just saying that I have a serious doubt that it works as you expect. That's why doing this overtime homework after reading your mail. Firstly - the argument that time adjustments aren't used is simply not true. Here's a simple snippet which can check what the time adjustment is while (1) { DWORD adj, inc; BOOL disabled; GetSystemTimeAdjustment(&adj, &inc, &disabled) /* Increment is 156250 which is the number of the ns units, so move it to ms: * 156250 * 100 ns / 1000 == 156250/10 us/s == 15625 us/s = 15.625 ms/s * */ if (!disabled) { printf("adjustment=%fms increment=%fms disabled=%d gain=%fms/s\n", adj/10000.0, inc/10000.0, disabled, 1000.0*((adj - inc)/(double)inc)); } else { printf("time adjustment disabled\n"); } } The windowstimestamp.com describes the increment values which apply to most systems I test on, 156250 is one of those and is described as RTC Periodic Interrupt at 64 Hz (see the document from my first mail). To my experience, the system clock will advance up to +/-90us every lpTimeIncrement, the smallest I've seen was like 20us. The lpTimeAdjustment can change after any update. However when a system uses HPET, it'll replace the RTC interrupt. So that's for one. Furthermore, sometimes GetSystemTimeAdjustment() switches to disabled, sometimes to enabled. The doc says, lpTimeIncrement and lpTimeAdjustment only make sense when lpTimeAdjustmentDisabled is false. GetSystemTimeAsFileTime() to my experience costs up to 20us or more. Just to call it. Here is the snippet to illustrate: __int64 freq, start, end; double elapsed FILETIME tod; if (!QueryPerformanceFrequency((LARGE_INTEGER *)&freq)) { return 1; } QueryPerformanceCounter((LARGE_INTEGER *)&start); GetSystemTimeAsFileTime(&tod); QueryPerformanceCounter((LARGE_INTEGER *)&end); elapsed = ((end - start)*1000000000)/freq; printf("GetSystemTimeAsFileTime took %fns\n", elapsed); Measuring it with QPC. Despite MSDN says QPC might cost like 20ns, in my test I can't even measure it in nanoseconds, so it's a matter of ticks. It's easy to modify this snippet to measure QPC as well. So the scenario like - get QPC (cheap) - get time-of-day (20ms uncertainty) - get correct time adjustment (have to measure that as well, but still some ms lost) - calculate IMHO the result isn't usable by the best will, looks more like a random value within some range. Here at the end a bit of the ASCII art I've prepared as an example: QPC 200us sample +/- 1 tick -|----------|----------|----------|----------|----------|----------|----------|---- PM clock advance with adjustment + 20us to get the time + Xms to get the adjustment -|----------|----------|----------|----------|----------|----------|----------|---- 50us 50us -50us 0us 90us 90us ..... PM clock interrupts ---|------|------|------|------|------|------|------|------|------|------|------|--- 1 2 3 4 5 6 7 8 9 10 11 12 You can see that while sampling on seemingly same interval with QPC and PM clock, a situation is possible that the time-of-day clock could have been updated more than once inbetween. So this is a hardware issue, say one is using TSC and the other PM interrupt or HPET, plus time adjustment. As you mentioned, whether QPC can rollover, also a big question (which adds more instability actually) as it depends on the source used. On the modern HW it usually uses TSC which is stable, on different systems however it can have drifts with the system clock, even on linux as mentioned in earlier mails. QPC will anyway care about anything like multicore or threads. I understand your intention to fulfill the user expectation, however for the performance tests much more qualitative results can be achieved with QPC and relative time intervals. That's for the answer what worse could it make as it was or is. For the normal usage, the microtime improvement were of course important, however with some feasible scenario. Also, even if they blame the current solution - it works well for many systems and scenarios. Just unlucky it doesn't work for yours ;) . Also don't forget that win8 is there. Anyway, I would suggest to do more homework and come up with a plausible algorithm that can be trusted to some higher level. Maybe doing it into a PECL ext so users can rate it. Regards Anatol