# [PATCH] make it possible to skip very slow tests - Thread: https://externals.io/message/38365 - Original archive: https://news-web.php.net/php.internals/38365 - Started by Steph Fox on 2008-06-18T20:37:08+00:00 - Messages: 20 - Reply by email to internals@lists.php.net - More for agents: search with https://externals.io/search.md?q={query} · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt --- ## #38365 — Steph Fox — 2008-06-18T20:37:08+00:00 URL: https://externals.io/message/38365#38365 Hi again, I'm using this locally because two of our tests take over 10 minutes each to run on my laptop, and I run the relevant bits of test suite every time I make a change. All it does is adds another option, -x, to run-tests.php. This sets an environmental variable which can then be checked for in the SKIPIF section of very slow-running tests. Any objections if I commit it in 5_3/HEAD? - Steph --- ## #38366 — Stanislav Malyshev — 2008-06-18T21:05:56+00:00 URL: https://externals.io/message/38365#38366 · In reply to: #38365 Hi! > + $environment['EXEMPT_SLOW_TESTS'] = 1; Maybe "SKIP_SLOW_TESTS"? If it's checked in skip section... :) Otherwise - good idea! -- Stanislav Malyshev, Zend Software Architect stas@zend.com http://www.zend.com/ (408)253-8829 MSN: stas@zend.com --- ## #38367 — Travis Swicegood — 2008-06-18T22:09:29+00:00 URL: https://externals.io/message/38365#38367 · In reply to: #38365 Hey Steph; > I'm using this locally because two of our tests take over 10 > minutes each to run on my laptop, and I run the relevant bits of > test suite every time I make a change. > > All it does is adds another option, -x, to run-tests.php. This sets > an environmental variable which can then be checked for in the > SKIPIF section of very slow-running tests. How do you specify that "test A is slow"? Is there a certain skipif message you include, or...? -T --- ## #38374 — Christopher Jones — 2008-06-19T03:46:57+00:00 URL: https://externals.io/message/38365#38374 · In reply to: #38365 Steph Fox wrote: > Hi again, > > I'm using this locally because two of our tests take over 10 minutes > each to run on my laptop, and I run the relevant bits of test suite > every time I make a change. > > All it does is adds another option, -x, to run-tests.php. This sets an > environmental variable which can then be checked for in the SKIPIF > section of very slow-running tests. > > Any objections if I commit it in 5_3/HEAD? I'd prefer a run-tests.php option that sets the timeout limit in seconds. Chris -- Christopher Jones, Oracle Email: christopher.jones@oracle.com Tel: +1 650 506 8630 Blog: http://blogs.oracle.com/opal/ Free PHP Book: http://tinyurl.com/f8jad --- ## #38378 — Steph Fox — 2008-06-19T11:02:40+00:00 URL: https://externals.io/message/38365#38378 · In reply to: #38374 > I'd prefer a run-tests.php option that sets the timeout limit in seconds. Nice idea, but I'm not sure it's achievable under CLI. - Steph --- ## #38379 — Steph Fox — 2008-06-19T11:17:22+00:00 URL: https://externals.io/message/38365#38379 · In reply to: #38367 Hi Travis, > > […] > > How do you specify that "test A is slow"? Is there a certain skipif > message you include, or...? Yep. The two that bug me most are actually labelled 'slow test', so having a skipif condition for those wouldn't be too obscure. Note that it's only intended for when you're repeatedly running the same group of tests... I like Chris's idea better, but can't see a way to implement it. There's probably something really obvious I'm missing, let me know if. - Steph --- ## #38380 — Steph Fox — 2008-06-19T11:21:30+00:00 URL: https://externals.io/message/38365#38380 · In reply to: #38366 Hey Stas, > > […] > > Maybe "SKIP_SLOW_TESTS"? If it's checked in skip section... :) > Otherwise - good idea! You can tell I'm reading my mail backwards today... I used 'EXEMPT' because the option is 'x' and I wanted it to be easy to remember. 's' wasn't available. - Steph --- ## #38381 — Nuno Lopes — 2008-06-19T11:56:11+00:00 URL: https://externals.io/message/38365#38381 · In reply to: #38378 > > […] > > Nice idea, but I'm not sure it's achievable under CLI. Yes, it is. Check the system_with_timeout() function in the run-tests.php script. There you've the timeout hardcoded ('$leak_check ? 300 : 60'). You would just need to make it configurable by some environment var. Nuno --- ## #38382 — Steph Fox — 2008-06-19T12:07:41+00:00 URL: https://externals.io/message/38365#38382 · In reply to: #38381 > Yes, it is. Check the system_with_timeout() function in the run-tests.php > script. > There you've the timeout hardcoded ('$leak_check ? 300 : 60'). You would > just need to make it configurable by some environment var. I already tried hard-coding both tv_sec and tv_usec to 0 and it makes no difference here. - Steph --- ## #38390 — Travis Swicegood — 2008-06-19T13:14:26+00:00 URL: https://externals.io/message/38365#38390 · In reply to: #38382 > > > […] > > I already tried hard-coding both tv_sec and tv_usec to 0 and it > makes no difference here. I can add this to PHPT - it uses a timeout based on reading the open proc. The only problem is that it is currently treated as an error. I could maybe add a new reporting level of timeout, though I do like the idea of having some sort of meta-data to conditionally skip tests. Maybe a better solution is to add an --exclude and ask people to either place potentially slow tests in tests/slow/, or name then .slow.phpt? Being able to exclude a pattern of test names definitely has more use than just setting a timeout. Thoughts? -T --- ## #38392 — Nuno Lopes — 2008-06-19T13:19:51+00:00 URL: https://externals.io/message/38365#38392 · In reply to: #38382 > > […] > > I already tried hard-coding both tv_sec and tv_usec to 0 and it makes no > difference here. uhm, file a bug report then. It was supposed to work.. Nuno --- ## #38393 — Steph Fox — 2008-06-19T13:26:42+00:00 URL: https://externals.io/message/38365#38393 · In reply to: #38390 > I can add this to PHPT - it uses a timeout based on reading the open > proc. That's what the current run-tests does too. The problem is it isn't reliable cross-platform - we don't have any control over the select() function used by `stream_select()` except under Windows, where it doesn't work (I'm looking into that but can't see why). > The only problem is that it is currently treated as an error. I could > maybe add a new reporting level of timeout, though I do like the idea of > having some sort of meta-data to conditionally skip tests. > > Maybe a better solution is to add an --exclude and ask people > to either place potentially slow tests in tests/slow/, or name then case>.slow.phpt? Being able to exclude a pattern of test names > definitely has more use than just setting a timeout. So what was wrong with the simple skipif and env var approach again? If you're going to mark a test to get run-tests to respond to it, it doesn't make a lot of difference whether you mark that test by its name or with a skipif line. The only difference is an untidy CVS directory. I think the naming thing would be a better idea for something that's just starting out. It's a less good idea for a test suite that already exists IMHO. - Steph > > Thoughts? > > -T --- ## #38394 — Steph Fox — 2008-06-19T13:30:46+00:00 URL: https://externals.io/message/38365#38394 · In reply to: #38392 > uhm, file a bug report then. It was supposed to work.. There are some open on this already. Also - Windows is the only environment where we actually have control over whether it works or not. Everything else relies on a system call. - Steph --- ## #38396 — Travis Swicegood — 2008-06-19T13:43:22+00:00 URL: https://externals.io/message/38365#38396 · In reply to: #38393 > > […] > > That's what the current run-tests does too. The problem is it isn't > reliable cross-platform - we don't have any control over the select > () function used by `stream_select()` except under Windows, where it > doesn't work (I'm looking into that but can't see why). This doesn't help you yet, but that's a moot point in PHPT as it uses COM under Windows as proc_* functions and the various stream functions are less than reliable. > So what was wrong with the simple skipif and env var approach > again? If you're going to mark a test to get run-tests to respond > to it, it doesn't make a lot of difference whether you mark that > test by its name or with a skipif line. The only difference is an > untidy CVS directory. There's nothing wrong with that approach. I'm trying to find something that addresses the problem (i.e., "how can I not run tests that are going to take a long time to run?"), while providing enough flexibility to answer other problems (i.e., "how can I skip X tests that I don't care about?"). If there's a way to address the problem without making it so specific, I'm all for it. > I think the naming thing would be a better idea for something > that's just starting out. It's a less good idea for a test suite > that already exists IMHO. Agreed. If we were to go this route, I'd actually consider adding a run-tests.ini or some such that allows you to configure things like excluded file patterns. -T --- ## #38398 — Steph Fox — 2008-06-19T14:03:00+00:00 URL: https://externals.io/message/38365#38398 · In reply to: #38396 > There's nothing wrong with that approach. I'm trying to find something > that addresses the problem (i.e., "how can I not run tests that are going > to take a long time to run?"), while providing enough flexibility to > answer other problems (i.e., "how can I skip X tests that I don't care > about?"). > > If there's a way to address the problem without making it so specific, > I'm all for it. I mostly agree - I'm just looking at 'here and now' rather than 'when the nice new test suite stuff is done'. 'Here and now', there isn't a reliable way to set this up and skipif looks like the cleanest option. > > […] > > Agreed. If we were to go this route, I'd actually consider adding a > run-tests.ini or some such that allows you to configure things like > excluded file patterns. That might not be a bad idea. - Steph --- ## #38401 — Christopher Jones — 2008-06-19T14:31:14+00:00 URL: https://externals.io/message/38365#38401 · In reply to: #38393 Steph Fox wrote: > So what was wrong with the simple skipif and env var approach again? The problem is it only skips the test! I'd like my slow tests to run (this generally occurs when I use a very remote DB). I end up manually increasing the timeout in `stream_select()` in run-tests.sh. This works for me on Linux. If we make the timeout value adjustable, you can set it to a low value so your slow tests are quickly aborted, and I can set it to a high value so my tests are run. Chris -- Christopher Jones, Oracle Email: christopher.jones@oracle.com Tel: +1 650 506 8630 Blog: http://blogs.oracle.com/opal/ Free PHP Book: http://tinyurl.com/f8jad --- ## #38404 — Steph Fox — 2008-06-19T14:52:45+00:00 URL: https://externals.io/message/38365#38404 · In reply to: #38401 Hi Chris, > If we make the timeout value adjustable, you can set it to a low value > so your slow tests are quickly aborted, and I can set it to a high > value so my tests are run. It's easily enough done, it's just a separate issue. I can easily put a workaround in the Windows version of select.c to make a 0-second timeout work, but that's as far as it goes. The problem in run-tests (if anyone's interested - Travis seems to have a better solution) is that the actual read is always only 8192 bytes, and the actual API call never times out because `stream_select()` is reset at every iteration and called afresh. We'd need to be able to set a timeout limit in the run-tests script rather than in the call to `stream_select()`. And if we do that the tests of course fail, which isn't a good outcome. So 'skipif' suits my needs better, but not yours. I'll add both. - Steph > > Chris > > -- > Christopher Jones, Oracle > Email: christopher.jones@oracle.com Tel: +1 650 506 8630 > Blog: http://blogs.oracle.com/opal/ Free PHP Book: > http://tinyurl.com/f8jad --- ## #38407 — Travis Swicegood — 2008-06-19T15:05:54+00:00 URL: https://externals.io/message/38365#38407 · In reply to: #38398 > > > […] > > I mostly agree - I'm just looking at 'here and now' rather than > 'when the nice new test suite stuff is done'. 'Here and now', there > isn't a reliable way to set this up and skipif looks like the > cleanest option. The only problem with that is everything you add I've got to add to my GSoC project so PHPT :-) > That might not be a bad idea. I think the long-term goal should be the ability to "force skip" files based on an --exclude parameter, an ini conf file (looks for -- ini-file or /tests.ini), and an ENV variable. The first and last would just be separated by the `PATH_SEPARATOR` for regex patterns. In addition, we can definitely make the time-out something that's settable via the command line and conf, but as you noted in your next email, that is a separate issue. I've created a few tickets on these so we can make sure to track these issues: * add exclude: http://phpt.info/ticket/69 * add timeout: http://phpt.info/ticket/70 -T --- ## #38413 — Christopher Jones — 2008-06-19T16:01:44+00:00 URL: https://externals.io/message/38365#38413 · In reply to: #38404 Steph Fox wrote: > So 'skipif' suits my needs better, but not yours. I'll add both. Thanks Steph. Chris -- Christopher Jones, Oracle Email: christopher.jones@oracle.com Tel: +1 650 506 8630 Blog: http://blogs.oracle.com/opal/ Free PHP Book: http://tinyurl.com/f8jad --- ## #38418 — Greg Beaver — 2008-06-19T17:37:48+00:00 URL: https://externals.io/message/38365#38418 · In reply to: #38407 Travis Swicegood wrote: > > > > […] > > The only problem with that is everything you add I've got to add to my > GSoC project so PHPT :-) > > > […] > > I think the long-term goal should be the ability to "force skip" files > based on an --exclude parameter, an ini conf file (looks for --ini-file > or /tests.ini), and an ENV variable. The first and last > would just be separated by the `PATH_SEPARATOR` for regex patterns. The ini file is a great idea, as one can create groupings of test suites independent of the tests. Greg