# New run-tests.php feature - Thread: https://externals.io/message/55054 - Original archive: https://news-web.php.net/php.internals/55054 - Started by Hannes Magnusson on 2011-09-01T11:35:17+00:00 - Messages: 12 - Reply by email to internals@lists.php.net - More for agents: search with https://externals.io/search.md?q={query} · JSON API at https://externals.io/api · documentation at https://externals.io/llms.txt --- ## #55054 — Hannes Magnusson — 2011-09-01T11:35:17+00:00 URL: https://externals.io/message/55054#55054 Throw qa and internals@ into the loop to. I'd also like to move the XFAIL section (printed out in the end) above the FAIL section. We have bucketloads of xfailed tests, I actually have to scroll up to see the failed tests which is really annoying and silly. Anyone have strong opinion on the printout order? -Hannes ---------- Forwarded message ---------- From: Hannes Magnusson Date: Thu, Sep 1, 2011 at 13:32 Subject: Re: [PHP-CVS] svn: /php/php-src/ branches/PHP_5_3/run-tests.php branches/PHP_5_4/run-tests.php trunk/run-tests.php To: Pierre Joye Cc: php-cvs@lists.php.net I did want to use -v at first, but that has a completely different meaning already. Its really only useful when looking at what is going on in a single testcase to debug run-tests :] -Hannes > ah right, I miss this part of the patch. I would suggest -v instead > and add a level of verbosity (less means no output but failed tests > for example). > > > > […] > > > > -- > Pierre > > @pierrejoye | http://blog.thepimp.net | http://www.libgd.org > > -- > PHP CVS Mailing List (http://www.php.net/) --- ## #55055 — johannes@schlueters.de — 2011-09-01T12:05:53+00:00 URL: https://externals.io/message/55054#55055 · In reply to: #55054 Hi, > Throw qa and internals@ into the loop to. > > I'd also like to move the XFAIL section (printed out in the end) above > the FAIL section. > We have bucketloads of xfailed tests, I actually have to scroll up to > see the failed tests which is really annoying and silly. Good thing to do! johannes > Anyone have strong opinion on the printout order? > > -Hannes > > > ---------- Forwarded message ---------- > From: Hannes Magnusson > Date: Thu, Sep 1, 2011 at 13:32 > Subject: Re: [PHP-CVS] svn: /php/php-src/ > branches/PHP_5_3/run-tests.php branches/PHP_5_4/run-tests.php > trunk/run-tests.php > To: Pierre Joye > Cc: php-cvs@lists.php.net > > > I did want to use -v at first, but that has a completely different > meaning already. > Its really only useful when looking at what is going on in a single > testcase to debug run-tests :] > > -Hannes > > > > […] --- ## #55056 — Ferenc Kovacs — 2011-09-01T12:09:09+00:00 URL: https://externals.io/message/55054#55056 · In reply to: #55054 > Throw qa and internals@ into the loop to. > > I'd also like to move the XFAIL section (printed out in the end) above > the FAIL section. > We have bucketloads of xfailed tests, I actually have to scroll up to > see the failed tests which is really annoying and silly. > > Anyone have strong opinion on the printout order? yeah, strongly agree -- Ferenc Kovács @Tyr43l - http://tyrael.hu --- ## #55057 — Alexey Shein — 2011-09-01T13:53:58+00:00 URL: https://externals.io/message/55054#55057 · In reply to: #55056 2011/9/1 Ferenc Kovacs : > > > […] > > yeah, strongly agree Great addition. While you're at it, could you also add an option to print a list of skipped tests (maybe above XFAIL section), that's also currently can't be done with other options in run-tests.php. It could also be useful because some skipped tests maybe skipped for the wrong reason and be actually failed. It's also a great spot to notice obsolete tests that need to be removed from the suite and make running all process faster. -- Regards, Shein Alexey --- ## #55059 — Hannes Magnusson — 2011-09-01T14:08:03+00:00 URL: https://externals.io/message/55054#55059 · In reply to: #55057 > 2011/9/1 Ferenc Kovacs : > > […] > > Great addition. > While you're at it, could you also add an option to print a list of > skipped tests (maybe above XFAIL section), that's also currently can't > be done with other options in run-tests.php. > It could also be useful because some skipped tests maybe skipped for > the wrong reason and be actually failed. It's also a great spot to > notice obsolete tests that need to be removed from the suite and make > running all process faster. Depending on how you run the test suite, you could get buuuuuuuunch of skipped tests.. Unsure if its that good idea to include them in the report. To see the skipped tests while running the test suite, and nothing else, you can use the new switch/environment variable as mentioned earlier in this thread.. -Hannes --- ## #55060 — johannes@schlueters.de — 2011-09-01T14:09:42+00:00 URL: https://externals.io/message/55054#55060 · In reply to: #55057 > Great addition. > While you're at it, could you also add an option to print a list of > skipped tests (maybe above XFAIL section), that's also currently can't > be done with other options in run-tests.php. > It could also be useful because some skipped tests maybe skipped for > the wrong reason and be actually failed. It's also a great spot to > notice obsolete tests that need to be removed from the suite and make > running all process faster. Let's open a can of worms: When designing this I'd like to keep in mind a feature I had on the list for the rewrite of run-tests by Zoe (anybody knows the status of that?) which is skipping entire directories with a single skipif. Benefit there is that for instance we don't have to create two PHP process for each and every test and don't have to try connecting to databases two times for each test (during skipif and the actual test). johannes > -- > Regards, > Shein Alexey --- ## #55061 — Hannes Magnusson — 2011-09-01T14:17:14+00:00 URL: https://externals.io/message/55054#55061 · In reply to: #55060 2011/9/1 Johannes Schlüter : > > > […] > > Let's open a can of worms: When designing this I'd like to keep in mind > a feature I had on the list for the rewrite of run-tests by Zoe (anybody > knows the status of that?) which is skipping entire directories with a > single skipif. > Benefit there is that for instance we don't have to create two PHP > process for each and every test and don't have to try connecting to > databases two times for each test (during skipif and the actual test). Create a magical "run-me-first" file which run-tests will execute first, and if it fails it will skip all the tests in that directory.. should be relatively easy to implement. Being able to run the tests for two directories in parallel however.. that would be awesome. -Hannes --- ## #55062 — Ferenc Kovacs — 2011-09-01T14:36:32+00:00 URL: https://externals.io/message/55054#55062 · In reply to: #55061 2011/9/1 Hannes Magnusson : > 2011/9/1 Johannes Schlüter : > > […] > > Create a magical "run-me-first" file which run-tests will execute > first, and if it fails it will skip all the tests in that directory.. > should be relatively easy to implement. > > Being able to run the tests for two directories in parallel however.. > that would be awesome. > > -Hannes > Hi guys, I think it would be useful to add this to the issue tracker, and if nobody else, I will look into this. -- Ferenc Kovács @Tyr43l - http://tyrael.hu --- ## #55063 — johannes@schlueters.de — 2011-09-01T14:45:57+00:00 URL: https://externals.io/message/55054#55063 · In reply to: #55061 > Create a magical "run-me-first" file which run-tests will execute > first, and if it fails it will skip all the tests in that directory.. > should be relatively easy to implement. See patch at http://schlueters.de/~johannes/php/run-tests-directory-skip.diff ;-) This will look for a file called SKIP, which has to be a PHP file, in each directory while looking for tests. There's at least one bug left: INI options aren't passed to the SKIP run so shared modules etc. will not be found. If somebody could extend it accordingly: great. :-) > Being able to run the tests for two directories in parallel however.. > that would be awesome. Ack. While that's a bit more complex as we can't run tests for a native database and the corresponding PDO driver at the same time, most likely. So we need some locking mechanism. johannes --- ## #55064 — Kalle Sommer Nielsen — 2011-09-01T14:47:43+00:00 URL: https://externals.io/message/55054#55064 · In reply to: #55061 Hi 2011/9/1 Hannes Magnusson : > Create a magical "run-me-first" file which run-tests will execute > first, and if it fails it will skip all the tests in that directory.. > should be relatively easy to implement. > > Being able to run the tests for two directories in parallel however.. > that would be awesome. I reckon Felipe had a patch for this back in the early 5.3 development days, Felipe? -- regards, Kalle Sommer Nielsen kalle@php.net --- ## #55065 — Felipe Pena — 2011-09-01T15:37:33+00:00 URL: https://externals.io/message/55054#55065 · In reply to: #55064 2011/9/1 Kalle Sommer Nielsen : > Hi > > 2011/9/1 Hannes Magnusson : > > […] > > I reckon Felipe had a patch for this back in the early 5.3 development > days, Felipe? > > > > -- > regards, > > Kalle Sommer Nielsen > kalle@php.net > Here's the patch: http://felipe.ath.cx/diff/run-tests.diff -- Regards, Felipe Pena --- ## #55067 — Hannes Magnusson — 2011-09-01T18:02:12+00:00 URL: https://externals.io/message/55054#55067 · In reply to: #55065 > 2011/9/1 Kalle Sommer Nielsen : > > […] > > Here's the patch: http://felipe.ath.cx/diff/run-tests.diff Hmh. This patch seems to be similar to johannes' patch, being able to skip entire directories based on a "configure file". I don't see a way to execute a check though. f.e. running the mysql tests won't do you any good unless you have the environment properly setup, so the skipif section needs to be able to check if it can connect to a db, and if not then you can skip the entire folder.. -Hannes