Concurrency support for run-tests

9 years ago by Andrea Faulds — view source

unread

I also have a patch for run-tests sitting around for quite some time, which adds concurrent test execution support. I already fixed a lot of non re-entrant tests in the past, but there might still be quite some of them.

Did you, or will you, add a new directive to say a test can’t be run concurrently?

Andrea Faulds
http://ajf.me/

9 years ago by Michael Wallner — view source

unread

Hi Andrea!

I also have a patch for run-tests sitting around for quite some time,
which adds concurrent test execution support. I already fixed a lot of non
re-entrant tests in the past, but there might still be quite some of them.

Did you, or will you, add a new directive to say a test can’t be run
concurrently?

Hm... didn't think of that, seems overly complex. What would actually need
the singularity of the universe? :)

--
Regards,
Mike

9 years ago by Andrea Faulds — view source

unread

Hm... didn't think of that, seems overly complex. What would actually need the singularity of the universe? :)

I don’t think it’d be difficult. It wouldn’t mean you can’t run tests alongside it, just that you can’t run it concurrently with another test with the same no-concurrent directive. So, if you had, say, a non-concurrent filesystem test, and another one, and some unrelated tests, you’d have to run those two FS tests sequentially, but you could still run unrelated tests at the same time.

--
Andrea Faulds
http://ajf.me/

9 years ago by Laruence — view source

unread

Hi!

I also have a patch for run-tests sitting around for quite some time, which
adds concurrent test execution support. I already fixed a lot of non
re-entrant tests in the past, but there might still be quite some of them.

Tests in ./Zend take 8 seconds instead of 30 on my box.

Please find the hack attached. Thoughts, praises, death threats?

I have dreamed for it for a long time......

thanks!

--
Regards,
Mike

--

--
Laruence Xinchen Hui
http://www.laruence.com/

9 years ago by Laruence — view source

unread

Hey:

Hi!

I also have a patch for run-tests sitting around for quite some time, which
adds concurrent test execution support. I already fixed a lot of non
re-entrant tests in the past, but there might still be quite some of them.

Tests in ./Zend take 8 seconds instead of 30 on my box.

Please find the hack attached. Thoughts, praises, death threats?

seems it enable concurrency by default ? it's not configurable?

maybe:

TEST_PHP_ARGS="-c 3" make test

means run 3 process

thanks

--
Regards,
Mike

--

--
Laruence Xinchen Hui
http://www.laruence.com/

9 years ago by Pierre Joye — view source

unread

Hey:

Hi!

I also have a patch for run-tests sitting around for quite some time, which
adds concurrent test execution support. I already fixed a lot of non
re-entrant tests in the past, but there might still be quite some of them.

Tests in ./Zend take 8 seconds instead of 30 on my box.

Please find the hack attached. Thoughts, praises, death threats?

seems it enable concurrency by default ? it's not configurable?

maybe:

TEST_PHP_ARGS="-c 3" make test

means run 3 process

sounds too make specific, a simple concurrency argument would do it.

About the need of a flag for the tests, it would be useful as well.
Many tests can't be in run in parallel, like many mysql tests (or
other databases f.e.).

cheers,

Pierre

@pierrejoye | http://www.libgd.org

9 years ago by Michael Wallner — view source

unread

Hey:

Hi!

I also have a patch for run-tests sitting around for quite some time,
which
adds concurrent test execution support. I already fixed a lot of non
re-entrant tests in the past, but there might still be quite some of
them.

Tests in ./Zend take 8 seconds instead of 30 on my box.

Please find the hack attached. Thoughts, praises, death threats?

seems it enable concurrency by default ? it's not configurable?

maybe:

TEST_PHP_ARGS="-c 3" make test

means run 3 process

sounds too make specific, a simple concurrency argument would do it.

Yes, there's a --concurrency switch in run-tests.php

About the need of a flag for the tests, it would be useful as well.
Many tests can't be in run in parallel, like many mysql tests (or
other databases f.e.).

Why can't those tests use unique (to them) names?

--
Regards,
Mike

9 years ago by Pierre Joye — view source

unread

About the need of a flag for the tests, it would be useful as well.
Many tests can't be in run in parallel, like many mysql tests (or
other databases f.e.).

Why can't those tests use unique (to them) names?

mysql f.e. initialized a DB, may have initial data as well and tests
alter them. The setup scripts may be changed and allows one DB per
test but I am not sure it will save much time. However you can try,
maybe it is worth it :)

--
Pierre

@pierrejoye | http://www.libgd.org

9 years ago by Michael Wallner — view source

unread

Hey:

Hi!

I also have a patch for run-tests sitting around for quite some time,
which
adds concurrent test execution support. I already fixed a lot of non
re-entrant tests in the past, but there might still be quite some of
them.

Tests in ./Zend take 8 seconds instead of 30 on my box.

Please find the hack attached. Thoughts, praises, death threats?

seems it enable concurrency by default ? it's not configurable?

maybe:

TEST_PHP_ARGS="-c 3" make test

means run 3 process

Actually it has a --concurrency switch. If you didn't know, you can pass
any run-tests argument to make test with the TESTS variable:

$ make test TESTS="--concurrency 8 ~/src/php-master/Zend"

HTH

--
Regards,
Mike

9 years ago by Ferenc Kovacs — view source

unread

Hi!

I also have a patch for run-tests sitting around for quite some time,
which adds concurrent test execution support. I already fixed a lot of non
re-entrant tests in the past, but there might still be quite some of them.

Tests in ./Zend take 8 seconds instead of 30 on my box.

Please find the hack attached. Thoughts, praises, death threats?

--
Regards,
Mike

--

for the record, we had a rewrite of run-tests.php which supported parallel
execution of the tests:
http://git.php.net/?p=phpruntests.git;a=summary
docs:
https://wiki.php.net/qa/runtests/documentation

--
Ferenc Kovács
@Tyr43l - http://tyrael.hu

9 years ago by Michael Wallner — view source

unread

Hi!

I also have a patch for run-tests sitting around for quite some time,
which adds concurrent test execution support. I already fixed a lot of non
re-entrant tests in the past, but there might still be quite some of them.

Tests in ./Zend take 8 seconds instead of 30 on my box.

Please find the hack attached. Thoughts, praises, death threats?

--
Regards,
Mike

--

for the record, we had a rewrite of run-tests.php which supported parallel
execution of the tests:
http://git.php.net/?p=phpruntests.git;a=summary
docs:
https://wiki.php.net/qa/runtests/documentation

What's the status of this? Why don't we use it?
How about letting travis run it additionally for some time?

--
Regards,
Mike

9 years ago by Sara Golemon — view source

unread

I also have a patch for run-tests sitting around for quite some time, which
adds concurrent test execution support. I already fixed a lot of non
re-entrant tests in the past, but there might still be quite some of them.

HHVM uses PHP's test suite and runs them in parallel. We upstreamed a
number of fixes for tests to make these behave properly (many tests
used common, generic filenames like "test.txt" which would clobber
over each other when run concurrently). Would love to see PHP eat up
a few more cores too.

-Sara

9 years ago by Nuno Lopes — view source

unread

I also have a patch for run-tests sitting around for quite some time,
which
adds concurrent test execution support. I already fixed a lot of non
re-entrant tests in the past, but there might still be quite some of
them.

HHVM uses PHP's test suite and runs them in parallel. We upstreamed a
number of fixes for tests to make these behave properly (many tests
used common, generic filenames like "test.txt" which would clobber
over each other when run concurrently). Would love to see PHP eat up
a few more cores too.

The other problem regarding running tests in parallel are DB extensions.
Many DB tests use the same tables. Or extensions that create sockets on the
same port (e.g., ftp).
Having thought a bit about the subject, the heuristic I came up with was to

do not run tests from the same extension in parallel, and 2) do not run
tests from certain extensions in parallel (e.g., mysql and pdo_mysql).

BTW, I made this patch in 2007 for run-tests.php that was a naive
parallelization of the system:
http://gcov.php.net/~nlopess/multi_threaded_run_tests.txt

Nuno

9 years ago by Stas Malyshev — view source

unread

Hi!

Having thought a bit about the subject, the heuristic I came up with was to

do not run tests from the same extension in parallel, and 2) do not run
tests from certain extensions in parallel (e.g., mysql and pdo_mysql).

How about making it a whitelist? For tests/extensions we know it's easy
to paralellize (like ones not having external dependencies/servers/etc)
simple script could whitelist a lot of them. For those more tricky, we
could whitelist them gradually.

--
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/

9 years ago by Michael Wallner — view source

unread

Hi!

Having thought a bit about the subject, the heuristic I came up with was to

do not run tests from the same extension in parallel, and 2) do not run
tests from certain extensions in parallel (e.g., mysql and pdo_mysql).

Heuristics are not reliable enough.

How about making it a whitelist? For tests/extensions we know it's easy
to paralellize (like ones not having external dependencies/servers/etc)
simple script could whitelist a lot of them. For those more tricky, we
could whitelist them gradually.

Whitelists are very limitating by nature, nobody will go and add tests
to this whitelist, I think. I don't think a script is a viable way to
whitelist tests (think about all those tests with with includes or
config files, etc). Tests that fail due to concurrency will be easier
noticed and added to a concurrency group or even better be fixed.

I added a CONCURRENCY_GROUP section, see
https://github.com/m6w6/php-src/compare/parallel-run-tests?expand=1

If the next test to execute is in the same CONCURRENCY_GROUP as one
which is already/still running, it will be pushed back.

So tests with f.e. listening servers could use "port:9999" as
CONCURRENCY_GROUP or mysql tests could use "mysql:dbname", etc. As
example, I added the open_basedir tests to one group because they all
tamper around with the same subdirectories.

I also added a shuffle($test_files) if $concurrency is greater than 2
because that helps running the test suite concurrently actually.
Anybody got an idea why the tests are sorted at all?

--
Regards,
Mike

9 years ago by Stas Malyshev — view source

unread

Hi!

I added a CONCURRENCY_GROUP section, see
https://github.com/m6w6/php-src/compare/parallel-run-tests?expand=1

If the next test to execute is in the same CONCURRENCY_GROUP as one
which is already/still running, it will be pushed back.

So tests with f.e. listening servers could use "port:9999" as
CONCURRENCY_GROUP or mysql tests could use "mysql:dbname", etc. As
example, I added the open_basedir tests to one group because they all
tamper around with the same subdirectories.

The problem there is that for a lot of tests the dependency is in common
include file (like server.inc or connect.inc) and extracting it into the
main file may be a bit time-consuming and also means we'd have to keep
the information in sync in many places at once.

I also added a shuffle($test_files) if $concurrency is greater than 2
because that helps running the test suite concurrently actually.
Anybody got an idea why the tests are sorted at all?

To have a predictable order and reproducible test runs? Imagine we have
an inadvertent test dependency, in which test X fails after test Y is
run. If the order is stable, it is relatively easy to discover. If it is
random, reproducing the exact test alignment which caused the failure is
next to impossible.

--
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/

9 years ago by Michael Wallner — view source

unread

Hi!

Having thought a bit about the subject, the heuristic I came up with was to

do not run tests from the same extension in parallel, and 2) do not run
tests from certain extensions in parallel (e.g., mysql and pdo_mysql).

How about making it a whitelist? For tests/extensions we know it's easy
to paralellize (like ones not having external dependencies/servers/etc)
simple script could whitelist a lot of them. For those more tricky, we
could whitelist them gradually.

So how about adding the concurrency switch, but leaving it disabled by
default?

--
Regards,
Mike

9 years ago by Matt Ficken — view source

unread

When building PFTT, the PHP Full Test Tool, one goal was maximizing speed
so that we would be more likely to frequently Fully test PHP (Full-coverage
of many scenarios), and to do that I built it to run tests concurrently...

Concurrent/parallel execution can cause breaks that otherwise work with
normal serial execution, so definitely don't enable concurrency in
run-tests by default.

run-tests.php's two main benefits are that it supports running the huge
library of 15k+pecl PHPTs (compatiblity) and its a lightweight tool so that
anyone who just built PHP can run make test to test their build (whereas
PFTT Is a heavyweight tool).

In making/merging major changes (like concurrency) the most important thing
is to not break that compatibility and dependability:

Your patch will fail without the /proc filesystem(mac osx?). On Windows,
you can check the %NUMBER_OF_PROCESSORS% environment variable. For unices
that don't have /proc there may not be a single simple way to check.

For better performance, PFTT runs a pool of 3 to 4 processes per CPU, when
the processes finish quickly, it runs fewer and when they slow down (due to
pending IO ops), it runs more processes. See:
http://git.php.net/?p=pftt2.git;a=blob;f=src/com/mostc/pftt/scenario/CLIScenario.java;h=30c81aabcb56698eebe4884d915298c92332b50d;hb=refs/heads/master#l61
and for other SAPI scenarios like Apache it runs 2 or 3 processes, see:
http://git.php.net/?p=pftt2.git;a=blob;f=src/com/mostc/pftt/scenario/ProductionWebServerScenario.java;h=d509bddda2b3756f05321ddc273600aeb38372be;hb=refs/heads/master

For more speed, rather than grouping all, fe open_basedir tests into a
single Concurrency Group its better to have them all operate in separate
temporary directories and database tables (then you could run 32 or 64
openbase dir tests at once).

The ext/standard/tests/file and ext/mysql* PHPTs take the longest to run
and mostly do IO operations so they benefit the most from maximizing
concurrency.

For databases, PFTT will create a new database (and for phpunit tests
populate the database) for each test runner thread, See:
http://git.php.net/?p=pftt2.git;a=blob;f=src/com/mostc/pftt/scenario/MySQLScenario.java;h=f768f58c4494cb2705a571ece22d533bfdc15192;hb=refs/heads/master#l164

Hi!

Having thought a bit about the subject, the heuristic I came up with
was to

do not run tests from the same extension in parallel, and 2) do not
run
tests from certain extensions in parallel (e.g., mysql and pdo_mysql).

How about making it a whitelist? For tests/extensions we know it's easy
to paralellize (like ones not having external dependencies/servers/etc)
simple script could whitelist a lot of them. For those more tricky, we
could whitelist them gradually.

So how about adding the concurrency switch, but leaving it disabled by
default?

--
Regards,
Mike

9 years ago by Michael Wallner — view source

unread

In making/merging major changes (like concurrency) the most important
thing is to not break that compatibility and dependability:

Your patch will fail without the /proc filesystem(mac osx?). On Windows,
you can check the %NUMBER_OF_PROCESSORS% environment variable. For unices
that don't have /proc there may not be a single simple way to check.

It won't "fail", it just wouldn't enable reasonable concurrency by default.
But auto-enabling concurrency is out of question now anyway.

While the patch looks big, most of it is caused by indentation, because
some code was converted to a callback.

For more speed, rather than grouping all, fe open_basedir tests into a
single Concurrency Group its better to have them all operate in separate
temporary directories and database tables (then you could run 32 or 64
openbase dir tests at once).

Sure, I tried to unfuck the open_basedir tests, but failed once.

Ps: Have to look at PFTT yet.