multi-threaded run-tests.php

18 years ago by Nuno Lopes — view source

unread

Hi,

Recently we have moved http://gcov.php.net to a new (and much faster)
server. This allowed us to reduce the time to build all branches to "just" 2
days (compared with almost one week that would take previously).
However I'm still not happy :P The server has 2x2 CPU cores and I would like
to use them :) Currently I can speed-up the make process, because it can be
parallelized (with -j xx). But the major time is still spent with
run-tests.php.

So, what I would like to propose is a multi-threaded version of
run-tests.php. We could spawn x processes (configurable) and one of them
would test y extensions (I think it's better to have each extension's tests
tested serially or they may break, because they may use the same resources:
BDs, files, etc..).

Any comments?

Thanks,
Nuno

18 years ago by Richard Lynch — view source

unread

Recently we have moved http://gcov.php.net to a new (and much faster)
server. This allowed us to reduce the time to build all branches to
"just" 2
days (compared with almost one week that would take previously).
However I'm still not happy :P The server has 2x2 CPU cores and I
would like
to use them :) Currently I can speed-up the make process, because it
can be
parallelized (with -j xx). But the major time is still spent with
run-tests.php.

So, what I would like to propose is a multi-threaded version of
run-tests.php. We could spawn x processes (configurable) and one of
them
would test y extensions (I think it's better to have each extension's
tests
tested serially or they may break, because they may use the same
resources:
BDs, files, etc..).

Sounds good to this naive reader.

Set X high enough, and you might even find an multi-extension
thread-safe issue we've been missing. :-v

If I'm reading it right, for sure the pgsql extension tests are serial
within the extension, as it re-uses the DB connection and presumes
test db setup in early tests, so you're 100% right on the inability to
parallelize that.

--
Some people have a "gift" link here.
Know what I want?
I want you to buy a CD from some indie artist.
http://cdbaby.com/browse/from/lynch
Yeah, I get a buck. So?

18 years ago by David Wang — view source

unread

Please.

This would be a big help to me, since I'm running these tests many,
many times a day as I develop the reference cycle collector.

Recently we have moved http://gcov.php.net to a new (and much faster)
server. This allowed us to reduce the time to build all branches to
"just" 2
days (compared with almost one week that would take previously).
However I'm still not happy :P The server has 2x2 CPU cores and I
would like
to use them :) Currently I can speed-up the make process, because it
can be
parallelized (with -j xx). But the major time is still spent with
run-tests.php.

So, what I would like to propose is a multi-threaded version of
run-tests.php. We could spawn x processes (configurable) and one of
them
would test y extensions (I think it's better to have each extension's
tests
tested serially or they may break, because they may use the same
resources:
BDs, files, etc..).

Sounds good to this naive reader.

Set X high enough, and you might even find an multi-extension
thread-safe issue we've been missing. :-v

If I'm reading it right, for sure the pgsql extension tests are serial
within the extension, as it re-uses the DB connection and presumes
test db setup in early tests, so you're 100% right on the inability to
parallelize that.

--
Some people have a "gift" link here.
Know what I want?
I want you to buy a CD from some indie artist.
http://cdbaby.com/browse/from/lynch
Yeah, I get a buck. So?

18 years ago by Sebastian Bergmann — view source

unread

Nuno Lopes wrote:

Any comments?

It's on the TODO for the run-tests.php rewrite. I think I/we should
start a wiki for this somewhere. Lukas?

--
Sebastian Bergmann http://sebastian-bergmann.de/
GnuPG Key: 0xB85B5D69 / 27A7 2B14 09E4 98CD 6277 0E5B 6867 C514 B85B 5D69

18 years ago by Antony Dovgal — view source

unread

Nuno Lopes wrote:

Any comments?

It's on the TODO for the run-tests.php rewrite. I think I/we should
start a wiki for this somewhere. Lukas?

Good idea, let's gather all the idea there.

--
Wbr,
Antony Dovgal

18 years ago by Nuno Lopes — view source

unread

So here it is my first attempt:
http://gcov.php.net/~nlopess/multi_threaded_run_tests.txt
WARNING: The patch is ugly and the output of the script isn't the prettiest.

Anyway, for automated testing seems to be good enough.

Nuno

P.S.: after this little hack, I also start to feel that run-tests.php should
be rewritten :P

""Nuno Lopes"" nlopess@php.net wrote:

Hi,

Recently we have moved http://gcov.php.net to a new (and much faster)
server. This allowed us to reduce the time to build all branches to "just"
2 days (compared with almost one week that would take previously).
However I'm still not happy :P The server has 2x2 CPU cores and I would
like to use them :) Currently I can speed-up the make process, because it
can be parallelized (with -j xx). But the major time is still spent with
run-tests.php.

So, what I would like to propose is a multi-threaded version of
run-tests.php. We could spawn x processes (configurable) and one of them
would test y extensions (I think it's better to have each extension's
tests tested serially or they may break, because they may use the same
resources: BDs, files, etc..).

Any comments?

Thanks,
Nuno

18 years ago by Richard Quadling — view source

unread

So here it is my first attempt:
http://gcov.php.net/~nlopess/multi_threaded_run_tests.txt
WARNING: The patch is ugly and the output of the script isn't the prettiest.

Anyway, for automated testing seems to be good enough.

Nuno

P.S.: after this little hack, I also start to feel that run-tests.php should
be rewritten :P

""Nuno Lopes"" nlopess@php.net wrote:

Hi,

Recently we have moved http://gcov.php.net to a new (and much faster)
server. This allowed us to reduce the time to build all branches to "just"
2 days (compared with almost one week that would take previously).
However I'm still not happy :P The server has 2x2 CPU cores and I would
like to use them :) Currently I can speed-up the make process, because it
can be parallelized (with -j xx). But the major time is still spent with
run-tests.php.

So, what I would like to propose is a multi-threaded version of
run-tests.php. We could spawn x processes (configurable) and one of them
would test y extensions (I think it's better to have each extension's
tests tested serially or they may break, because they may use the same
resources: BDs, files, etc..).

Any comments?

Thanks,
Nuno

Hey Nuno, you should stop looking at things. PhD and now run-tests.
Soon you'll be taking a look at the php-src and thinking, "pah, look
what those idiots have done, time for a rewrite!". :-)

--

Richard Quadling
Zend Certified Engineer : http://zend.com/zce.php?c=ZEND002498&r=213474731
"Standing on the shoulders of some very clever giants!"

18 years ago by Richard Lynch — view source

unread

Hey Nuno, you should stop looking at things. PhD and now run-tests.
Soon you'll be taking a look at the php-src and thinking, "pah, look
what those idiots have done, time for a rewrite!". :-)

Deja-vu:
Isn't that exactly what Ze'ev and Andi did for PHP4? :-)

Only you have to convince one of your professors that re-writing it
instead of doing all your homework is worth a full course credit like
they did. :-)

Apologies in advance to Ze'ev and Andi for slightly distorting
history, I think. (But not much, I hope)

--
Some people have a "gift" link here.
Know what I want?
I want you to buy a CD from some indie artist.
http://cdbaby.com/browse/from/lynch
Yeah, I get a buck. So?

18 years ago by Antony Dovgal — view source

unread

So here it is my first attempt:
http://gcov.php.net/~nlopess/multi_threaded_run_tests.txt
WARNING: The patch is ugly and the output of the script isn't the prettiest.

Anyway, for automated testing seems to be good enough.

Nuno

P.S.: after this little hack, I also start to feel that run-tests.php should
be rewritten :P

Unfortunately, this is 100% truth, Sebastian have had some ideas about it.

--
Wbr,
Antony Dovgal

18 years ago by Andrey Hristov — view source

unread

Hi Tony,
Antony Dovgal wrote:

So here it is my first attempt:
http://gcov.php.net/~nlopess/multi_threaded_run_tests.txt
WARNING: The patch is ugly and the output of the script isn't the
prettiest.

Anyway, for automated testing seems to be good enough.

Nuno

P.S.: after this little hack, I also start to feel that run-tests.php
should be rewritten :P

Unfortunately, this is 100% truth, Sebastian have had some ideas about it.

I also have a hack, that makes it possible to re-run a test few times. I
call it iteration. For every iteration there is $_ENV['TEST_ITERATION'].
Based on which different consts are defined by the pre-include script. I
am using it for testing mysqlnd against a bunch of MySQL Server, the
minimalistic configuration is doing 3 iterations against 4.1, 5.0 and
5.1 Server. Not sure that other extensions could benefit from it, but at
least the DB connectivity ones..like testing OCI8....

Regards,
Andrey

18 years ago by David Coallier — view source

unread

Hi Tony,
Antony Dovgal wrote:

So here it is my first attempt:
http://gcov.php.net/~nlopess/multi_threaded_run_tests.txt
WARNING: The patch is ugly and the output of the script isn't the
prettiest.

Anyway, for automated testing seems to be good enough.

Nuno

P.S.: after this little hack, I also start to feel that run-tests.php
should be rewritten :P

Unfortunately, this is 100% truth, Sebastian have had some ideas about it.

I also have a hack, that makes it possible to re-run a test few times. I
call it iteration. For every iteration there is $_ENV['TEST_ITERATION'].
Based on which different consts are defined by the pre-include script. I
am using it for testing mysqlnd against a bunch of MySQL Server, the
minimalistic configuration is doing 3 iterations against 4.1, 5.0 and
5.1 Server. Not sure that other extensions could benefit from it, but at
least the DB connectivity ones..like testing OCI8....

Where ? :) I am interested in seeing it.

Regards,
Andrey

--

--
David Coallier,
Founder & Software Architect,
Agora Production (http://agoraproduction.com)
51.42.06.70.18

18 years ago by Richard Lynch — view source

unread

Hi Tony,
Antony Dovgal wrote:

So here it is my first attempt:
http://gcov.php.net/~nlopess/multi_threaded_run_tests.txt
WARNING: The patch is ugly and the output of the script isn't the
prettiest.

Anyway, for automated testing seems to be good enough.

Nuno

P.S.: after this little hack, I also start to feel that
run-tests.php
should be rewritten :P

Unfortunately, this is 100% truth, Sebastian have had some ideas
about it.

I also have a hack, that makes it possible to re-run a test few times.
I
call it iteration. For every iteration there is
$_ENV['TEST_ITERATION'].
Based on which different consts are defined by the pre-include script.
I
am using it for testing mysqlnd against a bunch of MySQL Server, the
minimalistic configuration is doing 3 iterations against 4.1, 5.0 and
5.1 Server. Not sure that other extensions could benefit from it, but
at
least the DB connectivity ones..like testing OCI8....

FWIW:

I spent about 10 to 20 minutes trying to figure out if it was possible
to easily run just a subset of tests and failed.

I dunno if that indicates that it's a feature to be added, one to be
better documented, or that I'm just being stupid (again), but there it
is. :-)

--
Some people have a "gift" link here.
Know what I want?
I want you to buy a CD from some indie artist.
http://cdbaby.com/browse/from/lynch
Yeah, I get a buck. So?

18 years ago by Rasmus Lerdorf — view source

unread

Richard Lynch wrote:

Hi Tony,
Antony Dovgal wrote:

So here it is my first attempt:
http://gcov.php.net/~nlopess/multi_threaded_run_tests.txt
WARNING: The patch is ugly and the output of the script isn't the
prettiest.

Anyway, for automated testing seems to be good enough.

Nuno

P.S.: after this little hack, I also start to feel that
run-tests.php
should be rewritten :P
Unfortunately, this is 100% truth, Sebastian have had some ideas
about it.

I also have a hack, that makes it possible to re-run a test few times.
I
call it iteration. For every iteration there is
$_ENV['TEST_ITERATION'].
Based on which different consts are defined by the pre-include script.
I
am using it for testing mysqlnd against a bunch of MySQL Server, the
minimalistic configuration is doing 3 iterations against 4.1, 5.0 and
5.1 Server. Not sure that other extensions could benefit from it, but
at
least the DB connectivity ones..like testing OCI8....

FWIW:

I spent about 10 to 20 minutes trying to figure out if it was possible
to easily run just a subset of tests and failed.

I dunno if that indicates that it's a feature to be added, one to be
better documented, or that I'm just being stupid (again), but there it
is. :-)

make test TESTS=ext/foo

Or, add this little shell script which I call phpt:

#!/bin/bash
TEST_PHP_EXECUTABLE=/usr/local/bin/php
php /usr/local/lib/php/build/run-tests.php $1

Then cd into whatever directory has your tests and type:

phpt .

-Rasmus
-Rasmus

18 years ago by Sebastian Bergmann — view source

unread

Rasmus Lerdorf wrote:

make test TESTS=ext/foo

Or, add this little shell script which I call phpt:

#!/bin/bash
TEST_PHP_EXECUTABLE=/usr/local/bin/php
php /usr/local/lib/php/build/run-tests.php $1

Then cd into whatever directory has your tests and type:

phpt .

Yeah, letting make take care of this sounds like the right thing to do.
It should also solve Ulf's problem of not running the tests for the
various MySQL extensions at the same time somehow.

--
Sebastian Bergmann http://sebastian-bergmann.de/
GnuPG Key: 0xB85B5D69 / 27A7 2B14 09E4 98CD 6277 0E5B 6867 C514 B85B 5D69

18 years ago by Marcus Boerger — view source

unread

Hello Richard,

erm, php run-tests.php -h or --help was to hard?

Friday, July 27, 2007, 5:56:10 AM, you wrote:

Hi Tony,
Antony Dovgal wrote:

So here it is my first attempt:
http://gcov.php.net/~nlopess/multi_threaded_run_tests.txt
WARNING: The patch is ugly and the output of the script isn't the
prettiest.

Anyway, for automated testing seems to be good enough.

Nuno

P.S.: after this little hack, I also start to feel that
run-tests.php
should be rewritten

Unfortunately, this is 100% truth, Sebastian have had some ideas
about it.

I also have a hack, that makes it possible to re-run a test few times.
I
call it iteration. For every iteration there is
$_ENV['TEST_ITERATION'].
Based on which different consts are defined by the pre-include script.
I
am using it for testing mysqlnd against a bunch of MySQL Server, the
minimalistic configuration is doing 3 iterations against 4.1, 5.0 and
5.1 Server. Not sure that other extensions could benefit from it, but
at
least the DB connectivity ones..like testing OCI8....

FWIW:

I spent about 10 to 20 minutes trying to figure out if it was possible
to easily run just a subset of tests and failed.

I dunno if that indicates that it's a feature to be added, one to be
better documented, or that I'm just being stupid (again), but there it
is.

--
Some people have a "gift" link here.
Know what I want?
I want you to buy a CD from some indie artist.
http://cdbaby.com/browse/from/lynch
Yeah, I get a buck. So?

Best regards,
Marcus

18 years ago by Marcus Boerger — view source

unread

Hello Richard,

that -h issue is actually understandable. If run-tests-pgp was exectuable
you would more likely be tempted to try -h. Maybe that means we should make
it so by adding a hash bang line or by adding a short shell script that will
do so call run-tests.pgp with the correct php version.

marcus

Tuesday, July 31, 2007, 5:40:53 AM, you wrote:

I did try variations on "make test --help" but that just got me make
help...

I would have expected php xxx.php -h to simply print the help for PHP
and exit and didn't think to try that.

Sorry!

Hello Richard,

erm, php run-tests.php -h or --help was to hard?

Friday, July 27, 2007, 5:56:10 AM, you wrote:

Hi Tony,
Antony Dovgal wrote:

So here it is my first attempt:
http://gcov.php.net/~nlopess/multi_threaded_run_tests.txt
WARNING: The patch is ugly and the output of the script isn't the
prettiest.

Anyway, for automated testing seems to be good enough.

Nuno

P.S.: after this little hack, I also start to feel that
run-tests.php
should be rewritten

Unfortunately, this is 100% truth, Sebastian have had some ideas
about it.

I also have a hack, that makes it possible to re-run a test few
times.
I
call it iteration. For every iteration there is
$_ENV['TEST_ITERATION'].
Based on which different consts are defined by the pre-include
script.
I
am using it for testing mysqlnd against a bunch of MySQL Server,
the
minimalistic configuration is doing 3 iterations against 4.1, 5.0
and
5.1 Server. Not sure that other extensions could benefit from it,
but
at
least the DB connectivity ones..like testing OCI8....

FWIW:

I spent about 10 to 20 minutes trying to figure out if it was
possible
to easily run just a subset of tests and failed.

I dunno if that indicates that it's a feature to be added, one to be
better documented, or that I'm just being stupid (again), but there
it
is.

--
Some people have a "gift" link here.
Know what I want?
I want you to buy a CD from some indie artist.
http://cdbaby.com/browse/from/lynch
Yeah, I get a buck. So?

Best regards,
Marcus

--
Best regards,
Marcus mailto:mail@marcus-boerger.de