13 years ago by zoe slattery — view source

unread

Over the past couple of weeks I have updated the parallel run-tests
(fixed a couple of minor bugs in the PHP code and the build.xml), it's
now almost at the point where I could go ahead and implement the last
pieces. Here is a summary and a few questions:

In rebasing the code the the dev't stream I found a number of tests
with non-standard sections. My code checks test case structure and
objects to anything non-standard, the current run-tests.php mainly
ignores this kind of thing. I fixed up about 15 of these tests (see
#62022) already - I'll fix the rest if there are no objections - I will
open another bug report first.
If there is agreement to use this code it would make sense to
replace the existing run-tests code with it, or rather, it would make
no sense to try and maintain both versions. The new code is OO PHP, it's
in http://git.php.net/repository/phpruntests.git, is there any problem
with it staying there long term and maybe copying a run-tests.phar into
the PHP source directory? I have no idea what the right answer is,
suggestions welcome.
I ran a couple of small tests on my dual core Mac yesterday. For a
standard set of tests, the parallel code ran in 207 seconds, sequential
in 293 seconds and the standard run-tests.php took 298 seconds. This is
an improvement but I suspect we could still do better by looking at the
scheduling algorithm.
At the moment it's very simple, we just assemble a list of directories
with tests in and hand them out to processors till everything is done.
Being able to handle tests that must be run in sequence (mysql, mysqli)
will mean making some changes to this. So, perhaps we give an explicit
list to p1 and let the scheduler distribute the rest of the tests? Or
maybe we should have a 'process map' for all tests for extensions that
are build by default? Again, suggestions welcome.
REDIRECTTEST still needs to be implemented, I understand how it works
and this isn't (afaict) a major issue.
Testing. I'm able to do basic testing on Mac OSX and Linux. I really
need access to an 8 way Linux system, or someone who has access and
would be interested in looking at performance? Any volunteers? This is
probably the most interesting part of the project :-)
Windows. I'm not in a position to do anything much with Windows
except some very basic checks to make sure that the sequential version
runs. The parallel code won't work because we used pcntl(), however I
know that Stefan and George were keen to design the code so that a
Windows solution could be implemented if anyone thought of one. If
anyone wants to pick up this aspect I'd be happy to get them started.

Zoe

13 years ago by Olivier Doucet — view source

unread

Testing. I'm able to do basic testing on Mac OSX and Linux

Hi Zoe,

Here is a quick test on my 4-cores Linux server :

tested on PHP 5.3.9
412 tests executed
Sequential : 115,3 seconds
Parallel (4 threads) : 43,8 seconds
Original run-tests.php : 115,4 seconds

Olivier

13 years ago by zoe slattery — view source

unread

Olivier - brilliant thanks!

So, we have two points now: http://zoah.co.uk/update-parallel-run-tests-php

Anyone do an 8 way?

Zoe

Testing. I'm able to do basic testing on Mac OSX and Linux
Hi Zoe,

Here is a quick test on my 4-cores Linux server :

tested on PHP 5.3.9
412 tests executed
Sequential : 115,3 seconds
Parallel (4 threads) : 43,8 seconds
Original run-tests.php : 115,4 seconds

Olivier

13 years ago by Miah Gregory — view source

unread

Hi Zoe,

Any chance you could write up the steps you took to test this - or point
me in the right direction so I can try myself?

Re 8 way, is that 8 physical or 8 threads? If I can get the tests
running I can give you a data point for a 4 core/8 thread i7 under
linux.

--
Regards,

Miah

-----Original Message-----
From: zoe slattery aparachic@gmail.com
To: internals@lists.php.net, PHP QA php-qa@lists.php.net
Subject: [PHP-DEV] Parallel run-tests
Date: Thu, 17 May 2012 13:00:04 +0100
Mailer: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:12.0)
Gecko/20120428 Thunderbird/12.0.1

In rebasing the code the the dev't stream I found a number of tests
with non-standard sections. My code checks test case structure and
objects to anything non-standard, the current run-tests.php mainly
ignores this kind of thing. I fixed up about 15 of these tests (see
#62022) already - I'll fix the rest if there are no objections - I will
open another bug report first.
If there is agreement to use this code it would make sense to
replace the existing run-tests code with it, or rather, it would make
no sense to try and maintain both versions. The new code is OO PHP, it's
in http://git.php.net/repository/phpruntests.git, is there any problem
with it staying there long term and maybe copying a run-tests.phar into
the PHP source directory? I have no idea what the right answer is,
suggestions welcome.
I ran a couple of small tests on my dual core Mac yesterday. For a
standard set of tests, the parallel code ran in 207 seconds, sequential
in 293 seconds and the standard run-tests.php took 298 seconds. This is
an improvement but I suspect we could still do better by looking at the
scheduling algorithm.
At the moment it's very simple, we just assemble a list of directories
with tests in and hand them out to processors till everything is done.
Being able to handle tests that must be run in sequence (mysql, mysqli)
will mean making some changes to this. So, perhaps we give an explicit
list to p1 and let the scheduler distribute the rest of the tests? Or
maybe we should have a 'process map' for all tests for extensions that
are build by default? Again, suggestions welcome.
REDIRECTTEST still needs to be implemented, I understand how it works
and this isn't (afaict) a major issue.
Testing. I'm able to do basic testing on Mac OSX and Linux. I really
need access to an 8 way Linux system, or someone who has access and
would be interested in looking at performance? Any volunteers? This is
probably the most interesting part of the project :-)
Windows. I'm not in a position to do anything much with Windows
except some very basic checks to make sure that the sequential version
runs. The parallel code won't work because we used pcntl(), however I
know that Stefan and George were keen to design the code so that a
Windows solution could be implemented if anyone thought of one. If
anyone wants to pick up this aspect I'd be happy to get them started.

Zoe

13 years ago by zoe slattery — view source

unread

Hi Miah -

Any chance you could write up the steps you took to test this - or point
me in the right direction so I can try myself?

Re 8 way, is that 8 physical or 8 threads? If I can get the tests
running I can give you a data point for a 4 core/8 thread i7 under
linux.
that would be brilliant! I have a the steps written down in various
places - a bit scattered around :-/
I'll assemble a list of instructions and put it on the PHP wiki later
tonight, then I'll send out a link.

Thanks so much for the offer!

Zoe

13 years ago by Olivier Doucet — view source

unread

Hi,

Any chance you could write up the steps you took to test this - or point
me in the right direction so I can try myself?

Here is a quick setup that use your system php binary. It was written
live, may have some glitches but gives you an idea :

cd /usr/src
git clone git://git.php.net/phpruntests.git
wget "http://fr2.php.net/get/php-5.3.9.tar.bz2/from/this/mirror";
tar xf php-5.3.9.tar.bz2

you need pcntl extension

echo "Sequential new phpruntests"
php run-tests.php -p /usr/bin/php /usr/src/php-5.3.9/tests
echo "parallel new phpruntests"
#replace THREADCOUNT by number of cores
php run-tests.php -z THREADCOUNT -p /usr/bin/php /usr/src/php-5.3.9/tests
echo "old run test"
cd /tmp/php-5.3.9
TEST_PHP_EXECUTABLE=/usr/bin/php php tests/

Olivier

13 years ago by Felipe Pena — view source

unread

Hi Zoe,

great work on this new run-tests! :D

I was testing it and noticed a problem when running using the following command:
$ ~/dev/php5_4/sapi/cli/php run-tests.php -z 8 -p
~/dev/php5_4/sapi/cli/php ~/dev/php5_4/

It shows the PHP messages:
[...]
Fatal error: Call to undefined function gzencode() in
/home/felipe/dev/phpruntests/src/testcase/sections/configurationsections/rtGzipPostSection.php
on line 23
[...]
Notice: Undefined offset: 0 in
/home/felipe/dev/phpruntests/src/testcase/rtTestDifference.php on line
91
Fatal error: Allowed memory size of 134217728 bytes exhausted at
/home/felipe/dev/php5_4/Zend/zend_hash.c:330 (tried to allocate 81
bytes) in /home/felipe/dev/phpruntests/src/testcase/rtPhpTestFile.php
on line 108
/home/felipe/dev/php5_4/Zend/zend_hash.c(551) : ht=0x7f96e6356a50 is
inconsistent
[...]
Fatal error: Allowed memory size of 134217728 bytes exhausted at
ext/standard/var_unserializer.re:290 (tried to allocate 32 bytes) in
/home/felipe/dev/phpruntests/src/taskScheduler/rtTaskSchedulerFile.php
on line 183

2012/5/17 zoe slattery aparachic@gmail.com:

Hi

Over the past couple of weeks I have updated the parallel run-tests (fixed a
couple of minor bugs in the PHP code and the build.xml), it's now almost at
the point where I could go ahead and implement the last pieces. Here is a
summary and a few questions:

In rebasing the code the the dev't stream I found a number of tests with
non-standard sections. My code checks test case structure and objects to
anything non-standard, the current run-tests.php mainly ignores this kind of
thing. I fixed up about 15 of these tests (see #62022) already - I'll fix
the rest if there are no objections - I will open another bug report first.

If there is agreement to use this code it would make sense to replace
the existing run-tests code with it, or rather, it would make no sense to
try and maintain both versions. The new code is OO PHP, it's in
http://git.php.net/repository/phpruntests.git, is there any problem with it
staying there long term and maybe copying a run-tests.phar into the PHP
source directory? I have no idea what the right answer is, suggestions
welcome.

I ran a couple of small tests on my dual core Mac yesterday. For a
standard set of tests, the parallel code ran in 207 seconds, sequential in
293 seconds and the standard run-tests.php took 298 seconds. This is an
improvement but I suspect we could still do better by looking at the
scheduling algorithm.
At the moment it's very simple, we just assemble a list of directories with
tests in and hand them out to processors till everything is done. Being able
to handle tests that must be run in sequence (mysql, mysqli) will mean
making some changes to this. So, perhaps we give an explicit list to p1 and
let the scheduler distribute the rest of the tests? Or maybe we should have
a 'process map' for all tests for extensions that are build by default?
Again, suggestions welcome.

REDIRECTTEST still needs to be implemented, I understand how it works and
this isn't (afaict) a major issue.

Testing. I'm able to do basic testing on Mac OSX and Linux. I really
need access to an 8 way Linux system, or someone who has access and would be
interested in looking at performance? Any volunteers? This is probably the
most interesting part of the project :-)

Windows. I'm not in a position to do anything much with Windows except
some very basic checks to make sure that the sequential version runs. The
parallel code won't work because we used pcntl(), however I know that Stefan
and George were keen to design the code so that a Windows solution could be
implemented if anyone thought of one. If anyone wants to pick up this aspect
I'd be happy to get them started.

Zoe

--
PHP Quality Assurance Mailing List http://www.php.net/

--
Regards,
Felipe Pena

13 years ago by zoe slattery — view source

unread

Hi Zoe,

great work on this new run-tests! :D

I was testing it and noticed a problem when running using the following command:
$ ~/dev/php5_4/sapi/cli/php run-tests.php -z 8 -p
~/dev/php5_4/sapi/cli/php ~/dev/php5_4/

It shows the PHP messages:
[...]
Fatal error: Call to undefined function gzencode() in
/home/felipe/dev/phpruntests/src/testcase/sections/configurationsections/rtGzipPostSection.php
on line 23
[...]
Thanks for that - couple of questions:

Is your ~/dev/php5_4/sapi/cli/php built --with-zlib?
This test uses the php-cgi, are you setting it with an env variable?
Or relying on run-tests.php to find it (it should do that)

Even if it is missing zlib function that is causing the initial error,
this should be caught by a SKIPIF section. There is only one test (as
far as I know) that uses GZIP_POST, this is ext/soap/tests/server019.phpt.

Would you be able to run that test on its own and send me the output? I
think this should do it:

~/dev/php5_4/sapi/cli/php run-tests.php -p ~/dev/php5_4/sapi/cli/php ~/dev/php5_4/ -vvv ~/dev/php5_4/ext/soap/tests/server019.php

Thanks, Zoe

Notice: Undefined offset: 0 in
/home/felipe/dev/phpruntests/src/testcase/rtTestDifference.php on line
91
Fatal error: Allowed memory size of 134217728 bytes exhausted at
/home/felipe/dev/php5_4/Zend/zend_hash.c:330 (tried to allocate 81
bytes) in /home/felipe/dev/phpruntests/src/testcase/rtPhpTestFile.php
on line 108
/home/felipe/dev/php5_4/Zend/zend_hash.c(551) : ht=0x7f96e6356a50 is
inconsistent
[...]
Fatal error: Allowed memory size of 134217728 bytes exhausted at
ext/standard/var_unserializer.re:290 (tried to allocate 32 bytes) in
/home/felipe/dev/phpruntests/src/taskScheduler/rtTaskSchedulerFile.php
on line 183

2012/5/17 zoe slatteryaparachic@gmail.com:

Hi

Over the past couple of weeks I have updated the parallel run-tests (fixed a
couple of minor bugs in the PHP code and the build.xml), it's now almost at
the point where I could go ahead and implement the last pieces. Here is a
summary and a few questions:

In rebasing the code the the dev't stream I found a number of tests with
non-standard sections. My code checks test case structure and objects to
anything non-standard, the current run-tests.php mainly ignores this kind of
thing. I fixed up about 15 of these tests (see #62022) already - I'll fix
the rest if there are no objections - I will open another bug report first.

If there is agreement to use this code it would make sense to replace
the existing run-tests code with it, or rather, it would make no sense to
try and maintain both versions. The new code is OO PHP, it's in
http://git.php.net/repository/phpruntests.git, is there any problem with it
staying there long term and maybe copying a run-tests.phar into the PHP
source directory? I have no idea what the right answer is, suggestions
welcome.

I ran a couple of small tests on my dual core Mac yesterday. For a
standard set of tests, the parallel code ran in 207 seconds, sequential in
293 seconds and the standard run-tests.php took 298 seconds. This is an
improvement but I suspect we could still do better by looking at the
scheduling algorithm.
At the moment it's very simple, we just assemble a list of directories with
tests in and hand them out to processors till everything is done. Being able
to handle tests that must be run in sequence (mysql, mysqli) will mean
making some changes to this. So, perhaps we give an explicit list to p1 and
let the scheduler distribute the rest of the tests? Or maybe we should have
a 'process map' for all tests for extensions that are build by default?
Again, suggestions welcome.

REDIRECTTEST still needs to be implemented, I understand how it works and
this isn't (afaict) a major issue.

Testing. I'm able to do basic testing on Mac OSX and Linux. I really
need access to an 8 way Linux system, or someone who has access and would be
interested in looking at performance? Any volunteers? This is probably the
most interesting part of the project :-)

Windows. I'm not in a position to do anything much with Windows except
some very basic checks to make sure that the sequential version runs. The
parallel code won't work because we used pcntl(), however I know that Stefan
and George were keen to design the code so that a Windows solution could be
implemented if anyone thought of one. If anyone wants to pick up this aspect
I'd be happy to get them started.

Zoe

--
PHP Quality Assurance Mailing List<http://www.php.net/

13 years ago by zoe slattery — view source

unread

HI Felipe - a couple of updates.

Is your ~/dev/php5_4/sapi/cli/php built --with-zlib?

This test uses the php-cgi, are you setting it with an env
variable? Or relying on run-tests.php to find it (it should do that)

Even if it is missing zlib function that is causing the initial error,
this should be caught by a SKIPIF section. There is only one test (as
far as I know) that uses GZIP_POST, this is
ext/soap/tests/server019.phpt.
I rebuilt PHP with soap and without zlib. The skipif section in this
test seems to be working fine for me.

Would you be able to run that test on its own and send me the output?
I think this should do it:

~/dev/php5_4/sapi/cli/php run-tests.php -p ~/dev/php5_4/sapi/cli/php
~/dev/php5_4/ -vvv ~/dev/php5_4/ext/soap/tests/server019.php
I'd still like to see this.

Do you see the same issue in a sequential run? (ie without -z). If this
is specific to the number of processes that are being run I'd need to
put some extra debug in I think.

Thanks, Zoe

Thanks, Zoe

Notice: Undefined offset: 0 in
/home/felipe/dev/phpruntests/src/testcase/rtTestDifference.php on line
91
Fatal error: Allowed memory size of 134217728 bytes exhausted at
/home/felipe/dev/php5_4/Zend/zend_hash.c:330 (tried to allocate 81
bytes) in /home/felipe/dev/phpruntests/src/testcase/rtPhpTestFile.php
on line 108
/home/felipe/dev/php5_4/Zend/zend_hash.c(551) : ht=0x7f96e6356a50 is
inconsistent
[...]
Fatal error: Allowed memory size of 134217728 bytes exhausted at
ext/standard/var_unserializer.re:290 (tried to allocate 32 bytes) in
/home/felipe/dev/phpruntests/src/taskScheduler/rtTaskSchedulerFile.php
on line 183

2012/5/17 zoe slatteryaparachic@gmail.com:

Hi

Over the past couple of weeks I have updated the parallel run-tests
(fixed a
couple of minor bugs in the PHP code and the build.xml), it's now
almost at
the point where I could go ahead and implement the last pieces.
Here is a
summary and a few questions:

In rebasing the code the the dev't stream I found a number of
tests with
non-standard sections. My code checks test case structure and
objects to
anything non-standard, the current run-tests.php mainly ignores this
kind of
thing. I fixed up about 15 of these tests (see #62022) already -
I'll fix
the rest if there are no objections - I will open another bug report
first.

If there is agreement to use this code it would make sense to
replace
the existing run-tests code with it, or rather, it would make no
sense to
try and maintain both versions. The new code is OO PHP, it's in
http://git.php.net/repository/phpruntests.git, is there any problem
with it
staying there long term and maybe copying a run-tests.phar into the PHP
source directory? I have no idea what the right answer is, suggestions
welcome.

I ran a couple of small tests on my dual core Mac yesterday. For a
standard set of tests, the parallel code ran in 207 seconds,
sequential in
293 seconds and the standard run-tests.php took 298 seconds. This is an
improvement but I suspect we could still do better by looking at the
scheduling algorithm.
At the moment it's very simple, we just assemble a list of
directories with
tests in and hand them out to processors till everything is done.
Being able
to handle tests that must be run in sequence (mysql, mysqli) will mean
making some changes to this. So, perhaps we give an explicit list to
p1 and
let the scheduler distribute the rest of the tests? Or maybe we
should have
a 'process map' for all tests for extensions that are build by default?
Again, suggestions welcome.

REDIRECTTEST still needs to be implemented, I understand how it
works and
this isn't (afaict) a major issue.

Testing. I'm able to do basic testing on Mac OSX and Linux. I
really
need access to an 8 way Linux system, or someone who has access and
would be
interested in looking at performance? Any volunteers? This is
probably the
most interesting part of the project :-)

Windows. I'm not in a position to do anything much with Windows
except
some very basic checks to make sure that the sequential version
runs. The
parallel code won't work because we used pcntl(), however I know
that Stefan
and George were keen to design the code so that a Windows solution
could be
implemented if anyone thought of one. If anyone wants to pick up
this aspect
I'd be happy to get them started.

Zoe

--
PHP Quality Assurance Mailing List<http://www.php.net/

13 years ago by Nuno Lopes — view source

unread

Hi Zoe,

Thanks for undertaking this project!

I just have a few concerns about this:

The speedup seems a bit low to me. Maybe with a higher number of
extensions enabled it will improve?..
Is there any developer documentation available? If I wanted to do a
change or a bug fix today, how could I do it?
Can it be packaged as a single drop-in file replacement for run-tests.php?
The deployment is very important to me. IMHO, the optimal solution would be
to have a drop-in replacement for the current script, so that most
developers wouldn't even notice the difference.
From previous emails exchanged in this thread, it seems that this new
version requires a few extensions to run (gzip, soap??). This is not
acceptable. The script must be able to run with --disable-all. Of course in
that case the parallel version will fail.

Nuno

-----Original Message-----
From: zoe slattery
Sent: Thursday, May 17, 2012 1:00 PM
To: internals@lists.php.net ; PHP QA
Subject: [PHP-QA] Parallel run-tests

In rebasing the code the the dev't stream I found a number of tests
with non-standard sections. My code checks test case structure and
objects to anything non-standard, the current run-tests.php mainly
ignores this kind of thing. I fixed up about 15 of these tests (see
#62022) already - I'll fix the rest if there are no objections - I will
open another bug report first.
If there is agreement to use this code it would make sense to
replace the existing run-tests code with it, or rather, it would make
no sense to try and maintain both versions. The new code is OO PHP, it's
in http://git.php.net/repository/phpruntests.git, is there any problem
with it staying there long term and maybe copying a run-tests.phar into
the PHP source directory? I have no idea what the right answer is,
suggestions welcome.
I ran a couple of small tests on my dual core Mac yesterday. For a
standard set of tests, the parallel code ran in 207 seconds, sequential
in 293 seconds and the standard run-tests.php took 298 seconds. This is
an improvement but I suspect we could still do better by looking at the
scheduling algorithm.
At the moment it's very simple, we just assemble a list of directories
with tests in and hand them out to processors till everything is done.
Being able to handle tests that must be run in sequence (mysql, mysqli)
will mean making some changes to this. So, perhaps we give an explicit
list to p1 and let the scheduler distribute the rest of the tests? Or
maybe we should have a 'process map' for all tests for extensions that
are build by default? Again, suggestions welcome.
REDIRECTTEST still needs to be implemented, I understand how it works
and this isn't (afaict) a major issue.
Testing. I'm able to do basic testing on Mac OSX and Linux. I really
need access to an 8 way Linux system, or someone who has access and
would be interested in looking at performance? Any volunteers? This is
probably the most interesting part of the project :-)
Windows. I'm not in a position to do anything much with Windows
except some very basic checks to make sure that the sequential version
runs. The parallel code won't work because we used pcntl(), however I
know that Stefan and George were keen to design the code so that a
Windows solution could be implemented if anyone thought of one. If
anyone wants to pick up this aspect I'd be happy to get them started.

Zoe

13 years ago by zoe slattery — view source

unread

Hi Zoe,

Thanks for undertaking this project!

I just have a few concerns about this:

The speedup seems a bit low to me. Maybe with a higher number of
extensions enabled it will improve?..
I don't think it will improve with a higher number of extensions. It
may improve as a result of someone doing some work on the tests are
scheduled - but I wouldn't count on it. Looking at improving performance
is important and I wanted to get some confirmation that it is faster, I
think we have that level of confirmation now. However, if I'm the only
person working on this, further performance work will come after
complete implementation + debugging.

Is there any developer documentation available? If I wanted to do a
change or a bug fix today, how could I do it?
Lots - https://wiki.php.net/qa/runtests. I have just updated the
"Development" section which has instructions on how to build and test
the code. The 'Documentation' section has not been updated recently but
I think it is still valid.

Can it be packaged as a single drop-in file replacement for
run-tests.php? The deployment is very important to me. IMHO, the
optimal solution would be to have a drop-in replacement for the
current script, so that most developers wouldn't even notice the
difference.
Yes - totally agree. I think I experimented with packaging it as a .phar
(so it would be called run-tests.phar, not run-tests.php), but it's so
long ago that I can't remember. I have added it to the to-do list
(https://wiki.php.net/qa/runtests/todos).

From previous emails exchanged in this thread, it seems that this
new version requires a few extensions to run (gzip, soap??). This is
not acceptable. The script must be able to run with --disable-all. Of
course in that case the parallel version will fail.
No, it doesn't. That was a stupid bug and I fixed it last night :-)

Zoe

Nuno

-----Original Message----- From: zoe slattery
Sent: Thursday, May 17, 2012 1:00 PM
To: internals@lists.php.net ; PHP QA
Subject: [PHP-QA] Parallel run-tests

Hi

Over the past couple of weeks I have updated the parallel run-tests
(fixed a couple of minor bugs in the PHP code and the build.xml), it's
now almost at the point where I could go ahead and implement the last
pieces. Here is a summary and a few questions:

In rebasing the code the the dev't stream I found a number of tests
with non-standard sections. My code checks test case structure and
objects to anything non-standard, the current run-tests.php mainly
ignores this kind of thing. I fixed up about 15 of these tests (see
#62022) already - I'll fix the rest if there are no objections - I will
open another bug report first.

If there is agreement to use this code it would make sense to
replace the existing run-tests code with it, or rather, it would make
no sense to try and maintain both versions. The new code is OO PHP, it's
in http://git.php.net/repository/phpruntests.git, is there any problem
with it staying there long term and maybe copying a run-tests.phar into
the PHP source directory? I have no idea what the right answer is,
suggestions welcome.

I ran a couple of small tests on my dual core Mac yesterday. For a
standard set of tests, the parallel code ran in 207 seconds, sequential
in 293 seconds and the standard run-tests.php took 298 seconds. This is
an improvement but I suspect we could still do better by looking at the
scheduling algorithm.
At the moment it's very simple, we just assemble a list of directories
with tests in and hand them out to processors till everything is done.
Being able to handle tests that must be run in sequence (mysql, mysqli)
will mean making some changes to this. So, perhaps we give an explicit
list to p1 and let the scheduler distribute the rest of the tests? Or
maybe we should have a 'process map' for all tests for extensions that
are build by default? Again, suggestions welcome.

REDIRECTTEST still needs to be implemented, I understand how it works
and this isn't (afaict) a major issue.

Testing. I'm able to do basic testing on Mac OSX and Linux. I really
need access to an 8 way Linux system, or someone who has access and
would be interested in looking at performance? Any volunteers? This is
probably the most interesting part of the project :-)

Windows. I'm not in a position to do anything much with Windows
except some very basic checks to make sure that the sequential version
runs. The parallel code won't work because we used pcntl(), however I
know that Stefan and George were keen to design the code so that a
Windows solution could be implemented if anyone thought of one. If
anyone wants to pick up this aspect I'd be happy to get them started.

Zoe

13 years ago by Devis Lucato — view source

unread

Hi,

nice work, out of curiosity I tried on a VM with 24 virtual cores:

Sequential new phpruntests: 29.6 seconds
Parallel -z 4: 12.1 - 12.3 secs
Parallel -z 8: 10.6 - 10.9 secs
Parallel -z 16: 10.6 - 11.3 secs
Parallel -z 24: 10.9 - 11.7 secs

Btw I'm seeing this error:

PHP Warning: Invalid argument supplied for foreach() in
phpruntests/src/testrun/rtPhpTestRun.php on line 107
PHP Notice: Undefined offset: 1 in
phpruntests/src/testrun/rtPhpTestRun.php on line 107

Devis

13 years ago by zoe slattery — view source

unread

Hi,

nice work, out of curiosity I tried on a VM with 24 virtual cores:

Sequential new phpruntests: 29.6 seconds
Parallel -z 4: 12.1 - 12.3 secs
Parallel -z 8: 10.6 - 10.9 secs
Parallel -z 16: 10.6 - 11.3 secs
Parallel -z 24: 10.9 - 11.7 secs
Cool - thanks! I'll add those points to the performance chart. At a
quick look that fits pretty much with the other data. Out of curiosity,
how many 'test groups' was it?

Btw I'm seeing this error:

PHP Warning: Invalid argument supplied for foreach() in
phpruntests/src/testrun/rtPhpTestRun.php on line 107
PHP Notice: Undefined offset: 1 in
phpruntests/src/testrun/rtPhpTestRun.php on line 107
My bad. I started to implement REDIRECTTEST and checked in some initial
changes, I can see what I've done wrong and will fix later today.

Zoe

Devis

13 years ago by Nuno Lopes — view source

unread

Alright, thanks for the reply!
I'll try to have a look at the code and give it a try next weekend.

Nuno

-----Original Message-----
From: zoe slattery
Sent: Monday, May 21, 2012 8:26 AM
Subject: Re: [PHP-QA] Parallel run-tests

Hi Zoe,

Thanks for undertaking this project!

I just have a few concerns about this:

The speedup seems a bit low to me. Maybe with a higher number of
extensions enabled it will improve?..
I don't think it will improve with a higher number of extensions. It
may improve as a result of someone doing some work on the tests are
scheduled - but I wouldn't count on it. Looking at improving performance
is important and I wanted to get some confirmation that it is faster, I
think we have that level of confirmation now. However, if I'm the only
person working on this, further performance work will come after
complete implementation + debugging.

Is there any developer documentation available? If I wanted to do a
change or a bug fix today, how could I do it?
Lots - https://wiki.php.net/qa/runtests. I have just updated the
"Development" section which has instructions on how to build and test
the code. The 'Documentation' section has not been updated recently but
I think it is still valid.

Can it be packaged as a single drop-in file replacement for
run-tests.php? The deployment is very important to me. IMHO, the
optimal solution would be to have a drop-in replacement for the
current script, so that most developers wouldn't even notice the
difference.
Yes - totally agree. I think I experimented with packaging it as a .phar
(so it would be called run-tests.phar, not run-tests.php), but it's so
long ago that I can't remember. I have added it to the to-do list
(https://wiki.php.net/qa/runtests/todos).

From previous emails exchanged in this thread, it seems that this
new version requires a few extensions to run (gzip, soap??). This is
not acceptable. The script must be able to run with --disable-all. Of
course in that case the parallel version will fail.
No, it doesn't. That was a stupid bug and I fixed it last night :-)

Zoe

Nuno

-----Original Message----- From: zoe slattery
Sent: Thursday, May 17, 2012 1:00 PM
To: internals@lists.php.net ; PHP QA
Subject: [PHP-QA] Parallel run-tests

Hi

Over the past couple of weeks I have updated the parallel run-tests
(fixed a couple of minor bugs in the PHP code and the build.xml), it's
now almost at the point where I could go ahead and implement the last
pieces. Here is a summary and a few questions:

In rebasing the code the the dev't stream I found a number of tests
with non-standard sections. My code checks test case structure and
objects to anything non-standard, the current run-tests.php mainly
ignores this kind of thing. I fixed up about 15 of these tests (see
#62022) already - I'll fix the rest if there are no objections - I will
open another bug report first.

If there is agreement to use this code it would make sense to
replace the existing run-tests code with it, or rather, it would make
no sense to try and maintain both versions. The new code is OO PHP, it's
in http://git.php.net/repository/phpruntests.git, is there any problem
with it staying there long term and maybe copying a run-tests.phar into
the PHP source directory? I have no idea what the right answer is,
suggestions welcome.

I ran a couple of small tests on my dual core Mac yesterday. For a
standard set of tests, the parallel code ran in 207 seconds, sequential
in 293 seconds and the standard run-tests.php took 298 seconds. This is
an improvement but I suspect we could still do better by looking at the
scheduling algorithm.
At the moment it's very simple, we just assemble a list of directories
with tests in and hand them out to processors till everything is done.
Being able to handle tests that must be run in sequence (mysql, mysqli)
will mean making some changes to this. So, perhaps we give an explicit
list to p1 and let the scheduler distribute the rest of the tests? Or
maybe we should have a 'process map' for all tests for extensions that
are build by default? Again, suggestions welcome.

REDIRECTTEST still needs to be implemented, I understand how it works
and this isn't (afaict) a major issue.

Testing. I'm able to do basic testing on Mac OSX and Linux. I really
need access to an 8 way Linux system, or someone who has access and
would be interested in looking at performance? Any volunteers? This is
probably the most interesting part of the project :-)

Windows. I'm not in a position to do anything much with Windows
except some very basic checks to make sure that the sequential version
runs. The parallel code won't work because we used pcntl(), however I
know that Stefan and George were keen to design the code so that a
Windows solution could be implemented if anyone thought of one. If
anyone wants to pick up this aspect I'd be happy to get them started.

Zoe

12 years ago by zoe slattery — view source

unread

Hi Nuno - did you ever get a chance to look at the parallel version of
run-tests? I had some free time to work on it last week so REDIRECT is
now implemented, I have tested it using ext/pdo_mysql and ext/pdo_sqlite.

I'm testing using about 8000 phpt tests and getting the same results as
run-tests. There is still work to do on performance but I think it might
at this stage be worth trying a bigger sample.

Zoe

Alright, thanks for the reply!
I'll try to have a look at the code and give it a try next weekend.

Nuno

-----Original Message----- From: zoe slattery Sent: Monday, May 21,
2012 8:26 AM
Subject: Re: [PHP-QA] Parallel run-tests

Hi Zoe,

Thanks for undertaking this project!

I just have a few concerns about this:

The speedup seems a bit low to me. Maybe with a higher number of
extensions enabled it will improve?..
I don't think it will improve with a higher number of extensions. It
may improve as a result of someone doing some work on the tests are
scheduled - but I wouldn't count on it. Looking at improving
performance is important and I wanted to get some confirmation that it
is faster, I think we have that level of confirmation now. However, if
I'm the only person working on this, further performance work will
come after complete implementation + debugging.

Is there any developer documentation available? If I wanted to do
a change or a bug fix today, how could I do it?
Lots - https://wiki.php.net/qa/runtests. I have just updated the
"Development" section which has instructions on how to build and test
the code. The 'Documentation' section has not been updated recently
but I think it is still valid.

Can it be packaged as a single drop-in file replacement for
run-tests.php? The deployment is very important to me. IMHO, the
optimal solution would be to have a drop-in replacement for the
current script, so that most developers wouldn't even notice the
difference.
Yes - totally agree. I think I experimented with packaging it as a
.phar (so it would be called run-tests.phar, not run-tests.php), but
it's so long ago that I can't remember. I have added it to the to-do
list (https://wiki.php.net/qa/runtests/todos).

From previous emails exchanged in this thread, it seems that this
new version requires a few extensions to run (gzip, soap??). This is
not acceptable. The script must be able to run with --disable-all. Of
course in that case the parallel version will fail.
No, it doesn't. That was a stupid bug and I fixed it last night :-)

Zoe

Nuno

-----Original Message----- From: zoe slattery
Sent: Thursday, May 17, 2012 1:00 PM
To: internals@lists.php.net ; PHP QA
Subject: [PHP-QA] Parallel run-tests

Hi

Over the past couple of weeks I have updated the parallel run-tests
(fixed a couple of minor bugs in the PHP code and the build.xml), it's
now almost at the point where I could go ahead and implement the last
pieces. Here is a summary and a few questions:

In rebasing the code the the dev't stream I found a number of tests
with non-standard sections. My code checks test case structure and
objects to anything non-standard, the current run-tests.php mainly
ignores this kind of thing. I fixed up about 15 of these tests (see
#62022) already - I'll fix the rest if there are no objections - I will
open another bug report first.

If there is agreement to use this code it would make sense to
replace the existing run-tests code with it, or rather, it would make
no sense to try and maintain both versions. The new code is OO PHP, it's
in http://git.php.net/repository/phpruntests.git, is there any problem
with it staying there long term and maybe copying a run-tests.phar into
the PHP source directory? I have no idea what the right answer is,
suggestions welcome.

I ran a couple of small tests on my dual core Mac yesterday. For a
standard set of tests, the parallel code ran in 207 seconds, sequential
in 293 seconds and the standard run-tests.php took 298 seconds. This is
an improvement but I suspect we could still do better by looking at the
scheduling algorithm.
At the moment it's very simple, we just assemble a list of directories
with tests in and hand them out to processors till everything is done.
Being able to handle tests that must be run in sequence (mysql, mysqli)
will mean making some changes to this. So, perhaps we give an explicit
list to p1 and let the scheduler distribute the rest of the tests? Or
maybe we should have a 'process map' for all tests for extensions that
are build by default? Again, suggestions welcome.

REDIRECTTEST still needs to be implemented, I understand how it works
and this isn't (afaict) a major issue.

Testing. I'm able to do basic testing on Mac OSX and Linux. I really
need access to an 8 way Linux system, or someone who has access and
would be interested in looking at performance? Any volunteers? This is
probably the most interesting part of the project :-)

Windows. I'm not in a position to do anything much with Windows
except some very basic checks to make sure that the sequential version
runs. The parallel code won't work because we used pcntl(), however I
know that Stefan and George were keen to design the code so that a
Windows solution could be implemented if anyone thought of one. If
anyone wants to pick up this aspect I'd be happy to get them started.

Zoe

12 years ago by Nuno Lopes — view source

unread

Hi,

Here you have a dump of a run of PHP_HEAD in the gcov machine (almost 13k
tests) without valgrind:
http://gcov.php.net/~nlopess/dump_PHP_HEAD_z4.txt

It was run with -z 4. However, the reported CPU usage is only 213%
(instead of ~400%).

As you can see in the dump, there are a few BORK'ed tests.

BTW, sometimes I get this error when launching run-tests.php
(non-deterministic):
Fatal error: Call to undefined method rtGroupResults::run() in
src/taskScheduler/rtTaskSchedulerFile.php on line 225

Nuno

----- Original Message -----

Hi Nuno - did you ever get a chance to look at the parallel version of
run-tests? I had some free time to work on it last week so REDIRECT is now
implemented, I have tested it using ext/pdo_mysql and ext/pdo_sqlite.

I'm testing using about 8000 phpt tests and getting the same results as
run-tests. There is still work to do on performance but I think it might
at this stage be worth trying a bigger sample.

Zoe

Alright, thanks for the reply!
I'll try to have a look at the code and give it a try next weekend.

Nuno

-----Original Message----- From: zoe slattery Sent: Monday, May 21, 2012
8:26 AM
Subject: Re: [PHP-QA] Parallel run-tests

Hi Zoe,

Thanks for undertaking this project!

I just have a few concerns about this:

The speedup seems a bit low to me. Maybe with a higher number of
extensions enabled it will improve?..
I don't think it will improve with a higher number of extensions. It
may improve as a result of someone doing some work on the tests are
scheduled - but I wouldn't count on it. Looking at improving performance
is important and I wanted to get some confirmation that it is faster, I
think we have that level of confirmation now. However, if I'm the only
person working on this, further performance work will come after
complete implementation + debugging.

Is there any developer documentation available? If I wanted to do a
change or a bug fix today, how could I do it?
Lots - https://wiki.php.net/qa/runtests. I have just updated the
"Development" section which has instructions on how to build and test the
code. The 'Documentation' section has not been updated recently but I
think it is still valid.

Can it be packaged as a single drop-in file replacement for
run-tests.php? The deployment is very important to me. IMHO, the
optimal solution would be to have a drop-in replacement for the current
script, so that most developers wouldn't even notice the difference.
Yes - totally agree. I think I experimented with packaging it as a .phar
(so it would be called run-tests.phar, not run-tests.php), but it's so
long ago that I can't remember. I have added it to the to-do list
(https://wiki.php.net/qa/runtests/todos).

From previous emails exchanged in this thread, it seems that this new
version requires a few extensions to run (gzip, soap??). This is not
acceptable. The script must be able to run with --disable-all. Of course
in that case the parallel version will fail.
No, it doesn't. That was a stupid bug and I fixed it last night :-)

Zoe

Nuno

-----Original Message----- From: zoe slattery
Sent: Thursday, May 17, 2012 1:00 PM
To: internals@lists.php.net ; PHP QA
Subject: [PHP-QA] Parallel run-tests

Hi

Over the past couple of weeks I have updated the parallel run-tests
(fixed a couple of minor bugs in the PHP code and the build.xml), it's
now almost at the point where I could go ahead and implement the last
pieces. Here is a summary and a few questions:

In rebasing the code the the dev't stream I found a number of tests
with non-standard sections. My code checks test case structure and
objects to anything non-standard, the current run-tests.php mainly
ignores this kind of thing. I fixed up about 15 of these tests (see
#62022) already - I'll fix the rest if there are no objections - I will
open another bug report first.

If there is agreement to use this code it would make sense to
replace the existing run-tests code with it, or rather, it would make
no sense to try and maintain both versions. The new code is OO PHP, it's
in http://git.php.net/repository/phpruntests.git, is there any problem
with it staying there long term and maybe copying a run-tests.phar into
the PHP source directory? I have no idea what the right answer is,
suggestions welcome.

I ran a couple of small tests on my dual core Mac yesterday. For a
standard set of tests, the parallel code ran in 207 seconds, sequential
in 293 seconds and the standard run-tests.php took 298 seconds. This is
an improvement but I suspect we could still do better by looking at the
scheduling algorithm.
At the moment it's very simple, we just assemble a list of directories
with tests in and hand them out to processors till everything is done.
Being able to handle tests that must be run in sequence (mysql, mysqli)
will mean making some changes to this. So, perhaps we give an explicit
list to p1 and let the scheduler distribute the rest of the tests? Or
maybe we should have a 'process map' for all tests for extensions that
are build by default? Again, suggestions welcome.

REDIRECTTEST still needs to be implemented, I understand how it works
and this isn't (afaict) a major issue.

Testing. I'm able to do basic testing on Mac OSX and Linux. I really
need access to an 8 way Linux system, or someone who has access and
would be interested in looking at performance? Any volunteers? This is
probably the most interesting part of the project :-)

Windows. I'm not in a position to do anything much with Windows
except some very basic checks to make sure that the sequential version
runs. The parallel code won't work because we used pcntl(), however I
know that Stefan and George were keen to design the code so that a
Windows solution could be implemented if anyone thought of one. If
anyone wants to pick up this aspect I'd be happy to get them started.

Zoe

12 years ago by zoe slattery — view source

unread

Hi Nuno

Hi,

Here you have a dump of a run of PHP_HEAD in the gcov machine (almost
13k tests) without valgrind:
http://gcov.php.net/~nlopess/dump_PHP_HEAD_z4.txt

It was run with -z 4. However, the reported CPU usage is only 213%
(instead of ~400%).
I spent a lot of time this weekend trying to figure this out, not
completely successfully. The main problem is the way that groups of
tests are allocated between processors. The allocation is done randomly
from a list of groups (directories of tests), so for example, on my two
way machine, if the "array" and the "file" tests get allocated to the
same processor then the whole run can take as long as long with two
processors as with one. The second processor just finishes its task list
and hangs around waiting for the first one to complete.

Ideally all the processors would do something like pop tasks from a
list, so they'd all be busy all the time. However, I think this might be
difficult (for me) to implement correctly (?). A simpler solution might
be to have an allocation map either hard coded or read from a
configuration file - the student who implemented the parallel code did
allow for this. Having observed the issue with 'array' and 'file' I was
going to try this next. Any thoughts on the right way to do this would
be helpful.

As you can see in the dump, there are a few BORK'ed tests.
BORK'ed tests don't worry me too much. This version BORKs on tests that
have 'unrecognised sections', the current implementation just ignores
them so there are a few tests (for example those which contain SERVER
test sections) which run-tests just silently does nothing with, I don't
really like that behaviour and think that BORKs are an improvement.

I'll have a look through your results and make sure that it's nothing
more than that.

BTW, sometimes I get this error when launching run-tests.php
(non-deterministic):
Fatal error: Call to undefined method rtGroupResults::run() in
src/taskScheduler/rtTaskSchedulerFile.php on line 225
That is probably a bug - sorry. I added that class at the weekend, I'll
look into it.

Nuno

----- Original Message -----

Hi Nuno - did you ever get a chance to look at the parallel version
of run-tests? I had some free time to work on it last week so
REDIRECT is now implemented, I have tested it using ext/pdo_mysql and
ext/pdo_sqlite.

I'm testing using about 8000 phpt tests and getting the same results
as run-tests. There is still work to do on performance but I think it
might at this stage be worth trying a bigger sample.

Zoe

Alright, thanks for the reply!
I'll try to have a look at the code and give it a try next weekend.

Nuno

-----Original Message----- From: zoe slattery Sent: Monday, May 21,
2012 8:26 AM
Subject: Re: [PHP-QA] Parallel run-tests

Hi Zoe,

Thanks for undertaking this project!

I just have a few concerns about this:

The speedup seems a bit low to me. Maybe with a higher number of
extensions enabled it will improve?..
I don't think it will improve with a higher number of extensions. It
may improve as a result of someone doing some work on the tests
are scheduled - but I wouldn't count on it. Looking at improving
performance is important and I wanted to get some confirmation that
it is faster, I think we have that level of confirmation now.
However, if I'm the only person working on this, further performance
work will come after complete implementation + debugging.

Is there any developer documentation available? If I wanted to
do a change or a bug fix today, how could I do it?
Lots - https://wiki.php.net/qa/runtests. I have just updated the
"Development" section which has instructions on how to build and
test the code. The 'Documentation' section has not been updated
recently but I think it is still valid.

Can it be packaged as a single drop-in file replacement for
run-tests.php? The deployment is very important to me. IMHO, the
optimal solution would be to have a drop-in replacement for the
current script, so that most developers wouldn't even notice the
difference.
Yes - totally agree. I think I experimented with packaging it as a
.phar (so it would be called run-tests.phar, not run-tests.php), but
it's so long ago that I can't remember. I have added it to the to-do
list (https://wiki.php.net/qa/runtests/todos).

From previous emails exchanged in this thread, it seems that this
new version requires a few extensions to run (gzip, soap??). This
is not acceptable. The script must be able to run with
--disable-all. Of course in that case the parallel version will fail.
No, it doesn't. That was a stupid bug and I fixed it last night :-)

Zoe

Nuno

-----Original Message----- From: zoe slattery
Sent: Thursday, May 17, 2012 1:00 PM
To: internals@lists.php.net ; PHP QA
Subject: [PHP-QA] Parallel run-tests

Hi

Over the past couple of weeks I have updated the parallel run-tests
(fixed a couple of minor bugs in the PHP code and the build.xml), it's
now almost at the point where I could go ahead and implement the last
pieces. Here is a summary and a few questions:

In rebasing the code the the dev't stream I found a number of tests
with non-standard sections. My code checks test case structure and
objects to anything non-standard, the current run-tests.php mainly
ignores this kind of thing. I fixed up about 15 of these tests (see
#62022) already - I'll fix the rest if there are no objections - I
will
open another bug report first.

If there is agreement to use this code it would make sense to
replace the existing run-tests code with it, or rather, it would
make
no sense to try and maintain both versions. The new code is OO PHP,
it's
in http://git.php.net/repository/phpruntests.git, is there any problem
with it staying there long term and maybe copying a run-tests.phar
into
the PHP source directory? I have no idea what the right answer is,
suggestions welcome.

I ran a couple of small tests on my dual core Mac yesterday. For a
standard set of tests, the parallel code ran in 207 seconds,
sequential
in 293 seconds and the standard run-tests.php took 298 seconds.
This is
an improvement but I suspect we could still do better by looking at
the
scheduling algorithm.
At the moment it's very simple, we just assemble a list of directories
with tests in and hand them out to processors till everything is done.
Being able to handle tests that must be run in sequence (mysql,
mysqli)
will mean making some changes to this. So, perhaps we give an explicit
list to p1 and let the scheduler distribute the rest of the tests? Or
maybe we should have a 'process map' for all tests for extensions that
are build by default? Again, suggestions welcome.

REDIRECTTEST still needs to be implemented, I understand how it
works
and this isn't (afaict) a major issue.

Testing. I'm able to do basic testing on Mac OSX and Linux. I
really
need access to an 8 way Linux system, or someone who has access and
would be interested in looking at performance? Any volunteers? This is
probably the most interesting part of the project :-)

Windows. I'm not in a position to do anything much with Windows
except some very basic checks to make sure that the sequential version
runs. The parallel code won't work because we used pcntl(), however I
know that Stefan and George were keen to design the code so that a
Windows solution could be implemented if anyone thought of one. If
anyone wants to pick up this aspect I'd be happy to get them started.

Zoe

12 years ago by zoe slattery — view source

unread

Nuno - just a PS to the last note. It is (mainly) the task allocation
across processors which means that running tests in parallel on a 4-way
machine is not 4 times as fast as running them in sequence.

Here are some results from a run on my 2-way Mac -
http://static.inky.ws/image/3257/image.jpg. The blocks of colour are
just representations of the time it takes a group to run - and Open
office is allocating the colors randomly so they don't have any
significance. I've annotated the chart to show which groups are taking a
long time....

The net is that P0 runs its half of the tasks and then just hangs about
waiting for P1 to finish :-/.

I have added a 'debug' flag to the code which will print information
about how tasks are allocated if anyone wants to try on a 4 or 8 way.
Given that there are not many groups that take a long time to run the
simplest thing seems to be to map these to specific processors - that's
easy enough and requires no difficult code. There are more elegant
solutions of course.

Zoe

Hi Nuno

Hi,

Here you have a dump of a run of PHP_HEAD in the gcov machine (almost
13k tests) without valgrind:
http://gcov.php.net/~nlopess/dump_PHP_HEAD_z4.txt

It was run with -z 4. However, the reported CPU usage is only 213%
(instead of ~400%).
I spent a lot of time this weekend trying to figure this out, not
completely successfully. The main problem is the way that groups of
tests are allocated between processors. The allocation is done
randomly from a list of groups (directories of tests), so for example,
on my two way machine, if the "array" and the "file" tests get
allocated to the same processor then the whole run can take as long as
long with two processors as with one. The second processor just
finishes its task list and hangs around waiting for the first one to
complete.

Ideally all the processors would do something like pop tasks from a
list, so they'd all be busy all the time. However, I think this might
be difficult (for me) to implement correctly (?). A simpler solution
might be to have an allocation map either hard coded or read from a
configuration file - the student who implemented the parallel code did
allow for this. Having observed the issue with 'array' and 'file' I
was going to try this next. Any thoughts on the right way to do this
would be helpful.

As you can see in the dump, there are a few BORK'ed tests.
BORK'ed tests don't worry me too much. This version BORKs on tests
that have 'unrecognised sections', the current implementation just
ignores them so there are a few tests (for example those which contain
SERVER test sections) which run-tests just silently does nothing with,
I don't really like that behaviour and think that BORKs are an
improvement.

I'll have a look through your results and make sure that it's nothing
more than that.

BTW, sometimes I get this error when launching run-tests.php
(non-deterministic):
Fatal error: Call to undefined method rtGroupResults::run() in
src/taskScheduler/rtTaskSchedulerFile.php on line 225
That is probably a bug - sorry. I added that class at the weekend,
I'll look into it.

Nuno

----- Original Message -----

Hi Nuno - did you ever get a chance to look at the parallel version
of run-tests? I had some free time to work on it last week so
REDIRECT is now implemented, I have tested it using ext/pdo_mysql
and ext/pdo_sqlite.

I'm testing using about 8000 phpt tests and getting the same results
as run-tests. There is still work to do on performance but I think
it might at this stage be worth trying a bigger sample.

Zoe

Alright, thanks for the reply!
I'll try to have a look at the code and give it a try next weekend.

Nuno

-----Original Message----- From: zoe slattery Sent: Monday, May 21,
2012 8:26 AM
Subject: Re: [PHP-QA] Parallel run-tests

Hi Zoe,

Thanks for undertaking this project!

I just have a few concerns about this:

The speedup seems a bit low to me. Maybe with a higher number of
extensions enabled it will improve?..
I don't think it will improve with a higher number of extensions.
It may improve as a result of someone doing some work on the
tests are scheduled - but I wouldn't count on it. Looking at
improving performance is important and I wanted to get some
confirmation that it is faster, I think we have that level of
confirmation now. However, if I'm the only person working on this,
further performance work will come after complete implementation +
debugging.

Is there any developer documentation available? If I wanted to
do a change or a bug fix today, how could I do it?
Lots - https://wiki.php.net/qa/runtests. I have just updated the
"Development" section which has instructions on how to build and
test the code. The 'Documentation' section has not been updated
recently but I think it is still valid.

Can it be packaged as a single drop-in file replacement for
run-tests.php? The deployment is very important to me. IMHO, the
optimal solution would be to have a drop-in replacement for the
current script, so that most developers wouldn't even notice the
difference.
Yes - totally agree. I think I experimented with packaging it as a
.phar (so it would be called run-tests.phar, not run-tests.php),
but it's so long ago that I can't remember. I have added it to the
to-do list (https://wiki.php.net/qa/runtests/todos).

From previous emails exchanged in this thread, it seems that
this new version requires a few extensions to run (gzip, soap??).
This is not acceptable. The script must be able to run with
--disable-all. Of course in that case the parallel version will fail.
No, it doesn't. That was a stupid bug and I fixed it last night :-)

Zoe

Nuno

-----Original Message----- From: zoe slattery
Sent: Thursday, May 17, 2012 1:00 PM
To: internals@lists.php.net ; PHP QA
Subject: [PHP-QA] Parallel run-tests

Hi

Over the past couple of weeks I have updated the parallel run-tests
(fixed a couple of minor bugs in the PHP code and the build.xml),
it's
now almost at the point where I could go ahead and implement the last
pieces. Here is a summary and a few questions:

In rebasing the code the the dev't stream I found a number of
tests
with non-standard sections. My code checks test case structure and
objects to anything non-standard, the current run-tests.php mainly
ignores this kind of thing. I fixed up about 15 of these tests (see
#62022) already - I'll fix the rest if there are no objections - I
will
open another bug report first.

If there is agreement to use this code it would make sense to
replace the existing run-tests code with it, or rather, it would
make
no sense to try and maintain both versions. The new code is OO
PHP, it's
in http://git.php.net/repository/phpruntests.git, is there any
problem
with it staying there long term and maybe copying a run-tests.phar
into
the PHP source directory? I have no idea what the right answer is,
suggestions welcome.

I ran a couple of small tests on my dual core Mac yesterday. For a
standard set of tests, the parallel code ran in 207 seconds,
sequential
in 293 seconds and the standard run-tests.php took 298 seconds.
This is
an improvement but I suspect we could still do better by looking
at the
scheduling algorithm.
At the moment it's very simple, we just assemble a list of
directories
with tests in and hand them out to processors till everything is
done.
Being able to handle tests that must be run in sequence (mysql,
mysqli)
will mean making some changes to this. So, perhaps we give an
explicit
list to p1 and let the scheduler distribute the rest of the tests? Or
maybe we should have a 'process map' for all tests for extensions
that
are build by default? Again, suggestions welcome.

REDIRECTTEST still needs to be implemented, I understand how it
works
and this isn't (afaict) a major issue.

Testing. I'm able to do basic testing on Mac OSX and Linux. I
really
need access to an 8 way Linux system, or someone who has access and
would be interested in looking at performance? Any volunteers?
This is
probably the most interesting part of the project :-)

Windows. I'm not in a position to do anything much with Windows
except some very basic checks to make sure that the sequential
version
runs. The parallel code won't work because we used pcntl(), however I
know that Stefan and George were keen to design the code so that a
Windows solution could be implemented if anyone thought of one. If
anyone wants to pick up this aspect I'd be happy to get them started.

Zoe

12 years ago by Pierre Joye — view source

unread

hi!

Nuno - just a PS to the last note. It is (mainly) the task allocation across
processors which means that running tests in parallel on a 4-way machine is
not 4 times as fast as running them in sequence.

Here are some results from a run on my 2-way Mac -
http://static.inky.ws/image/3257/image.jpg. The blocks of colour are just
representations of the time it takes a group to run - and Open office is
allocating the colors randomly so they don't have any significance. I've
annotated the chart to show which groups are taking a long time....

The net is that P0 runs its half of the tasks and then just hangs about
waiting for P1 to finish :-/.

I have added a 'debug' flag to the code which will print information about
how tasks are allocated if anyone wants to try on a 4 or 8 way. Given that
there are not many groups that take a long time to run the simplest thing
seems to be to map these to specific processors - that's easy enough and
requires no difficult code. There are more elegant solutions of course.

Have you considered doing it using a daemon and php as client (to
launch tasks and the tests)? Gearman or even a webserver could do a
better job without the hassle to deal with what you are experiencing
now. We do that using webservers in our labs to test all phpt and
various apps.

That being said, to multiply cores never bring a linear speed
increase, even less in this case.

Cheers,

Pierre

@pierrejoye | http://blog.thepimp.net | http://www.libgd.org