Newsgroups: php.internals,php.qa
Path: news.php.net
Xref: news.php.net php.internals:63405 php.qa:66757
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: unknown (pb1.pair.com: domain php.net does not designate 212.55.154.22 as permitted sender)
Message-ID: <038CBF95DEC5479DA771E23528A33FB3@pc07654>
To: "zoe slattery" <aparachic@gmail.com>
Cc: <internals@lists.php.net>,
	"PHP QA" <php-qa@lists.php.net>
References: <4FB4E844.2070305@gmail.com> <9AF025709ED6452F9E5A91834F82F788@PC07655> <4FB9EE1E.3050701@gmail.com> <BCC3B4980516429D9BF61D910B09548D@PC07655> <506E9442.6050501@gmail.com>
In-Reply-To: <506E9442.6050501@gmail.com>
Date: Sun, 14 Oct 2012 21:38:01 +0100
MIME-Version: 1.0
Content-Type: text/plain;
	format=flowed;
	charset="iso-8859-1";
	reply-type=response
Content-Transfer-Encoding: 7bit
Subject: Re: [PHP-QA] Parallel run-tests
From: nlopess@php.net ("Nuno Lopes")

Hi,

Here you have a dump of a run of PHP_HEAD in the gcov machine (almost 13k 
tests) without valgrind:
http://gcov.php.net/~nlopess/dump_PHP_HEAD_z4.txt

It was run with -z 4.  However, the reported CPU usage is only 213% 
(instead of ~400%).

As you can see in the dump, there are a few BORK'ed tests.

BTW, sometimes I get this error when launching run-tests.php 
(non-deterministic):
Fatal error: Call to undefined method rtGroupResults::run() in 
src/taskScheduler/rtTaskSchedulerFile.php on line 225

Nuno


----- Original Message -----
> Hi Nuno - did you ever get a chance to look at the parallel version of 
> run-tests? I had some free time to work on it last week so REDIRECT is now 
> implemented, I have tested it using ext/pdo_mysql and ext/pdo_sqlite.
>
> I'm testing using about 8000 phpt tests and getting the same results as 
> run-tests. There is still work to do on performance but I think it might 
> at this stage be worth trying a bigger sample.
>
> Zoe
>> Alright, thanks for the reply!
>> I'll try to have a look at the code and give it a try next weekend.
>>
>> Nuno
>>
>>
>> -----Original Message----- From: zoe slattery Sent: Monday, May 21, 2012 
>> 8:26 AM
>> Subject: Re: [PHP-QA] Parallel run-tests
>> On 21/05/2012 06:45, Nuno Lopes wrote:
>>> Hi Zoe,
>>>
>>> Thanks for undertaking this project!
>>>
>>> I just have a few concerns about this:
>>> - The speedup seems a bit low to me. Maybe with a higher number of 
>>> extensions enabled it will improve?..
>> I don't think it will improve with a higher number of extensions. It 
>> _may_ improve as a result of someone doing some work on the tests are 
>> scheduled - but I wouldn't count on it. Looking at improving performance 
>> is important and I wanted to get some confirmation that it is faster, I 
>> think we have that level of confirmation now. However, if I'm the only 
>> person working on this, further performance work will  come after 
>> complete implementation + debugging.
>>> - Is there any developer documentation available?  If I wanted to do a 
>>> change or a bug fix today, how could I do it?
>> Lots - https://wiki.php.net/qa/runtests. I have just updated the 
>> "Development" section which has instructions on how to build and test the 
>> code. The 'Documentation' section has not been updated recently but I 
>> think it is still valid.
>>
>>> - Can it be packaged as a single drop-in file replacement for 
>>> run-tests.php? The deployment is very important to me.  IMHO, the 
>>> optimal solution would be to have a drop-in replacement for the current 
>>> script, so that most developers wouldn't even notice the difference.
>> Yes - totally agree. I think I experimented with packaging it as a .phar 
>> (so it would be called run-tests.phar, not run-tests.php), but it's so 
>> long ago that I can't remember. I have added it to the to-do list 
>> (https://wiki.php.net/qa/runtests/todos).
>>> - From previous emails exchanged in this thread, it seems that this new 
>>> version requires a few extensions to run (gzip, soap??). This is not 
>>> acceptable. The script must be able to run with --disable-all. Of course 
>>> in that case the parallel version will fail.
>> No, it doesn't. That was a stupid bug and I fixed it last night :-)
>>
>> Zoe
>>>
>>> Nuno
>>>
>>>
>>> -----Original Message----- From: zoe slattery
>>> Sent: Thursday, May 17, 2012 1:00 PM
>>> To: internals@lists.php.net ; PHP QA
>>> Subject: [PHP-QA] Parallel run-tests
>>>
>>> Hi
>>>
>>> Over the past couple of weeks I have updated the parallel run-tests
>>> (fixed a couple of minor bugs in the PHP code and the build.xml), it's
>>> now almost at the point where I could go ahead and implement the last
>>> pieces.  Here is a summary and a few questions:
>>>
>>> 1. In rebasing the code the the dev't stream I found a number of tests
>>> with non-standard sections. My code checks test case structure and
>>> objects to anything non-standard, the current run-tests.php mainly
>>> ignores this kind of thing. I fixed up about 15 of these tests (see
>>> #62022) already - I'll fix the rest if there are no objections - I will
>>> open another bug report first.
>>>
>>> 2. If there is agreement to use this code it would make sense to
>>> replace  the existing run-tests code with it, or rather,  it would make
>>> no sense to try and maintain both versions. The new code is OO PHP, it's
>>> in http://git.php.net/repository/phpruntests.git, is there any problem
>>> with it staying there long term and maybe copying a run-tests.phar into
>>> the PHP source directory? I have no idea what the right answer is,
>>> suggestions welcome.
>>>
>>> 3. I ran a couple of small tests on my dual core Mac yesterday. For a
>>> standard set of tests, the parallel code ran in 207 seconds, sequential
>>> in 293 seconds and the standard run-tests.php took 298 seconds. This is
>>> an improvement but I suspect we could still do better by looking at the
>>> scheduling algorithm.
>>> At the moment it's very simple, we just assemble a list of directories
>>> with tests in and hand them out to processors till everything is done.
>>> Being able to handle tests that must be run in sequence (mysql, mysqli)
>>> will mean making some changes to this. So, perhaps we give an explicit
>>> list to p1 and let the scheduler distribute the rest of the tests? Or
>>> maybe we should have a 'process map' for all tests for extensions that
>>> are build by default? Again, suggestions welcome.
>>>
>>> 4. REDIRECTTEST still needs to be implemented, I understand how it works
>>> and this isn't (afaict) a major issue.
>>>
>>> 5. Testing. I'm able to do basic testing on Mac OSX  and Linux. I really
>>> need access to an 8 way Linux system, or someone who has access and
>>> would be interested in looking at performance? Any volunteers? This is
>>> probably the most interesting part of the project :-)
>>>
>>> 6. Windows. I'm not in a position to do anything much with Windows
>>> except some very basic checks to make sure that the sequential version
>>> runs. The parallel code won't work because we used pcntl(), however I
>>> know that Stefan and George were keen to design the code so that a
>>> Windows solution could be implemented if anyone thought of one. If
>>> anyone wants to pick up this aspect I'd be happy to get them started.
>>>
>>> Zoe