Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:47160
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: error (pb1.pair.com: domain iki.fi from 204.13.248.72 cause and error)
Message-ID: <4B9A4085.8020607@iki.fi>
Date: Fri, 12 Mar 2010 15:24:21 +0200
Reply-To: jani.taskinen@iki.fi
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.8) Gecko/20100301 Fedora/3.0.3-1.fc11 Thunderbird/3.0.3
MIME-Version: 1.0
To: Ulf Wendel <Ulf.Wendel@Sun.COM>
CC: Christopher Jones <christopher.jones@oracle.com>, 
 internals@lists.php.net
References: <4B9926E8.4080202@lerdorf.com> <4B995F83.1000605@prohost.org> <4B99639B.9030405@oracle.com> <4B996FC6.7040102@zend.com> <4B997381.1050300@oracle.com> <4B9A14FF.6000703@iki.fi> <4B9A2A20.8030405@sun.com>
In-Reply-To: <4B9A2A20.8030405@sun.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Subject: Re: [PHP-DEV] Tests repository
From: jani.taskinen@iki.fi (Jani Taskinen)

On 03/12/2010 01:48 PM, Ulf Wendel wrote:
> Jani Taskinen schrieb:
>> Having tests in multiple branches is PITA. Hasn't anyone considered
>> that the best way would be to move all tests into their own repository
>> (directory..whatever :) in SVN..? Considering they are supposed to be
>> used for testing against regressions and BC breaks, they should always
>> be runnable using any PHP version?

> What you save is the work of having to update multiple branches manually.

Not only update, don't forget the maintaining of them.

> What you risk is that not each and every test is prepared for being run
> with every version - although, maybe, in theory it should be. This is

It should not be theory for regression tests? If new release does not 
pass the old tests but the old versions still do, then it's quite likely 
the new version is buggy? Now we have different versions of same tests 
in each branch (in the worst cases) and thus the behaviour might really 
be different between them when it should not be.

> Also, you may end up shipping the same huge set of tests for every PHP
> version regardless if all tests you ship are compatible with that
> version. Again, some magic should solve that - practicalities.

It's still one package for all. Thus less work, less space..? :)

> There is one thing I fear, although it is desired to do. Many extensions
> link external libraries. If you throw all tests in one place and run all
> tests with all PHP versions, I strongly assume you'll get more reports
> on test failures. That is because the likeliness of someone out there
> running new tests designed for the latest version of an external library
> against an old library will increase.

This part I didn't quite understand..isn't this same issue with the 
current situation as well?

> versions. I am aware how much people love BC. But there's a point where
> keeping BC should be left as an exercise to those asking for BC.

The BC is our holy grail. And having been bitten by this myself 
sometimes in last 3 years, I rather be a bit anal about BC now than I 
was before. :)

--Jani