Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:47156
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: error (pb1.pair.com: domain Sun.COM from 192.18.6.21 cause and error)
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: text/plain; CHARSET=US-ASCII; format=flowed
Date: Fri, 12 Mar 2010 12:48:48 +0100
In-reply-to: <4B9A14FF.6000703@iki.fi>
Sender: Ulf.Wendel@Sun.COM
To: jani.taskinen@iki.fi
Cc: Christopher Jones <christopher.jones@oracle.com>, internals@lists.php.net
Message-ID: <4B9A2A20.8030405@sun.com>
References: <4B9926E8.4080202@lerdorf.com> <4B995F83.1000605@prohost.org>
 <4B99639B.9030405@oracle.com> <4B996FC6.7040102@zend.com>
 <4B997381.1050300@oracle.com> <4B9A14FF.6000703@iki.fi>
User-Agent: Thunderbird 2.0.0.23 (Windows/20090812)
Subject: Re: [PHP-DEV] Tests repository
From: Ulf.Wendel@Sun.COM (Ulf Wendel)

Jani Taskinen schrieb:
> Having tests in multiple branches is PITA. Hasn't anyone considered that 
> the best way would be to move all tests into their own repository 
> (directory..whatever :) in SVN..? Considering they are supposed to be 
> used for testing against regressions and BC breaks, they should always 
> be runnable using any PHP version?

It is doable, likely desired but it is going to be a bit of work.

What you save is the work of having to update multiple branches manually.

What you risk is that not each and every test is prepared for being run 
with every version - although, maybe, in theory it should be. This is 
certainly something that can be solved by adding some more magic to 
run-tests.php, e.g. "==SKIP-VERSION==" as suggested by Hannes - 
practicalities.

Also, you may end up shipping the same huge set of tests for every PHP 
version regardless if all tests you ship are compatible with that 
version. Again, some magic should solve that - practicalities.

There is one thing I fear, although it is desired to do. Many extensions 
link external libraries. If you throw all tests in one place and run all 
tests with all PHP versions, I strongly assume you'll get more reports 
on test failures. That is because the likeliness of someone out there 
running new tests designed for the latest version of an external library 
against an old library will increase.

Yes, you want to know about any such incompatibility. Yes, it helps you 
making your software better.

But it may mean significantly more work in particular if you follow the 
common paragidma of "test ample must always be green at any time on 
every system".

What may cure the extra work problem is to be more strict on compatible 
versions. I am aware how much people love BC. But there's a point where 
keeping BC should be left as an exercise to those asking for BC.

All in all, I like the idea but I fear extra work.

Ulf