Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:47196
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: pass (pb1.pair.com: domain gmail.com designates 74.125.82.170 as permitted sender)
DomainKey-Status: bad
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type;
        b=BkqdfQBECSsVn4YXnJKFjJA49MSY6lZany0VLc4vPBjoq2pLGuOp7GgQpSWUD+GQct
         zq/a0qKOYqv4QH3YjgAowf1M1IrFMJNm8CSjga7yXB5Ye+TZtjb26ZX0eXT55zXJIbOP
         /6bArCAcjCsohPnhln0DzdkACxXE+QnmsDehs=
MIME-Version: 1.0
In-Reply-To: <4B9AC13B.401@sci.fi>
References: <4B9926E8.4080202@lerdorf.com> <4B9A848C.3080701@iki.fi>
	 <fe05d1541003121106g1e5f0251hbfccedf0e4ffac1f@mail.gmail.com>
	 <F920F09D-893F-43E0-A2D7-49D12410DBEF@gmail.com>
	 <4B9A93EE.7080403@zend.com>
	 <1418253B-84A3-4CA6-965B-A2501A5D432F@gmail.com>
	 <4B9AB840.1010209@zend.com>
	 <ce6b011c1003121413p4b9fbff9qa79af5ee0db0778@mail.gmail.com>
	 <4B9ABDCE.2070904@zend.com> <4B9AC13B.401@sci.fi>
Date: Fri, 12 Mar 2010 23:39:15 +0100
Message-ID: <fe05d1541003121439y60838573j5206d41c238d0431@mail.gmail.com>
To: jani.taskinen@iki.fi
Cc: Stanislav Malyshev <stas@zend.com>, internals@lists.php.net
Content-Type: text/plain; charset=ISO-8859-1
Subject: Re: [PHP-DEV] Tests repository
From: pierre.php@gmail.com (Pierre Joye)

On Fri, Mar 12, 2010 at 11:33 PM, Jani Taskinen <jani.taskinen@sci.fi> wrote:
> 13.3.2010 0:18, Stanislav Malyshev wrote:
>>
>> Hi!
>>
>>> There are going to be some technical challenges. Some (maybe a lot) of
>>> test
>>> will need updates or rewriting. run-tests.php may need more improvements
>>> than what is already planned. Knowing this, I would still rather update
>>> run-tests.php and fix the tests, then continue to applying tests to
>>> different branches of the code base.
>>
>> I still have yet to hear *how* these tests are supposed to be updated,
>> to work everywhere. It's very easy to say "oh, we'll just fix them" -
>> but how exactly you're going to fix them if the test is supposed to do
>> one thing in 5.2 and another in 5.3? Are you going to have 2 versions of
>
> What tests are you really talking about here? I thought we have regression
> tests in there which test that stuff does not change between versions. Such
> test AFAICT help us to keep stuff to work like it worked before and after
> some change somewhere in the related code. So there should not be any need
> for any updates given the tests aren't for some reason different between
> branches in which case they aren't really the same test anymore.
>
> Short version: if test works in 5.2 it also has to (!) work in 5.3.
> Otherwise the test is pointless.
>
> Can you define the case you're referring to here or are we actually talking
> about totally different thing?

Did you read our replies? I cited two kind of tests failing even on
the same version. There are also many new errors or behaviors changes
between 5.2 and 5.3, some errors are based on system error messages or
using the underlying libraries errors.

As I said already, it is possible but the way we test things have to
be rewamped. Especially the "variationxyz" kind of tests with dozen of
cases in one single phpt, including all kind of errors, system, php,
etc. That's not a small move, but a huge task which will take some
months to be done.


Cheers,
-- 
Pierre

@pierrejoye | http://blog.thepimp.net | http://www.libgd.org