Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:59240
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: error (pb1.pair.com: domain oracle.com from 141.146.126.227 cause and error)
Message-ID: <4F75EA77.3010300@oracle.com>
Date: Fri, 30 Mar 2012 10:16:39 -0700
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:11.0) Gecko/20120327 Thunderbird/11.0.1
MIME-Version: 1.0
To: internals@lists.php.net
References: <CAAAa2sh3AbFcWa42aGgBh4DnzOnShKjY=P=2y2k6VCK6ZzqVhg@mail.gmail.com>
In-Reply-To: <CAAAa2sh3AbFcWa42aGgBh4DnzOnShKjY=P=2y2k6VCK6ZzqVhg@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Subject: Re: [PHP-DEV] Change all XFAIL tests to FAIL
From: christopher.jones@oracle.com (Christopher Jones)



On 3/29/12 3:00 PM, Alexey Shein wrote:
> Hi, internals!
>
> I've got a suggestion about refactoring our tests suite. I'd like to
> remove XFAIL institution and mark all failing tests just as FAIL.
> XFAIL has a problem that it hides attention from failing tests
> depending on not yet fixed bugs (most important), not yet implemented
> features (less important).
> Failed tests should make pain. They should bug you every day until you
> go and fix them.
> XFAILs serve now as a pain-killers, we've got about 50 of them in the
> repo, so devs (I assume) think this way: "It's failing, but it's
> EXPECTED to fail, so let's leave it as is".
> That's wrong thinking. Either tests are correct and if they fail you
> should fix the code and leave them failed until the code is fixed, or,
> if the tests are incorrect - fix the tests or remove them completely.

The XFAIL mechanism reflects the reality of open source that not all
bugs are fixed.  We need a simple, low maintenance way to have a
'clean' testsuite shipped which exhibits minimal noise so that users
don't waste time investigating known failures.

XFAIL also allows end users to see when something has broken that used
to work.

If the system is being overused, feel free to call people out on it.
I don't think it should be used for unimplemented features long term.

XFAIL is a simple mechanism.  Anything different like moving tests to
a special 'failed' directory adds burden.  I don't belive we have
extra cycles for this, but would be happy to be proved wrong.

Chris

-- 
Email: christopher.jones@oracle.com
Tel:  +1 650 506 8630
Blog:  http://blogs.oracle.com/opal/