Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:59208
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: pass (pb1.pair.com: domain sugarcrm.com designates 173.203.6.139 as permitted sender)
Message-ID: <4F74DFF6.7010507@sugarcrm.com>
Date: Thu, 29 Mar 2012 15:19:34 -0700
Organization: SugarCRM
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:10.0.2) Gecko/20120216 Thunderbird/10.0.2
MIME-Version: 1.0
To: Alexey Shein <confik@gmail.com>
CC: PHP Internals <internals@lists.php.net>
References: <CAAAa2sh3AbFcWa42aGgBh4DnzOnShKjY=P=2y2k6VCK6ZzqVhg@mail.gmail.com>
In-Reply-To: <CAAAa2sh3AbFcWa42aGgBh4DnzOnShKjY=P=2y2k6VCK6ZzqVhg@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Subject: Re: [PHP-DEV] Change all XFAIL tests to FAIL
From: smalyshev@sugarcrm.com (Stas Malyshev)

Hi!

> I've got a suggestion about refactoring our tests suite. I'd like to
> remove XFAIL institution and mark all failing tests just as FAIL.
> XFAIL has a problem that it hides attention from failing tests
> depending on not yet fixed bugs (most important), not yet implemented
> features (less important).
> Failed tests should make pain. They should bug you every day until you
> go and fix them.

Please note that we were in that position and we moved from there. So to
move back, we need some argument about what's different this time from
the place we were a year ago.

> XFAILs serve now as a pain-killers, we've got about 50 of them in the
> repo, so devs (I assume) think this way: "It's failing, but it's
> EXPECTED to fail, so let's leave it as is".

Leaving it as is was happening anyway. It's not like we had crowds of
devs descending on test fails but ignoring xfails. Most of xfails were
fails and were sitting there ignored for years. So the difference was
"running constantly with 50 fails" or "having some xfails but detecting
new fails easily since we have no or very little fails normally".
The problem is exactly that there are no devs thinking like you imagine
them to think.

> from a file) and use that. Failing tests should not be hidden.

They are not hidden. But they were not being fixed when they were just
fails - only thing that happened is that we constantly run with tons of
fails, so it was impossible to distinguish situation of "everything is
fine" from "the build is FUBAR".

> functions or not. We could also introduce "Incomplete" state like it's
> done in PHPUnit for these tests.

So what's the difference between xfail and incomplete?
-- 
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227