Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:59209
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: pass (pb1.pair.com: domain gmail.com designates 209.85.160.42 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <4F74DFF6.7010507@sugarcrm.com>
References: <CAAAa2sh3AbFcWa42aGgBh4DnzOnShKjY=P=2y2k6VCK6ZzqVhg@mail.gmail.com>
	<4F74DFF6.7010507@sugarcrm.com>
Date: Fri, 30 Mar 2012 01:15:43 +0200
Message-ID: <CAH-PCH68Q1XWssgS=mL_i5vnTvqd+UA=L_Y7TsEWthMWnx+7Gw@mail.gmail.com>
To: Stas Malyshev <smalyshev@sugarcrm.com>
Cc: Alexey Shein <confik@gmail.com>, PHP Internals <internals@lists.php.net>
Content-Type: multipart/alternative; boundary=e89a8ff1c3fc30de2504bc69e72a
Subject: Re: [PHP-DEV] Change all XFAIL tests to FAIL
From: tyra3l@gmail.com (Ferenc Kovacs)

--e89a8ff1c3fc30de2504bc69e72a
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

>
>
> Please note that we were in that position and we moved from there. So to
> move back, we need some argument about what's different this time from
> the place we were a year ago.
>
>
could you elaborate on this part? where were we a year ago?


> > XFAILs serve now as a pain-killers, we've got about 50 of them in the
> > repo, so devs (I assume) think this way: "It's failing, but it's
> > EXPECTED to fail, so let's leave it as is".
>
> Leaving it as is was happening anyway. It's not like we had crowds of
> devs descending on test fails but ignoring xfails. Most of xfails were
> fails and were sitting there ignored for years. So the difference was
> "running constantly with 50 fails" or "having some xfails but detecting
> new fails easily since we have no or very little fails normally".
> The problem is exactly that there are no devs thinking like you imagine
> them to think.
>
>
yeah, but as we did see, the current approach makes it very easy to "hide"
even the not so small issues (for example the bunch of date related XFAILS
which you personally asked multiple times to be fixed before the 5.4
release).
I think that in it's current form XFAIL hurts more than it helps.



> > from a file) and use that. Failing tests should not be hidden.
>
> They are not hidden. But they were not being fixed when they were just
> fails - only thing that happened is that we constantly run with tons of
> fails, so it was impossible to distinguish situation of "everything is
> fine" from "the build is FUBAR".
>

yeah, and the exact same thing happened with the 5.3.7/CVE-2011-3189, even
though that we have XFAILs.
I think that eliminating the failing tests and making the fails noisy would
be a better approach.
I think that those spontaneous  "TestFests" initiated by Rasmus really
helped, and now that we are on git/github, there would be even
greater audience.


>
> > functions or not. We could also introduce "Incomplete" state like it's
> > done in PHPUnit for these tests.
>
> So what's the difference between xfail and incomplete?
>

http://www.phpunit.de/manual/3.2/en/incomplete-and-skipped-tests.html
currently the XFAIL can mean that either the test is incomplete, or the
functionality is missing/the code is broken, but we are expecting that for
some reason.
killing the XFAIL and adding the Incomplete test feature would allow the
first, but the second should be a failing test.

--=20
Ferenc Kov=C3=A1cs
@Tyr43l - http://tyrael.hu

--e89a8ff1c3fc30de2504bc69e72a--