Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:59207
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: pass (pb1.pair.com: domain gmail.com designates 209.85.160.170 as permitted sender)
MIME-Version: 1.0
Date: Fri, 30 Mar 2012 03:00:14 +0500
Message-ID: <CAAAa2sh3AbFcWa42aGgBh4DnzOnShKjY=P=2y2k6VCK6ZzqVhg@mail.gmail.com>
To: PHP Internals <internals@lists.php.net>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Subject: Change all XFAIL tests to FAIL
From: confik@gmail.com (Alexey Shein)

Hi, internals!

I've got a suggestion about refactoring our tests suite. I'd like to
remove XFAIL institution and mark all failing tests just as FAIL.
XFAIL has a problem that it hides attention from failing tests
depending on not yet fixed bugs (most important), not yet implemented
features (less important).
Failed tests should make pain. They should bug you every day until you
go and fix them.
XFAILs serve now as a pain-killers, we've got about 50 of them in the
repo, so devs (I assume) think this way: "It's failing, but it's
EXPECTED to fail, so let's leave it as is".
That's wrong thinking. Either tests are correct and if they fail you
should fix the code and leave them failed until the code is fixed, or,
if the tests are incorrect - fix the tests or remove them completely.

Reasons introducing XFAILs were described in this article:
http://zoomsplatter.blogspot.com/2008/06/why-xfail.html
I'll quote some thoughts from there:
> The intention of XFAIL is to help people working on developing PHP. Consi=
der first the situation where you (as a PHP implementer) are working throug=
h a set of failing tests. You do some analysis on one test but you can't fi=
x the implementation until something else is fixed =E2=80=93 however =E2=80=
=93 you don't want to lose the analysis and it might be some time before yo=
u can get back to the failing test. In this case I think it's reasonable to=
 add an XFAIL section with a brief description of the analysis. This takes =
the test out of the list of reported failures making it easier for you to s=
ee what is really on your priority list but still leaving the test as a fai=
ling test.

If you need something else to be fixed, leave your failing test, it
would annoy everybody, so something else will be fixed faster. If you
"don't want to lose analysis" you can make a list of tests you need
with (run-tests.php has -l option which can read/write test filenames
from a file) and use that. Failing tests should not be hidden.

> The second place that I can see that XFAIL might be useful is when a grou=
p of people are working on the same development project. Essentially one pe=
rson on the project finds a missing feature or capability but it isn't some=
thing they can add immediately, or maybe another person has agreed to imple=
ment it. A really good way to document the omission is to write a test case=
 which is expected to fail but which will pass if the feature is implemente=
d. This assumes that there is general agreement that implementation is a go=
od idea and needs to be done at some stage.

These "feature tests" can be put in a separate branch to not pollute
main release branches until the feature is ready. These usually called
"acceptance tests" so they mark if a feature fully implements its
functions or not. We could also introduce "Incomplete" state like it's
done in PHPUnit for these tests.

What do you think?

--=20
Regards,
Shein Alexey