Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:59298
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: pass (pb1.pair.com: domain gmail.com designates 209.85.160.42 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <4F780637.7090702@oracle.com>
References: <CAAAa2sh3AbFcWa42aGgBh4DnzOnShKjY=P=2y2k6VCK6ZzqVhg@mail.gmail.com>
	<4F75EA77.3010300@oracle.com>
	<CAAAa2siwqqG6LtUToiCubGykAqn8SdVLB+TNT1bgR3iACq-gaA@mail.gmail.com>
	<4F76B73B.2060005@sugarcrm.com>
	<CAAAa2si1g9KrU9EH47HwKFCYvZb4YS7nDWwB44R0EjvOPtS8fQ@mail.gmail.com>
	<1333222079.9243.3864.camel@guybrush>
	<CAAAa2shvvaFDXsZXUU-YL4z4JwC2va956uRcUt+bQMAso7zG9Q@mail.gmail.com>
	<4F780637.7090702@oracle.com>
Date: Sun, 1 Apr 2012 11:27:23 +0200
Message-ID: <CAH-PCH5-Kexa5LVe8AViJjUZybmkD=SG2_5++YsgQh_WZ5nJOw@mail.gmail.com>
To: Christopher Jones <christopher.jones@oracle.com>
Cc: "internals@lists.php.net" <internals@lists.php.net>
Content-Type: multipart/alternative; boundary=047d7b15ae3b6a96b804bc9aae64
Subject: Re: [PHP-DEV] Change all XFAIL tests to FAIL
From: tyra3l@gmail.com (Ferenc Kovacs)

--047d7b15ae3b6a96b804bc9aae64
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On Sun, Apr 1, 2012 at 9:39 AM, Christopher Jones <
christopher.jones@oracle.com> wrote:

>
>
> On 3/31/12 2:38 PM, Alexey Shein wrote:
>
>> 1 =D0=B0=D0=BF=D1=80=D0=B5=D0=BB=D1=8F 2012 =D0=B3. 0:27 =D0=BF=D0=BE=D0=
=BB=D1=8C=D0=B7=D0=BE=D0=B2=D0=B0=D1=82=D0=B5=D0=BB=D1=8C Johannes Schl=C3=
=BCter
>> <johannes@schlueters.de>  =D0=BD=D0=B0=D0=BF=D0=B8=D1=81=D0=B0=D0=BB:
>>
>>> On Sat, 2012-03-31 at 13:21 +0500, Alexey Shein wrote:
>>>
>>>> By constantly publishing
>>>> newsletter with failed / xfail bugs you're telling them "That's our
>>>> current problems. Maybe you could help us with them". This way we
>>>> could convert that discussing energy into some good patches.
>>>>
>>>
>>> While many people will simply filter them out. At least that's my
>>> experience with such automated mails in different projects ;-)
>>>
>>> johannes
>>>
>>
>> Okay, let's find it out. I've created a poll here:
>> https://wiki.php.net/xfail_**poll <https://wiki.php.net/xfail_poll>.
>> Please, leave your voice. I'll close the poll in a week. Thank you.
>>
>>
> Is there anything in the Jenkins work that makes this discussion irreleva=
nt
> (or more relevant)? What other ways should we be running & reviewing test
> failures?
>
>
currently XFAILs are handled as passing tests.
I'm working on a solution for having email notifications about new test
failures.
There are 3 option, which I'm evaluating:
1, one can set thresholds if using the xunit plugin (
https://wiki.jenkins-ci.org/display/JENKINS/xUnit+Plugin) instead of the
built-in junit plugin, which allows to fail the build if a new test failure
is introduced.
2, the email-ext plugin allows you to set trigger for sending an email if a
"regression"(new test failure), and also allows sending email for
"improvement" (which is triggered when a previously failing test starts
passing.
3, creating a custom post-build action, where we compare the current
build's test result with the previous reports, and send out emails/mark the
build failed if a new test failure is introduced.

The problem with 1 is that it doesn't likes our junit.xml. First I debuged
out that it doesn't support nested test suites, but that should be fixed
with https://issues.jenkins-ci.org/browse/JENKINS-8460 I upgraded the
plugin, but still no joy, I have to debug further.
The problem with 2 is that the trigger only happens based on comparing the
previous and current build's test result numbers. So if you introduce a new
test failure and also fix a previously failing test, it won't fire the
"regression" trigger.
3. seems to be the best solution, but requires the most work on our part.

When we implement this (using one method, or another) we could change so
that the XFAILs are handled from the jenkins POV as test failures, as they
will be listed as such, but you still can see what are the ones failing
since ages and what are the new ones.
Jenkins provides the following informations regarding test failures:

 you can see the test failures introduced with a particular build through
the build Status page (
http://ci.qa.php.net/job/php-src-5.3-matrix-build/architecture=3Dx86,os=3Dl=
inux-debian-6.0/452/
for
example)
here you have a link to the Test Result page, plus the number of the
failing tests, plus the sum of the test changes with that build (+-)
under that line, Jenkins list the test failures introduced with that build
under that you have "Show all failed tests >>>" which will expand the test
list to include all failing tests, not just the new ones.

jenkins also provides us with a Test Result Trend on the project page (
http://ci.qa.php.net/job/php-src-5.3-matrix-build/ for example), where you
can toggle between showing all tests or just the failing ones.
you can also see the history for a build for a config through the History
link (
http://ci.qa.php.net/job/php-src-5.4-matrix-build/architecture=3Dx86,os=3Dl=
inux-debian-6.0/651/testReport/history/?
for
example)
you can also see the total test results for a specific build per config (
http://ci.qa.php.net/job/php-src-5.4-matrix-build/architecture=3Dx86,os=3Dl=
inux-debian-6.0/651/testReport/junit/?)
or aggregated (
http://ci.qa.php.net/job/php-src-5.4-matrix-build/651/testReport/?)
you can also see a single test result history via selecting a test and
clicking on the History link (
http://ci.qa.php.net/job/php-src-5.4-matrix-build/651/architecture=3Dx86,os=
=3Dlinux-debian-6.0/testReport/php-src.ext.libxml/tests/004_phpt___libxml_s=
et_streams_context__/history/?
for
example) here you can also see the status for that test in each build, so
you can see when was that introduced or fixed.
for example you can see that the libxml_set_streams_context() started
failing after build number 641, where you can see the Changes mentions a
libxml bugfix:
http://ci.qa.php.net/job/php-src-5.4-matrix-build/architecture=3Dx86,os=3Dl=
inux-debian-6.0/641/changes


having the email notification in place could make those kind of changes
more visible.

--=20
Ferenc Kov=C3=A1cs
@Tyr43l - http://tyrael.hu

--047d7b15ae3b6a96b804bc9aae64--