Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:59304
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: pass (pb1.pair.com: domain gmail.com designates 209.85.213.170 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <CAAAa2sjdfdNOZoErM6K7=vY3Kjuk0P5eTw=UiDz0WbQyf3wxNw@mail.gmail.com>
References: <CAAAa2sh3AbFcWa42aGgBh4DnzOnShKjY=P=2y2k6VCK6ZzqVhg@mail.gmail.com>
	<4F75EA77.3010300@oracle.com>
	<CAAAa2siwqqG6LtUToiCubGykAqn8SdVLB+TNT1bgR3iACq-gaA@mail.gmail.com>
	<4F76B73B.2060005@sugarcrm.com>
	<CAAAa2si1g9KrU9EH47HwKFCYvZb4YS7nDWwB44R0EjvOPtS8fQ@mail.gmail.com>
	<1333222079.9243.3864.camel@guybrush>
	<CAAAa2shvvaFDXsZXUU-YL4z4JwC2va956uRcUt+bQMAso7zG9Q@mail.gmail.com>
	<CAAAa2sjdfdNOZoErM6K7=vY3Kjuk0P5eTw=UiDz0WbQyf3wxNw@mail.gmail.com>
Date: Sun, 1 Apr 2012 20:13:15 +0500
Message-ID: <CAAAa2sgZ7zr3UH9M2kPYrecbY-9=EOBwjrgSw0+3+x4V1OZj_w@mail.gmail.com>
To: Ferenc Kovacs <tyra3l@gmail.com>
Cc: Christopher Jones <christopher.jones@oracle.com>, 
	"internals@lists.php.net" <internals@lists.php.net>
Content-Type: multipart/alternative; boundary=20cf302d4c5c5212b404bc9f83a7
Subject: Change all XFAIL tests to FAIL
From: confik@gmail.com (Alexey Shein)

--20cf302d4c5c5212b404bc9f83a7
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

1 =D0=B0=D0=BF=D1=80=D0=B5=D0=BB=D1=8F 2012 =D0=B3. 14:27 =D0=BF=D0=BE=D0=
=BB=D1=8C=D0=B7=D0=BE=D0=B2=D0=B0=D1=82=D0=B5=D0=BB=D1=8C Ferenc Kovacs
<tyra3l@gmail.com<javascript:;>>
=D0=BD=D0=B0=D0=BF=D0=B8=D1=81=D0=B0=D0=BB:
> On Sun, Apr 1, 2012 at 9:39 AM, Christopher Jones <
> christopher.jones@oracle.com <javascript:;>> wrote:
>
>>
>>
>> On 3/31/12 2:38 PM, Alexey Shein wrote:
>>
>>> 1 =D0=B0=D0=BF=D1=80=D0=B5=D0=BB=D1=8F 2012 =D0=B3. 0:27 =D0=BF=D0=BE=
=D0=BB=D1=8C=D0=B7=D0=BE=D0=B2=D0=B0=D1=82=D0=B5=D0=BB=D1=8C Johannes Schl=
=C3=BCter
>>> <johannes@schlueters.de <javascript:;>>  =D0=BD=D0=B0=D0=BF=D0=B8=D1=81=
=D0=B0=D0=BB:
>>>
>>>> On Sat, 2012-03-31 at 13:21 +0500, Alexey Shein wrote:
>>>>
>>>>> By constantly publishing
>>>>> newsletter with failed / xfail bugs you're telling them "That's our
>>>>> current problems. Maybe you could help us with them". This way we
>>>>> could convert that discussing energy into some good patches.
>>>>>
>>>>
>>>> While many people will simply filter them out. At least that's my
>>>> experience with such automated mails in different projects ;-)
>>>>
>>>> johannes
>>>>
>>>
>>> Okay, let's find it out. I've created a poll here:
>>> https://wiki.php.net/xfail_**poll <https://wiki.php.net/xfail_poll>.
>>> Please, leave your voice. I'll close the poll in a week. Thank you.
>>>
>>>
>> Is there anything in the Jenkins work that makes this discussion
irrelevant
>> (or more relevant)? What other ways should we be running & reviewing tes=
t
>> failures?
>>
>>
> currently XFAILs are handled as passing tests.
> I'm working on a solution for having email notifications about new test
> failures.

Wow, that was great post :)

>  snip

> you can also see a single test result history via selecting a test and
> clicking on the History link (
>
http://ci.qa.php.net/job/php-src-5.4-matrix-build/651/architecture=3Dx86,os=
=3Dlinux-debian-6.0/testReport/php-src.ext.libxml/tests/004_phpt___libxml_s=
et_streams_context__/history/
?
> for
> example) here you can also see the status for that test in each build, so
> you can see when was that introduced or fixed.
> for example you can see that the libxml_set_streams_context() started
> failing after build number 641, where you can see the Changes mentions a
> libxml bugfix:
>
http://ci.qa.php.net/job/php-src-5.4-matrix-build/architecture=3Dx86,os=3Dl=
inux-debian-6.0/641/changes

The one problem I see here is that we keep only 100 last builds, so if bug
is old enough you can't know build/revision where it's introduced.
Maybe we should not keep in jenkins build php binary but only lightweight
stuff like junit logs, coverage reports (if we have them) and etc. This way
build size become smaller, so we can keep more of them.

--
Regards,
Shein Alexey



--=20
Regards,
Shein Alexey

--20cf302d4c5c5212b404bc9f83a7--