Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:59163
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: error (pb1.pair.com: domain ez.no from 209.85.213.170 cause and error)
MIME-Version: 1.0
In-Reply-To: <1327019609-sup-8204@fewbar.com>
References: <4F189F5F.20109@sugarcrm.com> <4F18A8C5.9020301@phpgangsta.de>
 <4F18B07C.2010402@sugarcrm.com> <1327019609-sup-8204@fewbar.com>
Date: Mon, 26 Mar 2012 13:44:53 +0200
Message-ID: <CAGMw7vEjjp8Cn5COM331OhxS_ggm_YEGpjKTE6Ep0rkFT_ArMw@mail.gmail.com>
To: Clint Byrum <clint@ubuntu.com>
Cc: internals <internals@lists.php.net>
Content-Type: multipart/alternative; boundary=047d7b10d17b4c260104bc23e83b
Subject: Re: [PHP-DEV] 5.4.0 rc6 and release
From: ar@ez.no (=?UTF-8?B?QW5kcsOpIFLDuG1ja2U=?=)

--047d7b10d17b4c260104bc23e83b
Content-Type: text/plain; charset=UTF-8

On Fri, Jan 20, 2012 at 1:43 AM, Clint Byrum <clint@ubuntu.com> wrote:

> Excerpts from Stas Malyshev's message of Thu Jan 19 16:08:28 -0800 2012:
> > Hi!
> >
> > > - According to this website there are still 94 test failures in 5.4 .
> > > Can you confirm all of them are minor problems?
> > > http://gcov.php.net/viewer.php?version=PHP_5_4
> >
> > Most of them appear so, I'll go through them again to be sure and
> > encourage others to do so too and raise red flags if somebody sees
> > something bad there.
> > Unfortunately, some tests are environment-dependent or otherwise have
> > subtle dependencies or structure that make them work on one system and
> > fail on another not because of the bug in PHP but because of the test
> > itself. So, I have 0 fails on my Linux build but 6 fails on my Mac
> > build. Other times some systems may not support some capability, use old
> > version of the library, etc. and the test may not account for that.
>
> These tests should be skipped or marked as XFAIL on platforms they are
> known to fail on. Better to have no test than one that cannot be relied
> upon. All supported platforms should pass with 0 fails. These intentional
> skips should have open bugs that are documented in the test code so that
> a developer can find out why this test was disabled when trying to make
> a change covered by the test.
>
> >
> > I do not think it is practical to postpone release until we solve all of
> > such problems, since this being volunteer-driven open-source project
> > this means not having any release schedule at all. I prefer having the
> > schedule even if that means we'd have to release with some known
> > deficiencies.
> >
>
> Its pretty bad actually. For all of PHP's success, this is something that
> continues to baffle me, and many others I have talked to who are charged
> with measuring quality and with patching systems in a timely manner. How
> better to document unreliable tests than to skip them with something like
> "SKIPPED - known to fail on Mac".
>
> Its precisely this unreliability that forced me to take a conservative
> approach for Ubuntu 12.04 and recommend to the community that we ship
> 5.3.9 instead of 5.4.0. I would much rather have the new stuff in, but
> even if all the tests pass on the machine we run the test suite on,
> how can we be sure they won't fail in another time zone, or in some
> other strange configuration?
>


Given that 12.04 beta2 will be out on Thursday, and the unit tests where
fixed before shipping 5.4 (I naively assume), is this in or out?
ref: https://blueprints.launchpad.net/ubuntu/+spec/servercloud-p-php54

With beta freeze on march 22 I guess the "mistake" of bundeling 5.3 instead
of 5.4 is already made, and we will have to live with that for the next 2
years for prod, and the next 7 months for dev.





>
> > > - There was this problem with 5.3.7 and the crypt() bug. Has there been
> > > some improvement in the process of handling test failures? For example
> > > mark expected failures as expected failures, and fix the tests or the
> > > code? Or are the failing tests "stable" since month and all of them are
> > > minor problems?
> >
> > Yes, there was work done on these. Most of those were fixed, but few
> > still remain, especially across various environments (i.e. test may be
> > fine on some but not others). I of course am all for fixing that further
> > and welcome any help on that.
> >
> > > - There have been 319 unique failed tests in RC5, reported by user
> > > tests. Is someone looking into them and trying to classify and/or fix
> them?
> > > http://qa.php.net/reports/
> >
> > Non-reproducible failures usually mean the problem is with the test
> > itself, or with the difference of expectations in the test and local
> > environment, not with PHP. It may still be PHP problem, of course, so
> > the person running the test should check it out and submit a bug if
> > appropriate and if it's bad enough, also send a message to this list.
> >
> > > All in all the number of test failures still feels very high, I would
> be
> > > interested in your opinion. Is this "normal" for big projects like
> this?
> >
> > I do think it should be reduced, however if the choice is between
> > waiting forever and have release with some bugs, I think it is practical
> > to choose the latter. Of course, if we discover a major problem that
> > makes PHP unusable or seriously impacts many PHP users, it will be dealt
> > with immediately, and had been so in the past, but otherwise we have to
> > work within the realities of a big project with limited resources and
> > realize that while we strive for 0 bugs in every release, it may never
> > be possible.
>
> All software will have bugs. The test suite, however, should reflect
> the bits of code that you know work reliably... not the bits of code
> you know work most of the time.
>
> The fact that its all being running regularly is a fantastic improvement.
> I'd like to see a commitment to getting 100% pass/xfail/skip for every
> release/tested environment in future releases though.
>
> --
> PHP Internals - PHP Runtime Development Mailing List
> To unsubscribe, visit: http://www.php.net/unsub.php
>
>

--047d7b10d17b4c260104bc23e83b--