Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:57440
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: error (pb1.pair.com: domain fewbar.com from 65.98.207.160 cause and error)
Content-Type: text/plain; charset=UTF-8
To: internals <internals@lists.php.net>
In-reply-to: <4F18B07C.2010402@sugarcrm.com>
References: <4F189F5F.20109@sugarcrm.com> <4F18A8C5.9020301@phpgangsta.de> <4F18B07C.2010402@sugarcrm.com>
Date: Thu, 19 Jan 2012 16:43:22 -0800
Message-ID: <1327019609-sup-8204@fewbar.com>
User-Agent: Sup/git
Content-Transfer-Encoding: 8bit
Subject: Re: [PHP-DEV] 5.4.0 rc6 and release
From: clint@ubuntu.com (Clint Byrum)

Excerpts from Stas Malyshev's message of Thu Jan 19 16:08:28 -0800 2012:
> Hi!
> 
> > - According to this website there are still 94 test failures in 5.4 .
> > Can you confirm all of them are minor problems?
> > http://gcov.php.net/viewer.php?version=PHP_5_4
> 
> Most of them appear so, I'll go through them again to be sure and 
> encourage others to do so too and raise red flags if somebody sees 
> something bad there.
> Unfortunately, some tests are environment-dependent or otherwise have 
> subtle dependencies or structure that make them work on one system and 
> fail on another not because of the bug in PHP but because of the test 
> itself. So, I have 0 fails on my Linux build but 6 fails on my Mac 
> build. Other times some systems may not support some capability, use old 
> version of the library, etc. and the test may not account for that.

These tests should be skipped or marked as XFAIL on platforms they are
known to fail on. Better to have no test than one that cannot be relied
upon. All supported platforms should pass with 0 fails. These intentional
skips should have open bugs that are documented in the test code so that
a developer can find out why this test was disabled when trying to make
a change covered by the test.

> 
> I do not think it is practical to postpone release until we solve all of 
> such problems, since this being volunteer-driven open-source project 
> this means not having any release schedule at all. I prefer having the 
> schedule even if that means we'd have to release with some known 
> deficiencies.
> 

Its pretty bad actually. For all of PHP's success, this is something that
continues to baffle me, and many others I have talked to who are charged
with measuring quality and with patching systems in a timely manner. How
better to document unreliable tests than to skip them with something like
"SKIPPED - known to fail on Mac".

Its precisely this unreliability that forced me to take a conservative
approach for Ubuntu 12.04 and recommend to the community that we ship
5.3.9 instead of 5.4.0. I would much rather have the new stuff in, but
even if all the tests pass on the machine we run the test suite on,
how can we be sure they won't fail in another time zone, or in some
other strange configuration?

> > - There was this problem with 5.3.7 and the crypt() bug. Has there been
> > some improvement in the process of handling test failures? For example
> > mark expected failures as expected failures, and fix the tests or the
> > code? Or are the failing tests "stable" since month and all of them are
> > minor problems?
> 
> Yes, there was work done on these. Most of those were fixed, but few 
> still remain, especially across various environments (i.e. test may be 
> fine on some but not others). I of course am all for fixing that further 
> and welcome any help on that.
> 
> > - There have been 319 unique failed tests in RC5, reported by user
> > tests. Is someone looking into them and trying to classify and/or fix them?
> > http://qa.php.net/reports/
> 
> Non-reproducible failures usually mean the problem is with the test 
> itself, or with the difference of expectations in the test and local 
> environment, not with PHP. It may still be PHP problem, of course, so 
> the person running the test should check it out and submit a bug if 
> appropriate and if it's bad enough, also send a message to this list.
> 
> > All in all the number of test failures still feels very high, I would be
> > interested in your opinion. Is this "normal" for big projects like this?
> 
> I do think it should be reduced, however if the choice is between 
> waiting forever and have release with some bugs, I think it is practical 
> to choose the latter. Of course, if we discover a major problem that 
> makes PHP unusable or seriously impacts many PHP users, it will be dealt 
> with immediately, and had been so in the past, but otherwise we have to 
> work within the realities of a big project with limited resources and 
> realize that while we strive for 0 bugs in every release, it may never 
> be possible.

All software will have bugs. The test suite, however, should reflect
the bits of code that you know work reliably... not the bits of code
you know work most of the time.

The fact that its all being running regularly is a fantastic improvement.
I'd like to see a commitment to getting 100% pass/xfail/skip for every
release/tested environment in future releases though.