Hi!
We'll be announcing RC6 very soon, and we're planning the final release
for February 2nd. The two weeks in between are meant for testing and
finding out any critical issues that could prevent us from releasing 5.4.0.
If we have none of these, we will proceed with the release. To ensure
this release would not have any "surprises", I am asking again to hold
your commits to 5.4 branch unless you have a critical bug fix AND it is
approved by the RMs (myself and David).
Thanks for your help!
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227
Hi,
great to see progress!
I have some small questions, please don't feel offended, I just want to
know and understand:
- According to this website there are still 94 test failures in 5.4 .
Can you confirm all of them are minor problems?
http://gcov.php.net/viewer.php?version=PHP_5_4 - There was this problem with 5.3.7 and the
crypt()bug. Has there been
some improvement in the process of handling test failures? For example
mark expected failures as expected failures, and fix the tests or the
code? Or are the failing tests "stable" since month and all of them are
minor problems? - There have been 319 unique failed tests in RC5, reported by user
tests. Is someone looking into them and trying to classify and/or fix them?
http://qa.php.net/reports/
All in all the number of test failures still feels very high, I would be
interested in your opinion. Is this "normal" for big projects like this?
Again, please take these questions as normal questions, I don't want to
attack anybody, I'm just interested in the process and relatively new to
internal PHP development.
Thanks for your work on PHP!
Michael
Am 19.01.2012 23:55, schrieb Stas Malyshev:
Hi!
We'll be announcing RC6 very soon, and we're planning the final
release for February 2nd. The two weeks in between are meant for
testing and finding out any critical issues that could prevent us from
releasing 5.4.0.
If we have none of these, we will proceed with the release. To ensure
this release would not have any "surprises", I am asking again to hold
your commits to 5.4 branch unless you have a critical bug fix AND it
is approved by the RMs (myself and David).
Thanks for your help!
Hi!
- According to this website there are still 94 test failures in 5.4 .
Can you confirm all of them are minor problems?
http://gcov.php.net/viewer.php?version=PHP_5_4
Most of them appear so, I'll go through them again to be sure and
encourage others to do so too and raise red flags if somebody sees
something bad there.
Unfortunately, some tests are environment-dependent or otherwise have
subtle dependencies or structure that make them work on one system and
fail on another not because of the bug in PHP but because of the test
itself. So, I have 0 fails on my Linux build but 6 fails on my Mac
build. Other times some systems may not support some capability, use old
version of the library, etc. and the test may not account for that.
I do not think it is practical to postpone release until we solve all of
such problems, since this being volunteer-driven open-source project
this means not having any release schedule at all. I prefer having the
schedule even if that means we'd have to release with some known
deficiencies.
- There was this problem with 5.3.7 and the
crypt()bug. Has there been
some improvement in the process of handling test failures? For example
mark expected failures as expected failures, and fix the tests or the
code? Or are the failing tests "stable" since month and all of them are
minor problems?
Yes, there was work done on these. Most of those were fixed, but few
still remain, especially across various environments (i.e. test may be
fine on some but not others). I of course am all for fixing that further
and welcome any help on that.
- There have been 319 unique failed tests in RC5, reported by user
tests. Is someone looking into them and trying to classify and/or fix them?
http://qa.php.net/reports/
Non-reproducible failures usually mean the problem is with the test
itself, or with the difference of expectations in the test and local
environment, not with PHP. It may still be PHP problem, of course, so
the person running the test should check it out and submit a bug if
appropriate and if it's bad enough, also send a message to this list.
All in all the number of test failures still feels very high, I would be
interested in your opinion. Is this "normal" for big projects like this?
I do think it should be reduced, however if the choice is between
waiting forever and have release with some bugs, I think it is practical
to choose the latter. Of course, if we discover a major problem that
makes PHP unusable or seriously impacts many PHP users, it will be dealt
with immediately, and had been so in the past, but otherwise we have to
work within the realities of a big project with limited resources and
realize that while we strive for 0 bugs in every release, it may never
be possible.
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227
Excerpts from Stas Malyshev's message of Thu Jan 19 16:08:28 -0800 2012:
Hi!
- According to this website there are still 94 test failures in 5.4 .
Can you confirm all of them are minor problems?
http://gcov.php.net/viewer.php?version=PHP_5_4Most of them appear so, I'll go through them again to be sure and
encourage others to do so too and raise red flags if somebody sees
something bad there.
Unfortunately, some tests are environment-dependent or otherwise have
subtle dependencies or structure that make them work on one system and
fail on another not because of the bug in PHP but because of the test
itself. So, I have 0 fails on my Linux build but 6 fails on my Mac
build. Other times some systems may not support some capability, use old
version of the library, etc. and the test may not account for that.
These tests should be skipped or marked as XFAIL on platforms they are
known to fail on. Better to have no test than one that cannot be relied
upon. All supported platforms should pass with 0 fails. These intentional
skips should have open bugs that are documented in the test code so that
a developer can find out why this test was disabled when trying to make
a change covered by the test.
I do not think it is practical to postpone release until we solve all of
such problems, since this being volunteer-driven open-source project
this means not having any release schedule at all. I prefer having the
schedule even if that means we'd have to release with some known
deficiencies.
Its pretty bad actually. For all of PHP's success, this is something that
continues to baffle me, and many others I have talked to who are charged
with measuring quality and with patching systems in a timely manner. How
better to document unreliable tests than to skip them with something like
"SKIPPED - known to fail on Mac".
Its precisely this unreliability that forced me to take a conservative
approach for Ubuntu 12.04 and recommend to the community that we ship
5.3.9 instead of 5.4.0. I would much rather have the new stuff in, but
even if all the tests pass on the machine we run the test suite on,
how can we be sure they won't fail in another time zone, or in some
other strange configuration?
- There was this problem with 5.3.7 and the
crypt()bug. Has there been
some improvement in the process of handling test failures? For example
mark expected failures as expected failures, and fix the tests or the
code? Or are the failing tests "stable" since month and all of them are
minor problems?Yes, there was work done on these. Most of those were fixed, but few
still remain, especially across various environments (i.e. test may be
fine on some but not others). I of course am all for fixing that further
and welcome any help on that.
- There have been 319 unique failed tests in RC5, reported by user
tests. Is someone looking into them and trying to classify and/or fix them?
http://qa.php.net/reports/Non-reproducible failures usually mean the problem is with the test
itself, or with the difference of expectations in the test and local
environment, not with PHP. It may still be PHP problem, of course, so
the person running the test should check it out and submit a bug if
appropriate and if it's bad enough, also send a message to this list.All in all the number of test failures still feels very high, I would be
interested in your opinion. Is this "normal" for big projects like this?I do think it should be reduced, however if the choice is between
waiting forever and have release with some bugs, I think it is practical
to choose the latter. Of course, if we discover a major problem that
makes PHP unusable or seriously impacts many PHP users, it will be dealt
with immediately, and had been so in the past, but otherwise we have to
work within the realities of a big project with limited resources and
realize that while we strive for 0 bugs in every release, it may never
be possible.
All software will have bugs. The test suite, however, should reflect
the bits of code that you know work reliably... not the bits of code
you know work most of the time.
The fact that its all being running regularly is a fantastic improvement.
I'd like to see a commitment to getting 100% pass/xfail/skip for every
release/tested environment in future releases though.
Hi!
These tests should be skipped or marked as XFAIL on platforms they are
known to fail on. Better to have no test than one that cannot be relied
upon. All supported platforms should pass with 0 fails. These intentional
Of course they should, and you (or anybody else) are welcome to make the
patches that make them so :) I promise they'll be in 5.4.1.
Its precisely this unreliability that forced me to take a conservative
approach for Ubuntu 12.04 and recommend to the community that we ship
5.3.9 instead of 5.4.0. I would much rather have the new stuff in, but
5.4.0 is better, not worse, than 5.3.9 in this regard - especially
because of the work that was done in 5.4 branch to fix or improve tests
that were failing and improve test coverage. I would advise to give your
users a choice once 5.4.0 is released and support both packages. On my
Linux environment, I have now 0 fails (though I don't run all modules so
some of non-default ones may be missing).
even if all the tests pass on the machine we run the test suite on,
how can we be sure they won't fail in another time zone, or in some
other strange configuration?
Well, you never can be sure - tests can test only things they know
about, unless somebody invents a way to make unit test test every
possible combination of environments at once. So far I never heard about
such tests.
The fact that its all being running regularly is a fantastic improvement.
I'd like to see a commitment to getting 100% pass/xfail/skip for every
release/tested environment in future releases though.
I'd like that too. I hope more people will step up and help to make this
a reality.
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227
Excerpts from Stas Malyshev's message of Thu Jan 19 16:08:28 -0800 2012:
Hi!
- According to this website there are still 94 test failures in 5.4 .
Can you confirm all of them are minor problems?
http://gcov.php.net/viewer.php?version=PHP_5_4Most of them appear so, I'll go through them again to be sure and
encourage others to do so too and raise red flags if somebody sees
something bad there.
Unfortunately, some tests are environment-dependent or otherwise have
subtle dependencies or structure that make them work on one system and
fail on another not because of the bug in PHP but because of the test
itself. So, I have 0 fails on my Linux build but 6 fails on my Mac
build. Other times some systems may not support some capability, use old
version of the library, etc. and the test may not account for that.These tests should be skipped or marked as XFAIL on platforms they are
known to fail on. Better to have no test than one that cannot be relied
upon. All supported platforms should pass with 0 fails. These intentional
skips should have open bugs that are documented in the test code so that
a developer can find out why this test was disabled when trying to make
a change covered by the test.I do not think it is practical to postpone release until we solve all of
such problems, since this being volunteer-driven open-source project
this means not having any release schedule at all. I prefer having the
schedule even if that means we'd have to release with some known
deficiencies.Its pretty bad actually. For all of PHP's success, this is something that
continues to baffle me, and many others I have talked to who are charged
with measuring quality and with patching systems in a timely manner. How
better to document unreliable tests than to skip them with something like
"SKIPPED - known to fail on Mac".Its precisely this unreliability that forced me to take a conservative
approach for Ubuntu 12.04 and recommend to the community that we ship
5.3.9 instead of 5.4.0. I would much rather have the new stuff in, but
even if all the tests pass on the machine we run the test suite on,
how can we be sure they won't fail in another time zone, or in some
other strange configuration?
Given that 12.04 beta2 will be out on Thursday, and the unit tests where
fixed before shipping 5.4 (I naively assume), is this in or out?
ref: https://blueprints.launchpad.net/ubuntu/+spec/servercloud-p-php54
With beta freeze on march 22 I guess the "mistake" of bundeling 5.3 instead
of 5.4 is already made, and we will have to live with that for the next 2
years for prod, and the next 7 months for dev.
- There was this problem with 5.3.7 and the
crypt()bug. Has there been
some improvement in the process of handling test failures? For example
mark expected failures as expected failures, and fix the tests or the
code? Or are the failing tests "stable" since month and all of them are
minor problems?Yes, there was work done on these. Most of those were fixed, but few
still remain, especially across various environments (i.e. test may be
fine on some but not others). I of course am all for fixing that further
and welcome any help on that.
- There have been 319 unique failed tests in RC5, reported by user
tests. Is someone looking into them and trying to classify and/or fix
them?
http://qa.php.net/reports/Non-reproducible failures usually mean the problem is with the test
itself, or with the difference of expectations in the test and local
environment, not with PHP. It may still be PHP problem, of course, so
the person running the test should check it out and submit a bug if
appropriate and if it's bad enough, also send a message to this list.All in all the number of test failures still feels very high, I would
be
interested in your opinion. Is this "normal" for big projects like
this?I do think it should be reduced, however if the choice is between
waiting forever and have release with some bugs, I think it is practical
to choose the latter. Of course, if we discover a major problem that
makes PHP unusable or seriously impacts many PHP users, it will be dealt
with immediately, and had been so in the past, but otherwise we have to
work within the realities of a big project with limited resources and
realize that while we strive for 0 bugs in every release, it may never
be possible.All software will have bugs. The test suite, however, should reflect
the bits of code that you know work reliably... not the bits of code
you know work most of the time.The fact that its all being running regularly is a fantastic improvement.
I'd like to see a commitment to getting 100% pass/xfail/skip for every
release/tested environment in future releases though.
hi,
With beta freeze on march 22 I guess the "mistake" of bundeling 5.3 instead
of 5.4 is already made, and we will have to live with that for the next 2
years for prod, and the next 7 months for dev.
It was long planed as far as I remember, to use 5.3 and not 5.4 in
this ubuntu release. We tried to convince them but sadly failed.
Debian however will use or uses already 5.4, if nothing has changed :)
--
Pierre
@pierrejoye | http://blog.thepimp.net | http://www.libgd.org
Excerpts from André Rømcke's message of Mon Mar 26 04:44:53 -0700 2012:
Excerpts from Stas Malyshev's message of Thu Jan 19 16:08:28 -0800 2012:
Hi!
- According to this website there are still 94 test failures in 5.4 .
Can you confirm all of them are minor problems?
http://gcov.php.net/viewer.php?version=PHP_5_4Most of them appear so, I'll go through them again to be sure and
encourage others to do so too and raise red flags if somebody sees
something bad there.
Unfortunately, some tests are environment-dependent or otherwise have
subtle dependencies or structure that make them work on one system and
fail on another not because of the bug in PHP but because of the test
itself. So, I have 0 fails on my Linux build but 6 fails on my Mac
build. Other times some systems may not support some capability, use old
version of the library, etc. and the test may not account for that.These tests should be skipped or marked as XFAIL on platforms they are
known to fail on. Better to have no test than one that cannot be relied
upon. All supported platforms should pass with 0 fails. These intentional
skips should have open bugs that are documented in the test code so that
a developer can find out why this test was disabled when trying to make
a change covered by the test.I do not think it is practical to postpone release until we solve all of
such problems, since this being volunteer-driven open-source project
this means not having any release schedule at all. I prefer having the
schedule even if that means we'd have to release with some known
deficiencies.Its pretty bad actually. For all of PHP's success, this is something that
continues to baffle me, and many others I have talked to who are charged
with measuring quality and with patching systems in a timely manner. How
better to document unreliable tests than to skip them with something like
"SKIPPED - known to fail on Mac".Its precisely this unreliability that forced me to take a conservative
approach for Ubuntu 12.04 and recommend to the community that we ship
5.3.9 instead of 5.4.0. I would much rather have the new stuff in, but
even if all the tests pass on the machine we run the test suite on,
how can we be sure they won't fail in another time zone, or in some
other strange configuration?Given that 12.04 beta2 will be out on Thursday, and the unit tests where
fixed before shipping 5.4 (I naively assume), is this in or out?
ref: https://blueprints.launchpad.net/ubuntu/+spec/servercloud-p-php54With beta freeze on march 22 I guess the "mistake" of bundeling 5.3 instead
of 5.4 is already made, and we will have to live with that for the next 2
years for prod, and the next 7 months for dev.
Our hands are tied, as the security team still does not feel comfortable
shipping a PHP without Suhosin. Perhaps more can be done to convince the
world that this is a safe thing to do now, but for now, we're taking the
extremely conservative stance and shipping 5.3.10 with the Suhosin patch.
Thanks everyone for chiming in, and especially thanks to Ondrej for
pushing hard to get things tested and rebuilt.
Our hands are tied, as the security team still does not feel
comfortable shipping a PHP without Suhosin. Perhaps more can be done
to convince the world that this is a safe thing to do now, but for
now, we're taking the extremely conservative stance and shipping
5.3.10 with the Suhosin patch.Thanks everyone for chiming in, and especially thanks to Ondrej for
pushing hard to get things tested and rebuilt.
Thinking loud: One could also ship both. Yes this doubles the effort but
gives users a choice :-)
johannes
Excerpts from Johannes Schlüter's message of Mon Mar 26 16:09:20 -0700 2012:
Our hands are tied, as the security team still does not feel
comfortable shipping a PHP without Suhosin. Perhaps more can be done
to convince the world that this is a safe thing to do now, but for
now, we're taking the extremely conservative stance and shipping
5.3.10 with the Suhosin patch.Thanks everyone for chiming in, and especially thanks to Ondrej for
pushing hard to get things tested and rebuilt.Thinking loud: One could also ship both. Yes this doubles the effort but
gives users a choice :-)
This simply won't happen in the main archive of Ubuntu. The whole point
of having a version from the archive in an LTS is that it receives security
updates for 5 years, regardless of upstream releasing fixes or not.
If users want something unsupported, an effort can be made to setup a PPA:
https://help.launchpad.net/Packaging/PPA
In fact, Ondrej already went through the trouble of creating one for testing
purposes:
https://launchpad.net/~ondrej/+archive/php5
Ubuntu's paid (by Canonical) security team does not have the resources
to support two versions of anything really. Often times two versions of
something are provided (like python 2.6 and 2.7) during a transition
like we see in PHP right now. However, one is generally in universe,
which means it is only supported by the community.
I think the lesson here is to get the necessary bits from Suhosin into
PHP's core so that users can feel safe when using stock PHP, rather
than needing to wait for the good and generous folks at the hardened
PHP project to catch up.
hi,
hi,
I think the lesson here is to get the necessary bits from Suhosin into
PHP's core so that users can feel safe when using stock PHP, rather
than needing to wait for the good and generous folks at the hardened
PHP project to catch up.
I disagree. The lesson here is that the Ubuntu's security team should
have discussed with us to see what are actually their worries instead
of not following what is actually a good move for everyone.
Cheers,
Pierre
@pierrejoye | http://blog.thepimp.net | http://www.libgd.org
hi,
hi,I think the lesson here is to get the necessary bits from Suhosin into
PHP's core so that users can feel safe when using stock PHP, rather
than needing to wait for the good and generous folks at the hardened
PHP project to catch up.I disagree. The lesson here is that the Ubuntu's security team should
have discussed with us to see what are actually their worries instead
of not following what is actually a good move for everyone.
Why? It is common practice to avoid .0 releases, including those from PHP :)
Actually, this is not a bad thing at all.
It makes sure lots of PHP projects and Frameworks stays on 5.3 as a common
base for the next 1-3 years, introp wise that is a good thing.
And those that want the extra 5.4 stuff can get it somehow anyway, and as
part of the distro in 7 months.
Cheers,
Pierre
@pierrejoye | http://blog.thepimp.net | http://www.libgd.org
I disagree. The lesson here is that the Ubuntu's security team should
have discussed with us to see what are actually their worries instead
of not following what is actually a good move for everyone.Why?
Non vanilla php.
Cheers,
Pierre
@pierrejoye | http://blog.thepimp.net | http://www.libgd.org
André Rømcke wrote:
I disagree. The lesson here is that the Ubuntu's security team should
have discussed with us to see what are actually their worries instead
of not following what is actually a good move for everyone.Why? It is common practice to avoid .0 releases, including those from PHP:)
Actually, this is not a bad thing at all.
It makes sure lots of PHP projects and Frameworks stays on 5.3 as a common
base for the next 1-3 years, introp wise that is a good thing.And those that want the extra 5.4 stuff can get it somehow anyway, and as
part of the distro in 7 months.
Or another 3 years time ...
I've taken over support for a number of php websites across a couple of ISP's.
Currently they are running on PHP5.2.10 because that is what has been provided.
And the upgrade by one of these has been announced for May ... to 5.3.10
Changing that to 5.4 was ruled out by the ISP because they would need to rerun
all of their tests, and they had given up waiting for 5.4 at the end of last year.
Real life users need 6 months to bed new things in, so I can well understand
Ubuntu's stance. It makes perfect sense NOT to rush something new in. I've got
to check all of my customers sites will still work in May so finishing work on
moving other sites to 5.4 is on hold .... I've already had the panics by 5.3.10
being installed on another ISP and having to get sites on that machine working
again. Nothing has changed on what the SITES are doing, only config settings
screwing up BC :(
--
Lester Caine - G8HFL
Contact - http://lsces.co.uk/wiki/?page=contact
L.S.Caine Electronic Services - http://lsces.co.uk
EnquirySolve - http://enquirysolve.com/
Model Engineers Digital Workshop - http://medw.co.uk//
Firebird - http://www.firebirdsql.org/index.php
Hi!
Actually, this is not a bad thing at all.
It makes sure lots of PHP projects and Frameworks stays on 5.3 as a common
base for the next 1-3 years, introp wise that is a good thing.
How this is good? 5.4 is superior to 5.3 in... well, all aspects, why
denying people chance to use it for years is a good thing?
And those that want the extra 5.4 stuff can get it somehow anyway, and as
part of the distro in 7 months.
People of course will get it "somehow", but then why bother with having
Linux distribution? This stuff is open source, people can download it
somehow anyway. Obviously, there's some value in getting software as
part of organized process, not "somehow".
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227
Hi!
I think the lesson here is to get the necessary bits from Suhosin into
PHP's core so that users can feel safe when using stock PHP, rather
than needing to wait for the good and generous folks at the hardened
PHP project to catch up.
Unfortunately, the good and generous leader of Suhosin project expressed
his complete opposition to the cooperation with PHP team on the topic of
getting the features into the core. It still can be done, I guess, but
I'm not sure if we will have a volunteer to do it, especially given this
situation.
As for users not feeling safe using stock PHP, I have a feeling you are
overestimating the number of users feeling so. Millions of users are
running stock PHP and we got no indication that they are suffering from
any particular strong feelings of unsafety. If the security team has any
specific concerns, of course, they can be discussed.
Without doubt, Suhosin adds a layer of protection, but I do not see why
this layer is so absolutely crucial that you are unable to release a
version of PHP without it. What would happen is that users would just
use third-party packages of 5.4 or build their own - with all issues
that follow that.
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227