CI for 5.4

13 years ago by Laruence — view source

unread

+1

thanks

2011/9/7 Stas Malyshev smalyshev@sugarcrm.com:

Hi!

Since we started to pay real attention to our unit tests now, I wonder if we
could set up some kind of frequently-running CI system that could be used to
screen commits and identify breakage early? That'd help with 5.4 process I
think.
We have http://gcov.php.net/ but it doesn't run with the frequency I'd like
and since it says the run takes 44 hours it's kind of understandable. So I
wonder if we could have something that just builds it and runs unit tests
and we could see it in the same format as on gcov? Ideally after each commit
would be nice, but say once an hour or two (even fullest unit tests run
should take more than that, I think) would be OK too. If we could have two
of them, like Linux & Windows, it'd be even better, but at least one would
be nice.
What do you think?

Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227

--

--
Laruence Xinchen Hui
http://www.laruence.com/

13 years ago by Ferenc Kovacs — view source

unread

Hi!

Since we started to pay real attention to our unit tests now, I wonder if we
could set up some kind of frequently-running CI system that could be used to
screen commits and identify breakage early? That'd help with 5.4 process I
think.
We have http://gcov.php.net/ but it doesn't run with the frequency I'd like
and since it says the run takes 44 hours it's kind of understandable. So I
wonder if we could have something that just builds it and runs unit tests
and we could see it in the same format as on gcov? Ideally after each commit
would be nice, but say once an hour or two (even fullest unit tests run
should take more than that, I think) would be OK too. If we could have two
of them, like Linux & Windows, it'd be even better, but at least one would
be nice.
What do you think?

Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227

--

Hi, I also support the idea.
I would suggest setting up a jenkins(ex-hudson) cluster, it is the
leading CI product on the market (and a really successful open source
project), and it is really well-known in the php community as well,
and because I have used for my other projects in the past(not just for
php, but for some C apps also).
Jenkins supports having multiple slave instances running on different
platforms, nicely bound together, so you only access the master
instance through your browser, but the builds itself can be off-loaded
to the slaves (
https://wiki.jenkins-ci.org/display/JENKINS/Distributed+builds ).
I would be happy to help setting it up.

--
Ferenc Kovács
@Tyr43l - http://tyrael.hu

13 years ago by Hannes Magnusson — view source

unread

Hi!

Since we started to pay real attention to our unit tests now, I wonder if we
could set up some kind of frequently-running CI system that could be used to
screen commits and identify breakage early? That'd help with 5.4 process I
think.
We have http://gcov.php.net/ but it doesn't run with the frequency I'd like
and since it says the run takes 44 hours it's kind of understandable. So I
wonder if we could have something that just builds it and runs unit tests
and we could see it in the same format as on gcov? Ideally after each commit
would be nice, but say once an hour or two (even fullest unit tests run
should take more than that, I think) would be OK too. If we could have two
of them, like Linux & Windows, it'd be even better, but at least one would
be nice.
What do you think?

Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227

--

Hi, I also support the idea.
I would suggest setting up a jenkins(ex-hudson) cluster, it is the
leading CI product on the market (and a really successful open source
project), and it is really well-known in the php community as well,
and because I have used for my other projects in the past(not just for
php, but for some C apps also).
Jenkins supports having multiple slave instances running on different
platforms, nicely bound together, so you only access the master
instance through your browser, but the builds itself can be off-loaded
to the slaves (
https://wiki.jenkins-ci.org/display/JENKINS/Distributed+builds ).
I would be happy to help setting it up.

We have wanted to setup some buildbots and testing environments for
years now, and I think we even have several boxes still laying around
for that purposes.
If you think you can handle the initial installation, go for it.
I'll see if I can hunt down the credentials somewhere.

-Hannes

13 years ago by Ferenc Kovacs — view source

unread

On Wed, Sep 7, 2011 at 11:29 AM, Hannes Magnusson <
hannes.magnusson@gmail.com> wrote:

On Wed, Sep 7, 2011 at 3:29 AM, Stas Malyshev smalyshev@sugarcrm.com
wrote:

Hi!

Since we started to pay real attention to our unit tests now, I wonder
if we
could set up some kind of frequently-running CI system that could be
used to
screen commits and identify breakage early? That'd help with 5.4
process I
think.
We have http://gcov.php.net/ but it doesn't run with the frequency I'd
like
and since it says the run takes 44 hours it's kind of understandable.
So I
wonder if we could have something that just builds it and runs unit
tests
and we could see it in the same format as on gcov? Ideally after each
commit
would be nice, but say once an hour or two (even fullest unit tests run
should take more than that, I think) would be OK too. If we could have
two
of them, like Linux & Windows, it'd be even better, but at least one
would
be nice.
What do you think?

Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227

--

Hi, I also support the idea.
I would suggest setting up a jenkins(ex-hudson) cluster, it is the
leading CI product on the market (and a really successful open source
project), and it is really well-known in the php community as well,
and because I have used for my other projects in the past(not just for
php, but for some C apps also).
Jenkins supports having multiple slave instances running on different
platforms, nicely bound together, so you only access the master
instance through your browser, but the builds itself can be off-loaded
to the slaves (
https://wiki.jenkins-ci.org/display/JENKINS/Distributed+builds ).
I would be happy to help setting it up.

We have wanted to setup some buildbots and testing environments for
years now, and I think we even have several boxes still laying around
for that purposes.
If you think you can handle the initial installation, go for it.
I'll see if I can hunt down the credentials somewhere.

-Hannes

Hi.

We ended up creating a Proof-of-concept Jenkins installation using the
server which was originally planned(
https://wiki.php.net/internals/buildbot https://wiki.php.net/internals/buildbot?s[]=buildbot
)
to be used for the buildbot setup(obviously Hannes managed to get the
credentials \o/), and created an RFC based on the current setup and the
ways that it could be extended:
https://wiki.php.net/rfc/jenkins
Basically it continuously(when the are changes in SVN since the last build)
builds our supported branches, runs the phpt testsuite(nicely integrated
the test results with jenkins), and also executes the Symfony2 phpunit
testsuite for each branch.
I think that using Jenkins would be really useful, as it is takes care of
the "hard" part of the CI process(distributed build support, integration
with every QA tool out there), and the php community is pretty much
familiar with it (at least those who are knew what the CI acronym means)
plus it is easy to integrate the userland testsuites, as many of those
projects already using Jenkins or phpunit, which is nicely integrated with
Jenkins, thanks to Sebastian.

We knew that Pierre and Dan is working on a distributed build/test
environment also, so we are well aware the fact that our work is maybe
redundant, or could end up as wasted, but we thought that it still worth
the effort.

Please check out the RFC, and if you have any question or opinion on the
matter, feel free to discuss in this thread.
Oh, and of course check out http://ci.qa.php.net/

--
Ferenc Kovács
@Tyr43l - http://tyrael.hu

13 years ago by Ferenc Kovacs — view source

unread

On Wed, Sep 7, 2011 at 11:29 AM, Hannes Magnusson <
hannes.magnusson@gmail.com> wrote:

On Wed, Sep 7, 2011 at 3:29 AM, Stas Malyshev smalyshev@sugarcrm.com
wrote:

Hi!

Since we started to pay real attention to our unit tests now, I wonder
if we
could set up some kind of frequently-running CI system that could be
used to
screen commits and identify breakage early? That'd help with 5.4
process I
think.
We have http://gcov.php.net/ but it doesn't run with the frequency
I'd like
and since it says the run takes 44 hours it's kind of understandable.
So I
wonder if we could have something that just builds it and runs unit
tests
and we could see it in the same format as on gcov? Ideally after each
commit
would be nice, but say once an hour or two (even fullest unit tests run
should take more than that, I think) would be OK too. If we could have
two
of them, like Linux & Windows, it'd be even better, but at least one
would
be nice.
What do you think?

Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227

--

Hi, I also support the idea.
I would suggest setting up a jenkins(ex-hudson) cluster, it is the
leading CI product on the market (and a really successful open source
project), and it is really well-known in the php community as well,
and because I have used for my other projects in the past(not just for
php, but for some C apps also).
Jenkins supports having multiple slave instances running on different
platforms, nicely bound together, so you only access the master
instance through your browser, but the builds itself can be off-loaded
to the slaves (
https://wiki.jenkins-ci.org/display/JENKINS/Distributed+builds ).
I would be happy to help setting it up.

We have wanted to setup some buildbots and testing environments for
years now, and I think we even have several boxes still laying around
for that purposes.
If you think you can handle the initial installation, go for it.
I'll see if I can hunt down the credentials somewhere.

-Hannes

Hi.

We ended up creating a Proof-of-concept Jenkins installation using the
server which was originally planned(
https://wiki.php.net/internals/buildbot https://wiki.php.net/internals/buildbot?s[]=buildbot )
to be used for the buildbot setup(obviously Hannes managed to get the
credentials \o/), and created an RFC based on the current setup and the
ways that it could be extended:
https://wiki.php.net/rfc/jenkins
Basically it continuously(when the are changes in SVN since the last
build) builds our supported branches, runs the phpt testsuite(nicely
integrated the test results with jenkins), and also executes the Symfony2
phpunit testsuite for each branch.
I think that using Jenkins would be really useful, as it is takes care of
the "hard" part of the CI process(distributed build support, integration
with every QA tool out there), and the php community is pretty much
familiar with it (at least those who are knew what the CI acronym means)
plus it is easy to integrate the userland testsuites, as many of those
projects already using Jenkins or phpunit, which is nicely integrated with
Jenkins, thanks to Sebastian.

We knew that Pierre and Dan is working on a distributed build/test
environment also, so we are well aware the fact that our work is maybe
redundant, or could end up as wasted, but we thought that it still worth
the effort.

Please check out the RFC, and if you have any question or opinion on the
matter, feel free to discuss in this thread.
Oh, and of course check out http://ci.qa.php.net/

--
Ferenc Kovács
@Tyr43l - http://tyrael.hu

A little update, I finished the master.php.net integration, so now you
should be able to login using your svn credentials.

ps: you can access the site using https, but it will use a self signed
cert, this can be fixed in the future, but unfortunately our
*.php.netwildcard cert won't work for
ci.qa.php.net :(

--
Ferenc Kovács
@Tyr43l - http://tyrael.hu

13 years ago by Klaus Silveira — view source

unread

What are the current obstacles in a possible Jenkins implementation for the
PHP codebase?

--
Klaus Silveira
(011) 8564-2492
www.klaussilveira.com

13 years ago by Ferenc Kovacs — view source

unread

On Thu, Nov 3, 2011 at 6:10 PM, Klaus Silveira contato@klaussilveira.comwrote:

What are the current obstacles in a possible Jenkins implementation for
the PHP codebase?

--
Klaus Silveira
(011) 8564-2492
www.klaussilveira.com

Sorry, I don't really follow you.
As I mentioned in my previous email and the RFC, the php-src(all of the
supported branches: 5.3, 5.4 and trunk) is already integrated into Jenkins.
I also listed the current problems/limitations in
https://wiki.php.net/rfc/jenkins#current_problems and there were other
obstacles (extending the run-tests.php to generate a test result format
supported by Jenkins), but most of them were solved (except those in the
current_problems section).
Of course there are ways to improve the current setup, I listed those ideas
at https://wiki.php.net/rfc/jenkins#future_plans

--
Ferenc Kovács
@Tyr43l - http://tyrael.hu

13 years ago by Stefan Marr — view source

unread

Hi Ferenc:

Of course there are ways to improve the current setup, I listed those ideas
at https://wiki.php.net/rfc/jenkins#future_plans

Very nice.
I don't know Jenkins, but would it be possible to mail the author of a change in case it breaks something? I just had such a case, where my local setup was insufficient to catch it.

Thanks a lot
Stefan

--
Stefan Marr
Software Languages Lab
Vrije Universiteit Brussel
Pleinlaan 2 / B-1050 Brussels / Belgium
http://soft.vub.ac.be/~smarr
Phone: +32 2 629 2974
Fax: +32 2 629 3525

13 years ago by Klaus Silveira — view source

unread

Yes, it is. But not only email notifications are possible, other
interesting integrations are possible. For example, Jabber.

https://wiki.jenkins-ci.org/display/JENKINS/Instant+Messaging+Plugin
https://wiki.jenkins-ci.org/display/JENKINS/Jabber+Plugin

Hi Ferenc:

Of course there are ways to improve the current setup, I listed those
ideas
at https://wiki.php.net/rfc/jenkins#future_plans

Very nice.
I don't know Jenkins, but would it be possible to mail the author of a
change in case it breaks something? I just had such a case, where my local
setup was insufficient to catch it.

Thanks a lot
Stefan

--
Stefan Marr
Software Languages Lab
Vrije Universiteit Brussel
Pleinlaan 2 / B-1050 Brussels / Belgium
http://soft.vub.ac.be/~smarr
Phone: +32 2 629 2974
Fax: +32 2 629 3525

--

--
Klaus Silveira
(011) 8564-2492
www.klaussilveira.com

13 years ago by Ferenc Kovacs — view source

unread

Hi Ferenc:

Of course there are ways to improve the current setup, I listed those
ideas
at https://wiki.php.net/rfc/jenkins#future_plans

Very nice.
I don't know Jenkins, but would it be possible to mail the author of a
change in case it breaks something? I just had such a case, where my local
setup was insufficient to catch it.

Thanks a lot
Stefan

It is, and as I mentioned in the RFC the email notification can be enabled
when needed, I didn't wanted to do that before introducing the RFC.
There is a slight problem however: if we don't build each and every
revision (which can happen, if two commit happens before the build starts,
which can happen more easily if another build is in progress for that job,
because then jenkins will just queue the build, not execute it right away
for obvious reasons), so there is a slight chance for "blaming" the wrong
person.
The usual setup is that you have a mailing list/alias which always gets a
mail about the build results (which can also be customized in detail, when
to mail, etc.) and optionally you can notify the person whose commit break
the build.

As Klaus pointed out, there are other possibilities for notification,
currently we were using the irc plugin, so if you hang around on #php.qa on
efnet, you could see the build results or ask phpcibot about the status of
the builds, However I just noticed that there is no way to restrict who can
start builds through the irc bot(see
https://issues.jenkins-ci.org/browse/JENKINS-5931), so I turned it off.

20:13 -!- phpcibot [~PircBotX@80.82.121.132] has quit [Remote host closed
the connection]

:(

--
Ferenc Kovács
@Tyr43l - http://tyrael.hu

13 years ago by Stefan Marr — view source

unread

Hi:

The usual setup is that you have a mailing list/alias which always gets a mail about the build results (which can also be customized in detail, when to mail, etc.) and optionally you can notify the person whose commit break the build.

If I get a personal notification via mail, if I break something, that kind of guarantees my attention. Using IRC is to distracting, so that is not an option for me. Sorry.

If I am the only one who would use it, then don't bother. But if I can simply opt-in for it somewhere, that would be great.

Thanks to people like Antony, obviously stupid bugs get caught pretty fast.

Thanks
Stefan

--
Stefan Marr
Software Languages Lab
Vrije Universiteit Brussel
Pleinlaan 2 / B-1050 Brussels / Belgium
http://soft.vub.ac.be/~smarr
Phone: +32 2 629 2974
Fax: +32 2 629 3525

13 years ago by Klaus Silveira — view source

unread

That's kind of a general setup, Stefan. Sending an email to the commiter
that broke the build with details of the build process, as well sending an
email to a mailing list.

I'll be looking into this Jenkins issue (JENKINS-5931). Maybe having an
access control based on IRC operators and voices?

Hi:

The usual setup is that you have a mailing list/alias which always gets
a mail about the build results (which can also be customized in detail,
when to mail, etc.) and optionally you can notify the person whose commit
break the build.

If I get a personal notification via mail, if I break something, that kind
of guarantees my attention. Using IRC is to distracting, so that is not an
option for me. Sorry.

If I am the only one who would use it, then don't bother. But if I can
simply opt-in for it somewhere, that would be great.

Thanks to people like Antony, obviously stupid bugs get caught pretty fast.

Thanks
Stefan

--
Stefan Marr
Software Languages Lab
Vrije Universiteit Brussel
Pleinlaan 2 / B-1050 Brussels / Belgium
http://soft.vub.ac.be/~smarr
Phone: +32 2 629 2974
Fax: +32 2 629 3525

--
Klaus Silveira
(011) 8564-2492
www.klaussilveira.com

13 years ago by Ferenc Kovacs — view source

unread

On Thu, Nov 3, 2011 at 8:41 PM, Klaus Silveira contato@klaussilveira.comwrote:

That's kind of a general setup, Stefan. Sending an email to the commiter
that broke the build with details of the build process, as well sending an
email to a mailing list.

I'll be looking into this Jenkins issue (JENKINS-5931). Maybe having an
access control based on IRC operators and voices?

yeah, using voices would work, but then that would mean that it is only a
read only channel for most people, which imo would be a bad move.
I mean it would be really nice, if people could drop by to our qa related
irc room and start talking about that pesky build failure.
another solution would be to set up a custom irc bot, independently from
jenkins, which could fetch the build results through the mailing list or
rss and send it to the channel.
of course extending the jenkins irc bot is also a possibility, the easiest
way would be having a configuration option for disabling the whole control
stuff, as we don't really need the interactivity there, you can use the web
interface for that.

--
Ferenc Kovács
@Tyr43l - http://tyrael.hu

13 years ago by Felipe Pena — view source

unread

2011/11/3 Ferenc Kovacs tyra3l@gmail.com:

On Thu, Nov 3, 2011 at 8:41 PM, Klaus Silveira contato@klaussilveira.comwrote:

That's kind of a general setup, Stefan. Sending an email to the commiter
that broke the build with details of the build process, as well sending an
email to a mailing list.

I'll be looking into this Jenkins issue (JENKINS-5931). Maybe having an
access control based on IRC operators and voices?

yeah, using voices would work, but then that would mean that it is only a
read only channel for most people, which imo would be a bad move.
I mean it would be really nice, if people could drop by to our qa related
irc room and start talking about that pesky build failure.
another solution would be to set up a custom irc bot, independently from
jenkins, which could fetch the build results through the mailing list or
rss and send it to the channel.
of course extending the jenkins irc bot is also a possibility, the easiest
way would be having a configuration option for disabling the whole control
stuff, as we don't really need the interactivity there, you can use the web
interface for that.

--
Ferenc Kovács
@Tyr43l - http://tyrael.hu

Nice work with this Jenkins stuff! The system seems pretty useful.
I like the idea to have a mailing list for build result notification,
as well as send mail to the build breaker.

--
Regards,
Felipe Pena

13 years ago by Klaus Silveira — view source

unread

Yes, it's a wonderful setup. Great work, Ferenc! :D

This does give a nice motivation to write more tests and increase the code
coverage, doesn't it?

2011/11/3 Ferenc Kovacs tyra3l@gmail.com:

On Thu, Nov 3, 2011 at 8:41 PM, Klaus Silveira <
contato@klaussilveira.com>wrote:

That's kind of a general setup, Stefan. Sending an email to the commiter
that broke the build with details of the build process, as well sending
an
email to a mailing list.

I'll be looking into this Jenkins issue (JENKINS-5931). Maybe having an
access control based on IRC operators and voices?

yeah, using voices would work, but then that would mean that it is only a
read only channel for most people, which imo would be a bad move.
I mean it would be really nice, if people could drop by to our qa related
irc room and start talking about that pesky build failure.
another solution would be to set up a custom irc bot, independently from
jenkins, which could fetch the build results through the mailing list or
rss and send it to the channel.
of course extending the jenkins irc bot is also a possibility, the
easiest
way would be having a configuration option for disabling the whole
control
stuff, as we don't really need the interactivity there, you can use the
web
interface for that.

--
Ferenc Kovács
@Tyr43l - http://tyrael.hu

Nice work with this Jenkins stuff! The system seems pretty useful.
I like the idea to have a mailing list for build result notification,
as well as send mail to the build breaker.

--
Regards,
Felipe Pena

--

--
Klaus Silveira
(011) 8564-2492
www.klaussilveira.com

13 years ago by Ferenc Kovacs — view source

unread

On Thu, Nov 3, 2011 at 10:25 PM, Klaus Silveira
contato@klaussilveira.comwrote:

Yes, it's a wonderful setup. Great work, Ferenc! :D

This does give a nice motivation to write more tests and increase the code
coverage, doesn't it?

Yes, and first of all, to fix the currently failing tests.
We also had a discussion with Hannes, that the build machines could also be
used for hunting down bugs, for which the developer doesn't have an
environment to test and reproduce the problem.

--
Ferenc Kovács
@Tyr43l - http://tyrael.hu

13 years ago by Ferenc Kovacs — view source

unread

2011/11/3 Ferenc Kovacs tyra3l@gmail.com:

On Thu, Nov 3, 2011 at 8:41 PM, Klaus Silveira <
contato@klaussilveira.com>wrote:

That's kind of a general setup, Stefan. Sending an email to the commiter
that broke the build with details of the build process, as well sending
an
email to a mailing list.

I'll be looking into this Jenkins issue (JENKINS-5931). Maybe having an
access control based on IRC operators and voices?

yeah, using voices would work, but then that would mean that it is only a
read only channel for most people, which imo would be a bad move.
I mean it would be really nice, if people could drop by to our qa related
irc room and start talking about that pesky build failure.
another solution would be to set up a custom irc bot, independently from
jenkins, which could fetch the build results through the mailing list or
rss and send it to the channel.
of course extending the jenkins irc bot is also a possibility, the
easiest
way would be having a configuration option for disabling the whole
control
stuff, as we don't really need the interactivity there, you can use the
web
interface for that.

--
Ferenc Kovács
@Tyr43l - http://tyrael.hu

Nice work with this Jenkins stuff! The system seems pretty useful.
I like the idea to have a mailing list for build result notification,
as well as send mail to the build breaker.

Thanks for the feedback.
We could set up the email notification any time, we would just have to
agree where to send (internals, php-qa, creating a dedicated mailing list?)
and when to send (as I added to the RFC, having failing tests can be a
problem from the notification POV).
We have to find a good middle way, else we will either miss the
notifications, or we will be overwhelmed, so people will just ignore it.

--
Ferenc Kovács
@Tyr43l - http://tyrael.hu

13 years ago by Rasmus Lerdorf — view source

unread

We could set up the email notification any time, we would just have to
agree where to send (internals, php-qa, creating a dedicated mailing list?)
and when to send (as I added to the RFC, having failing tests can be a
problem from the notification POV).
We have to find a good middle way, else we will either miss the
notifications, or we will be overwhelmed, so people will just ignore it.

Ideally they should go to internals if you can get them to be non-noisy.

As long as it only reports a test status change from the previous run,
hopefully that wouldn't create much noise.

-Rasmus

13 years ago by Ferenc Kovacs — view source

unread

We could set up the email notification any time, we would just have to
agree where to send (internals, php-qa, creating a dedicated mailing
list?)
and when to send (as I added to the RFC, having failing tests can be a
problem from the notification POV).
We have to find a good middle way, else we will either miss the
notifications, or we will be overwhelmed, so people will just ignore it.

Ideally they should go to internals if you can get them to be non-noisy.

true

As long as it only reports a test status change from the previous run,
hopefully that wouldn't create much noise.

yeah, and that is a little bit of a problem, because as I mentioned by
default the jenkins way is that your tests passes:
http://jenkins.361315.n4.nabble.com/Test-failure-baseline-td386476.html
so we have three option as far as I can see:
A, it will report every test failure, but we fix all of our tests so we are
cool. (my preference)
B, same as A, but until we fix all of the tests, we mark the failing test
as XFAIL, which won't be counted as failed test (I'm against it, as time
taught us that we are lazy, and XFAILs will be kept that way. :/)
C, Somebody(me) do some scripting, and our currently failing tests won't
fail the build, only make it unstable, but any other test failure will fail
the build.

I think I didn't explained in the RFC, because currently we don't use it,
but in jenkins there is a third result type for a build apart success and
failure: unstable.
Most plugins support this, as does the email notification plugin also, so
this should also work.

--
Ferenc Kovács
@Tyr43l - http://tyrael.hu

13 years ago by Rasmus Lerdorf — view source

unread

A, it will report every test failure, but we fix all of our tests so we
are cool. (my preference)

But even if we do that, when a test does fail, it may take a couple of
days to fix it. It would be nice if we didn't get an internals email for
every commit that triggers a build during that time. So even in this
ideal case I think the notification mechanism needs to be smarter.

-Rasmus

13 years ago by Ferenc Kovacs — view source

unread

A, it will report every test failure, but we fix all of our tests so we
are cool. (my preference)

But even if we do that, when a test does fail, it may take a couple of
days to fix it. It would be nice if we didn't get an internals email for
every commit that triggers a build during that time. So even in this
ideal case I think the notification mechanism needs to be smarter.

-Rasmus

using the email-ext plugin:
https://wiki.jenkins-ci.org/display/JENKINS/Email-ext+plugin one can bind
email notification on the following events:

Failure
Unstable
Still-Failing
Success
Fixed
Still-Unstable
(Before-Build)

So we could configure to only send emails on Failure and Fixed(I ignored
the Unstable on purpose for the sake of simplicity, of course we also use
this for option C).
That way we would only see the notification on the first error, and the
next mail would be when we finally fixed it.
This is still suboptimal, as we wouldn't get email notifications about the
other test failures introduced between the first failure and the fix, but I
think that having a failed build would mean that somebody/everybody is
working on fixing the build, so those test failures would also be
spotted(on the web interface) and fixed (without that, the build won't
succeed).

I almost forget to mention, but the email notification also supports
defining different recipients for each event, so for example the commiters
could still get the notification for each of their commits/builds until the
build is back to normal, that way the list wouldn't be spammed, but some
active contributors would be still continuously bugged.

How does it sounds to you?

--
Ferenc Kovács
@Tyr43l - http://tyrael.hu

13 years ago by Ferenc Kovacs — view source

unread

Hi:

The usual setup is that you have a mailing list/alias which always gets
a mail about the build results (which can also be customized in detail,
when to mail, etc.) and optionally you can notify the person whose commit
break the build.

If I get a personal notification via mail, if I break something, that kind
of guarantees my attention. Using IRC is to distracting, so that is not an
option for me. Sorry.

agree, this is why this option is present.

If I am the only one who would use it, then don't bother. But if I can
simply opt-in for it somewhere, that would be great.

I think it would be useful, I just wanted to mention the only downside that
I can think of.
Of course if everybody else would be against the idea, you could still
subscribe to the mailing list where the general report goes, and filter the
messages to only receive those mail to your inbox which reports a build
which was triggered by you (if I remember correctly, by default the email
would contain who started the build, either through SCM or manually).

--
Ferenc Kovács
@Tyr43l - http://tyrael.hu

13 years ago by Ferenc Kovacs — view source

unread

Hi Ferenc:

Of course there are ways to improve the current setup, I listed those
ideas
at https://wiki.php.net/rfc/jenkins#future_plans

Very nice.
I don't know Jenkins, but would it be possible to mail the author of a
change in case it breaks something? I just had such a case, where my local
setup was insufficient to catch it.

Thanks a lot
Stefan

It is, and as I mentioned in the RFC the email notification can be enabled
when needed, I didn't wanted to do that before introducing the RFC.
There is a slight problem however: if we don't build each and every
revision (which can happen, if two commit happens before the build starts,
which can happen more easily if another build is in progress for that job,
because then jenkins will just queue the build, not execute it right away
for obvious reasons), so there is a slight chance for "blaming" the wrong
person.
The usual setup is that you have a mailing list/alias which always gets a
mail about the build results (which can also be customized in detail, when
to mail, etc.) and optionally you can notify the person whose commit break
the build.

As Klaus pointed out, there are other possibilities for notification,
currently we were using the irc plugin, so if you hang around on #php.qaon efnet, you could see the build results or ask phpcibot about the status
of the builds, However I just noticed that there is no way to restrict who
can start builds through the irc bot(see
https://issues.jenkins-ci.org/browse/JENKINS-5931), so I turned it off.

20:13 -!- phpcibot [~PircBotX@80.82.121.132] has quit [Remote host closed
the connection]

:(

--
Ferenc Kovács
@Tyr43l - http://tyrael.hu

I reported https://issues.jenkins-ci.org/browse/JENKINS-11606 on the
jenkins issue tracker, and they fixed it super fast, so phpcibot is back
again. \o/

--
Ferenc Kovács
@Tyr43l - http://tyrael.hu

13 years ago by Ferenc Kovacs — view source

unread

Hi Ferenc:

Of course there are ways to improve the current setup, I listed those
ideas
at https://wiki.php.net/rfc/jenkins#future_plans

Very nice.
I don't know Jenkins, but would it be possible to mail the author of a
change in case it breaks something? I just had such a case, where my local
setup was insufficient to catch it.

Thanks a lot
Stefan

It is, and as I mentioned in the RFC the email notification can be
enabled when needed, I didn't wanted to do that before introducing the RFC.
There is a slight problem however: if we don't build each and every
revision (which can happen, if two commit happens before the build starts,
which can happen more easily if another build is in progress for that job,
because then jenkins will just queue the build, not execute it right away
for obvious reasons), so there is a slight chance for "blaming" the wrong
person.
The usual setup is that you have a mailing list/alias which always gets a
mail about the build results (which can also be customized in detail, when
to mail, etc.) and optionally you can notify the person whose commit break
the build.

As Klaus pointed out, there are other possibilities for notification,
currently we were using the irc plugin, so if you hang around on #php.qaon efnet, you could see the build results or ask phpcibot about the status
of the builds, However I just noticed that there is no way to restrict who
can start builds through the irc bot(see
https://issues.jenkins-ci.org/browse/JENKINS-5931), so I turned it off.

20:13 -!- phpcibot [~PircBotX@80.82.121.132] has quit [Remote host
closed the connection]

:(

--
Ferenc Kovács
@Tyr43l - http://tyrael.hu

I reported https://issues.jenkins-ci.org/browse/JENKINS-11606 on the
jenkins issue tracker, and they fixed it super fast, so phpcibot is back
again. \o/

--
Ferenc Kovács
@Tyr43l - http://tyrael.hu

Hi.

I would like to set up a windows slave also, as I mentioned in the RFC.
I was informed that oti1 is mostly idle, so if somebody could give me
access there, I would set up a jenkins slave, and the neccessary
dependencies there.
Thanks!

--
Ferenc Kovács
@Tyr43l - http://tyrael.hu

13 years ago by Ferenc Kovacs — view source

unread

Hi Ferenc:

Of course there are ways to improve the current setup, I listed those
ideas
at https://wiki.php.net/rfc/jenkins#future_plans

Very nice.
I don't know Jenkins, but would it be possible to mail the author of a
change in case it breaks something? I just had such a case, where my local
setup was insufficient to catch it.

Thanks a lot
Stefan

It is, and as I mentioned in the RFC the email notification can be
enabled when needed, I didn't wanted to do that before introducing the RFC.
There is a slight problem however: if we don't build each and every
revision (which can happen, if two commit happens before the build starts,
which can happen more easily if another build is in progress for that job,
because then jenkins will just queue the build, not execute it right away
for obvious reasons), so there is a slight chance for "blaming" the wrong
person.
The usual setup is that you have a mailing list/alias which always gets
a mail about the build results (which can also be customized in detail,
when to mail, etc.) and optionally you can notify the person whose commit
break the build.

As Klaus pointed out, there are other possibilities for notification,
currently we were using the irc plugin, so if you hang around on #php.qaon efnet, you could see the build results or ask phpcibot about the status
of the builds, However I just noticed that there is no way to restrict who
can start builds through the irc bot(see
https://issues.jenkins-ci.org/browse/JENKINS-5931), so I turned it off.

20:13 -!- phpcibot [~PircBotX@80.82.121.132] has quit [Remote host
closed the connection]

:(

--
Ferenc Kovács
@Tyr43l - http://tyrael.hu

I reported https://issues.jenkins-ci.org/browse/JENKINS-11606 on the
jenkins issue tracker, and they fixed it super fast, so phpcibot is back
again. \o/

--
Ferenc Kovács
@Tyr43l - http://tyrael.hu

Hi.

I would like to set up a windows slave also, as I mentioned in the RFC.
I was informed that oti1 is mostly idle, so if somebody could give me
access there, I would set up a jenkins slave, and the neccessary
dependencies there.
Thanks!

--
Ferenc Kovács
@Tyr43l - http://tyrael.hu

Yesterday we were talking about enabling more extensions and adding pecl
into the mix:
internals@lists.php.net/msg54220.html" rel="nofollow" target="_blank">http://www.mail-archive.com/internals@lists.php.net/msg54220.html
I think it would be a good idea to try to fix the currently failing tests.
http://ci.qa.php.net/job/php-src-5.3-matrix-tests/35/testReport/?
http://ci.qa.php.net/job/php-src-5.4-matrix-tests/88/testReport/?
http://ci.qa.php.net/job/php-src-trunk-matrix-tests/52/testReport/?

There is only three failing tests left on the debian slaves:

For 5.3
ext/phar/tests/phar_oo_005.phpt.Phar and RecursiveDirectoryIterator 0.01 20
ext/spl/tests/bug60082.phpt.Bug #60082 (100% CPU / when using references
with ArrayObject(&$ref))

For 5.4 and trunk
ext/standard/tests/general_functions/bug39322.phpt.Bug #39322
(proc_terminate() loosing process resource)

For the freebsd failures, some of them can be tackled from the test (the
locale related tests), but I see some things that need fixing in the code,
rather than in the tests:
https://bugs.php.net/bug.php?id=60186
https://bugs.php.net/bug.php?id=60222

I think I also come up with a solution for the xfail issue (where do we
count those):
I would add support to run-tests to be able only run the XFAILS, or skip
them.
This way I could run the testsuite twice(either adding another axis to the
matrix config, or duplicating the php-src-$version-matrix-tests jobs in
two), and we could get a distinct report about the failing tests and the
xfails.
Having an xfail would mean a test success(in the junit.xml), having an
xfail test pass would be a fail((in the junit.xml)) but wouldn't fail the
build, only make it unstable.
This would mean that if we have failing tests ( not xfail tests, but tests
which are expected to pass failing instead)
our php-src-$version-matrix-tests build would result in a failure (red
ball).
Having only passing xfails, and no other failure would make our
php-src-$version-matrix-tests would result in an unstable build (yellow
ball).
Having no xfail tests present, and everything else passing would result
in successful and stable build (blue ball).

--
Ferenc Kovács
@Tyr43l - http://tyrael.hu

13 years ago by Dirk Haun — view source

unread

Quoting Ferenc Kovacs tyra3l@gmail.com:

There is only three failing tests left on the debian slaves:

For 5.3
ext/phar/tests/phar_oo_005.phpt.Phar and RecursiveDirectoryIterator 0.01 20
ext/spl/tests/bug60082.phpt.Bug #60082 (100% CPU / when using references
with ArrayObject(&$ref))

FWIW, the test for Bug #60082 also seems to let the PHP CLI running,
even after "make test" has finished. I've reported this here:
https://bugs.php.net/bug.php?id=60225

bye, Dirk

13 years ago by Ferenc Kovacs — view source

unread

Quoting Ferenc Kovacs tyra3l@gmail.com:

There is only three failing tests left on the debian slaves:

For 5.3
ext/phar/tests/phar_oo_005.**phpt.Phar and RecursiveDirectoryIterator
0.01 20
ext/spl/tests/bug60082.phpt.**Bug #60082 (100% CPU / when using
references
with ArrayObject(&$ref))

FWIW, the test for Bug #60082 also seems to let the PHP CLI running, even
after "make test" has finished. I've reported this here:
https://bugs.php.net/bug.php?**id=60225 https://bugs.php.net/bug.php?id=60225

The run-tests.php should kill it when it timeouts(it does a proc_terminate).
On FreeBSD I experienced that the timeout never happen, but in you case it
seems it does, but it seems that the proc_terminate didn't killed the test.
Maybe that is related to ext/standard/tests/general_functions/bug39322.phpt
"Bug #39322 (proc_terminate() loosing process resource)"
Albeit that (https://bugs.php.net/bug.php?id=39322) should be fixed,
according to Rasmus.
Rasmus do you have any idea why is that test failing on 5.4 and trunk for
debian x86?
see:
http://ci.qa.php.net/job/php-src-5.4-matrix-tests/88/architecture=x86,os=linux-debian-6.0/testReport/ext_standard_tests_general_functions_bug39322/phpt/Bug__39322__proc_terminate___loosing_process_resource_/
http://ci.qa.php.net/job/php-src-trunk-matrix-tests/52/architecture=x86,os=linux-debian-6.0/testReport/ext_standard_tests_general_functions_bug39322/phpt/Bug__39322__proc_terminate___loosing_process_resource_/

Dirk: could you please update your bug report with more info (which disto,
output of uname -a) as I didn't see that happening on the machines that I'm
building php on.

Ferenc Kovács
@Tyr43l - http://tyrael.hu

13 years ago by Ferenc Kovacs — view source

unread

Yesterday we were talking about enabling more extensions and adding pecl

into the mix:
internals@lists.php.net/msg54220.html" rel="nofollow" target="_blank">http://www.mail-archive.com/internals@lists.php.net/msg54220.html
I think it would be a good idea to try to fix the currently failing tests.
http://ci.qa.php.net/job/php-src-5.3-matrix-tests/35/testReport/?
http://ci.qa.php.net/job/php-src-5.4-matrix-tests/88/testReport/?
http://ci.qa.php.net/job/php-src-trunk-matrix-tests/52/testReport/?

There is only three failing tests left on the debian slaves:

For 5.3
ext/phar/tests/phar_oo_005.phpt.Phar and RecursiveDirectoryIterator 0.01
20
ext/spl/tests/bug60082.phpt.Bug #60082 (100% CPU / when using references
with ArrayObject(&$ref))

For 5.4 and trunk
ext/standard/tests/general_functions/bug39322.phpt.Bug #39322
(proc_terminate() loosing process resource)

For the freebsd failures, some of them can be tackled from the test (the
locale related tests), but I see some things that need fixing in the code,
rather than in the tests:
https://bugs.php.net/bug.php?id=60186
https://bugs.php.net/bug.php?id=60222

I think I also come up with a solution for the xfail issue (where do we
count those):
I would add support to run-tests to be able only run the XFAILS, or skip
them.
This way I could run the testsuite twice(either adding another axis to the
matrix config, or duplicating the php-src-$version-matrix-tests jobs in
two), and we could get a distinct report about the failing tests and the
xfails.
Having an xfail would mean a test success(in the junit.xml), having an
xfail test pass would be a fail((in the junit.xml)) but wouldn't fail the
build, only make it unstable.
This would mean that if we have failing tests ( not xfail tests, but tests
which are expected to pass failing instead)
our php-src-$version-matrix-tests build would result in a failure (red
ball).
Having only passing xfails, and no other failure would make our
php-src-$version-matrix-tests would result in an unstable build (yellow
ball).
Having no xfail tests present, and everything else passing would result
in successful and stable build (blue ball).

--
Ferenc Kovács
@Tyr43l - http://tyrael.hu

Hi.

A quick update:

Notification mails:

As you probably remember, I mentioned that currently jenkins can only send
notification emails for each configuration build (debian x86, debian
x86_64, freebsd x86 etc.) which isn't what we want.
In the meantime I poked people on irc, and got
https://issues.jenkins-ci.org/browse/JENKINS-8590 fixed.
So that is a good news, it means that we can send email notifications per
upstream build, not for each of the configuration builds.
The bad news is that the latest email-ext plugin requires a higher jenkins
version than the current LTS, to make it even worst, the
soon-to-be-released next LTS will be lower than necessary.
So I have to do some scripting to set up the mail notifications.
I think we all agreed in the following:

we only send out mails for the aggregated results, not for each
configuration build.
the commiter(s) for the build will always be notified about the result of
the compilation and test build results.
if there are changes in the result of the build compared to the previous
build, the list will be also notified:
-- the build failed
-- the build recovered
-- there are "new" failing tests
-- previously failing tests are passing

does that sound reasonable?
(some scripting also required for the test change notification)

Windows:

It seems that we will have some windows slaves offered by OmniTI (thanks to
Elizabeth Marie Smith).

FreeBSD:

As I mentioned previously, the IO perf is really bad for those two vm's, so
if we can't find a solution, or other place to run those, I will have to
turn those off, as they really slows down the build(they are like 5-10
times slower, deleting a 100M file takes seconds, checking out the php
source for a branch take literally hours).
I will ran another round with the hoster, maybe we can still fix that.

Other platforms:

There were some suggestions that we should set up other slaves like Fedora,
RHEL, etc. I think it is a good idea to support at least the major distros,
but we have to still find a place to run those.

Speeding up the build process:

I'm also looking into other options to speed up the compilation but most
importantly the testsuite builds.
An obvious solution for that problem would be to not run each test in the
testrun sequentially, but allow it to be executed in parallel.
This isn't the first time, that this is brought up, we even have some code
for it, see:
https://wiki.php.net/qa/runtests
http://svn.php.net/viewvc/php/phpruntests/trunk/

I plan to look into that, how far is that from the completion. I think that
the current run-tests.php is too messy, so adding parallel test execution
support there would require a major cleanup/refactoring, but it is possible.

Improving the visualization of the test results

I plan to change my current patch for run-tests.php so that the tests will
be groupped into testsuites(Zend, ext/mysql, ext/session, etc.) so the All
Tests list won't be that cluttered with thousands of lines as it is
currently:
http://ci.qa.php.net/job/php-src-5.3-matrix-tests/79/architecture=x86,os=linux-debian-6.0/testReport/?

XFAIL separation:

I already mentioned this, the concept of XFAILS isn't really fit into the
generic test result concept, but I think that we should somehow separate
those from the other tests.
My idea is that the XFAILs would be logged into a separate testsuite(even
separat junit.xml), so that we can handle those separately from the other
test results.

Support more extensions:

Currently only the standard exts are compiled and tested, the only
impediment extending that is currently we use the same build target for all
configuration job, so if we enable a new extension, it should be available
on all slave.
This and the soon arriving windows slaves brings us to the next point.

Different build target for each configuration job/platform.

As I mentioned in the previous in the previous task, currently we use the
same build target for every platform, which obviously isn't feasible on the
long run.
I plan to extend our ant build.xml so that it can executes a separate
target based on the OS/platform, this way we can use the same job
configuration in jenkins, but the platforms can have
different capabilities or means to execute the build.

gcov/static analysis/coverage

I mentioned in the original RFC, that it would be possible to use Jenkins
for static analysis also, and from the feedback it seems that many of you
think that it would be a good idea.
Personally I see this a little bit lower priority as the other tasks, as we
have working (albeit not perfect) solutions for the static analysis stuff,
but I will look into this as well, and I will poke Nuno the
author/maintainer of gcov.php.net.
(and static analysis tend to be pretty resource intensive so to tell the
truth we don't really have the place to run those kind of jobs. yet.)

adding other php projects:

PEAR: they guys at PEAR are working on fixing up the tests for PEAR,
Daniel Convissor told me that I should wait that before putting that up.
Other projects: I have some ideas (Horde, ZF, ZF2, Doctrine, Propel,
PHPUnit, etc.) what would make sense to add, but I would be curious what
would you see to be tested against our snapshots.

Thats it folks, I will update the wiki accordingly, if anybody has any
question (or would like to join the efforts) feel free to shot me here or
on #php.qa on efnet.

ps: I was really busy recently with my day-job and other stuff, but I will
try to do as much as I can, but if someone willing to help me out with some
of the tasks mentioned here (or some other cool idea) please step up! :)

--
Ferenc Kovács
@Tyr43l - http://tyrael.hu

13 years ago by Ferenc Kovacs — view source

unread

Hi,

Recent activity:

Alexey Shein dug through the failing symfony2 tests, and it turned out that
3 tests were bugged, he went ahead and fixed it (
https://github.com/symfony/symfony/pull/2750), one from the other two is a
sporadic test, it can fail randomly, the last one fails in our specific
setup (we don't do make install so their test won't find our php binary
at PHP_BINDIR).

The new Jenkins LTS got released, so I went ahead and updated our cluster,
there were a couple of hiccups, but everything should be back to normal now.

I also simplified our setup, I merged the build and test jobs, this will
make our life much easier.
This also made possible to simplify our build.xml, and I used that chance
to complicate it a little bit more :P
I added a condition and an environment variable to use gmake (gnu make) on
FreeBSD instead of make, which also made it possible to add -j2(a little
bit concurrency) to our make process.

The biggest news is that we polished up my original JUnit patch for
run-tests.php and with the help of Felipe, we have that in the svn
repository.
Felipe, Alexey, thanks again!

--
Ferenc Kovács
@Tyr43l - http://tyrael.hu

13 years ago by Matteo Beccati — view source

unread

Hi,

FreeBSD:

As I mentioned previously, the IO perf is really bad for those two vm's, so
if we can't find a solution, or other place to run those, I will have to
turn those off, as they really slows down the build(they are like 5-10
times slower, deleting a 100M file takes seconds, checking out the php
source for a branch take literally hours).
I will ran another round with the hoster, maybe we can still fix that.

I do have FreeBSD 6.2 amd64 box that is mostly idle. I'm using Atlassian
Bamboo for my own builds, and I don't know Jenkins, but pls let me know
if I could help.

Cheers

Matteo Beccati

Development & Consulting - http://www.beccati.com/

13 years ago by Ferenc Kovacs — view source

unread

Hi,

FreeBSD:

As I mentioned previously, the IO perf is really bad for those two vm's,
so
if we can't find a solution, or other place to run those, I will have to
turn those off, as they really slows down the build(they are like 5-10
times slower, deleting a 100M file takes seconds, checking out the php
source for a branch take literally hours).
I will ran another round with the hoster, maybe we can still fix that.

I do have FreeBSD 6.2 amd64 box that is mostly idle. I'm using Atlassian
Bamboo for my own builds, and I don't know Jenkins, but pls let me know
if I could help.

Hi Matteo,

What we would really need is a(two) server with some recent FreeBSD version
(6.2 is a little bit old), it seems that virtualization and Freebsd doesn't
mix well yet (albeit there seems to be some experimantal improvements
http://lists.freebsd.org/pipermail/freebsd-current/2011-January/022036.html)
We will try to sort this out with the current hosting provider of that
server, but if that doesn't work out, we would be more than happy to accept
a generous offer. :P

--
Ferenc Kovács
@Tyr43l - http://tyrael.hu

13 years ago by Pierre Joye — view source

unread

hi Stas,

A new box is being setup with numerous VMs (Dan is on it). Each of of
them will cover one supported platform (debian, rh, bsd, win) and will
run the builds and tests using rmtools in controlled environment, like
what I began to do with windows a while back already. Build and tests
log will be available more than daily (will see how fast the tests can
be run, but 3-4x daily if necessary should be possible).

Code coverage won't be available for all run tho', as it slow down the
process and it is not necessary on each revision or change :)

Cheers,

Hi!

Since we started to pay real attention to our unit tests now, I wonder if we
could set up some kind of frequently-running CI system that could be used to
screen commits and identify breakage early? That'd help with 5.4 process I
think.
We have http://gcov.php.net/ but it doesn't run with the frequency I'd like
and since it says the run takes 44 hours it's kind of understandable. So I
wonder if we could have something that just builds it and runs unit tests
and we could see it in the same format as on gcov? Ideally after each commit
would be nice, but say once an hour or two (even fullest unit tests run
should take more than that, I think) would be OK too. If we could have two
of them, like Linux & Windows, it'd be even better, but at least one would
be nice.
What do you think?

Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227

--

--
Pierre

@pierrejoye | http://blog.thepimp.net | http://www.libgd.org

Dirk: could you please update your bug report with more info (which disto, output of uname -a) as I didn't see that happening on the machines that I'm building php on.

Cheers

Dirk: could you please update your bug report with more info (which disto,
output of uname -a) as I didn't see that happening on the machines that I'm
building php on.