[RFC] keeping unit tests green

11 years ago by Sebastian Bergmann — view source

unread

Am 14.04.2014 19:56, schrieb Stas Malyshev:

As promised, I have created an RFC on unit test policy, here:
https://wiki.php.net/rfc/travis_ci

Nitpicking: To the best of my knowledge, PHP does not have a single
unit test. PHPT tests, of which we have many, are integration tests.

11 years ago by Stas Malyshev — view source

unread

Hi!

Nitpicking: To the best of my knowledge, PHP does not have a single
unit test. PHPT tests, of which we have many, are integration tests.

OK, I'll change the RFC to reflect that.

--
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227

11 years ago by Florian Anderiasch — view source

unread

Am 14.04.2014 19:56, schrieb Stas Malyshev:

As promised, I have created an RFC on unit test policy, here:
https://wiki.php.net/rfc/travis_ci

Nitpicking: To the best of my knowledge, PHP does not have a single
unit test. PHPT tests, of which we have many, are integration tests.

If
https://github.com/php/php-src/blob/master/ext/standard/tests/strings/bin2hex.phpt
(and most of the other function tests) is not a unit test I'm not sure
how you would set the requirements.

Greetings,
Florian

11 years ago by Ferenc Kovacs — view source

unread

On Tue, Apr 15, 2014 at 10:46 AM, Florian Anderiasch ml@anderiasch.dewrote:

Am 14.04.2014 19:56, schrieb Stas Malyshev:

As promised, I have created an RFC on unit test policy, here:
https://wiki.php.net/rfc/travis_ci

Nitpicking: To the best of my knowledge, PHP does not have a single
unit test. PHPT tests, of which we have many, are integration tests.

If

https://github.com/php/php-src/blob/master/ext/standard/tests/strings/bin2hex.phpt
(and most of the other function tests) is not a unit test I'm not sure
how you would set the requirements.

Greetings,
Florian

--

they are still testing our PHP_FUNCTION() macro, and
zend_parse_parameters(), etc. instead of testing the php_bin2hex() function.
from the userland POV, they could be seen as unit tests, but they still
test the given unit from a mile away through multiple layers of cruft.

--
Ferenc Kovács
@Tyr43l - http://tyrael.hu

11 years ago by Sebastian Bergmann — view source

unread

Am 15.04.2014 10:46, schrieb Florian Anderiasch:

If
https://github.com/php/php-src/blob/master/ext/standard/tests/strings/bin2hex.phpt
(and most of the other function tests) is not a unit test I'm not sure
how you would set the requirements.

A PHPT test is a PHP script that is executed using the PHP interpreter.
This constitutes an integration test. A unit test would be a test that
is written in C that exercises a single C unit of code in isolation
from the rest of the interpreter.

11 years ago by Julien Pauli — view source

unread

Hi!

As promised, I have created an RFC on unit test policy, here:
https://wiki.php.net/rfc/travis_ci

Please discuss. Any comments/suggestions/corrections welcome.

Hi,

This looks like something we already do nowadays, except we (well, I)
don't go to watch for the CI but look after the tests in the branch
I'm as RM managing.

So I'm OK, as soon as it is simple.
For example, I dont know Travis. So when opening the Travis page about
PHP, I see many things, red or green, and feel like lost in all those
informations.
I don't know where to look to find quickly that something went wrong, and why.
So if we could have on the wiki a page explaining how to manage Travis
builds, I would be more than happy :-)

Thx.

Julien.P

11 years ago by Crypto Compress — view source

unread

Hi,

So I'm OK, as soon as it is simple.
For example, I dont know Travis. So when opening the Travis page about
PHP, I see many things, red or green, and feel like lost in all those
informations.

Travis-CI is as trivial as this:

a (github) notification hook queue a testrun
as resources are free a preconfigured vm starts and
all the things in .travis.yml are executed

As a RM you can let the VM execute whatever you want and how you want.

https://github.com/php/php-src/blob/master/.travis.yml
http://docs.travis-ci.com/user/languages/cpp/

11 years ago by Crypto Compress — view source

unread

Am 15.04.2014 14:02, schrieb Crypto Compress:

Hi,

So I'm OK, as soon as it is simple.
For example, I dont know Travis. So when opening the Travis page about
PHP, I see many things, red or green, and feel like lost in all those
informations.

Travis-CI is as trivial as this:

a (github) notification hook queue a testrun

as resources are free a preconfigured vm starts and

all the things in .travis.yml are executed

As a RM you can let the VM execute whatever you want and how you want.

https://github.com/php/php-src/blob/master/.travis.yml
http://docs.travis-ci.com/user/languages/cpp/
ops: http://docs.travis-ci.com/user/languages/c/

11 years ago by Stas Malyshev — view source

unread

Hi!

So I'm OK, as soon as it is simple.
For example, I dont know Travis. So when opening the Travis page about
PHP, I see many things, red or green, and feel like lost in all those
informations.

Travis is pretty simple as such - you go to
https://travis-ci.org/php/php-src/builds and check out your branch
(branch name is in Commit column). If it's not green, you click on the
build number and get the log which says what went wrong. The failed
tests are usually at the very end, and once you know the names of the
tests failed, look them up in the log to find the details.

With pulls on github, you just click on "Details" link and it gets you
to the same log page.

Setting up a fork requires some configs, but Travis docs are good
enough. I might also add a wiki page on how to set up the fork in Travis
a bit later.

--
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227

11 years ago by Julien Pauli — view source

unread

Hi!

So I'm OK, as soon as it is simple.
For example, I dont know Travis. So when opening the Travis page about
PHP, I see many things, red or green, and feel like lost in all those
informations.

Travis is pretty simple as such - you go to
https://travis-ci.org/php/php-src/builds and check out your branch
(branch name is in Commit column). If it's not green, you click on the
build number and get the log which says what went wrong. The failed
tests are usually at the very end, and once you know the names of the
tests failed, look them up in the log to find the details.

With pulls on github, you just click on "Details" link and it gets you
to the same log page.

Setting up a fork requires some configs, but Travis docs are good
enough. I might also add a wiki page on how to set up the fork in Travis
a bit later.

Yes I spent some time to see how it works, it seems pretty easy.

In fact, if we could add more analyze tools, it would be nice, because
actually, it just acts as an autommated "make test".

Julien

11 years ago by Stas Malyshev — view source

unread

Hi!

In fact, if we could add more analyze tools, it would be nice, because
actually, it just acts as an autommated "make test".

Yes, for now. We can definitely expand it. Only downside of simplicity
is that we have basic yes/no trigger and build artifacts are only
available as a log AFAIK. So for some tools it may not be the best and
Jenkins (which has more extensive reporting and build artifacts support)
could be better. But suggestions of course are most welcome, and we can
put anything we want in .travis.yml if Travis supports it.

Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227

11 years ago by Pierre Joye — view source

unread

Hi Stas,

Hi!

As promised, I have created an RFC on unit test policy, here:
https://wiki.php.net/rfc/travis_ci

Please discuss. Any comments/suggestions/corrections welcome.

I very much like this idea, a must do.

I also like to extend it more:

for stable branches, I would like a staging branche where all of us
can commit, RM can then cherry pick stable patches for a given branche
add static analyzers to the staging branch (gcc, clang, VC and PVS,
that will covers much issues including vicious 64bit issues
. it could be possible to do it per commit
try as much as possible to get warning free commits, linux and
partially windows. Windows may have false positive while it will be
drastically reduced once we have the int64 patch in

Doing so we will always be able to do security releases very
efficiently and safely

I do not care much about the wording, unit tests vs whatever the name is :)

Cheers,

Pierre

@pierrejoye | http://www.libgd.org

11 years ago by Stas Malyshev — view source

unread

Hi!

for stable branches, I would like a staging branche where all of us
can commit, RM can then cherry pick stable patches for a given branche

Travis does pull requests already, and you can hook up your fork to it
easily so it would test any branch you push into your fork. E.g. see
https://travis-ci.org/smalyshev/php-src/builds

add static analyzers to the staging branch (gcc, clang, VC and PVS,
that will covers much issues including vicious 64bit issues
. it could be possible to do it per commit

That would be most welcome, not sure how easy/hard it is to do it. If we
get our own Jenkins running and stable, that would be of course possible.

--
Stanislav Malyshev, Software Architect
SugarCRM: http://www.sugarcrm.com/
(408)454-6900 ext. 227

11 years ago by Stas Malyshev — view source

unread

Hi!

Speaking on Windows, Travis does not support it - so what could be use
instead?

As for warnings, I was planning to take a look into this soon, clang has a
lot of things to complain about our code :)