Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:73820
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: pass (pb1.pair.com: domain gmail.com designates 209.85.192.43 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <535AAC49.9050004@sugarcrm.com>
References: <535A13E1.3050106@sugarcrm.com>
	<CAH-PCH7ZM-RLyYUkCz6OeWMS74zrMiFj4eu8RBLcjTjWuQ8Stg@mail.gmail.com>
	<535AAC49.9050004@sugarcrm.com>
Date: Mon, 28 Apr 2014 01:01:24 +0200
Message-ID: <CAH-PCH7UWSgdZN5O=pRk8C0iF+M-UV8tefKWBWm5K0fs=aWSow@mail.gmail.com>
To: Stas Malyshev <smalyshev@sugarcrm.com>
Cc: PHP Internals <internals@lists.php.net>
Content-Type: multipart/alternative; boundary=001a11c1244c9663f204f80e2de1
Subject: Re: [PHP-DEV] [VOTE] CI tests RFC
From: tyra3l@gmail.com (Ferenc Kovacs)

--001a11c1244c9663f204f80e2de1
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On Fri, Apr 25, 2014 at 8:41 PM, Stas Malyshev <smalyshev@sugarcrm.com>wrot=
e:

> Hi!
>
> > Would be nice if we could also create some policy regarding the usage o=
f
> > XFAILs (when should be a failing test marked as XFAIL, should we add th=
e
> > tests from the open bugreports as XFAIL by default, whose responsibilit=
y
> > is to make sure that it will be fixed eventually, etc.).
>
> I agree. I didn't get into this but it definitely makes sense to have
> some rules there.
>

yeah, I'm glad that we managed to reach a state where we have
0/close-to-zero number of failing tests for the default builds on the major
platforms, but the number of XFAILS are somewhat scarry, and some of there
are pretty old too (there are a couple of them which I remember discussed
to be fixed before 5.4.0 final...).
the reason why we introduced the XFAILs were to be able to distinguish
between the old/known/low-priority problems from new/important test
failures.
but now if we want to keep the failing test numbers on zero, sooner or
later people will start putting XFAILs on everything to satisfy the CI, and
we will end up in the same situation where the less important XFAILS will
cause the important ones to slip under the radar.


>
> > I would also like to extend the current travis config a bit (we could
> > have more exts, more axes for stuff like ts/nts builds, enable debug
> > builds, so memory leaks are also triggering the test failures, etc.),
>
> You're more than welcome :) I've planned to get to some of it next -
> i.e. going through the list of exts and see which ones we can support on
> Travis - but any help would be great.
>

I've created a PR with some travis related changes:
https://github.com/php/php-src/pull/654

I've also looked into having both 32 and 64 bit builds, but it seems travis
only supports 64bit platforms atm:
https://github.com/travis-ci/travis-ci/issues/986

and also looked into enabling the email notifications, but it seems that it
is a bit complicated:
http://docs.travis-ci.com/user/notifications/#Email-notifications
the default behavior is:
"By default, email notifications will be sent to the committer and the
commit author, if they are members of the repository (that is, they have
push or admin permissions for public repositories, or if they have pull,
push or admin permissions for private repositories).
...
By default, a build email is sent to the committer and the author, but only
if they have access to the repository the commit was pushed to. This
prevents forks active on Travis CI from notifying the upstream repository's
owners when they're pushing any upstream changes to their fork. It also
prevents build notifications from going to folks not registered on Travis
CI.
...
The email address is then determined based on the email address in the
commit, but only if it matches one of the email addresses in our database.
We synchronize all your email addresses from GitHub, solely for the purpose
of build notifications."

which we can't really use, because almost nobody has push/admin rights to
our github repo, and adding an explicit email address (like internals@ or
php-qa@) instead of using the default strategy would mean that any fork
which get's activated on travis will start spamming those lists. :/

--=20
Ferenc Kov=C3=A1cs
@Tyr43l - http://tyrael.hu

--001a11c1244c9663f204f80e2de1--