Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:47368
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: pass (pb1.pair.com: domain googlemail.com designates 72.14.220.155 as permitted sender)
DomainKey-Status: bad
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=cc:message-id:from:to:in-reply-to:content-type
         :content-transfer-encoding:mime-version:subject:date:references
         :x-mailer;
        b=cK4wN6pABrVYe7P8Qtw4uW36EpA9ZSERTsOetWz+QfF2NyF1RaSWrcUrtsLal1RK0/
         fJ9tqcvUK0NICFFf9QxasyDc7nlUukv5hC2F3PhoqapAIUr2PeCUt5B/JUaGq4IpgMhs
         DrDOKAB4pN5SFd5lAUA1ImHUQlSaA+NGF8czo=
Cc: Alexey Zakhlestin <indeyets@gmail.com>,
 jani.taskinen@iki.fi,
 Stanislav Malyshev <stas@zend.com>,
 internals@lists.php.net
Message-ID: <7AB41029-94CC-4E02-B33F-09C460331C43@googlemail.com>
To: Pierre Joye <pierre.php@gmail.com>
In-Reply-To: <fe05d1541003121123k30300f35o3b7305b2b6b591d9@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed; delsp=yes
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (Apple Message framework v936)
Date: Wed, 17 Mar 2010 00:05:27 +0100
References: <4B9926E8.4080202@lerdorf.com> <4B995F83.1000605@prohost.org> <4B99639B.9030405@oracle.com> <4B996FC6.7040102@zend.com> <4B997381.1050300@oracle.com> <4B9A14FF.6000703@iki.fi> <4B9A833A.8040402@zend.com> <4B9A848C.3080701@iki.fi> <fe05d1541003121106g1e5f0251hbfccedf0e4ffac1f@mail.gmail.com> <F920F09D-893F-43E0-A2D7-49D12410DBEF@gmail.com> <fe05d1541003121123k30300f35o3b7305b2b6b591d9@mail.gmail.com>
Subject: Re: [PHP-DEV] Tests repository
From: foolistbar@googlemail.com (Geoffrey Sneddon)


On 12 Mar 2010, at 20:23, Pierre Joye wrote:

> On Fri, Mar 12, 2010 at 8:10 PM, Alexey Zakhlestin =20
> <indeyets@gmail.com> wrote:
>>
>> On 12.03.2010, at 22:06, Pierre Joye wrote:
>>
>>> Many tests fail because they are written for a given platform, or =20=

>>> even
>>> worst (from a portability point of view), only for a given
>>> configuration (library version, system version,etc.). And that's not
>>> about windows vs other, tests can work on a Debian/ubuntu version =20=

>>> and
>>> fail on another.
>>
>> Well, these tests should just be removed/rewritten.
>> Php-tests should test php, not libraries
>
> I would be interested to know how can we test the file API without
> testing libc, for example.
>
> But yes, that's the idea. Redesign the tests in a way that we output
> something when the results of the test is not what was expected,
> instead of testing random outputs from random libraries (in various
> versions and flavours). No big change in run-tests (none?) but a lot
> of necessary changes in existing tests.

To prefix the following comment: I've spent the previous six months =20
working full-time on Opera's new ECMAScript (i.e., JavaScript) engine, =20=

and have spent a fair bit of time looking at ECMAScript test suites.

I'm somewhat doubtful about how we can ever test PHP that well: by and =20=

large PHP-the-language is defined by PHP-the-interpreter; as long as =20
this is the case, how can a test have a pass/fail condition: the =20
interpreter is by definition correct as it is the definition of the =20
language you are testing?

To take an example from ECMAScript, Math.sin is defined as, "Returns =20
an implementation-dependent approximation to the sine of x. The =20
argument is expressed in radians." This is followed by a definition =20
which defines exact return values for the case of the argument being =20
=B10, =B1Inf, or NaN. The intention of the ECMAScript spec saying such a =
=20
thing is /exactly/ that of using system libraries: it is designed so =20
that you can use whatever the sin function of the maths library you =20
are using returns.

In the PHP manual, sin is defined as, "sin() returns the sine of the =20
arg parameter. The arg parameter is in radians." This is similar, but =20=

contains one vital difference: the PHP function says it returns the =20
_exact_ sine (or, at the very least, does not permit any variation =20
from it). If the PHP interpreter is to actually follow the manual in =20
this case, it is pretty much impossible for it to use any system =20
library, as they could easily vary from the value of sine, which the =20
manual does not permit.

If nothing else, I think PHP testing would become a lot easier if we =20
had some clear specification of _what_ PHP was actually expected to =20
do. At the moment, all there is to go by is what is in the manual =20
(almost always far too vague to actually exhaustively test) and what =20
the implementation itself does (and it's never a good situation to be =20=

working out what the implementation /should/ do from what it actually =20=

does).

The first thing that needs to be done with a lot of tests that fail on =20=

certain platforms currently within PHP is work out what result is =20
expected, and that is currently almost impossible for the above =20
reasons. I don't see how we can make any large gains in testing until =20=

we have some clear way of knowing what behaviour is expected.

I'm certainly willing to help spec out more of PHP to the extent =20
necessary to be able to decently test it, if there is interest in =20
doing so.


--
Geoffrey Sneddon
<http://gsnedders.com/>