Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:28374
Mailing-List: contact internals-help@lists.php.net; run by ezmlm
Received-SPF: error (pb1.pair.com: domain php.net from 85.214.94.56 cause and error)
Date: Fri, 16 Mar 2007 18:39:14 +0100
Reply-To: Marcus Boerger <helly@php.net>
Message-ID: <755864469.20070316183914@marcus-boerger.de>
To: Zoe Slattery <zoe.slattery@googlemail.com>
CC: internals@lists.php.net
In-Reply-To: <45FAA90C.7070006@googlemail.com>
References: <45F8173F.3010902@googlemail.com> <622207866.20070314203432@marcus-boerger.de> <45FAA90C.7070006@googlemail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Subject: Re: [PHP-DEV] php.ini and PHPT
From: helly@php.net (Marcus Boerger)

Hello Zoe,

Friday, March 16, 2007, 3:26:20 PM, you wrote:

> Hi Marcus

> Thanks - that answers the question ...and test cases are on their way:-)

> I have just committed 3 tests in the PHP 6 stream (strcasecmp, strcmp,=20
> strlen) and would very much appreciate any feedback on them. We are=20
> aiming for both line coverage and functional coverage, it would be good=
=20
> to know if we are missing obvious things.

Well line coverage can be seen by gcov.php.net. Domain coverage can't hardly
be detcted.

From=20what I can see you did a good job of trying to capture all kinds of
input to those functions. You also used echo lines to group the test blocks
which makes it easier to find failing locations.

Then I saw you changed a hell lot of expectations from "(-%d)" to "(%i)".
While this pretty much works it drops an important detail of the expectatio=
n.
The old version was aware of the sign of the result while the new one is no=
t.
That means that now you might fix a bug in those functions that lead to an
erroneous change in the sign and not being able to detect that. Though a lot
os code is relying on the sign (e.g. any sorting algorithm).

> For the future I think we will probably split out the test cases into=20
> good-path tests and error-path tests, putting both into one test case=20
> has resulted in the test cases being quite big.

That sounds like a good idea. Long expect sections make it hard to identify
the spot that causes the test failure.

> I have one specific question though, I notice that the string that=20
> follows Warning: is changing in PHP6, for example in PHP5 one would expec=
t:

> Warning: Wrong parameter count for strcasecmp() in blah.php on line 3

> In PHP6:

> Warning: strcasecmp() expects exactly 2 parameters, 0 given in blah.php=
=20
> on line 3

> The PHP6 message is clearly an improvement and the test cases that I=20
> have just put in PHP6 check the Warning: like this:

> --EXPECTF--
> Warning: strcasecmp() expects exactly 2 parameters, 0 given in %s on line=
 %d

> But of course, this will only pass on PHP6.  We can deal with this in=20
> two ways:

> (a) Use "Warning: %s" in the EXPECTF section. This will pass test on=20
> PHP5 and PHP6 but it won't check that PHP is giving the right error=20
> mesage - ie "exactly 2 parameters" as opposed to "exactly 42 parameters"

> (b) We can have different test cases for PHP5 and PHP6.

> I don't particularly like either of these, but have a slight preference
> for (b) because  it's a more satisfactory test of the implementation.

> What's your view? Or anyone else's?

Do b) and be as precise as possible. That means we usually even test for the
name of the testfile in the message excluding the directory and any
(back)slash.

best regards
marcus

> Marcus Boerger wrote:
>> Hello Zoe,
>>
>>   we decided to go the --INI-- route to test as many configurations as
>> possible. And only for specific tests change specific ini settings as
>> necessary. If one usees phpt as a test framework for his on web stuff
>> then it might be a good idea to do as you suggested. You would simply
>> do "php run-tests.php -c <ini-file>". Hope this satisfies you :-)
>>
>> best regards
>> marcus
>>
>> p.s.: Looking forward for new tests from your side
>>
>> Wednesday, March 14, 2007, 4:39:43 PM, you wrote:
>>
>>  =20
>>> Hi - I wonder if anyone could help me with a testing question?
>>>    =20
>>
>>  =20
>>> We have written a few additional test cases for string functions which =
I=20
>>> was about to check-in, then I noticed that the test cases pass OK if I=
=20
>>> use the php.ini-dist file and fail if I don't specify an file (ie use=
=20
>>> all default settings).
>>>    =20
>>
>>  =20
>>> The difference turns out to the be the use of precision =3D12 in the=20
>>> php.ini-dist, rather than the the default value of 14.
>>>    =20
>>
>>  =20
>>> It's easy enough to work around this and ensure that test cases pass in=
=20
>>> both environments with an --INI-- section in the test case, however - I=
=20
>>> wondered if there would be any value in having a php-ini file that was=
=20
>>> explicitly used for tests?
>>>    =20
>>
>>  =20
>>> We had been using the php.ini-dist file because it's slightly easier to=
=20
>>> be precise about the test environment if you can say what you are using=
=20
>>> rather than saying "I'm using the default values". I also find it rathe=
r=20
>>> too easy to get the wrong php.ini file accidentally :-( so specifying=
=20
>>> one is easier, even if it only contains default values.

Best regards,
 Marcus