php.ini and PHPT

Hello Zoe,

Friday, March 16, 2007, 3:26:20 PM, you wrote:

Hi Marcus

Thanks - that answers the question ...and test cases are on their way:-)

I have just committed 3 tests in the PHP 6 stream (strcasecmp, strcmp,
strlen) and would very much appreciate any feedback on them. We are
aiming for both line coverage and functional coverage, it would be good
to know if we are missing obvious things.

Well line coverage can be seen by gcov.php.net. Domain coverage can't hardly
be detcted.

From what I can see you did a good job of trying to capture all kinds of
input to those functions. You also used echo lines to group the test blocks
which makes it easier to find failing locations.

Then I saw you changed a hell lot of expectations from "(-%d)" to "(%i)".
While this pretty much works it drops an important detail of the expectation.
The old version was aware of the sign of the result while the new one is not.
That means that now you might fix a bug in those functions that lead to an
erroneous change in the sign and not being able to detect that. Though a lot
os code is relying on the sign (e.g. any sorting algorithm).

For the future I think we will probably split out the test cases into
good-path tests and error-path tests, putting both into one test case
has resulted in the test cases being quite big.

That sounds like a good idea. Long expect sections make it hard to identify
the spot that causes the test failure.

I have one specific question though, I notice that the string that
follows Warning: is changing in PHP6, for example in PHP5 one would expect:

Warning: Wrong parameter count for strcasecmp() in blah.php on line 3

In PHP6:

Warning: strcasecmp() expects exactly 2 parameters, 0 given in blah.php
on line 3

The PHP6 message is clearly an improvement and the test cases that I
have just put in PHP6 check the Warning: like this:

--EXPECTF--
Warning: strcasecmp() expects exactly 2 parameters, 0 given in %s on line %d

But of course, this will only pass on PHP6. We can deal with this in
two ways:

(a) Use "Warning: %s" in the EXPECTF section. This will pass test on
PHP5 and PHP6 but it won't check that PHP is giving the right error
mesage - ie "exactly 2 parameters" as opposed to "exactly 42 parameters"

(b) We can have different test cases for PHP5 and PHP6.

I don't particularly like either of these, but have a slight preference
for (b) because it's a more satisfactory test of the implementation.

What's your view? Or anyone else's?

Do b) and be as precise as possible. That means we usually even test for the
name of the testfile in the message excluding the directory and any
(back)slash.

best regards
marcus

Marcus Boerger wrote:

Hello Zoe,

we decided to go the --INI-- route to test as many configurations as
possible. And only for specific tests change specific ini settings as
necessary. If one usees phpt as a test framework for his on web stuff
then it might be a good idea to do as you suggested. You would simply
do "php run-tests.php -c <ini-file>". Hope this satisfies you :-)

best regards
marcus

p.s.: Looking forward for new tests from your side

Wednesday, March 14, 2007, 4:39:43 PM, you wrote:

Hi - I wonder if anyone could help me with a testing question?

We have written a few additional test cases for string functions which I
was about to check-in, then I noticed that the test cases pass OK if I
use the php.ini-dist file and fail if I don't specify an file (ie use
all default settings).

The difference turns out to the be the use of precision =12 in the
php.ini-dist, rather than the the default value of 14.

It's easy enough to work around this and ensure that test cases pass in
both environments with an --INI-- section in the test case, however - I
wondered if there would be any value in having a php-ini file that was
explicitly used for tests?

We had been using the php.ini-dist file because it's slightly easier to
be precise about the test environment if you can say what you are using
rather than saying "I'm using the default values". I also find it rather
too easy to get the wrong php.ini file accidentally :-( so specifying
one is easier, even if it only contains default values.

Best regards,
Marcus