Tests - and a question

18 years ago by Zoe Slattery — view source

unread

Hi all

We added a few tests for array, string and variable handling functions
at the weekend. As always - any feedback will be welcome.

One thing we had some difficulty with was keeping the test output small
enough to be easily read. We chose to divide up tests like this:

test_name_b.phpt b=basic function
test_name_v.phpt v=variation
test_name_e.phpt e=error cases
test_name_o.phpt o=object specific

This may not be the right scheme and I'm happy to change it if anyone
has better suggestions, although it is useful to seperate out the error
case tests because quite a lot of the differences between php5 and php6
seem to be in the exact form of the error message. Using this doesn't
completely solve the problem anyway - I think some of the "v" tests will
will end up being test_name_v1, test_name_v2 etc.

Thanks, Zoe
IBM UK

18 years ago by Antony Dovgal — view source

unread

Hi all

We added a few tests for array, string and variable handling functions
at the weekend. As always - any feedback will be welcome.

One thing we had some difficulty with was keeping the test output small
enough to be easily read. We chose to divide up tests like this:

test_name_b.phpt b=basic function
test_name_v.phpt v=variation
test_name_e.phpt e=error cases
test_name_o.phpt o=object specific

That would work for me, though I don't really see much sense in this.
001.phpt, test_name_001.phpt, func_1.phpt, test_name_v.phpt - they're all cryptic enough
and are not supposed to carry any information.
I believe they're just filenames and the real test name is in the --TEST-- section.

This may not be the right scheme and I'm happy to change it if anyone
has better suggestions, although it is useful to seperate out the error
case tests because quite a lot of the differences between php5 and php6
seem to be in the exact form of the error message. Using this doesn't
completely solve the problem anyway - I think some of the "v" tests will
will end up being test_name_v1, test_name_v2 etc.

I would personally prefer having more small tests with names like "001.phpt" than four 50kb "func_name_001_[b|v|e|0].phpt"..

--
Wbr,
Antony Dovgal

18 years ago by Marcus Boerger — view source

unread

Hello Zoe,

that's pretty fine. Instead of _[bveo][0-9]?.phpt you could do
_(basic|variation|error|object)[0-9]{3}.phpt. Which is a tad bit better.
However those are just filenames helping you to find the correct test
file while developing. For that reason we didn not do so much about
generating those files. We Only made it briefly and suggested a three
digit numbering scheme. After all th ename in --TEST.. is the important
thing.

Thanks for the effort into test writing!!!

best regards
marcus

Monday, May 14, 2007, 6:43:51 PM, you wrote:

Hi all

We added a few tests for array, string and variable handling functions
at the weekend. As always - any feedback will be welcome.

One thing we had some difficulty with was keeping the test output small
enough to be easily read. We chose to divide up tests like this:

test_name_b.phpt b=basic function
test_name_v.phpt v=variation
test_name_e.phpt e=error cases
test_name_o.phpt o=object specific

This may not be the right scheme and I'm happy to change it if anyone
has better suggestions, although it is useful to seperate out the error
case tests because quite a lot of the differences between php5 and php6
seem to be in the exact form of the error message. Using this doesn't
completely solve the problem anyway - I think some of the "v" tests will
will end up being test_name_v1, test_name_v2 etc.

Thanks, Zoe
IBM UK

Best regards,
Marcus

18 years ago by Zoe Slattery — view source

unread

Marcus Boerger wrote:

Hi Marcus & Tony

Hello Zoe,

that's pretty fine. Instead of _[bveo][0-9]?.phpt you could do
_(basic|variation|error|object)[0-9]{3}.phpt. Which is a tad bit better.
However those are just filenames helping you to find the correct test
file while developing. For that reason we didn not do so much about
generating those files. We Only made it briefly and suggested a three
digit numbering scheme. After all th ename in --TEST.. is the important
thing.

Thanks for the responses - you have both made the same point that it's
the --TEST-- section that should really give a descriptive title. Marcus

I like your suggestion of basic|variation etc better than our
current b|v etc scheme. Raghu - are you OK to switch to longer test case
names?

I think the test names are important to some degree - if you are trying
to look for a testcase for a particular function it is easier to look at
file names than to have to grep for the function name in a test title.
Is there a mapping between the functions listed in the functional ref
and their test cases? This kind of thing would be useful to us so I was
thinking I'd write one. It's probably lazy of me :-) but I'd just like
to click on a function name and go straight to a list of existing test
cases.

You also both made the same point that shorter tests are better and I
think we can improve here - although we have been making an effort to
make the output readable.

Thanks for the effort into test writing!!!

best regards
marcus

Monday, May 14, 2007, 6:43:51 PM, you wrote:

Hi all

We added a few tests for array, string and variable handling functions
at the weekend. As always - any feedback will be welcome.

One thing we had some difficulty with was keeping the test output small
enough to be easily read. We chose to divide up tests like this:

test_name_b.phpt b=basic function
test_name_v.phpt v=variation
test_name_e.phpt e=error cases
test_name_o.phpt o=object specific

This may not be the right scheme and I'm happy to change it if anyone
has better suggestions, although it is useful to seperate out the error
case tests because quite a lot of the differences between php5 and php6
seem to be in the exact form of the error message. Using this doesn't
completely solve the problem anyway - I think some of the "v" tests will
will end up being test_name_v1, test_name_v2 etc.

Thanks, Zoe
IBM UK

Best regards,
Marcus

18 years ago by SoftVirus — view source

unread

Hello Zoe, Marcus & Tony,

Files names having _(basic|variation|error|object).phpt looks much better.
File names _[0-9]*.phpt does not give much detail about what are the
functions being testing by this testcase until one looks at the --TEST--
section of the file. It would be nice to have the name of the testcase
accomodate name of the functions that are being tested. For example if
009.phpt is written to test basic functionality of current() and next()
functions, the better file name could be current_next_basic.phpt.

Regards,
Raghubansh, IBM India

Marcus Boerger wrote:

Hi Marcus & Tony

Hello Zoe,

that's pretty fine. Instead of _[bveo][0-9]?.phpt you could do
_(basic|variation|error|object)[0-9]{3}.phpt. Which is a tad bit better.
However those are just filenames helping you to find the correct test
file while developing. For that reason we didn not do so much about
generating those files. We Only made it briefly and suggested a three
digit numbering scheme. After all th ename in --TEST.. is the important
thing.

Thanks for the responses - you have both made the same point that it's
the --TEST-- section that should really give a descriptive title. Marcus

I like your suggestion of basic|variation etc better than our
current b|v etc scheme. Raghu - are you OK to switch to longer test case
names?

I think the test names are important to some degree - if you are trying
to look for a testcase for a particular function it is easier to look at
file names than to have to grep for the function name in a test title.
Is there a mapping between the functions listed in the functional ref
and their test cases? This kind of thing would be useful to us so I was
thinking I'd write one. It's probably lazy of me :-) but I'd just like
to click on a function name and go straight to a list of existing test
cases.

You also both made the same point that shorter tests are better and I
think we can improve here - although we have been making an effort to
make the output readable.

Thanks for the effort into test writing!!!

best regards
marcus

Monday, May 14, 2007, 6:43:51 PM, you wrote:

Hi all

We added a few tests for array, string and variable handling functions
at the weekend. As always - any feedback will be welcome.

One thing we had some difficulty with was keeping the test output small
enough to be easily read. We chose to divide up tests like this:

test_name_b.phpt b=basic function
test_name_v.phpt v=variation
test_name_e.phpt e=error cases
test_name_o.phpt o=object specific

This may not be the right scheme and I'm happy to change it if anyone
has better suggestions, although it is useful to seperate out the error
case tests because quite a lot of the differences between php5 and php6
seem to be in the exact form of the error message. Using this doesn't
completely solve the problem anyway - I think some of the "v" tests
will
will end up being test_name_v1, test_name_v2 etc.

Thanks, Zoe
IBM UK

Best regards,
Marcus

--

--
with Regards,
Raghubansh

18 years ago by Antony Dovgal — view source

unread

Hello Zoe, Marcus & Tony,

Files names having _(basic|variation|error|object).phpt looks much better.
File names _[0-9]*.phpt does not give much detail about what are the
functions being testing by this testcase until one looks at the --TEST--
section of the file.
It would be nice to have the name of the testcase
accomodate name of the functions that are being tested. For example if
009.phpt is written to test basic functionality of current() and next()
functions, the better file name could be current_next_basic.phpt.

Well, aesthetically - yes, "current_next_basic" might look better to those who don't like numbers.
But you still have to open the test and read its --TEST-- section contents to learn what exactly it's testing.

That said, I'm happy with both func_b.phpt and func_basic.phpt (and even 009.phpt) =)
We just need more tests, doesn't matter what their names look like.

--
Wbr,
Antony Dovgal

18 years ago by Zoe Slattery — view source

unread

Antony Dovgal wrote:

Hello Zoe, Marcus & Tony,

Files names having _(basic|variation|error|object).phpt looks much
better.
File names _[0-9]*.phpt does not give much detail about what are the
functions being testing by this testcase until one looks at the --TEST--
section of the file. It would be nice to have the name of the testcase
accomodate name of the functions that are being tested. For example if
009.phpt is written to test basic functionality of current() and next()
functions, the better file name could be current_next_basic.phpt.

Well, aesthetically - yes, "current_next_basic" might look better to
those who don't like numbers.
But you still have to open the test and read its --TEST-- section
contents to learn what exactly it's testing.

That said, I'm happy with both func_b.phpt and func_basic.phpt (and
even 009.phpt) =)
We just need more tests, doesn't matter what their names look like.

We're on it :-) Will probably have a few more questions on Unicode
testing later, noticed that you are putting UEXPECT section in tests-
but need to understand the implemenentation plan first...

18 years ago by Antony Dovgal — view source

unread

We're on it :-) Will probably have a few more questions on Unicode
testing later, noticed that you are putting UEXPECT section in tests-
but need to understand the implemenentation plan first...

We have two operation modes in PHP6 - "native" and Unicode.
In these modes expected output of the same test may differ very much.
That's why in some cases we have to use two expected output sections - EXPECT(F) for native and UEXPECT(F) for Unicode mode.

There is also UEXPECTREGEX - similar to EXPECTREGEX, but used in Unicode mode.

--
Wbr,
Antony Dovgal

18 years ago by Zoe Slattery — view source

unread

Antony Dovgal wrote:

We're on it :-) Will probably have a few more questions on Unicode
testing later, noticed that you are putting UEXPECT section in tests-
but need to understand the implemenentation plan first...

We have two operation modes in PHP6 - "native" and Unicode. In these
modes expected output of the same test may differ very much.
That's why in some cases we have to use two expected output sections -
EXPECT(F) for native and UEXPECT(F) for Unicode mode.
Thanks - I've just been looking at this and would like to confirm that
I've understood it. We can have both --EXEPECT(F)-- and --UEXPECT(F)--
in a PHPT file. In PHP5 run-tests.php will just ignore the --U.--
sections. In PHP6 run-tests.php will use the --U.-- sections if
unicode_semantics=on and ignore the corresponding --EXPECT(F)--
sections. If unicode_semantics=off PHP6 will behave the same way as PHP5.

Correct? Or is it more complicated that that?

There is also UEXPECTREGEX - similar to EXPECTREGEX, but used in
Unicode mode.

18 years ago by Zoe Slattery — view source

unread

Zoe Slattery wrote:

Antony Dovgal wrote:

We're on it :-) Will probably have a few more questions on Unicode
testing later, noticed that you are putting UEXPECT section in
tests- but need to understand the implemenentation plan first...

We have two operation modes in PHP6 - "native" and Unicode. In these
modes expected output of the same test may differ very much.
That's why in some cases we have to use two expected output sections

EXPECT(F) for native and UEXPECT(F) for Unicode mode.
Thanks - I've just been looking at this and would like to confirm that
I've understood it. We can have both --EXEPECT(F)-- and --UEXPECT(F)--
in a PHPT file. In PHP5 run-tests.php will just ignore the --U.--
sections. In PHP6 run-tests.php will use the --U.-- sections if
unicode_semantics=on and ignore the corresponding --EXPECT(F)--
sections. If unicode_semantics=off PHP6 will behave the same way as PHP5.

Correct? Or is it more complicated that that?
Hmm - this isn't quite right. I get the test to behave differently with
an --INI-- section which uses unicode_semantics=on/off, but not when I
use a php.ini file...

There is also UEXPECTREGEX - similar to EXPECTREGEX, but used in
Unicode mode.

18 years ago by Marcus Boerger — view source

unread

Hello Zoe,

it ignores the php.ini more or less. You should be using run-tests.php
switches -u, -U and -N.

Try: php run-tests.php -h

best regards
marcus

Tuesday, May 15, 2007, 1:33:28 PM, you wrote:

Zoe Slattery wrote:

Antony Dovgal wrote:

We're on it :-) Will probably have a few more questions on Unicode
testing later, noticed that you are putting UEXPECT section in
tests- but need to understand the implemenentation plan first...

We have two operation modes in PHP6 - "native" and Unicode. In these
modes expected output of the same test may differ very much.
That's why in some cases we have to use two expected output sections

EXPECT(F) for native and UEXPECT(F) for Unicode mode.
Thanks - I've just been looking at this and would like to confirm that
I've understood it. We can have both --EXEPECT(F)-- and --UEXPECT(F)--
in a PHPT file. In PHP5 run-tests.php will just ignore the --U.--
sections. In PHP6 run-tests.php will use the --U.-- sections if
unicode_semantics=on and ignore the corresponding --EXPECT(F)--
sections. If unicode_semantics=off PHP6 will behave the same way as PHP5.

Correct? Or is it more complicated that that?
Hmm - this isn't quite right. I get the test to behave differently with
an --INI-- section which uses unicode_semantics=on/off, but not when I
use a php.ini file...

There is also UEXPECTREGEX - similar to EXPECTREGEX, but used in
Unicode mode.

Best regards,
Marcus

18 years ago by Antony Dovgal — view source

unread

Antony Dovgal wrote:

We're on it :-) Will probably have a few more questions on Unicode
testing later, noticed that you are putting UEXPECT section in tests-
but need to understand the implemenentation plan first...

We have two operation modes in PHP6 - "native" and Unicode. In these
modes expected output of the same test may differ very much.
That's why in some cases we have to use two expected output sections -
EXPECT(F) for native and UEXPECT(F) for Unicode mode.
Thanks - I've just been looking at this and would like to confirm that
I've understood it. We can have both --EXEPECT(F)-- and --UEXPECT(F)--
in a PHPT file. In PHP5 run-tests.php will just ignore the --U.--
sections. In PHP6 run-tests.php will use the --U.-- sections if
unicode_semantics=on and ignore the corresponding --EXPECT(F)--
sections. If unicode_semantics=off PHP6 will behave the same way as PHP5.

Correct? Or is it more complicated that that?

Yup, everything is correct.

--
Wbr,
Antony Dovgal

18 years ago by Stanislav Malyshev — view source

unread

That said, I'm happy with both func_b.phpt and func_basic.phpt (and even
009.phpt) =)
We just need more tests, doesn't matter what their names look like.

I think it does. If one needs to see which tests there are or are there
any tests that test particular thing or inspect certain aspects of a
functionality, navigating a maze of func_01234.phpt, all alike, is a
nightmare. Of course, I could open all of them one by one and remember
which one does what - that is if I were a being with unlimited lifespan,
infallible memory and never felt bored - but for regular human it's a
painful experience. So when writing a new tests it would be very good to
give them names that make sense, it doesn't cost anything and helps
people. Renaming old ones might be somewhat boring task, so not sure
anybody ever would do that...

Stanislav Malyshev, Zend Products Engineer
stas@zend.com http://www.zend.com/

18 years ago by Antony Dovgal — view source

unread

That said, I'm happy with both func_b.phpt and func_basic.phpt (and even
009.phpt) =)
We just need more tests, doesn't matter what their names look like.

I think it does. If one needs to see which tests there are or are there
any tests that test particular thing or inspect certain aspects of a
functionality, navigating a maze of func_01234.phpt, all alike, is a
nightmare. Of course, I could open all of them one by one and remember
which one does what - that is if I were a being with unlimited lifespan,
infallible memory and never felt bored - but for regular human it's a
painful experience.

Well, I'm pretty sure regular humans know how to run find . -name "*.phpt" | xargs grep "func.

So when writing a new tests it would be very good to
give them names that make sense, it doesn't cost anything and helps
people. Renaming old ones might be somewhat boring task, so not sure
anybody ever would do that...

--
Wbr,
Antony Dovgal

18 years ago by Richard Lynch — view source

unread

Well, I'm pretty sure regular humans know how to run find . -name "*.phpt" | xargs grep "func.

I dunno...

It took me years to figure out that xargs thing and how to make it go...

I'd read the man page and see that it was what I wanted, and read the
tutorials and all that, and end up trying a few zillion different
things that failed before ever getting to the | and xargs bit...

Not that I'm claiming to be a normal human. :-)

[back on topic]
I'm okay with any of the naming conventions, or possibly even a
mixture that makes sense (e.g., bug report format could be unique from
more generalized test names)

--
Some people have a "gift" link here.
Know what I want?
I want you to buy a CD from some indie artist.
http://cdbaby.com/browse/from/lynch
Yeah, I get a buck. So?

18 years ago by Zoe Slattery — view source

unread

Stanislav Malyshev wrote:

I think it does. If one needs to see which tests there are or are
there any tests that test particular thing or inspect certain aspects
of a functionality, navigating a maze of func_01234.phpt, all alike,
is a nightmare. Of course, I could open all of them one by one and
remember which one does what - that is if I were a being with
unlimited lifespan, infallible memory and never felt bored - but for
regular human it's a painful experience. So when writing a new tests
it would be very good to give them names that make sense, it doesn't
cost anything and helps people. Renaming old ones might be somewhat
boring task, so not sure anybody ever would do that...
I'm happy to rename as we work through - if we can see an obvious and
more descriptive name. Also, what about making the test directory
structure match the functional reference for core functions? Some of it
does at the moment (strings, array) but it would be good to put all the
variable handling tests in ext/standard/tests/variable_handling for
example. It's another thing we could do as we go along...

Zoe Slattery
IBM UK

18 years ago by Raghubansh — view source

unread

Hello Zoe,
It would be nice thing to have mapping between categorization of the
functions in the php doc and the source/test tree structure but keeping the
tree structure as per the source code is better thing to do. If i have got
it right as of now the testcases are places based on the location of the
source file. For example testcases testing functions in the
ext/standard/array.c are placed in ext/standard/tests/array dir,
ext/standard/tests/reg.c is placed in ext/standar/tests/reg/ dir and so on.
Testcases for functions found in the Zend dir are placed in Zend/tests dir.
It would be nice to have the following addition to ext/standard/tests dir
with respect to variable handling functions ( varible handling funnctions
are implemented by different source file,e.g : type.c, var.c,
Zend/zend_builtin_functions.c, etc) :
A sub dir "type" (ext/standard/tests/type) where all the testcases testing
functions in type.c can be placed, currently its placed in
general_functions(ext/standard/tests/general_funtions) dir
A sub dir "var" (ext/standard/tests/var) where all the testcases testing
functions in var.c can be placed, currently they are placed in
general_functions(ext/standard/tests/general_functions) dir, and so on.

Doing this way its makes is very easy for locating a testcase and give
better dir structure for test organization. It would be nice if these
details were documented on php.net, May be right place is qa.php.net under
the doc "how to write test".

Marcus, Tony and others, Any comments from you ?

Also notice that we have done commit of some string functions(strcasecmp,
strcmp, strncmp) in string dir, which should have been committed in
Zend/tests dir because those functions are implemented by source file placed
in Zend dir.

Best Regards,
Raghubansh, IBM India

Stanislav Malyshev wrote:

I think it does. If one needs to see which tests there are or are
there any tests that test particular thing or inspect certain aspects
of a functionality, navigating a maze of func_01234.phpt, all alike,
is a nightmare. Of course, I could open all of them one by one and
remember which one does what - that is if I were a being with
unlimited lifespan, infallible memory and never felt bored - but for
regular human it's a painful experience. So when writing a new tests
it would be very good to give them names that make sense, it doesn't
cost anything and helps people. Renaming old ones might be somewhat
boring task, so not sure anybody ever would do that...
I'm happy to rename as we work through - if we can see an obvious and
more descriptive name. Also, what about making the test directory
structure match the functional reference for core functions? Some of it
does at the moment (strings, array) but it would be good to put all the
variable handling tests in ext/standard/tests/variable_handling for
example. It's another thing we could do as we go along...

Zoe Slattery
IBM UK

18 years ago by Marcus Boerger — view source

unread

Hello Raghubansh,

if such a mapping becomes more important i'd say we go with a xml file
mapping maintained in either php-src or phpdoc cvs module. That way we do
not have to rename all files and loose history.

best regards
marcus

Wednesday, May 16, 2007, 1:58:42 PM, you wrote:

Hello Zoe,
It would be nice thing to have mapping between categorization of the
functions in the php doc and the source/test tree structure but keeping the
tree structure as per the source code is better thing to do. If i have got
it right as of now the testcases are places based on the location of the
source file. For example testcases testing functions in the
ext/standard/array.c are placed in ext/standard/tests/array dir,
ext/standard/tests/reg.c is placed in ext/standar/tests/reg/ dir and so on.
Testcases for functions found in the Zend dir are placed in Zend/tests dir.
It would be nice to have the following addition to ext/standard/tests dir
with respect to variable handling functions ( varible handling funnctions
are implemented by different source file,e.g : type.c, var.c,
Zend/zend_builtin_functions.c, etc) :
A sub dir "type" (ext/standard/tests/type) where all the testcases testing
functions in type.c can be placed, currently its placed in
general_functions(ext/standard/tests/general_funtions) dir
A sub dir "var" (ext/standard/tests/var) where all the testcases testing
functions in var.c can be placed, currently they are placed in
general_functions(ext/standard/tests/general_functions) dir, and so on.

Doing this way its makes is very easy for locating a testcase and give
better dir structure for test organization. It would be nice if these
details were documented on php.net, May be right place is qa.php.net under
the doc "how to write test".

Marcus, Tony and others, Any comments from you ?

Also notice that we have done commit of some string functions(strcasecmp,
strcmp, strncmp) in string dir, which should have been committed in
Zend/tests dir because those functions are implemented by source file placed
in Zend dir.

Best Regards,
Raghubansh, IBM India

Stanislav Malyshev wrote:

I think it does. If one needs to see which tests there are or are
there any tests that test particular thing or inspect certain aspects
of a functionality, navigating a maze of func_01234.phpt, all alike,
is a nightmare. Of course, I could open all of them one by one and
remember which one does what - that is if I were a being with
unlimited lifespan, infallible memory and never felt bored - but for
regular human it's a painful experience. So when writing a new tests
it would be very good to give them names that make sense, it doesn't
cost anything and helps people. Renaming old ones might be somewhat
boring task, so not sure anybody ever would do that...
I'm happy to rename as we work through - if we can see an obvious and
more descriptive name. Also, what about making the test directory
structure match the functional reference for core functions? Some of it
does at the moment (strings, array) but it would be good to put all the
variable handling tests in ext/standard/tests/variable_handling for
example. It's another thing we could do as we go along...

Zoe Slattery
IBM UK

--

Best regards,
Marcus