Hi internals,
As most of you know, there is this script called run-tests.php which is huge in size and hard to maintain.
I also heard some wanted it to support parallel tests execution and probably other things.
There is a project I've created a long time ago and still maintaining (https://github.com/nazar-pc/phpt-tests-runner) that should be able to run most of the tests without changes already with colored output and the whole thing is a fraction of run-tests.php's size.
How I'd like to work towards mainlining it into php-src in case there is a desire and an agreement on such transition.
If this seems like a good idea, it would be very useful if someone can enumerate the list of features that are currently used/needed from run-tests.php and which replacement should offer, alongside with features on top that might be useful (like parallel execution).
If there was similar discussion already and there is an accepted solution there, let me know.
--
Sincerely, Nazar Mokrynskyi
github.com/nazar-pc
This is a reimplementation of run-test.php, not just a wrapper (so its
another parser for the PHPT file format). However, its missing support for
many PHPT sections (ie its PHPT format support is incomplete).
See:
https://github.com/nazar-pc/phpt-tests-runner/blob/master/bin/phpt-tests-runner#L117
I created PFTT as a multi-SAPI (CLI, IIS/FastCGI, CLI-WWW, Apache)
multiplatform test tool reimplementing PHPT and PHPUnit(and support for
other common tasks)...
This may give you some clarification of what's missing
http://git.php.net/?p=pftt2.git;a=blob_plain;f=src/com/mostc/pftt/model/core/EPhptSection.java;hb=refs/heads/master
PFTT is a big heavyweight tool (so PFTT shouldn't be in core).
run-tests.php is nice because its a lightweight tool, not overly complex
and minimal to download, so I can easily do make test for a custom
build... its just not as good at seriously pounding PHP across multiple
scenarios, SAPIs, filesystems, etc... (built in part to hammer PHP on
Windows to ensure it works well). Do we really need a 3rd PHPT engine?
It also supports parallel execution (it tests many different scenarios, so
faster execution is critical)... that's a feature that's hard to get right.
-Regards
-M
On Mon, Jan 29, 2018 at 4:57 AM, Nazar Mokrynskyi nazar@mokrynskyi.com
wrote:
Hi internals,
As most of you know, there is this script called run-tests.php which is
huge in size and hard to maintain.I also heard some wanted it to support parallel tests execution and
probably other things.There is a project I've created a long time ago and still maintaining (
https://github.com/nazar-pc/phpt-tests-runner) that should be able to run
most of the tests without changes already with colored output and the whole
thing is a fraction of run-tests.php's size.How I'd like to work towards mainlining it into php-src in case there is a
desire and an agreement on such transition.If this seems like a good idea, it would be very useful if someone can
enumerate the list of features that are currently used/needed from
run-tests.php and which replacement should offer, alongside with features
on top that might be useful (like parallel execution).If there was similar discussion already and there is an accepted solution
there, let me know.--
Sincerely, Nazar Mokrynskyi
github.com/nazar-pc
It was never intended to be a wrapper or to be 100% compatible.
It was created as a simpler and modern replacement with additional features like colored output (tried to add this to run-tests.php initially, even submitted PR with some fixes and refactoring, but found it too complex to deal with) and simpler CLI interface to work with.
It is intentionally missing a few PHPT sections that seem to be redundant and have slight tweaks to some that are present. At the end of the day so many time have passed since run-tests.php was initially written.
For instance, it is painful to work with tests in PHP source tree when they are called 001.phpt, 002.phpt and so on - you need to open each file and read it to see what it intends to test. My tool uses file names instead, which makes browsing source tree with tests more productive and makes --TEST-- section redundant (simply ignored).
This tool is 6.5 times smaller than run-tests.php and imho is better structured, which leads to lower cost for introducing new features when/if needed.
Sincerely, Nazar Mokrynskyi
github.com/nazar-pc
30.01.18 00:10, Matt Ficken пише:
This is a reimplementation of run-test.php, not just a wrapper (so its another parser for the PHPT file format). However, its missing support for many PHPT sections (ie its PHPT format support is incomplete).
See: https://github.com/nazar-pc/phpt-tests-runner/blob/master/bin/phpt-tests-runner#L117
I created PFTT as a multi-SAPI (CLI, IIS/FastCGI, CLI-WWW, Apache) multiplatform test tool reimplementing PHPT and PHPUnit(and support for other common tasks)...
This may give you some clarification of what's missing
http://git.php.net/?p=pftt2.git;a=blob_plain;f=src/com/mostc/pftt/model/core/EPhptSection.java;hb=refs/heads/masterPFTT is a big heavyweight tool (so PFTT shouldn't be in core). run-tests.php is nice because its a lightweight tool, not overly complex and minimal to download, so I can easily do
make testfor a custom build... its just not as good at seriously pounding PHP across multiple scenarios, SAPIs, filesystems, etc... (built in part to hammer PHP on Windows to ensure it works well). Do we really need a 3rd PHPT engine?It also supports parallel execution (it tests many different scenarios, so faster execution is critical)... that's a feature that's hard to get right.
-Regards
-MHi internals, As most of you know, there is this script called run-tests.php which is huge in size and hard to maintain. I also heard some wanted it to support parallel tests execution and probably other things. There is a project I've created a long time ago and still maintaining (https://github.com/nazar-pc/phpt-tests-runner <https://github.com/nazar-pc/phpt-tests-runner>) that should be able to run most of the tests without changes already with colored output and the whole thing is a fraction of run-tests.php's size. How I'd like to work towards mainlining it into php-src in case there is a desire and an agreement on such transition. If this seems like a good idea, it would be very useful if someone can enumerate the list of features that are currently used/needed from run-tests.php and which replacement should offer, alongside with features on top that might be useful (like parallel execution). If there was similar discussion already and there is an accepted solution there, let me know. -- Sincerely, Nazar Mokrynskyi github.com/nazar-pc <http://github.com/nazar-pc
Hi!
For instance, it is painful to work with tests in PHP source tree when they are called 001.phpt, 002.phpt and so on - you need to open each file and read it to see what it intends to test. My tool uses file names instead, which makes browsing source tree with tests more productive and makes --TEST-- section redundant (simply ignored).
I think --TEST-- should absolutely be supported. Filename is not nearly
enough to properly describe the content of the test. Something like
"Test whether foo(bar($baz)) crashes if called more than twice (bug
#1234)" is not going to be captured in a filename. Filenames and
informative descriptions are not the same thing.
I'll look more into what other requirements are needed (one suggestion
would be try to keep option compatibility with the old tool if possible,
so people could use their old habits) and write later, but I think
supporting proper test descriptions would definitely be one of them.
--
Stas Malyshev
smalyshev@gmail.com
The idea is to make filenames more descriptive and sufficient for displaying in test results. This doesn't prevent or discourage anyone from including description in test file itself when more details need to be included.
Moreover, all of the characters from mentioned title are allowed for use in file name (at least on Linux) and it would be much more useful to see them instead of 001.phpt or bug-1234.phpt.
Sincerely, Nazar Mokrynskyi
github.com/nazar-pc
31.01.18 00:26, Stanislav Malyshev пише:
Hi!
For instance, it is painful to work with tests in PHP source tree when they are called 001.phpt, 002.phpt and so on - you need to open each file and read it to see what it intends to test. My tool uses file names instead, which makes browsing source tree with tests more productive and makes --TEST-- section redundant (simply ignored).
I think --TEST-- should absolutely be supported. Filename is not nearly
enough to properly describe the content of the test. Something like
"Test whether foo(bar($baz)) crashes if called more than twice (bug
#1234)" is not going to be captured in a filename. Filenames and
informative descriptions are not the same thing.I'll look more into what other requirements are needed (one suggestion
would be try to keep option compatibility with the old tool if possible,
so people could use their old habits) and write later, but I think
supporting proper test descriptions would definitely be one of them.
The idea is to make filenames more descriptive and sufficient for
displaying in test results. This doesn't prevent or discourage anyone from
including description in test file itself when more details need to be
included.
Would you propose to apply a bulk rename of all .phpt files currently in
the tree to use their TEST blocks as filenames, or a manual review of each
test? If automating, how would you handle tests whose descriptions are not
unique within a directory? If manual, what kind of naming scheme would you
propose?
Moreover, all of the characters from mentioned title are allowed for use
in file name (at least on Linux) and it would be much more useful to see
them instead of 001.phpt or bug-1234.phpt.
While most characters are allowed in names, some can be irritating to work
with - test descriptions are likely to include things like ", $, * etc
which have to be escaped carefully whenever manipulating the files.
Different filesystems and environments may not handle the same set of
characters, so care would need to be taken to ensure portability of the
test suite.
It seems like a tool which lists the file name and description for each
.phpt file in a directory would bring most of the advantages of such a mass
renaming without the problems.
Regards,
Rowan Collins
[IMSoP]
Hi!
The idea is to make filenames more descriptive and sufficient for
displaying in test results.
Descriptive - sure, that'd be great, why not. Sufficient - won't happen.
Filenames are not meant to describe complex things, are not suitable for
it and we should not try to put it there. It just won't work.
This doesn't prevent or discourage anyone from including description
in test file itself when more details need to be included.
Thus, the necessity of supporting --TEST--.
Moreover, all of the characters from mentioned title are allowed for
use in file name (at least on Linux) and it would be much more useful
to see them instead of 001.phpt or bug-1234.phpt.
Yes, but I do not want to have files with such names. And neither, I
expect, do our users - weird filenames are hard to work with, break some
common tools and require extraordinary care to handle them properly. To
illustrate, right now git (tool that is behind Linux kernel, which is
behind pretty much everything on the Internet) can't properly process
some of our test filenames. We don't need more trouble in this
department. We certainly don't need to force people to name their files
"Test whether foo(bar($baz)) crashes if it's called more than twice (bug
#1234).phpt" - that would be a nightmare to work with. We should not be
using filenames for things they weren't meant to be used for.
--
Stas Malyshev
smalyshev@gmail.com
Hi!
There is a project I've created a long time ago and still maintaining (https://github.com/nazar-pc/phpt-tests-runner) that should be able to run most of the tests without changes already with colored output and the whole thing is a fraction of run-tests.php's size.
Some notes:
-
I already wrote about TEST support, that stays unchanged.
-
It would be nice to preserve option-compatibility with the old
version, if possible. People have scripts and trained keyboard-memory,
and migrating would be easier without looking up new options each time.
That doesn't mean every option should be supported, but for those that
are, it'd be nice to keep the same names. -
Would be nice to support in options:
-- Passing options to PHP binary (at least -n and -d, but potentially
all others)
-- Keeping temp files -
Section support - I think besides TEST, the following will be needed:
- XFAIL: unfortunately, those are still needed
- ENV: there are a lot of environment-dependent tests as it seems
- FILE_EXTERNAL: is pretty widely used
- The group of
GET,POST,PUT,POST_RAW,GZIP_POST,GZIP_POST,COOKIE,HEADERS,CGI - necessary
for some functionality related to HTTP processing. But most of the tests
--
Stas Malyshev
smalyshev@gmail.com
- Section support - I think besides TEST, the following will be
needed:
- XFAIL: unfortunately, those are still needed
- ENV: there are a lot of environment-dependent tests as it seems
- FILE_EXTERNAL: is pretty widely used
- The group of
GET,POST,PUT,POST_RAW,GZIP_POST,GZIP_POST,COOKIE,HEADERS,CGI -
necessary
for some functionality related to HTTP processing. But most of the
tests
Also --REDIRECTTEST-- as used in PDO drivers is needed or requires a
replacement.
johannes