Hi again,
I'm using this locally because two of our tests take over 10 minutes each to
run on my laptop, and I run the relevant bits of test suite every time I
make a change.
All it does is adds another option, -x, to run-tests.php. This sets an
environmental variable which can then be checked for in the SKIPIF section
of very slow-running tests.
Any objections if I commit it in 5_3/HEAD?
- Steph
Hi!
$environment['EXEMPT_SLOW_TESTS'] = 1;
Maybe "SKIP_SLOW_TESTS"? If it's checked in skip section... :)
Otherwise - good idea!
Stanislav Malyshev, Zend Software Architect
stas@zend.com http://www.zend.com/
(408)253-8829 MSN: stas@zend.com
Hey Stas,
$environment['EXEMPT_SLOW_TESTS'] = 1;Maybe "SKIP_SLOW_TESTS"? If it's checked in skip section... :)
Otherwise - good idea!
You can tell I'm reading my mail backwards today...
I used 'EXEMPT' because the option is 'x' and I wanted it to be easy to
remember. 's' wasn't available.
- Steph
Hey Steph;
I'm using this locally because two of our tests take over 10
minutes each to run on my laptop, and I run the relevant bits of
test suite every time I make a change.All it does is adds another option, -x, to run-tests.php. This sets
an environmental variable which can then be checked for in the
SKIPIF section of very slow-running tests.
How do you specify that "test A is slow"? Is there a certain skipif
message you include, or...?
-T
Hi Travis,
All it does is adds another option, -x, to run-tests.php. This sets an
environmental variable which can then be checked for in the SKIPIF
section of very slow-running tests.How do you specify that "test A is slow"? Is there a certain skipif
message you include, or...?
Yep. The two that bug me most are actually labelled 'slow test', so having a
skipif condition for those wouldn't be too obscure. Note that it's only
intended for when you're repeatedly running the same group of tests...
I like Chris's idea better, but can't see a way to implement it. There's
probably something really obvious I'm missing, let me know if.
- Steph
Steph Fox wrote:
Hi again,
I'm using this locally because two of our tests take over 10 minutes
each to run on my laptop, and I run the relevant bits of test suite
every time I make a change.All it does is adds another option, -x, to run-tests.php. This sets an
environmental variable which can then be checked for in the SKIPIF
section of very slow-running tests.Any objections if I commit it in 5_3/HEAD?
I'd prefer a run-tests.php option that sets the timeout limit in seconds.
Chris
--
Christopher Jones, Oracle
Email: christopher.jones@oracle.com Tel: +1 650 506 8630
Blog: http://blogs.oracle.com/opal/ Free PHP Book: http://tinyurl.com/f8jad
I'd prefer a run-tests.php option that sets the timeout limit in seconds.
Nice idea, but I'm not sure it's achievable under CLI.
- Steph
I'd prefer a run-tests.php option that sets the timeout limit in seconds.
Nice idea, but I'm not sure it's achievable under CLI.
Yes, it is. Check the system_with_timeout() function in the run-tests.php
script.
There you've the timeout hardcoded ('$leak_check ? 300 : 60'). You would
just need to make it configurable by some environment var.
Nuno
Yes, it is. Check the system_with_timeout() function in the run-tests.php
script.
There you've the timeout hardcoded ('$leak_check ? 300 : 60'). You would
just need to make it configurable by some environment var.
I already tried hard-coding both tv_sec and tv_usec to 0 and it makes no
difference here.
- Steph
Yes, it is. Check the system_with_timeout() function in the run-
tests.php script.
There you've the timeout hardcoded ('$leak_check ? 300 : 60'). You
would just need to make it configurable by some environment var.I already tried hard-coding both tv_sec and tv_usec to 0 and it
makes no difference here.
I can add this to PHPT - it uses a timeout based on reading the open
proc. The only problem is that it is currently treated as an error.
I could maybe add a new reporting level of timeout, though I do like
the idea of having some sort of meta-data to conditionally skip tests.
Maybe a better solution is to add an --exclude <pattern> and ask
people to either place potentially slow tests in tests/slow/, or name
then <test case>.slow.phpt? Being able to exclude a pattern of test
names definitely has more use than just setting a timeout.
Thoughts?
-T
I can add this to PHPT - it uses a timeout based on reading the open
proc.
That's what the current run-tests does too. The problem is it isn't reliable
cross-platform - we don't have any control over the select() function used
by stream_select() except under Windows, where it doesn't work (I'm looking
into that but can't see why).
The only problem is that it is currently treated as an error. I could
maybe add a new reporting level of timeout, though I do like the idea of
having some sort of meta-data to conditionally skip tests.Maybe a better solution is to add an --exclude <pattern> and ask people
to either place potentially slow tests in tests/slow/, or name then <test case>.slow.phpt? Being able to exclude a pattern of test names
definitely has more use than just setting a timeout.
So what was wrong with the simple skipif and env var approach again? If
you're going to mark a test to get run-tests to respond to it, it doesn't
make a lot of difference whether you mark that test by its name or with a
skipif line. The only difference is an untidy CVS directory.
I think the naming thing would be a better idea for something that's just
starting out. It's a less good idea for a test suite that already exists
IMHO.
- Steph
Thoughts?
-T
I can add this to PHPT - it uses a timeout based on reading the
open proc.That's what the current run-tests does too. The problem is it isn't
reliable cross-platform - we don't have any control over the select
() function used bystream_select()except under Windows, where it
doesn't work (I'm looking into that but can't see why).
This doesn't help you yet, but that's a moot point in PHPT as it uses
COM under Windows as proc_* functions and the various stream
functions are less than reliable.
So what was wrong with the simple skipif and env var approach
again? If you're going to mark a test to get run-tests to respond
to it, it doesn't make a lot of difference whether you mark that
test by its name or with a skipif line. The only difference is an
untidy CVS directory.
There's nothing wrong with that approach. I'm trying to find
something that addresses the problem (i.e., "how can I not run tests
that are going to take a long time to run?"), while providing enough
flexibility to answer other problems (i.e., "how can I skip X tests
that I don't care about?").
If there's a way to address the problem without making it so
specific, I'm all for it.
I think the naming thing would be a better idea for something
that's just starting out. It's a less good idea for a test suite
that already exists IMHO.
Agreed. If we were to go this route, I'd actually consider adding a
run-tests.ini or some such that allows you to configure things like
excluded file patterns.
-T
There's nothing wrong with that approach. I'm trying to find something
that addresses the problem (i.e., "how can I not run tests that are going
to take a long time to run?"), while providing enough flexibility to
answer other problems (i.e., "how can I skip X tests that I don't care
about?").If there's a way to address the problem without making it so specific,
I'm all for it.
I mostly agree - I'm just looking at 'here and now' rather than 'when the
nice new test suite stuff is done'. 'Here and now', there isn't a reliable
way to set this up and skipif looks like the cleanest option.
I think the naming thing would be a better idea for something that's
just starting out. It's a less good idea for a test suite that already
exists IMHO.Agreed. If we were to go this route, I'd actually consider adding a
run-tests.ini or some such that allows you to configure things like
excluded file patterns.
That might not be a bad idea.
- Steph
There's nothing wrong with that approach. I'm trying to find
something that addresses the problem (i.e., "how can I not run
tests that are going to take a long time to run?"), while
providing enough flexibility to answer other problems (i.e., "how
can I skip X tests that I don't care about?").If there's a way to address the problem without making it so
specific, I'm all for it.I mostly agree - I'm just looking at 'here and now' rather than
'when the nice new test suite stuff is done'. 'Here and now', there
isn't a reliable way to set this up and skipif looks like the
cleanest option.
The only problem with that is everything you add I've got to add to
my GSoC project so PHPT :-)
That might not be a bad idea.
I think the long-term goal should be the ability to "force skip"
files based on an --exclude parameter, an ini conf file (looks for --
ini-file <file> or <cwd>/tests.ini), and an ENV variable. The first
and last would just be separated by the PATH_SEPARATOR for regex
patterns.
In addition, we can definitely make the time-out something that's
settable via the command line and conf, but as you noted in your next
email, that is a separate issue.
I've created a few tickets on these so we can make sure to track
these issues:
- add exclude: http://phpt.info/ticket/69
- add timeout: http://phpt.info/ticket/70
-T
Travis Swicegood wrote:
There's nothing wrong with that approach. I'm trying to find
something that addresses the problem (i.e., "how can I not run tests
that are going to take a long time to run?"), while providing enough
flexibility to answer other problems (i.e., "how can I skip X tests
that I don't care about?").If there's a way to address the problem without making it so
specific, I'm all for it.I mostly agree - I'm just looking at 'here and now' rather than 'when
the nice new test suite stuff is done'. 'Here and now', there isn't a
reliable way to set this up and skipif looks like the cleanest option.The only problem with that is everything you add I've got to add to my
GSoC project so PHPT :-)That might not be a bad idea.
I think the long-term goal should be the ability to "force skip" files
based on an --exclude parameter, an ini conf file (looks for --ini-file
<file> or <cwd>/tests.ini), and an ENV variable. The first and last
would just be separated by thePATH_SEPARATORfor regex patterns.
The ini file is a great idea, as one can create groupings of test suites
independent of the tests.
Greg
Steph Fox wrote:
So what was wrong with the simple skipif and env var approach again?
The problem is it only skips the test!
I'd like my slow tests to run (this generally occurs when I use a very
remote DB). I end up manually increasing the timeout in stream_select()
in run-tests.sh. This works for me on Linux.
If we make the timeout value adjustable, you can set it to a low value
so your slow tests are quickly aborted, and I can set it to a high
value so my tests are run.
Chris
--
Christopher Jones, Oracle
Email: christopher.jones@oracle.com Tel: +1 650 506 8630
Blog: http://blogs.oracle.com/opal/ Free PHP Book: http://tinyurl.com/f8jad
Hi Chris,
If we make the timeout value adjustable, you can set it to a low value
so your slow tests are quickly aborted, and I can set it to a high
value so my tests are run.
It's easily enough done, it's just a separate issue.
I can easily put a workaround in the Windows version of select.c to make a
0-second timeout work, but that's as far as it goes. The problem in
run-tests (if anyone's interested - Travis seems to have a better solution)
is that the actual read is always only 8192 bytes, and the actual API call
never times out because stream_select() is reset at every iteration and
called afresh. We'd need to be able to set a timeout limit in the run-tests
script rather than in the call to stream_select(). And if we do that the
tests of course fail, which isn't a good outcome.
So 'skipif' suits my needs better, but not yours. I'll add both.
- Steph
Chris
--
Christopher Jones, Oracle
Email: christopher.jones@oracle.com Tel: +1 650 506 8630
Blog: http://blogs.oracle.com/opal/ Free PHP Book:
http://tinyurl.com/f8jad
Steph Fox wrote:
So 'skipif' suits my needs better, but not yours. I'll add both.
Thanks Steph.
Chris
--
Christopher Jones, Oracle
Email: christopher.jones@oracle.com Tel: +1 650 506 8630
Blog: http://blogs.oracle.com/opal/ Free PHP Book: http://tinyurl.com/f8jad
Yes, it is. Check the system_with_timeout() function in the run-tests.php
script.
There you've the timeout hardcoded ('$leak_check ? 300 : 60'). You would
just need to make it configurable by some environment var.I already tried hard-coding both tv_sec and tv_usec to 0 and it makes no
difference here.
uhm, file a bug report then. It was supposed to work..
Nuno
uhm, file a bug report then. It was supposed to work..
There are some open on this already.
Also - Windows is the only environment where we actually have control over
whether it works or not. Everything else relies on a system call.
- Steph