Hi,
I invested about half an hour this morning to refactor bench.php. It now
starts a new PHP process for each benchmark and uses microtime(true) and
memory_get_peak_usage() to measure benchmark data.
Have a look at
-
This is the main script.
-
These are the iterators used by the main script.
-
This code is shared between the benchmark scripts.
-
This is an example of a benchmark script.
Here is an example output:
sb@vmware benchmark % php run.php
Benchmark Time Peak Memory Usage
ackermann 3.067452 715080
mandel 2.242472 127840
mandel2 2.185040 123448
simple 1.031363 117184
simplecall 3.849651 115528
simpleucall 4.226199 116968
simpleudcall 4.394062 117176
Note that I have not yet ported all benchmarks from bench.php.
At this point I would like to know whether or not people are interested
in this. No point in working on this if nobody cares, right? :-)
Best,
Sebastian
--
Sebastian Bergmann http://sebastian-bergmann.de/
GnuPG Key: 0xB85B5D69 / 27A7 2B14 09E4 98CD 6277 0E5B 6867 C514 B85B 5D69
Here is an example output:
sb@vmware benchmark % php run.php
Benchmark Time Peak Memory Usage
ackermann 3.067452 715080
mandel 2.242472 127840
mandel2 2.185040 123448
simple 1.031363 117184
simplecall 3.849651 115528
simpleucall 4.226199 116968
simpleudcall 4.394062 117176Note that I have not yet ported all benchmarks from bench.php.
At this point I would like to know whether or not people are interested
in this. No point in working on this if nobody cares, right? :-)
I like it. Please continue.
--
Wbr,
Antony Dovgal
Antony Dovgal wrote:
I like it. Please continue.
Okay, great. Now: where should I put it? I think the current approach of
having bench.php duplicated in each branch of the Zend Engine is sub-
optimal. Maybe we could create a new top-level CVS module for this?
--
Sebastian Bergmann http://sebastian-bergmann.de/
GnuPG Key: 0xB85B5D69 / 27A7 2B14 09E4 98CD 6277 0E5B 6867 C514 B85B 5D69
Antony Dovgal wrote:
I like it. Please continue.
Okay, great. Now: where should I put it? I think the current approach of
having bench.php duplicated in each branch of the Zend Engine is sub-
optimal. Maybe we could create a new top-level CVS module for this?
Maybe.
But separate CVS module would make sense only if we're going to release it separately.
I guess qa.php.net would be a good place for such releases.
Opinions?
--
Wbr,
Antony Dovgal
Hello Antony,
QA sounds fine. Top-level is also fine as we have no real mathing place.
When we offer it through QA we might aswell use qw-web as the cvs module
but a seperate one sounds better.
best regards
marcus
Tuesday, February 27, 2007, 5:58:41 PM, you wrote:
Antony Dovgal wrote:
I like it. Please continue.
Okay, great. Now: where should I put it? I think the current approach of
having bench.php duplicated in each branch of the Zend Engine is sub-
optimal. Maybe we could create a new top-level CVS module for this?
Maybe.
But separate CVS module would make sense only if we're going to release it separately.
I guess qa.php.net would be a good place for such releases.
Opinions?
--
Wbr,
Antony Dovgal
Best regards,
Marcus
Hello Antony,
QA sounds fine. Top-level is also fine as we have no real mathing place.
I meant using qa.php.net for releases, I don't care where exactly it lives in our CVS.
When we offer it through QA we might aswell use qw-web as the cvs module
but a seperate one sounds better.
--
Wbr,
Antony Dovgal
Sebastian Bergmann wrote:
Have a look at
I uploaded a snapshot of what I have now (I added fallbacks for missing
functionality in PHP 4.4 and PHP 5.0 and the rest of the benchmarks) to
http://static.phpunit.de/benchmark.tbz2.
--
Sebastian Bergmann http://sebastian-bergmann.de/
GnuPG Key: 0xB85B5D69 / 27A7 2B14 09E4 98CD 6277 0E5B 6867 C514 B85B 5D69
Hello Sebastian,
looks pretty fine. Only you don't need fallbacks right now. Simply add a
check for PHP >= 5.2.0 since you are using Iterators and I don't see any
sense in testing 5.0 or 5.1. You could further more make the run script
operate a bit like run-tests.php and have the PHP executable be
configurable and get version tested. So you would first check that the PHP
runiing the run script is new enough and that second the tests are executed
by an acceptable PHP version.
When you want to spend more work, ever thought of making the bench files
have a layout similar to *.phpt? So that we could split up in sections like
name, actual bench, clean-up, etc? Or do you prefer to do all from within
one file and call bench funcs manually?
best regards
marcus
Tuesday, February 27, 2007, 5:53:53 PM, you wrote:
Sebastian Bergmann wrote:
Have a look at
I uploaded a snapshot of what I have now (I added fallbacks for missing
functionality in PHP 4.4 and PHP 5.0 and the rest of the benchmarks) to
http://static.phpunit.de/benchmark.tbz2.
--
Sebastian Bergmann http://sebastian-bergmann.de/
GnuPG Key: 0xB85B5D69 / 27A7 2B14 09E4 98CD 6277 0E5B 6867 C514 B85B 5D69
Best regards,
Marcus
Marcus Boerger wrote:
Only you don't need fallbacks right now. Simply add a check for
PHP >= 5.2.0 since you are using Iterators and I don't see any sense
in testing 5.0 or 5.1.
Already did that yesterday.
You could further more make the run script operate a bit like
run-tests.php and have the PHP executable be configurable and get
version tested.
I am taking baby steps at the moment :) Of course the PHP binary will
be configurable.
When you want to spend more work, ever thought of making the bench
files have a layout similar to *.phpt? So that we could split up in
sections like name, actual bench, clean-up, etc?
The question is: is a separate system such as mine needed or could we
extend run-tests.php? If we choose to extend run-tests.php, suddenly
every *.phpt test would be a potential benchmark.
--
Sebastian Bergmann http://sebastian-bergmann.de/
GnuPG Key: 0xB85B5D69 / 27A7 2B14 09E4 98CD 6277 0E5B 6867 C514 B85B 5D69
Sebastian Bergmann wrote:
I uploaded a snapshot of what I have now (I added fallbacks for missing
functionality in PHP 4.4 and PHP 5.0 and the rest of the benchmarks) to
http://static.phpunit.de/benchmark.tbz2.
I updated the snapshot. This is what the output looks like now:
Benchmark Time Peak Memory Peak Memory (Real)
ackermann 3.050 718336 786432
array 0.104 11949880 12058624
array2 0.101 11961848 12058624
array3 1.285 648016 786432
fibonacci 14.293 121936 262144
hash 0.653 7347400 7602176
hash2 0.115 247728 262144
heapsort 0.753 2941128 3145728
mandel 3.534 131920 262144
mandel2 2.806 124616 262144
matrix 0.703 636512 786432
nestedloop 2.844 125568 262144
sieve 0.796 2529288 2883584
simple 1.607 121320 262144
simplecall 5.828 119608 262144
simpleucall 6.526 121048 262144
simpleudcall 6.760 121256 262144
strcat 0.117 898008 1310720
--
Sebastian Bergmann http://sebastian-bergmann.de/
GnuPG Key: 0xB85B5D69 / 27A7 2B14 09E4 98CD 6277 0E5B 6867 C514 B85B 5D69
Hello Sebastian.
What's the status of this?
Do you have anything working?
Hi,
I invested about half an hour this morning to refactor bench.php. It now
starts a new PHP process for each benchmark and uses microtime(true) and
memory_get_peak_usage()to measure benchmark data.Have a look at
This is the main script.
These are the iterators used by the main script.
This code is shared between the benchmark scripts.
This is an example of a benchmark script.
Here is an example output:
sb@vmware benchmark % php run.php
Benchmark Time Peak Memory Usage
ackermann 3.067452 715080
mandel 2.242472 127840
mandel2 2.185040 123448
simple 1.031363 117184
simplecall 3.849651 115528
simpleucall 4.226199 116968
simpleudcall 4.394062 117176Note that I have not yet ported all benchmarks from bench.php.
At this point I would like to know whether or not people are interested
in this. No point in working on this if nobody cares, right? :-)Best,
Sebastian
--
Wbr,
Antony Dovgal
Antony Dovgal wrote:
Hello Sebastian.
Hello Antony,
What's the status of this?
Do you have anything working?
back in February/March when I first posted about this, I discussed this
with Marcus. The conclusion was that we wanted to rewrite run-tests.php
based on the Iterator approach I used for the benchmark system's
prototype. Benchmarks would then be .phpb in addition to .phpt for
tests.
I did some work on this but stopped when I got negative feedback (on
IRC) towards the idea of rewriting "something that works"
(run-tests.php).
I see several options to continue:
-
Add .phpb style support for benchmarks to the current run-tests.php.
-
Improve run-tests.php to also collect Peak Memory Usage, store the
data in SQLite, and have a test (optionally) fail when it takes more
memory than it did in the past (one of the ideas that I had for the
rewrite). -
Do the rewrite.
Opinions?
--
Sebastian Bergmann http://sebastian-bergmann.de/
GnuPG Key: 0xB85B5D69 / 27A7 2B14 09E4 98CD 6277 0E5B 6867 C514 B85B 5D69
Antony Dovgal wrote:
Hello Sebastian.
Hello Antony,
What's the status of this?
Do you have anything working?back in February/March when I first posted about this, I discussed this
with Marcus. The conclusion was that we wanted to rewrite run-tests.php
based on the Iterator approach I used for the benchmark system's
prototype.
Require SPL for a test suite?
Doesn't seem to be good idea..
Benchmarks would then be .phpb in addition to .phpt for
tests.I did some work on this but stopped when I got negative feedback (on
IRC) towards the idea of rewriting "something that works"
(run-tests.php).
Well, I personally think we'll have to do it sooner or later.
The current version run-tests.php is really hard to maintain and improve.
We can have both for some time and drop the old one when we're sure that the new one is stable.
It's a developers tool anyway..
I see several options to continue:
Add .phpb style support for benchmarks to the current run-tests.php.
Improve run-tests.php to also collect Peak Memory Usage, store the
data in SQLite, and have a test (optionally) fail when it takes more
memory than it did in the past (one of the ideas that I had for the
rewrite).
Oh, that would be too much IMO..
- Do the rewrite.
Opinions?
I'm for the rewrite and I'm ready to help.
--
Wbr,
Antony Dovgal
Antony Dovgal wrote:
Antony Dovgal wrote:
Hello Sebastian.
Hello Antony,
What's the status of this?
Do you have anything working?back in February/March when I first posted about this, I discussed this
with Marcus. The conclusion was that we wanted to rewrite run-tests.php
based on the Iterator approach I used for the benchmark system's
prototype.
I did not catch the start of this. This is about performance regression
testing?
Require SPL for a test suite?
Doesn't seem to be good idea..
Well SPL should not be allowed to be disabled anyways, imho.
regards,
Lukas
Antony Dovgal wrote:
Antony Dovgal wrote:
Hello Sebastian.
Hello Antony,
What's the status of this?
Do you have anything working?back in February/March when I first posted about this, I discussed this
with Marcus. The conclusion was that we wanted to rewrite run-tests.php
based on the Iterator approach I used for the benchmark system's
prototype.I did not catch the start of this. This is about performance regression
testing?Require SPL for a test suite?
Doesn't seem to be good idea..Well SPL should not be allowed to be disabled anyways, imho.
I agree, but that's out of the topic.
--
Wbr,
Antony Dovgal
Well SPL should not be allowed to be disabled anyways, imho.
I agree, but that's out of the topic.
Well, as we may require for the tests suite, and it looks like a non
sense to be able to disable part of it, it is the right time to fix
solve this issue :)
It should not be possible to disable it. Or even better, we should
move what we consider as core from SPL to Zend.
I don't think the "hundreds" of php script and numerous iterators
should be moved but the required ones. A couple of functions or
features (I don't understand what they do in SPL) should be moved as
well:
class_implements, class_parents, spl_object_hash
--Pierre
Pierre wrote:
It should not be possible to disable it. Or even better, we should
move what we consider as core from SPL to Zend.
Amen.
regards,
Lukas
I'd like to say something involving smoke and carcass but that'll bite me later
so.. :)
I'm all for moving the useful stuff to Zend where it belongs.
But I'm totally against making SPL another exception to the rule.
Along with ext/date and ext/standard that is. Extension is an extension.
It's not core, it's not "standard". If there are parts in the extension that
should always be available -> move to core!
--Jani
Pierre kirjoitti:
Well SPL should not be allowed to be disabled anyways, imho.
I agree, but that's out of the topic.
Well, as we may require for the tests suite, and it looks like a non
sense to be able to disable part of it, it is the right time to fix
solve this issue :)It should not be possible to disable it. Or even better, we should
move what we consider as core from SPL to Zend.I don't think the "hundreds" of php script and numerous iterators
should be moved but the required ones. A couple of functions or
features (I don't understand what they do in SPL) should be moved as
well:
class_implements, class_parents, spl_object_hash--Pierre
Antony Dovgal wrote:
Antony Dovgal wrote:
What's the status of this?
Do you have anything working?back in February/March when I first posted about this, I
discussed this with Marcus. The conclusion was that we wanted to
rewrite run-tests.php based on the Iterator approach I used for
the benchmark system's prototype.I did not catch the start of this. This is about performance
regression testing?Require SPL for a test suite?
Doesn't seem to be good idea..Well SPL should not be allowed to be disabled anyways, imho.
I agree, but that's out of the topic.
Then what is the issue with using this in the tests suite?
Derick
Require SPL for a test suite?
Doesn't seem to be good idea..Well SPL should not be allowed to be disabled anyways, imho.
I agree, but that's out of the topic.
Then what is the issue with using this in the tests suite?
<@tony2001> I know that ext/pcre can be disabled, but it's good idea to keep the number of required extensions as low as we can
--
Wbr,
Antony Dovgal