Hi Everyone,
It's been a while when I sent my email about introducing an automatic
benchmark suite I had been working:
https://externals.io/message/116323#116323 Now I would like to share an
important milestone of this project:
After doing lots of improvements and fixes throughout the year, the suite
now reached the
required maturity level to be able to use it within PHP's nightly CI jobs
(experimentally for now).
Previously, consistency and reliability issues, as well as the lack of
financial support prevented
us from doing so. As these issues were addressed by now, we will be able to
track the
performance changes of PHP, starting with 8.5.
The initial results are very promising for real-word app based tests (Laravel,
Symfony,
Wordpress), indicating that the results are consistent and seemingly
reliable. For example run to
run variation of the median absolute results typically don't exceed 0.001
sec. When
benchmarking a php-src commit against itself, the relative difference for
each real-word test
varied between 0.01% and 0.06% (while 0.0% would have been the best
possible value).
Unfortunately, unlike real-word apps, micro benchmarks (bench.php and
micro_bench.php)
sometimes give inconsistent results, so there is still room for
improvements. Besides making
micro benchmarks more reliable, future plans include adding visualization,
and coming up with
some notification system which tells us if there is a notable performance
regression or progression.
For anyone interested in the project and the results, the benchmark suite
is available at
https://github.com/kocsismate/php-version-benchmarks, while the nightly
results are committed
into https://github.com/php/real-time-benchmark-data in both human friendly
(markdown) and in
machine processable format (tsv). Additionally, a more detailed explanation
about the benchmark's methodology can be found at
https://github.com/php/php-src/pull/15696.
Regards,
Máté