Automatic performance benchmarking: PHP 8.1 is ~30% faster than PHP 7.4

Le 12 novembre 2021 11:18:50 GMT+01:00, "Máté Kocsis" kocsismate90@gmail.com a écrit :

Furthermore, I'd have some questions regarding this project:

• Would you welcome automatic emails on internals (or on a dedicated
  mailing list), just like how
  Intel did it? I'm not sure about the frequency, but in my opinion,
  1-2/month would be a sensible one, given there is interest in it.

I would favor having this on a website, with an atom/rss feed. But I'm not against having it on the ML, but more like when there is an RC or when some change have significant impact.

• As always, I'm happy to receive feedback and improvements (ideally
  along
  with an implementation :) ). For example, currently I've started
  working on
  some visualization and chart support, but frontend stuff is not my area
  of
  expertise, so I'm progressing a bit slower than normally. Also, the
  statistical foundations could be reviewed, and possibly improved.

The link to the results in the readme points to https://kocsismate.github.io/php-version-benchmarks/index.html which is empty. Maybe this is known.
Charts

This quite nice!
Côme

Furthermore, I'd have some questions regarding this project:

• Would you welcome automatic emails on internals (or on a dedicated
  mailing list), just like how Intel did it? I'm not sure about the
  frequency, but in my opinion, 1-2/month would be a sensible one, given
  there is interest in it.

I'm sure you know this already, but I'd be interested... I quite like
knowing when regressions have happened as soon as possible; and having
something which is keeping an eye on things, would hopefully highlight
issues sooner.

• Does anyone know how to get some sponsorship from AWS...

Very funny, Amazon paying :-P

• As always, I'm happy to receive feedback and improvements (ideally along

with an implementation :) ). For example, currently I've started working on
some visualization and chart support, but frontend stuff is not my area of
expertise...

For my sins, I can do some frontend stuff... are we talking CSS, SVG, and
maybe a bit of the old JavaScript? If so, feel free to contact me off-list,
I probably can't do anything in the next week or two, but I'm
optimistically (naively) hoping I can clear some of my todo list :-)

Craig

Hey Máté,

Hello Internals,

I'm writing this email because lately, I've been working on an automatic
benchmarking framework for PHP, and I'd like to share some news regarding
it. The initial implementation
was sponsored by Craig Francis, and it was used for the evaluation of the
performance aspects of the is_literal() RFC. Since then, I fixed numerous
issues and implemented many new features, so with the very close advent of
PHP 8.1, the time has come to unveil it.

I'm sure that many of you still remember Intel's automatic benchmarks from
a couple of years ago (e.g. https://externals.io/message/89843#89843). I
loved these emails, so this project served as a great inspiration for me
to start working on something similar. I have to admit
though that I won't be able to replicate their extremely advanced setup.

My main goal was to develop a framework (
https://github.com/kocsismate/php-version-benchmarks) which is:

• fully automatic so that it can be easily run regularly, and the
  benchmarks are reproducible
• it's possible to try it out locally via Docker, but it can be run in the
  cloud (currently, only AWS is
  supported), and the instance type is configurable
• supports different CPU platforms (X86-64 and ARM64), and advanced options
  whether
  turbo boost/hyper threading/deeper CPU C-states are enabled is configurable
• supports any version of PHP since PHP 7.4, including any git branches or
  commits
• supports the most important PHP configurations, including whether
  OPcache/JIT/preloading
  is enabled
• supports multiple tests: currently, the micro benchmarks bundled with
  php-src, as well as the
  Symfony and the Laravel demo sites as "real-life" tests are included with
  some customizability (number of warmups, iterations number, number of
  requests)
• results are measured via using PHP-CGI which is a lightweight web server
  with very a small overhead

The most current benchmark is available at

https://github.com/kocsismate/php-version-benchmarks/blob/main/docs/results/2021_11_11_09_20_1_aws_arm64_c6g_4xlarge/result.md
. I'm happy to share that the results suggest PHP 8.1.0 is around 28-32%
faster than PHP 7.4 in real-life tests, and a few percent faster when it
comes to micro benchmarks. The benchmark was performed on an ARM64 instance
because this platform provided much more stable results than X86-64-based
ones did, mainly due to their fixed CPU frequency.

Unfortunately, the benchmark is not yet suitable for detecting minor
performance changes between commits as the run-to-run variation of the
results is a bit too high; in some cases, it can be up to 1-2%. I hope that
this problem can be mitigated to an acceptable level in the future, but
most probably I'll need some help to achieve this. So any help is
appreciated!

Furthermore, I'd have some questions regarding this project:

• Would you welcome automatic emails on internals (or on a dedicated
  mailing list), just like how
  Intel did it? I'm not sure about the frequency, but in my opinion,
  1-2/month would be a sensible one, given there is interest in it.
• Does anyone know how to get some sponsorship from AWS (or from some other
  cloud
  provider at last resort)? I already tried to apply for AWS Activate's free
  credits, but my application was rejected due to different reasons
  (normally, this program is available for startups). I have some hope that
  running the benchmark on a dedicated instance would decrease variation of
  the results, but enabling this feature is relatively expensive compared to
  other costs, so I'd be very happy if I wouldn't have to sponsor all of
  this, when I already dedicate a great deal of my time to the cause.
• As always, I'm happy to receive feedback and improvements (ideally along
  with an implementation :) ). For example, currently I've started working on
  some visualization and chart support, but frontend stuff is not my area of
  expertise, so I'm progressing a bit slower than normally. Also, the
  statistical foundations could be reviewed, and possibly improved.

Regards:
Máté

Nice results, and nice project!

Overall against having it on the ML: I set up a filter to ignore those
pesky emails by Intel.
As much as I appreciate their effort, they were really noisy, and not
really useful to the larger group.

If your tool could export the results in machine format with something like
an attached XSD or JSON-Schema, it would be possible to build UIs like
https://arewefastyet.com/ , without the need to resort to a notification
model.
What's needed is to save them in a branch on the repo (or similar approach)
for later consumption.

Greets,

Marco Pivetta

http://twitter.com/Ocramius

http://ocramius.github.com/

Hello Internals,

I'm writing this email because lately, I've been working on an automatic
benchmarking framework for PHP, and I'd like to share some news regarding
it. The initial implementation
was sponsored by Craig Francis, and it was used for the evaluation of the
performance aspects of the is_literal() RFC. Since then, I fixed numerous
issues and implemented many new features, so with the very close advent of
PHP 8.1, the time has come to unveil it.

I'm sure that many of you still remember Intel's automatic benchmarks from
a couple of years ago (e.g. https://externals.io/message/89843#89843). I
loved these emails, so this project served as a great inspiration for me
to start working on something similar. I have to admit
though that I won't be able to replicate their extremely advanced setup.

My main goal was to develop a framework (
https://github.com/kocsismate/php-version-benchmarks) which is:

• fully automatic so that it can be easily run regularly, and the
  benchmarks are reproducible
• it's possible to try it out locally via Docker, but it can be run in the
  cloud (currently, only AWS is
  supported), and the instance type is configurable
• supports different CPU platforms (X86-64 and ARM64), and advanced options
  whether
  turbo boost/hyper threading/deeper CPU C-states are enabled is configurable
• supports any version of PHP since PHP 7.4, including any git branches or
  commits
• supports the most important PHP configurations, including whether
  OPcache/JIT/preloading
  is enabled
• supports multiple tests: currently, the micro benchmarks bundled with
  php-src, as well as the
  Symfony and the Laravel demo sites as "real-life" tests are included with
  some customizability (number of warmups, iterations number, number of
  requests)
• results are measured via using PHP-CGI which is a lightweight web server
  with very a small overhead

The most current benchmark is available at

https://github.com/kocsismate/php-version-benchmarks/blob/main/docs/results/2021_11_11_09_20_1_aws_arm64_c6g_4xlarge/result.md
. I'm happy to share that the results suggest PHP 8.1.0 is around 28-32%
faster than PHP 7.4 in real-life tests, and a few percent faster when it
comes to micro benchmarks. The benchmark was performed on an ARM64 instance
because this platform provided much more stable results than X86-64-based
ones did, mainly due to their fixed CPU frequency.

Unfortunately, the benchmark is not yet suitable for detecting minor
performance changes between commits as the run-to-run variation of the
results is a bit too high; in some cases, it can be up to 1-2%. I hope that
this problem can be mitigated to an acceptable level in the future, but
most probably I'll need some help to achieve this. So any help is
appreciated!

Furthermore, I'd have some questions regarding this project:

• Would you welcome automatic emails on internals (or on a dedicated
  mailing list), just like how
  Intel did it? I'm not sure about the frequency, but in my opinion,
  1-2/month would be a sensible one, given there is interest in it.
• Does anyone know how to get some sponsorship from AWS (or from some other
  cloud
  provider at last resort)? I already tried to apply for AWS Activate's free
  credits, but my application was rejected due to different reasons
  (normally, this program is available for startups). I have some hope that
  running the benchmark on a dedicated instance would decrease variation of
  the results, but enabling this feature is relatively expensive compared to
  other costs, so I'd be very happy if I wouldn't have to sponsor all of
  this, when I already dedicate a great deal of my time to the cause.
• As always, I'm happy to receive feedback and improvements (ideally along
  with an implementation :) ). For example, currently I've started working on
  some visualization and chart support, but frontend stuff is not my area of
  expertise, so I'm progressing a bit slower than normally. Also, the
  statistical foundations could be reviewed, and possibly improved.

Regards:
Máté