Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:116323
MIME-Version: 1.0
Date: Fri, 12 Nov 2021 11:18:50 +0100
Message-ID: <CAH5C8xUXONMBPBYav1rU5ese0G0kE5sXZCxqqrSTy1njU2kbzg@mail.gmail.com>
To: PHP Internals List <internals@lists.php.net>
Content-Type: multipart/alternative; boundary="000000000000b95d3605d094c9b0"
Subject: Automatic performance benchmarking: PHP 8.1 is ~30% faster than PHP 7.4
From: kocsismate90@gmail.com (=?UTF-8?B?TcOhdMOpIEtvY3Npcw==?=)

--000000000000b95d3605d094c9b0
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hello Internals,

I'm writing this email because lately, I've been working on an automatic
benchmarking framework for PHP, and I'd like to share some news regarding
it. The initial implementation
was sponsored by Craig Francis, and it was used for the evaluation of the
performance aspects of the is_literal() RFC. Since then, I fixed numerous
issues and implemented many new features, so with the very close advent of
PHP 8.1, the time has come to unveil it.

I'm sure that many of you still remember Intel's automatic benchmarks from
a couple of years ago (e.g. https://externals.io/message/89843#89843). I
loved these emails, so this project served as a great inspiration for me
to start working on something similar. I have to admit
though that I won't be able to replicate their extremely advanced setup.

My main goal was to develop a framework (
https://github.com/kocsismate/php-version-benchmarks) which is:
- fully automatic so that it can be easily run regularly, and the
benchmarks are reproducible
- it's possible to try it out locally via Docker, but it can be run in the
cloud (currently, only AWS is
supported), and the instance type is configurable
- supports different CPU platforms (X86-64 and ARM64), and advanced options
whether
turbo boost/hyper threading/deeper CPU C-states are enabled is configurable
- supports any version of PHP since PHP 7.4, including any git branches or
commits
- supports the most important PHP configurations, including whether
OPcache/JIT/preloading
is enabled
- supports multiple tests: currently, the micro benchmarks bundled with
php-src, as well as the
Symfony and the Laravel demo sites as "real-life" tests are included with
some customizability (number of warmups, iterations number, number of
requests)
- results are measured via using PHP-CGI which is a lightweight web server
with very a small overhead

The most current benchmark is available at
https://github.com/kocsismate/php-version-benchmarks/blob/main/docs/results=
/2021_11_11_09_20_1_aws_arm64_c6g_4xlarge/result.md
. I'm happy to share that the results suggest PHP 8.1.0 is around 28-32%
faster than PHP 7.4 in real-life tests, and a few percent faster when it
comes to micro benchmarks. The benchmark was performed on an ARM64 instance
because this platform provided much more stable results than X86-64-based
ones did, mainly due to their fixed CPU frequency.

Unfortunately, the benchmark is not yet suitable for detecting minor
performance changes between commits as the run-to-run variation of the
results is a bit too high; in some cases, it can be up to 1-2%. I hope that
this problem can be mitigated to an acceptable level in the future, but
most probably I'll need some help to achieve this. So any help is
appreciated!

Furthermore, I'd have some questions regarding this project:
- Would you welcome automatic emails on internals (or on a dedicated
mailing list), just like how
Intel did it? I'm not sure about the frequency, but in my opinion,
1-2/month would be a sensible one, given there is interest in it.
- Does anyone know how to get some sponsorship from AWS (or from some other
cloud
provider at last resort)? I already tried to apply for AWS Activate's free
credits, but my application was rejected due to different reasons
(normally, this program is available for startups). I have some hope that
running the benchmark on a dedicated instance would decrease variation of
the results, but enabling this feature is relatively expensive compared to
other costs, so I'd be very happy if I wouldn't have to sponsor all of
this, when I already dedicate a great deal of my time to the cause.
- As always, I'm happy to receive feedback and improvements (ideally along
with an implementation :) ). For example, currently I've started working on
some visualization and chart support, but frontend stuff is not my area of
expertise, so I'm progressing a bit slower than normally. Also, the
statistical foundations could be reviewed, and possibly improved.

Regards:
M=C3=A1t=C3=A9

--000000000000b95d3605d094c9b0--