Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:116342
MIME-Version: 1.0
References: <CAH5C8xUXONMBPBYav1rU5ese0G0kE5sXZCxqqrSTy1njU2kbzg@mail.gmail.com>
In-Reply-To: <CAH5C8xUXONMBPBYav1rU5ese0G0kE5sXZCxqqrSTy1njU2kbzg@mail.gmail.com>
Date: Sat, 13 Nov 2021 13:37:12 +0100
Message-ID: <CADyq6s+0Jf=npkqWBRNE11WAryxp1mbPxbHku29v9Qk+Owrvaw@mail.gmail.com>
To: =?UTF-8?B?TcOhdMOpIEtvY3Npcw==?= <kocsismate90@gmail.com>
Cc: PHP Internals List <internals@lists.php.net>
Content-Type: multipart/alternative; boundary="000000000000860c8405d0aad697"
Subject: Re: [PHP-DEV] Automatic performance benchmarking: PHP 8.1 is ~30%
 faster than PHP 7.4
From: ocramius@gmail.com (Marco Pivetta)

--000000000000860c8405d0aad697
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hey M=C3=A1t=C3=A9,

On Fri, Nov 12, 2021 at 11:19 AM M=C3=A1t=C3=A9 Kocsis <kocsismate90@gmail.=
com> wrote:

> Hello Internals,
>
> I'm writing this email because lately, I've been working on an automatic
> benchmarking framework for PHP, and I'd like to share some news regarding
> it. The initial implementation
> was sponsored by Craig Francis, and it was used for the evaluation of the
> performance aspects of the is_literal() RFC. Since then, I fixed numerous
> issues and implemented many new features, so with the very close advent o=
f
> PHP 8.1, the time has come to unveil it.
>
> I'm sure that many of you still remember Intel's automatic benchmarks fro=
m
> a couple of years ago (e.g. https://externals.io/message/89843#89843). I
> loved these emails, so this project served as a great inspiration for me
> to start working on something similar. I have to admit
> though that I won't be able to replicate their extremely advanced setup.
>
> My main goal was to develop a framework (
> https://github.com/kocsismate/php-version-benchmarks) which is:
> - fully automatic so that it can be easily run regularly, and the
> benchmarks are reproducible
> - it's possible to try it out locally via Docker, but it can be run in th=
e
> cloud (currently, only AWS is
> supported), and the instance type is configurable
> - supports different CPU platforms (X86-64 and ARM64), and advanced optio=
ns
> whether
> turbo boost/hyper threading/deeper CPU C-states are enabled is configurab=
le
> - supports any version of PHP since PHP 7.4, including any git branches o=
r
> commits
> - supports the most important PHP configurations, including whether
> OPcache/JIT/preloading
> is enabled
> - supports multiple tests: currently, the micro benchmarks bundled with
> php-src, as well as the
> Symfony and the Laravel demo sites as "real-life" tests are included with
> some customizability (number of warmups, iterations number, number of
> requests)
> - results are measured via using PHP-CGI which is a lightweight web serve=
r
> with very a small overhead
>
> The most current benchmark is available at
>
> https://github.com/kocsismate/php-version-benchmarks/blob/main/docs/resul=
ts/2021_11_11_09_20_1_aws_arm64_c6g_4xlarge/result.md
> . I'm happy to share that the results suggest PHP 8.1.0 is around 28-32%
> faster than PHP 7.4 in real-life tests, and a few percent faster when it
> comes to micro benchmarks. The benchmark was performed on an ARM64 instan=
ce
> because this platform provided much more stable results than X86-64-based
> ones did, mainly due to their fixed CPU frequency.
>
> Unfortunately, the benchmark is not yet suitable for detecting minor
> performance changes between commits as the run-to-run variation of the
> results is a bit too high; in some cases, it can be up to 1-2%. I hope th=
at
> this problem can be mitigated to an acceptable level in the future, but
> most probably I'll need some help to achieve this. So any help is
> appreciated!
>
> Furthermore, I'd have some questions regarding this project:
> - Would you welcome automatic emails on internals (or on a dedicated
> mailing list), just like how
> Intel did it? I'm not sure about the frequency, but in my opinion,
> 1-2/month would be a sensible one, given there is interest in it.
> - Does anyone know how to get some sponsorship from AWS (or from some oth=
er
> cloud
> provider at last resort)? I already tried to apply for AWS Activate's fre=
e
> credits, but my application was rejected due to different reasons
> (normally, this program is available for startups). I have some hope that
> running the benchmark on a dedicated instance would decrease variation of
> the results, but enabling this feature is relatively expensive compared t=
o
> other costs, so I'd be very happy if I wouldn't have to sponsor all of
> this, when I already dedicate a great deal of my time to the cause.
> - As always, I'm happy to receive feedback and improvements (ideally alon=
g
> with an implementation :) ). For example, currently I've started working =
on
> some visualization and chart support, but frontend stuff is not my area o=
f
> expertise, so I'm progressing a bit slower than normally. Also, the
> statistical foundations could be reviewed, and possibly improved.
>
> Regards:
> M=C3=A1t=C3=A9
>

Nice results, and nice project!

Overall against having it on the ML: I set up a filter to ignore those
pesky emails by Intel.
As much as I appreciate their effort, they were really noisy, and not
really useful to the larger group.

If your tool could export the results in machine format with something like
an attached XSD or JSON-Schema, it would be possible to build UIs like
https://arewefastyet.com/ , without the need to resort to a notification
model.
What's needed is to save them in a branch on the repo (or similar approach)
for later consumption.

Greets,

Marco Pivetta

http://twitter.com/Ocramius

http://ocramius.github.com/

On Fri, Nov 12, 2021 at 11:19 AM M=C3=A1t=C3=A9 Kocsis <kocsismate90@gmail.=
com> wrote:

> Hello Internals,
>
> I'm writing this email because lately, I've been working on an automatic
> benchmarking framework for PHP, and I'd like to share some news regarding
> it. The initial implementation
> was sponsored by Craig Francis, and it was used for the evaluation of the
> performance aspects of the is_literal() RFC. Since then, I fixed numerous
> issues and implemented many new features, so with the very close advent o=
f
> PHP 8.1, the time has come to unveil it.
>
> I'm sure that many of you still remember Intel's automatic benchmarks fro=
m
> a couple of years ago (e.g. https://externals.io/message/89843#89843). I
> loved these emails, so this project served as a great inspiration for me
> to start working on something similar. I have to admit
> though that I won't be able to replicate their extremely advanced setup.
>
> My main goal was to develop a framework (
> https://github.com/kocsismate/php-version-benchmarks) which is:
> - fully automatic so that it can be easily run regularly, and the
> benchmarks are reproducible
> - it's possible to try it out locally via Docker, but it can be run in th=
e
> cloud (currently, only AWS is
> supported), and the instance type is configurable
> - supports different CPU platforms (X86-64 and ARM64), and advanced optio=
ns
> whether
> turbo boost/hyper threading/deeper CPU C-states are enabled is configurab=
le
> - supports any version of PHP since PHP 7.4, including any git branches o=
r
> commits
> - supports the most important PHP configurations, including whether
> OPcache/JIT/preloading
> is enabled
> - supports multiple tests: currently, the micro benchmarks bundled with
> php-src, as well as the
> Symfony and the Laravel demo sites as "real-life" tests are included with
> some customizability (number of warmups, iterations number, number of
> requests)
> - results are measured via using PHP-CGI which is a lightweight web serve=
r
> with very a small overhead
>
> The most current benchmark is available at
>
> https://github.com/kocsismate/php-version-benchmarks/blob/main/docs/resul=
ts/2021_11_11_09_20_1_aws_arm64_c6g_4xlarge/result.md
> . I'm happy to share that the results suggest PHP 8.1.0 is around 28-32%
> faster than PHP 7.4 in real-life tests, and a few percent faster when it
> comes to micro benchmarks. The benchmark was performed on an ARM64 instan=
ce
> because this platform provided much more stable results than X86-64-based
> ones did, mainly due to their fixed CPU frequency.
>
> Unfortunately, the benchmark is not yet suitable for detecting minor
> performance changes between commits as the run-to-run variation of the
> results is a bit too high; in some cases, it can be up to 1-2%. I hope th=
at
> this problem can be mitigated to an acceptable level in the future, but
> most probably I'll need some help to achieve this. So any help is
> appreciated!
>
> Furthermore, I'd have some questions regarding this project:
> - Would you welcome automatic emails on internals (or on a dedicated
> mailing list), just like how
> Intel did it? I'm not sure about the frequency, but in my opinion,
> 1-2/month would be a sensible one, given there is interest in it.
> - Does anyone know how to get some sponsorship from AWS (or from some oth=
er
> cloud
> provider at last resort)? I already tried to apply for AWS Activate's fre=
e
> credits, but my application was rejected due to different reasons
> (normally, this program is available for startups). I have some hope that
> running the benchmark on a dedicated instance would decrease variation of
> the results, but enabling this feature is relatively expensive compared t=
o
> other costs, so I'd be very happy if I wouldn't have to sponsor all of
> this, when I already dedicate a great deal of my time to the cause.
> - As always, I'm happy to receive feedback and improvements (ideally alon=
g
> with an implementation :) ). For example, currently I've started working =
on
> some visualization and chart support, but frontend stuff is not my area o=
f
> expertise, so I'm progressing a bit slower than normally. Also, the
> statistical foundations could be reviewed, and possibly improved.
>
> Regards:
> M=C3=A1t=C3=A9
>

--000000000000860c8405d0aad697--