Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:125708
Precedence: bulk
MIME-Version: 1.0
Date: Sun, 29 Sep 2024 13:55:48 +0200
Message-ID: <CAH5C8xVbHe-gKu6_Tob4kwBxicXrQf1YxiCu2P_qxHZckOZ9-g@mail.gmail.com>
Subject: [PHP-DEV] Real-time benchmark suite has just been added
To: PHP Internals List <internals@lists.php.net>
Content-Type: multipart/alternative; boundary="0000000000009d9cf7062340c58c"
From: kocsismate90@gmail.com (=?UTF-8?B?TcOhdMOpIEtvY3Npcw==?=)

--0000000000009d9cf7062340c58c
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hi Everyone,

It's been a while when I sent my email about introducing an automatic
benchmark suite I had been working:
https://externals.io/message/116323#116323 Now I would like to share an
important milestone of this project:

After doing lots of improvements and fixes throughout the year, the suite
now reached the
required maturity level to be able to use it within PHP's nightly CI jobs
(experimentally for now).
Previously, consistency and reliability issues, as well as the lack of
financial support prevented
us from doing so. As these issues were addressed by now, we will be able to
track the
performance changes of PHP, starting with 8.5.

The initial results are very promising for real-word app based tests (Larav=
el,
Symfony,
Wordpress), indicating that the results are consistent and seemingly
reliable. For example run to
run variation of the median absolute results typically don't exceed 0.001
sec. When
benchmarking a php-src commit against itself, the relative difference for
each real-word test
varied between 0.01% and 0.06% (while 0.0% would have been the best
possible value).

Unfortunately, unlike real-word apps, micro benchmarks (bench.php and
micro_bench.php)
sometimes give inconsistent results, so there is still room for
improvements. Besides making
micro benchmarks more reliable, future plans include adding visualization,
and coming up with
some notification system which tells us if there is a notable performance
regression or progression.

For anyone interested in the project and the results, the benchmark suite
is available at
https://github.com/kocsismate/php-version-benchmarks, while the nightly
results are committed
into https://github.com/php/real-time-benchmark-data in both human friendly
(markdown) and in
machine processable format (tsv). Additionally, a more detailed explanation
about the benchmark's methodology can be found at
https://github.com/php/php-src/pull/15696.

Regards,
M=C3=A1t=C3=A9

--0000000000009d9cf7062340c58c
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi Everyone,<div><br></div><div>It&#39;s been a while when=
 I sent my email about introducing an automatic benchmark suite I had been =
working:=C2=A0<a href=3D"https://externals.io/message/116323#116323">https:=
//externals.io/message/116323#116323</a> Now I would like to share an</div>=
<div>important milestone of this project:</div><div><br></div><div>After do=
ing lots of improvements and fixes throughout the year, the=C2=A0suite now =
reached the</div><div>required maturity level to be able to use it within P=
HP&#39;s nightly CI jobs (experimentally for now).</div><div>Previously, co=
nsistency and reliability issues, as well as the lack of financial support =
prevented</div><div>us from doing so. As these issues were addressed by now=
, we will be able to track the</div><div>performance changes of PHP, starti=
ng with 8.5.</div><div><br></div><div>The initial results are very promisin=
g for real-word app based tests=C2=A0<span style=3D"color:rgb(191,191,191)"=
>(Laravel, Symfony,</span></div><div><span style=3D"color:rgb(191,191,191)"=
>Wordpress)</span><span style=3D"color:rgb(191,191,191)">, indicating that =
the results are consistent and seemingly reliable. For example run to</span=
></div><div><span style=3D"color:rgb(191,191,191)">run variation of the med=
ian absolute results typically=C2=A0</span><span style=3D"color:rgb(191,191=
,191)">don&#39;t exceed 0.001 sec. When</span></div><div><span style=3D"col=
or:rgb(191,191,191)">benchmarking a php-src commit against itself, the rela=
tive difference for each real-word test</span></div><div><span style=3D"col=
or:rgb(191,191,191)">varied between 0.01% and 0.06% (while 0.0% would have =
been the best possible value).</span></div><div><span style=3D"color:rgb(19=
1,191,191)"><br></span></div><div><span style=3D"color:rgb(191,191,191)">Un=
fortunately, unlike real-word apps, micro benchmarks (bench.php and micro_b=
ench.php)</span></div><div><span style=3D"color:rgb(191,191,191)">sometimes=
 give inconsistent results, so there is still room for improvements. Beside=
s making</span></div><div><span style=3D"color:rgb(191,191,191)">micro </sp=
an>benchmarks more<span style=3D"color:rgb(191,191,191)"> reliable, future=
=C2=A0plans include adding visualization, and coming up with</span></div><d=
iv><span style=3D"color:rgb(191,191,191)">some notification system which te=
lls us if there is a notable performance regression or progression.</span><=
/div><div><br></div><div>For anyone interested in the project and the resul=
ts, the benchmark suite is available at</div><div><a href=3D"https://github=
.com/kocsismate/php-version-benchmarks">https://github.com/kocsismate/php-v=
ersion-benchmarks</a>, while the nightly results are committed</div><div>in=
to <a href=3D"https://github.com/php/real-time-benchmark-data">https://gith=
ub.com/php/real-time-benchmark-data</a> in both human friendly (markdown) a=
nd in</div><div>machine processable format (tsv).=C2=A0<span style=3D"color=
:rgb(191,191,191)">Additionally, a=C2=A0more detailed explanation about the=
 benchmark&#39;s methodology can be found at <a href=3D"https://github.com/=
php/php-src/pull/15696">https://github.com/php/php-src/pull/15696</a>.</spa=
n></div><div><br></div><div>Regards,</div><div>M=C3=A1t=C3=A9</div><div><br=
></div></div>

--0000000000009d9cf7062340c58c--