Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:126127
Precedence: bulk
MIME-Version: 1.0
Date: Mon, 16 Dec 2024 14:18:46 +0100
Message-ID: <CAEKnhAEFJKPWWL0EKu_JBPgwBEe+Pk3GDzR7HQm3J-g7=Wk0BA@mail.gmail.com>
Subject: [PHP-DEV] C Unit testing and mocking
To: PHP internals list <internals@lists.php.net>
Content-Type: multipart/alternative; boundary="000000000000de12110629630562"
From: bukka@php.net (Jakub Zelenka)

--000000000000de12110629630562
Content-Type: text/plain; charset="UTF-8"

Hi,

I have been looking into how to test some cases where integration tests are
very difficult or even impossible to create for. Those are often found in
networking related and system specific code code (network.c, streams, FPM
and more). I was recently fixing one such bug and decided to give a try
which resulted in this PR: https://github.com/php/php-src/pull/16987 .

There was a suggestion of RFC but that might be a bit too much as it's just
an internal change / addition. But certainly some overview on internals
should be done so writing this instead.

I decided to use cmocka in that PR because I had some experience with that.
It's quite small and still very powerful and allow vast mocking options.
It's a bit manual but it gives a bigger control over the mock. It relies on
--wrap linking option that replaces original functions with wraps. This is
however available only on Linux (or maybe some other Unix variants) but
doesn't work on MacOS or Windows. The developers that want to use it on
those platforms would need to use some Linux Virtualisation option (e.g.
Docker). It also requires static library which is supported by embed SAPI
that can be compiled statically. That limits number of extensions to use
but the main use cases don't really have deps so it should be fine.

I did also some research into the other mocking libraries in C. There is a
Unity with CMock, FFF and some C++ libs like GUnit, Criterion and
Trompeloeil that I looked into. I quickly discarded GUnit and Trompeloeil
as they relay on C++ virtual methods and require wrapping C code to C++
which is very inconvenient. FFF seems too simple and maybe quite inflexible
for our needs as well. Criterion also optionally uses wrap so I didn't see
much advantages compare to cmocka. So it left Unity with CMock that allows
generating custom mocks using a Ruby script. That seemed initially quite
nice but after spending around two hours with trying to make it works for
PHP codebase, I just gave up. It gets quite messy for complex scenarios and
I just didn't figure out how to nicely mock libc functions without any
modification to php-src.

In terms of CI. It has got its own build which is very simple and it tests
just specific parts so we could just limit it to run only for changed files
which might be quite convenient.

So the proposed PR is probably the only reasonable unit testing that I can
come up with. I think it should be completely optional initially for people
to use - more like an experiment. If it becomes used, then good of course.
And if it becomes pain, we can just get rid of it. Has anyone got any
objections to get this merged? If not I plan to merge it early in January.

Cheers

Jakub

--000000000000de12110629630562
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi,<div><br></div><div>I have been looking into how to tes=
t some cases where integration tests are very difficult or even impossible =
to create for. Those are often found in networking related and system speci=
fic code code (network.c, streams, FPM and more). I was recently fixing one=
 such bug and decided to give a try which resulted in this PR:=C2=A0<a href=
=3D"https://github.com/php/php-src/pull/16987">https://github.com/php/php-s=
rc/pull/16987</a> .</div><div><br></div><div>There was a suggestion of RFC =
but that might be a bit too much as it&#39;s just an internal change / addi=
tion. But certainly some overview on internals should be done so writing th=
is instead.</div><div><br></div><div>I decided to use cmocka in that PR bec=
ause I had some experience with that. It&#39;s quite small and still very p=
owerful and allow vast mocking options. It&#39;s a bit manual but it gives =
a bigger control over the mock. It relies on --wrap linking option that rep=
laces original functions with wraps. This is however available only on Linu=
x (or maybe some other Unix variants) but doesn&#39;t work on MacOS or Wind=
ows. The developers that want to use it on those platforms would need to us=
e some Linux Virtualisation option (e.g. Docker). It also requires static l=
ibrary which is supported by embed SAPI that can be compiled statically. Th=
at limits number of extensions to use but the main use cases don&#39;t real=
ly have deps so it should be fine.</div><div><br></div><div>I did also some=
 research into the other mocking libraries in C. There is a Unity with CMoc=
k, FFF and some C++ libs like GUnit, Criterion and Trompeloeil that I looke=
d into. I quickly discarded=C2=A0GUnit and=C2=A0Trompeloeil as they relay o=
n C++ virtual methods and require wrapping C code to C++ which is very inco=
nvenient. FFF seems too simple and maybe quite inflexible for our needs as =
well. Criterion also optionally uses wrap so I didn&#39;t see much advantag=
es compare to cmocka. So it left Unity with CMock that allows generating cu=
stom mocks using a Ruby script. That seemed initially quite nice but after =
spending around two hours with trying to make it works for PHP codebase, I =
just gave up. It gets quite messy for complex scenarios and I just didn&#39=
;t figure out how to nicely mock libc functions without any modification to=
 php-src.</div><div><br></div><div>In terms of CI. It has got its own build=
 which is very simple and it tests just specific parts so we could just lim=
it it to run only for changed files which might be quite convenient.</div><=
div><br></div><div>So the proposed PR is probably the only reasonable unit =
testing that I can come up with. I think it should be completely optional i=
nitially for people to use - more like an experiment. If it becomes used, t=
hen good of course. And if it becomes pain, we can just get rid of it. Has =
anyone got any objections to get this merged? If not I plan to merge it ear=
ly in January.</div><div><br></div><div>Cheers</div><div><br></div><div>Jak=
ub</div><div><br></div></div>

--000000000000de12110629630562--