Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:126175
Content-Type: multipart/alternative; boundary=Apple-Mail-39C07BC4-B061-493D-9851-473825241EEB
Content-Transfer-Encoding: 7bit
Precedence: bulk
Mime-Version: 1.0 (1.0)
Subject: Re: [PHP-DEV] C Unit testing and mocking
Date: Tue, 24 Dec 2024 13:22:06 -0600
Message-ID: <D1158884-F704-4727-AF3C-E840C5B1A9A8@benramsey.com>
References: <CAEKnhAEFJKPWWL0EKu_JBPgwBEe+Pk3GDzR7HQm3J-g7=Wk0BA@mail.gmail.com>
Cc: PHP internals list <internals@lists.php.net>
In-Reply-To: <CAEKnhAEFJKPWWL0EKu_JBPgwBEe+Pk3GDzR7HQm3J-g7=Wk0BA@mail.gmail.com>
To: Jakub Zelenka <bukka@php.net>
From: ben@benramsey.com (Ben Ramsey)


--Apple-Mail-39C07BC4-B061-493D-9851-473825241EEB
Content-Type: text/plain;
	charset=utf-8
Content-Transfer-Encoding: quoted-printable

> On Dec 16, 2024, at 07:21, Jakub Zelenka <bukka@php.net> wrote:
>=20
> =EF=BB=BF
> Hi,
>=20
> I have been looking into how to test some cases where integration tests ar=
e very difficult or even impossible to create for. Those are often found in n=
etworking related and system specific code code (network.c, streams, FPM and=
 more). I was recently fixing one such bug and decided to give a try which r=
esulted in this PR: https://github.com/php/php-src/pull/16987 .
>=20
> There was a suggestion of RFC but that might be a bit too much as it's jus=
t an internal change / addition. But certainly some overview on internals sh=
ould be done so writing this instead.
>=20
> I decided to use cmocka in that PR because I had some experience with that=
. It's quite small and still very powerful and allow vast mocking options. I=
t's a bit manual but it gives a bigger control over the mock. It relies on -=
-wrap linking option that replaces original functions with wraps. This is ho=
wever available only on Linux (or maybe some other Unix variants) but doesn'=
t work on MacOS or Windows. The developers that want to use it on those plat=
forms would need to use some Linux Virtualisation option (e.g. Docker). It a=
lso requires static library which is supported by embed SAPI that can be com=
piled statically. That limits number of extensions to use but the main use c=
ases don't really have deps so it should be fine.
>=20
> I did also some research into the other mocking libraries in C. There is a=
 Unity with CMock, FFF and some C++ libs like GUnit, Criterion and Trompeloe=
il that I looked into. I quickly discarded GUnit and Trompeloeil as they rel=
ay on C++ virtual methods and require wrapping C code to C++ which is very i=
nconvenient. FFF seems too simple and maybe quite inflexible for our needs a=
s well. Criterion also optionally uses wrap so I didn't see much advantages c=
ompare to cmocka. So it left Unity with CMock that allows generating custom m=
ocks using a Ruby script. That seemed initially quite nice but after spendin=
g around two hours with trying to make it works for PHP codebase, I just gav=
e up. It gets quite messy for complex scenarios and I just didn't figure out=
 how to nicely mock libc functions without any modification to php-src.
>=20
> In terms of CI. It has got its own build which is very simple and it tests=
 just specific parts so we could just limit it to run only for changed files=
 which might be quite convenient.
>=20
> So the proposed PR is probably the only reasonable unit testing that I can=
 come up with. I think it should be completely optional initially for people=
 to use - more like an experiment. If it becomes used, then good of course. A=
nd if it becomes pain, we can just get rid of it. Has anyone got any objecti=
ons to get this merged? If not I plan to merge it early in January.
>=20
> Cheers
>=20
> Jakub


FWIW, as someone still very new to C, I found Criterion quite easy to use, a=
nd I was able to quickly grasp its concepts and start using it right away. I=
 can=E2=80=99t say the same for other C testing libraries I tried.

Cheers,
Ben


--Apple-Mail-39C07BC4-B061-493D-9851-473825241EEB
Content-Type: text/html;
	charset=utf-8
Content-Transfer-Encoding: quoted-printable

<html><head><meta http-equiv=3D"content-type" content=3D"text/html; charset=3D=
utf-8"></head><body dir=3D"auto"><div dir=3D"ltr"><blockquote type=3D"cite">=
On Dec 16, 2024, at 07:21, Jakub Zelenka &lt;bukka@php.net&gt; wrote:<br><br=
></blockquote></div><blockquote type=3D"cite"><div dir=3D"ltr">=EF=BB=BF<div=
 dir=3D"ltr">Hi,<div><br></div><div>I have been looking into how to test som=
e cases where integration tests are very difficult or even impossible to cre=
ate for. Those are often found in networking related and system specific cod=
e code (network.c, streams, FPM and more). I was recently fixing one such bu=
g and decided to give a try which resulted in this PR:&nbsp;<a href=3D"https=
://github.com/php/php-src/pull/16987">https://github.com/php/php-src/pull/16=
987</a> .</div><div><br></div><div>There was a suggestion of RFC but that mi=
ght be a bit too much as it's just an internal change / addition. But certai=
nly some overview on internals should be done so writing this instead.</div>=
<div><br></div><div>I decided to use cmocka in that PR because I had some ex=
perience with that. It's quite small and still very powerful and allow vast m=
ocking options. It's a bit manual but it gives a bigger control over the moc=
k. It relies on --wrap linking option that replaces original functions with w=
raps. This is however available only on Linux (or maybe some other Unix vari=
ants) but doesn't work on MacOS or Windows. The developers that want to use i=
t on those platforms would need to use some Linux Virtualisation option (e.g=
. Docker). It also requires static library which is supported by embed SAPI t=
hat can be compiled statically. That limits number of extensions to use but t=
he main use cases don't really have deps so it should be fine.</div><div><br=
></div><div>I did also some research into the other mocking libraries in C. T=
here is a Unity with CMock, FFF and some C++ libs like GUnit, Criterion and T=
rompeloeil that I looked into. I quickly discarded&nbsp;GUnit and&nbsp;Tromp=
eloeil as they relay on C++ virtual methods and require wrapping C code to C=
++ which is very inconvenient. FFF seems too simple and maybe quite inflexib=
le for our needs as well. Criterion also optionally uses wrap so I didn't se=
e much advantages compare to cmocka. So it left Unity with CMock that allows=
 generating custom mocks using a Ruby script. That seemed initially quite ni=
ce but after spending around two hours with trying to make it works for PHP c=
odebase, I just gave up. It gets quite messy for complex scenarios and I jus=
t didn't figure out how to nicely mock libc functions without any modificati=
on to php-src.</div><div><br></div><div>In terms of CI. It has got its own b=
uild which is very simple and it tests just specific parts so we could just l=
imit it to run only for changed files which might be quite convenient.</div>=
<div><br></div><div>So the proposed PR is probably the only reasonable unit t=
esting that I can come up with. I think it should be completely optional ini=
tially for people to use - more like an experiment. If it becomes used, then=
 good of course. And if it becomes pain, we can just get rid of it. Has anyo=
ne got any objections to get this merged? If not I plan to merge it early in=
 January.</div><div><br></div><div>Cheers</div><div><br></div><div>Jakub</di=
v></div>
</div></blockquote><br><div><br></div><div>FWIW, as someone still very new t=
o C, I found Criterion quite easy to use, and I was able to quickly grasp it=
s concepts and start using it right away. I can=E2=80=99t say the same for o=
ther C testing libraries I tried.</div><div><br></div><div>Cheers,</div><div=
>Ben</div><div><br></div></body></html>=

--Apple-Mail-39C07BC4-B061-493D-9851-473825241EEB--