Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:126177
Precedence: bulk
MIME-Version: 1.0
References: <CAEKnhAEFJKPWWL0EKu_JBPgwBEe+Pk3GDzR7HQm3J-g7=Wk0BA@mail.gmail.com>
 <D1158884-F704-4727-AF3C-E840C5B1A9A8@benramsey.com>
In-Reply-To: <D1158884-F704-4727-AF3C-E840C5B1A9A8@benramsey.com>
Date: Thu, 26 Dec 2024 13:19:27 +0100
Message-ID: <CAEKnhAEuZyiSLFOVg+A=iDRmMbib5uKxEjCwTqaw0wp1zDx9EA@mail.gmail.com>
Subject: Re: [PHP-DEV] C Unit testing and mocking
To: Ben Ramsey <ben@benramsey.com>
Cc: PHP internals list <internals@lists.php.net>
Content-Type: multipart/alternative; boundary="00000000000027234b062a2b5ca4"
From: bukka@php.net (Jakub Zelenka)

--00000000000027234b062a2b5ca4
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Tue, Dec 24, 2024 at 8:22=E2=80=AFPM Ben Ramsey <ben@benramsey.com> wrot=
e:

> On Dec 16, 2024, at 07:21, Jakub Zelenka <bukka@php.net> wrote:
>
> =EF=BB=BF
> Hi,
>
> I have been looking into how to test some cases where integration tests
> are very difficult or even impossible to create for. Those are often foun=
d
> in networking related and system specific code code (network.c, streams,
> FPM and more). I was recently fixing one such bug and decided to give a t=
ry
> which resulted in this PR: https://github.com/php/php-src/pull/16987 .
>
> There was a suggestion of RFC but that might be a bit too much as it's
> just an internal change / addition. But certainly some overview on
> internals should be done so writing this instead.
>
> I decided to use cmocka in that PR because I had some experience with
> that. It's quite small and still very powerful and allow vast mocking
> options. It's a bit manual but it gives a bigger control over the mock. I=
t
> relies on --wrap linking option that replaces original functions with
> wraps. This is however available only on Linux (or maybe some other Unix
> variants) but doesn't work on MacOS or Windows. The developers that want =
to
> use it on those platforms would need to use some Linux Virtualisation
> option (e.g. Docker). It also requires static library which is supported =
by
> embed SAPI that can be compiled statically. That limits number of
> extensions to use but the main use cases don't really have deps so it
> should be fine.
>
> I did also some research into the other mocking libraries in C. There is =
a
> Unity with CMock, FFF and some C++ libs like GUnit, Criterion and
> Trompeloeil that I looked into. I quickly discarded GUnit and Trompeloeil
> as they relay on C++ virtual methods and require wrapping C code to C++
> which is very inconvenient. FFF seems too simple and maybe quite inflexib=
le
> for our needs as well. Criterion also optionally uses wrap so I didn't se=
e
> much advantages compare to cmocka. So it left Unity with CMock that allow=
s
> generating custom mocks using a Ruby script. That seemed initially quite
> nice but after spending around two hours with trying to make it works for
> PHP codebase, I just gave up. It gets quite messy for complex scenarios a=
nd
> I just didn't figure out how to nicely mock libc functions without any
> modification to php-src.
>
> In terms of CI. It has got its own build which is very simple and it test=
s
> just specific parts so we could just limit it to run only for changed fil=
es
> which might be quite convenient.
>
> So the proposed PR is probably the only reasonable unit testing that I ca=
n
> come up with. I think it should be completely optional initially for peop=
le
> to use - more like an experiment. If it becomes used, then good of course=
.
> And if it becomes pain, we can just get rid of it. Has anyone got any
> objections to get this merged? If not I plan to merge it early in January=
.
>
> Cheers
>
> Jakub
>
>
>
> FWIW, as someone still very new to C, I found Criterion quite easy to use=
,
> and I was able to quickly grasp its concepts and start using it right awa=
y.
> I can=E2=80=99t say the same for other C testing libraries I tried.
>

I just checked Criterion a bit more and it actually does not have built-in
mocking. It could be used with --wrap but it doesn't have any expectations
like cmocka. There's Mimick [1] from the same author which is indendent but
its docs still say that it's experimental so probably not a good idea to
use an experimental tool as its API can change. So all in all it's not
probably an option for us.

[1] https://github.com/Snaipe/Mimick

Cheers

Jakub

--00000000000027234b062a2b5ca4
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div dir=3D"ltr">On Tue, Dec 24, 2024 at 8:22=E2=80=AFPM B=
en Ramsey &lt;<a href=3D"mailto:ben@benramsey.com">ben@benramsey.com</a>&gt=
; wrote:</div><div class=3D"gmail_quote gmail_quote_container"><blockquote =
class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px sol=
id rgb(204,204,204);padding-left:1ex"><div dir=3D"auto"><div dir=3D"ltr"><b=
lockquote type=3D"cite">On Dec 16, 2024, at 07:21, Jakub Zelenka &lt;<a hre=
f=3D"mailto:bukka@php.net" target=3D"_blank">bukka@php.net</a>&gt; wrote:<b=
r><br></blockquote></div><blockquote type=3D"cite"><div dir=3D"ltr">=EF=BB=
=BF<div dir=3D"ltr">Hi,<div><br></div><div>I have been looking into how to =
test some cases where integration tests are very difficult or even impossib=
le to create for. Those are often found in networking related and system sp=
ecific code code (network.c, streams, FPM and more). I was recently fixing =
one such bug and decided to give a try which resulted in this PR:=C2=A0<a h=
ref=3D"https://github.com/php/php-src/pull/16987" target=3D"_blank">https:/=
/github.com/php/php-src/pull/16987</a> .</div><div><br></div><div>There was=
 a suggestion of RFC but that might be a bit too much as it&#39;s just an i=
nternal change / addition. But certainly some overview on internals should =
be done so writing this instead.</div><div><br></div><div>I decided to use =
cmocka in that PR because I had some experience with that. It&#39;s quite s=
mall and still very powerful and allow vast mocking options. It&#39;s a bit=
 manual but it gives a bigger control over the mock. It relies on --wrap li=
nking option that replaces original functions with wraps. This is however a=
vailable only on Linux (or maybe some other Unix variants) but doesn&#39;t =
work on MacOS or Windows. The developers that want to use it on those platf=
orms would need to use some Linux Virtualisation option (e.g. Docker). It a=
lso requires static library which is supported by embed SAPI that can be co=
mpiled statically. That limits number of extensions to use but the main use=
 cases don&#39;t really have deps so it should be fine.</div><div><br></div=
><div>I did also some research into the other mocking libraries in C. There=
 is a Unity with CMock, FFF and some C++ libs like GUnit, Criterion and Tro=
mpeloeil that I looked into. I quickly discarded=C2=A0GUnit and=C2=A0Trompe=
loeil as they relay on C++ virtual methods and require wrapping C code to C=
++ which is very inconvenient. FFF seems too simple and maybe quite inflexi=
ble for our needs as well. Criterion also optionally uses wrap so I didn=
9;t see much advantages compare to cmocka. So it left Unity with CMock that=
 allows generating custom mocks using a Ruby script. That seemed initially =
quite nice but after spending around two hours with trying to make it works=
 for PHP codebase, I just gave up. It gets quite messy for complex scenario=
s and I just didn&#39;t figure out how to nicely mock libc functions withou=
t any modification to php-src.</div><div><br></div><div>In terms of CI. It =
has got its own build which is very simple and it tests just specific parts=
 so we could just limit it to run only for changed files which might be qui=
te convenient.</div><div><br></div><div>So the proposed PR is probably the =
only reasonable unit testing that I can come up with. I think it should be =
completely optional initially for people to use - more like an experiment. =
If it becomes used, then good of course. And if it becomes pain, we can jus=
t get rid of it. Has anyone got any objections to get this merged? If not I=
 plan to merge it early in January.</div><div><br></div><div>Cheers</div><d=
iv><br></div><div>Jakub</div></div>
</div></blockquote><br><div><br></div><div>FWIW, as someone still very new =
to C, I found Criterion quite easy to use, and I was able to quickly grasp =
its concepts and start using it right away. I can=E2=80=99t say the same fo=
r other C testing libraries I tried.</div></div></blockquote><div><br></div=
><div>I just checked Criterion a bit more and it actually does not have bui=
lt-in mocking. It could be used with --wrap but it doesn&#39;t have any exp=
ectations like cmocka. There&#39;s Mimick [1] from the same author which is=
 indendent but its docs still say that it&#39;s experimental so probably no=
t a good idea to use an experimental tool as its API can change. So all in =
all it&#39;s not probably an option for us.</div><div><br></div><div>[1]=C2=
=A0<a href=3D"https://github.com/Snaipe/Mimick">https://github.com/Snaipe/M=
imick</a></div><div><br></div><div>Cheers</div><div><br></div><div>Jakub</d=
iv></div></div>

--00000000000027234b062a2b5ca4--