C Unit testing and mocking

Hi,

I have been looking into how to test some cases where integration tests are very difficult or even impossible to create for. Those are often found in networking related and system specific code code (network.c, streams, FPM and more). I was recently fixing one such bug and decided to give a try which resulted in this PR: https://github.com/php/php-src/pull/16987 .

There was a suggestion of RFC but that might be a bit too much as it's just an internal change / addition. But certainly some overview on internals should be done so writing this instead.

I decided to use cmocka in that PR because I had some experience with that. It's quite small and still very powerful and allow vast mocking options. It's a bit manual but it gives a bigger control over the mock. It relies on --wrap linking option that replaces original functions with wraps. This is however available only on Linux (or maybe some other Unix variants) but doesn't work on MacOS or Windows. The developers that want to use it on those platforms would need to use some Linux Virtualisation option (e.g. Docker). It also requires static library which is supported by embed SAPI that can be compiled statically. That limits number of extensions to use but the main use cases don't really have deps so it should be fine.

I did also some research into the other mocking libraries in C. There is a Unity with CMock, FFF and some C++ libs like GUnit, Criterion and Trompeloeil that I looked into. I quickly discarded GUnit and Trompeloeil as they relay on C++ virtual methods and require wrapping C code to C++ which is very inconvenient. FFF seems too simple and maybe quite inflexible for our needs as well. Criterion also optionally uses wrap so I didn't see much advantages compare to cmocka. So it left Unity with CMock that allows generating custom mocks using a Ruby script. That seemed initially quite nice but after spending around two hours with trying to make it works for PHP codebase, I just gave up. It gets quite messy for complex scenarios and I just didn't figure out how to nicely mock libc functions without any modification to php-src.

In terms of CI. It has got its own build which is very simple and it tests just specific parts so we could just limit it to run only for changed files which might be quite convenient.

So the proposed PR is probably the only reasonable unit testing that I can come up with. I think it should be completely optional initially for people to use - more like an experiment. If it becomes used, then good of course. And if it becomes pain, we can just get rid of it. Has anyone got any objections to get this merged? If not I plan to merge it early in January.

Cheers

Jakub

I'm assuming that uses ELF symbol interposition or something like that,
which is why it seems Linux/BSD specific. That seems fragile to me.

I think currently for wanting to test C functions, we're adding custom
functions into ext/zend_test and writing PHPT. Would this work? We
already have that, after all. If not, it'd be helpful to list the
challenges that approach faces.

I have been looking into how to test some cases where integration tests are
very difficult or even impossible to create for. Those are often found in
networking related and system specific code code (network.c, streams, FPM
and more). I was recently fixing one such bug and decided to give a try
which resulted in this PR: https://github.com/php/php-src/pull/16987 .

It makes sense to me to have some unit tests for code which is otherwise
hard or even impossible to test via integration tests.

There was a suggestion of RFC but that might be a bit too much as it's just
an internal change / addition. But certainly some overview on internals
should be done so writing this instead.

I'm fine with not going through the RFC process, although the policy[1]
police might come after us. :)

I decided to use cmocka in that PR because I had some experience with that.
It's quite small and still very powerful and allow vast mocking options.
It's a bit manual but it gives a bigger control over the mock. It relies on
--wrap linking option that replaces original functions with wraps. This is
however available only on Linux (or maybe some other Unix variants) but
doesn't work on MacOS or Windows. The developers that want to use it on
those platforms would need to use some Linux Virtualisation option (e.g.
Docker).

Especially on Windows, where we have different code paths, and sometimes
even completely different code, it would be great to also have these
unit tests. Given that link.exe supports /alternatename, a bit of
additional macro magic might do the trick[2]. I'll try to have a stab
at this soon.

I did also some research into the other mocking libraries in C.

Thank you! I barely have any experience with unit testing in C (only
one time, long ago, wrote part of unit test for timelib which is using
Cpputest), but from a cursory glance cmocka appears to be suitable for
our purposes.

In terms of CI. It has got its own build which is very simple and it tests
just specific parts so we could just limit it to run only for changed files
which might be quite convenient.

If the tests won't take long, in my opinion we could run them for every
push. Or at least in nightly (plus when relevant files changed), to
avoid that the suite doesn't run for a long time (possibly weeks), and
then suddenly is broken for external reasons.

[1] https://github.com/php/policies/blob/main/third-party-code.rst
[2]
https://stackoverflow.com/questions/33790425/visual-studio-c-linker-wrap-option

Christoph