Newsgroups: php.internals Path: news.php.net Xref: news.php.net php.internals:126129 X-Original-To: internals@lists.php.net Delivered-To: internals@lists.php.net Received: from php-smtp4.php.net (php-smtp4.php.net [45.112.84.5]) by qa.php.net (Postfix) with ESMTPS id B38CB1A00BD for ; Mon, 16 Dec 2024 15:33:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=php.net; s=mail; t=1734363012; bh=5gRzQm+8pmdambiSQauxa7vKXmkYWCiYPSLvU/VCzmE=; h=References:In-Reply-To:From:Date:Subject:To:Cc:From; b=aiuhR5EmaL8+gyVk6kdoD4FzTRp2U48OhvCQ87+BM11RL9FirHsY2IZlqqX/KD+il ZaAkZSoAEYjzVf9ruPMFnRM841JiT+EL/PfWm6JXPwAcD/3vU32H2stDuIg5AQPM4u UHrkwd9Lc8KKv7GkLHlyTuyGCpruMyasiyao7Cp0KeKrdNqGZa6FymlOPuZvma8SH9 GCVLptTbfUNxdyBpPm2kHqIRzt58xFqqRGXLi73baFK6pew8aucgtzR8wnMzbsJ+eX QkyANMeJKG8uZHFLMsbNgKH8soQRNnCusgFzkunCtG09FJ3bmNbx1nyJsEmZCPAyOo i8VU4Ef7dwDsw== Received: from php-smtp4.php.net (localhost [127.0.0.1]) by php-smtp4.php.net (Postfix) with ESMTP id 988B0180042 for ; Mon, 16 Dec 2024 15:30:11 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-13) on php-smtp4.php.net X-Spam-Level: * X-Spam-Status: No, score=2.0 required=5.0 tests=BAYES_50,DMARC_NONE, FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS, HTML_MESSAGE,RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=4.0.0 X-Spam-Virus: No X-Envelope-From: Received: from mail-oo1-f49.google.com (mail-oo1-f49.google.com [209.85.161.49]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by php-smtp4.php.net (Postfix) with ESMTPS for ; Mon, 16 Dec 2024 15:30:11 +0000 (UTC) Received: by mail-oo1-f49.google.com with SMTP id 006d021491bc7-5f304ac59b9so2098984eaf.0 for ; Mon, 16 Dec 2024 07:33:14 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734363193; x=1734967993; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=5gRzQm+8pmdambiSQauxa7vKXmkYWCiYPSLvU/VCzmE=; b=Y+ku4aprlORpFB1BuZb7XUbMwD5qagjg3W5j5Gzi90Y2BF0M6Pgav2P3p7QVDClCin g3XCcD6rZAUfeEUznsoL65RpD7rGiLRWsUGSePhTDLUnNvyBnoHX+KAO4P9D2seTEf/7 Dca5vhpVXyelX6EIcB0FKdVgbdcb6yKmYpXpmY7sa5k93IsvOpgnPyiZP68h127l++Ap JLIrgEqwTce5uvSYNMv6Yl7QrxQkRMxIKuTgxFTClK/+B0YyynpEZaE3TqYgu2QuSeOQ CgYD2T2LHAVzreJnbeKrf4E4sdPPprU/ApD242vvxPhjrg5jgOBBvfrFbh3dyuoy0Pea kOxQ== X-Gm-Message-State: AOJu0YyK4SZqpEpkZ16o24susswGS6inN/jvReNUqTLQaYdhG5hdXzjo AEtmFf0gJr9O0f4Y+hItROKSTYaTZTULdy5ivhHjKAuL17WLvkV2sjHN5IOzd+hYOHpoeTX47kl tbKuvpPPiamK6uKC9SChlv6BzQ6t0lpDl X-Gm-Gg: ASbGncvLZl4mnGH7EMEVAmD0wmMxhyXk42JDoyXFKcT57TdtjoIAxLA/Xe2+hPdI587 Lj8K5sACrWGF4SOtPUVMSBtQA2gRUnjHW0Klbhg== X-Google-Smtp-Source: AGHT+IFyVjFp32OeoUcIlYGIRmRxaoEjyRJ2luMd4/MolvOTHUVWFM+APUWQyD789emKyAketp2Vil70BhZf8ilxJ+U= X-Received: by 2002:a05:6870:224b:b0:29e:9dc:8ba with SMTP id 586e51a60fabf-2a3ac9973aemr7590731fac.27.1734363193213; Mon, 16 Dec 2024 07:33:13 -0800 (PST) Precedence: bulk list-help: list-post: List-Id: internals.lists.php.net x-ms-reactions: disallow MIME-Version: 1.0 References: In-Reply-To: Date: Mon, 16 Dec 2024 16:33:02 +0100 Message-ID: Subject: Re: [PHP-DEV] C Unit testing and mocking To: Calvin Buckley Cc: PHP internals list Content-Type: multipart/alternative; boundary="0000000000000b0c03062964e68e" From: bukka@php.net (Jakub Zelenka) --0000000000000b0c03062964e68e Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Mon, Dec 16, 2024 at 4:04=E2=80=AFPM Calvin Buckley = wrote: > On Dec 16, 2024, at 9:18=E2=80=AFAM, Jakub Zelenka wrote: > > > > Hi, > > > > I have been looking into how to test some cases where integration tests > are very difficult or even impossible to create for. Those are often foun= d > in networking related and system specific code code (network.c, streams, > FPM and more). I was recently fixing one such bug and decided to give a t= ry > which resulted in this PR: > https://github.com/php/php-src/pexpectionsull/16987 > . > > > > There was a suggestion of RFC but that might be a bit too much as it's > just an internal change / addition. But certainly some overview on > internals should be done so writing this instead. > > > > I decided to use cmocka in that PR because I had some experience with > that. It's quite small and still very powerful and allow vast mocking > options. It's a bit manual but it gives a bigger control over the mock. I= t > relies on --wrap linking option that replaces original functions with > wraps. This is however available only on Linux (or maybe some other Unix > variants) but doesn't work on MacOS or Windows. The developers that want = to > use it on those platforms would need to use some Linux Virtualisation > option (e.g. Docker). It also requires static library which is supported = by > embed SAPI that can be compiled statically. That limits number of > extensions to use but the main use cases don't really have deps so it > should be fine. > > > > I did also some research into the other mocking libraries in C. There i= s > a Unity with CMock, FFF and some C++ libs like GUnit, Criterion and > Trompeloeil that I looked into. I quickly discarded GUnit and Trompeloeil > as they relay on C++ virtual methods and require wrapping C code to C++ > which is very inconvenient. FFF seems too simple and maybe quite inflexib= le > for our needs as well. Criterion also optionally uses wrap so I didn't se= e > much advantages compare to cmocka. So it left Unity with CMock that allow= s > generating custom mocks using a Ruby script. That seemed initially quite > nice but after spending around two hours with trying to make it works for > PHP codebase, I just gave up. It gets quite messy for complex scenarios a= nd > I just didn't figure out how to nicely mock libc functions without any > modification to php-src. > > > > In terms of CI. It has got its own build which is very simple and it > tests just specific parts so we could just limit it to run only for chang= ed > files which might be quite convenient. > > > > So the proposed PR is probably the only reasonable unit testing that I > can come up with. I think it should be completely optional initially for > people to use - more like an experiment. If it becomes used, then good of > course. And if it becomes pain, we can just get rid of it. Has anyone got > any objections to get this merged? If not I plan to merge it early in > January. > > > > Cheers > > > > Jakub > > > > I'm assuming that uses ELF symbol interposition or something like that, > which is why it seems Linux/BSD specific. That seems fragile to me. > > It is replaced when linking the program by ld (see https://man7.org/linux/man-pages/man1/ld.1.html - the --wrap option for more info). The flag should be stable on linux and it is not really fragile as few mocking libraries are built around it. > I think currently for wanting to test C functions, we're adding custom > functions into ext/zend_test and writing PHPT. Would this work? We > already have that, after all. If not, it'd be helpful to list the > challenges that approach faces. ext/zend_test is just for normal ext testing that does not require any mocking. What I need is to test certain flow where mocked function returns specific response / set output params in certain way. That requires mocking that function and setting exceptions. This is not possible in zend_test. The only exception there is https://github.com/php/php-src/blob/284c4e3e318a75b8133b52c153bc912b48d1bba= b/ext/zend_test/test.c#L1499-L1514 which mocks copy_file_range but it uses dlsym with RTLD_NEXT to replace the function in runtime. I was actually checking if any mocking lib uses dlsym but it is not as it's probably a bit more fragile than wrap. And without the mocking lib, it's not really useful because we would have to re-implement what they do to support expectations from different tests (some global state manager) and other features. So I didn't really go there... Regards Jakub --0000000000000b0c03062964e68e Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On Mon, Dec 16, 2024 at 4:04=E2=80=AFPM Calvin Buckle= y <calvin@cmpct.info> wrote:=
On Dec 16, 2024, at 9:18=E2=80=AFAM, Jakub Zelenka <bukka@php.net> wrote:
>
> Hi,
>
> I have been looking into how to test some cases where integration test= s are very difficult or even impossible to create for. Those are often foun= d in networking related and system specific code code (network.c, streams, = FPM and more). I was recently fixing one such bug and decided to give a try= which resulted in this PR: https://github.com/php/php-src/= pexpectionsull/16987 .
>
> There was a suggestion of RFC but that might be a bit too much as it&#= 39;s just an internal change / addition. But certainly some overview on int= ernals should be done so writing this instead.
>
> I decided to use cmocka in that PR because I had some experience with = that. It's quite small and still very powerful and allow vast mocking o= ptions. It's a bit manual but it gives a bigger control over the mock. = It relies on --wrap linking option that replaces original functions with wr= aps. This is however available only on Linux (or maybe some other Unix vari= ants) but doesn't work on MacOS or Windows. The developers that want to= use it on those platforms would need to use some Linux Virtualisation opti= on (e.g. Docker). It also requires static library which is supported by emb= ed SAPI that can be compiled statically. That limits number of extensions t= o use but the main use cases don't really have deps so it should be fin= e.
>
> I did also some research into the other mocking libraries in C. There = is a Unity with CMock, FFF and some C++ libs like GUnit, Criterion and Trom= peloeil that I looked into. I quickly discarded GUnit and Trompeloeil as th= ey relay on C++ virtual methods and require wrapping C code to C++ which is= very inconvenient. FFF seems too simple and maybe quite inflexible for our= needs as well. Criterion also optionally uses wrap so I didn't see muc= h advantages compare to cmocka. So it left Unity with CMock that allows gen= erating custom mocks using a Ruby script. That seemed initially quite nice = but after spending around two hours with trying to make it works for PHP co= debase, I just gave up. It gets quite messy for complex scenarios and I jus= t didn't figure out how to nicely mock libc functions without any modif= ication to php-src.
>
> In terms of CI. It has got its own build which is very simple and it t= ests just specific parts so we could just limit it to run only for changed = files which might be quite convenient.
>
> So the proposed PR is probably the only reasonable unit testing that I= can come up with. I think it should be completely optional initially for p= eople to use - more like an experiment. If it becomes used, then good of co= urse. And if it becomes pain, we can just get rid of it. Has anyone got any= objections to get this merged? If not I plan to merge it early in January.=
>
> Cheers
>
> Jakub
>

I'm assuming that uses ELF symbol interposition or something like that,=
which is why it seems Linux/BSD specific. That seems fragile to me.


It is replaced when linking the progra= m by ld (see=C2=A0https://man7.org/linux/man-pages/man1/ld.1.html - the --wrap option = for more info). The flag should be stable on linux and it is not really fra= gile as few mocking libraries are built around it.
=C2=A0
I think currently for wanting to test C functions, we're adding custom<= br> functions into ext/zend_test and writing PHPT. Would this work? We
already have that, after all. If not, it'd be helpful to list the
challenges that approach faces.

ext/zend_te= st is just for normal ext testing that does not require any mocking. What I= need is to test certain flow where mocked function returns specific respon= se / set output params in certain way. That requires mocking that function = and setting exceptions. This is not possible in zend_test. The only excepti= on there is https://github.c= om/php/php-src/blob/284c4e3e318a75b8133b52c153bc912b48d1bbab/ext/zend_test/= test.c#L1499-L1514 which mocks=C2=A0copy_file_range but it uses=C2=A0dl= sym with RTLD_NEXT to replace the function in runtime. I was actually check= ing if any mocking lib uses dlsym but it is not as it's probably a bit = more fragile than wrap. And without the mocking lib, it's not really us= eful because we would have to re-implement what they do to support expectat= ions from different tests (some global state manager) and other features. S= o I didn't really go there...

Regards

Jakub
--0000000000000b0c03062964e68e--