Newsgroups: php.internals
Path: news.php.net
Xref: news.php.net php.internals:125966
Message-ID: <00831f49-ae36-4c4e-a60e-94a28333bc98@gmail.com>
Date: Fri, 15 Nov 2024 19:34:34 +0100
Precedence: bulk
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: [PHP-DEV] A new fuzz testing tool for PHP
To: internals@lists.php.net
References: <79C53085-9AD8-4E6D-ADAA-38AC1660A57E@gmail.com>
Content-Language: en-US
In-Reply-To: <79C53085-9AD8-4E6D-ADAA-38AC1660A57E@gmail.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
From: dossche.niels@gmail.com (Niels Dossche)

On 15/11/2024 14:20, Yuancheng Jiang wrote:
> Hi all,
> 
> 
> I have been submitting hundreds of bugs (see https://github.com/php/php-src/issues/created_by/YuanchengJiang <https://github.com/php/php-src/issues/created_by/YuanchengJiang>) during the past months and I first thank all the developers who take time to fix these issues to make PHP better.
> 
> 
> I am thrilled to introduce one fully automated fuzz testing tool, FlowFusion, for discovering various bugs of the PHP interpreter.
> 
> 
> The core idea behind FlowFusion is to leverage dataflow as an effective representation of test cases (.phpt files) maintained by PHP developers, merging two (or more) test cases to produce fused test cases with more complex code semantics. We connect two (or more) test cases via interleaving their dataflows, i.e., bringing the code context from one test case to another. This enables interactions among existing test cases, which are mostly the unit tests verifying one single functionality, making fused test cases interesting with merging code semantics.
> 
> 
> FlowFusion additionally fuzzes all defined functions and class methods using the code contexts of fused test cases. Available functions, classes, and methods are pre-collected and stored in sqlite3 with necessary information like the number of parameters. FlowFusion will be automatically upgrading if phpt files keep updating. Any new single test can bring thousands of new fused tests.
> 
> 
> The search space of FlowFusion is huge, which means it can cover various corner cases. Reasons for the huge search space are three-fold: (i) two random combinations of around 20,000 test cases can generate 400,000,000 test cases, and we can combine even more; (ii) the interleaving has randomness, given two test cases, there could be multiple ways to connect them; and (iii) FlowFusion also mutates the test case, fuzzes the runtime environment/configuration like JIT.
> 
> 
> *I can open-source the tool under my personal repository. I wonder by any chance if I can contribute it as the official PHP tool under https://github.com/php <https://github.com/php>, and I would be happy to maintain it for a long time.*
> 
> *
> *
> 
> Best,
> 
> Yuancheng
> 

Hi Yuancheng

Thanks for all the reports you made, certainly an impressive feat!
I don't know what other maintainers think, but FWIW I'd be in favor incorporating this into our toolchain.

Kind regards
Niels