A new fuzz testing tool for PHP

[YuanchengJiang]

Description

Hi all,

I have submitted hundreds of bugs during the past months and I first thank all the developers who take time to fix these issues to make PHP better.

I am thrilled to introduce one fully automated fuzz testing tool, FlowFusion, for discovering various bugs of the PHP interpreter.

The core idea behind FlowFusion is to leverage dataflow as an effective representation of test cases (.phpt files) maintained by PHP developers, merging two (or more) test cases to produce fused test cases with more complex code semantics. We connect two (or more) test cases via interleaving their dataflows, i.e., bringing the code context from one test case to another. This enables interactions among existing test cases, which are mostly the unit tests verifying one single functionality, making fused test cases interesting with merging code semantics.

FlowFusion additionally fuzzes all defined functions and class methods using the code contexts of fused test cases. Available functions, classes, and methods are pre-collected and stored in sqlite3 with necessary information like the number of parameters. FlowFusion will be automatically upgrading if phpt files keep updating. Any new single test can bring thousands of new fused tests.

The search space of FlowFusion is huge, which means it can cover various corner cases. Reasons for the huge search space are three-fold: (i) two random combinations of around 20,000 test cases can generate 400,000,000 test cases, and we can combine even more; (ii) the interleaving has randomness, given two test cases, there could be multiple ways to connect them; and (iii) FlowFusion also mutates the test case, fuzzes the runtime environment/configuration like JIT.

I can open-source the tool under my personal repository. I wonder by any chance if I can contribute it as the official PHP tool under https://github.com/php, and I would be happy to maintain it for a long time.

[devnexen]

And thanks for all your reports. I invite you to discuss this matter in the internal mail list, in addition you will garner more attention.

[TimWolla]

For reference: https://externals.io/message/125963

[pospjan]

@YuanchengJiang I just read your paper https://arxiv.org/pdf/2410.21713 and it's super interesting!

[YuanchengJiang]

@pospjan thanks! the paper gives more details though it still needs many updates

[danog]

Hi @YuanchengJiang,

Awesome! I noticed the impressive number of opened issues, and given your background I guessed you were working on some new fuzzer.

I’ve been thinking that the fuzzing corpus of this new fuzzer (and of the existing oss-fuzz fuzzer SAPI, currently being ran by Google) may be improved by also adding all of the code of the community libraries (i.e. those defined by the nightly GitHub workflow at https://github.com/php/php-src/blob/master/.github/workflows/nightly.yml#L485).

I understand this might cause issues due to the volume of additional code being permutated by the fuzzer, but even running the community tests themselves without fuzzing already uncovers multiple segfaults (some of which are still being failing on master as of today).

Some time ago, I submitted #12406, which does multiple things to improve the coverage of the JIT compiler:

• Add a few more popular CPU-intensive community libs like phpseclib, psalm, phpstan, etc: all prime candidates for addition to the nightly tests, as they're all extremely CPU-intensive libraries which benefit a lot from JIT, but suffer from its bugs (i.e. the psalm/phpseclib phpunit testsuites currently fail with segfaults with tracing JIT, phpseclib even explicitly disables JIT on windows to avoid a yet unsolved bug, etc...).

• Parallelise community tests using a custom new runner, addressing concerns from Ilya which was worried about CI run times caused by the addition of new community tests

• Add --repeat 2 to all tests (including community tests), which manages to catch some nasty JIT bugs by re-invoking the same script twice without actually recompiling the PHP code and zend byte code twice, managing to find issues caused by side effects of the first compilation.

• Improve JIT flags of community and phpt tests to detect more JIT bugs, mainly copying them from https://github.com/danog/jit_bugs/blob/master/php.ini

I don’t currently have the free time to clean up the pull request and fix the numerous JIT bugs it currently detects (at least without a support contract), but all of these approaches may be reused if you or anyone else decides to upstream FlowFusion (though I would love at least a @danog mention in the pull request :)

[YuanchengJiang]

Hi @danog,

Good point! Thanks, Involving the peripheral libraries makes the corpus more diverse.

I do have some similar insights. I patched the run-tests.php. I also noticed that repeated executions can yield different results. I will surely refer to your experience when updating FlowFusion later ;)