[Fuzzing] Create script to run all fuzz targets

[fleupold]

We currently have two fuzzing targets:

• element_read_all
• orderbook

Our goal is to run fuzzing continuously after each push. This PR adds a shell script that fetches the list of fuzz targets via cargo fuzz list and runs each target sequentially for a certain amount of time.

It loops over all targets indefinitely until it finds a crash.

Test Plan

cd pricegraph/fuzz
./fuzz_all_targets.sh 20

See targets getting fuzzed for 20s each indefinitely. Now, add a panic in one of the targets (e.g. orderbook.rs) and save. As soon as the target is compiled and run next everything should stop.

[e00E]

I don't want to delete the existing targets. cargo fuzz is made with the intention of having multiple targets. I don't want to limit this because a level higher up (running the fuzzer) cannot handle this well. We should improve that layer instead.

I would rather have a more complicated kubernetes script and have to update it. We can get rid of manually updating by parsing the output of cargo +nightly fuzz list.

[nlordell]

Cool!

[nlordell]

I don't want to delete the existing targets. cargo fuzz is made with the intention of having multiple targets.

I agree, we can keep our old targets, but what is the issue of adding a third target to have a single entry point for kubernetes?

[e00E]

It's not unreasonable but it has the drawback of duplicating the code into that target. When we make changes to the original targets we have to update this target too. Or we could refactor the targets into functions in a module so the code does not have to be duplicated.

On the other hand writing a script that parses the list of fuzzing targets and then loops through them forever running each target for 5 minutes is the better solution imo.

[fleupold]

I agree that having to add a target in two places is a bit error prone and annoying. We can use cargo fuzz list to get the targets. I'd probably write a rust binary similar to https://github.com/rust-fuzz/targets/blob/master/cli.rs that queries the targets and then spawns a command to run them (will try in parallel, but sequentially would also work).

I'll add this script into the services repo as it's inconvenient to have non-trivial code in the gnosis-staging repo which should really only contain configuration.

[e00E]

I like a rust binary. If you prefer a shell script you can try something like

cargo +nightly fuzz list | xargs -L1 -I{} cargo +nightly fuzz run {} -- -max_total_time=300

That runs each target for 300 seconds, once. Not sure how it behaves if the fuzzer finds a panic.

will try in parallel, but sequentially would also work

I don't think we need parallel, if that makes the script more complicated but if not then there is no drawback.

[fleupold]

If you prefer a shell script you can try something like

I looked into that but I would have to do something like this to make xargs fail if one of the inner targets fail and wrap this in some endless loop running until the first failure. It's possible in bash but maybe a bit too magical, and it might be good to have a simple target that can run all fuzz targets locally as well.

[fleupold]

It turns out that writing a rust binary to do this is actually harder than I thought (a binary that lives in the fuzz subfolder will automatically be considered a fuzzing target itself + I wasn't able to kill the processes after the timeout using Child.kill and thus the script become quite large and hacky)

I therefore ended up writing it in bash at the end 🙈

Much time wasted...

[bh2smith]

I find this shell script nice and easy to read!