Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve Flux Fuzz tests reliability #3346

Closed
27 of 30 tasks
pjbgf opened this issue Nov 23, 2022 · 4 comments
Closed
27 of 30 tasks

Improve Flux Fuzz tests reliability #3346

pjbgf opened this issue Nov 23, 2022 · 4 comments
Assignees
Labels
area/ci CI related issues and pull requests

Comments

@pjbgf
Copy link
Member

pjbgf commented Nov 23, 2022

Recently a change upstream have broken all builds across fluxcd.
This is broken both at upstream build time (for our project) as well as locally within our projects. Here's the output error:

+ cd /tmp/oss_fuzz-9iS27e/go-118-fuzz-build
+ go build -o /root/go/bin/go-118-fuzz-build
+ cd addimport
/root/go/src/github.com/fluxcd/image-reflector-controller/tests/fuzz/oss_fuzz_build.sh: line 37: cd: addimport: No such file or directory
+ cleanup
+ rm -rf /tmp/oss_fuzz-9iS27e
ERROR:root:Building fuzzers failed.

This is due to changes on https://github.com/AdamKorcz/go-118-fuzz-build/ which were not backwards compatible. This package is the basis of all Go support on oss-fuzz, and therefore any changes to it will affect us.

For improved reliability we shall remove any unsupported features upstream and trying to rely solely on their setup to run smoke tests.
We will still suffer from issues when upstream changes are introduced, but this should decrease their impact.

We may also have to change the way we approach fuzz testing. Once of the key changes being to avoid sharing code amongst fuzz and non-fuzz tests. Although this is supported by Go fuzz, this isn't working well with oss-fuzz, and attempts to improve upstream support are currently stuck.

Outstanding Tasks

@AdamKorcz
Copy link

AdamKorcz commented Nov 24, 2022

Sorry for missing this. I checked all Go projects in OSS-Fuzz for breakages but must have missed Flux since the build was broken since September: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=51202&q=proj%3Dfluxcd, or is the OSS-Fuzz build status reporting incorrect results?

If there is anything I can help with, please let me know.

@pjbgf
Copy link
Member Author

pjbgf commented Nov 25, 2022

@AdamKorcz Don't worry about it. Due to things like libgit2, Flux had a lot of custom code to run its fuzzers and at times it got out of sync with upstream. For example, our CI checks are based on local Dockerfiles. With the changes on this issue we should make our use of oss-fuzz more streamlined and completely aligned with oss-fuzz.

I have been working through some of the upstream build failures, but most recently (and what I referred to in the issue was) our internal CI builds also broke. Once we are done aligning the fuzz implementation all of them should be fixed.

@stefanprodan
Copy link
Member

@pjbgf can this be closed?

@pjbgf
Copy link
Member Author

pjbgf commented Jul 3, 2023

Yes, closing it now.

@pjbgf pjbgf closed this as completed Jul 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/ci CI related issues and pull requests
Projects
Status: Done
Development

No branches or pull requests

3 participants