go-fuzz-build: parallelize build #172
Conversation
Add crashers in github.com/arolek/ase
Update README
Add confl trophy
Add bencode trophy
Added stalecucumber trophy
Update README.md
- remove suppresstions for fixed bugs - add suppressions for new bugs - switch to stantard package go/types - add new inputs
Update README.md
Added github.com/eaburns/flac to trophies
trophies: add github.com/digitalocean/captainslog
Update to the current location of the freetype library
Currently we have hardcoded list of go-fuzz-dep dependencies. This list constantly changes due to changes in std lib (due various new internal packages). Extract go-fuzz-dep dependencies automatically with 'go list'. Fixes dvyukov#157
Update list of trophies in README file
Add two trophies to README file
Add a new external article
fixes build error in png example, g1.9
Add github.com/spenczar/tdigest trophies
Update README.md
…er library. Specifically: * Fixed the issues related to API changes. * Changed to using the ParseNext() API (which can parse multiple statements). * Changed to use the io.Reader interface. * Added a class to artificially reduce the chunks that can be read, to increase the chance of buffer underrun/overrun errors.
Updated the sqlparser fuzzer to use the latest version of the library.
Parallelize some build steps to improve build speed. On my particular test on a macosx system, the build time goes from 2m05 to 1m17.
go-fuzz-build/main.go
Outdated
| wg.Add(1) | ||
| sem <- true | ||
| go func(p string) { | ||
| res := goListBool(p, "Standard") |
There was a problem hiding this comment.
If go list invocation is the problem, then we should extract all interesting properties and once. Currently we run go list 4 times per-package: to check Standart, extract Import, GoFiles CgoFiles.
go-fuzz-build/main.go
Outdated
| @@ -194,9 +213,20 @@ func buildInstrumentedBinary(pkg string, deps map[string]bool, lits map[Literal] | |||
| // Cross-compilation is not implemented. | |||
| copyDir(filepath.Join(GOROOT, "pkg", runtime.GOOS+"_"+runtime.GOARCH), filepath.Join(workdir, "goroot", "pkg", runtime.GOOS+"_"+runtime.GOARCH), true, nil) | |||
| } | |||
|
|
|||
| wg := sync.WaitGroup{} | |||
There was a problem hiding this comment.
var wg sync.WaitGroup please as more standard.
In one codebase I counted 2598 uses of var wg sync.WaitGroup vs 1 for wg := sync.WaitGroup{}.
go-fuzz-build/main.go
Outdated
| defer wg.Done() | ||
| defer func() { <-sem }() | ||
|
|
||
| impList := goListList(pkg, "Imports") |
There was a problem hiding this comment.
Here we only parallelize go list, right? So if we call go list once to extract all properties, this part will go away.
go-fuzz-build/main.go
Outdated
| @@ -362,7 +407,11 @@ func instrumentPackages(workdir string, deps map[string]bool, lits map[Literal]s | |||
| } | |||
| path := filepath.Join(workdir, root, "src", p.name) | |||
| var files []*ast.File | |||
| for _, fn := range append(goListList(p.name, "GoFiles"), goListList(p.name, "CgoFiles")...) { | |||
| xgoFiles := make(chan []string, 2) | |||
| go func() { xgoFiles <- goListList(p.name, "GoFiles") }() | |||
There was a problem hiding this comment.
This will also go away if we call go list once.
go-fuzz-build/main.go
Outdated
| @@ -71,10 +75,25 @@ func main() { | |||
| } | |||
|
|
|||
| deps := make(map[string]bool) | |||
| mu := sync.Mutex{} | |||
|
Thanks for your feedback. I'll try to find some free time to improve on this. |
|
@josselin-c Please note that go-fuzz repository history was rewritten to reduce total repository size and download time. The easiest migration path for this PR is to |
|
#211 substantively changed go-fuzz-build, making it faster in the process. I can imagine that you don't want to revisit this, but it might be interesting to look at the new go-fuzz-build, and see whether your optimizations still apply. |
|
I looked through the relevant commit, and found that all but one of the improvements have been incorporated into go-fuzz, by way of using go/packages. The outstanding one is parallelizing package cloning. In my experiments, that is not a significant build time factor. So I'm going to close this PR. Thanks again for taking the time to work on it. |
Parallelize some build steps to improve build speed.
On my particular test on a macosx system, the build time goes from 2m05
to 1m17.