Tools: Add benchmark warnings for PRs and push graphs for commits into master

[algochoi]

Summary

This PR adds a github action to track performance regressions for the BenchmarkUintMath benchmark.

Workflows

The PR introduces two github workflows:

• One to run the benchmark on a PR branch against the master branch and comment on the PR if it is over a particular threshold (default: 200%). An example comment for the performance alert can be seen [here].(Check workflow against commits on master algochoi/go-algorand#10 (comment)).
• One to run the benchmark between commits on the master branch, push and persist performance metrics to the gh-pages branch ,and visualize any improvements or regressions over time (will be available at: https://algorand.github.io/go-algorand/dev/bench/).

Variance

There is some variance between runs because github actions might spin up a different machine each time (e.g. Intel Xeon8370C vs 8171M). Empirically, the variance seems to be 10~30% for the most part. We tried putting some baseline benchmarks (e.g. sleep(1s)), but the variance in sleeping between machines is extremely low to provide any useful metric.

Time series visualization

The github-actions-benchmark tool we used provides a graph to visualize performance benchmark. It requires permissions to push to the gh-pages branch and can show trends over commits in the master branch. An example in a local repo can be seen here.

Future work

• Currently, this benchmark only runs for BenchmarkUintMath, but it could be useful to track other benchmarks in the future to see any regressions/improvements over time.
• A way to reduce or track variance between CI runs. Currently, there seems to be limited support for this in the github-action-benchmark tool we used.

Test Plan

Tested on a local fork and branch. Example PR for the performance alert can be seen here.

Closes an internal ticket

[codecov]

Codecov Report

Merging #3998 (bb26b6a) into master (91095fa) will increase coverage by 4.67%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #3998      +/-   ##
==========================================
+ Coverage   49.77%   54.45%   +4.67%     
==========================================
  Files         409      390      -19     
  Lines       69157    48500   -20657     
==========================================
- Hits        34426    26411    -8015     
+ Misses      31011    19868   -11143     
+ Partials     3720     2221    -1499     

Impacted Files	Coverage Δ
catchup/service.go	68.14% <0.00%> (-1.49%)	⬇️
ledger/acctupdates.go	69.43% <0.00%> (-0.14%)	⬇️
network/wsNetwork.go	62.79% <0.00%> (ø)
data/committee/msgp_gen.go
node/msgp_gen.go
protocol/msgp_gen.go
data/transactions/msgp_gen.go
crypto/merklesignature/msgp_gen.go
ledger/ledgercore/msgp_gen.go
data/account/msgp_gen.go
... and 16 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 91095fa...bb26b6a. Read the comment docs.

[algochoi]

@michaeldiamant thanks for the quick review and explanations for your reasoning - I agreed with all the comments and addressed them in the latest PR

[michaeldiamant]

@algochoi Thanks for the effort here - I'm excited to collect data and gain experience with the tooling! ☕