Future-proof Benchmarking ⏱

[p-shahi]

eta: 2023Q2

• perf protocol
  • spec Introduce perf spec specs#478
  • feat(perf): implement libp2p perf protocol rust-libp2p#3508 (rust-libp2p)
  • go-libp2p feat(perf): add (automation, provision, build, run) tooling #184
  • feat(@libp2p/protocol-perf): Implement perf protocol js-libp2p#1604 (js-libp2p)
  • perf: throughput test (TCP, QUIC, libp2p, but not iperf) never exits slow start #261
  • Add vanilla Go http in order to measure overhead of TLS #204
• Create performance tooling to run perf protocol implementation on cloud instances
  • First iteration feat(perf): add (automation, provision, build, run) tooling #184
  • Only use a single seconds measurement. Don't split in connection establishment, upload and download. These are difficult to differentiate. refactor(perf): expose single latency measurement  #207
  • Add js-libp2p feat(@libp2p/protocol-perf): Implement perf protocol js-libp2p#1604 feat: add js-libp2p perf tests #244
  • Add back zig-libp2p
  • Run iperf on both UDP and TCP
  • Requests per second (for some request/response protocol)
  • Handshakes per second (useful to identify overhead in connection initialization).
  • Support testing other security protocols, i.e. Noise
  • Resource utilization benchmark?
  • Don't blindly trust server and clients ssh key (mitm attack)
• Automation see Automate performance benchmarking #183
• Visualize data emitted by above performance tooling Create a performance dashboard #156
  • ideally mirror advanced filtering in the URL for better sharing
• Create test to show libp2p's performance compared to http #27
• Benchmark js-libp2p transfer performance #65
• blog about performance benchmarking in libp2p blog#74
• Diverse benchmarks e.g. multiple streams or multiple connections
• Write documentation

[BigLep]

@mxinden : I know you're doing blog drafting at https://observablehq.com/@mxinden-workspace/libp2p-perf - thanks!

I did leave some comments inline.

A couple of general things that came to mind while looking at it:

1. Is this just where we're drafting, or something we expect to point users to?
2. If pointing users to it, can we create a libp2p Observable namespace?
3. What is going to be the presence on the libp2p blog about this?

(no pressure to respond now - I don't mean to be catching you in-progress. If this has been though through and you just haven't gotten to any of this yet, that is fine. I am just leaving notes now while it's coming to mind.)

[mxinden]

Status Update

#184 and thus a first iteration of the libp2p performance benchmark setup is merged. Follow-up / future work in tracked in the description of this issue.

[mxinden]

Crossposting a Slack message here, for those following through GitHub only.

First iteration of the libp2p performance benchmarking setup merged

More specifically #184.

Let's try it out. We recently released a new version of rust-libp2p, v0.52.0. Now we can add it to the benchmarking setup, see how well it performs.

1. Create a pull request against libp2p/test-plans adding the new version fix(perf/rust-libp2p): update to latest v0.52.0 release #202. (Note that in this particular case we are updating an existing version, as we have been already testing a pre-release of v0.52.0.)
2. Trigger the libp2p perf test GitHub workflow for the branch of the pull request https://github.com/libp2p/test-plans/actions/workflows/perf.yml
3. Follow the workflow run, which eventually pushes the benchmark results to the branch https://github.com/libp2p/test-plans/actions/runs/5351995316/jobs/9706612810
4. See the new commit on your pull request fix(perf/rust-libp2p): update to latest v0.52.0 release #202 b94960f
5. Visualize the benchmark results via our ObservableHQ dashboard. Just make sure to choose the right branch, in this case perf-rust-libp2p-v0.52 in the Branch input https://observablehq.com/@mxinden-workspace/libp2p-performance-dashboard

Note that this is the first iteration only. There are lots of follow-ups, especially around the dashboard. Future work tracked on #63.

Thanks to Marco, Piotr and Marten for all the help on the project!

https://filecoinproject.slack.com/archives/C03K82MU486/p1687492998525109

[mxinden]

Status Update

ideally mirror advanced filtering in the URL for better sharing

You can now embed the repository and branch of the benchmark data in the dashboard URL. For example the dashboard link below displays the benchmarking data of #241.

https://observablehq.com/@libp2p-workspace/performance-dashboard?branch=perf-iperf-tcp