update roadmap #56

p-shahi · 2022-10-18T23:48:21Z

This PR

updates the roadmap with multi-dimension test/interoperability efforts
pulls in testing and benchmarking from libp2p implementation roadmaps
organizes in style of the go and js documents

TODO:

if the projects defined are agreed upon, create Epics/issues for items
assign completion best guess

p-shahi · 2022-10-19T01:09:29Z

ROADMAP.md

- We fixed every known stability issue with the `libp2p/test-plans` test suite
-    - [Issue 36](https://github.com/libp2p/test-plans/issues/36)


I chose to remove this as it's ongoing maintenance work

p-shahi · 2022-10-19T01:11:20Z

ROADMAP.md

- `libp2p/interop` and `libp2p/test-plans` are working together
-    - They are either merged or somehow know about each other.


I couldn't tell the difference between this goal vs

We have ported the tests from libpp2/interop

So I chose to subsume it under "Be the home for all interop tests"

p-shahi · 2022-10-19T01:15:03Z

ROADMAP.md

+<!-- TODO: Assign a quarter -->
+<!-- TODO: Create issue -->
+**Why:** Having interoperability tests with lots of transports, encryption mechanisms, and stream muxers is great. However, we need to stay backwards-compatible with legacy libp2p releases, with other libp2p implementations, and less advanced libp2p stacks.


This is listed last here but likely merits being much higher up. Rough idea is to itemize our test backlog (across implementations), prioritize, and then commit to chipping away at it over the 2023 H1 and H2.

p-shahi · 2022-10-19T01:17:18Z

ROADMAP.md

+**Why**: For libp2p to be competitive, it needs to delivers comparable performance to widely used protocols on the internet, namely HTTP/2 and HTTP/3.
+
+**Goal**: We have a test suite that runs libp2p transfers between nodes located at different locations all over the world, proving that libp2p is able to achieve performance on par with HTTP. The test suite is run on a continuous basis and results are published to a public performance dashboard.


Deliverables for both milestones are in Testground, but idea is that it lives here so we can use test-plans and multi-dimension test framework to measure transfer performance for all implementations.

ROADMAP.md

p-shahi · 2022-10-19T01:20:32Z

DESIGN.md

@@ -58,6 +61,11 @@ This document describes our process for testing interoperability & backward comp

 # How do we test libp2p interop at ProtocolLabs?

+---
+


I think this is basically the multi-dimension tests (semantics and process)

BigLep

Feel free to merge from my perspective if comments are incorporated.

ROADMAP.md

BigLep · 2022-10-19T03:15:46Z

ROADMAP.md

+#### 3. Rock solid libp2p releases
+<!-- TODO: Create issue -->
+The full `libp2p/test-plans` test suite is run before releasing a version of go-libp2p,  rust-libp2p, and js-libp2p.


I think we should run the interoperability testplan today (or at least soon) before releases. It's fine if it's manual.

Can we pull this portion in sooner?

I think run the interoperability testplan should occur today (or at least soon) before releases.

I'm having trouble parsing this sentence.

It's fine if it's manual.

The problem is that we have many parts of our stack that are currently untested (other than unit tests): hole punching, autonat, autorelay etc. We manually test these parts when we make changes to the respective code paths, but we don't rerun these manual tests on every release. It's just too much stuff to test.
This is not ideal, and we should have automated interoperability tests for all these components. Requiring manual testing though would add hours of testing effort to our release process.

I think we all agree that these tests are needed like yesterday. But we don’t have this yet, so what’s the fastest way we can get to the world where we have this? I’m afraid manual smoke testing will cut into the effort to build this.

I’m also afraid that adding more friction to the release process will cause patches to be unnecessarily delayed.

If we weren’t prioritizing these projects, I would agree that we should at least have a manual test checklist. But since we are, it seems to me we can live with the status quo for a bit longer if it gets us this faster.

@marten-seemann @MarcoPolo : My bad on incoherent sentence. Fixed now.

Correct me if this is wrong, but my understanding is that we have some degree of interoperability testing run as part of CI but there is also a "big test" that doesn't run as part of CI because it takes too long.

What I'm suggesting we do today in our release process is run the big test manually as one of the release checklist items.

I'm not suggesting we do any manual smoke testing features that don't have a test-plan (unless of course we made changes in one of those areas). I'm suggesting we do make sure we execute the test-plan we have written that is flexing interoperability.

In future, I expect the "big interoperability test" either gets run as part of CI (because we have a cluster than can execute it quickly enough) or it runs as a nightly cron job and the release process is one of verifying that last night's run passed.

Does that make sense?

Removed "3. Rock solid libp2p releases" and combined with "2. Interop test plans for all existing/developing libp2p transports"

3's intention was to run the transport interop tests as CI nightly job, serving as regression tests for new release candidates.

BigLep · 2022-10-19T03:17:47Z

ROADMAP.md

+A GitHub workflow is added so that these suites run for any PR against master and run against the `master` branch on a nightly basis (updating the status check.)
+
+#### 4. Canonical interop tests & dashboard


I would vote to do this sooner - maybe as item 2. It gives a very clear signal to the community of what we're aiming for and how far we have to go. It also becomes a clear way for us to show progress each month. I'd love at monthly all hands for us to show the month over month change of this.

Isn’t point 2 a prereq for this issue?

@MarcoPolo : I don't think it's a prerequ. I'm viewing this as analogous to other behaviours in software develop;meant like putting in the metric before you make the performance improvement or writing the test before the implementation. The dashboard makes clear what we're aiming for. It's our baseline. To start with it will be red/empty for all/most the combinations. Then as we do A.2 "Interop test plans for all existing/developing features" we'll start showing the matrix progressively filling more-and-more each week.

I think this came up in a verbal conversation with @mxinden as well (which is making me think it would be good to use sync time in Lisbon to be on the same page here), but I'm not suggesting to prioritize the dashboard so we can claim interoperability or pad ourselves on the back. I want so we have a clear target that we can show steady progress against.

Does that make sense?

To start with it will be red/empty for all/most the combinations.

IIUC, I think this should still be possible with A.1's simple dashboard, but it'll be a markdown output and require manual effort to: trigger to run the interop test plans and generate/update the resulting dashboard.

The comprehensiveness of the simple dashboard would be dependent on the permutations specified in the resource/composition file. From what I understand, I think we can specify all permutations as soon as the tooling is ready. It should also be possible to list incompatible/unimplemented combinations in the resource file (the testground run will fail and we can display the failure how we like.)

It also becomes a clear way for us to show progress each month

Having said that I think A.1 can meet this goal.
From what I understood, it really depends on the inputs provided and massaging the output csv file to markdown.

A.3 as I intended to describe it is: once all interop test-plans are written and running in CI, we can focus on generating a nicer output that can be shared on a website. An output that is updated from the CI run without manual intervention; maybe even hosted on interoperability.libp2p.io

I'm good to be scrappy here. At the minimum I want to make sure we can produce/show the interoperability dashboard with its dimensionality monthly during our all-hands (and ideally highlight the deltas). Even better (but not required) would be if this was a primary "metric" reported by libp2p in the weekly sitrep. I'm good with however we get there. If it's a major manual effort then it won't happen, so I'm good to invest in automation early here. Manually starting a github action on a monthly basis to generate the data seems good enough to me to start. Thanks all.

ROADMAP.md

marten-seemann · 2022-10-19T09:06:54Z

ROADMAP.md

+<!-- TODO: Create issue -->
+We have ported the tests from `libp2p/interop`
+- This repository implement tests `connect`, `dht`, `pubsub`, `steam` ([ref](https://github.com/libp2p/interop/blob/ce0aa3749c9c53cf5ad53009b273847b94106d40/src/index.ts#L32-L35))


Suggested change

- This repository implement tests `connect`, `dht`, `pubsub`, `steam` ([ref](https://github.com/libp2p/interop/blob/ce0aa3749c9c53cf5ad53009b273847b94106d40/src/index.ts#L32-L35))

- This repository implement tests `connect`, `dht`, `pubsub` ([ref](https://github.com/libp2p/interop/blob/ce0aa3749c9c53cf5ad53009b273847b94106d40/src/index.ts#L32-L35))

It's stream, not steam. This test just seems to be a simple echo handler, so this should be handled by our ping test already.

marten-seemann · 2022-10-19T09:09:02Z

ROADMAP.md

+<!-- TODO: Create issue -->
+We have a more stable build process that doesn't risk breaking
+- We generate artifacts for old versions during merges to the libp2p repositories https://github.com/libp2p/test-plans/issues/35#issuecomment-1254991985


I'm not sure if a stable build process deserves its own point. This seems like the bare minimum that's needed for testing anyway.

Generating artifacts, if I understand correctly, would just be an optimization to speed up build times.

I'm not 100% sure whether to keep or remove this.
@laurentsenta do you have any suggestions?

@marten-seemann has a point, no strong feelings against removing these

ROADMAP.md

p-shahi · 2022-10-20T16:10:02Z

Discussed in person with Laurent, will be merging this into his branch no.w

update roadmap

d8d72ce

p-shahi mentioned this pull request Oct 18, 2022

roadmap: reorder priority, reorganize sections libp2p/go-libp2p#1831

Merged

update

b2e52c2

p-shahi requested review from laurentsenta and BigLep October 19, 2022 00:54

p-shahi mentioned this pull request Oct 19, 2022

go-libp2p 2022 Q4/2023 Q1 Roadmap libp2p/go-libp2p#1806

Closed

p-shahi marked this pull request as ready for review October 19, 2022 01:00

p-shahi commented Oct 19, 2022

View reviewed changes

BigLep reviewed Oct 19, 2022

View reviewed changes

marten-seemann reviewed Oct 19, 2022

View reviewed changes

ROADMAP.md Outdated Show resolved Hide resolved

marten-seemann reviewed Oct 19, 2022

View reviewed changes

ROADMAP.md Outdated Show resolved Hide resolved

marten-seemann reviewed Oct 19, 2022

View reviewed changes

ROADMAP.md Outdated Show resolved Hide resolved

p-shahi added 2 commits October 19, 2022 17:58

updates

8a78d85

update

1791cd1

p-shahi mentioned this pull request Oct 20, 2022

test-plans 2022 Q4/2023 Q1 Roadmap #58

Open

p-shahi added 2 commits October 19, 2022 19:49

add browser test interop dependency

a436711

remove stable build

880b59d

p-shahi merged commit f8cf56b into feat/add-design-doc Oct 20, 2022

p-shahi mentioned this pull request Oct 20, 2022

Organization: add PROCESS.md and ROADMAP.md #44

Merged

4 tasks

p-shahi deleted the roadmap-updates branch October 21, 2022 06:40

p-shahi mentioned this pull request Nov 9, 2022

Update ROADMAP.md based on previous feedback. libp2p/rust-libp2p#3091

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

update roadmap #56

update roadmap #56

p-shahi commented Oct 18, 2022 •

edited

Loading

p-shahi Oct 19, 2022

p-shahi Oct 19, 2022

p-shahi Oct 19, 2022

p-shahi Oct 19, 2022

p-shahi Oct 19, 2022

BigLep left a comment

BigLep Oct 19, 2022 •

edited

Loading

marten-seemann Oct 19, 2022

MarcoPolo Oct 19, 2022

BigLep Oct 20, 2022

p-shahi Oct 20, 2022

BigLep Oct 19, 2022 •

edited

Loading

MarcoPolo Oct 19, 2022 •

edited

Loading

BigLep Oct 20, 2022

p-shahi Oct 20, 2022 •

edited

Loading

BigLep Oct 20, 2022

marten-seemann Oct 19, 2022

marten-seemann Oct 19, 2022

p-shahi Oct 20, 2022

laurentsenta Oct 20, 2022

p-shahi commented Oct 20, 2022

		- We fixed every known stability issue with the `libp2p/test-plans` test suite
		- [Issue 36](https://github.com/libp2p/test-plans/issues/36)

		- `libp2p/interop` and `libp2p/test-plans` are working together
		- They are either merged or somehow know about each other.

		Why: For libp2p to be competitive, it needs to delivers comparable performance to widely used protocols on the internet, namely HTTP/2 and HTTP/3.

		Goal: We have a test suite that runs libp2p transfers between nodes located at different locations all over the world, proving that libp2p is able to achieve performance on par with HTTP. The test suite is run on a continuous basis and results are published to a public performance dashboard.

		@@ -58,6 +61,11 @@ This document describes our process for testing interoperability & backward comp

		# How do we test libp2p interop at ProtocolLabs?

		---

		A GitHub workflow is added so that these suites run for any PR against master and run against the `master` branch on a nightly basis (updating the status check.)

		#### 4. Canonical interop tests & dashboard

	- This repository implement tests `connect`, `dht`, `pubsub`, `steam` ([ref](https://github.com/libp2p/interop/blob/ce0aa3749c9c53cf5ad53009b273847b94106d40/src/index.ts#L32-L35))
	- This repository implement tests `connect`, `dht`, `pubsub` ([ref](https://github.com/libp2p/interop/blob/ce0aa3749c9c53cf5ad53009b273847b94106d40/src/index.ts#L32-L35))

update roadmap #56

update roadmap #56

Conversation

p-shahi commented Oct 18, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BigLep left a comment

Choose a reason for hiding this comment

BigLep Oct 19, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BigLep Oct 19, 2022 • edited Loading

Choose a reason for hiding this comment

MarcoPolo Oct 19, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

p-shahi Oct 20, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

p-shahi commented Oct 20, 2022

p-shahi commented Oct 18, 2022 •

edited

Loading

BigLep Oct 19, 2022 •

edited

Loading

BigLep Oct 19, 2022 •

edited

Loading

MarcoPolo Oct 19, 2022 •

edited

Loading

p-shahi Oct 20, 2022 •

edited

Loading