Decommission wakuv2 fleets

[jakubgs]

After discussion in Doha we've come to a conclusion it makes no sense to migrate the wakuv2.* fleets to use Postgres if we already have waku.test that uses it correctly. We have an overabundance of fleets and this chaos needs to be curtailed.

The wakuv2.test fleet is not depended upon the same way as wakuv2.prod is so it should not require a grace period.
In order to decommission wakuv2.prod extra caution will be necessary, due to it's usage in documentation and by 3rd parties.
Only the waku.sandbox fleet should be referenced in public-facing documentation in lieu of wakuv2.prod.

• Deploy a new waku.stanbox fleet with autosharding which will be used by js-waku and reference in docs.
  • Generate ENR records for the new waku.sandbox fleet.
  • Update fleets.json
  • Update canary checks
  • create CI job and push images
• Replace references to wakuv2.test with waku.test and wakuv2.prod with waku.sandbox following repos (also search for IP addresses and enr / enrtrees ids):
  • chore: switch wakuv2 to waku fleet waku-org/nwaku#2519
  • chore: switch wakuv2 fleet to waku waku-org/go-waku#1049
  • chore: switch wakuv2 fleet to waku waku-org/js-waku#1910
  • chore: switch wakuv2 fleet to waku waku-org/docs.waku.org#181
  • chore: switch wakuv2 fleet to waku waku-org/waku-tests#6
  • chore_: drop wakuv2 fleets, use waku fleets status-go#4933
  • chore: drop wakuv2 fleets, use waku fleets status-mobile#19288
  • chore: switch wakuv2 fleet to waku status-desktop#14033
  • And any others that are relevant.
  • AOFTICU2XWDULNLZGRMQS4RIZPAZEHYMV4FYHAPW563HNRAOERP7C and test.nodes.vac.dev for enrtrees look outdated
• Communicate decommissioning of both fleets to Waku community and relevant projects with a grace period (2 weeks?).
• Decommission wakuv2.test first.
• Decommission wakuv2.prod after a communicated grace period.
• Remove CI jobs
• fleets.json cleanup
• Canary checks cleanup
• Terraform backend cleanup

[jakubgs]

We don't need to fix ALL references in ALL repos, just the ones that matter, the rest can be fixed when the need arises.

[NagyZoltanPeter]

@yakimant : Can we wait decomissioning wakuv2.test till this Wed EOB, 14th of Feb?
Last weekend we ran a stress test on our upcoming release and encountered a strange peer disconnect and unsuccessful reconnects afterwards. We would like to ensure if its a continuous issue to trace and/or related to the fleet maybe.
Thank you.

[yakimant]

Ok, please don't forget to update, when you finish.

[yakimant]

We have a lot of multiaddr records like this in repostories (this one is for node-01.do-ams3.wakuv2.test.statusim.net):

/ip4/134.209.139.210/tcp/60000/p2p/16Uiu2HAmJb2e28qLXxT5kZxVUUoJt72EMzNGXB47Rxx5hw3q4YjS

I wonde if we can replace them with dns ones (for waku fleet)?

/dns4/node-01.do-ams3.waku.test.statusim.net/tcp/30303/p2p/16Uiu2HAkykgaECHswi3YKJ5dMLbq2kPVCo89fcyTd38UcQD6ej5W

[jakubgs]

It should be supported.

[NagyZoltanPeter]

Ok, please don't forget to update, when you finish.

@yakimant : nwaku 0.25.0 Release testing is ready. You can continue with decommissioning. Thank you for the support.

[yakimant]

Issues reappeared during waku.sandbox bootstrap:

• https://github.com/status-im/infra-role-bootstrap-linux/issues/37
• bootstrapping fails for non-jakubgs user infra-tf-alibaba-cloud#1

[yakimant]

New Google Cloud images introduced a clash with one of our Linux group ids.
Fixed by moving to another id:
https://github.com/status-im/infra-role-bootstrap-linux/commit/888e765be1c4bfdd464a07572d4494f18afb270c

[yakimant]

@Ivansete-status please help to figure out waku.sandbox parameters.

Here are ones from waku.test:
https://github.com/status-im/infra-waku/blob/master/ansible/group_vars/node.yml

[Ivansete-status]

Here are ones from waku.test: https://github.com/status-im/infra-waku/blob/master/ansible/group_vars/node.yml

Hi Anton!
For now, we could use similar values but we should separate the waku.test network from the waku.sandbox network.

I vote for just having The Waku Network in waku.sandbox and then waku.test to be connected in a cluster-id != 1.

Then, we need to adapt the following settings accordingly in both fleets:
https://github.com/status-im/infra-waku/blob/master/ansible/group_vars/node.yml#L11-L23
https://github.com/status-im/infra-waku/blob/85e8ad0f7ed46377aec301cf70ee6d2c342867c1/ansible/group_vars/node.yml#L45

WDYT about this fleet separation @jm-clius @alrevuelta ?

[alrevuelta]

What is waku.sandbox? Does it refer to the sandbox machine? If so, IMHO that's a machine, not a fleet nor a network.

I vote for just having The Waku Network in waku.sandbox and then waku.test to be connected in a cluster-id != 1.
Why do we want these 2 fleets? And when you refer to cluster-id!=1 to which cluster are you referring to?
Why moving TWN to waku.sandbox? afaik that's in waku.test?

[alrevuelta]

Deploy a new waku.stanbox fleet with autosharding which will be used by js-waku and reference in docs.

Okay, found this in the tittle of the issue. Unsure why this is needed. @fryorcraken ?

[Ivansete-status]

Okay, found this in the tittle of the issue. Unsure why this is needed. @fryorcraken ?

This task is part of the wakuv2.* decommission and therefore, we need replacements beforehand for those fleets.

As an analogy with the current wakuv2.* fleets, it is interesting to keep a stable fleet, waku.sandbox, which users and partners can work with while having a separate testing fleet, waku.test, which receives more regular updates and can be broken more frequently.

The waku.test fleet would be updated on every merged PR.
The waku.sandbox fleet will be updated monthly, per each new nwaku version.

[yakimant]

Then, we need to adapt the following settings accordingly in both fleets:
https://github.com/status-im/infra-waku/blob/master/ansible/group_vars/node.yml#L11-L23
https://github.com/status-im/infra-waku/blob/85e8ad0f7ed46377aec301cf70ee6d2c342867c1/ansible/group_vars/node.yml#L45

@Ivansete-status, can you please help to figure out the correct values for the sandbox fleet?

[yakimant]

wakuv2 -> waku substitutions:
https://docs.google.com/spreadsheets/d/1bkg9x4Zf6GweqC69CIXJPtpQX9ZBtehuaq6-yrK7EyI

[yakimant]

https://github.com/waku-org/waku-tests is not updated for years.

@jakubgs, @fryorcraken, is it worth it?
waku-org/waku-tests#6

[fryorcraken]

waku-org/waku-tests is not updated for years.

@jakubgs, @fryorcraken, is it worth it? waku-org/waku-tests#6

Repo archived. It's unused.

[jakubgs]

I've decomissioned wakuv2.test:

• infra-nim-waku#cc2d870c - wakuv2-test: decomission unused fleet

[jakubgs]

I have also decomissioned most nodes in wakuv2.prod:

• infra-nim-waku#8c51855f - prod: drop 2/3 hosts, keeping only do-ams3 one

The only host remaining is node-01.do-ams3.wakuv2.prod.status.im and will remain until js-waku supports RLN and sharding.