[CI] Add bootnode checking CI jobs

[s3krit]

Fixes paritytech/ci_cd#735
This PR creates two new Github workflows that check whether some given set of bootnodes are contactable. It does this by rewriting the chainspec for a given runtime such that it only includes the bootnode we wish to test. It then runs a polkadot node using that new chainspec, waits a little while and polls the local node's health_check endpoint to make sure it has been able to peer with other nodes.

The two tests are:

• One that runs at release-time (i.e., when a release candidate branch is pushed) that iterates over all of the bootnodes in the kusama, westend and polkadot runtimes and ensures that each of those bootnodes are contactable.
• One that runs every time a chainspec file in node/service/chain-specs/ is updated that only tests the connectivity of the newly-added bootnodes.

P.S., @paritytech/release-engineering I wasn't sure what to call the on-release workflow file, I noticed you've got some pattern for your github workflows, so lmk what I should change it to (i.e., what number I should prepend the file with).

EDIT: Oh also, here are some examples of the jobs being run against bootnodes. Notice that it is already finding a couple of uncontactable bootnodes (verified manually so we know it's working):

• Westend: https://github.com/paritytech/polkadot/actions/runs/4428685731/jobs/7768127546
• Kusama: https://github.com/paritytech/polkadot/actions/runs/4428685731/jobs/7768127726
• Polkadot: https://github.com/paritytech/polkadot/actions/runs/4428685731/jobs/7768128137

For all three runtimes, the bootnodes by polkadotters.com and metaspan.io are unresponsive. Also for kusama, one of our own bootnodes (/dns/kusama-bootnode-0.paritytech.net/tcp/30334/ws/p2p/12D3KooWSueCPH3puP2PcvqPJdNaDNF3jMZjtJtDiSy35pWrbt5h) is also uncontactable.

Currently failing bootnodes:

Westend:

[!] Bad bootnodes found for westend:
    /dns/westend-bootnode.polkadotters.com/tcp/30333/p2p/12D3KooWHPHb64jXMtSRJDrYFATWeLnvChL8NtWVttY67DCH1eC5
    /dns/boot-westend.metaspan.io/tcp/33016/wss/p2p/12D3KooWNTau7iG4G9cUJSwwt2QJP1W88pUf2SgqsHjRU2RL8pfa

Kusama:

 [!] Bad bootnodes found for kusama:
    /dns/kusama-bootnode-0.paritytech.net/tcp/30334/ws/p2p/12D3KooWSueCPH3puP2PcvqPJdNaDNF3jMZjtJtDiSy35pWrbt5h
    /dns/kusama-bootnode.polkadotters.com/tcp/30333/p2p/12D3KooWHB5rTeNkQdXNJ9ynvGz8Lpnmsctt7Tvp7mrYv6bcwbPG
    /dns/boot-kusama.metaspan.io/tcp/23016/wss/p2p/12D3KooWE1tq9ZL9AAxMiUBBqy1ENmh5pwfWabnoBPMo8gFPXhn6

Polkadot:

[!] Bad bootnodes found for polkadot:
    /dns/polkadot-bootnode.polkadotters.com/tcp/30333/p2p/12D3KooWPAVUgBaBk6n8SztLrMk8ESByncbAfRKUdxY1nygb9zG3
    /dns/boot-polkadot.metaspan.io/tcp/13016/wss/p2p/12D3KooWRjHFApinuqSBjoaDjQHvxwubQSpEVy5hrgC9Smvh92WF

Don't know if we want a process for removing them, should we remove them as part of another PR or contact the owners of said bootnodes first?

[senseless]

Thanks for this, we were trying to figure out a way to roll our own decentralized test and hadn't considered using custom chain spec files. We'll be rolling this into the decentralized IBP monitor to monitor our own nodes.

In the case of polkadotters, I believe they had an issue with their network security key. There is an open PR from them to correct this. In the case of metaspan, I reached out to let them know that their bootnodes were failing.

[dcolley]

I corrected an error in the proxy config for the wss connection. Using this command I'm able now to get a test node to start syncing:

docker run --rm parity/polkadot \
  --base-path /data \
  --no-hardware-benchmarks --no-mdns \
  --chain westend \
  --reserved-only \
  --reserved-nodes "/dns/boot-westend.metaspan.io/tcp/33016/wss/p2p/12D3KooWNTau7iG4G9cUJSwwt2QJP1W88pUf2SgqsHjRU2RL8pfa"

[s3krit]

I corrected an error in the proxy config for the wss connection. Using this command I'm able now to get a test node to start syncing:

docker run --rm parity/polkadot \
  --base-path /data \
  --no-hardware-benchmarks --no-mdns \
  --chain westend \
  --reserved-only \
  --reserved-nodes "/dns/boot-westend.metaspan.io/tcp/33016/wss/p2p/12D3KooWNTau7iG4G9cUJSwwt2QJP1W88pUf2SgqsHjRU2RL8pfa"

Awesome! I reran our job and the metaspan bootnode no longer appears uncontactable for the three runtimes we're checking, cheers!

[PierreBesson]

In my independent testing the Parity bootnode is UP: /dns/kusama-bootnode-0.paritytech.net/tcp/30334/ws/p2p/12D3KooWSueCPH3puP2PcvqPJdNaDNF3jMZjtJtDiSy35pWrbt5h

[s3krit]

In my independent testing the Parity bootnode is UP: /dns/kusama-bootnode-0.paritytech.net/tcp/30334/ws/p2p/12D3KooWSueCPH3puP2PcvqPJdNaDNF3jMZjtJtDiSy35pWrbt5h

So testing with --no-hardware-benchmarks --tmp --no-mdns --chain kusama --reserved-only --reserved-nodes "/dns/kusama-bootnode-0.paritytech.net/tcp/30334/ws/p2p/12D3KooWSueCPH3puP2PcvqPJdNaDNF3jMZjtJtDiSy35pWrbt5h", this will indeed connect to it as a peer (and the only peer), but when creating a chainspec with only that bootnode, we will never find any peers. Sounds to me like this bootnode is not well connected, which indicates to me a problem with the bootnode (and also confirms that Basti's suggested approach of alterning the chainspec to just include the bootnode we want to test was the correct approach to take).

[s3krit]

bot rebase

[paritytech-processbot]

Rebased

[s3krit]

bot merge