fix: possible netsplits and chokes #2635
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Bring current
…hould add their seed nodes to init.go. Inbound peer settings have been significantly increased because I theorize that the network is choked (symptom: low peer counts & slow sync speed)
update docs for 5.0.7 with embedded seeds
faddat
requested review from
alexanderbez,
zmanian,
crodriguezvega,
jackzampolin and
a team
as code owners
faddat
changed the title
This was referenced Jul 2, 2023
Closed
|
@mmulji-ic sir, something is wierd here. The changes I've made in this branch should not affect the tests that it seems to break. Is it possible you can take a look? I may have found it -- I added the other debug commands, but the CI output above may be worth having a peek at, because I can't figure out why adding other debug commands would ever make that happen. |
|
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR removes depguard and cleans up root.go
Related issues:
(I'm serious about the related issues -- they all relate back to the reality that the cosmos hub network is choked by default settings)
Recent tweet that could be related unless all of these nodes are run by the same VaaS:
Complimented by
This PR is complimented by iavl fast node because enabling iavl allows for better query handling and fixes a condition that is well known in tendermint: you can make p2p stop by hammering on the rpc endpoint, demonstrated here:
There's another related issue I opened on the tendermint repository before it was moved to comet, please inquire privately about that.
Combined, these could be used to destabilize the hub's p2p network.
Issues discovered while working on this
note
I am unsure why this is making the E2E tests fail. It seems like it's just one of them, and if somebody could have a peek that would be great, because I am back on baby duty 🙂.
other possibilities
It's perfectly possible that all of this was caused by a failure at a single VAS provider or by an interruption in the internet service links between the validators who experience downtime, and the rest of the world. In all cases, increasing the default number of inbound peers will improve performance on Gaia.