Skip to content
This repository has been archived by the owner on Jun 11, 2024. It is now read-only.

Unstable broadhash consensus #3574

Closed
4miners opened this issue May 9, 2019 · 5 comments
Closed

Unstable broadhash consensus #3574

4miners opened this issue May 9, 2019 · 5 comments
Assignees
@ishantiw
Labels
type: bug

Comments

@4miners
Copy link
Contributor

4miners commented May 9, 2019

Expected behavior

Broadhash consensus should be stable.

Actual behavior

Broadhash consensus is unstable which leads to missed blocks.

Steps to reproduce

Tested on network with 10 peers, observe logs.

Which version(s) does this affect? (Environment, OS, etc...)

2.0.0-alpha.0
@4miners
Copy link
Contributor Author

4miners commented May 9, 2019

Version 1.6.0:

[dbg] 2019-05-09 13:45:07 | Broadhash consensus: 100 %
[inf] 2019-05-09 13:45:10 | Broadhash consensus before forging a block: 100 %
[dbg] 2019-05-09 13:45:12 | Broadhash consensus: 100 %
[dbg] 2019-05-09 13:45:17 | Broadhash consensus: 100 %
[dbg] 2019-05-09 13:45:22 | Broadhash consensus: 100 %
[dbg] 2019-05-09 13:45:27 | Broadhash consensus: 100 %
[dbg] 2019-05-09 13:45:32 | Broadhash consensus: 100 %
[dbg] 2019-05-09 13:45:37 | Broadhash consensus: 100 %
[dbg] 2019-05-09 13:45:42 | Broadhash consensus: 100 %
[dbg] 2019-05-09 13:45:47 | Broadhash consensus: 100 %
[dbg] 2019-05-09 13:45:52 | Broadhash consensus: 100 %
[dbg] 2019-05-09 13:45:57 | Broadhash consensus: 100 %
[dbg] 2019-05-09 13:46:02 | Broadhash consensus: 100 %
[dbg] 2019-05-09 13:46:07 | Broadhash consensus: 100 %
[dbg] 2019-05-09 13:46:12 | Broadhash consensus: 100 %
[dbg] 2019-05-09 13:46:17 | Broadhash consensus: 100 %
[dbg] 2019-05-09 13:46:22 | Broadhash consensus: 100 %
[dbg] 2019-05-09 13:46:27 | Broadhash consensus: 100 %
[inf] 2019-05-09 13:46:30 | Broadhash consensus before forging a block: 100 %
[dbg] 2019-05-09 13:46:32 | Broadhash consensus: 100 %
[dbg] 2019-05-09 13:46:37 | Broadhash consensus: 100 %
[dbg] 2019-05-09 13:46:42 | Broadhash consensus: 100 %
[dbg] 2019-05-09 13:46:47 | Broadhash consensus: 100 %
[dbg] 2019-05-09 13:46:52 | Broadhash consensus: 100 %
[dbg] 2019-05-09 13:46:57 | Broadhash consensus: 100 %
[dbg] 2019-05-09 13:47:02 | Broadhash consensus: 100 %
[dbg] 2019-05-09 13:47:07 | Broadhash consensus: 100 %
[inf] 2019-05-09 13:47:10 | Broadhash consensus before forging a block: 100 %
[dbg] 2019-05-09 13:47:12 | Broadhash consensus: 100 %
[dbg] 2019-05-09 13:47:17 | Broadhash consensus: 100 %
[dbg] 2019-05-09 13:47:22 | Broadhash consensus: 100 %
[dbg] 2019-05-09 13:47:27 | Broadhash consensus: 100 %
[inf] 2019-05-09 13:47:30 | Broadhash consensus before forging a block: 100 %
[dbg] 2019-05-09 13:47:32 | Broadhash consensus: 100 %
[dbg] 2019-05-09 13:47:37 | Broadhash consensus: 100 %
[dbg] 2019-05-09 13:47:42 | Broadhash consensus: 100 %
[dbg] 2019-05-09 13:47:47 | Broadhash consensus: 100 %
[dbg] 2019-05-09 13:47:52 | Broadhash consensus: 100 %
[dbg] 2019-05-09 13:47:57 | Broadhash consensus: 100 %
[inf] 2019-05-09 13:48:00 | Broadhash consensus before forging a block: 100 %
[dbg] 2019-05-09 13:48:02 | Broadhash consensus: 100 %
[dbg] 2019-05-09 13:48:07 | Broadhash consensus: 100 %
[dbg] 2019-05-09 13:48:12 | Broadhash consensus: 100 %

Version 2.0.0-alpha.0:

13:46:12.794Z DEBUG lisk-framework: Broadhash consensus: 0 %
13:46:17.796Z DEBUG lisk-framework: Broadhash consensus: 0 %
13:46:22.801Z DEBUG lisk-framework: Broadhash consensus: 90 %
13:46:27.803Z DEBUG lisk-framework: Broadhash consensus: 90 %
13:46:32.805Z DEBUG lisk-framework: Broadhash consensus: 90 %
13:46:37.809Z DEBUG lisk-framework: Broadhash consensus: 90 %
13:46:42.811Z DEBUG lisk-framework: Broadhash consensus: 0 %
13:46:47.814Z DEBUG lisk-framework: Broadhash consensus: 0 %
13:46:50.834Z ERROR lisk-framework: Failed to generate block within delegate slot Inadequate broadhash consensus before forging a block: 0 %
13:46:51.840Z ERROR lisk-framework: Failed to generate block within delegate slot Inadequate broadhash consensus before forging a block: 0 %
13:46:52.816Z DEBUG lisk-framework: Broadhash consensus: 0 %
13:46:52.847Z ERROR lisk-framework: Failed to generate block within delegate slot Inadequate broadhash consensus before forging a block: 0 %
13:46:53.852Z ERROR lisk-framework: Failed to generate block within delegate slot Inadequate broadhash consensus before forging a block: 0 %
13:46:54.858Z ERROR lisk-framework: Failed to generate block within delegate slot Inadequate broadhash consensus before forging a block: 0 %
13:46:55.867Z ERROR lisk-framework: Failed to generate block within delegate slot Inadequate broadhash consensus before forging a block: 0 %
13:46:56.875Z ERROR lisk-framework: Failed to generate block within delegate slot Inadequate broadhash consensus before forging a block: 0 %
13:46:57.818Z DEBUG lisk-framework: Broadhash consensus: 0 %
13:46:57.880Z ERROR lisk-framework: Failed to generate block within delegate slot Inadequate broadhash consensus before forging a block: 0 %
13:46:58.886Z ERROR lisk-framework: Failed to generate block within delegate slot Inadequate broadhash consensus before forging a block: 0 %
13:46:59.889Z ERROR lisk-framework: Failed to generate block within delegate slot Inadequate broadhash consensus before forging a block: 0 %
13:47:02.821Z DEBUG lisk-framework: Broadhash consensus: 10 %
13:47:05.469Z DEBUG lisk-framework: Establishing broadhash consensus before sync: 40 %
13:47:05.647Z DEBUG lisk-framework: Establishing broadhash consensus after sync: 40 %
13:47:07.822Z DEBUG lisk-framework: Broadhash consensus: 50 %
13:47:10.932Z  INFO lisk-framework: Broadhash consensus before forging a block: 90 %
13:47:12.824Z DEBUG lisk-framework: Broadhash consensus: 0 %
13:47:17.826Z DEBUG lisk-framework: Broadhash consensus: 0 %
13:47:22.828Z DEBUG lisk-framework: Broadhash consensus: 90 %
13:47:27.831Z DEBUG lisk-framework: Broadhash consensus: 90 %
13:47:32.834Z DEBUG lisk-framework: Broadhash consensus: 90 %
13:47:37.836Z DEBUG lisk-framework: Broadhash consensus: 90 %
13:47:42.838Z DEBUG lisk-framework: Broadhash consensus: 0 %
13:47:47.840Z DEBUG lisk-framework: Broadhash consensus: 0 %
13:47:52.842Z DEBUG lisk-framework: Broadhash consensus: 10 %
13:47:55.654Z DEBUG lisk-framework: Establishing broadhash consensus before sync: 40 %
13:47:55.772Z DEBUG lisk-framework: Establishing broadhash consensus after sync: 40 %
13:47:57.845Z DEBUG lisk-framework: Broadhash consensus: 50 %
13:48:02.847Z DEBUG lisk-framework: Broadhash consensus: 90 %
13:48:07.849Z DEBUG lisk-framework: Broadhash consensus: 90 %
13:48:12.851Z DEBUG lisk-framework: Broadhash consensus: 0 %
13:48:17.853Z DEBUG lisk-framework: Broadhash consensus: 0 %
13:48:20.213Z ERROR lisk-framework: Failed to generate block within delegate slot Inadequate broadhash consensus before forging a block: 0 %
13:48:21.217Z ERROR lisk-framework: Failed to generate block within delegate slot Inadequate broadhash consensus before forging a block: 0 %
13:48:22.220Z ERROR lisk-framework: Failed to generate block within delegate slot Inadequate broadhash consensus before forging a block: 0 %
13:48:22.856Z DEBUG lisk-framework: Broadhash consensus: 0 %
13:48:23.224Z ERROR lisk-framework: Failed to generate block within delegate slot Inadequate broadhash consensus before forging a block: 0 %
13:48:24.229Z ERROR lisk-framework: Failed to generate block within delegate slot Inadequate broadhash consensus before forging a block: 0 %
13:48:25.236Z ERROR lisk-framework: Failed to generate block within delegate slot Inadequate broadhash consensus before forging a block: 0 %
13:48:26.243Z ERROR lisk-framework: Failed to generate block within delegate slot Inadequate broadhash consensus before forging a block: 0 %
13:48:27.248Z ERROR lisk-framework: Failed to generate block within delegate slot Inadequate broadhash consensus before forging a block: 0 %
13:48:27.858Z DEBUG lisk-framework: Broadhash consensus: 0 %
13:48:28.255Z ERROR lisk-framework: Failed to generate block within delegate slot Inadequate broadhash consensus before forging a block: 0 %
13:48:29.259Z ERROR lisk-framework: Failed to generate block within delegate slot Inadequate broadhash consensus before forging a block: 0 %
13:48:32.860Z DEBUG lisk-framework: Broadhash consensus: 90 %
13:48:37.862Z DEBUG lisk-framework: Broadhash consensus: 90 %
13:48:40.331Z  INFO lisk-framework: Broadhash consensus before forging a block: 90 %
13:48:42.864Z DEBUG lisk-framework: Broadhash consensus: 0 %
13:48:47.865Z DEBUG lisk-framework: Broadhash consensus: 0 %

@4miners
Copy link
Contributor Author

4miners commented May 9, 2019

This issue can be linked to #3533

@shuse2
Copy link
Collaborator

shuse2 commented May 9, 2019

In here, no need to use newPeers.
https://github.com/LiskHQ/lisk-sdk/blob/release/2.0.0/framework/src/modules/network/filter_peers.js#L170
Also, state property needs to be updated to match the framework usage.
if it's connectedPeer, state should be 2 and if it's triedPeer, it should be 0.

Maybe another issue but connectedPeer and triedPeer seems to create duplicate information and triedPeer has outdated information.
Also, newPeer include itself

@ishantiw ishantiw mentioned this issue May 10, 2019
Merged
@ishantiw ishantiw self-assigned this May 10, 2019
@shuse2 shuse2 closed this as completed May 10, 2019
@4miners
Copy link
Contributor Author

4miners commented May 13, 2019 •
edited
Loading

Now the consensus is too good (2.0.0-alpha.2):

13:34:26.938Z DEBUG lisk-framework: Establishing broadhash consensus before sync: 110 %
13:34:27.023Z DEBUG lisk-framework: Establishing broadhash consensus after sync: 110 %
13:34:37.030Z DEBUG lisk-framework: Establishing broadhash consensus before sync: 110 %
13:34:37.050Z DEBUG lisk-framework: Establishing broadhash consensus after sync: 110 %
13:34:47.057Z DEBUG lisk-framework: Establishing broadhash consensus before sync: 110 %
13:34:47.176Z DEBUG lisk-framework: Establishing broadhash consensus after sync: 110 %
13:34:57.184Z DEBUG lisk-framework: Establishing broadhash consensus before sync: 110 %
13:34:57.281Z DEBUG lisk-framework: Establishing broadhash consensus after sync: 110 %
13:35:07.289Z DEBUG lisk-framework: Establishing broadhash consensus before sync: 110 %
13:35:07.381Z DEBUG lisk-framework: Establishing broadhash consensus after sync: 110 %
13:35:17.394Z DEBUG lisk-framework: Establishing broadhash consensus before sync: 110 %
13:35:17.498Z DEBUG lisk-framework: Establishing broadhash consensus after sync: 110 %
13:35:27.502Z DEBUG lisk-framework: Establishing broadhash consensus before sync: 110 %
13:35:27.547Z DEBUG lisk-framework: Establishing broadhash consensus after sync: 110 %
13:35:37.552Z DEBUG lisk-framework: Establishing broadhash consensus before sync: 110 %
13:35:37.594Z DEBUG lisk-framework: Establishing broadhash consensus after sync: 110 %
13:35:47.604Z DEBUG lisk-framework: Establishing broadhash consensus before sync: 110 %
13:35:47.698Z DEBUG lisk-framework: Establishing broadhash consensus after sync: 110 %
13:35:57.702Z DEBUG lisk-framework: Establishing broadhash consensus before sync: 110 %
13:35:57.734Z DEBUG lisk-framework: Establishing broadhash consensus after sync: 110 %
13:36:07.740Z DEBUG lisk-framework: Establishing broadhash consensus before sync: 110 %
13:36:07.837Z DEBUG lisk-framework: Establishing broadhash consensus after sync: 110 %
13:36:17.845Z DEBUG lisk-framework: Establishing broadhash consensus before sync: 110 %
13:36:17.929Z DEBUG lisk-framework: Establishing broadhash consensus after sync: 110 %
13:36:27.935Z DEBUG lisk-framework: Establishing broadhash consensus before sync: 110 %
13:36:28.036Z DEBUG lisk-framework: Establishing broadhash consensus after sync: 110 %
13:36:38.043Z DEBUG lisk-framework: Establishing broadhash consensus before sync: 110 %
13:36:38.063Z DEBUG lisk-framework: Establishing broadhash consensus after sync: 110 %

Additionally, probably there is still some issue regarding connecting to itself:

13:37:14.744Z  INFO lisk-framework: Received inbound request for procedure updateMyself
13:37:15.382Z DEBUG lisk-framework: No delegates enabled
13:37:16.383Z DEBUG lisk-framework: No delegates enabled
13:37:16.557Z  INFO lisk-framework: Transaction pool - received size: 0 validated size: 0 verified size: 0 pending size: 0 ready size: 0
13:37:17.331Z DEBUG lisk-framework: Outbound connection of peer 134.209.75.228:5000 was closed with code 4101 and reason: Peer cannot connect to itself
13:37:17.332Z ERROR lisk-framework: Event 'rpc-request' was aborted due to a bad connection
13:37:17.333Z DEBUG lisk-framework: Socket connection closed with status code 4101 and reason: Peer cannot connect to itself
13:37:17.385Z DEBUG lisk-framework: No delegates enabled
13:37:18.345Z  INFO lisk-framework: Updated info of peer 159.203.112.223:5000 to {"os":"linux4.15.0-48-generic","version":"2.0.0-alpha.2","wsPort":5000,"httpPort":4000,"minVersion":"1.1.0-rc.0","protocolVersion":"1.0","height":1,"nethash":"198f2b61a8eb95fbeed58b8216780b68f697f26b849acf00c8c93bb9b24f783d","broadhash":"198f2b61a8eb95fbeed58b8216780b68f697f26b849acf00c8c93bb9b24f783d","nonce":"T5GkLqiYxD3CAxXL","state":2,"ipAddress":"159.203.112.223"}
13:37:18.347Z  INFO lisk-framework: Received inbound request for procedure updateMyself
13:37:18.361Z  INFO lisk-framework: Starting sync
13:37:18.365Z DEBUG lisk-framework: Establishing broadhash consensus before sync: 110 %

@4miners 4miners reopened this May 13, 2019
@ishantiw ishantiw mentioned this issue May 15, 2019
Merged
@ishantiw
Copy link
Contributor

PR #3634 will fix the too good consensus,

@4miners Regarding the comment about the peer connecting to itself. if you are referring to the log below then its because its the expected behavior, when a node tries to connect to itself, it is rejected with this error and drops the connection based on nonce so it's not accounted in the calculation of broadhash consensus.

13:37:17.331Z DEBUG lisk-framework: Outbound connection of peer 134.209.75.228:5000 was closed with code 4101 and reason: Peer cannot connect to itself

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@ishantiw ishantiw

Labels
type: bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@pablitovicente @shuse2 @ishantiw @4miners