Functional WS tests constantly failing

[pablitovicente]

Expected behavior

• Functional WS tests do not fail

Actual behavior

• Functional WS tests constantly fail with

[2019-05-14T08:23:55.073Z]      TimeoutError: Event response for 'rpc-request' timed out

[2019-05-14T08:23:55.073Z]       at SCClientSocket._handleEventAckTimeout (/home/lisk/workspace/lisk-sdk-framework_PR-3603/node_modules/socketcluster-client/lib/scclientsocket.js:700:17)

[2019-05-14T08:23:55.073Z]       at Timeout._onTimeout (/home/lisk/workspace/lisk-sdk-framework_PR-3603/node_modules/socketcluster-client/lib/scclientsocket.js:730:10)

Steps to reproduce

• Check any ongoing PR
• This behaviour is more common in Jenkins than locally but it does happen locally too with less frequency

Which version(s) does this affect? (Environment, OS, etc...)

[shuse2]

Development branch seems to be stable now, so closing this issue for now

[pablitovicente]

Initial investigation might suggest that when the node running the tests is under heavy load the ws tests are failing; the behaviour is random so still investigating how to solve.

[pablitovicente]

Information gathered about this issue:

• Although the error thrown by P2P seems to imply that the connection is dropped it's actually NOT open; for testing this I added debug points to socket cluster and when the failure occurs the connection is never made to the node

• The only way to reproduce this is to load all cores of the CPU to the max and then run the WS tests until they fail;

It's important to note that this only happens when running tests and the SDK is not actually affected by this issue.

Leaving issue open for more research in the future.