Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Release 4.8.0 - Alpha 1 - Workload benchmarks metrics #20721

Closed
2 tasks done
GGP1 opened this issue Dec 11, 2023 · 5 comments
Closed
2 tasks done

Release 4.8.0 - Alpha 1 - Workload benchmarks metrics #20721

GGP1 opened this issue Dec 11, 2023 · 5 comments
Assignees
@GGP1
Labels
level/task type/test

Comments

@GGP1
Copy link
Member

GGP1 commented Dec 11, 2023 •
edited by damarisg
Loading

The following issue aims to run all workload benchmarks for the current release candidate, report the results, and open new issues for any encountered errors.

Workload benchmarks metrics information
Main release candidate issue #20667
Version 4.8.0
Release candidate # Alpha 1
Tag v4.8.0-alpha1
Previous Workload benchmarks metrics issue #19845

Test configuration

All tests will be run and workload performance metrics will be obtained for the following clustered environment configurations:
# Agents # Worker nodes
50000 25

Test report procedure

All individual test checks must be marked as:
Pass The test ran successfully.
Xfail The test was expected to fail and it failed. It must be properly justified and reported in an issue.
Skip The test was not run. It must be properly justified and reported in an issue.
Fail The test failed. A new issue must be opened to evaluate and address the problem.

All test results must have one the following statuses:
🟢 All checks passed.
🔴 There is at least one failed check.
🟡 There is at least one expected fail or skipped test and no failures.

Any failing test must be properly addressed with a new issue, detailing the error and the possible cause. It must be included in the Fixes section of the current release candidate main issue.

Any expected fail or skipped test must have an issue justifying the reason. All auditors must validate the justification for an expected fail or skipped test.

An extended report of the test results must be attached as a zip or txt. This report can be used by the auditors to dig deeper into any possible failures and details.

Conclusions

All tests have been executed and the results can be found here and here.

All tests have passed and the fails have been reported or justified. I therefore conclude that this issue is finished and OK for this release candidate.

New issues

Known issues

Auditors validation

The definition of done for this one is the validation of the conclusions and the test results from all auditors.

All checks from below must be accepted in order to close this issue.
@GGP1 GGP1 self-assigned this Dec 11, 2023
@damarisg damarisg mentioned this issue Dec 11, 2023
Closed
@GGP1
Copy link
Member Author

GGP1 commented Dec 12, 2023 •
edited by Selutario
Loading

API Performance Tests

Artifacts

artifacts.zip

Tests report

api_performance_tests.zip

Tests results

Endpoint test name Status Issues Ref.
GET /cluster/local/info 🟢
GET /cluster/nodes 🟢
GET /cluster/healthcheck 🟢
GET /cluster/status 🟢
GET /cluster/local/config 🟢
GET /cluster/api/config 🟢
GET /cluster/configuration/validation 🟢
GET /manager/status 🟢
GET /manager/info 🟢
GET /manager/configuration 🟢
GET /manager/logs 🟢
GET /manager/api/config 🟢
GET /manager/configuration/validation 🟢
GET /mitre/groups 🟢
GET /mitre/metadata 🟢
GET /mitre/mitigations 🟢
GET /mitre/references 🟢
GET /mitre/software 🟢
GET /mitre/tactics 🟢
GET /mitre/techniques 🟢
GET /overview/agents 🟢
GET /tasks/status 🟢
PUT /active-response 🟡 XFAIL. wazuh/wazuh-qa#1266
PUT /rootcheck 🟢
PUT /syscheck 🟢
POST /groups 🟢
POST /security/users 🟢
POST /security/roles 🟢
POST /security/policies 🟢
POST /security/rules 🟢
PUT /agents/group 🟡 XFAIL. #13872
PUT /agents/group/new_test_group/restart 🟢
PUT /agents/restart 🟢
POST /agents 🟢
POST /agents/insert​ 🟢
POST /agents/insert​/quick 🟢
GET /agents 🟢
GET /agents/no_group 🟢
GET /agents/outdated 🟢
GET /decoders 🟢
GET /groups 🟢
GET /lists 🟢
GET /rules 🟢
GET /security/users 🟢
GET /security/roles 🟢
GET /security/policies 🟢
GET /security/rules 🟢
DELETE /groups 🟢
DELETE /agents 🟢
DELETE /security/users 🟢
DELETE /security/roles 🟢
DELETE /security/policies 🟢
DELETE /security/rules 🟢
PUT /manager/restart 🟢
PUT /cluster/restart 🟢

Note

XFAIL tests list

@GGP1 GGP1 mentioned this issue Dec 13, 2023
Closed
4 tasks
@GGP1
Copy link
Member Author

GGP1 commented Dec 12, 2023 •
edited by Selutario
Loading

Cluster

Performance 🟡

The test_cluster_performance test is failing because of wazuh/wazuh-qa#4298

Artifacts: test_cluster_performance.zip

Reliability 🔴

Artifacts: test_cluster_reliability.zip

Test Status Issues Ref.
test_cluster_connection 🟡 wazuh/wazuh-qa#4385
test_cluster_error_logs 🔴 New issue wazuh/wazuh-qa#4767
test_check_logs_order_workers 🟡 #20162 wazuh/wazuh-qa#4385
test_check_logs_order_master 🟢
test_cluster_sync 🟢
test_cluster_task_order 🟢

Logs

No errors were found in the master logs. On the other hand, the workers did contain some connection errors, some of them are expected due to the workers' disconnections during the tests.

However, the errors

Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
Error during connection with '587879': [Errno 104] Connection reset by peer.

were not present in the last release testing. They are causing the reliability test test_cluster_error_logs to fail. I've opened the issue #20739 to address this.

Similar issue: wazuh/wazuh-qa#4385

The results were obtained with a modified version of the script given here.

cluster logs 
Searching for errors in /home/gasti/Downloads/artifacts/master/logs/cluster.log:
-------------------------------------------
Searching for errors in /home/gasti/Downloads/artifacts/worker_10/logs/cluster.log:
2023/12/11 17:52:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:42 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:52 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:02 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:12 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
-------------------------------------------
Searching for errors in /home/gasti/Downloads/artifacts/worker_11/logs/cluster.log:
2023/12/11 17:52:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:42 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:52 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:02 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:12 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:46 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:51 ERROR: [Local 500773] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:52 ERROR: [Local 171216] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:53 ERROR: [Local 866293] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:54 ERROR: [Local 112240] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:55 ERROR: [Local 855821] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:56 ERROR: [Local 195756] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
-------------------------------------------
Searching for errors in /home/gasti/Downloads/artifacts/worker_12/logs/cluster.log:
2023/12/11 17:52:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:42 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:52 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:02 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:12 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:46 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:51 ERROR: [Local 829497] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:52 ERROR: [Local 461876] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:53 ERROR: [Local 226390] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:54 ERROR: [Local 95276] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:55 ERROR: [Local 227064] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:56 ERROR: [Local 646483] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
-------------------------------------------
Searching for errors in /home/gasti/Downloads/artifacts/worker_13/logs/cluster.log:
2023/12/11 17:52:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:42 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:52 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:02 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:12 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
-------------------------------------------
Searching for errors in /home/gasti/Downloads/artifacts/worker_14/logs/cluster.log:
2023/12/11 17:52:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:42 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:52 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:02 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:12 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:46 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:51 ERROR: [Local 658312] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:52 ERROR: [Local 760677] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:53 ERROR: [Local 273922] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:54 ERROR: [Local 420308] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:55 ERROR: [Local 859596] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:56 ERROR: [Local 432294] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
-------------------------------------------
Searching for errors in /home/gasti/Downloads/artifacts/worker_15/logs/cluster.log:
2023/12/11 17:52:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:42 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:52 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:02 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:12 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:45 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:51 ERROR: [Local 434522] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:52 ERROR: [Local 192752] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:53 ERROR: [Local 583872] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:54 ERROR: [Local 791664] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:55 ERROR: [Local 1041366] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
-------------------------------------------
Searching for errors in /home/gasti/Downloads/artifacts/worker_16/logs/cluster.log:
2023/12/11 17:52:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:42 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:52 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:02 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:12 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:45 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:51 ERROR: [Local 255862] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:52 ERROR: [Local 903770] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:53 ERROR: [Local 436372] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:54 ERROR: [Local 302414] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:55 ERROR: [Local 168358] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:55:28 ERROR: [Local 905521] [Main] Error during connection with '905521': [Errno 104] Connection reset by peer.
-------------------------------------------
Searching for errors in /home/gasti/Downloads/artifacts/worker_17/logs/cluster.log:
2023/12/11 17:52:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:42 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:52 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:02 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:12 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:45 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:51 ERROR: [Local 761192] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:52 ERROR: [Local 273414] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:53 ERROR: [Local 333686] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:54 ERROR: [Local 543107] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:55 ERROR: [Local 399125] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
-------------------------------------------
Searching for errors in /home/gasti/Downloads/artifacts/worker_18/logs/cluster.log:
2023/12/11 17:52:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:42 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:52 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:02 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:12 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:45 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:51 ERROR: [Local 877870] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:52 ERROR: [Local 1972] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:53 ERROR: [Local 535491] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:54 ERROR: [Local 891742] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:55 ERROR: [Local 291281] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
-------------------------------------------
Searching for errors in /home/gasti/Downloads/artifacts/worker_19/logs/cluster.log:
2023/12/11 17:52:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:42 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:52 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:02 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:12 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:46 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:51 ERROR: [Local 493528] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:52 ERROR: [Local 690468] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:53 ERROR: [Local 1041190] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:54 ERROR: [Local 782428] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:55 ERROR: [Local 711730] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:56 ERROR: [Local 669595] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
-------------------------------------------
Searching for errors in /home/gasti/Downloads/artifacts/worker_1/logs/cluster.log:
2023/12/11 17:52:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:42 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:52 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:02 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:12 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:46 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:51 ERROR: [Local 544633] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:52 ERROR: [Local 585147] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:53 ERROR: [Local 125562] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:54 ERROR: [Local 535538] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:55 ERROR: [Local 127205] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:56 ERROR: [Local 80983] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
-------------------------------------------
Searching for errors in /home/gasti/Downloads/artifacts/worker_20/logs/cluster.log:
2023/12/11 17:52:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:42 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:52 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:02 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:12 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:46 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:51 ERROR: [Local 989779] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:52 ERROR: [Local 245692] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:53 ERROR: [Local 871920] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:54 ERROR: [Local 915938] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:55 ERROR: [Local 7128] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
-------------------------------------------
Searching for errors in /home/gasti/Downloads/artifacts/worker_21/logs/cluster.log:
2023/12/11 17:52:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:42 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:52 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:02 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:12 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:45 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:51 ERROR: [Local 870999] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:52 ERROR: [Local 991766] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:53 ERROR: [Local 545448] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:54 ERROR: [Local 297102] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:55 ERROR: [Local 631678] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
-------------------------------------------
Searching for errors in /home/gasti/Downloads/artifacts/worker_22/logs/cluster.log:
2023/12/11 17:52:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:42 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:52 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:02 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:12 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:46 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:51 ERROR: [Local 208564] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:52 ERROR: [Local 847761] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:53 ERROR: [Local 317984] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:54 ERROR: [Local 534473] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:55 ERROR: [Local 995057] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:56 ERROR: [Local 904090] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
-------------------------------------------
Searching for errors in /home/gasti/Downloads/artifacts/worker_23/logs/cluster.log:
2023/12/11 17:52:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:42 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:52 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:02 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:12 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:46 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:51 ERROR: [Local 368165] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:52 ERROR: [Local 977840] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:53 ERROR: [Local 901146] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:54 ERROR: [Local 269108] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:55 ERROR: [Local 450064] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:56 ERROR: [Local 258082] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
-------------------------------------------
Searching for errors in /home/gasti/Downloads/artifacts/worker_24/logs/cluster.log:
2023/12/11 17:52:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:42 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:52 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:02 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:12 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:46 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:51 ERROR: [Local 562763] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:52 ERROR: [Local 352565] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:53 ERROR: [Local 799843] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:54 ERROR: [Local 942882] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:55 ERROR: [Local 968786] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:56 ERROR: [Local 491719] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
-------------------------------------------
Searching for errors in /home/gasti/Downloads/artifacts/worker_25/logs/cluster.log:
2023/12/11 17:52:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:42 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:52 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:02 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:12 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
-------------------------------------------
Searching for errors in /home/gasti/Downloads/artifacts/worker_2/logs/cluster.log:
2023/12/11 17:52:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:42 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:52 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:02 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:12 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:46 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:51 ERROR: [Local 640630] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:52 ERROR: [Local 308644] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:53 ERROR: [Local 926559] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:54 ERROR: [Local 339583] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:55 ERROR: [Local 325687] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:56 ERROR: [Local 143448] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
-------------------------------------------
Searching for errors in /home/gasti/Downloads/artifacts/worker_3/logs/cluster.log:
2023/12/11 17:52:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:42 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:52 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:02 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:12 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:46 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:51 ERROR: [Local 865074] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:52 ERROR: [Local 146116] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:53 ERROR: [Local 261746] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:54 ERROR: [Local 398269] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:55 ERROR: [Local 980098] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:56 ERROR: [Local 240434] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
-------------------------------------------
Searching for errors in /home/gasti/Downloads/artifacts/worker_4/logs/cluster.log:
2023/12/11 17:52:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:42 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:52 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:02 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:12 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:46 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:51 ERROR: [Local 423350] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:52 ERROR: [Local 806038] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:53 ERROR: [Local 337877] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:54 ERROR: [Local 231627] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:55 ERROR: [Local 336608] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:56 ERROR: [Local 113612] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
-------------------------------------------
Searching for errors in /home/gasti/Downloads/artifacts/worker_5/logs/cluster.log:
2023/12/11 17:52:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:42 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:52 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:02 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:12 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:45 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:51 ERROR: [Local 418258] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:52 ERROR: [Local 180930] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:53 ERROR: [Local 254583] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:54 ERROR: [Local 710860] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:55 ERROR: [Local 914281] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:55:23 ERROR: [Local 15598] [Main] Error during connection with '15598': [Errno 104] Connection reset by peer.
-------------------------------------------
Searching for errors in /home/gasti/Downloads/artifacts/worker_6/logs/cluster.log:
2023/12/11 17:52:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:42 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:52 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:02 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:12 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:45 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:51 ERROR: [Local 470992] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:52 ERROR: [Local 537158] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:53 ERROR: [Local 316117] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:54 ERROR: [Local 987535] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:55 ERROR: [Local 276403] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
-------------------------------------------
Searching for errors in /home/gasti/Downloads/artifacts/worker_7/logs/cluster.log:
2023/12/11 17:52:12 ERROR: [Worker CLUSTER-Workload_benchmarks_metrics_B316_manager_7] [Main] Connection closed due to an unhandled error: [Errno 104] Connection reset by peer
2023/12/11 17:52:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:42 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:52 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:02 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:12 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:45 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:51 ERROR: [Local 648169] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:52 ERROR: [Local 889249] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:53 ERROR: [Local 608884] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:54 ERROR: [Local 769641] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:55 ERROR: [Local 122493] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:54:15 ERROR: [Local 587879] [Main] Error during connection with '587879': [Errno 104] Connection reset by peer.
-------------------------------------------
Searching for errors in /home/gasti/Downloads/artifacts/worker_8/logs/cluster.log:
2023/12/11 17:52:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:42 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:52 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:02 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:12 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:45 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:51 ERROR: [Local 540634] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:52 ERROR: [Local 190983] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:53 ERROR: [Local 346689] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:54 ERROR: [Local 203176] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:55 ERROR: [Local 971721] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
-------------------------------------------
Searching for errors in /home/gasti/Downloads/artifacts/worker_9/logs/cluster.log:
2023/12/11 17:52:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:32 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:42 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:52:52 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:02 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:12 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:22 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:46 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:51 ERROR: [Local 179337] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:52 ERROR: [Local 196727] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:53 ERROR: [Local 874430] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:54 ERROR: [Local 176873] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:55 ERROR: [Local 402924] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:53:56 ERROR: [Local 968555] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
-------------------------------------------

@Selutario
Copy link
Contributor

Benchmark review

Logs

Regarding these errors:

2023/12/11 17:53:55 ERROR: [Local 122493] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master
2023/12/11 17:54:15 ERROR: [Local 587879] [Main] Error during connection with '587879': [Errno 104] Connection reset by peer.

Although they have not appeared on the last release testing, they are not worrying. Usually it is simply a matter of chance. In this case, the master was restarted at 17:52:09, and then the entire cluster at 17:53:34:

2023/12/11 17:52:09 INFO: wazuh 172.31.16.36 "PUT /manager/restart" with parameters {} and body {} done in 0.052s: 200
2023/12/11 17:52:39 INFO: unknown_user 172.31.16.36 "GET /" with parameters {} and body {} done in 0.017s: 400
2023/12/11 17:52:44 INFO: unknown_user 172.31.16.36 "GET /" with parameters {} and body {} done in 0.016s: 400
2023/12/11 17:52:50 INFO: unknown_user 172.31.16.36 "GET /" with parameters {} and body {} done in 0.016s: 400
2023/12/11 17:52:55 INFO: unknown_user 172.31.16.36 "GET /" with parameters {} and body {} done in 0.017s: 400
2023/12/11 17:53:00 INFO: unknown_user 172.31.16.36 "GET /" with parameters {} and body {} done in 0.016s: 400
2023/12/11 17:53:05 INFO: unknown_user 172.31.16.36 "GET /" with parameters {} and body {} done in 0.016s: 400
2023/12/11 17:53:10 INFO: unknown_user 172.31.16.36 "GET /" with parameters {} and body {} done in 0.017s: 400
2023/12/11 17:53:16 INFO: Checking RBAC database integrity...
2023/12/11 17:53:16 INFO: /var/ossec/api/configuration/security/rbac.db file was detected
2023/12/11 17:53:16 INFO: RBAC database integrity check finished successfully
2023/12/11 17:53:20 INFO: Listening on 0.0.0.0:55000..
2023/12/11 17:53:20 INFO: unknown_user 172.31.16.36 "GET /" with parameters {} and body {} done in 0.201s: 400
2023/12/11 17:53:25 INFO: unknown_user 172.31.16.36 "GET /" with parameters {} and body {} done in 0.019s: 400
2023/12/11 17:53:33 INFO: wazuh 172.31.16.36 "GET /" with parameters {} and body {} done in 3.181s: 200
2023/12/11 17:53:34 INFO: wazuh 172.31.16.36 "PUT /cluster/restart" with parameters {} and body {} done in 0.121s: 200

It seems that workers took less time to restart than the master, so they couldn't send sendsync commands when they were requested from other services in the workers hosts (probably remoted). Worker1 logs when restarting (17:53:35 - 17:53:46):

2023/12/11 17:53:35 INFO: [Cluster] [Main] SIGNAL [(15)-(SIGTERM)] received. Exit...
2023/12/11 17:53:46 DEBUG: [Cluster] [Main] Removing '/var/ossec/queue/cluster/'.
2023/12/11 17:53:46 DEBUG: [Cluster] [Main] Removed '/var/ossec/queue/cluster/'.
2023/12/11 17:53:46 INFO: [Local Server] [Main] Serving on /var/ossec/queue/cluster/c-internal.sock
2023/12/11 17:53:46 DEBUG: [Local Server] [Keep alive] Calculating.
2023/12/11 17:53:46 DEBUG: [Local Server] [Keep alive] Calculated.
2023/12/11 17:53:46 ERROR: [Local Server] [Main] Could not connect to master. Trying again in 10 seconds.
2023/12/11 17:53:51 DEBUG: [Local 544633] [Main] Connection received in local server.
2023/12/11 17:53:51 ERROR: [Local 544633] [Main] Internal error processing request 'b'sendsync'': Error 3023 - Worker node is not connected to master

Master logs when restarting (17:53:35 - 17:53:47)

2023/12/11 17:53:35 INFO: [Cluster] [Main] SIGNAL [(15)-(SIGTERM)] received. Exit...
2023/12/11 17:53:47 DEBUG: [Cluster] [Main] Removing '/var/ossec/queue/cluster/'.
2023/12/11 17:53:47 DEBUG: [Cluster] [Main] Removed '/var/ossec/queue/cluster/'.
2023/12/11 17:53:47 INFO: [Local Server] [Main] Serving on /var/ossec/queue/cluster/c-internal.sock
2023/12/11 17:53:47 DEBUG: [Local Server] [Keep alive] Calculating.
2023/12/11 17:53:47 DEBUG: [Local Server] [Keep alive] Calculated.
2023/12/11 17:53:47 INFO: [Master] [Main] Serving on ('0.0.0.0', 1516)

The master started listening one second after the workers, so they had already tried to connect unsuccessfully. They wait 10 seconds before trying again, leading to connection errors for any sendsync request received in the meantime.

Reliability

Currently, the reliability tests are not prepared to analyze logs where there has been a restarted node, and this is exactly what happens when launching API performance tests, so it is expected that many of them will fail. We need however to improve those tests so they work in situations like this.

Conclusion

🟢 Everything seems fine, fixes are required mainly in the tests, not in the cluster.

@damarisg
Copy link
Member

Please add a detailed conclusion.

@damarisg
Copy link
Member

LGTM!

@EduLeon12 EduLeon12 mentioned this issue Jan 11, 2024
Closed
1 task
@GGP1 GGP1 mentioned this issue Feb 26, 2024
Closed
2 tasks
This was referenced Mar 13, 2024
Closed
Closed
@fdalmaup fdalmaup mentioned this issue Apr 8, 2024
Closed
2 tasks
@GGP1 GGP1 mentioned this issue Apr 22, 2024
Closed
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@GGP1 GGP1

Labels
level/task type/test
Projects
No open projects
Release 4.8.0
Archived in project
Milestone
No milestone
Development

No branches or pull requests
3 participants
@damarisg @Selutario @GGP1