Release 4.8.0 - Beta 4 - System tests

[Rebits]

The following issue aims to run all system tests for the current release candidate, report the results, and open new issues for any encountered errors.

System tests information

Main release candidate issue	#22457
Version	4.8.0
Release candidate #	Beta 4
Tag	https://github.com/wazuh/wazuh-qa/tree/v4.8.0-beta4
Previous system tests issue	#22123

Instructions

To run tests in an AWS EC2 virtual environment, the following requirements will need to be met:

Environment	EC2
Basic_cluster	Ubuntu 22.04.2 LTS C5.XLarge 15GB SSD
Big_cluster_40_agents	Ubuntu 22.04.2 LTS T3.Large 60GB SSD
Agentless_cluster	Ubuntu 22.04.2 LTS T3.Large 30GB SSD
Four_manager_disconnected_node	Ubuntu 22.04.2 LTS T3.Large 30GB SSD
One_manager_agent	Ubuntu 22.04.2 LTS T3.Large 30GB SSD
Manager_agent	Ubuntu 22.04.2 LTS T3.Large 30GB SSD
Enrollment_cluster	Ubuntu 22.04.2 LTS T3.Large 30GB SSD
Basic_environment	Ubuntu 22.04.2 LTS T3.Large 30GB SSD

These requirements should be requested to the @wazuh/cicd team via https://github.com/wazuh/internal-devel-requests.

• Instances requested in https://github.com/wazuh/internal-devel-requests/issues/972

For further information, check https://github.com/wazuh/wazuh-qa/tree/master/tests/system/README.md

Test report procedure

All individual test checks must be marked as:

Pass	The test ran successfully.
Xfail	The test was expected to fail and it failed. It must be properly justified and reported in an issue.
Skip	The test was not run. It must be properly justified and reported in an issue.
Fail	The test failed. A new issue must be opened to evaluate and address the problem.

All test results must have one the following statuses:

🟢	All checks passed.
🔴	There is at least one failed check.
🟡	There is at least one expected fail or skipped test and no failures.

Any failing test must be properly addressed with a new issue, detailing the error and the possible cause. It must be included in the Fixes section of the current release candidate main issue.

Any expected fail or skipped test must have an issue justifying the reason. All auditors must validate the justification for an expected fail or skipped test.

An extended report of the test results must be attached as a zip or txt. This report can be used by the auditors to dig deeper into any possible failures and details.

Conclusion 🔴

• System one_manager_agent false positive in 4.8.0-beta4 wazuh-qa#5099
• System agentless_cluster tests fail in 4.8.0-beta4 wazuh-qa#5100
• System enrollment_cluster tests fail in 4.8.0-beta4 wazuh-qa#5101
• System basic_environment tests fail in 4.8.0-beta4 wazuh-qa#5102

Auditors validation

The definition of done for this one is the validation of the conclusions and the test results from all auditors.

All checks from below must be accepted in order to close this issue.

• @juliamagan

[Rebits]

Currently launching basic_environment, one_manager_agent, manager_agent and enrollment_cluster tests.
Results will be uploaded 13/03/2024

cd provisioning/basic_environment
ansible-playbook -i inventory.yml playbook.yml --extra-vars='{"package_repository":"packages-dev", "repository": "pre-release", "package_version": "4.8.0", "package_revision": "1", "wazuh_qa_branch":"v4.8.0-beta4"}'
cd ../..
python3 -m pytest -v -m basic_environment_env --html=basic_env.html

cd provisioning/one_manager_agent
ansible-playbook -i inventory.yml playbook.yml --extra-vars='{"package_repository":"packages-dev", "repository": "pre-release", "package_version": "4.8.0", "package_revision": "1", "wazuh_qa_branch":"v4.8.0-beta4"}'
cd ../..
python3 -m pytest -v -m one_manager_agent_env --html=one_manager_agent.html

cd provisioning/enrollment_cluster
ansible-playbook -i inventory.yml playbook.yml --extra-vars='{"package_repository":"packages-dev", "repository": "pre-release", "package_version": "4.8.0", "package_revision": "1", "wazuh_qa_branch":"v4.8.0-beta4"}'
cd ../..
python3 -m pytest -v -m enrollment_cluster_env --html=enrollment_cluster.html

cd provisioning/manager_agent
ansible-playbook -i inventory.yml playbook.yml --extra-vars='{"package_repository":"packages-dev", "repository": "pre-release", "package_version": "4.8.0", "package_revision": "1", "wazuh_qa_branch":"v4.8.0-beta4"}'
cd ../..
python3 -m pytest -v -m manager_agent_env --html=manager_agent.html

[Rebits]

Test results 🔴

Reports

one_manager_agent.zip
four_manager_disconnected.zip
agentless_cluster.zip
enrollment_cluster.zip
manager_agent.zip
basic_cluster.zip

![NOTE]
test_agent_groups_forced_change unstable due to lack of resources.

basic_environment.zip
big_cluster.zip

---

	Test name	Status	Pass	XFail/Skip	Fail	Issues Ref.
	One Manager Agent
1	test_cluster/test_correct_merged_file_generation/test_correct_merged_file_generation.py	🟢	12	0	0
2	test_fim/test_files/test_files_cud.py	🔴	8	0	1	wazuh/wazuh-qa#5099
3	test_fim/test_synchronization/test_synchronization.py	🟢	6	0	6
	Four Manager Disconnected Node
4	test_cluster/test_agent_groups/test_agent_groups_new_cluster_node.py	🟢	2	0	0
	Agentless Cluster
5	test_cluster/test_integrity_sync/test_integrity_sync.py	🔴	3	0	1	wazuh/wazuh-qa#5100
6	test_cluster/test_ruleset_sync_status/test_ruleset_sync_status.py	🟢	1	0	0
7	test_jwt_invalidation/test_change_rbac_mode.py	🟢	4	0	0
8	test_jwt_invalidation/test_change_security_resources.py	🟢	4	0	0
9	test_jwt_invalidation/test_disconnected_nodes.py	🟢	1	0	0
10	test_jwt_invalidation/test_revoke_endpoint.py	🟢	6	0	0
11	test_jwt_invalidation/test_update_password.py	🟢	4	0	0
	Enrollment Cluster
12	test_cluster/test_agent_enrollment/test_agent_enrollment.py	🟢	1	0
13	test_cluster/test_agent_groups/test_agent_default_group_added.py	🟢	2	0	0
14	test_cluster/test_agent_groups/test_assign_agent_group_with_enrollment.py	🟢	1	0	0
15	test_cluster/test_agent_groups/test_assign_agent_never_connected_to_group.py	🟢	2	0	0
16	test_cluster/test_agent_groups/test_assign_agent_to_a_group_api.py	🟢	4	0	0
17	test_cluster/test_agent_groups/test_assign_agent_to_a_group_by_tool.py	🟢	2	0	0
18	test_cluster/test_agent_groups/test_assign_agent_to_a_group.py	🟢	4	0	0
19	test_cluster/test_agent_groups/test_assign_groups_guess.py	🔴	11	0	1	wazuh/wazuh-qa#5101
20	test_cluster/test_agent_groups/test_agent_groups.py	🟢	1	0	0
21	test_cluster/test_agent_groups/test_remove_group.py	🔴	9	0	1	wazuh/wazuh-qa#5101
22	test_cluster/test_agent_groups/test_group_sync_status.py	🔴	23	0	4	wazuh/wazuh-qa#5101
23	test_cluster/test_agent_groups/test_group_hash.py	🟢	8	0	0
	Manager Agent
24	test_active_response_log_format/test_active_response_log_format.py	🟢	3	0	0
	Basic Cluster
25	test_cluster/test_agent_info_sync/test_agent_info_sync.py	🟢	2	0	0
26	test_cluster/test_agent_key_polling/test_agent_key_polling.py	🟢	1	0	0
27	test_cluster/test_agent_files_deletion/test_agent_files_deletion.py	🟢	1	0	0
28	test_multigroups/test_multigroups.py	🟢	4	0	0
29	test_cluster/test_agent_groups/test_agent_groups_forced_change.py	🟢	3	0	0
	Big Cluster 40 Agents
30	test_cluster/test_agent_groups/test_groups_sync_default.py	🟢	2	0	0
31	test_cluster/test_agent_groups/test_groups_sync_time.py	🟢	2	0	0
32	test_shutdown_message/test_shutdown_message.py	🟢	1	0	0
	Basic Environment
33	test_agent_auth/test_agent_auth.py	🔴	15	0	5	wazuh/wazuh-qa#5102
34	test_enrollment/test_enrollment.py	🔴	4	10	5	wazuh/wazuh-qa#5102

Conclusion 🔴

• Reported Issues:
  • System one_manager_agent false positive in 4.8.0-beta4 wazuh-qa#5099
  • System agentless_cluster tests fail in 4.8.0-beta4 wazuh-qa#5100
  • System enrollment_cluster tests fail in 4.8.0-beta4 wazuh-qa#5101
  • System basic_environment tests fail in 4.8.0-beta4 wazuh-qa#5102

[MARCOSD4]

GJ! Some review notes:

• Missing link to download the report for one_manager_agent.zip
• The links of the new issues are missing in the Issues Ref. column of the report.
• the basic_environment.zip file does not contain the assets directory, although it does appear in the file attached to the issue System basic_environment tests fail in 4.8.0-beta4 wazuh-qa#5102.

[MARCOSD4]

LGTM

[juliamagan]

LGTM