Add new flags to just setup, teardown or run tests only #1250

mrodm · 2023-05-05T09:14:42Z

Closes #582

This PR is intended to add the required subcommands and/or parameters to allow just run the setup, no-provision or teardown of tests. Focused on system tests.

Added flags:

--setup
- Creates all the resources , install the package, enrolls the agent
- It ensures that some docs are stored in ES
- Writes into ServiceSetupDir folder the original policy assigned to the agent as well as the policy created by elastic-package
--no-provision
- Reads both policies from ServiceSetupDir
- Install the package (in case there are changes locally)
- Executes all the system tests
- Does not remove the logs folder to avoid issues in service writing to those files (they should be truncated instead)
--tear-down
- Reads both policies from ServiceSetupDir
- Runs all the handlers to remove all resources created
- Stop the service

Pending:

Add creation of a new folder under the current profile directory for test resources
Store needed resources between Setup, NoProvision and Teardown in a folder under the current profile (.e.g. `~/.elastic-package/profiles/default/testing/*). Examples of resources to be stored:
- Original policy assigned before changing it in the setup method, so TearDown step can restore it
```
 $ ls -l ~/.elastic-package/profiles/default/stack/state/
total 4
-rw-r--r-- 1 user user 773 feb 14 17:15 service.json
```
Add methods and CLI parameters to Setup and TearDown for just one config and variant in system test
- Those new parameters could be added just
  - if a new method in TestRunner interface like CanRunSetupTeardownIndependent() or CanRunSeparateTests returns true
  - if the runner implements a new interface (e.g. TestRunnerSetterUp)
```
type TestRunnerSetterUp interface {
 	// Setup installs package and starts the service
 	Configure(TestOptions) error

	// Setup installs package and starts the service
	Setup(TestOptions) error
}
```
Ensure this works also for input packages - tested with sql_input
CI step
- Build step link: https://buildkite.com/elastic/elastic-package/builds/2262#018d6a96-5e74-4e00-8633-e25536c5a1d1/476
Add new section in docs:
- Rendered version of the docs: https://github.com/mrodm/elastic-package/blob/add_flag_setup_testrunner/docs/howto/system_testing.md#running-system-tests-without-cleanup-technical-preview

Checklist

Test with an integration package (e.g. nginx - integrations)
Test setting a non-default variant (e.g. mysql percona_8_0_13 - integrations)
Test with an input package (e.g. sql_input - integrations)
Test with a package using custom agents (e.g. auditd_manager - elastic-package)
Test with a package using kind as a service deployer (e.g. kubernetes - elastic-package)

How to test locally

cd /path/integrations/packages/mysql

# Start Elastic stack as usual
elastic-package stack down -v
elastic-package stack up -v -d

# Go to a package from integrations repository with variants, taken as example `mysql`
elastic-package test system -v --config-file data_stream/status/_dev/test/system/test-default-config.yml --variant percona_8_0_36 --setup
elastic-package test system -v --no-provision
elastic-package test system -v --tear-down

# Go to a package from integrations repository without variants, taken as example `hashicorp_vault`
elastic-package test system -v --config-file data_stream/audit/_dev/test/system/test-tcp-config.yml --setup

# this command can be executed several times
elastic-package test system -v --no-provision

elastic-package test system -v --tear-down


# Go to an input package, taken as example `sql_input`
elastic-package test system -v --config-file _dev/test/system/test-mysql-config.yml --setup

# this command can be executed several times
elastic-package test system -v --no-provision

elastic-package test system -v --tear-down

elastic-package stack down -v

sharbuz · 2023-07-10T14:51:26Z

Hi everyone,
Please update the branch because we merged an important change to the main branch: #1347

Refactor runTest to extract a method with all the information about the scenario (policies, service, component templates, etc.)

mrodm · 2024-02-15T08:47:13Z

internal/servicedeployer/factory.go

+		if options.RunSetup || options.RunTearDown || options.RunTestsOnly {
+			return nil, errors.New("terraform service deployer not supported to run by steps")
+		}


For now, this service deployer is not supported. I'd prefer to just leave this option (test flags) for stacks/services started locally for now.

This the error that is going to be shown to the user:

2024/02/15 09:44:36 DEBUG setting up service... Error: error running package system tests: could not complete test run: could not create service runner: terraform service deployer not supported to run by steps

👍 we can add this later

jsoriano

I think we can ship it ⛵
Thanks!

jsoriano · 2024-02-15T10:04:08Z

internal/servicedeployer/factory.go

+		if options.RunSetup || options.RunTearDown || options.RunTestsOnly {
+			return nil, errors.New("terraform service deployer not supported to run by steps")
+		}


👍 we can add this later

jsoriano · 2024-02-15T10:05:49Z

internal/testrunner/runners/system/runner.go

+	dirPath := filepath.Dir(r.serviceStateFilePath)
 	err := os.RemoveAll(dirPath)


Maybe we should remove the file only, in case we use the directory later for other states. Leaving the directory there should not be a problem.

That makes totally sense.
Update this function to just delete the file, and update the tests accordingly.

jsoriano · 2024-02-15T10:35:42Z

cmd/testrunner.go

+			// variant flag should not be used with tear-down or no-provision flags
+			// cannot be defined here using MarkFlagsMutuallyExclusive as in --config-file


This comment seems a bit lonely around here, not clear to what it refers to. Or is another mutual exclusion missing here?

I wanted to add a note that I could not set the mutually exclusive using testTypeCmd.MarkFlagsMutuallyExclusive. Variant flag is defined in the cmd variable and not in the testTypeCmd.

This is managed afterwards in the code:
https://github.com/elastic/elastic-package/pull/1250/files#diff-33701d098205d3537f071235876d903ddbfa851c492917e49852c9212bc1d4fdR316-R320

I'll update this comment with that reference

elasticmachine · 2024-02-15T11:45:33Z

💚 Build Succeeded

Buildkite Build
Commit: b22f48a

History

💚 Build #2404 succeeded 7c86cf2
💔 Build #2401 failed 5264086
💚 Build #2400 succeeded 14e1e2f
💚 Build #2397 succeeded ee91de4
💔 Build #2396 failed 40c5c87
💚 Build #2392 succeeded bd8acba

cc @mrodm

mrodm added 5 commits January 24, 2024 12:17

Add Setup method to interface

20fbf41

Add setup for system

adad2d6

Add flag for config file

d1af618

Fix field name

63ecb86

Update runners

684ee9b

mrodm force-pushed the add_flag_setup_testrunner branch from 0050990 to 684ee9b Compare January 24, 2024 11:57

mrodm added 23 commits January 24, 2024 16:01

Execute Configure/Setup if it is implemented interface

628216b

Add new flags for setup / teardown

681f8af

Refactor runTest to extract a method with all the information about the scenario (policies, service, component templates, etc.)

Add CanRunSetupTeardownIndependent method to the interface

da5272a

Refactor - add initRun

2345470

Use variables from runner struct

d86240f

Fix data stream root path

cf23c5d

Add specific policy suffix when running setup flag

5add400

Add first PoC for setup and teardown

58dde46

Add check about variants

17542ef

Remove prefix from policy - those policies are saved after each setup

84ea79a

Do not delete old docs in tear-down - just in the wipe handler

6ad085f

If any error in setup, run teardown

2b6027f

Use constants

eb23780

Ensure two setup commands are run sequentially

f0d955c

Update folder path for service setup files

e0f2b39

Merge remote upstream/main into add_flag_setup_testrunner

9e0b394

Not hide errors from prepare scenario

029ad0b

Show wait for data timeout in message

e6e0a7a

Review log messages

13f0c97

Manage errors about setup service directory for setup and teardown

d2f3498

Remove commented code

7065d57

Use the expected config file if set

87c56f5

Just create folder and files if setup flag is set

3e5cb14

mrodm added 18 commits February 14, 2024 14:21

Move to function to read config file path string

32ae291

Change switch by ifs

9856557

Add more checks for config-file and data-streams flags

1393831

Raise error if variant is not defined

7fcad77

Add functions to manage serviceSetupDir

4934517

Use function to read setup data

e4e5b81

Move state file to profile (stack) folder

827c7d7

Rename variables to refer to state

de6acb3

Add doc for StateFolderPath

baad699

Merge remote upstream/main into add_flag_setup_testrunner

90b8e79

Update path in test script

bd8acba

Update script with packages using custom agents

40c5c87

Remove state folder field from struct

f57a653

Fix test script with package folder paths

ee91de4

Raise error for unsupported service deployers

14e1e2f

Add support for kind service deployer

5264086

Fix function name

7d4ca6a

Install kind and kubectl in system test flags step

d7818bd

mrodm commented Feb 15, 2024

View reviewed changes

Fix function name

7c86cf2

mrodm requested a review from jsoriano February 15, 2024 10:15

jsoriano approved these changes Feb 15, 2024

View reviewed changes

mrodm added 2 commits February 15, 2024 12:04

Not delete state folder, just service state

7c43ec9

Add note to comment about variant flag

b22f48a

jsoriano approved these changes Feb 15, 2024

View reviewed changes

mrodm merged commit 81289fa into elastic:main Feb 15, 2024
3 checks passed

mrodm deleted the add_flag_setup_testrunner branch February 15, 2024 12:09

This was referenced Feb 15, 2024

[Feature Request] Adding flag to run system tests without cleanup #582

Closed

Remove leftovers after changing path for service state #1689

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add new flags to just setup, teardown or run tests only #1250

Add new flags to just setup, teardown or run tests only #1250

mrodm commented May 5, 2023 •

edited

Loading

sharbuz commented Jul 10, 2023

mrodm Feb 15, 2024

jsoriano Feb 15, 2024

jsoriano left a comment

jsoriano Feb 15, 2024

jsoriano Feb 15, 2024

mrodm Feb 15, 2024

jsoriano Feb 15, 2024

mrodm Feb 15, 2024

elasticmachine commented Feb 15, 2024

		dirPath := filepath.Dir(r.serviceStateFilePath)
		err := os.RemoveAll(dirPath)

		// variant flag should not be used with tear-down or no-provision flags
		// cannot be defined here using MarkFlagsMutuallyExclusive as in --config-file

Add new flags to just setup, teardown or run tests only #1250

Add new flags to just setup, teardown or run tests only #1250

Conversation

mrodm commented May 5, 2023 • edited Loading

Checklist

How to test locally

sharbuz commented Jul 10, 2023

mrodm Feb 15, 2024

Choose a reason for hiding this comment

jsoriano Feb 15, 2024

Choose a reason for hiding this comment

jsoriano left a comment

Choose a reason for hiding this comment

jsoriano Feb 15, 2024

Choose a reason for hiding this comment

jsoriano Feb 15, 2024

Choose a reason for hiding this comment

mrodm Feb 15, 2024

Choose a reason for hiding this comment

jsoriano Feb 15, 2024

Choose a reason for hiding this comment

mrodm Feb 15, 2024

Choose a reason for hiding this comment

elasticmachine commented Feb 15, 2024

💚 Build Succeeded

History

mrodm commented May 5, 2023 •

edited

Loading