From 98cea309793ef40a973927ba9ef70481166f43b0 Mon Sep 17 00:00:00 2001 From: Dirk Avery Date: Fri, 14 Jan 2022 10:44:26 -0500 Subject: [PATCH 1/5] docs: Long-running test guards --- .../running-and-writing-acceptance-tests.md | 61 +++++++++++++++++++ 1 file changed, 61 insertions(+) diff --git a/docs/contributing/running-and-writing-acceptance-tests.md b/docs/contributing/running-and-writing-acceptance-tests.md index fff5d0eeb65..6fa3b65a1d1 100644 --- a/docs/contributing/running-and-writing-acceptance-tests.md +++ b/docs/contributing/running-and-writing-acceptance-tests.md @@ -4,6 +4,7 @@ - [Running an Acceptance Test](#running-an-acceptance-test) - [Running Cross-Account Tests](#running-cross-account-tests) - [Running Cross-Region Tests](#running-cross-region-tests) + - [Running Only Short Tests](#running-only-short-tests) - [Writing an Acceptance Test](#writing-an-acceptance-test) - [Anatomy of an Acceptance Test](#anatomy-of-an-acceptance-test) - [Resource Acceptance Testing](#resource-acceptance-testing) @@ -20,6 +21,7 @@ - [ErrorChecks](#errorchecks) - [Common ErrorCheck](#common-errorcheck) - [Service-Specific ErrorChecks](#service-specific-errorchecks) + - [Long-Running Test Guards](#long-running-test-guards) - [Disappears Acceptance Tests](#disappears-acceptance-tests) - [Per Attribute Acceptance Tests](#per-attribute-acceptance-tests) - [Cross-Account Acceptance Tests](#cross-account-acceptance-tests) @@ -173,6 +175,26 @@ export AWS_ALTERNATE_REGION=... export AWS_THIRD_REGION=... ``` +### Running Only Short Tests + +Some tests have been manually marked as long-running and can be skipped using the `-short` flag. However, implementation of the long-running guards is a work in progress and many services have no tests guarded. + +Where guards have been implemented, do not always skip long-running tests. However, for intermediate test runs during development, or to verify functionality unrelated to the specific long-running tests, skipping long-running tests makes work more efficient. We recommend that for the final test run before submitting a PR that you run all tests without the `-short` flag. + +If you want to only run short-running tests, you can use either one of these equivalent statements. Note the use of `-short`. + +For example: + +```console +% make testacc TESTS='TestAccECSTaskDefinition_' PKG=ecs TESTARGS=-short +``` + +Or: + +```console +% TF_ACC=1 go test ./internal/service/ecs/... -v -count 1 -parallel 20 -run='TestAccECSTaskDefinition_' -short -timeout 180m +``` + ## Writing an Acceptance Test Terraform has a framework for writing acceptance tests which minimises the @@ -446,6 +468,22 @@ resource "aws_example_thing" "test" { Typically the `rName` is always the first argument to the test configuration function, if used, for consistency. +Note that if `rName` is used multiple times in the `fmt.Sprintf()` statement, _do not_ repeat `rName` in the `fmt.Sprintf()` arguments. Using `fmt.Sprintf(..., rName, rName)`, for example, would not be correct. Instead, use the indexed `%[1]q` verb multiple times. For example: + +```go +func testAccExampleThingConfigName(rName string) string { + return fmt.Sprintf(` +resource "aws_example_thing" "test" { + name = %[1]q + + tags = { + Name = %[1]q + } +} +`, rName) +} +``` + #### Other Recommended Variables We also typically recommend saving a `resourceName` variable in the test that contains the resource reference, e.g., `aws_example_thing.test`, which is repeatedly used in the checks. @@ -665,6 +703,29 @@ func testAccErrorCheckSkipService(t *testing.T) resource.ErrorCheckFunc { } ``` +#### Long-Running Test Guards + +For any acceptance tests that typically run longer than 300 seconds (5 minutes), add a `-short` test guard before the `resource.ParallelTest()` (or `resource.Test()`) statement. + +For example: + +```go +func TestAccExampleThing_longRunningTest(t *testing.T) { + rName := sdkacctest.RandomWithPrefix(acctest.ResourcePrefix) + resourceName := "aws_example_thing.test" + + if testing.Short() { + t.Skip("skipping long-running test in short mode") + } + + resource.ParallelTest(t, resource.TestCase{ + // ... omitted for brevity ... + }) +} +``` + +When running acceptances tests, tests with these guards can be skipped using the Go `-short` flag. See [Running Only Short Tests](#running-only-short-tests) for examples. + #### Disappears Acceptance Tests This test is generally implemented second. It is straightforward to setup once the basic test is passing since it can reuse that test configuration. It prevents a common bug report with Terraform resources that error when they can not be found (e.g., deleted outside Terraform). From f6e564c7d2efa55f336bc40300a5457d9c06e09b Mon Sep 17 00:00:00 2001 From: Dirk Avery Date: Fri, 14 Jan 2022 10:45:58 -0500 Subject: [PATCH 2/5] Update language --- docs/contributing/running-and-writing-acceptance-tests.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing/running-and-writing-acceptance-tests.md b/docs/contributing/running-and-writing-acceptance-tests.md index 6fa3b65a1d1..8b7f3c6e452 100644 --- a/docs/contributing/running-and-writing-acceptance-tests.md +++ b/docs/contributing/running-and-writing-acceptance-tests.md @@ -179,7 +179,7 @@ export AWS_THIRD_REGION=... Some tests have been manually marked as long-running and can be skipped using the `-short` flag. However, implementation of the long-running guards is a work in progress and many services have no tests guarded. -Where guards have been implemented, do not always skip long-running tests. However, for intermediate test runs during development, or to verify functionality unrelated to the specific long-running tests, skipping long-running tests makes work more efficient. We recommend that for the final test run before submitting a PR that you run all tests without the `-short` flag. +Where guards have been implemented, do not always skip long-running tests. However, for intermediate test runs during development, or to verify functionality unrelated to the specific long-running tests, skipping long-running tests makes work more efficient. We recommend that for the final test run before submitting a PR that you run affected tests without the `-short` flag. If you want to only run short-running tests, you can use either one of these equivalent statements. Note the use of `-short`. From a970cbf029248966a5d09e6fea85300c3eee709a Mon Sep 17 00:00:00 2001 From: Dirk Avery Date: Fri, 14 Jan 2022 10:46:34 -0500 Subject: [PATCH 3/5] Update language --- docs/contributing/running-and-writing-acceptance-tests.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing/running-and-writing-acceptance-tests.md b/docs/contributing/running-and-writing-acceptance-tests.md index 8b7f3c6e452..b9ae830ee5d 100644 --- a/docs/contributing/running-and-writing-acceptance-tests.md +++ b/docs/contributing/running-and-writing-acceptance-tests.md @@ -181,7 +181,7 @@ Some tests have been manually marked as long-running and can be skipped using th Where guards have been implemented, do not always skip long-running tests. However, for intermediate test runs during development, or to verify functionality unrelated to the specific long-running tests, skipping long-running tests makes work more efficient. We recommend that for the final test run before submitting a PR that you run affected tests without the `-short` flag. -If you want to only run short-running tests, you can use either one of these equivalent statements. Note the use of `-short`. +If you want to run only short-running tests, you can use either one of these equivalent statements. Note the use of `-short`. For example: From 6350ae268b9c425ee7278d238026c138b5528ec7 Mon Sep 17 00:00:00 2001 From: Dirk Avery Date: Fri, 14 Jan 2022 10:48:18 -0500 Subject: [PATCH 4/5] Update language --- docs/contributing/running-and-writing-acceptance-tests.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing/running-and-writing-acceptance-tests.md b/docs/contributing/running-and-writing-acceptance-tests.md index b9ae830ee5d..f357af28cd8 100644 --- a/docs/contributing/running-and-writing-acceptance-tests.md +++ b/docs/contributing/running-and-writing-acceptance-tests.md @@ -177,7 +177,7 @@ export AWS_THIRD_REGION=... ### Running Only Short Tests -Some tests have been manually marked as long-running and can be skipped using the `-short` flag. However, implementation of the long-running guards is a work in progress and many services have no tests guarded. +Some tests have been manually marked as long-running (longer than 300 seconds) and can be skipped using the `-short` flag. However, implementation of the long-running guards is a work in progress and many services have no tests guarded. Where guards have been implemented, do not always skip long-running tests. However, for intermediate test runs during development, or to verify functionality unrelated to the specific long-running tests, skipping long-running tests makes work more efficient. We recommend that for the final test run before submitting a PR that you run affected tests without the `-short` flag. From fced8feb5b6dd7cd51e7aedd1d547bdbd2dd7228 Mon Sep 17 00:00:00 2001 From: Dirk Avery Date: Fri, 14 Jan 2022 10:51:58 -0500 Subject: [PATCH 5/5] Update language --- docs/contributing/running-and-writing-acceptance-tests.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing/running-and-writing-acceptance-tests.md b/docs/contributing/running-and-writing-acceptance-tests.md index f357af28cd8..f6d0a4d8b31 100644 --- a/docs/contributing/running-and-writing-acceptance-tests.md +++ b/docs/contributing/running-and-writing-acceptance-tests.md @@ -468,7 +468,7 @@ resource "aws_example_thing" "test" { Typically the `rName` is always the first argument to the test configuration function, if used, for consistency. -Note that if `rName` is used multiple times in the `fmt.Sprintf()` statement, _do not_ repeat `rName` in the `fmt.Sprintf()` arguments. Using `fmt.Sprintf(..., rName, rName)`, for example, would not be correct. Instead, use the indexed `%[1]q` verb multiple times. For example: +Note that if `rName` (or any other variable) is used multiple times in the `fmt.Sprintf()` statement, _do not_ repeat `rName` in the `fmt.Sprintf()` arguments. Using `fmt.Sprintf(..., rName, rName)`, for example, would not be correct. Instead, use the indexed `%[1]q` (or `%[x]q`, `%[x]s`, `%[x]t`, or `%[x]d`, where `x` represents the index number) verb multiple times. For example: ```go func testAccExampleThingConfigName(rName string) string {