Add acceptance tests for how provider handles `project` arguments #11607

SarahFrench · 2024-09-02T14:40:19Z

This PR adds acc tests to test how the provider handles the project argument, using the new datasources google_provider_config_plugin_framework and google_provider_config_sdk.

In the existing unit tests there is a test case in the TestFrameworkProvider_LoadAndValidateFramework_project test called "when project is an unknown value, the provider treats it as if it's unset and uses an environment variable instead".

I added a test case called "when project is unknown in the config, environment variables are used" in the new SDK and PF acc tests. However the test case is flakey and has different outcomes on different runs. Because of this I'm going to leave testing unknown handling to unit tests only.

Release Note Template for Downstream PRs (will be copied)

modular-magician · 2024-09-02T14:52:44Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 12 files changed, 1472 insertions(+), 709 deletions(-))
google-beta provider: Diff ( 12 files changed, 1472 insertions(+), 709 deletions(-))

modular-magician · 2024-09-02T16:03:45Z

Tests analytics

Total tests: 3883
Passed tests: 3469
Skipped tests: 410
Affected tests: 4

Click here to see the affected service packages

All service packages are affected

Action taken

Found 4 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

TestAccBackupDRBackupVault_fullUpdate

TestAccComputeInstanceTemplate_withNamePrefix

TestAccProjectIamPolicy_invalidMembers

TestAccSdkProvider_project

Get to know how VCR tests work

modular-magician · 2024-09-02T16:09:36Z

$\textcolor{green}{\textsf{Tests passed during RECORDING mode:}}$
TestAccBackupDRBackupVault_fullUpdate[Debug log]
TestAccComputeInstanceTemplate_withNamePrefix[Debug log]
TestAccProjectIamPolicy_invalidMembers[Debug log]
$\textcolor{red}{\textsf{Tests failed when rerunning REPLAYING mode:}}$
TestAccComputeInstanceTemplate_withNamePrefix[Error message] [Debug log]

Tests failed due to non-determinism or randomness when the VCR replayed the response after the HTTP request was made.

Please fix these to complete your PR. If you believe these test failures to be incorrect or unrelated to your change, or if you have any questions, please raise the concern with your reviewer.

$\textcolor{red}{\textsf{Tests failed during RECORDING mode:}}$
TestAccSdkProvider_project[Error message] [Debug log]

$\textcolor{red}{\textsf{Errors occurred during RECORDING mode. Please fix them to complete your PR.}}$

View the build log or the debug log for each test

SarahFrench · 2024-09-02T18:14:20Z

TestAccFwProvider_project did run, but wasn't reported here. Look in gcs bucket for file

SarahFrench · 2024-09-02T18:20:53Z

This failure is weird, as I can get the original version of the acctest's checks to pass locally. I assume that there's a difference in the CI environment and my laptop that affects the data source being refreshed or not? I assume the plugin-framework version won't be as flaky as unknown handling is more explicit and less of "lucky timing" where a data source refreshes before you try to use its attributes in dependent resources

modular-magician · 2024-09-04T18:44:41Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 12 files changed, 1492 insertions(+), 710 deletions(-))
google-beta provider: Diff ( 12 files changed, 1492 insertions(+), 710 deletions(-))

modular-magician · 2024-09-04T19:52:36Z

Tests analytics

Total tests: 3907
Passed tests: 3494
Skipped tests: 410
Affected tests: 3

Click here to see the affected service packages

All service packages are affected

Action taken

Found 3 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

TestAccBackupDRBackupVault_fullUpdate

TestAccComputeInstanceTemplate_withNamePrefix

TestAccSdkProvider_project

Get to know how VCR tests work

modular-magician · 2024-09-04T19:56:48Z

$\textcolor{green}{\textsf{Tests passed during RECORDING mode:}}$
TestAccBackupDRBackupVault_fullUpdate[Debug log]
TestAccComputeInstanceTemplate_withNamePrefix[Debug log]
$\textcolor{red}{\textsf{Tests failed when rerunning REPLAYING mode:}}$
TestAccComputeInstanceTemplate_withNamePrefix[Error message] [Debug log]

Tests failed due to non-determinism or randomness when the VCR replayed the response after the HTTP request was made.

Please fix these to complete your PR. If you believe these test failures to be incorrect or unrelated to your change, or if you have any questions, please raise the concern with your reviewer.

$\textcolor{red}{\textsf{Tests failed during RECORDING mode:}}$
TestAccSdkProvider_project[Error message] [Debug log]

$\textcolor{red}{\textsf{Errors occurred during RECORDING mode. Please fix them to complete your PR.}}$

View the build log or the debug log for each test

modular-magician · 2024-09-04T21:40:59Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 12 files changed, 1492 insertions(+), 710 deletions(-))
google-beta provider: Diff ( 12 files changed, 1492 insertions(+), 710 deletions(-))

modular-magician · 2024-09-04T22:55:26Z

Tests analytics

Total tests: 3804
Passed tests: 3393
Skipped tests: 410
Affected tests: 1

Click here to see the affected service packages

All service packages are affected

Action taken

Found 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

TestAccComputeInstanceTemplate_withNamePrefix

Get to know how VCR tests work

modular-magician · 2024-09-04T22:58:35Z

$\textcolor{green}{\textsf{Tests passed during RECORDING mode:}}$
TestAccComputeInstanceTemplate_withNamePrefix[Debug log]
$\textcolor{red}{\textsf{Tests failed when rerunning REPLAYING mode:}}$
TestAccComputeInstanceTemplate_withNamePrefix[Error message] [Debug log]

Tests failed due to non-determinism or randomness when the VCR replayed the response after the HTTP request was made.

Please fix these to complete your PR. If you believe these test failures to be incorrect or unrelated to your change, or if you have any questions, please raise the concern with your reviewer.

$\textcolor{green}{\textsf{All tests passed!}}$

View the build log or the debug log for each test

…ameworkProvider_LoadAndValidateFramework_project`

mmv1/third_party/terraform/fwprovider/framework_provider_project_test.go

zli82016 · 2024-09-11T21:08:24Z

mmv1/third_party/terraform/fwtransport/go/framework_config_test.go

@@ -14,168 +14,6 @@ import (
 	transport_tpg "github.com/hashicorp/terraform-provider-google/google/transport"
 )

-func TestFrameworkProvider_LoadAndValidateFramework_project(t *testing.T) {


Do you mind adding this function back? Thanks.

Added in f66e3ad

Ah sorry, this is the go rewrite file - I'll push it back

modular-magician · 2024-09-11T21:27:40Z

Tests analytics

Total tests: 3996
Passed tests: 3584
Skipped tests: 409
Affected tests: 3

Click here to see the affected service packages

All service packages are affected

Action taken

Found 3 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

TestAccBackupDRBackupVault_fullUpdate

TestAccComputeInstance_confidentialHyperDiskBootDisk

TestAccSdkProvider_project

Get to know how VCR tests work

modular-magician · 2024-09-11T21:35:52Z

$\textcolor{green}{\textsf{Tests passed during RECORDING mode:}}$
TestAccBackupDRBackupVault_fullUpdate[Debug log]
TestAccComputeInstance_confidentialHyperDiskBootDisk[Debug log]
$\textcolor{red}{\textsf{Tests failed when rerunning REPLAYING mode:}}$
TestAccComputeInstance_confidentialHyperDiskBootDisk[Error message] [Debug log]

Tests failed due to non-determinism or randomness when the VCR replayed the response after the HTTP request was made.

Please fix these to complete your PR. If you believe these test failures to be incorrect or unrelated to your change, or if you have any questions, please raise the concern with your reviewer.

$\textcolor{red}{\textsf{Tests failed during RECORDING mode:}}$
TestAccSdkProvider_project[Error message] [Debug log]

$\textcolor{red}{\textsf{Errors occurred during RECORDING mode. Please fix them to complete your PR.}}$

View the build log or the debug log for each test

…unit-test level. Add comments to group test cases by type.

modular-magician · 2024-09-12T13:41:23Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 2 files changed, 391 insertions(+))
google-beta provider: Diff ( 2 files changed, 391 insertions(+))

modular-magician · 2024-09-12T14:49:21Z

Tests analytics

Total tests: 3998
Passed tests: 3587
Skipped tests: 409
Affected tests: 2

Click here to see the affected service packages

All service packages are affected

Action taken

Found 2 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

TestAccBackupDRBackupVault_fullUpdate

TestAccFrameworkProviderBasePath_setInvalidBasePath

Get to know how VCR tests work

modular-magician · 2024-09-12T14:52:38Z

$\textcolor{green}{\textsf{Tests passed during RECORDING mode:}}$
TestAccBackupDRBackupVault_fullUpdate[Debug log]

$\textcolor{green}{\textsf{No issues found for passed tests after REPLAYING rerun.}}$

$\textcolor{red}{\textsf{Tests failed during RECORDING mode:}}$
TestAccFrameworkProviderBasePath_setInvalidBasePath[Error message] [Debug log]

$\textcolor{red}{\textsf{Errors occurred during RECORDING mode. Please fix them to complete your PR.}}$

View the build log or the debug log for each test

zli82016

LGTM. The failed test looks like unrelated.

…ersion of framework_config_test.go

SarahFrench · 2024-09-13T20:18:01Z

I realised that I was confused before - once that latest push completes it's automation I'll merge the PR.

Thank you for the review & bearing with me on the weird Unknown behaviour!

modular-magician · 2024-09-13T20:27:57Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 2 files changed, 391 insertions(+))
google-beta provider: Diff ( 2 files changed, 391 insertions(+))

modular-magician · 2024-09-13T21:37:28Z

Tests analytics

Total tests: 4036
Passed tests: 3622
Skipped tests: 411
Affected tests: 3

Click here to see the affected service packages

All service packages are affected

Action taken

Found 3 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

TestAccBackupDRBackupVault_fullUpdate

TestAccComputeInstance_confidentialHyperDiskBootDisk

TestAccFrameworkProviderBasePath_setInvalidBasePath

Get to know how VCR tests work

modular-magician · 2024-09-13T21:45:37Z

$\textcolor{green}{\textsf{Tests passed during RECORDING mode:}}$
TestAccBackupDRBackupVault_fullUpdate[Debug log]
TestAccComputeInstance_confidentialHyperDiskBootDisk[Debug log]

$\textcolor{green}{\textsf{No issues found for passed tests after REPLAYING rerun.}}$

$\textcolor{red}{\textsf{Tests failed during RECORDING mode:}}$
TestAccFrameworkProviderBasePath_setInvalidBasePath[Error message] [Debug log]

$\textcolor{red}{\textsf{Errors occurred during RECORDING mode. Please fix them to complete your PR.}}$

View the build log or the debug log for each test

modular-magician · 2024-09-16T10:12:32Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 2 files changed, 391 insertions(+))
google-beta provider: Diff ( 2 files changed, 391 insertions(+))

modular-magician · 2024-09-16T11:22:22Z

Tests analytics

Total tests: 4043
Passed tests: 3630
Skipped tests: 412
Affected tests: 1

Click here to see the affected service packages

All service packages are affected

Action taken

Found 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

TestAccBackupDRBackupVault_fullUpdate

Get to know how VCR tests work

modular-magician · 2024-09-16T11:25:39Z

$\textcolor{green}{\textsf{Tests passed during RECORDING mode:}}$
TestAccBackupDRBackupVault_fullUpdate[Debug log]

$\textcolor{green}{\textsf{No issues found for passed tests after REPLAYING rerun.}}$

$\textcolor{green}{\textsf{All tests passed!}}$

View the build log or the debug log for each test

SarahFrench · 2024-09-16T11:45:45Z

That resolves the missing test, merging now!

…ogleCloudPlatform#11607)

This comment was marked as outdated.

Sign in to view

SarahFrench force-pushed the mux-refactor-3-replace-project-tests branch from b86f569 to 83a0020 Compare September 4, 2024 17:46

This comment has been minimized.

Sign in to view

SarahFrench added 6 commits September 5, 2024 12:06

Replace plugin-framework tests for project with acceptance tests

18f93b9

Add acc test case for handling unknown values, remove rest of `TestFr…

d02e0b1

…ameworkProvider_LoadAndValidateFramework_project`

Replace SDK tests for project with acceptance tests

6f3f209

Update SDK test to reflect its flakiness

e2b21d2

Add missing var declaration

48680c0

Test is being flaky

d33c692

SarahFrench force-pushed the mux-refactor-3-replace-project-tests branch from 62ce0dd to d33c692 Compare September 5, 2024 11:06

This comment has been minimized.

Sign in to view

SarahFrench commented Sep 5, 2024

View reviewed changes

mmv1/third_party/terraform/fwprovider/framework_provider_project_test.go Show resolved Hide resolved

This comment has been minimized.

Sign in to view

Fix bad rebase (?)

632b8b8

zli82016 reviewed Sep 11, 2024

View reviewed changes

Remove unknown value handling test cases, as these will be tested at …

1b07e9b

…unit-test level. Add comments to group test cases by type.

zli82016 approved these changes Sep 12, 2024

View reviewed changes

Return TestFrameworkProvider_LoadAndValidateFramework_project to Go v…

65caffd

…ersion of framework_config_test.go

Re-add missing test cases to Go rewrite file

4ab85df

SarahFrench merged commit 0bfab18 into GoogleCloudPlatform:main Sep 16, 2024
10 checks passed

SarahFrench mentioned this pull request Sep 16, 2024

Re-add SDK tests TestProvider_ProviderConfigure_region, TestProvider_ProviderConfigure_zone, remove unused config from acc tests too #11719

Merged

abd-goog pushed a commit to abd-goog/magic-modules that referenced this pull request Sep 23, 2024

Add acceptance tests for how provider handles project arguments (Go…

5138a35

…ogleCloudPlatform#11607)

niharika-98 pushed a commit to niharika-98/magic-modules that referenced this pull request Oct 7, 2024

Add acceptance tests for how provider handles project arguments (Go…

ce729d6

…ogleCloudPlatform#11607)

Add acceptance tests for how provider handles project arguments #11607

Add acceptance tests for how provider handles project arguments #11607

Conversation

SarahFrench commented Sep 2, 2024 • edited Loading

modular-magician commented Sep 2, 2024

Diff report

modular-magician commented Sep 2, 2024

Tests analytics

Action taken

modular-magician commented Sep 2, 2024

SarahFrench commented Sep 2, 2024 • edited Loading

SarahFrench commented Sep 2, 2024

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment has been minimized.

modular-magician commented Sep 4, 2024

Diff report

This comment has been minimized.

This comment has been minimized.

modular-magician commented Sep 4, 2024

Tests analytics

Action taken

modular-magician commented Sep 4, 2024

modular-magician commented Sep 4, 2024

Diff report

modular-magician commented Sep 4, 2024

Tests analytics

Action taken

modular-magician commented Sep 4, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

zli82016 Sep 11, 2024

Choose a reason for hiding this comment

SarahFrench Sep 12, 2024

Choose a reason for hiding this comment

SarahFrench Sep 13, 2024

Choose a reason for hiding this comment

modular-magician commented Sep 11, 2024

Tests analytics

Action taken

modular-magician commented Sep 11, 2024

modular-magician commented Sep 12, 2024

Diff report

modular-magician commented Sep 12, 2024

Tests analytics

Action taken

modular-magician commented Sep 12, 2024

zli82016 left a comment

Choose a reason for hiding this comment

SarahFrench commented Sep 13, 2024

modular-magician commented Sep 13, 2024

Diff report

modular-magician commented Sep 13, 2024

Tests analytics

Action taken

modular-magician commented Sep 13, 2024

modular-magician commented Sep 16, 2024

Diff report

modular-magician commented Sep 16, 2024

Tests analytics

Action taken

modular-magician commented Sep 16, 2024

SarahFrench commented Sep 16, 2024

Add acceptance tests for how provider handles `project` arguments #11607

Add acceptance tests for how provider handles `project` arguments #11607

SarahFrench commented Sep 2, 2024 •

edited

Loading

SarahFrench commented Sep 2, 2024 •

edited

Loading