Finish missing sources from Datastream Stream #7115

iperetz-goo · 2023-01-12T12:42:53Z

Add missing fields and behavior from recently implemented Datastream Stream resource PR link

Changes:

Add Postgres & Oracle source configs
Call an additional update after create stream if the desired_state was RUNNING.
Add support for validation results in operation errors

If this PR is for Terraform, I acknowledge that I have:

Searched through the issue tracker for an open issue that this either resolves or contributes to, commented on it to claim it, and written "fixes {url}" or "part of {url}" in this PR description. If there were no relevant open issues, I opened one and commented that I would like to work on it (not necessary for very small changes).
Generated Terraform, and ran make test and make lint to ensure it passes unit and linter tests.
Ensured that all new fields I added that can be set by a user appear in at least one example (for generated resources) or third_party test (for handwritten resources or update tests).
Ran relevant acceptance tests (If the acceptance tests do not yet pass or you are unable to run them, please let your reviewer know).
Read the Release Notes Guide before writing my release note below.

Release Note Template for Downstream PRs (will be copied)

modular-magician · 2023-01-12T12:43:08Z

Oops! It looks like you're using an unknown release-note type in your changelog entries:

REPLACEME

Please only use the types listed in https://github.com/GoogleCloudPlatform/magic-modules/blob/master/.ci/RELEASE_NOTES_GUIDE.md.

modular-magician · 2023-01-12T12:43:58Z

Hello! I am a robot who works on Magic Modules PRs.

I've detected that you're a community contributor. @c2thorn, a repository maintainer, has been assigned to assist you and help review your changes.

❓ First time contributing? Click here for more details

Your assigned reviewer will help review your code by:

Ensuring it's backwards compatible, covers common error cases, etc.
Summarizing the change into a user-facing changelog note.
Passes tests, either our "VCR" suite, a set of presubmit tests, or with manual test runs.

You can help make sure that review is quick by running local tests and ensuring they're passing in between each push you make to your PR's branch. Also, try to leave a comment with each push you make, as pushes generally don't generate emails.

If your reviewer doesn't get back to you within a week after your most recent change, please feel free to leave a comment on the issue asking them to take a look! In the absence of a dedicated review dashboard most maintainers manage their pending reviews through email, and those will sometimes get lost in their inbox.

modular-magician · 2023-01-12T12:58:54Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 2 files changed, 2072 insertions(+), 337 deletions(-))
Terraform Beta: Diff ( 2 files changed, 2072 insertions(+), 337 deletions(-))
TF Validator: Diff ( 3 files changed, 645 insertions(+), 3 deletions(-))

modular-magician · 2023-01-12T13:47:48Z

Tests analytics

Total tests: 2406
Passed tests 2154
Skipped tests: 251
Failed tests: 1

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests

TestAccContainerCluster_withInvalidGatewayApiConfigChannel

modular-magician · 2023-01-12T13:48:04Z

Tests passed during RECORDING mode:
TestAccContainerCluster_withInvalidGatewayApiConfigChannel[Debug log]

All tests passed
View the build log or the debug log for each test

modular-magician · 2023-01-12T15:05:56Z

Oops! It looks like you're using an unknown release-note type in your changelog entries:

REPLACEME

Please only use the types listed in https://github.com/GoogleCloudPlatform/magic-modules/blob/master/.ci/RELEASE_NOTES_GUIDE.md.

modular-magician · 2023-01-12T15:25:06Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 2 files changed, 4988 insertions(+), 1386 deletions(-))
Terraform Beta: Diff ( 2 files changed, 4988 insertions(+), 1386 deletions(-))
TF Validator: Diff ( 3 files changed, 1541 insertions(+), 191 deletions(-))

modular-magician · 2023-01-12T16:13:11Z

Tests analytics

Total tests: 2406
Passed tests 2153
Skipped tests: 251
Failed tests: 2

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests

TestAccFirebaserulesRelease_BasicRelease|TestAccContainerCluster_withInvalidGatewayApiConfigChannel

modular-magician · 2023-01-12T16:13:42Z

Tests passed during RECORDING mode:
TestAccFirebaserulesRelease_BasicRelease[Debug log]
TestAccContainerCluster_withInvalidGatewayApiConfigChannel[Debug log]

All tests passed
View the build log or the debug log for each test

modular-magician · 2023-01-15T07:11:58Z

Oops! It looks like you're using an unknown release-note type in your changelog entries:

REPLACEME

Please only use the types listed in https://github.com/GoogleCloudPlatform/magic-modules/blob/master/.ci/RELEASE_NOTES_GUIDE.md.

modular-magician · 2023-01-15T07:15:20Z

Oops! It looks like you're using an unknown release-note type in your changelog entries:

REPLACEME

Please only use the types listed in https://github.com/GoogleCloudPlatform/magic-modules/blob/master/.ci/RELEASE_NOTES_GUIDE.md.

modular-magician · 2023-01-15T07:15:46Z

Oops! It looks like you're using an unknown release-note type in your changelog entries:

REPLACEME

Please only use the types listed in https://github.com/GoogleCloudPlatform/magic-modules/blob/master/.ci/RELEASE_NOTES_GUIDE.md.

modular-magician · 2023-01-15T07:32:56Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 2 files changed, 4988 insertions(+), 1386 deletions(-))
Terraform Beta: Diff ( 2 files changed, 4988 insertions(+), 1386 deletions(-))
TF Validator: Diff ( 3 files changed, 1541 insertions(+), 191 deletions(-))

modular-magician · 2023-01-15T08:27:29Z

Tests analytics

Total tests: 2419
Passed tests 2165
Skipped tests: 252
Failed tests: 2

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests

TestAccFirebaserulesRelease_BasicRelease|TestAccComputeForwardingRule_update

modular-magician · 2023-01-15T08:28:54Z

Tests passed during RECORDING mode:
TestAccFirebaserulesRelease_BasicRelease[Debug log]
TestAccComputeForwardingRule_update[Debug log]

All tests passed
View the build log or the debug log for each test

modular-magician · 2023-01-15T09:12:23Z

Oops! It looks like you're using an unknown release-note type in your changelog entries:

REPLACEME

Please only use the types listed in https://github.com/GoogleCloudPlatform/magic-modules/blob/master/.ci/RELEASE_NOTES_GUIDE.md.

modular-magician · 2023-01-15T09:27:47Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 2 files changed, 4988 insertions(+), 1386 deletions(-))
Terraform Beta: Diff ( 2 files changed, 4988 insertions(+), 1386 deletions(-))
TF Validator: Diff ( 3 files changed, 1541 insertions(+), 191 deletions(-))

modular-magician · 2023-01-15T10:15:07Z

Tests analytics

Total tests: 2419
Passed tests 2166
Skipped tests: 252
Failed tests: 1

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests

TestAccFirebaserulesRelease_BasicRelease

modular-magician · 2023-01-15T10:15:34Z

Tests passed during RECORDING mode:
TestAccFirebaserulesRelease_BasicRelease[Debug log]

All tests passed
View the build log or the debug log for each test

modular-magician · 2023-01-18T14:56:12Z

Oops! It looks like you're using an unknown release-note type in your changelog entries:

REPLACEME

Please only use the types listed in https://github.com/GoogleCloudPlatform/magic-modules/blob/master/.ci/RELEASE_NOTES_GUIDE.md.

modular-magician · 2023-01-18T15:12:21Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 3 files changed, 5042 insertions(+), 1395 deletions(-))
Terraform Beta: Diff ( 3 files changed, 5042 insertions(+), 1395 deletions(-))
TF Validator: Diff ( 3 files changed, 1541 insertions(+), 191 deletions(-))

modular-magician · 2023-01-18T15:40:58Z

Oops! It looks like you're using an unknown release-note type in your changelog entries:

REPLACEME

Please only use the types listed in https://github.com/GoogleCloudPlatform/magic-modules/blob/master/.ci/RELEASE_NOTES_GUIDE.md.

modular-magician · 2023-01-18T15:54:51Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 4 files changed, 5474 insertions(+), 1395 deletions(-))
Terraform Beta: Diff ( 4 files changed, 5474 insertions(+), 1395 deletions(-))
TF Validator: Diff ( 3 files changed, 1541 insertions(+), 191 deletions(-))
TF OiCS: Diff ( 8 files changed, 373 insertions(+))

modular-magician · 2023-01-18T16:43:28Z

Tests analytics

Total tests: 2425
Passed tests 2168
Skipped tests: 254
Failed tests: 3

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests

TestAccComposerEnvironment_UpdateComposerV2|TestAccComposerEnvironment_UpdateComposerV2WithTriggerer|TestAccComposerEnvironment_ComposerV2

modular-magician · 2023-01-18T18:01:10Z

Tests failed during RECORDING mode:
TestAccComposerEnvironment_UpdateComposerV2[Error message] [Debug log]
TestAccComposerEnvironment_UpdateComposerV2WithTriggerer[Error message] [Debug log]
TestAccComposerEnvironment_ComposerV2[Error message] [Debug log]

Please fix these to complete your PR
View the build log or the debug log for each test

c2thorn

Apologies for the delay. I have a couple of questions. I have not run tests related to this PR yet, since I think they can be run in the automatic VCR tests instead of being skipped. Let me know

c2thorn · 2023-01-23T18:59:25Z

mmv1/products/datastream/terraform.yaml

+        # Random provider
+        skip_vcr: true


I'm not seeing the random provider used in either of these examples. Should we remove these lines?

c2thorn · 2023-01-25T19:15:38Z

mmv1/templates/terraform/post_create/datastream_stream.go.erb

@@ -14,4 +14,11 @@
 -%>
 if err := waitForDatastreamStreamReady(d, config, d.Timeout(schema.TimeoutCreate) - time.Minute); err != nil {
  return fmt.Errorf("Error waiting for Stream %q to be NOT_STARTED or RUNNING during creation: %q", d.Get("name").(string), err)
+}
+
+if d.Get("state") != d.Get("desired_state") {


This seems to be completely unrelated to adding the other fields. Is the purpose of doing another Update call just a retry of sorts? Have you tested that this is effective?

This is a bug fix, datastream requires an additional update call when desired_state=RUNNING

c2thorn · 2023-01-25T19:16:47Z

scripts/make-in-container.sh

Even though it is trivial, can we remove this change to not affect the file history?

modular-magician · 2023-01-26T08:11:58Z

Oops! It looks like you're using an unknown release-note type in your changelog entries:

REPLACEME

Please only use the types listed in https://github.com/GoogleCloudPlatform/magic-modules/blob/master/.ci/RELEASE_NOTES_GUIDE.md.

modular-magician · 2023-01-26T08:13:26Z

Oops! It looks like you're using an unknown release-note type in your changelog entries:

REPLACEME

Please only use the types listed in https://github.com/GoogleCloudPlatform/magic-modules/blob/master/.ci/RELEASE_NOTES_GUIDE.md.

modular-magician · 2023-01-26T08:15:08Z

Oops! It looks like you're using an unknown release-note type in your changelog entries:

REPLACEME

Please only use the types listed in https://github.com/GoogleCloudPlatform/magic-modules/blob/master/.ci/RELEASE_NOTES_GUIDE.md.

modular-magician · 2023-01-26T08:16:06Z

Oops! It looks like you're using an unknown release-note type in your changelog entries:

REPLACEME

Please only use the types listed in https://github.com/GoogleCloudPlatform/magic-modules/blob/master/.ci/RELEASE_NOTES_GUIDE.md.

modular-magician · 2023-01-26T08:20:54Z

Oops! It looks like you're using an unknown release-note type in your changelog entries:

REPLACEME

Please only use the types listed in https://github.com/GoogleCloudPlatform/magic-modules/blob/master/.ci/RELEASE_NOTES_GUIDE.md.

modular-magician · 2023-01-26T08:27:44Z

Oops! It looks like you're using an unknown release-note type in your changelog entries:

REPLACEME

Please only use the types listed in https://github.com/GoogleCloudPlatform/magic-modules/blob/master/.ci/RELEASE_NOTES_GUIDE.md.

roaks3 · 2023-01-26T12:55:06Z

FYI @c2thorn this was accidentally closed, and the new PR is #7175

modular-magician requested a review from c2thorn January 12, 2023 12:44

iperetz-goo force-pushed the postgres-support branch from ebd3459 to 967eb02 Compare January 15, 2023 07:15

roaks3 requested a review from lido-google January 17, 2023 16:10

iperetz-goo force-pushed the postgres-support branch from 4ab1dfd to e246699 Compare January 18, 2023 14:55

c2thorn requested changes Jan 25, 2023

View reviewed changes

modular-magician requested a review from c2thorn January 26, 2023 08:12

iperetz-goo closed this Jan 26, 2023

iperetz-goo force-pushed the postgres-support branch from 8544dac to 0ae27f3 Compare January 26, 2023 08:27

This was referenced Jan 26, 2023

Finish datastream #7174

Closed

Finish missing sources from Datastream Stream #7175

Merged

Finish missing sources from Datastream Stream #7115

Finish missing sources from Datastream Stream #7115

Conversation

iperetz-goo commented Jan 12, 2023 • edited Loading

modular-magician commented Jan 12, 2023

modular-magician commented Jan 12, 2023

modular-magician commented Jan 12, 2023

Diff report

modular-magician commented Jan 12, 2023

Tests analytics

Action taken

modular-magician commented Jan 12, 2023

modular-magician commented Jan 12, 2023

modular-magician commented Jan 12, 2023

Diff report

modular-magician commented Jan 12, 2023

Tests analytics

Action taken

modular-magician commented Jan 12, 2023

modular-magician commented Jan 15, 2023

modular-magician commented Jan 15, 2023

modular-magician commented Jan 15, 2023

modular-magician commented Jan 15, 2023

Diff report

modular-magician commented Jan 15, 2023

Tests analytics

Action taken

modular-magician commented Jan 15, 2023

modular-magician commented Jan 15, 2023

modular-magician commented Jan 15, 2023

Diff report

modular-magician commented Jan 15, 2023

Tests analytics

Action taken

modular-magician commented Jan 15, 2023

modular-magician commented Jan 18, 2023

modular-magician commented Jan 18, 2023

Diff report

modular-magician commented Jan 18, 2023

modular-magician commented Jan 18, 2023

Diff report

modular-magician commented Jan 18, 2023

Tests analytics

Action taken

modular-magician commented Jan 18, 2023

c2thorn left a comment

Choose a reason for hiding this comment

c2thorn Jan 23, 2023

Choose a reason for hiding this comment

iperetz-goo Jan 26, 2023

Choose a reason for hiding this comment

c2thorn Jan 25, 2023

Choose a reason for hiding this comment

iperetz-goo Jan 26, 2023

Choose a reason for hiding this comment

c2thorn Jan 25, 2023

Choose a reason for hiding this comment

iperetz-goo Jan 26, 2023

Choose a reason for hiding this comment

modular-magician commented Jan 26, 2023

modular-magician commented Jan 26, 2023

modular-magician commented Jan 26, 2023

modular-magician commented Jan 26, 2023

modular-magician commented Jan 26, 2023

modular-magician commented Jan 26, 2023

roaks3 commented Jan 26, 2023

iperetz-goo commented Jan 12, 2023 •

edited

Loading