feat: Support for Dataproc on GKE #6143

prauc · 2022-06-16T23:05:16Z

Adds Support for Dataproc on GKE

If this PR is for Terraform, I acknowledge that I have:

Searched through the issue tracker for an open issue that this either resolves or contributes to, commented on it to claim it, and written "fixes {url}" or "part of {url}" in this PR description. If there were no relevant open issues, I opened one and commented that I would like to work on it (not necessary for very small changes).
Generated Terraform, and ran make test and make lint to ensure it passes unit and linter tests.
Ensured that all new fields I added that can be set by a user appear in at least one example (for generated resources) or third_party test (for handwritten resources or update tests).
Ran relevant acceptance tests (If the acceptance tests do not yet pass or you are unable to run them, please let your reviewer know).
Read the Release Notes Guide before writing my release note below.

Release Note Template for Downstream PRs (will be copied)

dataproc: Added Support for Dataproc on GKE in `google_dataproc_cluster` (GA only)

modular-magician · 2022-06-16T23:05:32Z

Oops! It looks like you're using an unknown release-note type in your changelog entries:

note:enhancement

Please only use the types listed in https://github.com/GoogleCloudPlatform/magic-modules/blob/master/.ci/RELEASE_NOTES_GUIDE.md.

modular-magician · 2022-06-16T23:06:11Z

Hello! I am a robot who works on Magic Modules PRs.

I've detected that you're a community contributor. @melinath, a repository maintainer, has been assigned to assist you and help review your changes.

❓ First time contributing? Click here for more details

Your assigned reviewer will help review your code by:

Ensuring it's backwards compatible, covers common error cases, etc.
Summarizing the change into a user-facing changelog note.
Passes tests, either our "VCR" suite, a set of presubmit tests, or with manual test runs.

You can help make sure that review is quick by running local tests and ensuring they're passing in between each push you make to your PR's branch. Also, try to leave a comment with each push you make, as pushes generally don't generate emails.

If your reviewer doesn't get back to you within a week after your most recent change, please feel free to leave a comment on the issue asking them to take a look! In the absence of a dedicated review dashboard most maintainers manage their pending reviews through email, and those will sometimes get lost in their inbox.

modular-magician · 2022-06-16T23:17:18Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 2 files changed, 413 insertions(+), 2 deletions(-))
Terraform Beta: Diff ( 2 files changed, 413 insertions(+), 2 deletions(-))
TF Validator: Diff ( 2 files changed, 3 insertions(+), 3 deletions(-))

modular-magician · 2022-06-16T23:18:35Z

Tests analytics

Total tests: 0
Passed tests 0
Skipped tests: 0
Failed tests: 0

Errors occurred during REPLAYING mode. Please fix them to complete your PR
View the build log

modular-magician · 2022-06-16T23:59:30Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 2 files changed, 413 insertions(+), 2 deletions(-))
Terraform Beta: Diff ( 1 file changed, 5 insertions(+), 1 deletion(-))
TF Validator: Diff ( 2 files changed, 3 insertions(+), 3 deletions(-))

modular-magician · 2022-06-17T00:40:38Z

Tests analytics

Total tests: 2053
Passed tests 1825
Skipped tests: 226
Failed tests: 2

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests

TestAccContainerCluster_withConfidentialNodes|TestAccContainerCluster_withAddons

modular-magician · 2022-06-17T01:03:23Z

Tests failed during RECORDING mode:
TestAccContainerCluster_withAddons[view]
TestAccContainerCluster_withConfidentialNodes[view]

Please fix these to complete your PR
View the build log or the debug log for each test

modular-magician · 2022-06-20T17:16:15Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 2 files changed, 522 insertions(+), 2 deletions(-))
Terraform Beta: Diff ( 1 file changed, 5 insertions(+), 1 deletion(-))
TF Validator: Diff ( 2 files changed, 3 insertions(+), 3 deletions(-))

prauc · 2022-06-20T17:25:53Z

some missing features added after alignment call with customer

modular-magician · 2022-06-20T18:04:05Z

Tests analytics

Total tests: 2053
Passed tests 1825
Skipped tests: 226
Failed tests: 2

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests

TestAccContainerCluster_withConfidentialNodes|TestAccContainerCluster_withAddons

modular-magician · 2022-06-20T18:26:40Z

Tests failed during RECORDING mode:
TestAccContainerCluster_withAddons[view]
TestAccContainerCluster_withConfidentialNodes[view]

Please fix these to complete your PR
View the build log or the debug log for each test

c2thorn

Have some high-level questions about the schema. Also, the documentation file should be updated with the new fields, but I'd hold off on that until confirmation on the schema level questions so you don't duplicate work

c2thorn · 2022-06-28T16:54:50Z

mmv1/third_party/terraform/resources/resource_dataproc_cluster.go.erb

@@ -200,6 +218,168 @@ func resourceDataprocCluster() *schema.Resource {
 							Description:  `The Cloud Storage temp bucket used to store ephemeral cluster and jobs data, such as Spark and MapReduce history files. Note: If you don't explicitly specify a temp_bucket then GCP will auto create / assign one for you.`,
 						},

+<% if version == "ga" -%>


Why is this GA only? What's preventing it from being in beta?
It is rare for something to be in GA and completely non-existent in beta (in some form).

I was also very surprised, but the beta-package seems not to have this implement. VirtualClusterConfig does not exist in type Cluster for package v1beta2: https://pkg.go.dev/google.golang.org/api/dataproc/v1beta2#Cluster

c2thorn · 2022-06-28T17:00:20Z

mmv1/third_party/terraform/resources/resource_dataproc_cluster.go.erb

@@ -200,6 +218,168 @@ func resourceDataprocCluster() *schema.Resource {
 							Description:  `The Cloud Storage temp bucket used to store ephemeral cluster and jobs data, such as Spark and MapReduce history files. Note: If you don't explicitly specify a temp_bucket then GCP will auto create / assign one for you.`,
 						},

+<% if version == "ga" -%>
+						"gke_cluster_config": {


This seems to combine VirtualClusterConfig, GKEClusterConfig, GKENodePoolConfig and potentially more (it's hard to follow) from the API. We aim to be as in line as possible to the API schema and deviating from that could lead to challenges updating the resource in the future as schema changes are breaking changes. What is the reasoning here?

makes sense! I wanted to simplify the structure and make it easier for users to use. But I get your point. I will change this according to the API

c2thorn · 2022-06-28T17:01:44Z

mmv1/third_party/terraform/resources/resource_dataproc_cluster.go.erb

+										Elem: &schema.Resource{
+											Schema: map[string]*schema.Schema{
+
+												"metastore_service": {


Child fields that are not ForceNew imply that they are updatable. If this is the case, a test step showing the updatability of these fields is preferred

Thanks for the hint! Will add tests to mentioned child fields

modular-magician · 2022-07-05T19:02:00Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 2 files changed, 736 insertions(+), 5 deletions(-))
Terraform Beta: Diff ( 1 file changed, 5 insertions(+), 1 deletion(-))
TF Validator: Diff ( 2 files changed, 3 insertions(+), 3 deletions(-))

modular-magician · 2022-07-05T19:42:52Z

Tests analytics

Total tests: 2076
Passed tests 1847
Skipped tests: 226
Failed tests: 3

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests

TestAccFirebaserulesRelease_BasicRelease|TestAccPrivatecaCertificateAuthority_privatecaCertificateAuthoritySubordinateExample|TestAccPrivatecaCertificateAuthority_subordinateCaActivatedByFirstPartyIssuerOnCreation

modular-magician · 2022-07-05T19:43:36Z

Tests passed during RECORDING mode:
TestAccFirebaserulesRelease_BasicRelease[view]

Tests failed during RECORDING mode:
TestAccPrivatecaCertificateAuthority_subordinateCaActivatedByFirstPartyIssuerOnCreation[view]
TestAccPrivatecaCertificateAuthority_privatecaCertificateAuthoritySubordinateExample[view]

Please fix these to complete your PR
View the build log or the debug log for each test

modular-magician · 2022-07-06T07:50:22Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 2 files changed, 731 insertions(+), 5 deletions(-))
Terraform Beta: Diff ( 1 file changed, 5 insertions(+), 1 deletion(-))
TF Validator: Diff ( 2 files changed, 3 insertions(+), 3 deletions(-))

modular-magician · 2022-07-06T08:34:55Z

Tests analytics

Total tests: 2076
Passed tests 1846
Skipped tests: 226
Failed tests: 4

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests

TestAccSqlUser_mysqlDisabled|TestAccPrivatecaCertificateAuthority_subordinateCaActivatedByFirstPartyIssuerOnCreation|TestAccPrivatecaCertificateAuthority_privatecaCertificateAuthoritySubordinateExample|TestAccFirebaserulesRelease_BasicRelease

modular-magician · 2022-07-06T08:49:32Z

Tests passed during RECORDING mode:
TestAccFirebaserulesRelease_BasicRelease[view]
TestAccSqlUser_mysqlDisabled[view]

Tests failed during RECORDING mode:
TestAccPrivatecaCertificateAuthority_subordinateCaActivatedByFirstPartyIssuerOnCreation[view]
TestAccPrivatecaCertificateAuthority_privatecaCertificateAuthoritySubordinateExample[view]

Please fix these to complete your PR
View the build log or the debug log for each test

…odules

modular-magician · 2022-07-09T14:01:16Z

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 3 files changed, 896 insertions(+), 5 deletions(-))
Terraform Beta: Diff ( 2 files changed, 163 insertions(+), 1 deletion(-))
TF Validator: Diff ( 2 files changed, 3 insertions(+), 3 deletions(-))

modular-magician · 2022-07-09T14:44:55Z

Tests analytics

Total tests: 2090
Passed tests 1862
Skipped tests: 226
Failed tests: 2

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests

TestAccPrivatecaCertificateAuthority_subordinateCaActivatedByFirstPartyIssuerOnCreation|TestAccPrivatecaCertificateAuthority_privatecaCertificateAuthoritySubordinateExample

c2thorn · 2022-07-11T19:28:37Z

Ran the test manually since VCR tests are beta-only, and it passed. Just taking another look before merging

c2thorn · 2022-07-11T20:21:01Z

mmv1/third_party/terraform/resources/resource_dataproc_cluster.go.erb

@@ -89,6 +119,25 @@ func resourceDataprocLabelDiffSuppress(k, old, new string, d *schema.ResourceDat
 	return false
 }

+<% if version == "ga" -%>
+const resourceDataprocGoogleProvidedDPGKEPrefix = "virtual_cluster_config.0.kubernetes_cluster_config.0.kubernetes_software_config.0.properties.dpgke"


This could be written as a list of prefixes, and the diff suppress could iterate through the list to check. Not the biggest deal now, but if more sever-generated properties are found later, it will be worthwhile to switch.

prauc added 12 commits May 30, 2022 16:53

add dataproc on gke

02cde24

add dataproc on gke cluster

74a0d8d

remove unnecessary items

ad94735

add config

97b49d5

revoke changes

fd3ec39

add dataproc on gke

da3018a

revoke api changes

e499522

add last line

f21857d

add last line

9b1ed98

add tests

fc76e9d

enhance config

7bf295f

remove unused func

031973a

modular-magician requested a review from melinath June 16, 2022 23:06

remove features for beta-provider

f65c78e

prauc added 2 commits June 20, 2022 18:59

add missing properties and auxiliary_services fields

11338b4

add missing funcs

ed50d69

melinath requested review from c2thorn and removed request for melinath June 22, 2022 22:04

c2thorn requested changes Jun 28, 2022

View reviewed changes

fix lint err

09191b1

prauc added 12 commits July 8, 2022 14:58

suppress properties and make component version required

88c4fb9

remove computed

9dc8e0d

add docs for virtual cluster config

1f9324c

fmt

01deb90

fix

109bda2

fix fmt

5e25ede

revoke change

492a1af

add docs

1f276b4

fix var naming

0efac63

fix regex

d950599

Merge branch 'main' of https://github.com/GoogleCloudPlatform/magic-m…

407f864

…odules

fix duplicate func name

f3f7bcc

prauc requested a review from c2thorn July 9, 2022 14:54

c2thorn reviewed Jul 11, 2022

View reviewed changes

c2thorn approved these changes Jul 11, 2022

View reviewed changes

c2thorn merged commit b631a10 into GoogleCloudPlatform:main Jul 11, 2022

This was referenced Jul 11, 2022

feat: Support for Dataproc on GKE hashicorp/terraform-provider-google-beta#4474

Merged

feat: Support for Dataproc on GKE hashicorp/terraform-provider-google#12076

Merged

geojaz pushed a commit to geojaz/magic-modules that referenced this pull request Jul 12, 2022

feat: Support for Dataproc on GKE (GoogleCloudPlatform#6143)

02d717b

melinath mentioned this pull request May 4, 2023

Sigsev error with Google Dataproc creation error with Virtual Cluster Config and Gke Cluster Config hashicorp/terraform-provider-google#14457

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Support for Dataproc on GKE #6143

feat: Support for Dataproc on GKE #6143

prauc commented Jun 16, 2022 •

edited by c2thorn

Loading

modular-magician commented Jun 16, 2022

modular-magician commented Jun 16, 2022

modular-magician commented Jun 16, 2022

modular-magician commented Jun 16, 2022

modular-magician commented Jun 16, 2022

modular-magician commented Jun 17, 2022

modular-magician commented Jun 17, 2022

modular-magician commented Jun 20, 2022

prauc commented Jun 20, 2022

modular-magician commented Jun 20, 2022

modular-magician commented Jun 20, 2022

c2thorn left a comment

c2thorn Jun 28, 2022

prauc Jun 28, 2022

c2thorn Jun 28, 2022

prauc Jun 28, 2022

c2thorn Jun 28, 2022

prauc Jun 28, 2022

modular-magician commented Jul 5, 2022

modular-magician commented Jul 5, 2022

modular-magician commented Jul 5, 2022

modular-magician commented Jul 6, 2022

modular-magician commented Jul 6, 2022

modular-magician commented Jul 6, 2022

modular-magician commented Jul 9, 2022

modular-magician commented Jul 9, 2022

c2thorn commented Jul 11, 2022

c2thorn Jul 11, 2022

feat: Support for Dataproc on GKE #6143

feat: Support for Dataproc on GKE #6143

Conversation

prauc commented Jun 16, 2022 • edited by c2thorn Loading

modular-magician commented Jun 16, 2022

modular-magician commented Jun 16, 2022

modular-magician commented Jun 16, 2022

Diff report:

modular-magician commented Jun 16, 2022

Tests analytics

modular-magician commented Jun 16, 2022

Diff report:

modular-magician commented Jun 17, 2022

Tests analytics

Action taken

modular-magician commented Jun 17, 2022

modular-magician commented Jun 20, 2022

Diff report:

prauc commented Jun 20, 2022

modular-magician commented Jun 20, 2022

Tests analytics

Action taken

modular-magician commented Jun 20, 2022

c2thorn left a comment

Choose a reason for hiding this comment

c2thorn Jun 28, 2022

Choose a reason for hiding this comment

prauc Jun 28, 2022

Choose a reason for hiding this comment

c2thorn Jun 28, 2022

Choose a reason for hiding this comment

prauc Jun 28, 2022

Choose a reason for hiding this comment

c2thorn Jun 28, 2022

Choose a reason for hiding this comment

prauc Jun 28, 2022

Choose a reason for hiding this comment

modular-magician commented Jul 5, 2022

Diff report:

modular-magician commented Jul 5, 2022

Tests analytics

Action taken

modular-magician commented Jul 5, 2022

modular-magician commented Jul 6, 2022

Diff report:

modular-magician commented Jul 6, 2022

Tests analytics

Action taken

modular-magician commented Jul 6, 2022

modular-magician commented Jul 9, 2022

Diff report:

modular-magician commented Jul 9, 2022

Tests analytics

Action taken

c2thorn commented Jul 11, 2022

c2thorn Jul 11, 2022

Choose a reason for hiding this comment

prauc commented Jun 16, 2022 •

edited by c2thorn

Loading