New Resource(s)/Data Sources for HDInsights Cluster #3196

tombuildsstuff · 2019-04-05T19:48:52Z

This PR adds support for HDInsights, in all of its various incarnations - in particular this PR adds the following:

Data Source: azurerm_hdinsight_cluster
Resource: azurerm_hdinsight_kafka_cluster
Resource: azurerm_hdinsight_kbase_cluster
Resource: azurerm_hdinsight_hadoop_cluster
Resource: azurerm_hdinsight_interactive_query_cluster
Resource: azurerm_hdinsight_ml_services_cluster
Resource: azurerm_hdinsight_rserver_cluster
Resource: azurerm_hdinsight_spark_cluster
Resource: azurerm_hdinsight_storm_cluster

The Data Source azurerm_hdinsight_cluster intentionally only exports a subset of fields since there's sufficient differences between the different kinds of HDInsights Cluster that this is both practical and (what I believe) are the relevant fields that most users are likely to need.

Due to the HDInsights API being very pedantic about the result structure - rather than try and ship a single HDInsights Resource (as has been previously attempted in #1267, #924 and #814) this PR introduces separate resources for each "kind" of HDInsights Cluster, due to the number of these the resources in this PR follow a very different structure to other resources.

Remaining Tasks

~~Validation~~ Documentation for the Node password field
Validation for the Cluster Version
Filing an API bug about the Sku being returned in the incorrect case

Fixes #37

…the right field

…Server

``` === RUN TestHDInsightClusterVersionDiffSuppress === RUN TestHDInsightClusterVersionDiffSuppress/empty_name === RUN TestHDInsightClusterVersionDiffSuppress/missing_user_input === RUN TestHDInsightClusterVersionDiffSuppress/missing_api_response === RUN TestHDInsightClusterVersionDiffSuppress/major_minor_user_input === RUN TestHDInsightClusterVersionDiffSuppress/full_version_user_input --- PASS: TestHDInsightClusterVersionDiffSuppress (0.00s) --- PASS: TestHDInsightClusterVersionDiffSuppress/empty_name (0.00s) --- PASS: TestHDInsightClusterVersionDiffSuppress/missing_user_input (0.00s) --- PASS: TestHDInsightClusterVersionDiffSuppress/missing_api_response (0.00s) --- PASS: TestHDInsightClusterVersionDiffSuppress/major_minor_user_input (0.00s) --- PASS: TestHDInsightClusterVersionDiffSuppress/full_version_user_input (0.00s) PASS ```

``` === RUN TestHDInsightClusterVersion === RUN TestHDInsightClusterVersion/empty_name === RUN TestHDInsightClusterVersion/major_only === RUN TestHDInsightClusterVersion/major_minor === RUN TestHDInsightClusterVersion/major_minor_large === RUN TestHDInsightClusterVersion/major_minor_build === RUN TestHDInsightClusterVersion/major_minor_build_revision === RUN TestHDInsightClusterVersion/real-world-example --- PASS: TestHDInsightClusterVersion (0.00s) --- PASS: TestHDInsightClusterVersion/empty_name (0.00s) --- PASS: TestHDInsightClusterVersion/major_only (0.00s) --- PASS: TestHDInsightClusterVersion/major_minor (0.00s) --- PASS: TestHDInsightClusterVersion/major_minor_large (0.00s) --- PASS: TestHDInsightClusterVersion/major_minor_build (0.00s) --- PASS: TestHDInsightClusterVersion/major_minor_build_revision (0.00s) --- PASS: TestHDInsightClusterVersion/real-world-example (0.00s) PASS ```

tombuildsstuff · 2019-04-08T08:39:01Z

Tests pass for Kafka, HBase, Hadoop, ML Services, R Server and Storm

Changes have been pushed for Interactive Query and Spark - running atm

tombuildsstuff · 2019-04-08T09:02:08Z

Tests pass for Interactive Query Clusters:

tombuildsstuff · 2019-04-08T10:40:30Z

HDInsight Spark Cluster tests pass:

…parsing the uri

tombuildsstuff · 2019-04-08T18:09:19Z

Ignoring a failed assertion (which has been fixed in a follow up commit) the data source tests pass:

katbyte

Aside from a couple minor comments LGTM @tombuildsstuff 👍

azurerm/common_hdinsight.go

azurerm/data_source_hdinsight_cluster.go

azurerm/helpers/azure/hdinsight.go

ofirshtrull · 2019-04-17T07:39:15Z

hi,
I couldnt find in the new configurations,
will there be an option to add cluster configurations like for hive-site.xml
i.e https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-hadoop-customize-cluster-bootstrap

Thanks

ghost · 2019-04-17T10:38:35Z

This has been released in version 1.25.0 of the provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading. As an example:

provider "azurerm" {
	version = "~> 1.25.0"
}
# ... other configuration ...

widgetpl · 2019-04-17T12:06:57Z

website/docs/d/hdinsight_cluster.html.markdown

+
+* `username` - The username used for the Ambari Portal.
+
+* `username` - The password used for the Ambari Portal.


shouldn't be password here?

@widgetpl - yes - would you mind opening a new issue for this? thanks!

@tombuildsstuff done #3287 but should we make docs per cluster type like azurerm_hdinsight_kafka_cluster, azurerm_hdinsight_hadoop_cluster, azurerm_hdinsight_spark_cluster I see there are some small differences?

sorry I mixed data with resource documentation, everything is in place ;)

ghost · 2019-05-09T13:50:30Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!

tombuildsstuff added new-resource new-data-source service/hdinsight labels Apr 5, 2019

tombuildsstuff added this to the v1.25.0 milestone Apr 5, 2019

tombuildsstuff requested review from katbyte and mbfrahry April 5, 2019 19:48

ghost added size/XXL dependencies labels Apr 5, 2019

tombuildsstuff mentioned this pull request Apr 5, 2019

Feature Request: HDinsight Cluster Deployment #37

Closed

tombuildsstuff added 20 commits April 8, 2019 04:58

registering the required resource providers for HDInsight

6656450

Vendoring the HDInsights Clients

71d274f

HDInsight: common schema and helpers

ba9b42f

New Resource: azurerm_hdinsight_hadoop_cluster

659f851

New Resource: azurerm_hdinsight_hbase_cluster

e00e84f

New Resource: azurerm_hdinsight_interactive_query_cluster

9c46324

New Resource: azurerm_hdinsight_kafka_cluster

3f4b4b8

New Resource: azurerm_hdinsight_ml_services_cluster

1c9a524

New Resource: azurerm_hdinsight_rserver_cluster

d263fc8

New Resource: azurerm_hdinsight_spark_cluster

bd6a505

New Resource: azurerm_hdinsight_storm_cluster

bd196f0

New Data Source: azurerm_hdinsight_cluster

5370125

Registering/Linking to the HDInsight Resources

d57f3ee

Fixing a crash where the edge node didn't exist in the schema

bb59833

r/hdinsight: removing irrelevant todo's

61fd71f

r/hdinsight_interactive_query_cluster: loading the hive version from …

1f08567

…the right field

r/hdinsight: setting rserver at the right location for MLServices/R…

ba68216

…Server

r/hdinsight: standardizing the vm sku's

01dd34b

r/hdinsight: ignoring the casing on sku

bec362d

r/hdinsights_interactive_query: fixing the cluster kind

a4a4936

tombuildsstuff force-pushed the f/hdinsights branch from 19e5484 to a4a4936 Compare April 8, 2019 02:58

tombuildsstuff added 7 commits April 8, 2019 05:49

r/hdinsights: allowing Standard_A3 as a type

453af3a

r/hdinsight_interactive_query: using Standard_A4_V2 instances

efab846

hdinsight: switching out the example vm sku's

2ca7596

r/hdinsight: documenting the password requirements

1023dad

r/hdinsights_interactive_query: switching to large instances.

a9d1413

r/hdinsight_spark: adding a fixed target count

b6c7024

tombuildsstuff added 4 commits April 8, 2019 13:03

d/hdinsight_cluster: setting the id

8552769

d/hdinsight_cluster: loading the values from the config, rather than …

58a57b7

…parsing the uri

d/hdinsight_cluster: fixing the casing on the test

1a22656

d/hdinsight: force lower-casing the kind

63304cc

tombuildsstuff force-pushed the f/hdinsights branch from 28fe95f to 63304cc Compare April 8, 2019 13:22

katbyte approved these changes Apr 9, 2019

View reviewed changes

azurerm/common_hdinsight.go Outdated Show resolved Hide resolved

azurerm/data_source_hdinsight_cluster.go Show resolved Hide resolved

azurerm/helpers/azure/hdinsight.go Show resolved Hide resolved

azurerm/helpers/azure/hdinsight.go Show resolved Hide resolved

tombuildsstuff added 3 commits April 9, 2019 07:58

r/hdinsights: addressing comments from PR review

2740d05

r/hdinsight: documenting why this pattern is different

ac37aa2

fixing the import

455dc3d

tombuildsstuff force-pushed the f/hdinsights branch from 08d9a59 to 455dc3d Compare April 9, 2019 07:07

tombuildsstuff merged commit b18af3a into master Apr 9, 2019

tombuildsstuff deleted the f/hdinsights branch April 9, 2019 07:16

tombuildsstuff added a commit that referenced this pull request Apr 9, 2019

Updating to include #3196

091a1b5

widgetpl reviewed Apr 17, 2019

View reviewed changes

ghost locked and limited conversation to collaborators May 9, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New Resource(s)/Data Sources for HDInsights Cluster #3196

New Resource(s)/Data Sources for HDInsights Cluster #3196

tombuildsstuff commented Apr 5, 2019 •

edited

Loading

tombuildsstuff commented Apr 8, 2019

tombuildsstuff commented Apr 8, 2019

tombuildsstuff commented Apr 8, 2019

tombuildsstuff commented Apr 8, 2019

katbyte left a comment

ofirshtrull commented Apr 17, 2019 •

edited

Loading

ghost commented Apr 17, 2019

widgetpl Apr 17, 2019

tombuildsstuff Apr 17, 2019

widgetpl Apr 19, 2019

widgetpl Apr 19, 2019

ghost commented May 9, 2019


		* `username` - The username used for the Ambari Portal.

		* `username` - The password used for the Ambari Portal.

New Resource(s)/Data Sources for HDInsights Cluster #3196

New Resource(s)/Data Sources for HDInsights Cluster #3196

Conversation

tombuildsstuff commented Apr 5, 2019 • edited Loading

Remaining Tasks

tombuildsstuff commented Apr 8, 2019

tombuildsstuff commented Apr 8, 2019

tombuildsstuff commented Apr 8, 2019

tombuildsstuff commented Apr 8, 2019

katbyte left a comment

Choose a reason for hiding this comment

ofirshtrull commented Apr 17, 2019 • edited Loading

ghost commented Apr 17, 2019

widgetpl Apr 17, 2019

Choose a reason for hiding this comment

tombuildsstuff Apr 17, 2019

Choose a reason for hiding this comment

widgetpl Apr 19, 2019

Choose a reason for hiding this comment

widgetpl Apr 19, 2019

Choose a reason for hiding this comment

ghost commented May 9, 2019

tombuildsstuff commented Apr 5, 2019 •

edited

Loading

ofirshtrull commented Apr 17, 2019 •

edited

Loading