TPG 3.0.0 (#2470)

* cloudbuild build trigger's trigger template should be required in 3.0.0. (#2352)

* cloudbuild build trigger's trigger template should be required in 3.0.0.

* Make backend_service.backends.group required. (#2373)

* Remove Removed fields (#2391)

* deprecated-->removed event notification config (#2390)

* deprecated-->removed event notification config

* remove singular config from tests/docs

* pluralize

* Update third_party/terraform/tests/resource_cloudiot_registry_test.go

Co-Authored-By: Riley Karson <[email protected]>

* Delete google_project_services in 3.0.0. (#2403)

* Changing IAM audit config to be authoritative (#2438)

* Changing IAM audit config to be authoritative

* Remove unused code + test

* Removing deprecated fields (#2436)

* Add validation for scratch disks in Instance Template (#2282)

* Add validation for scratch disks

* Remove source from scratch disk

* Use hardcoded image

* Add reverse logic

* Change default id format from {{name}} to self_link_uri (#2461)

* Change default id format from {{name}} to self_link_uri

* Updates to generated id formats (#2460)

* Update ID fields for various MM-generated resources

* gofmt -s -w extra file

* Add userinfo.email to default scopes (#2473)

* Set GKE Stackdriver defaults to GKE Stackdriver Monitoring (#2471)

* only allow instance templates with 375gb scratch disks (#2495)

* Update cloudfunction id for 3.0.0 release (#2501)

* handle cloudfunctions deprecating nodejs6 (#2499)

* Handwritten id updates for several compute resources (#2502)

* Composer environment and compute attached disk id updates

* Add compute instance

* Add instance_from_group and instance_group

* IGM id

* Instance template tests passing

* Add new id format to other calls of parseImportId in IGM

* Small refactor to memoize var

* Refactor, remove parseImportId in IGM

* handle legacy network deprecation (#2508)

* Fix gofmt diff in Cloud Functions (#2542)

* More handwritten ids (#2527)

* Fix cloudfunction formatting, migrate RIGM id

* Update security policy resource id

* Update target pool id format

* Add new id for container cluster

* Node pool id updates

* PR feedback, using parseImportId for container resources

* Formatting

* fix container cluster (#2550)

* Datasource id updates (#2544)

* Update datasources

* formatting

* Even more handwritten ids (#2540)

* Dataproc cluster, job, google project

* Update sql ssl cert, database instance ids

* Project id comparison include projects/

* remove kubernetes_dashboard from google_container_cluster (#2551)

* Add back encoder that always sends autoCreateSubnetworks (#2558)

* Allow defining empty taint, remove old DSF (#2537)

* Remove encoder, can be done as send_empty_value on autoCreateSubnetworks (#2559)

* deprecate 0.11 syntax in docs (datasources) (#2573)

* deprecate 0.11 syntax in docs (resources) (#2574)

* deprecate 0.11 syntax in docs (magic modules examples) (#2579)

* remove long name behavior for pubsub subscriptions, clean up cu… (#2561)

* remove long name behavior for pubsub subscriptions, clean up custom expanders

* fix build

* fix tests

* Fix merge conflict in rigm test

* Remove unused regexes from rigm (#2602)

* Fix missing paren + bad var in resourcepolicy datasource 3.0.0 (#2600)

* Test fixes based on integration tests, id formats + other easy fixes (#2605)

* Test fixes based on integration tests, id formats + other easy fixes

* Compute network id test update

* Fix region disk test id

* Update subnetwork to remove enable_flow_logs and depend on log_config instead (#2597)

* Test updates to not depend on id format. Also test fixes (#2609)

* Test updates to not depend on id format. Also test fixes

* Typo

* location -> region in node pool test

* Cluster test fixes (#2611)

* Fix container cluster tests, and signed url key, instance group

* Fix invalid accessor

* Update hand-written resources with `AtLeastOneOf` and `ExactlyOneOf` attributes (#2608)

* Rigm/igm field removals for 3.0.0 (#2595)

* Remove deprecated IGM fields for 3.0

* fix imports (#2619)

* Remove automatic subnetwork creation in GKE (#2615)

* Remove automatic subnetwork creation in GKE

* Update docs, add Computed

* More integration test fixes (#2617)

* Fix bigtable for real, scratch disk size, target pool checks

* Proxy test fixes

* CloudIOT registry set removed field to nil. Add specific id format for access context manager

* (r)igm imports for customdiff (#2623)

* (r)igm imports for customdiff

* Fix schema issues

* Fix IGM custom diff and reuse method between resources (#2624)

* Forwardingrule ip address (#2620)

add the validation function

* Remove unused sort import from rigm (#2625)

* add composer test fixes, backend bucket signed url key (#2631)

* Stop users from specifying bigquery-json in 3.0.0 (#2626)

* Send null logConfig if subnetwork is L7ILB (#2635)

* Send null logConfig if subnetwork is L7ILB

* Move subnet ilb check to only if logConfig unspecified

* deprecate 0.11 syntax in tests (datasources) (#2593)

* Make master_authorized_networks_config.cidr_blocks Optional in… (#2642)

* Make master_authorized_networks_config.cidr_blocks Optional in GKE

* Add note

* update magic module-generated resources with at_least_one_of (#2639)

* add exactly_one_of to magic modules

update comment

* Update Data Fusion Instance to conform with 3.0.0 id updates (#2649)

* Fix IAM doc id_format to new post 3.0.0 style

* Add note on id format changes (#2656)

* Add upgrade guide for subnetwork log_config (#2653)

* Add upgrade guide for subnetwork log_config

* Add examples for subnet logConfig update

* Wording on upgrade guide

* Add note on audit log config (#2658)

* Add misc entries to the upgrade guide (#2657)

* Add misc entries to the upgrade guide

* Remove extra space

* add info on scratch disk size and remove duplicate kms_key_self… (#2659)

* Add upgrade guide notes. (#2663)

* Add upgrade guide for forwarding rule validation (#2666)

* Convert old import formats (using non-/) to use /. (#2638)

* Convert old import formats (using non-/) to use /.

* Upgrade proofreading (#2668)

* Add upgrade note for IoT registry event_notification_configs (#2390) (#2651)

Merged PR #2651.

* sort upgrade guide; change wording in a few places

* Password is not actually required. (#2670)

* Add improved for_each non-module variant to project service upg… (#2671)

* Update bigtable import & tests to 3.0 (#2673)

* update upgrade guide (#2665)

* update formatting for upgrade doc (#2674)

* Add rigm to upgrade guide (#2667)

Update docs to be accurate for 3.0

* proofreading changes (#2676)

* Add back lost newlines

api/type.rb

@@ -70,6 +70,12 @@ module Fields
       # A list of properties that conflict with this property.
       attr_reader :conflicts
 
+      # A list of properties that at least one of must be set.
+      attr_reader :at_least_one_of
+
+      # A list of properties that exactly one of must be set.
+      attr_reader :exactly_one_of
+
       # Can only be overridden - we should never set this ourselves.
       attr_reader :new_type
 
@@ -109,6 +115,8 @@ def validate
 
       check_default_value_property
       check_conflicts
+      check_at_least_one_of
+      check_exactly_one_of
     end
 
     def to_s
@@ -135,6 +143,10 @@ def to_json(opts = nil)
       instance_variables.each do |v|
         if v == :@conflicts && instance_variable_get(v).empty?
           # ignore empty conflict arrays
+        elsif v == :@at_least_one_of && instance_variable_get(v).empty?
+          # ignore empty at_least_one_of arrays
+        elsif v == :@exactly_one_of && instance_variable_get(v).empty?
+          # ignore empty exactly_one_of arrays
         elsif instance_variable_get(v) == false || instance_variable_get(v).nil?
           # ignore false booleans as non-existence indicates falsey
         elsif !ignored_fields.include? v
@@ -194,6 +206,38 @@ def conflicting
        @__resource.all_user_properties.select { |p| p.conflicts.include?(@api_name) }).uniq
     end
 
+    # Checks that all properties that needs at least one of their fields actually exist.
+    # This currently just returns if empty, because we don't want to do the check, since
+    # this list will have a full path for nested attributes.
+    def check_at_least_one_of
+      check :at_least_one_of, type: ::Array, default: [], item_type: ::String
+
+      return if @at_least_one_of.empty?
+    end
+
+    # Returns list of properties that needs at least one of their fields set.
+    def at_least_one_of_list
+      return [] unless @__resource
+
+      @at_least_one_of
+    end
+
+    # Checks that all properties that needs exactly one of their fields actually exist.
+    # This currently just returns if empty, because we don't want to do the check, since
+    # this list will have a full path for nested attributes.
+    def check_exactly_one_of
+      check :exactly_one_of, type: ::Array, default: [], item_type: ::String
+
+      return if @exactly_one_of.empty?
+    end
+
+    # Returns list of properties that needs exactly one of their fields set.
+    def exactly_one_of_list
+      return [] unless @__resource
+
+      @exactly_one_of
+    end
+
     def type
       self.class.name.split('::').last
     end
@@ -289,6 +333,8 @@ class FetchedExternal < Type
 
       def validate
         @conflicts ||= []
+        @at_least_one_of ||= []
+        @exactly_one_of ||= []
       end
 
       def api_name

overrides/terraform/property_override.rb

@@ -69,6 +69,12 @@ def self.attributes
           # Names of attributes that can't be set alongside this one
           :conflicts_with,
 
+          # Names of attributes that at least one of must be set
+          :at_least_one_of,
+
+          # Names of attributes that exactly one of must be set
+          :exactly_one_of,
+
           # Names of fields that should be included in the updateMask.
           :update_mask_fields,
 

overrides/terraform/resource_override.rb

@@ -76,7 +76,7 @@ def validate
         @examples ||= []
 
         check :legacy_name, type: String
-        check :id_format, type: String, default: '{{name}}'
+        check :id_format, type: String
         check :examples, item_type: Provider::Terraform::Examples, type: Array, default: []
         check :virtual_fields,
               item_type: Provider::Terraform::VirtualFields,

products/accesscontextmanager/api.yaml

@@ -267,14 +267,15 @@ objects:
                           Format: "major.minor.patch" such as "10.5.301", "9.2.1".
                       - !ruby/object:Api::Type::Enum
                         name: 'osType'
+                        required: true
                         description: |
                           The operating system type of the device.
                         values:
-                        - :OS_UNSPECIFIED
-                        - :DESKTOP_MAC
-                        - :DESKTOP_WINDOWS
-                        - :DESKTOP_LINUX
-                        - :DESKTOP_CHROME_OS
+                          - :OS_UNSPECIFIED
+                          - :DESKTOP_MAC
+                          - :DESKTOP_WINDOWS
+                          - :DESKTOP_LINUX
+                          - :DESKTOP_CHROME_OS
   - !ruby/object:Api::Resource
     name: 'ServicePerimeter'
     # This is an unusual API, so we need to use a few fields to map the methods
@@ -376,6 +377,10 @@ objects:
               A list of GCP resources that are inside of the service perimeter.
               Currently only projects are allowed.
               Format: projects/{project_number}
+            at_least_one_of:
+              - status.0.resources
+              - status.0.access_levels
+              - status.0.restricted_services
             item_type: Api::Type::String
           - !ruby/object:Api::Type::Array
             name: 'accessLevels'
@@ -390,6 +395,10 @@ objects:
               be empty.
 
               Format: accessPolicies/{policy_id}/accessLevels/{access_level_name}
+            at_least_one_of:
+              - status.0.resources
+              - status.0.access_levels
+              - status.0.restricted_services
             item_type: Api::Type::String
           - !ruby/object:Api::Type::Array
             name: 'restrictedServices'
@@ -399,4 +408,8 @@ objects:
               `storage.googleapis.com` is specified, access to the storage
               buckets inside the perimeter must meet the perimeter's access
               restrictions.
+            at_least_one_of:
+              - status.0.resources
+              - status.0.access_levels
+              - status.0.restricted_services
             item_type: Api::Type::String

products/accesscontextmanager/terraform.yaml

@@ -19,6 +19,7 @@ overrides: !ruby/object:Overrides::ResourceOverrides
       update_minutes: 6
       delete_minutes: 6
     autogen_async: true
+    id_format: "{{name}}"
     import_format: ["{{name}}"]
     examples:
       - !ruby/object:Provider::Terraform::Examples

products/appengine/api.yaml

@@ -94,6 +94,7 @@ objects:
               Example: 12345.
           - !ruby/object:Api::Type::Enum
             name: 'sslManagementType'
+            required: true
             description: |
               SSL management type for this domain. If `AUTOMATIC`, a managed certificate is automatically provisioned.
               If `MANUAL`, `certificateId` must be manually specified in order to configure SSL for this domain.
@@ -331,16 +332,19 @@ objects:
               - :REDIRECT_HTTP_RESPONSE_CODE_307
           - !ruby/object:Api::Type::NestedObject
             name: 'script'
+            # TODO (mbang): Exactly one of script, staticFiles, or apiEndpoint must be set
             description: |
               Executes a script to handle the requests that match this URL pattern. 
               Only the auto value is supported for Node.js in the App Engine standard environment, for example "script:" "auto".
             properties:
             - !ruby/object:Api::Type::String
               name: 'scriptPath'
+              required: true
               description: |
                 Path to the script from the application root directory.
           - !ruby/object:Api::Type::NestedObject
             name: 'staticFiles'
+            # TODO (mbang): Exactly one of script, staticFiles, or apiEndpoint must be set
             description: |
               Files served directly to the user for a given URL, such as images, CSS stylesheets, or JavaScript source files. Static file handlers describe which files in the application directory are static files, and which URLs serve them.
             properties:
@@ -403,10 +407,14 @@ objects:
             name: 'zip'
             description: 'Zip File'
             required: false
+            at_least_one_of:
+              - deployment.0.zip
+              - deployment.0.files
             properties:
               - !ruby/object:Api::Type::String
                 name: 'sourceUrl'
                 description: 'Source URL'
+                required: true
               - !ruby/object:Api::Type::Integer
                 name: 'filesCount'
                 description: 'files count'
@@ -417,6 +425,9 @@ objects:
               Manifest of the files stored in Google Cloud Storage that are included as part of this version.
               All files must be readable using the credentials supplied with this call.
             required: false
+            at_least_one_of:
+              - deployment.0.zip
+              - deployment.0.files
             key_name: 'name'
             key_description: |
               name of file
@@ -428,6 +439,7 @@ objects:
                     SHA1 checksum of the file
                 - !ruby/object:Api::Type::String
                   name: 'sourceUrl'
+                  required: true
                   description: |
                     Source URL
       - !ruby/object:Api::Type::NestedObject
@@ -438,6 +450,7 @@ objects:
         properties:
           - !ruby/object:Api::Type::String
             name: 'shell'
+            required: true
             description: |
               The format should be a shell command that can be fed to bash -c.
       - !ruby/object:Api::Type::String

products/appengine/terraform.yaml

@@ -14,8 +14,7 @@
 --- !ruby/object:Provider::Terraform::Config
 overrides: !ruby/object:Overrides::ResourceOverrides
   FirewallRule: !ruby/object:Overrides::Terraform::ResourceOverride
-    id_format: "{{project}}/{{priority}}"
-    import_format: ["{{project}}/{{priority}}"]
+    import_format: ["apps/{{project}}/firewall/ingressRules/{{priority}}"]
     examples:
       - !ruby/object:Provider::Terraform::Examples
         name: "app_engine_firewall_rule_basic"
@@ -25,7 +24,6 @@ overrides: !ruby/object:Overrides::ResourceOverrides
         test_env_vars:
           org_id: :ORG_ID
   StandardAppVersion: !ruby/object:Overrides::Terraform::ResourceOverride
-    id_format: "apps/{{project}}/services/{{service}}/versions/{{version_id}}"
     import_format: ["apps/{{project}}/services/{{service}}/versions/{{version_id}}"]
     mutex: "apps/{{project}}"
     parameters:
@@ -73,8 +71,8 @@ overrides: !ruby/object:Overrides::ResourceOverrides
     exclude: true
   DomainMapping: !ruby/object:Overrides::Terraform::ResourceOverride
     self_link: 'apps/{{project}}/domainMappings/{{domain_name}}'
-    id_format: "{{domain_name}}"
-    import_format: ["{{domain_name}}"]
+    id_format: 'apps/{{project}}/domainMappings/{{domain_name}}'
+    import_format: ['apps/{{project}}/domainMappings/{{domain_name}}']
     examples:
       - !ruby/object:Provider::Terraform::Examples
         name: "app_engine_domain_mapping_basic"

products/bigquery/api.yaml

@@ -52,6 +52,7 @@ objects:
               description: An email address of a Google Group to grant access to.
             - !ruby/object:Api::Type::String
               name: 'role'
+              required: true
               description: |
                 Describes the rights granted to the user specified by the other
                 member of the access object. Primitive, Predefined and custom

products/bigquery/terraform.yaml

@@ -15,8 +15,7 @@
 legacy_name: 'bigquery'
 overrides: !ruby/object:Overrides::ResourceOverrides
   Dataset: !ruby/object:Overrides::Terraform::ResourceOverride
-    id_format: "{{project}}:{{dataset_id}}"
-    import_format: ["{{project}}:{{dataset_id}}", "{{project}}/{{dataset_id}}", "{{dataset_id}}"]
+    import_format: ["projects/{{project}}/datasets/{{dataset_id}}"]
     delete_url: projects/{{project}}/datasets/{{dataset_id}}?deleteContents={{delete_contents_on_destroy}}
     examples:
       - !ruby/object:Provider::Terraform::Examples

products/bigtable/api.yaml

@@ -64,23 +64,26 @@ objects:
           Long form description of the use case for this app profile.
       - !ruby/object:Api::Type::Boolean
         name: 'multiClusterRoutingUseAny'
-        conflicts:
-          - singleClusterRouting
+        exactly_one_of:
+          - single_cluster_routing
+          - multi_cluster_routing_use_any
         description: |
           If true, read/write requests are routed to the nearest cluster in the instance, and will fail over to the nearest cluster that is available
           in the event of transient errors or delays. Clusters in a region are considered equidistant. Choosing this option sacrifices read-your-writes
           consistency to improve availability.
         input: true
       - !ruby/object:Api::Type::NestedObject
         name: 'singleClusterRouting'
-        conflicts:
-          - multiClusterRoutingUseAny
+        exactly_one_of:
+          - single_cluster_routing
+          - multi_cluster_routing_use_any
         description: |
           Use a single-cluster routing policy.
         input: true
         properties:
             - !ruby/object:Api::Type::String
               name: 'clusterId'
+              required: true
               description: |
                 The cluster to which read/write requests should be routed.
             - !ruby/object:Api::Type::Boolean

products/bigtable/terraform.yaml

@@ -14,7 +14,7 @@
 --- !ruby/object:Provider::Terraform::Config
 overrides: !ruby/object:Overrides::ResourceOverrides
   AppProfile: !ruby/object:Overrides::Terraform::ResourceOverride
-    id_format: "{{project}}/{{instance}}/{{app_profile_id}}"
+    id_format: "projects/{{project}}/instances/{{instance}}/appProfiles/{{app_profile_id}}"
     import_format: ["projects/{{project}}/instances/{{instance}}/appProfiles/{{app_profile_id}}"]
     examples:
       - !ruby/object:Provider::Terraform::Examples

products/binaryauthorization/api.yaml

@@ -96,6 +96,7 @@ objects:
                     See the documentation on publicKey cases below for details.
                 - !ruby/object:Api::Type::String
                   name: asciiArmoredPgpPublicKey
+                  # TODO (mbang): Exactly one of asciiArmoredPgpPublicKey or pkixPublicKey must be set
                   description: |
                     ASCII-armored representation of a PGP public key, as the
                     entire output by the command
@@ -108,6 +109,7 @@ objects:
                     be overwritten by the API-calculated ID.
                 - !ruby/object:Api::Type::NestedObject
                   name: pkixPublicKey
+                  # TODO (mbang): Exactly one of asciiArmoredPgpPublicKey or pkixPublicKey must be set
                   description: |
                     A raw PKIX SubjectPublicKeyInfo format public key.
                     
@@ -178,6 +180,7 @@ objects:
           properties:
             - !ruby/object:Api::Type::String
               name: namePattern
+              required: true
               description: |
                 An image name pattern to whitelist, in the form
                 `registry/path/to/image`. This supports a trailing * as a
@@ -202,6 +205,7 @@ objects:
           properties:
             - !ruby/object:Api::Type::Enum
               name: evaluationMode
+              required: true
               description: How this admission rule will be evaluated.
               values:
                 - :ALWAYS_ALLOW
@@ -221,6 +225,7 @@ objects:
               item_type: Api::Type::String
             - !ruby/object:Api::Type::Enum
               name: enforcementMode
+              required: true
               description: |
                 The action when a pod creation is denied by the admission rule.
               values:

products/binaryauthorization/terraform.yaml

@@ -15,7 +15,6 @@
 overrides: !ruby/object:Overrides::ResourceOverrides
   Attestor: !ruby/object:Overrides::Terraform::ResourceOverride
     import_format: ["projects/{{project}}/attestors/{{name}}"]
-    id_format: "{{project}}/{{name}}"
     examples:
      - !ruby/object:Provider::Terraform::Examples
       name: "binary_authorization_attestor_basic"
@@ -39,7 +38,7 @@ overrides: !ruby/object:Overrides::ResourceOverrides
     properties:
       name: !ruby/object:Overrides::Terraform::PropertyOverride
         custom_flatten: 'templates/terraform/custom_flatten/name_from_self_link.erb'
-        custom_expand: 'templates/terraform/custom_expand/binaryauthorization_attestor_name.erb'
+        custom_expand: 'templates/terraform/custom_expand/shortname_to_url.go.erb'
       attestationAuthorityNote.noteReference: !ruby/object:Overrides::Terraform::PropertyOverride
         custom_expand: 'templates/terraform/custom_expand/container_analysis_note.erb'
         diff_suppress_func: 'compareSelfLinkOrResourceName'
@@ -54,7 +53,7 @@ overrides: !ruby/object:Overrides::ResourceOverrides
       attestationAuthorityNote.publicKeys.id: !ruby/object:Overrides::Terraform::PropertyOverride
         default_from_api: true
   Policy: !ruby/object:Overrides::Terraform::ResourceOverride
-    id_format: "{{project}}"
+    id_format: "projects/{{project}}"
     import_format: ["projects/{{project}}"]
     custom_code: !ruby/object:Provider::Terraform::CustomCode
       constants: 'templates/terraform/constants/binaryauthorization_policy.erb'