Retry billing update in project creation

[crhino]

HCPF-2286

🛠️ Description

Building the provider locally and running many project creations at once, I see:

2024-11-14T10:39:46.626-0500 [DEBUG] provider.terraform-provider-hcp: {"code":14, "message":"failed to update billing account", "details":[]}
2024-11-14T10:39:46.626-0500 [WARN]  unexpected data: localhost/providers/hcp:stdout="Error trying to update billing account. Retrying in 10 seconds..."

and that all projects create successfully given time:

hcp_project.proj["project16"]: Creation complete after 1s [name=project16]
hcp_project.proj["project10"]: Still creating... [10s elapsed]
hcp_project.proj["project12"]: Still creating... [10s elapsed]
hcp_project.proj["project12"]: Creation complete after 11s [name=project12]
hcp_project.proj["project10"]: Creation complete after 11s [name=project10]

🏗️ Acceptance tests

• Are there any feature flags that are required to use this functionality?
• Have you added an acceptance test for the functionality being added?
• Have you run the acceptance tests on this branch?

Output from acceptance testing:

$ make testacc TESTARGS='-run=TestAccXXX'

...
=== RUN   TestAccProjectResource
--- PASS: TestAccProjectResource (5.78s)
PASS
ok  	github.com/hashicorp/terraform-provider-hcp/internal/provider/resourcemanager	8.092s

[pierluc-codes]

@crhino can you explain why the retries are necessary? Especially on a 503.

[crhino]

@crhino can you explain why the retries are necessary? Especially on a 503.

@pierluc-codes

Sure. This essentially stems from a race condition in the billing system. When creating a large number of projects, each of those projects needs to be added to the same billing account, as the TF provider does here. Those UpdateBilling calls conflict with each other and some of them return an Unavailable error, which is then translated to a 503. In general, I believe a 503 status code is considered reasonable to retry on anyways.

[pierluc-codes]

Approving, but I would encourage you to fix your race condition instead.