Allow provider to get value from config or prior state explicitly

[rcohenma]

I work on the OCI terraform provider
We are encountering a problem in our update functions on lists of nested objects.
Terraform's GetOk/GetOkExists do not allow us to distinguish whether a property value comes from the customer's config or the statefile.
This is a problem when we try to update a list of nested objects that have optional parameters.
We had solved this problem before by converting the TypeList into a TypeSet but now we are adding a new feature where the order of the list matters to the backend service so we had to disable the update on that List because we were not passing to the service what the user had in its config.

Example of the problem:
Lets say i have a TypeList paramter "listPar" in which its elements have optional properties "optA" and "optB" of typeString
Lets say the user initially createad a resource with one element on the list

resource "..." "..." {
   listPar {
      optB = "1"
   }
}

now the user wants to update to add another element at the beginning of the list:

resource "..." "..." {
   listPar {
      optA = "0"
   }
   listPar {
      optB = "1"
   }
}

In this case because of the existing statefile and because getOk/getOkExists get the value of a property from the statefile if it is not in the config then we would be sending to the service something that the user did not specify in the config

UpdateRequest {
    listPar = [
        {optA=0, optB=1},
        {optB=1},
    ]
}

The statefile contains

 "... .listPar.0.optB" = 1

So on the Update when we do GetOk on the properties of the first element we will be getting a value for optB even if the user did not specify it in the config.

Thanks

[jbardin]

Hi @rcohenma,

This is a known limitation of the existing provider SDK, and due to the internal structure of helper/schema and the old protocol, it's not really solvable.

The new plugin protocol has provisions to supply the necessary information to providers, and a future SDK will make this distinction possible.

[mingfang]

This is critical problem.
The new SDK to fix this will take a very long time.
Until then, will it help if you made getRaw() public?

[jbardin]

Hi @mingfang,

There is no workaround for the existing SDK, because the information simply is not there to expose (and when it is, it's now consistent between execution phases and various configuration+state combinations). Exposing more details about the internals (which unfortunately GetOkExists does in a manner of speaking) only leads to more confusion when it only works in very narrow cases.

It may be possible to handle this in a more consistent way post-0.12 and before a new SDK is complete, but in the meantime we need to stick to the API that helper/schema provided, especially when there is already considerable change happening beneath it.

[mingfang]

@jbardin I think exposing more is better than less. GetOkExists is great because it doesn't check the Zero() value unlike GetOk. In fact, I want to call GetRaw to get to the getResult stuct and do the checking myself.

I think in general Terraform is trying to impose its rules on how other systems operate. In particular things like zeros, nulls, computed values, and in this case the actual config values, needs to be exposed to the provider and let it decide how to handle them.

So YES, we need a post-0.12 effort to simply expose more of the internals and not change helper/schema behavior so that providers can deal with the real world properly. Here's an example of how I had to use golang/reflect hacks to workaround the recent Terraform behavior change we talked about mingfang/terraform-provider-k8s@6c013c9. It's ugly but was necessary to my provider.

[mingfang]

@jbardin It looks like your fix here hashicorp/terraform#21068 also fixes this issue. I tested it and now GetOkExists only returns what's in the config. Thank you.

[mingfang]

Actually this is still a problem for me.
My schema has optional + computed fields.
For example, if my starting state looks like this

env {
    name  = "foo"
    value = "bar"
 }

And then in my config I replace value with value_from, then the plan ends up looking like this

env {
   name  = "foo"
   value = "bar"

   + value_from {
      + field_ref {
         + field_path = "status.podIP"
       }
    }
 }

This fails because I also need to remove value. However both GetOk/GetOkExists returns true for value.

Looking at https://github.com/hashicorp/terraform/blob/master/plans/objchange/plan_valid.go I know that the config data is there. I just need a way to get to it.

[apparentlymart]

As of right now, the SDK is working with Terraform 0.11 data types rather than 0.12 datatypes so that it can support both Terraform 0.11 and 0.12 and to minimize the changes required to existing provider codebases. The detail you're seeing in the 0.12 internal data types is lost in that conversion to Terraform 0.11 types, because similar concepts did not exist in 0.11.

The new type system is intentionally designed to be a superset of JSON so that remote all concepts from a remote JSON API can be represented faithfully, and we do intend to expose the full power of that type system for use by providers, but we need to finish the v0.12.0 release before we can start on the significant SDK rewrite required to make that be true.

[mingfang]

I'm sure the new SDK will be a huge improvement, but it's likely to be at least a year away.
What can we do in the meantime?
To debug, I created a CustomizeDiff function and printed out the resourceDiff values.
It looks like resourceDiff.config.Raw should contain the raw config but it doesn't.
Is that a bug?

Is there any way to get to the actual config?
I'm willing use reflection to get to non-public values.

[apparentlymart]

Developers of other providers have worked within the conventions the SDK expects and made compromises. That is the main path available right now.

Another possibility, if you can't work within the constraints of the existing SDK, is to ignore the SDK and code directly against the provider protocol. That is not an easy option by any means, but it is technically possible.

For a personal project (not an official HashiCorp project, and I make no promises about maintaining compatibility or maintaining it at all) I wrote my own SDK which only supports the Terraform v0.12 protocol and thus produces plugins that don't work with Terraform v0.11. In return, it has access to all of the information available in the new protocol. While I can't recommend this as a sustainable option, forking my repository there and using it as a dependency could be a compromise between using the existing SDK and writing directly to the protocol. (Don't depend on that repository directly, because I'm very likely to make breaking changes to it.)

If you think you can get the information you need using values already available in the internals of the existing SDK then a similar approach could apply there: fork the SDK and make whatever changes you need and use that in your provider.

None of these alternative approaches are appropriate for any provider that is participating in the provider development program -- official SDK use is required for that, so that HashiCorp engineers can collaborate on maintenance -- but for a third-party plugin, just talking the right protocol is all that's needed, no matter how that is achieved. The ability to install third-party plugins automatically via terraform init is planned too, which will make third-party plugins easier to use.

[mingfang]

@apparentlymart I submitted this PR hashicorp/terraform#21218 to pass the original config when making the call to the customizeDiff callback.

After looking through the Terraform codebase, it looks like changing GetOk/GetOkExists is not possible. Therefore my PR has the least impact but yet provides the critical information my(and probably others) provider needs.

Please review my PR.

[paultyng]

We are currently pursuing splitting the SDK from the main TF repo to simplify versioning / releasing, etc., so this will need to wait until we can setup the new repo.

[bflad]

Hi folks 👋

As of version 2.8.0 of the Terraform Plugin SDK, there is experimental functionality to read the raw configuration, plan, and state data that was stored the beginning of the ApplyResourceChange, PlanResourceChange, ReadDataSource, and ReadResource RPCs:

• https://pkg.go.dev/github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema#ResourceData.GetRawConfig
• https://pkg.go.dev/github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema#ResourceData.GetRawPlan
• https://pkg.go.dev/github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema#ResourceData.GetRawState
• https://pkg.go.dev/github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema#ResourceDiff.GetRawConfig
• https://pkg.go.dev/github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema#ResourceDiff.GetRawPlan
• https://pkg.go.dev/github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema#ResourceDiff.GetRawState

More details and functionality surrounding the cty.Value returned by these methods can be found in the github.com/zclconf/go-cty/cty package.

Familiarity with protocol version 5 and which data is available with each RPC is encouraged before attempting to use this functionality (e.g. ReadResource only has state information). Raw plan data is not updated by any changes caused by a resource's CustomizeDiff function.

The SDK may not make it trivial to work with this new data yet. If you have specific feature requests surrounding this, please reach out by creating a new issue. If you have questions around this advanced functionality, please reach out by creating a new topic in HashiCorp Discuss. Thanks.

---

Aside: The Terraform Plugin Framework, under active development, also provides direct access to the configuration, plan (which is updated during plan modifications), and state data. There is a HashiCorp Learn tutorial and template repository available if you are interested in trying out that refreshed provider development model. See also: Which SDK Should I Use?

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.