[WIP] Provider-contributed Functions

[apparentlymart]

This is the beginnings of work to evaluate the feasibility of and one possible language design for allowing normal functions to be contributed by providers.

This implementation exposes provider functions under the function namespace provider::NAME::, where NAME is the local name of the provider as declared in the required_providers block. For example, if hashicorp/aws had a hypothetical parse_arn function then it would appear like this under this implementation:

terraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"
    }
  }
}

locals {
  arn_parts = provider::aws::parse_arn("arn:aws:s3:::examplebucket/developers/design_info.doc")
}

output "arn_service" {
  value = local.arn_parts.service  
}

With what I have here so far, the basic mechanism works: providers can declare that they have certain functions available as part of their schema, and then Terraform will call a new CallFunction RPC each time on an unconfigured instance of the provider each it needs to evaluate one of the declared functions.

However, there's a fair amount more to do here before this could be shipped, including:

• It spins up an entirely new instance of the contributing provider for each individual function call, which is likely to be pretty slow and memory intensive for real plugins that run as child processes. We'll need to find some way to reuse these for multiple calls and then clean them up when we're finished.
• It doesn't make any attempts to guarantee that the provider-contributed functions correctly honor the contracts such as behaving as pure functions. Properly checking this is important because if a function doesn't uphold Terraform's expectations then it will cause confusing errors reported downstream, incorrectly blaming other components for the inconsistency.

There are also many external dependencies that will require some broader consensus on this path before we could possibly move forward to shipping, including but certainly not limited to:

• The plugin framework will need some reasonable way for a provider developer to implement and register functions, integrated with the same type system implementation it uses for other purposes.
• The language server will need awareness of provider-contributed functions in general and some way to learn about which specific functions are in scope for a particular module in order to offer code completion and hover tips.
• Terraform Registry will need some way to display documentation for provider-contributed functions as part of the documentation for a provider.

I'm sure there are other shortcomings that I've not thought of yet, too. I've also only wrote some minimal additional tests just to show the main case working, so it'll need considerably more automated tests before it could be shippable.

[flynnhandley]

Hey @crw @apparentlymart 👋

It looks like there are a few other PR's waiting on this PR.

Based on your comments @crw, it seems this feature is likely to be implemented.

Is there any way I can help progress it?

Cheers 🏝️

[crw]

The main thing is upvotes on the main issue. I am building an argument that Functions are the #1 most requested feature if you consider all of the upvotes on all of the various functions requests. However it is always a challenge to prioritize against other projects that also have a lot of user interest. In any case, thanks for the bump on this draft PR.

[apparentlymart]

This PR is essentially a feasibility prototype I built to see what seems solved already vs. what remains to be solved. I don't expect that this PR will actually ever be merged itself, and instead it's more likely that there will be another PR that achieves a similar effect but with different implementation details.

If you want to express interest in the general idea of provider-contributed functions I would suggest instead adding your upvotes to #2771, which is the issue representing the use-case as opposed to this PR which is representing only a prototype implementation of that use-case.

(No harm in voting here too if you want, but what I want to avoid is people only voting here and then those votes getting missed when I inevitably close out this PR without merging it.)

[A141864]

The main thing is upvotes on the main issue. I am building an argument that Functions are the #1 most requested feature if you consider all of the upvotes on all of the various functions requests. However it is always a challenge to prioritize against other projects that also have a lot of user interest. In any case, thanks for the bump on this draft PR.

That's a fair argument, however, I can see there are a lot of PRs for functions that are pending a feature that likely won't be delivered in the near future.

What are the tradeoffs of merging the existing PRs into terraform core?

[crw]

What are the tradeoffs of merging the existing PRs into terraform core?

Number one is the maintenance cost on an ever-increasingly more-complex code base. As an example, base36decode: once it is added, it cannot be removed, but this is a function that belongs in a bigger package of encoding/decoding functions. It will make it easier, in the long run, to hold the line now so that these functions can be more cleanly and clearly organized in the future.

The other problem that comes to mind are functions that do not have a clear one-size-fits-all implementation. As an example, deep merge. There are a couple of different deep merge proposals, it is clear different users have different expected behavior for a deep merge function. With external functions, you can choose the implementation that works best for your use case (or write your own).

We have accepted functions recently (startswith and endswith string functions), and we do have a process to champion functions we believe are core-enough to include. We are not taking an all-or-nothing approach, but are being fairly picky about what makes the cut for "core." Let me know if I can clarify any of the above. As always, this thinking is always in motion and changing, so please take this as a snapshot of the current thinking on this topic.

[apparentlymart]

This PR was only ever intended as a prototype to see what technical challenges might arise if we tried to do this, and it's become incredibly merge-conflicty as development has continued in the main branch.

I'm going to close this now just because this PR has served its purpose and will not see any more development in its current form. Others on the team are working towards a non-prototype design and implementation for this, and so although this PR might still serve as a useful reference point for them the future development in this area will happen in other PRs. My attention is elsewhere, so I'll leave those folks to decide how best to proceed and how much of this PR is useful beyond just being a feasibility prototype.