Allow plugins to export custom functions

[thegedge]

It would be interesting to allow the set of functions allowed in interpolations to be an extension point for plugins. For example, one could write a plugin that exports a function to encode a string into base64, split a root CIDR block into a list of smaller CIDR blocks that covers the root block, or read a string from a URL.

[thegedge]

I should note that you can create a plugin that exports resources to compute things for you right now, but I think functions would be more appropriate than read-only + compute-once resources in many/most cases.

[phinze]

Yep totally - this is something I've thought about as well. The use case is absolutely clear, but the implementation is very important to get right so we don't shoot our future selves in the foot with a difficult-to-support ecosystem of configs. I will join @mitchellh in the thinking tag here. 😀

[thegedge]

Just glad to hear that there's interest in this. Looking forward to what you come up with :)

[steve-jansen]

+1 on CIDR functions

[apparentlymart]

Heh... this is the first time I came across this issue, but funny that two of your three example use-cases have been implemented in the mean time:

• implements base64{enc,dec} interpolation funcs #3325 implemented Base64 encoding
• Various interpolation functions for CIDR range manipulation. #3127 will implement CIDR manipulation once it's merged

No URL parsing yet, but I think that's suitably generic that it could be in core too.

I think continuing to add things to core Terraform will be sufficient at least until a function comes along that is truly specific to one particular set of resources, e.g. a function for parsing AWS IAM policy documents.

[polvoazul]

Why not support a "sh" function in variable interpolation, and then the user can run anything he wants?

[theist]

This is also something I'll like to have. My use case is to remove account section and region section of AWS ARNs to store them in policy documents. This is provider specific and can not be a generic function, and I want this because this allows to reuse policies with resources across regions.

For example if I use in the resource section "${aws_sqs_queue.my_queue.arn}" I get a computed ARN like 
arn:aws:sqs:us-west-1:000111223334455:my-queue and I want something to transform it to arn:aws:sqs:::my-queue. This fine string manipulation looks good for a plugin (imho)

[jhoos]

As another example usecase: we have a standard set of tags that we like to apply across all of our AWS resources, and the values of some of the tags vary per-resource (usually on the names of things). Being able to create a function that returned a map of those tags would be an effective way to make sure we're tagging everything consistently.

For example, something like mytags("myenv", "myService") which could return { environment = "${arg.env}", name = "${arg.service}", selector="${arg.env}-${arg.service}" } so that for each AWS resource we could simply add tags = mytags("myenv", "myService") to each resource and be done with it. Doing this with a data source or module would be unwieldy.

[apparentlymart]

It seems that this issue has grown to represent a couple different use-cases. That's fine (we may choose to split them later if one gets addressed first, but we'll see), but I just wanted to write them down here for future reference:

Functions as part of a provider plugin

A few times now we've seen situations where it would be handy to have a function that does something provider-specific, like parsing an AWS ARN. It's felt weird to add such functions to Terraform Core, but they could potentially be at home in the aws provider itself.

A possible design we considered for this is to extend HCL with a "function namespace" syntax, allowing extension functions to be placed in a namespace named after the provider itself. For example, if we choose :: as the function namespace delimiter then we might see a function aws::parsearn for parsing AWS ARNs.

This is not something we plan to do in the very near future because it may require some changes to HCL and so we want to make sure those changes feel right for HCL in general (since HCL is used by more than just Terraform) before moving forward with it. But we do see how it would be useful, and would like to do something like it eventually.

User-defined functions in configuration

With the above implemented, you could potentially write a provider plugin that only includes functions and use that to deal with user-defined functions, which are truly specific to a particular configuration or set of configurations and make no sense to share with others.

However, we've also thought about offering some syntax for this inside the configuration language itself, allowing local functions to be written as well as local values.

Some challenges/questions down this path are:

• Once user-defined functions are possible, it seems inevitable that we'll want to enable sharing of them between modules and between configurations, so we need to figure out how to define them to enable such re-use. Perhaps they are exported as part of a module? or perhaps there's some other new reusability construct for functions that doesn't also imply the possibility of creating infrastructure?
• Are functions written in HCL itself, or in some other language? (Keeping in mind the general restrictions in the section below.)
• Do user-defined functions participate in the dependency graph? Functions being in the dependency graph is not part of the Terraform model today -- they run only as part of configuration evaluation. In order to avoid a drastic redesign of how evaluation works in Terraform (which may not actually be tractable in practice) we may have to limit functions to operating only on their own arguments, and not anything else in the configuration.

General constraints on extension functions

However they are implemented, there are some constraints that must hold for functions in Terraform's current model. While some of these may be able to change slightly, I think for initial design we should assume they are fixed design constraints:

• All functions in Terraform must behave as pure functions. If this constraint is not followed, Terraform cannot keep its promise that the actions taken during apply will match the actions planned (unless there is an error). Terraform has consistency checks that are likely to detect impure function behavior, but they are not very user-friendly so Terraform's existing functions are designed to behave as pure, at least under normal operating conditions.
• Functions must be local-only and not make any external API calls. Terraform evaluates functions at various points during its work and expects function evaluation to be relatively cheap in time and money.
• Functions cannot participate in the dependency graph, as described in the previous section. As a consequence, it's likely that we'd restrict them to produce results based only on their own arguments, and not on anything else that might normally be interpolatable from elsewhere in the configuration.

As a consequence of the above restrictions, I expect that functions-in-providers support would be implemented by running the functions in an unconfigured provider context, constrained similarly to the context used for validation, and that user-defined functions would be implemented in HCL itself or some other language that can guarantee "pure function" behavior.

---

As was implied in other earlier comments above, this is one of those issues where the design of it is the hardest part -- making sure whatever is added is reliable, sustainable, and interacts well with other features -- with the subsequent implementation then probably relatively straightforward. Therefore we're going to keep "thinking" on this for now, representing that there's still some more design work to do. We've been focused on configuration language work for a long time now, so I expect in the very near future we're going to cast our attention into some other areas that have been somewhat neglected during development of v0.12.0. We do still intend to return to this problem of extensible functions eventually, though.

[pll]

@jhoos I've accomplished this by doing something like the following:

tags = "${merge(var.tags, map("Name", "my unique name"))}"

[cc-stjm]

A use case I have would be for some string operations. A common pattern we have for resource names is "(current-git-branch)-(resource-specific-suffix)". These can get quite long, and sometimes long branch names will mean it goes over the limit of a given resource name (and of course each resource name has different limits).

What I'd love to be able to define is a function that would combine these, and truncate the git branch name as needed (possibly with a sha of the git branch name as a suffix so that 2 git branches differing in only the last character don't clash). This would be wonderful as a pure string manipulation function that I could define once and use everywhere, but is a nightmare if I have to do it in every single resource.

(Even better would be if somehow the function could know the resource name limits for the resource in which it was being invoked, but I imagine that's a step too far)

[bmaltais]

Indeed. I would like to create a function that would essentially do this and be able to call it to generate a compliant name for all the resources that need it by simply passign the right arguments:

locals {
  azurecaf_naming_convention-Project-law-replace = replace("${var.env}CLD-${var.group}-${var.project}", "_", "-")
  azurecaf_naming_convention-Project-law-regex   = regex("[0-9A-Za-z-]+", local.azurecaf_naming_convention-Project-law-replace)
  azurecaf_naming_convention-Project-law-54      = substr(local.azurecaf_naming_convention-Project-law-regex, 0, 54)
  azurecaf_naming_convention-Project-law-59      = substr("${local.azurecaf_naming_convention-Project-law-54}-${local.unique_Logs}", 0, 59)
  azurecaf_naming_convention-Project-law-result  = "${local.azurecaf_naming_convention-Project-law-59}-law"
}

resource "azurerm_log_analytics_workspace" "Project-law" {
  # name                = azurecaf_naming_convention.Project-law.result
  name                = local.azurecaf_naming_convention-Project-law-result
  location            = azurerm_resource_group.Logs-rg.location
  resource_group_name = azurerm_resource_group.Logs-rg.name
  sku                 = "PerGB2018"
  tags                = var.tags
}

That way I could create different functions for different Azure resources with different weird name rules instead of having to declare a local for every resource and repeat the code all over the place.

[Ardalan-Saberi]

How about extending hcl by a function that can pass a query into external data resources inline... something like the external_lookup function below:

data "external" "my_function" {
    program = ["python", "${path.module}/my_function.py"]
}

resource "instance" "foo" {
    name = external_lookup("my_function", {query="foo"}).result.instance_name
    ...
}

[ocervell]

I think the main use case here for enterprises is to simplify naming conventions and extracting things from them, to simplify end-user configs like shown in comment above. Custom functions would help greatly with that !

[ghostsquad]

regarding the need for functions to be pure, it makes sense that functions would really just be pre-baked expressions of all the existing "primitive" functions (and expression language) that is available.

Sounds like the only problem to solve here is distribution and how functions are referenced?

[mcintyre321]

#31419 add jsonindent function

[apparentlymart]

Hi @MallikharjunaTeja! Thanks for your interest in contributing.

The implementation of the feature described here is non-trivial and so if you are interested in working on it then I think we'd prefer to first discuss the design you have in mind. We wouldn't want to encourage you to implement it and then afterwards have to ask you to significantly change your approach if it were not a good fit for Terraform Core's architecture.

I previously wrote an initial prototype implementation of this in #31225, and my opening comment there includes some notes about some remaining design questions we need to answer before we could proceed with an implementation of this mechanism.

[schollii]

@polvoazul

Why not support a "sh" function in variable interpolation, and then the user can run anything he wants?

This can already be done with local_exec and introduces issue, namely platform dependency (now everywhere your terraform code is used needs to have the right shell or utilities called by the shell like jq or cat or sed which are not platform independent eg on macos sed has a few different switches than the one used on ubuntu).

[apparentlymart]

Hi all!

A first round of this is coming in the forthcoming Terraform v1.8. Of course, this is one of those things where having it just in Terraform Core doesn't really achieve anything because there are not yet any functions to call, and providers can't offer functions until the SDK and framework offer a way to do that, so the work in this repository is done for now (and so I'm going to close this issue) but it will take some time for the surrounding ecosystem to make use of this.

The provider framework documentation section Functions has information on how this feature is exposed for provider developers. Work on those docs will continue as this feature rolls out.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.