tfvarsdecode/tfvarsencode encoding functions

[gw0]

Current Terraform Version

Terraform v0.12.24

Use-cases

Built-in encoding functions are missing support for your HCL syntax format (missing hcldecode/hclencode). Because only other formats are supported, it is impossible to have everything in the same syntax as Terraform modules.

A common use case is to implement global/common values. The obvious option of using common_vars.tfvars is deprecated and results in Warning: Value for undeclared variable and in future this will be an error (#22004). The alternative of preparing a wrapper Bash script to export env variables looks like a hack. So the only way is to implement something like globally defined locals in common_vars.yaml (idea from Terragrunt) with:

locals {
  common_vars = yamldecode(file("../common_vars.yaml")),
}

In this situation it seems weird to use YAML for common values and HCL for everything else (Terraform files). The solution is to expose support for hcldecode (and maybe even hclencode), that are already implemented somewhere in Terraform source code.

Attempted Solutions

As mentioned, for that use case the only solution is to use YAML or JSON.

Proposal

Support for hcldecode (and maybe even hclencode) in order to load globally defined locals like:

locals {
  common_vars = hcldecode(file("../common_vars.hcl")),
}

[apparentlymart]

Hi @gw0,

We don't have hcldecode and hclencode functions because HCL is a schema-based meta-language and so there isn't a defined way to just parse arbitrary HCL into a data structure in HCL's type system without describing which HCL-based language you are intending to work with.

However, you mentioned .tfvars here which makes me think you are talking about the "tfvars language" rather than HCL in general. The "tfvars language" is defined as being a sequence of top-level HCL arguments that are evaluated in a context where no external variables or functions are available, and so that language is much more amenable to having a jsondecode-like function defined for it.

For that reason, I'd suggest as a compromise a function tfvarsdecode, which parses the string you give in the same way that Terraform would normally parse a .tfvars file, and returns the result as an object value where the top-level arguments in the string are attributes in the resulting object. If someone tried to pass a valid Terraform .tf file to tfvarsdecode then it would fail with a message like "resource blocks are not expected here", because the main Terraform language includes constructs that the "tfvars language" does not. This would also be true for other HCL-based languages, like HashiCorp Nomad's jobspec.

A further "design hazard" here is that the tfvars language comes in two variants: one based on HCL native syntax and the other based on JSON. However, the .tfvars.json format would, I think, always produce the same result as jsondecode of the same characters, so I think it would be okay to make tfvarsdecode support only the HCL native syntax version and have its documentation suggest using jsondecode if you want to load a .tfvars.json file.

---

FWIW, an existing pattern I've seen for "global values" is to write a data-only module and call it from each configuration using a module block. Although the documentation about data-only modules on that page discusses modules containing data resources, a module containing only output blocks is also a valid and extreme case of data-only module:

output "aws_account_id" {
  value = "235346456753456"
}

module "globals" {
  source = "../globals"
}

provider "aws" {
  # ...
  allowed_account_ids = [module.globals.aws_account_id]
}

This works with Terraform today, and potentially allows the module to further encapsulate details that might make later refactoring easier, such as potentially obtaining some values via data resources later, or dynamically generating certain values based on input variables for situations where the patterns are systematic across different environments.

[gw0]

Thank you for the explanation. If I understand correctly, it would then be possible to create Terraform-specific HCL functions hcltfdecode/hcltfencode?

Anyway, data-only modules for implementing "common values" are a viable option. But I ended up implementing "hierarchical common values" with the help of Terragrunt read_terragrunt_config(). It works great if you use Terragrunt.

[ericnorris]

Hey @apparentlymart, apologies for bumping an old thread - I've recently run into a need for a tfvarsencode and wanted to note that there's crossover between this and a tfe provider issue.

Since we're using terraform (via tfe) to upload HCL variables to Terraform Cloud, we end up having to do a hacky replace(jsonencode(foo), ... instead of something like hclencode(foo). It seems that other people are running into the same issue, and hopefully it's helpful to have another concrete example of where this would be useful.

[apparentlymart]

hashicorp/terraform-provider-tfe#188 seems to speak to a different meaning of tfvarsencode than we've been discussing here so far: instead of encoding an object into the syntax expected for an entire .tfvars file, I guess it would instead require a function to take an arbitrary value and return an HCL expression that would evaluate to that value.

It's unfortunate that the Terraform Cloud API was defined to take variables in HCL syntax instead of a more convenient API-friendly format, and therefore in turn the hashicorp/tfe provider needed to pass on that oddity in its interface. Given that this is an odd implementation detail of the Terraform Cloud API, which is unlikely to be true in any other context, I think it would be preferable for the hashicorp/tfe provider to itself accept arbitrary data structures and internally encode them to the expected HCL subset itself for submission to the remote API, rather than forcing a module author to explicitly encode into this Terraform Cloud-specific syntax. 🤔

Let's keep this issue about the original topic of a function for encoding and decoding the .tfvars file format, rather than arbitrary standalone HCL expressions. We could still potentially have a function for arbitrary expressions, but I think it's better to explore solutions for that in the hashicorp/tfe provider specifically first, because that's both easier to design (the hashicorp/tfe provider can "just know" what subset of HCL Terraform Cloud is expecting) and I think results in a more readable Terraform configuration in the end. If the maintainers of that provider disagree then please let us know by opening a separate issue to talk about that!

[ericnorris]

@apparentlymart I'm not sure I agree that this is different, but it may be the way I'm parsing (pun intended) the original poster.

So the only way is to implement something like globally defined locals in common_vars.yaml (idea from Terragrunt) with:

locals {
  common_vars = yamldecode(file("../common_vars.yaml")),
}

When common_vars is decoded, it will be an arbitrary value, likely a map / object type. For example, the file probably looks like:

foo: bar

baz:
  buzz: true

...which, when yamldecoded, can then be used by the user w/ local.common_vars.foo, etc. The OP asks for an hcldecode function that, if it were matching yamldecode, could do the exact same thing:

foo = "bar"

baz = {
  buzz = true
}

...i.e. this could be hcldecoded, and also be used by the user w/ local.common_vars.foo.

The OP also asks for the counterpart ("...(and maybe even hclencode) ..."), which I would expect would be the exact same as yamlencode, i.e. the opposite of yamldecode. This hypothetical hclencode function that I'm talking about is exactly what is needed for the terraform-provider-tfe issue I mentioned, in the same way that if that resource took YAML I would need the yamlencode function.

I believe that there may be an objection that hclencode and hcldecode functions are somewhat non-sensical since HCL is paired with a schema, but the primitives of HCL allow you to encode maps and objects, etc. just like YAML and JSON, so I don't fully see a problem here. That said, it seems you address that objection by talking about tfvarsencode, as a way to be explicit that you are only encoding these primitives, which brings me to:

We could still potentially have a function for arbitrary expressions, but I think it's better to explore solutions for that in the hashicorp/tfe provider specifically first, because that's both easier to design (the hashicorp/tfe provider can "just know" what subset of HCL Terraform Cloud is expecting)...

It may be worth clarifying the the "subset of HCL Terraform Cloud is expecting", is, unless I'm missing something, the exact same subset of a single expression in a .tfvars file. By definition, the proposed tfvarsencode would need to have this functionality inside of it.

That is, tfvarsencode could be implemented by calling tfvarencode (the name doesn't matter to me) like:

// this code is non-functional, and only meant to be illustrative
func tfvarsencode_impl() {
  for _, variable := range variables {
    fmt.Sprintf("%s = %s\n", variable.name, tfvarencode_impl(variable))
  }
}

...and then users like myself could call:

resource "tfe_variable" "foo" {
  key          = "foo"
  value        = tfvarencode(local.foo)
  category     = "terraform"
  hcl          = true

  // ...
}

It may be that I'm not seeing this the same way as you, but if tfvarsencode worked at all like the above, it would solve both problems at once. Hence why I think this is in fact on-topic with the original request.

[gw0]

If it is unclear how to write a generic HCL decoding function (equivalent to yamldecode), you can look at read_terragrunt_config` in Terragrunt (https://terragrunt.gruntwork.io/docs/reference/built-in-functions/#read_terragrunt_config).

[tullydwyer]

As a user it feels weird that Terraform is able to decode JSON/YAML but not tfvars.

Due to specific requirements we are trying to reference multiple tfvars files in multiple git submodules without adding them to the Terraform project input variables:

locals {
  git_submodule_vars = tfvarsdecode(file("../git_submodule_folder/git_submodule_vars.tfvars")),
}

But it looks like we may need to switch the remote vars to YAML to achieve this as simply as above (which is not ideal).

[rwblokzijl]

We have a usecase for hclencode. We generate some boilerplate terraform code (we want a human readable format). For this we want to take a dynamically generated datastructure and create a .tfvars based on that.

Currently we use the following workaround (requires jq)

data "external" "hclencode" {
  program = ["bash", "-c", "echo '${jsonencode(local.to_hcl)}' | jq | sed -r 's/\"(.*)\": /\\1 = /;s/^(  [^ ].*),$/\\1/' | tail -n +2 | head -n -1 | terraform fmt - | jq --raw-input --slurp '{\"hcl\":.}'"]
}
locals {
  to_hcl = { # some dynamically generated data stucture, this static data is an example
    a = 1,
    b = 2,
    c = [
      {
        d = 4,
        e = 5
      },
      {
        f = [
          6,
          7,
          8
        ]
      }
    ]
  }
  hcl_string = data.external.hclencode.result["hcl"]
}
resource "github_repository_file" "tfvars" {
  repository          = ...
  file                = "something.auto.tfvars"
  content             = <<-EOF
################################################################################
# GENERATED BY TERRAFORM, DO NOT EDIT!!!!
# This file will be automatically loaded by terraform
################################################################################

some_static_var = ...

${data.external.hclencode.result["hcl"]}
EOF
...
}

Here is a module that does the same:

variable "map" {
  type = any
}
data "external" "hclencode" {
  program = ["bash", "-c", "set -o pipefail; echo '${jsonencode(var.map)}' | jq | sed -r 's/^  \"(.*)\": /  \\1 = /;s/^  ( *)\"([a-zA-z0-9_-]*)\": /  \\1\\2 = /;s/^  ( *\".*\"): /  \\1 = /;s/^(  [^ ].*),$/\\1/' | tail -n +2 | head -n -1 | terraform fmt - | jq --raw-input --slurp '{\"hcl\":.}'"]
}
output "hcl" {
  value = data.external.hclencode.result["hcl"]
}

[giner]

Another use-case here.

We use complex variables in our modules (for good reason). In case we need to expose a complex module variable in the root module we have to re-define the same variable structure in the root and keep this in sync with the module. To overcome this limitation we have to define variables in the root module as strings using heredoc with yaml or json as content and then decode it in the main. With hcl decode we wouldn't have to convert the data to yaml or json.

[nick4fake]

Another very simple use-case. We use tfe_variable to manage variables for terraform cloud workspaces and it seems like there is simply no way to pass existing data to value parameter (it requires string).

[Tbohunek]

We use complex variables in our modules (for good reason). In case we need to expose a complex module variable in the root module we have to re-define the same variable structure in the root and keep this in sync with the module. To overcome this limitation we have to define variables in the root module as strings using heredoc with yaml or json as content and then decode it in the main. With hcl decode we wouldn't have to convert the data to yaml or json.

Maybe it helps you @giner, I've been using a map(string), which still requires parsing at main, but much simpler, i.e. only tobool(), tolist(split()) etc. Still you have to define that map in all modules, but with much less excess.

[Tbohunek]

Maybe it helps you @giner, I've been using a map(string), which still requires parsing at main, but much simpler, i.e. only tobool(), tolist(split()) etc. Still you have to define that map in all modules, but with much less excess.

I don't like its vague structure, but map(object) sadly can't be used unless I'd want to define every single attribute in those 200 modules that use it on my end. So I'm using the artificial module "globals" {} wherever I can.

But even this has a limitation, at least from security perspective where you want to be sure that certain variables can't be tampered with from outside the environment. And it's a cheat anyways. I would much prefer an actual concept of state-wide global_variable.
@apparentlymart what would you think of that?

For perspective, my usecase is this:
I have a module subnet that needs multiple inputs, but resource path to it is like this:
module.project.module.region.module.subnet in each of the 4 environments.
So I have to define these variables at 3 places, plus 1 for each projectxenvironment combination.
Either I do this, or I have a state per projectxenvironment and each time I make change to the project template, I run 300+ separate plans.

I wish I could use the computer to do some work for me... And while I understand the importance of defining variable type at the subnet module for validate purposes, I CBA to do that.

What I could imagine though is.. variable of type=inherit in the intermediate pass-through modules. It will take its type from the submodule into which it is being passed in this module.
Unless there is a duplicate, I can't see why parent module could not assume a variable has the same type as the submodule it's being passed into.

# subnet.tf
variable "complex" {
  type = map(object({}))
}

# region.tf
variable "complex" {
  type = inherit
}

module "subnet_1" {
  source = "./subnet"
  complex = var.complex
}

How about this? Would it help with some of the use-cases above?

[apparentlymart]

Using the new possibility for providers to contribute their own functions to the Terraform language, coming in Terraform v1.8, the built-in "terraform" provider (which previously offered only the terraform_remote_state data source and terraform_data resource type) will also now offer three functions covering the main use-cases discussed in this issue.

These are added as part of the provider, rather than as traditional built-in functions, as part of the new posture of using built-ins only for very broad, general features; moving forward, most new functions are likely to belong to providers, and so these are some early examples. Of course, since this provider is built-in itself Terraform will not need to download an external plugin to execute these ones.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.