Module Support #522

skorfmann · 2021-01-26T21:21:41Z

The overall goal of this issue is to extend the support for modules within cdktf.

At the moment, we only support generating (via cdktf get) packages for modules which are published in the Terraform registry. While this covers a good part of the public Terraform module ecosystem, it's limiting users who want to use modules in version-control systems, local disk or other sources.

We will add support for generating Typescript classes using all module sources currently supported by Terraform Core. The experience of using these generated classes should feel idiomatic within the respective target language of the user.

Implementation

At the moment we're using a dedicated Terraform Registry client to fetch the required meta information about a module to generate the code.

When looking at the various ways of consuming Terraform modules, it's better to delegate the work of fetching the modules to Terraform Core directly. This could be done via defining a module block in a temporary working directory, running terraform init and use Terraform Config Inspect to generate the metadata.

Compared to the direct Terraform Registry API call, this will certainly a bit slower and use more resources, e.g. the terraform init call will download all required module dependencies as well. On the upside, we don't have to reimplement functionality which already exists and we'll always support what Terraform Core is going to support in the future.

On a high level view, this will include:

Replace the Terraform Registry Client by a Terraform workflow similar to the Provider generator step
Figure out if we can use Terraform Config Inspect directly or if it's better to reimplement the relevant parts in Typescript
~~Make sure Module Outputs are properly typed and are using Tokens where suitable (related to Token System #525)~~

Issues Resolved by this Feature

Extend Support for Modules beyond the Terraform Registry Extend Support of Module Sources beyond Terraform Registry #16
Support Submodules (likely addressed in Extend Support of Module Sources beyond Terraform Registry #16) - Support Submodules in Terraform Registry #520
Fix error prone versioning in cdktf.json (likely addressed in Extend Support of Module Sources beyond Terraform Registry #16) - Specify module versions in cdktf.json #425
Variable default handling inconsistent (at least when generated from Terraform Registry) - Terraform Registry API Response Differs from Actual Implementation #27
Improve typing of module outputs - All Module Outputs Typed as String #287

The text was updated successfully, but these errors were encountered:

skorfmann · 2021-02-22T16:25:10Z

Getting Module Source Code

Given the following main.tf source:

provider "aws" {

}

module "iam_account" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-account"
  version = "~> 3.0"

  account_alias = "awesome-company"

  minimum_password_length = 37
  require_numbers         = false
}

module "consul_consul-iam-policies" {
  source  = "hashicorp/consul/aws//modules/consul-iam-policies"
  version = "0.8.4"
  iam_role_id = ""
}

module "s3_bucket" {
  source = "cloudposse/s3-bucket/aws"
  # Cloud Posse recommends pinning every module to a specific version
  # version     = "x.x.x"
  acl                      = "private"
  enabled                  = true
  user_enabled             = true
  versioning_enabled       = false
  allowed_bucket_actions   = ["s3:GetObject", "s3:ListBucket", "s3:GetBucketLocation"]
  name                     = "app"
  stage                    = "test"
  namespace                = "eg"
}

module "es" {
  source  = "git::https://github.com/terraform-community-modules/tf_aws_elasticsearch.git?ref=v1.1.0"

  domain_name                    = "my-elasticsearch-domain"
  management_public_ip_addresses = ["34.203.XXX.YYY"]
  instance_count                 = 16
  instance_type                  = "m4.2xlarge.elasticsearch"
  dedicated_master_type          = "m4.large.elasticsearch"
  es_zone_awareness              = true
  ebs_volume_size                = 100
}

A terraform init command will fetch all modules, including modules which are referenced in the modules which are defined explicitly. There's a .terraform/modules/modules.json file after init:

{
  "Modules": [
    {
      "Key": "",
      "Source": "",
      "Dir": "."
    },
    {
      "Key": "s3_bucket.s3_user",
      "Source": "cloudposse/iam-s3-user/aws",
      "Version": "0.15.1",
      "Dir": ".terraform/modules/s3_bucket.s3_user"
    },
    {
      "Key": "s3_bucket.s3_user.s3_user",
      "Source": "cloudposse/iam-system-user/aws",
      "Version": "0.20.1",
      "Dir": ".terraform/modules/s3_bucket.s3_user.s3_user"
    },
    {
      "Key": "s3_bucket.s3_user.s3_user.this",
      "Source": "cloudposse/label/null",
      "Version": "0.24.1",
      "Dir": ".terraform/modules/s3_bucket.s3_user.s3_user.this"
    },
    {
      "Key": "s3_bucket.this",
      "Source": "cloudposse/label/null",
      "Version": "0.24.1",
      "Dir": ".terraform/modules/s3_bucket.this"
    },
    {
      "Key": "consul_consul-iam-policies",
      "Source": "hashicorp/consul/aws//modules/consul-iam-policies",
      "Version": "0.8.4",
      "Dir": ".terraform/modules/consul_consul-iam-policies/modules/consul-iam-policies"
    },
    {
      "Key": "es",
      "Source": "git::https://github.com/terraform-community-modules/tf_aws_elasticsearch.git?ref=v1.1.0",
      "Dir": ".terraform/modules/es"
    },
    {
      "Key": "iam_account",
      "Source": "terraform-aws-modules/iam/aws//modules/iam-account",
      "Version": "3.9.0",
      "Dir": ".terraform/modules/iam_account/modules/iam-account"
    },
    {
      "Key": "s3_bucket",
      "Source": "cloudposse/s3-bucket/aws",
      "Version": "0.31.0",
      "Dir": ".terraform/modules/s3_bucket"
    },
    {
      "Key": "s3_bucket.s3_user.this",
      "Source": "cloudposse/label/null",
      "Version": "0.24.1",
      "Dir": ".terraform/modules/s3_bucket.s3_user.this"
    }
  ]
}

Based on this, we can process and analyse the source code of the given modules with one of the following options

hcl2json

Nice and simple, using this via wasm is likely rather straightforward.

./hcl2json ./.terraform/modules/es/variables.tf
{
    "variable": {
        "advanced_options": [
            {
                "default": {},
                "description": "Map of key-value string pairs to specify advanced configuration options. Note that the values for these configuration options must be strings (wrapped in quotes) or they may be wrong and cause a perpetual diff, causing Terraform to want to recreate your Elasticsearch domain on every apply.",
                "type": "${map(string)}"
            }
        ],

terraform config inspect

That's pretty much the output we're using right now directly from the Terraform registry. Using this via wasm isn't possible without modifications (due to disk access).

{
  "path": "/Users/sebastian/projects/cdk/tfmodules/.terraform/modules/es",
  "variables": {
    "advanced_options": {
      "name": "advanced_options",
      "type": "map(string)",
      "description": "Map of key-value string pairs to specify advanced configuration options. Note that the values for these configuration options must be strings (wrapped in quotes) or they may be wrong and cause a perpetual diff, causing Terraform to want to recreate your Elasticsearch domain on every apply.",
      "default": {},
      "required": false,
      "pos": {
        "filename": "/Users/sebastian/projects/cdk/tfmodules/.terraform/modules/es/variables.tf",
        "line": 119
      }
    }
}

@ts-terraform/hcl

This uses wasm behind the scenes to fully parse an AST of the given hcl content. It's an interesting approach, but seems to be too much at this point.

{
  "attributes": {},
  "blocks": [
    {
      "body": {
        "attributes": {
          "default": {
            "equalsRange": {
              "end": {
                "byte": 204,
                "column": 16,
                "line": 4
              },
              "filename": "foo",
              "start": {
                "byte": 203,
                "column": 15,
                "line": 4
              }
            },
            "expr": {
              "kind": "literalValueExpr",
              "srcRange": {
                "end": {
                  "byte": 209,
                  "column": 21,
                  "line": 4
                },
                "filename": "foo",
                "start": {
                  "byte": 205,
                  "column": 17,
                  "line": 4
                }
              },
              "type": "bool",
              "val": true
            },

Naming

Right now, the name of a generated module is derived from the response of the Terraform Registry. Since we're dealing with various unstructured module sources now, we'll have to come up with a naming schema and/or allow the user to override the name. That's still something to figure out.

cdktf.json

This is how the modules from above would be defined in the cdktf.json file:

{
  "language": "python",
  "app": "pipenv run python main.py",
  "terraformProviders": [
    "aws@~> 2.55"
  ],
  "terraformModules": [
    "terraform-aws-modules/iam/aws//modules/iam-account@~> 3.0",
    "hashicorp/consul/aws//modules/[email protected]",
    "cloudposse/s3-bucket/aws",
    "git::https://github.com/terraform-community-modules/tf_aws_elasticsearch.git?ref=v1.1.0"
  ],
  "codeMakerOutput": "imports"
}

skorfmann · 2021-02-22T19:20:12Z

Terraform Versions

With Terraform 0.12.x it's likely to see a few error messages when running terraform init - that's an excerpt for the configuration above:

There are some problems with the configuration, described below.

The Terraform configuration must be valid before initialization so that
Terraform can determine which modules and providers need to be installed.

Warning: Provider source not supported in Terraform v0.12

  on .terraform/modules/s3_bucket.s3_user.s3_user/versions.tf line 5, in terraform:
   5:     aws = {
   6:       source  = "hashicorp/aws"
   7:       version = ">= 2.0"
   8:     }

A source was declared for provider aws. Terraform v0.12 does not support the
provider source attribute. It will be ignored.

(and 2 more similar warnings elsewhere)


Error: Custom variable validation is experimental

  on .terraform/modules/s3_bucket.s3_user.s3_user.this/variables.tf line 27, in variable "context":
  27:   validation {

This feature is currently an opt-in experiment, subject to change in future
releases based on feedback.

Activate the feature for this module by adding variable_validation to the list
of active experiments.

Terraform > 0.12 doesn't have these issues. The .terraform/modules/modules.json file seems to be consistent for all versions >= 0.12.

skorfmann · 2021-02-23T09:48:44Z

Getting Module Source Code

For the sake of completeness, we could in theory ship platform specific binaries with the cdktf-cli npm package or even build them on-demand via node-gyp. However, I think this should be avoided if possible.

jsteinich · 2021-02-23T13:47:53Z

including modules which are referenced in the modules which are defined explicitly

Given how long source generation takes, I don't think we should be generating code for the referenced modules.

Do you know if hcl2json has anything to indicate required variables? I think that is definitely useful given how many variables many modules have.

I imagine we'll end up being fairly verbose on the naming to eliminate naming collisions. Perhaps could match current names for registry sourced modules.

I think it's pretty common for modules to have <=0.12 versions and >0.12 versions. We should just make sure errors are surfaced cleanly.

skorfmann · 2021-02-23T14:46:30Z

including modules which are referenced in the modules which are defined explicitly

Given how long source generation takes, I don't think we should be generating code for the referenced modules.

Yes, agree 👍

Do you know if hcl2json has anything to indicate required variables? I think that is definitely useful given how many variables many modules have.

I think that could be derived from a missing default value - or are you aware of cases where that's not sufficient?

I imagine we'll end up being fairly verbose on the naming to eliminate naming collisions. Perhaps could match current names for registry sourced modules.

Sounds good.

I think it's pretty common for modules to have <=0.12 versions and >0.12 versions. We should just make sure errors are surfaced cleanly.

👍

jsteinich · 2021-02-23T23:07:46Z

I think that could be derived from a missing default value - or are you aware of cases where that's not sufficient?

Most likely.

skorfmann · 2021-02-24T10:26:37Z

WebAssembly

For reference, here a quick link collection around WebAssembly (wasm)

Supported Platforms: https://webassembly.org/roadmap/

Non-Browser Dedicated Runtimes

Options

WASI (WebAssembly System Interface)

https://github.com/WebAssembly/WASI
https://nodejs.org/api/wasi.html (>= v12.16)
https://www.secondstate.io/articles/wasi-access-system-resources/
https://radu-matei.com/blog/nodejs-wasi/
WASI seems to be the emerging standard
No direct build support in Go yet - see wasm: support new WASI interface golang/go#31105
TinyGo supports it https://twitter.com/wasmerio/status/1261374695074492416

Go wasm_exec.js

Unrelated, but interesting

assemblyscript.org (see status as well)
https://webassembly.sh/ (wasmer playground)

skorfmann · 2021-03-09T22:04:14Z

Make sure Module Outputs are properly typed and are using Tokens where suitable (related to Token System #525)

Gonna out-scope this, since it's significant effort to infer the types for outputs and perhaps better off as some sort of Terraform core feature anyway.

nandac · 2021-11-02T13:40:34Z

Does cdktf support specifying custom modules stored in Terraform Cloud in cdktf.json. If so how would I go about it?

Right now the only workaround seems to be pointing to the git repository.

jsteinich · 2021-11-03T02:53:54Z

Does cdktf support specifying custom modules stored in Terraform Cloud in cdktf.json. If so how would I go about it?

Right now the only workaround seems to be pointing to the git repository.

You should be able to specify using the same format as supported by the source parameter in hcl. See https://www.terraform.io/docs/cloud/registry/using.html#using-modules-in-configurations for more information.

nandac · 2021-11-24T19:10:38Z

Does cdktf support specifying custom modules stored in Terraform Cloud in cdktf.json. If so how would I go about it?
Right now the only workaround seems to be pointing to the git repository.

You should be able to specify using the same format as supported by the source parameter in hcl. See https://www.terraform.io/docs/cloud/registry/using.html#using-modules-in-configurations for more information.

I did as you said and ended up with the following error when running cdktf get

⠇ downloading and generating modules and providers...
There are some problems with the configuration, described below.

The Terraform configuration must be valid before initialization so that
Terraform can determine which modules and providers need to be installed.
╷
│ Error: Invalid module instance name
│ 
│   on main.tf.json line 1, in module:
│    1: {"terraform":{},"module":{"app.terraform.io_<REDACTED>_application-load-balancer_aws":{"source":"app.terraform.io/REDACTED/application-load-balancer/aws"}}}
│ 
│ A name must start with a letter or underscore and may contain only letters,
│ digits, underscores, and dashes.
╵
Error: non-zero exit code 1

Any help is appreciated.

Cheers.

jsteinich · 2021-11-27T16:36:13Z

@nandac I moved to a separate issue (#1360) for better tracking.

github-actions · 2022-12-01T01:48:21Z

I'm going to lock this issue because it has been closed for 30 days. This helps our maintainers find and focus on the active issues. If you've found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

skorfmann added enhancement New feature or request feature modules and removed enhancement New feature or request labels Jan 26, 2021

skorfmann added this to the 0.2 milestone Jan 26, 2021

danieldreier modified the milestones: 0.2, beta Jan 26, 2021

danieldreier added the release-blocking label Feb 4, 2021

skorfmann mentioned this issue Feb 15, 2021

JSII compilation failed {cloudwatch,iam,route53} error TS6133: 'options' is declared but its value is never read. #563

Closed

skorfmann self-assigned this Feb 22, 2021

This was referenced Mar 1, 2021

Build Modules from Various Sources #584

Merged

Transform Modules to Constructs #231

Open

skorfmann closed this as completed in #584 Mar 11, 2021

This was referenced Mar 16, 2021

Extend Support of Module Sources beyond Terraform Registry #16

Closed

Support Submodules in Terraform Registry #520

Closed

Specify module versions in cdktf.json #425

Closed

jsteinich mentioned this issue Nov 27, 2021

Error specifying custom modules stored in Terraform Cloud in cdktf.json. #1360

Closed

github-actions bot locked as resolved and limited conversation to collaborators Dec 1, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Module Support #522

Module Support #522

skorfmann commented Jan 26, 2021 •

edited

Loading

skorfmann commented Feb 22, 2021

skorfmann commented Feb 22, 2021

skorfmann commented Feb 23, 2021

jsteinich commented Feb 23, 2021

skorfmann commented Feb 23, 2021

jsteinich commented Feb 23, 2021

skorfmann commented Feb 24, 2021

skorfmann commented Mar 9, 2021

nandac commented Nov 2, 2021

jsteinich commented Nov 3, 2021

nandac commented Nov 24, 2021

jsteinich commented Nov 27, 2021

github-actions bot commented Dec 1, 2022

Module Support #522

Module Support #522

Comments

skorfmann commented Jan 26, 2021 • edited Loading

Implementation

Issues Resolved by this Feature

skorfmann commented Feb 22, 2021

Getting Module Source Code

hcl2json

terraform config inspect

@ts-terraform/hcl

Naming

cdktf.json

skorfmann commented Feb 22, 2021

Terraform Versions

skorfmann commented Feb 23, 2021

jsteinich commented Feb 23, 2021

skorfmann commented Feb 23, 2021

jsteinich commented Feb 23, 2021

skorfmann commented Feb 24, 2021

WebAssembly

Non-Browser Dedicated Runtimes

Options

WASI (WebAssembly System Interface)

Go wasm_exec.js

Unrelated, but interesting

skorfmann commented Mar 9, 2021

nandac commented Nov 2, 2021

jsteinich commented Nov 3, 2021

nandac commented Nov 24, 2021

jsteinich commented Nov 27, 2021

github-actions bot commented Dec 1, 2022

skorfmann commented Jan 26, 2021 •

edited

Loading