Add support for maps with non-primitive types (rich / object map support)

[radeksimko]

Terraform Version

Terraform v0.6.14

Terraform Configuration Files

resource "aws_api_gateway_stage" "boo" {
    rest_api_id = "${aws_api_gateway_rest_api.demo.id}"
    name = "test"

    method_setting "yada" {
        metrics_enabled = true
        logging_level = "DEBUG"
    }

    method_setting "bada" {
        metrics_enabled = false
        logging_level = "ERROR"
    }
}

Expected Behavior

+ aws_api_gateway_stage.boo
    rest_api_id:                         "" => "${aws_api_gateway_rest_api.demo.id}"
    name:                                "" => "test"
    method_setting.#:                    "" => "2"
    method_setting.yada.metrics_enabled: "" => "1"
    method_setting.yada.logging_level:   "" => "DEBUG"
    method_setting.bada.metrics_enabled: "" => "0"
    method_setting.bada.logging_level:   "" => "ERROR"

Actual Behavior

Error running plan: 1 error(s) occurred:

* method_setting: 2 error(s) decoding:

* '[bada]' expected type 'string', got unconvertible type '[]map[string]interface {}'
* '[yada]' expected type 'string', got unconvertible type '[]map[string]interface {}'

Steps to Reproduce

            "method_setting": &schema.Schema{
                Type:     schema.TypeMap,
                Optional: true,
                Elem: &schema.Resource{
                    Schema: map[string]*schema.Schema{
                        "metrics_enabled": &schema.Schema{
                            Type:     schema.TypeBool,
                            Optional: true,
                        },
                        "logging_level": &schema.Schema{
                            Type:     schema.TypeString,
                            Optional: true,
                        },
                        "data_trace_enabled": &schema.Schema{
                            Type:     schema.TypeBool,
                            Optional: true,
                        },
                    },
                },
            },

$ terraform plan

The current schema does support TypeMap which translates into map[string]interface{}. The interface is then decoded via mapstructure:
https://github.com/mitchellh/mapstructure/blob/master/mapstructure.go#L70

which seems to support slice of maps, but somehow expects string.

---

I'm creating this issue as I'm working around this by putting the key inside TypeSet as another field and I plan to link here.

[hugomd]

@radeksimko How did you end up working around this issue?

[glenjamin]

I just ran into this when trying to pass list/hash types to the azurerm_template_deployment resource.

AzureRM templates and the go wrapper around them both support arbitrary types, but it looks like the diffMap function in schema.go assumes the values will be strings, and causes an error

resource "azurerm_template_deployment" "webapp" {
    name = "${var.app-name}"
    resource_group_name = "${azurerm_resource_group.group.name}"
    deployment_mode = "Incremental"
    template_body = "${file("../webapp.template.json")}"
    parameters {
        config {
            A = "1"
            B = "2"
        }
    }
}

* azurerm_template_deployment.webapp: parameters (config): '' expected type 'string', got unconvertible type '[]map[string]interface {}'

Is anyone aware of a workaround for this?

[radeksimko]

@hugomd I think the easiest workaround - if it's a new resource - is to use Type: TypeList; MaxItems: 1. The only caveat is that addresses will then (have to) contain the index, i.e. resource_type.ref_name.list_attribute.0.nested_attr.

The other option is to just use TypeMap with no Elem which has the downside of relaxed restrictions on keys on the schema level - i.e. user can define any map keys and it would be then up to the CRUD code to do any validation. The upside is reasonable address formats (resource_type.ref_name.map_attribute.nested_attr)

[benjvi]

@radeksimko Do you know if this planned to be addressed as part of the hcl2 work? I've run into a couple of issues I've had to work around with maps now, one as you've described above, and also in trying to put heterogeneous items as values into a map which it looks like also can't work

[apparentlymart]

Hi @benjvi!

The initial set of work addresses the low-level problems that prevented us from supporting this before, but we're not planning to include new features for the helper/schema in the first release just to minimize the number of subsystems changing (there are already a lot!). Once the core changes are in place to make this possible we plan to add new features to the helper/schema API that will make use of these new capabilities as a separate step.

---

I know some readers appreciate having some more details on why certain things in Terraform behave the way they do, so what follows is a little background on why TypeMap doesn't support complex-typed elements in the current release. Feel free to skip this entirely if you're not interested in Terraform's internals; none of this knowledge is required for Terraform users.

In Terraform Core's current internal model, the state for a resource is modeled as a flat map from strings to strings, with complex data structured flattened into dot-separated keys as you can see in the plan output, like in Radek's example above:

    rest_api_id:                         "" => "${aws_api_gateway_rest_api.demo.id}"
    name:                                "" => "test"
    method_setting.#:                    "" => "2"
    method_setting.yada.metrics_enabled: "" => "1"
    method_setting.yada.logging_level:   "" => "DEBUG"
    method_setting.bada.metrics_enabled: "" => "0"
    method_setting.bada.logging_level:   "" => "ERROR"

This technique dates back to the earliest versions of Terraform, where lists and maps were not supported yet at all. Support for complex types has been gradually improved, with the most notable changes happening in 0.7, but the physical model internally has been unchanged due to the fact that this is a cross-cutting concern that touches almost all of Terraform's subsystems.

This flattening technique is lossy due to limitations of the key format. If there were any key in the path containing a period then the result would be ambiguous:

    method_setting "ya.da" {
        metrics_enabled = true
        logging_level = "DEBUG"
    }

    method_setting.ya.da.metrics_enabled: "" => "1"
    method_setting.ya.da.logging_level:   "" => "DEBUG"

In this scenario helper/schema could in principle walk backwards from the deeper schema and infer that da must be part of the key because there is no da attribute in the nested resource schema, but this would quickly become complex and possibly undecidable with many nested maps that may have identical intermediate attribute names.

In all cases except maps we know that periods cannot appear in the segments, because periods are not valid in attribute names and indexes into lists and sets are numeric and thus cannot contain periods. Maps of primitive types are possible because we know that primitives cannot have attributes or elements and so any remaining characters in the flattened key must be the key into the map structure.

The lower-level change made by our configuration language work is to represent these data structures internally without flattening them, using the type system of the new language interpreter. Retaining the individual collection objects removes the ambiguity, and avoids the need for helper/schema to infer the original structure itself based on the period-separated path.

[kilokahn]

hey @apparentlymart - I think you may have already explained this above, but wanted to check in if this support is available in the v0.12 alpha builds of the provider-sdk? Maybe not yet (since you said we'd not change it in the first release), but wanted to know. Thanks! 😄

[apparentlymart]

Indeed, the current state is that Terraform Core supports this but the SDK cannot, because the SDK is still working with the old-style representation. Our goal for the initial v0.12.0 release is to ensure that the existing SDK functionality is all still broadly working in spite of the type system changes, which has been tricky enough as it is, so we're not going to try to introduce any new functionality at the same time.

Once we have v0.12.0 final released, we'll begin discussions about how to improve the SDK itself, which should include making it use the new native Terraform type system instead of its own legacy representation, so that it can make use of all of the new capabilities.

[yann-soubeyrand]

Hi,

What is the state of the discussions on upgrading the SDK?

[radeksimko]

The problem described in my initial post could be solved either by #220 or #155 either of which may actually be better suited solution for that problem.

Both problems describe the need for complex/nested data structures, they just aim to solve it in different ways.

There may still be value in allowing more/variadic types in TypeMap though, so I'll keep this open for now.