aws_db_parameter_group shows diffs but doesn't apply. Ignore_changes has no effect

[yasin-amadmia-mck]

When one or more db_parameter_group parameters are modified outside of TF, TF always shows diff during 'plan' but 'apply' doesn't seem to revert the changed parameter. Also using 'ignore_changes' seems to NOT have any affect
Step to Reproduce

-> Create an aws_db_parameter_group with some parameters set.

resource "aws_db_parameter_group" "default" {
    name = "test1"
    family = "mysql5.6"
    description = "test parameter group"

    parameter {
        name = "character_set_server"
        value = "utf8"
    }
}

resource "aws_db_instance" "default" {
    identifier = "db-test1"
    allocated_storage = "5"
    engine = "mysql"
    engine_version = "5.6.22"
    instance_class = "db.t2.micro"
    storage_type = "gp2"

    username = "root"
    password = "123456789abcdefg"

    parameter_group_name = "${aws_db_parameter_group.default.id}"
}

-> Now modify a parameter outside of TF. Example using aws cli

aws rds modify-db-parameter-group --db-parameter-group-name test1 --parameters "ParameterName=autocommit,ParameterValue=1,ApplyMethod=pending-reboot"

-> After step 2, if we look into AWS Console, the DB_PARAMETER_GROUP 'test1' will have autocommit set to 1.

-> Run terraform plan.
This will detect the change to autocommit.

-> Apply terraform apply.
This will show that it has changed 1 resource but looking at the AWS Console, the 'autocommit' value will still be '1'.

-> Doing subsequent terraform plans will always show the diffs. 'apply' will always say it has changed the resource but doesn't seem to actually change it. i.e. autocommit always stays 1 as set in step 2 above.

-> Using lifecycle seems to have no effect. Expected behaviour is that it should not show any changed parameter values (autocommit in this case) as diffs during 'plan'

resource "aws_db_parameter_group" "default" {
    name = "test1"
    family = "mysql5.6"
    description = "test parameter group"

    parameter {
        name = "character_set_server"
        value = "utf8"
    }

    lifecycle {
        ignore_changes = ["parameter"]
   }
}

[yasin-amadmia-mck]

I believe what's happening is perhaps TF is trying to set the changed parameter back to <engine-default>, i.e. to null or not-defined but AWS doesn't allow that on a per-parameter basis and hence TF does that at each plan/apply and shows that as diffs.

Sample Plan output below. Here I manually changed autocommit = 0.
By default, autocommit is blank/null/engine-default with AWS Parameter Group, but once you change it to a set value (0 or 1 are allowed values), you cannot revert it back to blank/null/engine-default on its own alone (which is what TF is trying to do). You can reset the full db_parameter_group to default (ie all values).

~ aws_db_parameter_group.test
    parameter.#:                       "1" => "0"
    parameter.3870453284.apply_method: "" => ""
    parameter.3870453284.name:         "autocommit" => ""
    parameter.3870453284.value:        "0" => ""

Plan: 0 to add, 1 to change, 0 to destroy.

If this analysis is correct, to fix this temporarily, I need a way to ignore_changes to parameters, which is not working or I am not setting it correctly. See my sample code above.

[yasin-amadmia-mck]

Realised I was using TF V0.6.6. Upgraded to TF V0.6.6.7 and now I can get the ignore_changes to work (i..e It ignores the parameters changed outside TF and doesn't show them during diff/apply). However, now when I destroy the resource and try re-creating it, it complains (during apply)

* aws_db_parameter_group.test: diffs didn't match during apply. This is a bug with Terraform and should be reported.

The terraform.tf file is

provider "aws" {
    access_key = "xxx"
    secret_key = "xxx"
    region     = "us-east-1"
}

resource "aws_db_parameter_group" "test" {
  name        = "test"
  family      = "mysql5.6"
  description = "test"

  parameter {
    name  = "autocommit"
    value = 1
  }

  lifecycle {
    ignore_changes = ["parameter"]
  }
}

[heww]

@geek876 Yes, I have the save problem like yours using TF v0.6.7 .

Error applying plan:

1 error(s) occurred:

* openstack_compute_instance_v2.resource.4: diffs didn't match during apply. This is a bug with Terraform and should be reported.

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

and my resource is like this

resource "openstack_compute_instance_v2" "resource" {
  name = "${ var.short_name}-worker-${format("%03d", count.index+1) }"
  key_pair = "${ var.keypair_name }"
  image_name = "${ var.image_name }"
  flavor_name = "${ var.resource_flavor_name }"
  security_groups = [ "${ var.security_groups }" ]
  network = { uuid = "${ var.net_id }" }
  metadata = {
    dc = "${var.datacenter}"
    role = "worker"
    ssh_user = "${ var.ssh_user }"
  }
  count = "${ var.resource_count }"
  lifecycle {
    ignore_changes = ["volume"]
  }
}

[stack72]

Hi @geek876

I have just tested this and it's no longer an issue. We fixed a lot of the ignore_changes issues in terraform 0.7

So using your config:

resource "aws_db_parameter_group" "default" {
    name = "test1"
    family = "mysql5.6"
    description = "test parameter group"

    parameter {
        name = "character_set_server"
        value = "utf8"
    }
}

I was able to plan and apply as follows:

% terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but
will not be persisted to local or remote state storage.


The Terraform execution plan has been generated and is shown below.
Resources are shown in alphabetical order for quick scanning. Green resources
will be created (or destroyed and then created if an existing resource
exists), yellow resources are being changed in-place, and red resources
will be destroyed. Cyan entries are data sources to be read.

Note: You didn't specify an "-out" parameter to save this plan, so when
"apply" is called, Terraform can't guarantee this is what will execute.

+ aws_db_parameter_group.default
    arn:                               "<computed>"
    description:                       "test parameter group"
    family:                            "mysql5.6"
    name:                              "test1"
    parameter.#:                       "1"
    parameter.2421266705.apply_method: ""
    parameter.2421266705.name:         "character_set_server"
    parameter.2421266705.value:        "utf8"


Plan: 1 to add, 0 to change, 0 to destroy.
[stacko@Pauls-MacBook-Pro:~/Code/terraform-recreations/4045]
% terraform apply
aws_db_parameter_group.default: Creating...
  arn:                               "" => "<computed>"
  description:                       "" => "test parameter group"
  family:                            "" => "mysql5.6"
  name:                              "" => "test1"
  parameter.#:                       "" => "1"
  parameter.2421266705.apply_method: "" => ""
  parameter.2421266705.name:         "" => "character_set_server"
  parameter.2421266705.value:        "" => "utf8"
aws_db_parameter_group.default: Creation complete

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.

State path: terraform.tfstate

I then modified a parameter:

% aws rds modify-db-parameter-group --db-parameter-group-name test1 --parameters "ParameterName=autocommit,ParameterValue=1,ApplyMethod=pending-reboot"
{
    "DBParameterGroupName": "test1"
}

A subsequent plan looks as follows now:

% terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but
will not be persisted to local or remote state storage.

aws_db_parameter_group.default: Refreshing state... (ID: test1)

The Terraform execution plan has been generated and is shown below.
Resources are shown in alphabetical order for quick scanning. Green resources
will be created (or destroyed and then created if an existing resource
exists), yellow resources are being changed in-place, and red resources
will be destroyed. Cyan entries are data sources to be read.

Note: You didn't specify an "-out" parameter to save this plan, so when
"apply" is called, Terraform can't guarantee this is what will execute.

~ aws_db_parameter_group.default
    parameter.#:                       "2" => "1"
    parameter.2421266705.apply_method: "" => ""
    parameter.2421266705.name:         "character_set_server" => "character_set_server"
    parameter.2421266705.value:        "utf8" => "utf8"
    parameter.4289290085.apply_method: "" => ""
    parameter.4289290085.name:         "autocommit" => ""
    parameter.4289290085.value:        "1" => ""


Plan: 0 to add, 1 to change, 0 to destroy.

I then changed the config to look as follows:

resource "aws_db_parameter_group" "default" {
    name = "test1"
    family = "mysql5.6"
    description = "test parameter group"

    parameter {
        name = "character_set_server"
        value = "utf8"
    }

    lifecycle {
        ignore_changes = ["parameter"]
   }
}

a plan then does the following:

% terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but
will not be persisted to local or remote state storage.

aws_db_parameter_group.default: Refreshing state... (ID: test1)

No changes. Infrastructure is up-to-date. This means that Terraform
could not detect any differences between your configuration and
the real physical resources that exist. As a result, Terraform
doesn't need to do anything.

I can destroy it as follows:

% terraform destroy
Do you really want to destroy?
  Terraform will delete all your managed infrastructure.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes

aws_db_parameter_group.default: Refreshing state... (ID: test1)
aws_db_parameter_group.default: Destroying...
aws_db_parameter_group.default: Destruction complete

Destroy complete! Resources: 1 destroyed.

and I can recreate it as follows:

% terraform apply
aws_db_parameter_group.default: Creating...
  arn:                               "" => "<computed>"
  description:                       "" => "test parameter group"
  family:                            "" => "mysql5.6"
  name:                              "" => "test1"
  parameter.#:                       "" => "1"
  parameter.2421266705.apply_method: "" => ""
  parameter.2421266705.name:         "" => "character_set_server"
  parameter.2421266705.value:        "" => "utf8"
aws_db_parameter_group.default: Creation complete

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.

State path: terraform.tfstate

Sorry this has taken so long to get back to you on - but I am happy this is no longer an issue using the Terraform 0.7.x series :)

Paul

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.