The plan output contains an invalid byte sequence in UTF-8 when ID is multibyte characters

[minamijoyo]

Hi there,

I found a bug related to multibyte characters in the terraform plan output.

Terraform Version

$ terraform -v
Terraform v0.11.8
+ provider.aws v1.34.0

Terraform Configuration Files

provider "aws" {}

resource "aws_cloudwatch_metric_alarm" "hoge" {
  alarm_name = "あああああああああああああああああああああああああああ"

  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = "1"
  datapoints_to_alarm = "1"
  metric_name         = "UnHealthyHostCount"
  namespace           = "AWS/ELB"
  period              = "60"
  statistic           = "Sum"
  threshold           = "0"
}

Note that あ (e3 81 82) is a Japanese character consisting of 3 bytes.
The aws_cloudwatch_metric_alarm uses alarm_name as an ID of the resource.

Debug Output

Crash Output

Expected Behavior

The terraform plan output should not contain an invalid byte sequence in UTF-8.

Actual Behavior

When the length of ID is longer than 80 bytes, the terraform plan truncates the extra bytes.
If the ID is multibyte characters, It is truncated without considering the multibyte boundary, and this will result in an invalid byte sequence in UTF-8.

Steps to Reproduce

$ terraform init

$ terraform plan

$ terraform apply

$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

aws_cloudwatch_metric_alarm.hoge: Refreshing state... (ID: あああああああああああああ...ああああああああああああ)

------------------------------------------------------------------------

No changes. Infrastructure is up-to-date.

This means that Terraform did not detect any differences between your
configuration and real physical resources that exist. As a result, no
actions need to be performed.

Dumping the byte sequence of terraform plan output with the od command is as follows.
You can see that the bytes at the address 0x0000129 - 0x000012A is 81 82 and it is an invalid byte sequence.

$ terraform plan | od -Ax -tcx1
0000000  033   [   0   m 033   [   1   m   R   e   f   r   e   s   h   i
           1b  5b  30  6d  1b  5b  31  6d  52  65  66  72  65  73  68  69
0000010    n   g       T   e   r   r   a   f   o   r   m       s   t   a
           6e  67  20  54  65  72  72  61  66  6f  72  6d  20  73  74  61
0000020    t   e       i   n   -   m   e   m   o   r   y       p   r   i
           74  65  20  69  6e  2d  6d  65  6d  6f  72  79  20  70  72  69
0000030    o   r       t   o       p   l   a   n   .   .   . 033   [   0
           6f  72  20  74  6f  20  70  6c  61  6e  2e  2e  2e  1b  5b  30
0000040    m  \n   T   h   e       r   e   f   r   e   s   h   e   d
           6d  0a  54  68  65  20  72  65  66  72  65  73  68  65  64  20
0000050    s   t   a   t   e       w   i   l   l       b   e       u   s
           73  74  61  74  65  20  77  69  6c  6c  20  62  65  20  75  73
0000060    e   d       t   o       c   a   l   c   u   l   a   t   e
           65  64  20  74  6f  20  63  61  6c  63  75  6c  61  74  65  20
0000070    t   h   i   s       p   l   a   n   ,       b   u   t       w
           74  68  69  73  20  70  6c  61  6e  2c  20  62  75  74  20  77
0000080    i   l   l       n   o   t       b   e  \n   p   e   r   s   i
           69  6c  6c  20  6e  6f  74  20  62  65  0a  70  65  72  73  69
0000090    s   t   e   d       t   o       l   o   c   a   l       o   r
           73  74  65  64  20  74  6f  20  6c  6f  63  61  6c  20  6f  72
00000a0        r   e   m   o   t   e       s   t   a   t   e       s   t
           20  72  65  6d  6f  74  65  20  73  74  61  74  65  20  73  74
00000b0    o   r   a   g   e   .  \n 033   [   0   m  \n 033   [   0   m
           6f  72  61  67  65  2e  0a  1b  5b  30  6d  0a  1b  5b  30  6d
00000c0  033   [   1   m   a   w   s   _   c   l   o   u   d   w   a   t
           1b  5b  31  6d  61  77  73  5f  63  6c  6f  75  64  77  61  74
00000d0    c   h   _   m   e   t   r   i   c   _   a   l   a   r   m   .
           63  68  5f  6d  65  74  72  69  63  5f  61  6c  61  72  6d  2e
00000e0    h   o   g   e   :       R   e   f   r   e   s   h   i   n   g
           68  6f  67  65  3a  20  52  65  66  72  65  73  68  69  6e  67
00000f0        s   t   a   t   e   .   .   .       (   I   D   :      あ
           20  73  74  61  74  65  2e  2e  2e  20  28  49  44  3a  20  e3
0000100   **  **  あ  **  **  あ  **  **  あ  **  **  あ  **  **  あ  **
           81  82  e3  81  82  e3  81  82  e3  81  82  e3  81  82  e3  81
0000110   **  あ  **  **  あ  **  **  あ  **  **  あ  **  **  あ  **  **
           82  e3  81  82  e3  81  82  e3  81  82  e3  81  82  e3  81  82
0000120   あ  **  **  あ  **  **   .   .   . 201 202  あ  **  **  あ  **
           e3  81  82  e3  81  82  2e  2e  2e  81  82  e3  81  82  e3  81
0000130   **  あ  **  **  あ  **  **  あ  **  **  あ  **  **  あ  **  **
           82  e3  81  82  e3  81  82  e3  81  82  e3  81  82  e3  81  82
0000140   あ  **  **  あ  **  **  あ  **  **  あ  **  **  あ  **  **   )
           e3  81  82  e3  81  82  e3  81  82  e3  81  82  e3  81  82  29
0000150  033   [   0   m  \n  \n   -   -   -   -   -   -   -   -   -   -
           1b  5b  30  6d  0a  0a  2d  2d  2d  2d  2d  2d  2d  2d  2d  2d
0000160    -   -   -   -   -   -   -   -   -   -   -   -   -   -   -   -
           2d  2d  2d  2d  2d  2d  2d  2d  2d  2d  2d  2d  2d  2d  2d  2d
*
0000190    -   -   -   -   -   -   -   -   -   -   -   -   -   -  \n  \n
           2d  2d  2d  2d  2d  2d  2d  2d  2d  2d  2d  2d  2d  2d  0a  0a
00001a0  033   [   0   m 033   [   1   m 033   [   3   2   m   N   o
           1b  5b  30  6d  1b  5b  31  6d  1b  5b  33  32  6d  4e  6f  20
00001b0    c   h   a   n   g   e   s   .       I   n   f   r   a   s   t
           63  68  61  6e  67  65  73  2e  20  49  6e  66  72  61  73  74
00001c0    r   u   c   t   u   r   e       i   s       u   p   -   t   o
           72  75  63  74  75  72  65  20  69  73  20  75  70  2d  74  6f
00001d0    -   d   a   t   e   . 033   [   0   m 033   [   3   2   m  \n
           2d  64  61  74  65  2e  1b  5b  30  6d  1b  5b  33  32  6d  0a
00001e0   \n   T   h   i   s       m   e   a   n   s       t   h   a   t
           0a  54  68  69  73  20  6d  65  61  6e  73  20  74  68  61  74
00001f0        T   e   r   r   a   f   o   r   m       d   i   d       n
           20  54  65  72  72  61  66  6f  72  6d  20  64  69  64  20  6e
0000200    o   t       d   e   t   e   c   t       a   n   y       d   i
           6f  74  20  64  65  74  65  63  74  20  61  6e  79  20  64  69
0000210    f   f   e   r   e   n   c   e   s       b   e   t   w   e   e
           66  66  65  72  65  6e  63  65  73  20  62  65  74  77  65  65
0000220    n       y   o   u   r  \n   c   o   n   f   i   g   u   r   a
           6e  20  79  6f  75  72  0a  63  6f  6e  66  69  67  75  72  61
0000230    t   i   o   n       a   n   d       r   e   a   l       p   h
           74  69  6f  6e  20  61  6e  64  20  72  65  61  6c  20  70  68
0000240    y   s   i   c   a   l       r   e   s   o   u   r   c   e   s
           79  73  69  63  61  6c  20  72  65  73  6f  75  72  63  65  73
0000250        t   h   a   t       e   x   i   s   t   .       A   s
           20  74  68  61  74  20  65  78  69  73  74  2e  20  41  73  20
0000260    a       r   e   s   u   l   t   ,       n   o  \n   a   c   t
           61  20  72  65  73  75  6c  74  2c  20  6e  6f  0a  61  63  74
0000270    i   o   n   s       n   e   e   d       t   o       b   e
           69  6f  6e  73  20  6e  65  65  64  20  74  6f  20  62  65  20
0000280    p   e   r   f   o   r   m   e   d   . 033   [   0   m  \n
           70  65  72  66  6f  72  6d  65  64  2e  1b  5b  30  6d  0a
000028f

Additional Context

I had found this issue when parsing terraform plan output with a third party tool, terraform-landscape.

At first, I had thought it might be a landscape's bug, but after a while debugging, I found out that it was a terraform core's bug.

$ terraform plan | landscape --trace
/usr/local/Cellar/terraform_landscape/0.1.18/libexec/gems/terraform_landscape-0.1.18/lib/terraform_landscape/printer.rb:87:in `apply_prompt': invalid byte sequence in UTF-8 (ArgumentError)
        from /usr/local/Cellar/terraform_landscape/0.1.18/libexec/gems/terraform_landscape-0.1.18/lib/terraform_landscape/printer.rb:32:in `process_stream'
        from /usr/local/Cellar/terraform_landscape/0.1.18/libexec/gems/terraform_landscape-0.1.18/lib/terraform_landscape/cli.rb:48:in `print'
        from /usr/local/Cellar/terraform_landscape/0.1.18/libexec/gems/terraform_landscape-0.1.18/lib/terraform_landscape/cli.rb:34:in `block (2 levels) in define_commands'
        from /usr/local/Cellar/terraform_landscape/0.1.18/libexec/gems/commander-4.4.5/lib/commander/command.rb:182:in `call'
        from /usr/local/Cellar/terraform_landscape/0.1.18/libexec/gems/commander-4.4.5/lib/commander/command.rb:153:in `run'
        from /usr/local/Cellar/terraform_landscape/0.1.18/libexec/gems/commander-4.4.5/lib/commander/runner.rb:446:in `run_active_command'
        from /usr/local/Cellar/terraform_landscape/0.1.18/libexec/gems/commander-4.4.5/lib/commander/runner.rb:68:in `run!'
        from /usr/local/Cellar/terraform_landscape/0.1.18/libexec/gems/commander-4.4.5/lib/commander/delegates.rb:15:in `run!'
        from /usr/local/Cellar/terraform_landscape/0.1.18/libexec/gems/terraform_landscape-0.1.18/lib/terraform_landscape/cli.rb:25:in `run'
        from /usr/local/Cellar/terraform_landscape/0.1.18/libexec/gems/terraform_landscape-0.1.18/bin/landscape:7:in `<top (required)>'
        from /usr/local/Cellar/terraform_landscape/0.1.18/libexec/bin/landscape:22:in `load'
        from /usr/local/Cellar/terraform_landscape/0.1.18/libexec/bin/landscape:22:in `<main>'

References

#12261
#13292
#13317

Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.