Sidecar: Blocks not uploaded with KMS encryption enabled

[timkra]

Thanos, Prometheus and Golang version used:

• Thanos: v0.19.0
• Prometheus: v2.26.0
• Golang: 1.15

Object Storage Provider:
S3

What happened:
I configured S3 Server-Side Encryption, whenever the sidecar component tries to upload a new block an error is raised and the upload fails.

What you expected to happen:
Blocks are beeing uploaded to S3.

How to reproduce it (as minimally and precisely as possible):
Create a KMS key, configure S3 SSE-KMS with the previously created KMS key, Create an IAM role for Thanos, grant this role access to the KMS Key and configure the Objectstore S3 Server-Side Encryption with the KMS key.

Objectstore config:

objstore.yml

type: S3
config:
  bucket: MY-BUCKET
  endpoint: s3.eu-central-1.amazonaws.com
  region: eu-central-1
  insecure: false
  signature_version2: false
  put_user_metadata: {}
  http_config: null
  idle_conn_timeout: 90s
  response_header_timeout: 2m
  insecure_skip_verify: false
  trace:
    enable: false
  part_size: 134217728
  sse_config:
    type: SSE-KMS
    kms_key_id: MY-KEY-ID

IAM Policy

policy.json

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Statement",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetObject",
                "s3:DeleteObject",
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::MY-BUCKET/*",
                "arn:aws:s3:::MY-BUCKET"
            ]
        },
        {
            "Sid": "KMSAccess",
            "Effect": "Allow",
            "Action": [
                "kms:GenerateDataKey",
                "kms:Encrypt",
                "kms:Decrypt"
            ],
            "Resource": "arn:aws:kms:eu-central-1:MY-ACCOUNT-ID:key/MY-KEY-ID"
        }
    ]
}

Full logs to relevant components:

level=info ts=2021-05-18T11:50:16.519827134Z caller=shipper.go:337 msg="upload new block" id=01F5ZJ44TM8D0VWNNWEB4RVEY5
level=warn ts=2021-05-18T11:50:16.622554333Z caller=sidecar.go:277 err="upload 01F5ZJ44TM8D0VWNNWEB4RVEY5: upload debug meta file: upload s3 object: The header 'x-amz-server-side-encryption-context' shall be Base64-encoded UTF-8 string holding JSON which represents a string-string map" uploaded=0

Anything else we need to know:

The same error is raised on the compactor component.

[timkra]

I did some further testing and found that setting kms_encryption_context with any key/value pair results in the block beeing uploaded.

The current Objectstore config is as:

objstore.yml

type: S3
config:
  bucket: MY-BUCKET
  endpoint: s3.eu-central-1.amazonaws.com
  region: eu-central-1
  insecure: false
  signature_version2: false
  put_user_metadata: {}
  http_config: null
  idle_conn_timeout: 90s
  response_header_timeout: 2m
  insecure_skip_verify: false
  trace:
    enable: false
  part_size: 134217728
  sse_config:
    type: SSE-KMS
    kms_key_id: MY-KEY-ID
    kms_encryption_context:
      foo: bar

level=info ts=2021-05-21T09:00:14.249963107Z caller=shipper.go:337 msg="upload new block" id=01F672EAPWT1AJZFYWN0QKCF5P
level=debug ts=2021-05-21T09:00:14.542675693Z caller=objstore.go:206 msg="uploaded file" from=/prometheus/thanos/upload/01F672EAPWT1AJZFYWN0QKCF5P/chunks/000001 dst=01F672EAPWT1AJZFYWN0QKCF5P/chunks/000001 bucket="tracing: MY-BUCKET" 

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[stale]

Closing for now as promised, let us know if you need this to be reopened! 🤗

[avestuk]

I experienced the same issue as @timkra using v0.23.1 and I was able to narrow the issue down to the following:

When I had the following S3 config:

        objstoreConfig: |-
          type: S3
          config:
            sse_config:
              type: "SSE-KMS"
              kms_key_id: "redacted"
            trace:
              enable: true

Requests as follows were being made:

PUT /debug/metas/01FSVBDNFW9QAS5JH7XXPGMP7Z.json HTTP/1.1
X-Amz-Server-Side-Encryption-Context: bnVsbA==  aka:  null

prometheus-server-0 thanos-sc level=debug ts=2022-01-20T16:02:01.96560183Z caller=stdlib.go:105 s3TraceMsg="HTTP/1.1 400 Bad Request
The header 'x-amz-server-side-encryption-context' shall be Base64-encoded UTF-8 string holding JSON which represents a string-string map

After updating the config

        objstoreConfig: |-
          type: S3
          config:
            sse_config:
              type: "SSE-KMS"
              kms_key_id: "redacted"
              kms_encryption_context:
                foo: bar
            trace:
              enable: true

"PUT /01FSW00V1N5VQ1419R9S5R87C3/meta.json HTTP/1.1
X-Amz-Server-Side-Encryption-Context: Zm9vOiBiYXI= aka: foo: bar

When using the following config:

        objstoreConfig: |-
          type: S3
          config:
            sse_config:
              type: "SSE-KMS"
              kms_key_id: "redacted"
              kms_encryption_context: {}
            trace:
              enable: true

"PUT /01FSW00V1N5VQ1419R9S5R87C3/meta.json HTTP/1.1
X-Amz-Server-Side-Encryption-Context: e30= aka {}

tl;dr If you use SSE-KMS you must set a value for kms_encryption_context even if it is an empty map