S3: Unable to sanity check object store

[JBodkin-Amphora]

Describe the bug

When starting mimir with S3 configured as the block store, the sanity check is failing on our environment with a context deadline exeeded error message.

level=info ts=2022-07-30T15:29:52.08092532Z caller=sanity_check.go:40 msg="Checking object storage config"
level=warn ts=2022-07-30T15:33:57.254231096Z caller=sanity_check.go:116 msg="Unable to successfully connect to configured object storage (will retry)" err="2 errors: blocks storage: unable to successfully send a request to object storage: Get "https://removed.s3.dualstack.eu-west-2.amazonaws.com/blocks/sanity-check-at-startup\": context deadline exceeded; ruler storage: unable to successfully send a request to object storage: Get "https://removed.s3.dualstack.eu-west-2.amazonaws.com/ruler/sanity-check-at-startup\": context deadline exceeded"

Mimir is running inside docker using docker-compose, on an IPv4 AWS VPC.

To Reproduce

Steps to reproduce the behavior:

Uncertain

Expected behavior

The same configuration applied to Tempo is working correctly. I would expect the same to work here.

Environment

• Infrastructure: AWS, Virtual Machine, Docker, Docker Compose

Additional Context

The S3 endpoint is configured to use the non-dualstack endpoint, but the error message above seems to be wanting the dualstack endpoint. I'm wondering if its trying to use IPv6, which isn't supported in our environment.

blocks_storage:
  backend: s3
  storage_prefix: blocks
  s3:
    bucket_name: removed
    endpoint: s3.eu-west-2.amazonaws.com

[JBodkin-Amphora]

Okay, I figured out that this is an issue with fetching the credentials. After putting in a lot of logging statements, I found that the call to fetchIMDSToken in https://github.com/grafana/mimir/blob/main/vendor/github.com/minio/minio-go/v7/pkg/credentials/iam_aws.go#L292 is hanging. When this call times out, after the default of 30 seconds, it will return an empty string as an timeout error will occur.

I think the sanity check cancels the lookup earlier than the 30 seconds, so it doesn't get the chance to proceed. I tried increasing this up to 60, but didn't seem to make any difference. https://github.com/grafana/mimir/blob/main/pkg/mimir/sanity_check.go#L150

I couldn't see any in tempo does the same timeout check, so I expect that is why it works.

[pracucci]

Interesting! However, I would like to learn more about why increasing the hop limit fixes to the issue to you.

The timeout for the API call made by fetchIMDSToken() to the AWS metadata service is actually 1 second, set here:

mimir/vendor/github.com/minio/minio-go/v7/pkg/credentials/iam_aws.go

Lines 258 to 259 in e92aee1

	ctx, cancel := context.WithTimeout(context.Background(), time.Second)
	defer cancel()

What have you seen in your additional log statements? How long did fetchIMDSToken() take?

[JBodkin-Amphora]

Looking into the logs that I have stored, it was taking around a couple of minutes between the two function calls in that file.

e.g.

2022-07-30 18:42:42 | IAM
2022-07-30 18:42:42 | fetchIMDSToken
2022-07-30 18:44:45 | getIAMRoleURL

Sadly, I don't remember if I placed these specific logging statements into the code before increasing the timeout in sanity_check.go to 60*time.Second, in case there were any unintended consequences by changing this timeout. I didn't spot that the fetchIMDSToken had a 1 second timeout.

[JBodkin-Amphora]

I found out that Amazon monitor the hop count when connecting to the metadata service for a PUT request. The default is 1, whereas I need 2 hops since I'm running inside docker. This can be modified using the CLI to increase the hop limit.

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html#imds-considerations

aws ec2 modify-instance-metadata-options --instance-id removed --http-put-response-hop-limit 2

[ghost]

To enable full debug logging using https://github.com/grafana/mimir/blob/main/operations/helm/charts/mimir-distributed/values.yaml I tried adding:

  server:
    log_level: debug

Which did not seem to have much effect Is there some other value you are setting in the helm-chart to enable the same type of output you are seeing? I am using an S3 compatible bucket but having similar issues and want to try to resolve it myself before waisting to many peoples time for something I have misconfigured.

[JBodkin-Amphora]

I wasn't able to gain any useful information from the debug logs, hence I modified the code to add some print statements to stderr and pushed the docker image to our internal docker registry.