You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The compactor and other components that reach out to S3 are failing with this error on startup:
caller=sanity_check.go:115 level=warn msg="Unable to successfully connect to configured object storage (will retry)" err="blocks storage: unable to successfully send a request to object storage: Get \"https://<our-bucket>.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded
caller=seed.go:127 level=warn msg="failed to read cluster seed file from object storage" err="Get \"https://<our-bucket>.s3.dualstack.us-east-1.amazonaws.com/__mimir_cluster/mimir_cluster_seed.json\": dial tcp <ip>: i/o timeout
I have confirmed that the blocks_storage.s3.endpoint value is set correctly to s3-fips.us-gov-east-1.amazonaws.com by using kubectl exec to get into the pod and checking the etc/mimir/mimir.yaml file, but for some reason the pods are actually trying to hit the s3.dualstack.us-east-1.amazonaws.com endpoint instead.
I have been successfully running Mimir with the same configuration in non-gov regions, so it seems like a problem specific to the AWS GovCloud environment.
To Reproduce
Steps to reproduce the behavior:
Start Mimir (v2.9.0)
Deploy into an AWS EKS cluster in the us-gov-east-1 region with the configuration specified above
Expected behavior
I expect Mimir to be trying to hit the S3 endpoint set in the configuration (s3-fips.us-gov-east-1.amazonaws.com), not s3.dualstack.us-east-1.amazonaws.com.
Environment
Infrastructure: Kubernetes, AWS EKS
Deployment tool: ArgoCD
Additional Context
Mimir is granted access to the S3 bucket through IRSA.
I see dependabot updated the version in #5888 . Is there any chance another 2.10.0 release candidate version could be released with this change? Thanks so much!
Describe the bug
I'm running Mimir v2.9.0 on an AWS EKS cluster in the us-gov-east-1 region. I have my blocks storage configured as follows:
The compactor and other components that reach out to S3 are failing with this error on startup:
I have confirmed that the blocks_storage.s3.endpoint value is set correctly to s3-fips.us-gov-east-1.amazonaws.com by using kubectl exec to get into the pod and checking the etc/mimir/mimir.yaml file, but for some reason the pods are actually trying to hit the s3.dualstack.us-east-1.amazonaws.com endpoint instead.
I have been successfully running Mimir with the same configuration in non-gov regions, so it seems like a problem specific to the AWS GovCloud environment.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
I expect Mimir to be trying to hit the S3 endpoint set in the configuration (s3-fips.us-gov-east-1.amazonaws.com), not s3.dualstack.us-east-1.amazonaws.com.
Environment
Additional Context
Mimir is granted access to the S3 bucket through IRSA.
It is likely this is related to minio/minio-go#1880 which was patched in https://github.com/minio/minio-go/releases/tag/v7.0.63.
Can the minio-go version be upgrade to
7.0.63
to address this issue? Thanks!The text was updated successfully, but these errors were encountered: