S3 endpoint is defaulting to incorrect value in us-gov-east-1

[sam-mcbr]

Describe the bug

I'm running Mimir v2.9.0 on an AWS EKS cluster in the us-gov-east-1 region. I have my blocks storage configured as follows:

blocks_storage:
  backend: s3
  s3:
    bucket_name: <bucket name>
    endpoint: s3-fips.us-gov-east-1.amazonaws.com

The compactor and other components that reach out to S3 are failing with this error on startup:

caller=sanity_check.go:115 level=warn msg="Unable to successfully connect to configured object storage (will retry)" err="blocks storage: unable to successfully send a request to object storage: Get \"https://<our-bucket>.s3.dualstack.us-east-1.amazonaws.com/sanity-check-at-startup\": context deadline exceeded

caller=seed.go:127 level=warn msg="failed to read cluster seed file from object storage" err="Get \"https://<our-bucket>.s3.dualstack.us-east-1.amazonaws.com/__mimir_cluster/mimir_cluster_seed.json\": dial tcp <ip>: i/o timeout

I have confirmed that the blocks_storage.s3.endpoint value is set correctly to s3-fips.us-gov-east-1.amazonaws.com by using kubectl exec to get into the pod and checking the etc/mimir/mimir.yaml file, but for some reason the pods are actually trying to hit the s3.dualstack.us-east-1.amazonaws.com endpoint instead.

I have been successfully running Mimir with the same configuration in non-gov regions, so it seems like a problem specific to the AWS GovCloud environment.

To Reproduce

Steps to reproduce the behavior:

1. Start Mimir (v2.9.0)
2. Deploy into an AWS EKS cluster in the us-gov-east-1 region with the configuration specified above

Expected behavior

I expect Mimir to be trying to hit the S3 endpoint set in the configuration (s3-fips.us-gov-east-1.amazonaws.com), not s3.dualstack.us-east-1.amazonaws.com.

Environment

• Infrastructure: Kubernetes, AWS EKS
• Deployment tool: ArgoCD

Additional Context

Mimir is granted access to the S3 bucket through IRSA.

It is likely this is related to minio/minio-go#1880 which was patched in https://github.com/minio/minio-go/releases/tag/v7.0.63.

Can the minio-go version be upgrade to 7.0.63 to address this issue? Thanks!

[sam-mcbr]

I see dependabot updated the version in #5888 . Is there any chance another 2.10.0 release candidate version could be released with this change? Thanks so much!

[56quarters]

I've marked #5888 to be backported to 2.10: #5903

[sam-mcbr]

Thank you so much!