Unable to use insecure HTTP in S3 config

[crockk]

Hi,

I'm having trouble skipping HTTPS and using insecure HTTP. I am connecting to a local MinIO S3 bucket using the following config:

type: S3
prefix: "prometheus4"
config:
  bucket: "prometheus4"
  endpoint: "my.local.ip.here"
  access_key: "thanos4"
  secret_key: "xxxxxxxx"
  insecure: true
  trace:
    enable: true
  http_config:
    insecure_skip_verify: true
    tls_config:
      insecure_skip_verify: true

However in my Thanos Store container I can see it's still trying to use HTTPS:

level=info ts=2022-07-11T20:52:37.610274875Z caller=factory.go:50 msg="loading bucket configuration"
level=info ts=2022-07-11T20:52:37.612014695Z caller=inmemory.go:174 msg="created in-memory index cache" maxItemSizeBytes=131072000 maxSizeBytes=262144000 maxItems=maxInt
level=info ts=2022-07-11T20:52:37.613143475Z caller=options.go:27 protocol=gRPC msg="disabled TLS, key and cert must be set to enable"
level=info ts=2022-07-11T20:52:37.621642489Z caller=store.go:446 msg="starting store node"
level=info ts=2022-07-11T20:52:37.622502006Z caller=intrumentation.go:75 msg="changing probe status" status=healthy
level=info ts=2022-07-11T20:52:37.622546165Z caller=http.go:73 service=http/server component=store msg="listening for requests and metrics" address=0.0.0.0:10906
level=info ts=2022-07-11T20:52:37.623278221Z caller=store.go:364 msg="initializing bucket store"
level=info ts=2022-07-11T20:52:37.625150507Z caller=tls_config.go:195 service=http/server component=store msg="TLS is disabled." http2=false
level=warn ts=2022-07-11T20:52:37.62940911Z caller=intrumentation.go:67 msg="changing probe status" status=not-ready reason="bucket store initial sync: sync block: BaseFetcher: iter bucket: Get \"https://my.local.ip.here/prometheus4/?location=\": remote error: tls: internal error"
level=info ts=2022-07-11T20:52:37.630107614Z caller=http.go:84 service=http/server component=store msg="internal server is shutting down" err="bucket store initial sync: sync block: BaseFetcher: iter bucket: Get \"https://my.local.ip.here/prometheus4/?location=\": remote error: tls: internal error"
level=info ts=2022-07-11T20:52:37.630257266Z caller=intrumentation.go:56 msg="changing probe status" status=ready
level=info ts=2022-07-11T20:52:37.63040577Z caller=grpc.go:131 service=gRPC/server component=store msg="listening for serving gRPC" address=0.0.0.0:10905
level=info ts=2022-07-11T20:52:37.631600194Z caller=http.go:103 service=http/server component=store msg="internal server is shutdown gracefully" err="bucket store initial sync: sync block: BaseFetcher: iter bucket: Get \"https://my.local.ip.here/prometheus4/?location=\": remote error: tls: internal error"
level=info ts=2022-07-11T20:52:37.631821134Z caller=intrumentation.go:81 msg="changing probe status" status=not-healthy reason="bucket store initial sync: sync block: BaseFetcher: iter bucket: Get \"https://my.local.ip.here/prometheus4/?location=\": remote error: tls: internal error"
level=warn ts=2022-07-11T20:52:37.632217549Z caller=intrumentation.go:67 msg="changing probe status" status=not-ready reason="bucket store initial sync: sync block: BaseFetcher: iter bucket: Get \"https://my.local.ip.here/prometheus4/?location=\": remote error: tls: internal error"
level=info ts=2022-07-11T20:52:37.632271453Z caller=grpc.go:138 service=gRPC/server component=store msg="internal server is shutting down" err="bucket store initial sync: sync block: BaseFetcher: iter bucket: Get \"https://my.local.ip.here/prometheus4/?location=\": remote error: tls: internal error"
level=info ts=2022-07-11T20:52:37.632358865Z caller=grpc.go:151 service=gRPC/server component=store msg="gracefully stopping internal server"
level=info ts=2022-07-11T20:52:37.632463859Z caller=grpc.go:164 service=gRPC/server component=store msg="internal server is shutdown gracefully" err="bucket store initial sync: sync block: BaseFetcher: iter bucket: Get \"https://my.local.ip.here/prometheus4/?location=\": remote error: tls: internal error"
level=error ts=2022-07-11T20:52:37.633749171Z caller=main.go:158 err="Get \"https://my.local.ip.here/prometheus4/?location=\": remote error: tls: internal error\nBaseFetcher: iter bucket\ngithub.com/thanos-io/thanos/pkg/block.(*BaseFetcher).fetchMetadata\n\t/app/pkg/block/fetcher.go:375\ngithub.com/thanos-io/thanos/pkg/block.(*BaseFetcher).fetch.func2\n\t/app/pkg/block/fetcher.go:439\ngithub.com/golang/groupcache/singleflight.(*Group).Do\n\t/go/pkg/mod/github.com/golang/[email protected]/singleflight/singleflight.go:56\ngithub.com/thanos-io/thanos/pkg/block.(*BaseFetcher).fetch\n\t/app/pkg/block/fetcher.go:437\ngithub.com/thanos-io/thanos/pkg/block.(*MetaFetcher).Fetch\n\t/app/pkg/block/fetcher.go:497\ngithub.com/thanos-io/thanos/pkg/store.(*BucketStore).SyncBlocks\n\t/app/pkg/store/bucket.go:457\ngithub.com/thanos-io/thanos/pkg/store.(*BucketStore).InitialSync\n\t/app/pkg/store/bucket.go:526\nmain.runStore.func3\n\t/app/cmd/thanos/store.go:366\ngithub.com/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/[email protected]/group.go:38\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1571\nsync block\ngithub.com/thanos-io/thanos/pkg/store.(*BucketStore).InitialSync\n\t/app/pkg/store/bucket.go:527\nmain.runStore.func3\n\t/app/cmd/thanos/store.go:366\ngithub.com/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/[email protected]/group.go:38\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1571\nbucket store initial sync\nmain.runStore.func3\n\t/app/cmd/thanos/store.go:368\ngithub.com/oklog/run.(*Group).Run.func1\n\t/go/pkg/mod/github.com/oklog/[email protected]/group.go:38\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1571\nstore command failed\nmain.main\n\t/app/cmd/thanos/main.go:158\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1571"

For reference, the relevant docker-compose.yml:

  store:
    image: thanosio/thanos:main-2022-07-08-47dcda70
    container_name: thanos-store
    restart: always
    command:
      - "store"
      - "--grpc-address=0.0.0.0:10905"
      - "--http-address=0.0.0.0:10906"
      - "--data-dir=/data/store"
      - "--objstore.config-file=/bucket.yml"
    volumes:
      - store:/data
      - ./minio_bucket_config.yml:/bucket.yml
    networks:
      - thanos

I would imagine (hope) that this is not a bug but me just overlooking something here... So I'm posting this as a discussion question and not a bug report.

Any support would be appreciated!