how to limit/throttle the data fetched form the s3 bucket

[nbari]

I have am running thanos-store

 thanos-store --version
thanos, version 0.11.0 (branch: HEAD, revision: 31cacc4930cc9c9ca9452fa149f89ee06248bd29)
  build user:       root@1118a616e31e
  build date:       20200302-14:45:46
  go version:       go1.13.6

With this config:

thanos-store store --objstore.config-file /home/prometheus/thanos_s3_bucket_config.yaml \
--data-dir /home/prometheus/thanos_store_cache/ \
--http-address=localhost:9301 \
--grpc-address=localhost:9201 \
--max-time=-180d

Because of the amount of data fetched from the s3 bucket, I am frequently hitting the bandwidth limits of the VM, wondering if there is a way to rate-limit or prevent frequently downloading all data?

From my understanding, my query range is defined by the --max-time, if I decrease it may help, but is there a way to use the local disk as a cache so that I could only download less data?

[wiardvanrij]

There is at the moment not a way to throttle that. To limit it, implement caches on multiple levels. So, frontend query cache, bucket cache, index cache. Furthermore you can add 'limits' to other components on how much series they should return and queries to process. For this I would recommend the documentation and check all the flags for your possibilities (there are multiple ways basically)

[nbari]

Just in case help for others, my current workaround was to use an HTTP PROXY (tinyproxy) in FreeBSD with ALTQ, here are the basic PF rules that limit the proxy to use 20% bandwidth out of 10MB:

# The IP's of the thanos nodes
table <trust> {x.x.x.x y.y.y.y z.z.z.z}

altq on vtnet0 cbq bandwidth 10Mb queue { std, ssh }
queue std bandwidth 20% cbq(default)
queue ssh bandwidth 80%

block in proto { tcp, udp } from any to any port 8118
pass from <trust> to any