-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
compact: Introduce smart bucket caching for metadata with TTL; replace fetcher cache with it. #3408
Comments
Code to bucket caching: thanos/pkg/store/cache/caching_bucket.go Line 37 in 843307a
Fetcher dir and memory caching Line 149 in 2ad3805
|
Does it make sense to divide the tasks you mentioned above and send a PR for each one separately? |
Definitely (: |
or both! (: No, what I mean rather is what KIND of files to cache where. Currently, we cache the only meta.json per block in mem and optionally on disk (fetcher). Plus if you do bucket caching does its own thing. Currently caching bucket implementation is quite flexible and good, but I think we might want to improve configuring YAML and how it's used in our code. For example:
// Config for Exists and Get operations for metadata files.
MetafileExistsTTL time.Duration `yaml:"metafile_exists_ttl"`
MetafileDoesntExistTTL time.Duration `yaml:"metafile_doesnt_exist_ttl"`
MetafileContentTTL time.Duration `yaml:"metafile_content_ttl"`
MetafileMaxSize model.Bytes `yaml:"metafile_max_size"` Would be nice to specify this configuration per each "metadata" file. (make this more generic, per file name). We could then cache
cc @Sudhar287 and @pstibrany who wrote the caching bucket we want to reuse more (: |
You risk having too many config options. In Cortex we have these:
|
Yes. I would aim to define some defaults first, THEN allow configuring those only if really there is use case, but good point 👍 |
@bwplotka I'm trying to scope this issue. Can you please elaborate on
Do you mean additional test cases for the
|
No I mean cache jitter rather (See: http://neopatel.blogspot.com/2012/04/adding-jitter-to-cache-layer.html (point no 2)) The problem is that sync is done once and then all metadata are cached in the same time. With no jitter cache TTL (expiry) we revoke all blocks in the same time, so we redownload all in one go introducing spike in bucket APIs and potential slow downs or even rate limits. If we add some randomness to TTL it will allow us to potentially uniform cache revokes and cache refresh |
Hello 👋 Looks like there was no activity on this issue for the last two months. |
Closing for now as promised, let us know if you need this to be reopened! 🤗 |
This can be reopened as point 5 |
Hello 👋 Looks like there was no activity on this issue for the last two months. |
Closing for now as promised, let us know if you need this to be reopened! 🤗 |
Closing for now as promised, let us know if you need this to be reopened! 🤗 |
1 similar comment
Closing for now as promised, let us know if you need this to be reopened! 🤗 |
Hello 👋 Looks like there was no activity on this issue for the last two months. |
Closing for now as promised, let us know if you need this to be reopened! 🤗 |
Action Items:
The text was updated successfully, but these errors were encountered: