Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement lazy retrieval of series from object store. #5837

Merged
merged 14 commits into from
Nov 28, 2022
Merged

Implement lazy retrieval of series from object store. #5837

merged 14 commits into from
Nov 28, 2022

Commits on Nov 12, 2022

  1. Implement lazy retrieval of series from object store. 

    The bucket store fetches series in a single blocking operation from
object storage. This is likely not an ideal strategy when it comes to
latency and resource usage. In addition, it causes the store to buffer
everything in memory before starting to send results to queriers.

This commit modifies the series retrieval to use the proxy response heap
and take advantage of the k-way merge used in the proxy store.

Signed-off-by: Filip Petkovski <[email protected]>
    @fpetkovski
    fpetkovski committed Nov 12, 2022
    Configuration menu
    Copy the full SHA
    9ed9a91 View commit details
    Browse the repository at this point in the history

  2. Add batching 

    Signed-off-by: Filip Petkovski <[email protected]>
    @fpetkovski
    fpetkovski committed Nov 12, 2022
    Configuration menu
    Copy the full SHA
    ca12644 View commit details
    Browse the repository at this point in the history

  3. Preload series in batches 

    Signed-off-by: Filip Petkovski <[email protected]>
    @fpetkovski
    fpetkovski committed Nov 12, 2022
    Configuration menu
    Copy the full SHA
    1c5886b View commit details
    Browse the repository at this point in the history

  4. Emit proper stats 

    Signed-off-by: Filip Petkovski <[email protected]>
    @fpetkovski
    fpetkovski committed Nov 12, 2022
    Configuration menu
    Copy the full SHA
    f1e9538 View commit details
    Browse the repository at this point in the history

  5. Extract block series client 

    Signed-off-by: Filip Petkovski <[email protected]>
    @fpetkovski
    fpetkovski committed Nov 12, 2022
    Configuration menu
    Copy the full SHA
    e1327dc View commit details
    Browse the repository at this point in the history

  6. Fix CI 

    Signed-off-by: Filip Petkovski <[email protected]>
    @fpetkovski
    fpetkovski committed Nov 12, 2022
    Configuration menu
    Copy the full SHA
    6a8b06b View commit details
    Browse the repository at this point in the history

  7. Address review comments 

    Signed-off-by: Filip Petkovski <[email protected]>
    @fpetkovski
    fpetkovski committed Nov 12, 2022
    Configuration menu
    Copy the full SHA
    89d237f View commit details
    Browse the repository at this point in the history

  8. Use emptyPostingsCount in lazyRespSet 

    Signed-off-by: Filip Petkovski <[email protected]>
    @fpetkovski
    fpetkovski committed Nov 12, 2022
    Configuration menu
    Copy the full SHA
    774016a View commit details
    Browse the repository at this point in the history

  9. Reuse chunk metas 

    Signed-off-by: Filip Petkovski <[email protected]>
    @fpetkovski
    fpetkovski committed Nov 12, 2022
    Configuration menu
    Copy the full SHA
    30143bc View commit details
    Browse the repository at this point in the history

  10. Avoid overallocating for small responses 

    Signed-off-by: Filip Petkovski <[email protected]>
    @fpetkovski
    fpetkovski committed Nov 12, 2022
    Configuration menu
    Copy the full SHA
    f2c4ecc View commit details
    Browse the repository at this point in the history

  11. Add metric for chunk fetch time 

    Signed-off-by: Filip Petkovski <[email protected]>
    @fpetkovski
    fpetkovski committed Nov 12, 2022
    Configuration menu
    Copy the full SHA
    8946647 View commit details
    Browse the repository at this point in the history

Commits on Nov 13, 2022

  1. Regroup imports 

    Signed-off-by: Filip Petkovski <[email protected]>
    @fpetkovski
    fpetkovski committed Nov 13, 2022
    Configuration menu
    Copy the full SHA
    cbc3d23 View commit details
    Browse the repository at this point in the history

Commits on Nov 14, 2022

  1. Change counter to uint64 

    Signed-off-by: Filip Petkovski <[email protected]>
    @fpetkovski
    fpetkovski committed Nov 14, 2022
    Configuration menu
    Copy the full SHA
    25ed544 View commit details
    Browse the repository at this point in the history

Commits on Nov 27, 2022

  1. Merge branch 'main' into lazy-bucket-store 

    Signed-off-by: Filip Petkovski <[email protected]>
Signed-off-by: fpetkovski <[email protected]>
    @fpetkovski
    fpetkovski committed Nov 27, 2022
    Configuration menu
    Copy the full SHA
    57a3124 View commit details
    Browse the repository at this point in the history