Improvements on Thanos Compactor

[yashrsharma44]

Description of the Problem

Thanos Compactor has a large scope of improvement, particularly scaling out and running Compactor concurrently, so this umbrella issue tracks all the expected improvements, proof of concepts and design proposals for improving the Compactor.

Describe the solution you'd like

There are a few important issues/discussions that are relevant to this issue -

Design Docs

• Scalable Compactor Doc - https://docs.google.com/document/d/1xi0V8DB0hE54XgkogJRnNL6yH7C5JThJywlLFoC6dCQ/edit
• LFX Mentoring Proposal - Vertical Block Sharding - Vertical Block Sharding Proposal #4191
• Vertical Block Sharding - proposal: Added proposal for vertical block sharding (limit size of block) #3390

Pull Requests/Issues

• Compaction Grouper Concurrency - Compaction group concurrency #3807
• Streamed Block Reading - compact/downsampling/rewrite: Read chunks directly from object storage.  #3416
• Delete old data - Compactor should ignore dirty data or automatically delete corrupted data #4046
• Progress Calculation for Compactor - Have easier to way to tell the progress of compactor (and downsampling) #3985
• Upper Bound - compactor: Consider upper bound of block size #2340
• Penalty based deduplication (offline deduplication) - compactor: Add offline deduplication #4239

P.S. Feel free to add your Proof of Concepts/Suggestions/Discussions

cc @thanos-io/thanos-maintainers

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[yeya24]

Not stale.

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[stale]

Closing for now as promised, let us know if you need this to be reopened! 🤗

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[stale]

Closing for now as promised, let us know if you need this to be reopened! 🤗

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[bill3tt]

@yashrsharma44 this is a concise & well written umbrella issue with load of relevant context - kudos!