Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parallelize vertical compactions inside a single group #5936

Closed
wants to merge 22 commits into from
Closed

Parallelize vertical compactions inside a single group #5936

Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
912172b
Parallelize vertical compactions
fpetkovski Dec 1, 2022
d71c4d1
Run all tasks independently
fpetkovski Dec 2, 2022
c911229
Use errutil.MultiError instead of cortex MultiError
fpetkovski Dec 2, 2022
afe27df
Add test case for progress calculator
fpetkovski Dec 2, 2022
321efdc
Remove ULID return value
fpetkovski Dec 2, 2022
31885e1
Iterate through all errors for a compaction group
fpetkovski Dec 2, 2022
86af2a8
Add test case for large index size
fpetkovski Dec 2, 2022
1248a96
Parametrize concurrency inside single group
fpetkovski Dec 2, 2022
09bdb1a
Change test name
fpetkovski Dec 3, 2022
b58d2bf
Limit concurrency of tasks to global concurrency
fpetkovski Dec 4, 2022
c804045
Improve flag description
fpetkovski Dec 4, 2022
d754370
Fix linter
fpetkovski Dec 4, 2022
24fd336
Add vertical compaction check
fpetkovski Dec 4, 2022
a808840
Adjust e2e tests to not account for empty planning cycles
fpetkovski Dec 4, 2022
7c6dba5
Run make docs
fpetkovski Dec 4, 2022
a7d4a7b
Automatically infer group concurrency
fpetkovski Dec 7, 2022
f476edb
Run make docs
fpetkovski Dec 7, 2022
2662961
Fix tests
fpetkovski Dec 7, 2022
407e98a
Distribute tasks in round-robin manner
fpetkovski Dec 7, 2022
25705dc
Add e2e test for parallel compaction
fpetkovski Dec 7, 2022
e3e94c6
Plan at least one task from each group
fpetkovski Dec 7, 2022
ea766ad
Clean up scheduling
fpetkovski Dec 12, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion cmd/thanos/compact.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -352,7 +352,7 @@ func runCompact(
conf.blockFilesConcurrency,
conf.compactBlocksFetchConcurrency,
)
tsdbPlanner := compact.NewPlanner(logger, levels, noCompactMarkerFilter)
tsdbPlanner := compact.NewPlanner(logger, levels, noCompactMarkerFilter, conf.groupTasksConcurrency)
planner := compact.WithLargeTotalIndexSizeFilter(
tsdbPlanner,
bkt,
Expand Down Expand Up @@ -661,6 +661,7 @@ type compactConfig struct {
cleanupBlocksInterval time.Duration
compactionConcurrency int
downsampleConcurrency int
groupTasksConcurrency int
compactBlocksFetchConcurrency int
deleteDelay model.Duration
dedupReplicaLabels []string
Expand Down Expand Up @@ -746,6 +747,10 @@ func (cc *compactConfig) registerFlag(cmd extkingpin.FlagClause) {
"NOTE: This flag is ignored and (enabled) when --deduplication.replica-label flag is set.").
Hidden().Default("false").BoolVar(&cc.enableVerticalCompaction)

cmd.Flag("compact.group-concurrency", "The number of concurrent compactions from a single compaction group inside one compaction iteration. "+
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Can we add this flag next to compact.concurrency for readability?
  2. Are we sure we need this granuality of config? Is there a way to automatically detect this from compact.concurrency?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love the idea of detecting based on compact.concurrency.

"The absolute concurrency between all compactions is still limited by the compact.concurrency flag.").
Default("1").IntVar(&cc.groupTasksConcurrency)

cmd.Flag("deduplication.func", "Experimental. Deduplication algorithm for merging overlapping blocks. "+
"Possible values are: \"\", \"penalty\". If no value is specified, the default compact deduplication merger is used, which performs 1:1 deduplication for samples. "+
"When set to penalty, penalty based deduplication algorithm will be used. At least one replica label has to be set via --deduplication.replica-label flag.").
Expand Down
6 changes: 6 additions & 0 deletions docs/components/compact.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -308,6 +308,12 @@ Flags:
happen at the end of an iteration.
--compact.concurrency=1 Number of goroutines to use when compacting
groups.
--compact.group-concurrency=1
The number of concurrent compactions from a
single compaction group inside one compaction
iteration. The absolute concurrency between
all compactions is still limited by the
compact.concurrency flag.
--compact.progress-interval=5m
Frequency of calculating the compaction progress
in the background when --wait has been enabled.
Expand Down
Loading