Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make rollup update interval configurable when rollup-conf set to "auto". #186

Merged
merged 3 commits into from
Jul 11, 2022
Merged

Make rollup update interval configurable when rollup-conf set to "auto". #186

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
deca9d1
Introduce rollup-auto-interval config parameter.
lexx-bright Jul 5, 2022
9df9bf7
Mentioned rollup-auto-interval parameter in the documentation.
lexx-bright Jul 5, 2022
04fd016
Set rollup-auto-interval to 1 minute by default.
lexx-bright Jul 11, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion config/config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -148,6 +148,7 @@ type DataTable struct {
TargetMatchAllRegexp *regexp.Regexp `toml:"-" json:"-"`
RollupConf string `toml:"rollup-conf" json:"-" comment:"custom rollup.xml file for table, 'auto' and 'none' are allowed as well"`
RollupAutoTable string `toml:"rollup-auto-table" json:"rollup-auto-table" comment:"custom table for 'rollup-conf=auto', useful for Distributed or MatView"`
RollupAutoInterval *time.Duration `toml:"rollup-auto-interval" json:"rollup-auto-interval" comment:"rollup update interval for 'rollup-conf=auto'"`
RollupDefaultPrecision uint32 `toml:"rollup-default-precision" json:"rollup-default-precision" comment:"is used when none of rules match"`
RollupDefaultFunction string `toml:"rollup-default-function" json:"rollup-default-function" comment:"is used when none of rules match"`
RollupUseReverted bool `toml:"rollup-use-reverted" json:"rollup-use-reverted" comment:"should be set to true if you don't have reverted regexps in rollup-conf for reversed tables"`
Expand Down Expand Up @@ -270,10 +271,12 @@ func PrintDefaultConfig() error {
}

if len(cfg.DataTable) == 0 {
interval := time.Minute
cfg.DataTable = []DataTable{
{
Table: "graphite_data",
RollupConf: "auto",
RollupAutoInterval: &interval,
},
}
}
Expand Down Expand Up @@ -452,11 +455,15 @@ func (c *Config) ProcessDataTables() (err error) {
rdf := c.DataTable[i].RollupDefaultFunction
if c.DataTable[i].RollupConf == "auto" || c.DataTable[i].RollupConf == "" {
table := c.DataTable[i].Table
interval := time.Minute
if c.DataTable[i].RollupAutoTable != "" {
table = c.DataTable[i].RollupAutoTable
}
if c.DataTable[i].RollupAutoInterval != nil {
interval = *c.DataTable[i].RollupAutoInterval
}

c.DataTable[i].Rollup, err = rollup.NewAuto(c.ClickHouse.URL, table, time.Minute, rdp, rdf)
c.DataTable[i].Rollup, err = rollup.NewAuto(c.ClickHouse.URL, table, interval, rdp, rdf)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to keep it as it is, 1 minute. It doesn't harm, but everybody will be surprised if eventually, the auto-rollup update will require service restart

} else if c.DataTable[i].RollupConf == "none" {
c.DataTable[i].Rollup, err = rollup.NewDefault(rdp, rdf)
} else {
Expand Down
2 changes: 1 addition & 1 deletion deploy/doc/config.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ The rollup configuration is used for a proper metrics pre-aggregation. It conta

Historically, the way to define the config was `rollup-conf = "/path/to/the/conf/with/graphite_rollup.xml"`. The format is the same as [graphite_rollup](https://clickhouse.tech/docs/en/engines/table-engines/mergetree-family/graphitemergetree/#rollup-configuration) scheme for ClickHouse server.

For a quite long time it's recommended to use `rollup-conf = "auto"` to get the configuration from remote ClickHouse server. It will update itself each minute.
For a quite long time it's recommended to use `rollup-conf = "auto"` to get the configuration from remote ClickHouse server. It will update itself on each `rollup-auto-interval` (1 minute by default) or once on startup if set to "0s".

If you don't use a `GraphiteMergeTree` family engine, you can still use `rollup-conf = "auto"` by setting `rollup-auto-table="graphiteMergeTreeTable"` and get the proper config. In this case `graphiteMergeTreeTable` is a dummy table associated with proper [graphite_rollup](https://clickhouse.tech/docs/en/engines/table-engines/mergetree-family/graphitemergetree/#rollup-configuration). The cases when you may need it:

Expand Down
4 changes: 3 additions & 1 deletion doc/config.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ The rollup configuration is used for a proper metrics pre-aggregation. It conta

Historically, the way to define the config was `rollup-conf = "/path/to/the/conf/with/graphite_rollup.xml"`. The format is the same as [graphite_rollup](https://clickhouse.tech/docs/en/engines/table-engines/mergetree-family/graphitemergetree/#rollup-configuration) scheme for ClickHouse server.

For a quite long time it's recommended to use `rollup-conf = "auto"` to get the configuration from remote ClickHouse server. It will update itself each minute.
For a quite long time it's recommended to use `rollup-conf = "auto"` to get the configuration from remote ClickHouse server. It will update itself on each `rollup-auto-interval` (1 minute by default) or once on startup if set to "0s".

If you don't use a `GraphiteMergeTree` family engine, you can still use `rollup-conf = "auto"` by setting `rollup-auto-table="graphiteMergeTreeTable"` and get the proper config. In this case `graphiteMergeTreeTable` is a dummy table associated with proper [graphite_rollup](https://clickhouse.tech/docs/en/engines/table-engines/mergetree-family/graphitemergetree/#rollup-configuration). The cases when you may need it:

Expand Down Expand Up @@ -189,6 +189,8 @@ It's possible to set multiple loggers. See `Config` description in [config.go](h
rollup-conf = "auto"
# custom table for 'rollup-conf=auto', useful for Distributed or MatView
rollup-auto-table = ""
# rollup update interval for 'rollup-conf=auto'
rollup-auto-interval = "1m0s"
# is used when none of rules match
rollup-default-precision = 0
# is used when none of rules match
Expand Down
4 changes: 3 additions & 1 deletion helper/rollup/rollup.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,8 +104,10 @@ func (r *Rollup) updateWorker() {
// If we still have no rules - try every second to fetch them
if r.rules == nil {
time.Sleep(1 * time.Second)
} else {
} else if r.interval != 0 {
time.Sleep(r.interval)
} else {
break
}
}
}
Expand Down