Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

*: add session var 'tidb_ddl_reorg_worker_cnt' to control ddl reorg workers count #6441

Merged
merged 17 commits into from
Jun 27, 2018
Merged

*: add session var 'tidb_ddl_reorg_worker_cnt' to control ddl reorg workers count #6441

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
a2a95e6
*: add session var 'tidb_ddl_reorg_worker_cnt' to control ddl reorg w…
winkyao May 2, 2018
bd3e576
Merge branch 'master' into add_reorg_worker_cnt_variable
winkyao May 2, 2018
9944329
Merge branch 'master' into add_reorg_worker_cnt_variable
winkyao May 2, 2018
9a66045
move DDLReorgWorkerCount to vars.Concurrency
winkyao May 2, 2018
8ca59aa
Merge branch 'add_reorg_worker_cnt_variable' of github.com:winkyao/ti…
winkyao May 2, 2018
56928e3
make DDLReorgWorkerCounter be global variable
winkyao May 3, 2018
6265d6e
update comment
winkyao May 3, 2018
230285b
fix check
winkyao May 4, 2018
add416d
Merge remote-tracking branch 'upstream/master' into add_reorg_worker_…
winkyao May 4, 2018
0440006
Merge remote-tracking branch 'upstream/master' into add_reorg_worker_…
winkyao Jun 11, 2018
64e6cbc
Merge branch 'master' into add_reorg_worker_cnt_variable
winkyao Jun 11, 2018
81caeca
add upper limit of worker counter
winkyao Jun 11, 2018
31c0162
add test
winkyao Jun 11, 2018
680c202
Merge remote-tracking branch 'upstream/master' into add_reorg_worker_…
winkyao Jun 15, 2018
427acdf
Merge branch 'master' into add_reorg_worker_cnt_variable
winkyao Jun 15, 2018
195c46c
Merge branch 'master' into add_reorg_worker_cnt_variable
ciscoxll Jun 27, 2018
314e00c
Merge branch 'master' into add_reorg_worker_cnt_variable
ciscoxll Jun 27, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions ddl/index.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ import (
"github.com/pingcap/tidb/model"
"github.com/pingcap/tidb/mysql"
"github.com/pingcap/tidb/sessionctx"
"github.com/pingcap/tidb/sessionctx/variable"
"github.com/pingcap/tidb/store/tikv"
"github.com/pingcap/tidb/table"
"github.com/pingcap/tidb/table/tables"
Expand Down Expand Up @@ -404,7 +405,6 @@ func (d *ddl) onDropIndex(t *meta.Meta, job *model.Job) (ver int64, _ error) {

const (
defaultTaskHandleCnt = 128
defaultWorkers = 16
)

// indexRecord is the record information of an index.
Expand Down Expand Up @@ -806,10 +806,10 @@ func (d *ddl) addTableIndex(t table.Table, indexInfo *model.IndexInfo, reorgInfo
log.Infof("[ddl-reorg] addTableIndex, job:%s, reorgInfo:%#v", job, reorgInfo)
colFieldMap := makeupIndexColFieldMap(t, indexInfo)

// TODO: make workerCnt can be modified by system variable.
workerCnt := defaultWorkers
// variable.DDLReorgWorkerCounter can be modified by system variable "tidb_ddl_reorg_worker_cnt".
workerCnt := variable.GetDDLReorgWorkerCounter()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we support updating worker counter when the workers are running?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will be fixed later, not this PR.

workers := make([]*addIndexWorker, workerCnt)
for i := 0; i < workerCnt; i++ {
for i := 0; i < int(workerCnt); i++ {
sessCtx := d.newContext()
workers[i] = newAddIndexWorker(sessCtx, d, i, t, indexInfo, colFieldMap)
go workers[i].run()
Expand Down
29 changes: 29 additions & 0 deletions executor/ddl_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ import (
"github.com/pingcap/tidb/model"
"github.com/pingcap/tidb/mysql"
"github.com/pingcap/tidb/plan"
"github.com/pingcap/tidb/sessionctx/variable"
"github.com/pingcap/tidb/table"
"github.com/pingcap/tidb/types"
"github.com/pingcap/tidb/util/chunk"
Expand Down Expand Up @@ -373,3 +374,31 @@ func (s *testSuite) TestShardRowIDBits(c *C) {
c.Assert(count, Equals, 100)
c.Assert(hasShardedID, IsTrue)
}

func (s *testSuite) TestSetDDLReorgWorkerCnt(c *C) {
tk := testkit.NewTestKit(c, s.store)
tk.MustExec("use test")
c.Assert(variable.GetDDLReorgWorkerCounter(), Equals, int32(variable.DefTiDBDDLReorgWorkerCount))
tk.MustExec("set tidb_ddl_reorg_worker_cnt = 1")
c.Assert(variable.GetDDLReorgWorkerCounter(), Equals, int32(1))
tk.MustExec("set tidb_ddl_reorg_worker_cnt = 100")
c.Assert(variable.GetDDLReorgWorkerCounter(), Equals, int32(100))
tk.MustExec("set tidb_ddl_reorg_worker_cnt = invalid_val")
c.Assert(variable.GetDDLReorgWorkerCounter(), Equals, int32(variable.DefTiDBDDLReorgWorkerCount))
tk.MustExec("set tidb_ddl_reorg_worker_cnt = 100")
c.Assert(variable.GetDDLReorgWorkerCounter(), Equals, int32(100))
tk.MustExec("set tidb_ddl_reorg_worker_cnt = -1")
c.Assert(variable.GetDDLReorgWorkerCounter(), Equals, int32(variable.DefTiDBDDLReorgWorkerCount))

res := tk.MustQuery("select @@tidb_ddl_reorg_worker_cnt")
res.Check(testkit.Rows("-1"))
tk.MustExec("set tidb_ddl_reorg_worker_cnt = 100")
res = tk.MustQuery("select @@tidb_ddl_reorg_worker_cnt")
res.Check(testkit.Rows("100"))

res = tk.MustQuery("select @@global.tidb_ddl_reorg_worker_cnt")
res.Check(testkit.Rows(fmt.Sprintf("%v", variable.DefTiDBDDLReorgWorkerCount)))
tk.MustExec("set @@global.tidb_ddl_reorg_worker_cnt = 100")
res = tk.MustQuery("select @@global.tidb_ddl_reorg_worker_cnt")
res.Check(testkit.Rows("100"))
}
1 change: 1 addition & 0 deletions session/session.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1273,6 +1273,7 @@ const loadCommonGlobalVarsSQL = "select HIGH_PRIORITY * from mysql.global_variab
variable.TiDBIndexSerialScanConcurrency + quoteCommaQuote +
variable.TiDBHashJoinConcurrency + quoteCommaQuote +
variable.TiDBBackoffLockFast + quoteCommaQuote +
variable.TiDBDDLReorgWorkerCount + quoteCommaQuote +
variable.TiDBDistSQLScanConcurrency + quoteCommaQuote +
variable.TiDBRetryLimit + "')"

Expand Down
3 changes: 3 additions & 0 deletions sessionctx/variable/session.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -531,6 +531,9 @@ func (s *SessionVars) SetSystemVar(name string, val string) error {
s.EnableStreaming = TiDBOptOn(val)
case TiDBOptimizerSelectivityLevel:
s.OptimizerSelectivityLevel = tidbOptPositiveInt32(val, DefTiDBOptimizerSelectivityLevel)
case TiDBDDLReorgWorkerCount:
workerCnt := tidbOptPositiveInt32(val, DefTiDBDDLReorgWorkerCount)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to have an upper limit?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

SetDDLReorgWorkerCounter(int32(workerCnt))
}
s.systems[name] = val
return nil
Expand Down
1 change: 1 addition & 0 deletions sessionctx/variable/sysvar.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -642,6 +642,7 @@ var defaultSysVars = []*SysVar{
/* The following variable is defined as session scope but is actually server scope. */
{ScopeSession, TiDBGeneralLog, strconv.Itoa(DefTiDBGeneralLog)},
{ScopeSession, TiDBConfig, ""},
{ScopeGlobal | ScopeSession, TiDBDDLReorgWorkerCount, strconv.Itoa(DefTiDBDDLReorgWorkerCount)},
}

// SynonymsSysVariables is synonyms of system variables.
Expand Down
7 changes: 6 additions & 1 deletion sessionctx/variable/tidb_vars.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -156,6 +156,9 @@ const (

// tidb_backoff_lock_fast is used for tikv backoff base time in milliseconds.
TiDBBackoffLockFast = "tidb_backoff_lock_fast"

// tidb_ddl_reorg_worker_cnt defines the count of ddl reorg workers.
TiDBDDLReorgWorkerCount = "tidb_ddl_reorg_worker_cnt"
)

// Default TiDB system variable values.
Expand Down Expand Up @@ -189,9 +192,11 @@ const (
DefTiDBRetryLimit = 10
DefTiDBHashJoinConcurrency = 5
DefTiDBOptimizerSelectivityLevel = 0
DefTiDBDDLReorgWorkerCount = 16
)

// Process global variables.
var (
ProcessGeneralLog uint32
ProcessGeneralLog uint32
DDLReorgWorkerCounter int32 = DefTiDBDDLReorgWorkerCount
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/DDLReorgWorkerCounter/ddlReorgWorkerCounter ?
We use GetDDLReorgWorkerCounter and SetDDLReorgWorkerCounter to change its value. So I think we'd better use the lowercase variable.

)
10 changes: 10 additions & 0 deletions sessionctx/variable/varsutil.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,16 @@ import (
"github.com/pingcap/tidb/types"
)

// SetDDLReorgWorkerCounter sets DDLReorgWorkerCounter count.
func SetDDLReorgWorkerCounter(cnt int32) {
atomic.StoreInt32(&DDLReorgWorkerCounter, cnt)
}

// GetDDLReorgWorkerCounter gets DDLReorgWorkerCounter.
func GetDDLReorgWorkerCounter() int32 {
return atomic.LoadInt32(&DDLReorgWorkerCounter)
}

// GetSessionSystemVar gets a system variable.
// If it is a session only variable, use the default value defined in code.
// Returns error if there is no such variable.
Expand Down
7 changes: 7 additions & 0 deletions sessionctx/variable/varsutil_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -217,6 +217,13 @@ func (s *testVarsutilSuite) TestVarsutil(c *C) {
SetSessionSystemVar(v, TiDBOptimizerSelectivityLevel, types.NewIntDatum(1))
c.Assert(v.OptimizerSelectivityLevel, Equals, 1)

c.Assert(GetDDLReorgWorkerCounter(), Equals, int32(DefTiDBDDLReorgWorkerCount))
SetSessionSystemVar(v, TiDBDDLReorgWorkerCount, types.NewIntDatum(1))
c.Assert(GetDDLReorgWorkerCounter(), Equals, int32(1))

SetSessionSystemVar(v, TiDBDDLReorgWorkerCount, types.NewIntDatum(-1))
c.Assert(GetDDLReorgWorkerCounter(), Equals, int32(DefTiDBDDLReorgWorkerCount))

err = SetSessionSystemVar(v, TiDBRetryLimit, types.NewStringDatum("3"))
c.Assert(err, IsNil)
val, err = GetSessionSystemVar(v, TiDBRetryLimit)
Expand Down