Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

server: support resigning ddl owner, use http method ddl/owner/resign #7649

Merged
merged 16 commits into from
Sep 11, 2018
Merged

server: support resigning ddl owner, use http method ddl/owner/resign #7649

merged 16 commits into from
Sep 11, 2018

Conversation

winkyao
Copy link
Contributor

@winkyao winkyao commented Sep 9, 2018

What problem does this PR solve?

Support resigning DDL owner, let tidb start a new DDL owner election.

What is changed and how it works?

Add a new HTTP method ddl/owner/resign and this method will call concurrency.Election.Resign. concurrency.Election.Resign will delete the leader key /tidb/ddl/fg/owner/ on the etcd. And the owner of TiDB will exit watchOwner function, and TiDB cluster will start a new election by elec.Campaign.

Check List

Tests

  • Manual test.

use ansible to start a new cluster with two tidb instances. Run curl http://{TiDBIP}:10080/ddl/owner/resign in non-owner will response This node is not a ddl owner, can't be resigned., and the owner will return "success!".

And use select tidb_is_ddl_owner, we can find out the owner of the tidb is transferred to the other instance.

Code changes

  • Has exported function/method change
  • Has interface methods change

Side effects

  • Possible performance regression
  • Increased code complexity

Related changes

  • Need to update the documentation
  • Need to be included in the release note

defer s.Close()
}

dom := domain.GetDomain(s.(sessionctx.Context))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can use session.GetDomain().

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the difference? Use session.GetDomain() will look likes:

dom, err := session.GetDomain(s.GetStore())
	if err != nil {
		return errors.Trace(err)
	}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The difference is that we do not need to create a session.

1. Resign the ddl owner, let tidb start a new ddl owner election.

```shell
curl http://{TiDBIP}:10080/ddl/owner/resign
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be a POST request.

@@ -334,6 +353,11 @@ type ddlHistoryJobHandler struct {
*tikvHandlerTool
}

// ddlResignOwnerHandler is the handler for resigning ddl owner.
type ddlResignOwnerHandler struct {
*tikvHandlerTool
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not operated on TiKV.
How about we implement the method on ddlResignOwnerHandler directly?

@winkyao
Copy link
Contributor Author

winkyao commented Sep 10, 2018

@coocood @lamxTyler PTAL

@coocood
Copy link
Member

coocood commented Sep 10, 2018

Can we just use elec != nil to determine it is the owner?

@winkyao
Copy link
Contributor Author

winkyao commented Sep 11, 2018

@coocood use !isOwner() is more readable.

@winkyao
Copy link
Contributor Author

winkyao commented Sep 11, 2018

@coocood fine, if using elec != nil to determine, we can remove the mutex.

@winkyao
Copy link
Contributor Author

winkyao commented Sep 11, 2018

@coocood @lamxTyler Done, PTAL

owner/manager.go Outdated

func (m *ownerManager) retireOwner() {
m.SetOwner(false)
atomic.StorePointer(&m.elec, nil)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we only use the elec to determine if it is the owner? Then we do not need to maintain the owner anymore.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are some test cases use SetOwner(false) and just use elec to determin if it is the owner will not easy to mock in tests.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we change SetOwner(isOwner bool) to SetOwner(elec unsafe.Pointer)?
In the mockManager, we can use any not nil pointer to represent elec.

Copy link
Contributor

@alivxxx alivxxx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

return errors.Trace(err)
}

log.Warnf("%s Resign ddl owner success!", m.logPrefix)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we call RetireOwner here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, this is not the work of ResignOwner.

@coocood
Copy link
Member

coocood commented Sep 11, 2018

LGTM

@coocood coocood added status/LGT2 Indicates that a PR has LGTM 2. component/DDL-need-LGT3 labels Sep 11, 2018
owner/manager.go Outdated
@@ -45,12 +46,14 @@ type Manager interface {
ID() string
// IsOwner returns whether the ownerManager is the owner.
IsOwner() bool
// SetOwner sets whether the ownerManager is the owner.
SetOwner(isOwner bool)
// RetireOwner make the manager to be a not owner. It's exported for testing
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

end with .

owner/mock.go Outdated
atomic.StoreInt32(&m.owner, 1)
}

// setOwner implements Manager.RetireOwner interface.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/setOwner/RetireOwner

}

ownerMgr := dom.DDL().OwnerManager()
err = ownerMgr.ResignOwner(context.Background())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to add a timeout?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@winkyao Please answer this question.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, PTAL

@winkyao
Copy link
Contributor Author

winkyao commented Sep 11, 2018

@zimulala PTAL

Copy link
Contributor

@zimulala zimulala left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zimulala
Copy link
Contributor

/run-all-tests

@winkyao winkyao added status/all tests passed release-note Denotes a PR that will be considered when it comes time to generate release notes. labels Sep 11, 2018
@winkyao winkyao merged commit f497444 into pingcap:master Sep 11, 2018
@winkyao winkyao deleted the evict_owner_by_http branch September 11, 2018 07:18
@you06 you06 added the sig/sql-infra SIG: SQL Infra label Mar 4, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/sql-infra SIG: SQL Infra status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants