registry/rpc: use simpleBalancer instead of ClientConn.State() #1673

dongsupark · 2016-08-26T14:08:19Z

As ClientConn.State() of gRPC has disappeared, we need to avoid using ClientConn.State(). Instead we should make use of gRPC rebalancer mechanism, just like etcdv3 is doing. To do that, introduce simpleBalancer, as a minimum structure to be used for grpc.Balancer.

Note that this change could be possibly invasive to the current behavior of fleet's gRPC. A careful review is needed.

/cc @hectorj2f
Fixes #1672

hectorj2f · 2016-09-01T11:03:33Z

registry/rpc/rpcregistry.go

-		log.Infof("Status of rpc service: %d, connection state: %s", status, connState)
-		return status == pb.HealthCheckResponse_SERVING && err == nil
-	}
+	// NOTE: checking for readyc doesn't work as expected.


Is this part WIP ?

Yeah right. I need to have a deeper look into this one. Unfortunately checking the readyc doesn't work as expected. If you have a better idea, please let me know. TBH I'm just a beginner of the new API of gRPC v1.0 or etcdv3.

hectorj2f · 2016-09-01T11:08:22Z

@dongsupark could add a bigger description about what the balancer does ?
Also, this PR is a bit big. What do you think about a PR for the vendoring packages and another with the logic ? That would make the reviewing much easier and faster.

dongsupark · 2016-09-01T11:26:36Z

@hectorj2f Sure, I will split it into multiple PRs, and add more descriptions.

dongsupark · 2016-09-08T15:13:56Z

Force-pushed it by changing the following:

Fixed bugs in IsRegistryReady(). This function still needs to check r.Status() like before. It also needs to check if r.registryConn or r.balancer is nil.
Fixed a bug in rpcAcquireLeadership() by inserting reg.IsRegistryReady() to the condition for checking existing lease. This bug resulted in another failure in TestDynamicClusterNewMemberUnitMigration(), which got me locked up in a debugging nightmare. (Now it's solved)
Simplified balancer.go, removing unnecessary parts like getHost().
Added high-level descriptions on simpleBalancer.

And I created 2 PRs, #1676 (only vendor updates) and #1677 (relevant code), for easier reviews.

dongsupark · 2016-09-25T16:36:58Z

FYI.
Running benchmarks with this PR included, I can observe performance regressions. Nomi benchmarks starting thousands of units over a multi-node cluster, unit starts take much longer time than the current master branch.
I'm not sure why. (gRPC balancer issue? Bug in my patches?)
Until we figured out its reason, let's not merge this PR.

Update vendor/google.golang.org/grpc to the most recent master, as of 2016-08-26.

Now that gRPC as well as other dependencies were updated, we need to update fleet.pb.go too.

As ClientConn.State() of gRPC has disappeared, we need to avoid using ClientConn.State(). Instead we should make use of gRPC rebalancer mechanism, just like etcdv3 is doing. To do that, introduce simpleBalancer, as a minimum structure to be used for grpc.Balancer.

dongsupark · 2016-11-23T15:16:29Z

Running benchmarks with this PR included, I can observe performance regressions. Nomi benchmarks starting thousands of units over a multi-node cluster, unit starts take much longer time than the current master branch.

I finally got some time to do nomi benchmarks again. This time, I cannot see the regression any more.

When tested with etcd v3, results of master branch and this branch are nearly the same.
With etcd v2, this branch is even faster than master.
That's actually an expected result, as it's already known that performance got improved with gRPC v1.0, compared to the previous versions.

What I saw in September was probably just because of mistake in testing.
Tomorrow I'll run again benchmarks several times, and if it's still green, I'll merge this PR.

dongsupark · 2016-11-24T14:31:08Z

I can confirm that its performance is all right.
With etcd2, unit start time was improved by 16%. With etcd3, it's nearly the same.
Benchmark raw data is available on https://github.com/dongsupark/fleet-benchmarks.
I'll merge this PR. Thanks!

registry/rpc: use simpleBalancer instead of ClientConn.State()

dongsupark added dependency/external kind/friction technology/grpc labels Aug 26, 2016

dongsupark self-assigned this Aug 26, 2016

dongsupark added the needs review label Aug 26, 2016

dongsupark force-pushed the dongsu/grpc-fix-build-clientconn branch 9 times, most recently from af9b893 to 6d3a053 Compare August 26, 2016 17:48

dongsupark force-pushed the master branch 2 times, most recently from bdc94e8 to fa5aa3a Compare August 30, 2016 12:26

hectorj2f reviewed Sep 1, 2016
View reviewed changes

dongsupark added reviewed/needs rework and removed needs review labels Sep 1, 2016

dongsupark force-pushed the dongsu/grpc-fix-build-clientconn branch from 6d3a053 to f5f9223 Compare September 8, 2016 14:58

dongsupark changed the title ~~registry/rpc: fix build errors from ClientConn.State() in gRPC~~ registry/rpc: use simpleBalancer instead of ClientConn.State() Sep 8, 2016

This was referenced Sep 8, 2016

registry/rpc: vendor updates for using simpleBalancer #1676

Closed

registry/rpc: use simpleBalancer instead of ClientConn.State() #1677

Closed

dongsupark added needs review and removed reviewed/needs rework labels Sep 9, 2016

dongsupark mentioned this pull request Sep 12, 2016

api: support cAPI.UnitState() for a single unit #1679

Merged

Dongsu Park added 2 commits November 10, 2016 11:36

vendor: update grpc to master 2016-08-26

a6cff59

Update vendor/google.golang.org/grpc to the most recent master, as of 2016-08-26.

protobuf: regenerate fleet.pb.go

423e390

Now that gRPC as well as other dependencies were updated, we need to update fleet.pb.go too.

dongsupark force-pushed the dongsu/grpc-fix-build-clientconn branch 2 times, most recently from 4ef338e to ecb121a Compare November 10, 2016 11:26

dongsupark force-pushed the master branch 3 times, most recently from 20a3e96 to 3aaa1ab Compare November 10, 2016 15:24

dongsupark merged commit 8ecf7e8 into coreos:master Nov 24, 2016

dongsupark deleted the dongsu/grpc-fix-build-clientconn branch November 24, 2016 14:43

dongsupark pushed a commit that referenced this pull request Nov 24, 2016

Merge pull request #1673 from endocode/dongsu/grpc-fix-build-clientconn

86b14c2

registry/rpc: use simpleBalancer instead of ClientConn.State()

dongsupark pushed a commit that referenced this pull request Nov 24, 2016

Merge pull request #1673 from endocode/dongsu/grpc-fix-build-clientconn

eb6872f

registry/rpc: use simpleBalancer instead of ClientConn.State()

dongsupark pushed a commit that referenced this pull request Nov 24, 2016

Merge pull request #1673 from endocode/dongsu/grpc-fix-build-clientconn

365565e

registry/rpc: use simpleBalancer instead of ClientConn.State()

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

registry/rpc: use simpleBalancer instead of ClientConn.State() #1673

registry/rpc: use simpleBalancer instead of ClientConn.State() #1673

dongsupark commented Aug 26, 2016

hectorj2f Sep 1, 2016 •

edited

Loading

dongsupark Sep 1, 2016

hectorj2f commented Sep 1, 2016

dongsupark commented Sep 1, 2016

dongsupark commented Sep 8, 2016

dongsupark commented Sep 25, 2016

dongsupark commented Nov 23, 2016 •

edited

Loading

dongsupark commented Nov 24, 2016

registry/rpc: use simpleBalancer instead of ClientConn.State() #1673

registry/rpc: use simpleBalancer instead of ClientConn.State() #1673

Conversation

dongsupark commented Aug 26, 2016

hectorj2f Sep 1, 2016 • edited Loading

Choose a reason for hiding this comment

dongsupark Sep 1, 2016

Choose a reason for hiding this comment

hectorj2f commented Sep 1, 2016

dongsupark commented Sep 1, 2016

dongsupark commented Sep 8, 2016

dongsupark commented Sep 25, 2016

dongsupark commented Nov 23, 2016 • edited Loading

dongsupark commented Nov 24, 2016

hectorj2f Sep 1, 2016 •

edited

Loading

dongsupark commented Nov 23, 2016 •

edited

Loading