Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prevents watches from being orphaned when KVS blocking queries loop. #1632

Merged
merged 1 commit into from
Jan 21, 2016

Conversation

slackpad
Copy link
Contributor

This fixes #1626.

The basic problem is that we were returning the underlying NotifyGroup object here:

https://github.com/hashicorp/consul/blob/v0.6.0/consul/state/watch.go#L100-L112

This worked fine the first time through, but when a prefix was notified, this NotifyGroup would get deleted:

https://github.com/hashicorp/consul/blob/v0.6.0/consul/state/watch.go#L140-L143

So in a blocking RPC, once you looped around because the watch was notified, you'd end up calling Wait() on a NotifyGroup that was no longer part of the watch manager:

https://github.com/hashicorp/consul/blob/v0.6.0/consul/rpc.go#L342

The solution is to never expose the underlying NotifyGroup to the blocking RPC code. I created a wrapper object that re-registers the wait channel with whatever NotifyGorup is in the tree, or it creates one.

This bug was pretty subtle because you had to have these conditions:

  1. Make a blocking query on some key "A".
  2. Update some key "B" whose key is a prefix of "A".

This is probably not super common, but this would have been difficult to detect. The query timeout would eventually un-block the first query, but it would report stale results from the first time it looked at the state store.

/cc @armon or @ryanuber

@armon
Copy link
Member

armon commented Jan 20, 2016

LGTM

slackpad added a commit that referenced this pull request Jan 21, 2016
Prevents watches from being orphaned when KVS blocking queries loop.
@slackpad slackpad merged commit bfb1b7d into master Jan 21, 2016
@slackpad slackpad deleted the b-orphaned-watches branch January 21, 2016 00:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

KV watches don't fire properly when prefix keys are modified
2 participants