fix conflict between handleReload and antiEntropy critical sections #1235
Conversation
see: hashicorp#1173 hashicorp#1173 Reasoning: somewhere during consul development Pause()/Resume() and PauseSync()/ResumeSync() were added to protect larger changes to agent's localState. A few of the places that it tries to protect are: - (a *Agent) AddService(...) # part of the method - (c *Command) handleReload(...) # almost the whole method - (l *localState) antiEntropy(...)# isPaused() prevents syncChanges() The main problem is, that in the middle of handleReload(...)'s critical section it indirectly (loadServices()) calls AddService(...). AddService() in turn calls Pause() to protect itself against syncChanges(). At the end of AddService() a defered call to Resume() is made. With the current implementation, this releases isPaused() "lock" in the middle of handleReload() allowing antiEntropy to kick in while configuration reload is still in progress. Specifically almost all services and probably all check are unloaded when syncChanges() is allowed to run. This in turn can causes massive service/check de-/re-registration, and since checks are by default registered in the critical state, a majority of services on a node can be marked as failing. It's made worse with automation, often calling `consul reload` in close proximity on many nodes in the cluster. This change basically turns Pause()/Resume() into P()/V() of a garden-variety semaphore. Allowing Pause() to be called multiple times, and releasing isPaused() only after all matching/defered Resumes() are called as well. TODO/NOTE: as with many semaphore implementations, it might be reasonable to panic() if l.paused ever becomes negative.
|
This looks good. I don't think my patches under 19e10e7 are necessary since the lock is already there to protect critical sections. Want to take a crack at preventing multiple reloads as well? |
| } | ||
|
|
||
| // Resume is used to resume state synchronization | ||
| func (l *localState) Resume() { | ||
| atomic.StoreInt32(&l.paused, 0) | ||
| atomic.AddInt32(&l.paused, -1) |
There was a problem hiding this comment.
Since this returns the new value, I think I would put the panic in here to immediately catch any mismatch of these if it underflows.
There was a problem hiding this comment.
Proper check added in the c4537ed.
I'm more concerned about overflows TBH, there is no way to detect them at the moment and one missing defer to .Resume() can block antiEntropy forever.
There was a problem hiding this comment.
I was thinking about this as well. We could enforce an arbitrary upper limit on the semaphore to catch the case where someone is walking it up, which is probably the most likely programming error, but it won't catch an oddball bad path that doesn't get called often.
wuub
changed the title
I'm pretty sure It seems that all the interference was caused by |
fix conflict between handleReload and antiEntropy critical sections
|
Sounds good. Thanks for your fix! |
see: #1173 #1173
Reasoning: somewhere during consul development Pause()/Resume() and
PauseSync()/ResumeSync() were added to protect larger changes to
agent's localState. A few of the places that it tries to protect are:
(a *Agent) AddService(...) # part of the method
(c *Command) handleReload(...) # almost the whole method
(l *localState) antiEntropy(...)# isPaused() prevents syncChanges()
The main problem is, that in the middle of handleReload(...)'s
critical section it indirectly (loadServices()) calls AddService(...).
AddService() in turn calls Pause() to protect itself against
syncChanges(). At the end of AddService() a defered call to Resume() is
made.
With the current implementation, this releases
isPaused() "lock" in the middle of handleReload() allowing antiEntropy
to kick in while configuration reload is still in progress.
Specifically almost all services and probably all check are unloaded
when syncChanges() is allowed to run.
This in turn can causes massive service/check de-/re-registration,
and since checks are by default registered in the critical state,
a majority of services on a node can be marked as failing.
It's made worse with automation, often calling
consul reloadin closeproximity on many nodes in the cluster.
This change basically turns Pause()/Resume() into P()/V() of
a garden-variety semaphore. Allowing Pause() to be called multiple times,
and releasing isPaused() only after all matching/defered Resumes() are
called as well.
TODO/NOTE: as with many semaphore implementations, it might be reasonable
to panic() if l.paused ever becomes negativ