Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

storage: prevent node from restarting after consistency check fatal #42401

Merged
merged 2 commits into from
Nov 18, 2019
Merged

storage: prevent node from restarting after consistency check fatal #42401

merged 2 commits into from
Nov 18, 2019

Commits on Nov 18, 2019

  1. cli: fix typo 

    Release note: None
    @tbg
    tbg committed Nov 18, 2019
    Configuration menu
    Copy the full SHA
    7dbdd94 View commit details
    Browse the repository at this point in the history

  2. storage: prevent node from restarting after corruption 

    Many deployments auto-restart crashed nodes unconditionally. As a
result, we are often pulled into inconsistency failures relatively late,
which results in a loss of information (in particular if the source of
the problem turns out to be one of the nodes that did not actually
crash); if the node that crashes *is* the one with the problem,
restarting it lets it serve data again (until the next time the
consistency checker nukes it), which is bad as well.

This commit introduces a "death rattle" - if a node is told to terminate
as the result of a consistency check, it will write a marker file that
prevents subsequent restarts of the node with an informative message
alerting the operator that there is a serious problem that needs to be
addressed. We use the same strategy when a replica finds that its
internal invariants are violated.

Fixes cockroachdb#40442.

Release note (general change): nodes that have been terminated as the
result of a failed consistency check now refuse to restart, making it
more likely that the operator notices that there is a persistent issue
in a timely manner.
    @tbg
    tbg committed Nov 18, 2019
    Configuration menu
    Copy the full SHA
    ba6699b View commit details
    Browse the repository at this point in the history