New directory layout does not play nicely with upgrades/downgrades #52414
Labels
blocker
:Distributed/Cluster Coordination
Cluster formation and cluster state publication, including cluster membership and fault detection.
Team:Distributed
Meta label for distributed team
v8.0.0-beta1
Comments
DaveCTurner
added
the
:Distributed/Cluster Coordination
|
Pinging @elastic/es-distributed (:Distributed/Cluster Coordination) |
DaveCTurner
changed the title
DaveCTurner
added a commit
to DaveCTurner/elasticsearch
that referenced
this issue
Oct 2, 2021
Users sometimes attempt to downgrade a node in place, but downgrades are totally untested and unsupported and generally don't work. We protect against this by recording the node version in the metadata and refusing to start if we encounter metadata written by a future version. However in 8.0 (elastic#42489) we changed the directory layout so that a 7.x node won't find the upgraded metadata, or indeed any other data, and will proceed as if it's a fresh node. That's almost certainly not what the user wants, so with this commit we create a file at `${path.data}/nodes` at each startup, preventing an older node from starting. Closes elastic#52414
DaveCTurner
added a commit
that referenced
this issue
Oct 4, 2021
Users sometimes attempt to downgrade a node in place, but downgrades are totally untested and unsupported and generally don't work. We protect against this by recording the node version in the metadata and refusing to start if we encounter metadata written by a future version. However in 8.0 (#42489) we changed the directory layout so that a 7.x node won't find the upgraded metadata, or indeed any other data, and will proceed as if it's a fresh node. That's almost certainly not what the user wants, so with this commit we create a file at `${path.data}/nodes` at each startup, preventing an older node from starting. Closes #52414
|
Merged the fix too soon, I am reverting the fix and reopening this. |
DaveCTurner
added a commit
to DaveCTurner/elasticsearch
that referenced
this issue
Oct 4, 2021
Users sometimes attempt to downgrade a node in place, but downgrades are totally untested and unsupported and generally don't work. We protect against this by recording the node version in the metadata and refusing to start if we encounter metadata written by a future version. However in 8.0 (elastic#42489) we changed the directory layout so that a 7.x node won't find the upgraded metadata, or indeed any other data, and will proceed as if it's a fresh node. That's almost certainly not what the user wants, so with this commit we create a file at `${path.data}/nodes` at each startup, preventing an older node from starting. Closes elastic#52414
elasticsearchmachine
pushed a commit
that referenced
this issue
Oct 14, 2021
Users sometimes attempt to downgrade a node in place, but downgrades are totally untested and unsupported and generally don't work. We protect against this by recording the node version in the metadata and refusing to start if we encounter metadata written by a future version. However in 8.0 (#42489) we changed the directory layout so that a 7.x node won't find the upgraded metadata, or indeed any other data, and will proceed as if it's a fresh node. That's almost certainly not what the user wants, so with this commit we create a file at `${path.data}/nodes` at each startup, preventing an older node from starting. Closes #52414
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
In #42489 we introduced a new layout for the data directory, but this does not today play nicely with some corner cases of upgrades and downgrades.
Firstly, as noted in #44624 (comment), we reorganise the data directory before checking that we can load and upgrade the cluster metadata; if it turns out that the cluster metadata is unreadable and needs fixing in 7.x then the user is stuck since a downgrade to 7.x will not find the cluster metadata since was already moved. I think we should not reorganise the data directory until we are sure that the upgraded node is viable.
Secondly, if a user does try and downgrade an 8.x node to 7.x then the node will successfully start up but will appear to be completely empty since the upgrade to 8.x already reorganised the data directory. I think we should protect against this by, for instance, leaving a copy of the node metadata in place in
nodes/0/_stateto indicate that this node has been upgraded.The text was updated successfully, but these errors were encountered: