Update without downtime

[nikita-sakharin]

I have already asked my question in a letter. Also, I attended at a meeting on Monday and was advised to move that conversation to GitHub. The following text partially repeats the letter.

I would like to research the possibility of updating Ozone cluster from one version to higher one without downtime.

Let I have Ozone cluster configured High Availability, so there are multiple Ozone Managers and multiple Storage Container Managers.

There are two things to discuss:

• Guarantee of backward compatibility between releases. This is important because during such an update, OMs and SCMs of different versions will be present in the cluster. Having these guarantees formulated rolling update seems to be possible.
• Rolling update implies that Leader of consensus is got down at some moment of time. And due to that It is necessary to formulate availability warranties during such an update.

Under rolling update I mean the following:

1. Turn one OM down
2. Cluster establish new consensus
3. Do update on the machine turned down
4. Turn that machine up
5. Wait some amount of time to let new machine warm up
6. Repeat point 1-5 for the remains

I am looking forward to getting links clarifying the current status of that issue and some thoughts/ideas on the implementation.

Having that information got I will try to draft some concept or implementation plan. Having this plan reviewed by community I will try to implement it.

[kerneltime]

Thank you, @nikita-sakharin. We prefer using GitHub discussions for easier access and discovery than email lists or Slack.

Regarding rolling upgrades, the main concern is compatibility between different versions and the implications of running in mixed mode. Currently, we ensure all writes in OM/SCM are flushed, replicated and no new mutating requests are accepted before upgrading.

To enable rolling upgrades, we need to guarantee that any writes accepted during the upgrade are compatible with older versions to allow for downgrade and correct processing of requests across all nodes. We all need to correctly share version information for Datanodes to the client such that the client does not invoke a newer API against a Datanode that is running an older version.

There is a lot more to enabling rolling upgrade in addition to what I laid out but take a look at the current upgrade framework

Ref:
https://github.com/apache/ozone/blob/08b1e3f7463912a67c26e1acf4319b1756e7c03c/hadoop-hdds/docs/content/design/upgrade-dev-primer.md
https://github.com/apache/ozone/blob/08b1e3f7463912a67c26e1acf4319b1756e7c03c/hadoop-hdds/docs/content/feature/Nonrolling-Upgrade.md

cc @errose28 @fapifta

[fapifta]

Hi @nikita-sakharin, first of all, let me thank you for your interest and continued efforts to get this figured out.

You may look at the original JIRA for non-rolling upgrade under HDDS-3698, and the spillover things under HDDS-5444 to get a grasp on what problems we needed to think about and solve for the non-rolling upgrades to go smooth.
Once you go through it, I suggest you to pay attention to the details of the pre-finalization stage, and why we need a step to prepare to an upgrade during which we stop writes to happen.

With rolling upgrades, you need to think of an awful lot of things... Request/Response payload compatibility is somewhat solved at least for Ozone Manager under HDDS-6390, however that is a half baked solution the same things are applied in the request processing code of SCM and maybe the DN also, but without the proposed annotations and such.

The biggest problem is how we handle transactions in Ratis as far as I remember, and afaik it did not changed since then. In Ratis we have the statemachine that applies the transactions in order to the current leader, and the transaction data is sent to the other OMs and they also apply the transaction to their state separately. Once the majority applied the transaction, then the transaction is committed and acked back to the client. Now if we have two different versions of the code, then we need to solve that the transactions are processed identically in two OMs or SCMs and are applied exactly the same way to the DB from the two different codebase.
This is a real challange, as it involves not just the request processing code, but all the related code that is being called during request processing. With that it is a really complex problem to ensure that the new version of the code processes the transactions the same way as the old version of the code, while the old version is running on other OMs. We did not come up with a good solution to this problem so far.

There are a few other problems, like Recon, as Recon gets the OM DB changes via an API, and it has to be able to process all the changes that happen within OM, so if the DB is changing you need to guarantee that Recon is also upgraded before you finalize the upgrade in OMs. Similar problem occurs with heartbeats, as Datanodes send the heartbeat to the Recon server also, so compatibility needs to be taken care of there also before turning on the new features with finalizing the upgrade.

I am not sure, but maybe @szetszwo can add some thoughts on Ratis version differences and whether you need to consider that also, as Ratis is an embedded raft implementation, but it has its own protocols, which might need some care in case of different versions running within a raft ring.

I am certain that I forget to mention or ignorant about a few things, so I hope @errose28 will have the time to also correct and complement this list. At the time we worked on non-rolling upgrades, we had some discussions on problems with rolling upgrade but we did not go too deep to figure out a feasible solution for the rolling upgrades back at that time.

I hope I did not take away your ambitions with all the things listed above, I am sure this is a huge undertake, and I hope we can give you enough support to help you figure out a way and implement it.

[errose28]

Hi @nikita-sakharin, I can add to the explanations provided here already.

As you've probably gathered from the above responses, rolling upgrades will be large cross-cutting feature in Ozone touching every component in a different way. While it is on the roadmap to eventually be supported, design or code has not been started in the community and there is currently no timeline for when work will start and how long that will take. As an open source project we are open to any ideas or designs you would like to share, but I recommend starting with some smaller Ozone tasks to get familiar with the system before diving into a challenging task like this.

That said, I can fill you in on the current state of upgrades, downgrades, and compatibility within Ozone so you can start exploring this.

Currently Supported by Ozone

I did a presentation on this back in 2022. Unfortunately it was not recorded but the slides are here if you want to reference them. The current state of upgrades and cross compatibility support is still the same:

• Non-rolling upgrades from any version since 1.0.0 (GA release) until present.
• Non-rolling downgrades back to the previous version before the cluster has been finalized (see docs for details)
• Full client/server cross compatibility between mixed client and server versions from 1.0.0 (GA release) until present.

Current Testing

• Upgrades and downgrades between versions are tested in Docker with our acceptance test framework.
• Client/server cross compatibility is also tested in Docker.

If you would like to try these tests out locally and are having difficulty running them I can provide more details to help you get set up. Docker images allow us to test the current version of the code against previous releases by pulling them from Docker Hub exactly as they were released.

Current Challenges

I would summarize the following as the biggest challenges for rolling upgrades in Ozone. Ideally I would say that only "request versioning" would be the issue that needs to be solved, but due to architecture decisions that were made early on in the system there are other issues as well.

Ratis StateMachine#applyTransaction Handling

Pifta mentioned this above, and let me elaborate a bit more. When a system reaches consensus on an operation, there are two main choices for what to reach consensus on:

1. Consensus on a set of mutations to apply to the state machine
   1. The leader receives the request.
   2. The leader translates it into a set of mutations like ADD bucket TO bucketTable
   3. The leader submits the mutations as one transaction for consensus
   4. The quorum agrees that they will all apply ADD bucket TO bucketTable
   5. The quorum applies the same mutations as a transaction to their database.
2. Consensus on the operation to perform
   1. The leader receives the request and translates it to a named operation like createBucket
   2. The leader submits request for consensus
   3. The quorum agrees that they will all apply createBucket
   4. Each node individually converts createBucket into a set of database mutations like ADD bucket TO bucketTable
   5. Each node individually modifies their database with the mutations they computed themselves.

Ideally consensus in Ozone would have been implemented with option 1. This is the "textbook" way to do consensus from what I've seen. Unfortunately it was implmeneted using option 2 early on in the design and it is now deeply ingrained in the system. This causes problems that option 1 does not have:

• It is only consistent if all nodes are running the same version of the code
  • Say in new version of Ozone, we add a field to the bucket metadata called foo.
  • In a rolling upgrade scenario, nodes in the newer code will create the bucket with foo, and nodes in the older version will create the bucket without foo since their code to convert from an operation to mutations is in the old version.
  • Option 1 does not have this problem because only the leader determines the structure of the objects to commit.
    • The objects (or values stored in the DB key/value store) would then be pushed through consensus as binary eliminating the possibility of different node's protobuf versions messing with the fields after consensus.
• If there is a bug in converting the operation to DB mutations, it will crash all nodes in the ring and cause unavailability.
  • This is because the request has already been agreed on before the conversion happens, so Raft requires the quorum to successfully apply this request before any more can be processed.
    • Once quorum is reached, it is difficult to remove the request and the system will be stuck in a crash loop.
  • Conversely, if such a bug happens during conversion in option 1, the request can simply be dropped and an error returned to the client, because no consensus has been reached yet. Even if the request does somehow crash the leader, the followers will still be up because they have not seen it yet.
  • Given that the Ozone Manager supports about 100 different types of requests that each need their own translation to DB operations, bugs have a higher probablility of happening in this phase because it is unique to each request.
    • The other phases that involve reaching consensus on an item and modifying the DB are generic to all requests and therefore much more stable.
  • This problem is not exactly related to rolling upgrades but it is another reason option 1 is favorable.

While option 1 ensures the state machines are identical throughout the rolling upgrade, it still does not fully solve the problem, because the system must behave as if it is in the lowest version until all members of the quorum are upgraded. Otherwise leader changes between old and new nodes may cause non-deterministic behavior if a client tries to read or write new fields like foo from the example above. Of course you can devise some request versioning scheme to accomodate this, but in practice it is hard to enforce at the scale of Apache open source development without some CI automation to prevent unversioned changed from being committed and released.

Write Pipeline Membership

This is another issue that is not intrinsic to rolling upgrades in general but specific to Ozone due to an early architecture decision. Ozone uses Ratis pipelines on datanodes to handle 3 way replicated data. It does not change the membership of these pipelines after creation. This differs from HDFS which uses "chained" replication with ad-hoc/stateless pipelines of datanodes and no consensus on the data path. Consensus is only required for metadata in HDFS and temporary datanode unavailability is handled by retrying on a different set of datanodes.

In Ozone, this means when there is an issue on a datanode (or it is restarted to upgrade), the pipeline must be closed. There is no guarantee that the pipeline will still exist when the node comes back online and it will have any way to rejoin and get the writes that it missed. These pipelines interleave with each other in a setup that we call "multi-raft", which means that restarting a single node requires the destruction of all pipelines it is involved in. And of course the Ratis/Raft write pipelines are implemented using consensus as described in "option 2" above.

All this adds up to a very complicated problem, one that even if solved theoretically would likely break down in real world use cases due to the complexity of the implementation and amount of corner cases. There is no technical reason that chained replication would not work in Ozone the same way it works in HDFS. This would also simplify write path considerations for rolling upgrade to about the same magnitude of difficulty as HDFS rolling upgrade. I've said before that I think chained replication in Ozone is a pre-requisite to rolling upgrade support, but that is just my theoretical take on the problem and not a requirement.

Network Protocol/Request Versioning Among Servers

Now with the Ozone specific issues out of the way, we get to the fundamental aspect of that sets rolling upgrades apart from other issues: versioning within the Ozone servers.

When upgrading a stateful client/server system there are 3 points of contact where compatibility issues could arise:

• Server and its disk state from other versions
  • Ozone handles this with "layout features" to version how data is written to disk.
  • This allows Ozone to support non-rolling upgrades and downgrades.
• Client and server in different versions
  • Ozone handles this with "component versions" where the client and server inform each other about what version they are running in, and newer component makes adjustments to make communication work with the older one.
  • This allows Ozone to support client and server cross-compatibility in a disagregated compute/storage environment where each layer may be upgraded separately.
• Server and server in different versions
  • Ozone does not have handling for this right now.
  • It could be implemented with some extension of the "component version" framework, or using a new solution.

In summary, rolling upgrade is a fundamentally challenging feature for any system, but Ozone has extra roadblocks due to early design decisions that I would consider "tech debt". If I were drafting a rolling upgrade design I would want the tech debt resolved first, but that is just my personal opinion/approach. Maybe other solutions exist that could work around these issues effectively : )

[nikita-sakharin]

@kerneltime, @errose28, @fapifta
Many thanks to you for details provided!

It has taken for me some amount of time to get acquainted with the information you gave. I examined all Issues and presentation you mentioned. For now I have got some understanding of the problem. And as far as I have understood Update without downtime is pretty huge feature with respect to code changes to be made.

As I have already mentioned in this Monday meeting I am looking forward to splitting that feature for sub tasks.

Could I kindly ask you to point on some sub-issues/sub-tasks this feature depends on?

If it possible I need something small and simple to start with and implement something tiny to step forward.

[errose28]

Hi, thanks for joining the community sync today. Sorry I'm still refreshing my own memory as it has been a few years since I looked in this area myself. I went through and filled in some details on the task mentioned in #6984. I will need to go back to the issues mentioned in #6980 and HDDS-5525 then I can provide more context on those soon if those are areas you would like to investigate.

[fapifta]

Unfortunately I also don't have any small things that is easy to start with... Probably the best way is to collaborate with others on some area of the code, and work on changes/features that are relevant, for that I have a few ideas to share related to the problem areas @errose28 summarized.

As a starter I think you can join to @sumitagrawl on resolving the problems in OM's StateMachine, afaik he has some ideas already on what and how to do about it.

An other area is around request validators, as that feature is worked out for Ozone Manager and client communication, but I believe it should be extended to OM<->SCM and other communication channels as well, this would help to solve the protocol compatibility problems. @swamirishi has a draft PR (#6932) that adds some fundamental changes to how we handle these, you may take a look into the related discussions in linked PRs.

The ratis write pipeline related problems are somewhat related to a new feature called Ratis streaming, it is already integrated in the write path, @szetszwo may have additional information on how it works, and what might be done to get closer to more dynamic pipelines if that is possible.