Synchronization issues in Redis-QUIC

[ssinghaldev]

Hey!

We are doing a project where we are changing the cluster bus communication of Redis from TCP to QUIC using msquic (https://redis.io/topics/cluster-tutorial). We are running into some synchronization issues as now Redis has become multi-threaded. We have documented the problem and some solutions (which has their own problems). Looking for some leads/direction on how we could solve it. Any help is greatly appreciated.

Problem and Context:
We are facing synchronization issues. To give some context, normal Redis is single threaded and it uses its own event-loop library to handle incoming and outgoing connections within cluster nodes. As there is only one thread, there was no need for synchronization earlier. With msquic, now there are multiple threads - one for msquic and other for main redis. Both threads act on the same connection link (which is b/w two nodes on a cluster). Both threads either use this link to either send a message (‘read’) and/or delete the link (‘write’) when some logic is met etc. As read/write can happen simultaneously from multiple threads, when we run the code it is getting core dumped as one thread is trying to use this link and the other has deleted it. To synchronize, we thought of a couple of solutions but every one of them has problems.

Possible Solutions and their Problems:

• Have a global mutex lock. Whenever a thread wants to access any connection lock, take this and process. The problem is - msquic callback threads have to take a lock and if some other threads are taking a long time to process, then there could be a problem with the underlying connection as the QUIC protocol runs on the same callback thread and one should not spend much time on it: https://github.com/microsoft/msquic/blob/main/docs/API.md#execution-mode

• Have a global read-write lock. As a result, multiple readers can enter, thus preventing the above problem to some extent. But the writer thread may get starved as there will always be some reader accessing the lock (coz some msquic callback from other connections will be there and processing the packet received )

• Have a fine-grained version of the lock - where there are multiple locks - one for each connection link. It solves the above problem of the coarse grained variant above but it has its own struggles.

  • How does a thread find what is the lock for a particular link it is accessing. We can’t store that information in link structure (present in Redis) as we have to “access” as then it will defeat the whole purpose - one will be accessing it and other will be deleting at the same time.
  • Store in some other data structure that the lock to use for a particular link is this lock. We thought of many variants of it. But in all cases, we need to have synchronization for this structure itself to get the info (as this will be changed once we add/remove the links). Also there is no concept of some form of “link id” which could help us in mapping (there is another “node” structure in redis which hosts this link but this info has to be derived from “link” in some cases, thus making it unusable).

• There are some other expensive solutions which may not be possible to employ at this time - moving read/writing links to a separate thread - because that will require significant changes to existing Redis design.

  • But any suggestions on this is welcome. Like once we have packet data from the callbacks, can we move (or append?) it to some structure, so that redis thread could pick it up and process it - thus both reads/writes are done in a single thread. There might be other complications with this approach but any leads on some pattern around it would be helpful

Can you please suggest some other ideas which will be useful in solving this problem? Also, is there any way we could get away without locking (some other synchronization pattern we could employ?). Any help in this regard is greatly appreciated.

If something is not clear, please let us know. We could provide more details to clarify.

[nibanks]

Synchronization is hard! Whenever you have multiple threads sharing a resource (a connection) you're going to have to add some synchronization. Also, I'd strongly recommend staying away from any global mutex/lock design. As you already mentioned, if you have a lot of parallel threads trying to use it, you're going to have problems. Generally, the goal should be to have a lock specific to the smallest resource possible. This might be a whole connection. It might be for a "work queue" on a connection. When you have a shared object with multiple users of it, having a reference count on the object can also be a good idea. When a new user of the object comes along, you increment the ref count. When one goes away, you decrement it. When it goes to zero, you clean up.

As to exactly how you should do things in your scenario, I'd probably need some more details on exactly what types of operations are being performed on which threads, but without that extra info, I'd recommend something like this:

• For each connection you have your own object/class - MY_CONNECTION
• You create your object and associate it's pointer with the QUIC connection handle as the "context"
• You object at least has it's own lock
• Never hold that lock and call into MsQuic
• Don't hold the lock very long on a MsQuic callback
• If you have a model of a single thread doing all the work already, perhaps you extend that and add a "QUIC work queue"
  • Whenever you get a callback from QUIC, all you do is queue a work item for that into the work queue and return
  • Then your main thread is updated to drain that work queue along with everything else it already does

Bottom line, though, there are lots of ways to go about this. Feel free to provide more details and I can try to give you a better suggestion.

[ssinghaldev]

Thanks for the advice. In interest of time(have to submit this project later next week), for now we have increased our node-timeout for failure detection and it seems to get by without synchronization. We will revisit this after the semester and try to redesign the code to accommodate worker queues so that everything happens on a single thread