Horizontal scaling and multiple backends

[julienlau]

Hi,

If the DB is hoster in a postgres instance isolated from Lighter, can Lighter be scaled horizontally ?
For exemple :

• 2 instances of Lighter for better HA
• 2 instances of Lighter for different configuration : one with backend Local and the other with backed Kubernetes

Regards
Julien

[Minutis]

It can be scaled horizontally. We are using Shedlock for locks when running more than one instance of Lighter. So generally having a few Lighter instances could help with HA but there are some limitations:

• A single instance of the Lighter is doing tasks at a time. That means in case the load is really huge scaling several instances of Lighter is not going to help to balance the load.
• In case there is a big load for a single Lighter instance the HA setup can bring some issues with stability. Since locks for tasks have fixed time before it is unlocked by brute force and second instance could try to execute tasks that are already being processed in the first one.

Regarding the separate Lighter configurations and separate backends, it could probably be done. But I would recommend separating the Lighters logically (separate DBs and endpoints configured) to get better visibility and debugging options.

[julienlau]

I don't understand the lock strategy of Lighter... The locks are not there to avoid Leader election between several Lighter instances ?

As I understand, the lock is used to avoid concurrency between update tasks:
The locks are here to avoid bad operations form other task on a given Lighter instance : track session could interfere with create session.

If so, would it be possible to have a queue with a single thread doing update tasks and another queue with more threads for the reads or the concurrent-safe tasks ?
Note: I do not know micronaut

Regards

[Minutis]

You are right regarding the lock strategy. Current implementation is just a simple way to ensure that there is some kind of redundancy but it is far from optimal.

Of course it's possible to implement different concurrency mechanisms but it takes time in both development and additional effort to test it.

We are always open for contributions though :) Closing for now.