Horizontally scaled Web Applications (Dash, Django, etc) running Vaex

[mtwichan]

Hi @maartenbreddels,

I've been thinking lately about how Vaex would scale with a horizontally deployed Dash application. From our conversation together, what I understood is that Vaex caches utilized data into disk memory.

How would the caching work if multiple nodes (horizontally scaled) are running a Vaex Dash application? Also, do you know of any horizontally scaled Dash applications or web applications that are utilizing Vaex?

In short, does Vaex work in a horizontally scaled environment? If so, how does it work?

Thanks in advance for your time Maarten!

CC: @Kully

[maartenbreddels]

Hi Matthew,

what I understood is that Vaex caches utilized data into disk memory.

Yes, that is correct, I would say that if you run on EC2 and get your data from S3 (same region) that might not be needed for good performance (pass cache=False to vaex.open). But otherwise it's good to cache locally on a fast ssd if you data is much larger than memory (otherwise the OS filesystem caching should be enough).

I'd say horizontal scaling with Dash and Vaex is a good combination, provided each node handles each request by its own. So this assumes that you only scale up to serve more users. There are no components in Vaex or Dash that don't scale, so there is no issue at all. What you could consider, is enabling the new caching feature, and set up a central Redis server for caching: https://vaex.io/docs/api.html#module-vaex.cache

I hope that answers your question.
cheers,

Maarten

PS: I moved this to discussions

[mtwichan]

Yes, that is correct, I would say that if you run on EC2 and get your data from S3 (same region) that might not be needed for good performance (pass cache=False to vaex.open). But otherwise it's good to cache locally on a fast ssd if you data is much larger than memory (otherwise the OS filesystem caching should be enough).

How would the local caching work if there are multiple nodes hosting the application? Would each node cache the utilized data into its own disk space? If not, is there some sort of shared disk space that the nodes can share?

I'd say horizontal scaling with Dash and Vaex is a good combination, provided each node handles each request by its own.

Great. So it would be ideal to use something like sticky sessions when deploying the application?

Thanks Maarten!

Regards,
Matthew