Delta WASM / Web Support

[mmgeorge]

I was wondering if there are any known workflows currently for interacting with delta tables from a web client. I know there's also a rust kernel lib that can mostly be ported to WASM, but I'm not sure to what extent that library is officially maintained.

Anyone have any experience with consuming delta tables from the web?

[zen-data]

The delta-connect feature for Delta 4.0 could help with this. It provides Delta support for spark-connect, so you get a gRPC bridge between your Delta tables and your backend. This provides you with the glue to bind between whatever the Web based experience, and your Delta tables.

Take a look at https://connectrpc.com/. This provides additional hooks on top of classic gRPC to make working with gRPC data on the web simple. It has typescript bindings, so while there would be some work required to proxy from web connectrpc <-> delta-connect <-> delta-tableX. It is doable.

If you wanted to just send the arrow buffers back and forth - you could look at https://arrow.apache.org/docs/js/ as well.

[zen-data]

Also food for thought. If you are thinking about writing something like a notebook environment, you could look at https://github.com/newfront/spark-inception/tree/main from a talk I gave in 2022. The longer walkthrough is here (https://newfrontcreative.medium.com/data-ai-2020-spark-inception-8ef5b67b9acb). This does however pre-dates spark-connect.

[mmgeorge]

Thanks @zen-data! Had not heard of delta-connect before, worth looking into. I've used grpc-web exposed through an envoy proxy in the past, which sounds like it could be somewhat similar to connectrpc?

My specific use case is for data visualization and mapping some subset of a delta table with a geometry column.

[newfront]

I like the use case. I used to use Apache Zeppelin a lot and it always felt like magic to be able to access my dataframe data and easily populate the different visualization widgets. Have fun.

Also, the envoy proxy + grpc-web is essentially what connectrpc makes simpler. The client can use standard http or http/2 depending on the capabilities of the client - so you don't need the proxy.