[Discuss] Flink on Velox supports

[lsyldliu]

Hi, guys

I'm focusing on flink sql in flink community, now we are investigate the native vectorized execution solution of batch sql. We found that velox is an universal open vectorized execution plugin, it can support presto and spark now. Moreover, Flink is very similar to Spark, they are both rpc-based and JVM-based compute engine, also the overall architecture is similar, so we think flink on velox is workable. I want to ask whether velox community has interest to support flink? Maybe there are some effort we can do together.

[pedroerp]

Hi @lsyldliu, thanks for reaching out! As we work on expanding Velox's reach and growing our open soruce community, we are certainly interested in providing integrations with other widely adopted big data systems such as Flink.

Some of our close partners from Intel (cc: @davecohn) have expressed interest in the past in having an open source stream processing Velox integration available, so his team might be interested in helping. In addition, the Velox to Spark integration recently created by Intel (called Gluten - https://github.com/oap-project/gluten/) has a similar JVM/JNI integration mode, so maybe some of that work could be reused.

Let me know how we can help. The community usually communicate via Slack, so if you send me your work email me or @jijufb can send you an invite and we could pick up from there?

[jijufb]

Hi @lsyldliu, please send your work email to [email protected] and I can send you invite to access Velox Slack channel.

[FelixYBW]

In addition, the Velox to Spark integration recently created by Intel (called Gluten - oap-project/gluten) has a similar JVM/JNI integration mode, so maybe some of that work could be reused.

@lsyldliu Yes, we definitely can reuse the native part for Spark and Flink, maybe some JVM part as well. You may ping me (@binwei Yang) in Slack channel and we can have a sync offline.

[lsyldliu]

Hi @lsyldliu, thanks for reaching out! As we work on expanding Velox's reach and growing our open soruce community, we are certainly interested in providing integrations with other widely adopted big data systems such as Flink.

Some of our close partners from Intel (cc: @davecohn) have expressed interest in the past in having an open source stream processing Velox integration available, so his team might be interested in helping. In addition, the Velox to Spark integration recently created by Intel (called Gluten - https://github.com/oap-project/gluten/) has a similar JVM/JNI integration mode, so maybe some of that work could be reused.

Let me know how we can help. The community usually communicate via Slack, so if you send me your work email me or @jijufb can send you an invite and we could pick up from there?

Thanks for your attention, I'm very glad to hear that velox community is interested in integration with flink. Due to I'm not familiar with velox currently, so I want to get some help from community about how integrate velox with flink, it seems that Gluten is very good to me. I will study it first.

[lsyldliu]

Hi @lsyldliu, please send your work email to [email protected] and I can send you invite to access Velox Slack channel.

@jijufb Thank you very much, I'm from alibaba, my work email is [email protected].

[lsyldliu]

@FelixYBW Okay, it sounds like good news to me.

[Aitozi]

Hi @lsyldliu @FelixYBW, nice to see the discussion here. Are there any ongoing work around the Velox intergrated with Flink? We also try to explore the vector execution engine to accelerate the Flink batch/streaming sql

[FelixYBW]

Nothing from my side. @lsyldliu Did you started the work? You may take a look of this PR in substriat: substrait-io/substrait-java#90. It converts spark's logic plan into substrait plan. Currently Gluten used different way to create the substrait plan. We are checking if we can use the same way here.

The biggest issue is spark plan isn't 1:1 mapping to velox plan. In Gluten when we create substrait plan in Gluten, we customized the generated substrait plan to make sure it's 1:1 mapping between substrait plan and Velox's plan.

[qqibrow]

@FelixYBW This is Lu from Pinterest. We are also exploring whether we can bring velox in flink. could you add me into velox slack? my email is [email protected]. Thanks!

[qqibrow]

cc @pedroerp

[pedroerp]

Hi Lu, I just sent you a Slack invite.

[qqibrow]

Thanks!

[zoudan]

Hi @pedroerp, I am also interested in it, could you please invite me, my email is [email protected], thanks!

[wypb]

Hi @pedroerp, could you please invite me? [email protected], thanks.

[pedroerp]

Sent as well

[MrMonad]

can you please invite me as well [email protected]

[whutjs]

@pedroerp Could you please invite me as well, thanks! [email protected]

[pedroerp]

Invite sent!

[sharath1709]

Hi @pedroerp , This is Sharath from UBER and we're also interested in bringing velox to Flink. could you please invite me, my email is [email protected]

[pedroerp]

Invite sent.

[FangYongs]

Hi @pedroerp , I'm also interested in this issue, please invite me too, email [email protected] Thanks

[lorinlee]

Hi @pedroerp , Could you please invite me as well, thanks! email: [email protected]

[pedroerp]

Invite sent.

[waitinfuture]

Hi @pedroerp , Could you please invite me as well, thanks! [email protected]

[pedroerp]

Invite sent as well.

[liuyongvs]

@pedroerp ， Could you please invite me as well, thanks! [email protected]

[zjuwangg]

hi @pedroerp ， Could you please invite me as well, thanks! [email protected]

[Poorvankbhatia]

hi @pedroerp , Could you please send me an invite. Thanks [email protected]