Add new operator for ClickHouse

[bharatnc]

Description

Add a new operator for ClickHouse.

Use case / motivation

Use case similar to any other operators like Postgres, MySQL etc for working with ClickHouse.

Related Issues

As far as I have looked, there is already a plugin that can be installed.

But would be nice to have this part of airflow alongside other available operators.

If this feature sounds acceptable, then I can work on adding this to airflow (please feel free to assign this issue to me). However, I'm not sure if a similar feature has already been requested before (as far as I have searched through the issues, I didn't find one). Thank you!

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

Sure thing ! Go ahead @bharatnc

[mik-laj]

Should we use DB API? Thanks to this, we won't have to implement operators for this software, but we will be able to use generic operators for SQL.

[potiuk]

I think there is a value in using specific operators for each DB. There is not a lot of code and DBAPI is designed rather as a very low-level API that usually is implemented by driver specific for the Database. This is how most of the DB operators work and allow to implement specific features of each operator with the API's that are much easier to use.

[bharatnc]

Thank you! I haven't taken a look at DB generic API but sounds like what @potiuk describes is true and applicable here. The generic DB API will work for basic queries but a lot of operations and other queries used in Clickhouse have rather specific syntax / functionality unique to that database. The overlap of functionality b/w generic DB API operations and Clickhouse operations are rather not very extensive. So I think it would be better to have a separate operator for this use case so that operations can be made more extensive in future. Do let me know if I am correct or if I've completely missed the intention here.

[potiuk]

Regardless if we would like to implement DBAPI also for ClickHouse it is not a good idea to do it now IMHO. If we decide to do that - we should do it as a separate PR and see how much of the commonalities we can extract at all. But I really do not think adding db api when adding new database (and much different than the regular one) is a good idea now. @mik-laj let us know if you think otherwise, but please go ahead @bharatnc without it.

[eladkal]

For anyone who wants to pick this task:
Python driver that can be used: https://github.com/mymarilyn/clickhouse-driver
Reference for PR that added another influxdb provider #17068

[pateash]

@eladkal, I would like to pick this up.

[subkanthi]

Is this being worked on, happy to contribute to the Async version

[eladkal]

The issue here is that this means adding new provider and right now we are discussing in the mailing list about approch to new providers
https://lists.apache.org/thread/nvfc75kj2w1tywvvkw8ho5wkx1dcvgrn

Maybe you can share your thoughts on this in the mailing list?

[pateash]

Sorry for late response, I was OOH
As far as I understood from the conversation with @eladkal, that there is a wider conversation going on regarding whether we should add new providers or not.
I had code ready but decided not to raise a PR.
If we want to add a new provider I have changes ready.

[potiuk]

Yeah. We want to make sure we do the right thing - see the discussion. We will resume the discussion in the devlist after the Airflow Summit (and conversations during the Summit are most welcome).

[potiuk]

Hello @pateash @subkanthi @bharatnc It took a bit long but I think we got to a consensus regarding new providers, and we also propose some kind of mix-governance aproach, where (among the others) the stakeholders for the future providers (which we are going to technically split to separate repositories soon) should take a bit more responsibility for maintenance:

#24680

If that does not scare you away, and you still want to add the provider to Airflow community providers, feel free to make a PR after we merge #24672 - we are going to change the way how we keep depdencies for providers in order to prepare them to separate to different repository.

Let us know what you want to do, either close the issue or make PR and rebase, and lead it to completion after #24672 is merged.

[pateash]

thanks @potiuk, this makes sense.

I will raise the PR after #24672 is merged.

[potiuk]

It's merged.

[pateash]

It's merged.

Thanks @potiuk,
Let me have a look.

[eladkal]

Policy for adding new provider is explained at https://github.com/apache/airflow/blob/main/PROVIDERS.rst

I'm going to close this issue as the process of adding a provider require investment and commitments from the requester thus it's unlikely that someone will just pick up this request and implement it so no reason to keep this issue open.