Adding ClickHouse Provider

[pateash]

closes: #10893

---

Description

Adding ClickHouse provider based on its Python SDK https://clickhouse-driver.readthedocs.io/en/latest/

Users can create their own custom operators leveraging the ClickHouseHook directly
or building their operator on ClickHouseOperator by providing result_processor method,

operator = ClickHouseOperator(
    task_id='clickhouse_operator',
    sql="SELECT * FROM gettingstarted.clickstream",
    dag=dag,
    result_processor=lambda cursor: print(cursor)
)

The sensor can be implemented by SQL

sensor = ClickHouseSensor(
    task_id="clickhouse_sensor",
    sql="SELECT * FROM gettingstarted.clickstream where customer_id='customer1'",
    timeout=60,
    poke_interval=10,
    dag=dag,
)

[pateash]

[pateash]

Testing providers with sql and template files(*_tf)

[pateash]

[raphaelauv]

ClickHouseQueryOperator ?

also it could maybe be base on the https://pypi.org/project/apache-airflow-providers-common-sql/ ?

[pateash]

Its similar

[raphaelauv]

what is similar ?

[eladkal]

@pateash can you elaborate on this point?
I think ClickHouseHook should inherit from DbApiHook and ClickHouseOperator should inherit from BaseSQLOperator

Is there a reason why we shouldn't do it?

[pateash]

HI @eladkal, @raphaelauv ,
Sorry for the late response, I was OOO.

the reason I didn't used DBApiHook is because it's intended to be used with databases ( mostly transactional supporting sqlalchemy ),

and Clickhouse being a distributed OLAP, i really think that it will be better not to expose methods like insert_rows(), which uses sqlalchemy.
rather if someone wants to have this functionality they should use the underlying library ( clickhouse-driver ) and implement their own operator using the hook and connection object.

[eladkal]

I think this is a source for confusion? Maybe we should customize the connection in the like we do with other providers?

[pateash]

Could you please provide more information?

here, if someone wants to override the default schema ( provided by connection ), they can pass the database argument.

[eladkal]

Where is result_processor implemented?

[pateash]

result process is a callback which could be passed by the user to proess the result similar to arangodb provider.

[eladkal]

I asked it elsewhere but if ClickHouseHook will inhert from DBApiHook then we won't need this custom sensor because SqlSensor will support ClickHouse natively https://github.com/pateash/airflow/blob/f7e2ffe42bb7e957e4e16d0cb65e9541c87bd72b/airflow/providers/common/sql/sensors/sql.py#L80

[pateash]

HI @eladkal,
Sorry for the late response, I was OOO.

the reason I didn't used DBApiHook is because it's intended to be used with databases ( mostly transactional supporting sqlalchemy ),

and Clickhouse being a OLAP, i really think that it will be better not to expose methods like insert_rows(), which uses sqlalchemy.
rather if someone wants to have this functionality they should use the underlying library ( clickhouse-driver ) and implement their own operator using the hook and connection object.

[sachebotarev]

Hi @pateash

I'm interested in having airflow support ClickHouse, but I don't want to have such a standard implementation.

1. You realisation very similar Anton Bryzgalov's rather old code https://github.com/bryzgaloff/airflow-clickhouse-plugin. His code is suitable for plugins, but it looks strange in the provider
2. I'm very confused by calling self.get_conn() in a constructor. The connection is not closed anywhere.
3. clickhouse-driver supports DB API 2.0 so inheriting from DbApiHook and ClickHouseOperator will provide many out of the box solutions. insert_rows( ) can be overridden with Cursor.executemany or just rise exception.

cursor.executemany('INSERT INTO test (x) VALUES', [[200]])

4. Your ClickHouseHook provides very limited functionality, compare for example with ExasolHook (also an analytical database).
5. Your ClickHouseOperator does not provide the same capabilities (last result, many requests in template, xcom ).
   I would get these features (and more) just using SQLExecuteQueryOperator

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.