Implements SqlToSlackApiFileOperator

[Taragolis]

closes: #9145
follow-up: #24660

Rather than extend existed SqlToSlackOperator I've decided to create new transfer operator SqlToSlackApiFileOperator.

cc: @eladkal

[eladkal]

Rather than extend existed SqlToSlackOperator I've decided to create new transfer operator SqlToSlackApiFileOperator.

Can you explain why?

[Taragolis]

Can you explain why?

1. SqlToSlackOperator intends to use Slack Webhook ether based on Slack App or Slack Legacy Integrations but both won't support send file, which only allowed by Slack API. Slack Incoming Webhooks and Slack API are not not interchangeable and do not have common interface.
2. Expected different parameters, so if extend current operator than required 4-5 additional parameters which won't use in case of use send as a regular message.
3. I thought also easier maintain something simple rather than with complicated logic and a lot of additional conditions

[eladkal]

I thought also easier maintain something simple rather than with complicated logic and a lot of additional conditions

Im not sure this makes it simpler?
If I'm using current operator to send message and now I want the exact same results just in file it require to switch to another operator. Its not very intuative and a potential source for confusion.

But I'd like to hear also @alexkruc point of view as the author of the operator.

[Taragolis]

If I'm using current operator to send message and now I want the exact same results just in file it require to switch to another operator

As well as create new connection with different type.

If create all in once operator than:

• slack_filename, slack_initial_comment, slack_initial_comment, slack_title, df_kwargs - parameters won't use if user want to send in slack as a message
• slack_channels - multiple channels (0 to many) supported only by Slack API, Slack Incoming Webhook based on Slack APP do not supported change channels at all, Slack Webhooks based on Legacy Integration could assign only one channel (slack_channel)
• slack_webhook_token - token are different for Slack Incoming Webhook and Slack API
• slack_message, results_df_name - won't use if user want to send as a File

[alexkruc]

Hi all :)
Please note that I'm not a commiter, so my replay might lack details -
I think that this discussion is showing us a bit of an underlying problem with the way Airflow works with Slack. There are 2 hooks that are entirely different with the parameter they accept and their behavior.. (link)
I think that this is causing a lot of confusion with the development of new operators/flows, as there is no strict guideline on when to use which Slack hook or whether one of them is deprecated.
So I think that what should be done in the long run is to consolidate the Slack hook in a way that allows a single hook to be used with the webhook integration or with the Slack API.

My take on this specific matter is until we don't have a unified Slack hook, having 2 operators that both of them use different Slack hooks is valid because the hooks are different in their nature, and their configuration is different as well. If we want to solve this, the solution should be on the SlackHook layer IMO.

@eladkal WDYT?

[Taragolis]

Right now there are 3 ways to send formatted message to slack

1. Use Slack API and chat.postMessage. Supported by Python Slack SDK
2. Use Slack Incoming Webhook based on Slack App. Supported by Python Slack SDK
3. Use Incoming Webhook based on Slack Integration. Slack integration itself is legacy tech for a while.

And only Slack API use for other integration with slack.

Authentification

Slack API use access token based on the type of integration (e.g. for Bots xoxb-1234567890123-09876543210987-AbCdEfGhIjKlMnOpQrStUvWx) and send requests to https://www.slack.com/api/ (by default)

Slack Incoming Webhook and Incoming Webhook based on Slack Integration use URL (the same format: https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX) for auth.

The main differences between Slack Incoming Webhook and Legacy version that for legacy integration you could change username, channel, icon_url and icon_link. If you tried to send this parameters to Slack Incoming Webhook it just ignored and user wouldn't notified.

[eladkal]

Use Incoming Webhook based on Slack Integration. Slack integration itself is legacy tech for a while.

Does this mean that the SlackWebhookHook is deprecated?

[Taragolis]

Does this mean that the SlackWebhookHook is deprecated?

Not at all. It might use something that actually has no affect anymore and it is depend on what user actually use. I've tried to do some significant changes in SlackWebhookHook (#26452)

Personally I've also have an issue with slack in the past:

• 2019-2020: in one of the project we used SlackWebhookHook for send callbacks, and it work fine we could change channel and displayed username. We did not create webhook url by our own, someone just gave it to us
• 2021: another project, we were creating webhook url by our own.... and nothing actually worked as expected. So we just switch to Slack API
• end of 2021-early 2022: one guy told me "Did you want a magic? This is two Slack Webhook URL, looks similar use in the same Slack Workspace but one of them allow change channel and username and another not"

[potiuk]

What a mess :) . @eladkal - > I am ffne with this approach

[eladkal]

This is a mess but because this is a feature and not a bug fix I prefer maybe that we first get #26452 revewed and merged and sort out how to use slack in general and then revist this one?
However I'm OK also with moving a head with this and possible make changes/fixes later if we feel this is important to get it for next release

[eladkal]

What are our options if we want to avoid creating a new operator for files?

[Taragolis]

Let me just summarise what we have right not, what we use right now and what Slack supports

Send Message in Slack Channel supported by Airflow:

1. Slack API chat.postMessage method via SlackHook.call method
2. Slack Incoming Webhook via SlackHook.send_dict, SlackHook.send and SlackHook.send_text methods
3. Slack Webhook based on Legacy Integration via SlackHook.send_dict, SlackHook.send and SlackHook.send_text methods

Full list for what could be use for send message into Slack Channel

Send File in Slack Channel (or Workspace) supported by Airflow:

1. Slack API files.upload method via SlackHook.send_file or SlackHook.call methods

Which parameters could be provided for different methods

Slack API Method chat.postMessage (Mainstream)

parameter	required	scope
token	Yes	headers
channel	Yes	dict payload
attachments	At least one of attachments blocks text	dict payload
blocks	At least one of attachments blocks text	dict payload
text	At least one of attachments blocks text	dict payload
as_user	No	dict payload
icon_emoji	No	dict payload
icon_url	No	dict payload
link_names	No	dict payload
metadata	No	dict payload
mrkdwn	No	dict payload
boolean	No	dict payload
parse	No	dict payload
reply_broadcast	No	dict payload
thread_ts	No	dict payload
unfurl_links	No	dict payload
unfurl_media	No	dict payload
username	No	dict payload

Slack API Method files.upload (Mainstream)

parameter	required	scope
token	Yes	headers
channels	No	dict payload
content	No (if file provided)	dict payload
file	No (if content provided)	multipart/form-data
filename	No	dict payload
filetype	No	dict payload
initial_comment	No	dict payload
thread_ts	No	dict payload
title	No	dict payload

Slack Incoming Webhook (Mainstream)

There is no information about end list of parameters, due to the code of WebhookClient.send from slack_sdk only this parameters allowed (but not for 100% sure)

parameter	required	scope
token	Yes	URL
attachments	At least one of attachments blocks text	dict payload
blocks	At least one of attachments blocks text	dict payload
text	At least one of attachments blocks text	dict payload
response_type	No	dict payload
replace_original	No	dict payload
delete_original	No	dict payload
unfurl_links	No	dict payload
unfurl_media	No	dict payload

Slack Webhook based on Legacy Integration (Legacy)

Even less information than Slack Incoming Webhook. Due to investigation this parameters supported

parameter	required	scope
token	Yes	URL
channel	No	dict payload
attachments	At least one of attachments blocks text	dict payload
blocks	At least one of attachments blocks text	dict payload
text	At least one of attachments blocks text	dict payload
icon_emoji	No	dict payload
icon_url	No	dict payload
username	No	dict payload
unfurl_links	No	dict payload

And this additional parameters might supported

parameter	required	scope
response_type	No	dict payload
replace_original	No	dict payload
delete_original	No	dict payload
unfurl_media	No	dict payload

[potiuk]

Wow! . Quick question then - I am preparing for Provider's release. SHould I merge this one (code looks cool but I guess I need TL;DR; if the current state in this PR is "Releasable" if we merge it). I guess so, but wanted to get the feeling of others involved in the disucssion :)

[eladkal]

I dont think we should add operator just to send file. I understand the challange but I think we should try to mitigate this in the operator itself.

If Slack offers 3 diffrent approches than maybe we should have base class and 3 operators one per approch? Then each operator will be able to levrage the full capabilities of what slack offers?

[Taragolis]

I think we do not need to rush especially if we do not have agreement how to do it in a better way.

IMHO. For send SQL response as a message we could actually do by three different way without turn into the pain 🤣
most of the major parameters are presented in all 3 APIs requests.

For files situation are different we can use only Slack API and internally it do not have same parameters from different methods. Most close it text from chat.postMessage and initial_comment from files.upload even channel and channels working differently.

[eladkal]

@Taragolis Under the assumption that we do not want to add a new operator for this - what are our options?
I think we are having hard time here since the capabilities are not clear on the hook level.
If slack has 3 different APIs to send message maybe we should have 3 hooks ?

[Taragolis]

Initially when I started refactor slack provider I've also want to create additional hook but after deeply investigate I've found that Legacy and based on slack API Incoming Webhook has almost the same interface.
Legacy supports additional features which is not available in new one: change icons, username and channels.

And it is impossible (just my personal findings) to determine witch Webhook URL use for Legacy Incoming Webhook and which one for new one - even HTTP responses almost the same. That is why I combine usage in airflow.providers.slack.hooks.slack_webhook.SlackWebhookHook with warnings about parameters which supported by Legacy only

If we want to just send message as a text than we might create some of generic interface because all three methods support blocks and fallback messages.

With send as file (upload) it is quite difficult because it is different method of API with different parameters and it only supported by Slack API not Webhook. And in case of send as file we do not need overwrite render_template_fields for add Jinja filter in runtime

[potiuk]

@Taragolis @eladkal -> do you think that one is ready for prime time ? We could stil merge it and make it part of the November provider's wave.

[eladkal]

I think this is the best we can do at this point.

@Taragolis thank you for the through research!
(I am still not a fan of adding additional operator just to send files but this is not a reflaction on your work). Much appriciated!

Will merge when CI is green