Review connection/hook usage

[potiuk]

Description

The discussion in #12466 (comment) had shown that there is likely an expectation about making sure all the connections are 'registrable' in the UI of Airflow. This is not the case currently. Not all connections are visible in the UI (and possibly for a good reason) and currently connections are defined by Hook class variables.

Only a subset of the existing hooks provides this information, first of all because many of the connections/hooks do not require any custom fields, behaviour, they do not use automated Hook creation by connection type, but also it is because many of the hooks share the same connection type (For example almost all Google Hooks, most Amazon hooks etc).

Seems that we have 1 <> Many relationships Connection/Hook rather than 1-1 as initially designed. Also, there is an existing "standard" introduced by the DBApi Hook where this metadata is stored in the Hook class as class-level field. Which we carried to those hooks that required to register themselves in the UI:

For example:

class ImapHook(BaseHook):
    conn_name_attr = 'imap_conn_id'
    default_conn_name = 'imap_default'
    conn_type = 'imap'
    conn_name = 'IMAP'

We likely need to address that by defining clear rules for the hooks/providers:

• whether we want to be able to map all hooks to connections
• whether we still want to create hook class automatically based on connection type - especially where we have multiple hooks sharing one connection type
• should we maybe have connection aliases to alias same connection type for different hooks
• where to keep that information (in Hooks ? Elsewehere)
• how to name the connection types - especially when there are siblings and multiple hooks sharing connection type.

I think those questions need to be answered before we can think about starting any implementation.

Related Issues

#12466 #11429 #12465

[potiuk]

cc: @mik-laj @ashb -> if you want to tackle this, I described the situation and open questions we have for the Hook/Connections story. Marked it provisionally for 2.1. Feel free to take a lead on it!

[kaxil]

I think this is done now, right @potiuk ?

[potiuk]

I think the problems about some inconsistent connection have been handled already. Part of them in separate provider changes and final step was done with Provider's managers refactoring and import optimisations. I think all the cases with 1<>Many relations which I was aware of have been handled - without setting clear rules though.

I think it is handled in a "good-enough" way for now. Any rule setting and enforcing them should be done when we migrate-away from WTforms and declarative way of defining connections in the UI with the new modern UI (or abandon the idea of configurable connections via UI, but that does not seem likely).