-
Notifications
You must be signed in to change notification settings - Fork 14.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Adding support to hive hook for high availability Hive installations #38651
Adding support to hive hook for high availability Hive installations #38651
Conversation
Looks good :) |
Good catch! Updated the test, can you check once? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I added some nits, LGTM
@hussein-awala I have addressed the nits as well, can you take a look if it looks OK now? |
@potiuk @eladkal @hussein-awala if all looks good shall we send this one in? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My previous comments were nits and not blockers, LGTM
Merging this in favour of the reviews from Hussein and Elad. |
Right now there is a limitation where the HiveOperator incorrectly parses and constructs the beeline command if hive "high availability" url is given for host. I was able to work this around by adding this change to extra and changing the HiveHook code but a better fix would be to add a new field called HA, which when selected, parses the connection better in the hook itself.
I would like to propose adding this into the connection form for Hive CLI type in a better way. Recently some change was made to remove extra and integrate this into the form and simplify it: #37043.
Would propose adding similar fields for HA and if enabled, the beeline command construction would vary slightly.
MY HA URL looks somewhat like this btw:
jdbc:hive2://host1:port1,host2:port2,host3:port3/default;principal=hive/principal@REALM;serviceDiscoveryMode=zooKeeper;ssl=true;zooKeeperNamespace=hiveserver2
Tested using a HA setting, beeline works as expected.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rst
or{issue_number}.significant.rst
, in newsfragments.