Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[AIRFLOW-5582] AutoCommit in jdbc is missing get_autocommit #6232

Merged
merged 5 commits into from
Nov 17, 2019
Merged

[AIRFLOW-5582] AutoCommit in jdbc is missing get_autocommit #6232

merged 5 commits into from
Nov 17, 2019

Conversation

blindlf
Copy link
Contributor

@blindlf blindlf commented Oct 2, 2019

Make sure you have checked all steps below.

Jira

Description

  • Some of JDBC driver does not support autoCommit=false, such as org.apache.hive.jdbc.HiveDrive.
    JdbcHook.set_autocommit update by conn.jconn.setAutoCommit, DbApiHook.get_autocommit retrieve by conn.autocommit.
    After DbApiHook.execute(sql), conn.commit() will throws exception.

Tests

  • My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

  • My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
    1. Subject is separated from body by a blank line
    2. Subject is limited to 50 characters (not including Jira issue reference)
    3. Subject does not end with a period
    4. Subject uses the imperative mood ("add", not "adding")
    5. Body wraps at 72 characters
    6. Body explains "what" and "why", not "how"

@codecov-io
Copy link

codecov-io commented Oct 2, 2019

Codecov Report

Merging #6232 into master will increase coverage by 3.25%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #6232      +/-   ##
==========================================
+ Coverage   80.05%   83.31%   +3.25%     
==========================================
  Files         610      645      +35     
  Lines       35264    37293    +2029     
==========================================
+ Hits        28232    31071    +2839     
+ Misses       7032     6222     -810
Impacted Files Coverage Δ
airflow/hooks/jdbc_hook.py 100% <100%> (+5.88%) ⬆️
airflow/kubernetes/volume_mount.py 44.44% <0%> (-55.56%) ⬇️
airflow/kubernetes/volume.py 52.94% <0%> (-47.06%) ⬇️
airflow/kubernetes/pod_launcher.py 45.25% <0%> (-46.72%) ⬇️
airflow/example_dags/example_python_operator.py 63.33% <0%> (-31.12%) ⬇️
...rflow/contrib/operators/kubernetes_pod_operator.py 77.14% <0%> (-21.37%) ⬇️
airflow/executors/kubernetes_executor.py 58.99% <0%> (-6.24%) ⬇️
airflow/models/dagbag.py 86.88% <0%> (-5.5%) ⬇️
airflow/ti_deps/deps/pool_slots_available_dep.py 95.65% <0%> (-4.35%) ⬇️
airflow/gcp/operators/bigquery.py 84.18% <0%> (-2.43%) ⬇️
... and 256 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 76fe45e...5cfe487. Read the comment docs.

@kaxil kaxil force-pushed the master branch 2 times, most recently from 558f9f1 to d719e1f Compare October 2, 2019 15:30
Copy link
Member

@feluelle feluelle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Welcome @blindlf ! :)

airflow/hooks/jdbc_hook.py Show resolved Hide resolved
Return True if conn.autocommit is set to True.
Return False if conn.autocommit is not set or set to False

:param conn: The connection
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you also add the type, please? Either via the docs :type or in the function header directly to the arg conn:

@OmerJog
Copy link
Contributor

OmerJog commented Oct 23, 2019

@blindlf any progress with addressing the comments?

@blindlf
Copy link
Contributor Author

blindlf commented Oct 25, 2019

@blindlf any progress with addressing the comments?

Sorry for the late. Comments are done, I'm trying to work on unit test.

tests/contrib/hooks/test_jdbc_hook.py Outdated Show resolved Hide resolved
tests/contrib/hooks/test_jdbc_hook.py Outdated Show resolved Hide resolved
@blindlf blindlf closed this Nov 10, 2019
@blindlf blindlf reopened this Nov 10, 2019
Copy link
Member

@feluelle feluelle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can assert that a mock object has been called exactly once.

tests/contrib/hooks/test_jdbc_hook.py Outdated Show resolved Hide resolved
tests/contrib/hooks/test_jdbc_hook.py Outdated Show resolved Hide resolved
@blindlf
Copy link
Contributor Author

blindlf commented Nov 11, 2019

You can assert that a mock object has been called exactly once.

I'm not sure about this.
The getter/setter can be invoked multiple times without side effect. autocommit is not the operator like the SQL insert or HTTP post, why it should be restricted to be called exactly once?

@feluelle
Copy link
Member

why it should be restricted to be called exactly once?

Because that's what the function should do if you call it once.

Copy link
Member

@feluelle feluelle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your patience @blindlf :)

@feluelle feluelle merged commit d03fb28 into apache:master Nov 17, 2019
kaxil pushed a commit that referenced this pull request Dec 17, 2019
- add tests
- update docs

Co-Authored-By: Felix Uellendall <[email protected]>
(cherry picked from commit d03fb28)
ashb pushed a commit that referenced this pull request Dec 17, 2019
- add tests
- update docs

Co-Authored-By: Felix Uellendall <[email protected]>
(cherry picked from commit d03fb28)
ashb pushed a commit that referenced this pull request Dec 19, 2019
- add tests
- update docs

Co-Authored-By: Felix Uellendall <[email protected]>
(cherry picked from commit d03fb28)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants