[AIRFLOW-6685] ThresholdCheckOperator #7353
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst)
|
mik-laj
reviewed
Feb 3, 2020
alexzlue
force-pushed
the
AIRFLOW-6685
branch
7 times, most recently
from
5c3b6bc
to
e4d2930
Compare
Codecov Report
@@ Coverage Diff @@
## master #7353 +/- ##
==========================================
+ Coverage 86.35% 86.52% +0.17%
==========================================
Files 871 874 +3
Lines 40627 41841 +1214
==========================================
+ Hits 35083 36203 +1120
- Misses 5544 5638 +94
Continue to review full report at Codecov.
|
alexzlue
force-pushed
the
AIRFLOW-6685
branch
2 times, most recently
from
f6c003b
to
e85e036
Compare
alexzlue
force-pushed
the
AIRFLOW-6685
branch
from
e85e036
to
c1c243b
Compare
|
In general the operators in this PR sounds like enhancement of CheckOperator
|
|
@eladkal Thanks for bringing this to mind. I do see that there is some functionality that I have that |
alexzlue
changed the title
eladkal
reviewed
Feb 22, 2020
|
|
||
| def push(self, meta_data): | ||
| """ | ||
| Optional: Send data check info and metadata to an external database. |
There was a problem hiding this comment.
When inheriting from this class, push can be overwritten
eladkal
approved these changes
Mar 10, 2020
turbaszek
approved these changes
Mar 30, 2020
|
Awesome work, congrats on your first merged pull request! |
kaxil
pushed a commit
that referenced
this pull request
Apr 1, 2020
* [AIRFLOW-6685] Data Quality Check operators * removed .get_connection to get hook in get_sql_value * added tests for get_sql_value * threshold check operator and tests added to checkoperator file (cherry picked from commit 4c6ae18)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR includes a new operator in
CheckOperatorthat allows users to perform a threshold data quality check.ThresholdCheckOperatorwill check a single value, sql result against a threshold range, and will fail a task if it is outside this range. The lower and upper bound of the threshold can be defined as either a numeric values, or sql-statements that returns a numeric value.Issue link: AIRFLOW-6685
Make sure to mark the boxes below before creating PR: [x]
[AIRFLOW-NNNN]. AIRFLOW-NNNN = JIRA ID** For document-only changes commit message can start with
[AIRFLOW-XXXX].In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.