Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[AIRFLOW-6564] Additional diagnostics information on CI check failure #7172

Conversation

potiuk
Copy link
Member

@potiuk potiuk commented Jan 15, 2020


Issue link: AIRFLOW-6564

Make sure to mark the boxes below before creating PR: [x]

  • Description above provides context of the change
  • Commit message/PR title starts with [AIRFLOW-NNNN]. AIRFLOW-NNNN = JIRA ID*
  • Unit tests coverage for changes (not needed for documentation changes)
  • Commits follow "How to write a good git commit message"
  • Relevant documentation is updated including usage instructions.
  • I will engage committers as explained in Contribution Workflow Example.

* For document-only changes commit message can start with [AIRFLOW-XXXX].


In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.

@potiuk potiuk force-pushed the AIRFLOW-6564-additional-diagnostics-information-on-ci-check-failure branch from 886b91e to 87a0197 Compare January 15, 2020 09:10
@potiuk potiuk force-pushed the AIRFLOW-6564-additional-diagnostics-information-on-ci-check-failure branch from 2cd1c72 to 31762e5 Compare January 15, 2020 15:51
@boring-cyborg boring-cyborg bot added the area:Scheduler including HA (high availability) scheduler label Jan 15, 2020
@potiuk potiuk requested review from Fokko and kaxil January 15, 2020 16:05
@potiuk
Copy link
Member Author

potiuk commented Jan 15, 2020

For now I have not yet tried to improve entropy retrieval from Docker (I will see if that can be achieved easily) but for now I added a bit more sophisticated way of making sure the environment for tests is sane.

I test each db/integration in a loop n-times + I added the special case for MySQL with parsing the logs + we have full diagnostics in case any of the checks fails.

Pls. take a look!

@potiuk
Copy link
Member Author

potiuk commented Jan 15, 2020

I also updated mysql to 5.7 - I might remove that and make a separate PR.

Copy link
Contributor

@Fokko Fokko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two minor points, apart from that: LGTM

airflow/jobs/scheduler_job.py Outdated Show resolved Hide resolved
airflow/utils/db.py Show resolved Hide resolved
airflow/utils/db.py Outdated Show resolved Hide resolved
@potiuk potiuk force-pushed the AIRFLOW-6564-additional-diagnostics-information-on-ci-check-failure branch 2 times, most recently from 0ca023d to ead4811 Compare January 15, 2020 20:27
@codecov-io
Copy link

codecov-io commented Jan 15, 2020

Codecov Report

Merging #7172 into master will increase coverage by 0.13%.
The diff coverage is 50%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #7172      +/-   ##
==========================================
+ Coverage   85.24%   85.37%   +0.13%     
==========================================
  Files         710      723      +13     
  Lines       39485    39543      +58     
==========================================
+ Hits        33660    33761     +101     
+ Misses       5825     5782      -43
Impacted Files Coverage Δ
airflow/bin/cli.py 94.73% <ø> (ø) ⬆️
airflow/cli/commands/db_command.py 95.12% <50%> (-2.32%) ⬇️
airflow/utils/db.py 97.93% <50%> (-2.07%) ⬇️
airflow/contrib/hooks/azure_data_lake_hook.py 0% <0%> (-93.11%) ⬇️
airflow/contrib/sensors/azure_cosmos_sensor.py 0% <0%> (-81.25%) ⬇️
airflow/configuration.py 88.01% <0%> (-3.43%) ⬇️
...low/contrib/operators/wasb_delete_blob_operator.py 100% <0%> (ø) ⬆️
airflow/contrib/operators/adls_list_operator.py 100% <0%> (ø) ⬆️
airflow/contrib/hooks/azure_fileshare_hook.py 100% <0%> (ø) ⬆️
airflow/contrib/sensors/wasb_sensor.py 100% <0%> (ø) ⬆️
... and 25 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8729ca5...5f23840. Read the comment docs.

airflow/bin/cli.py Outdated Show resolved Hide resolved
airflow/bin/cli.py Outdated Show resolved Hide resolved
@potiuk potiuk force-pushed the AIRFLOW-6564-additional-diagnostics-information-on-ci-check-failure branch from ead4811 to 5f23840 Compare January 15, 2020 21:35
@potiuk potiuk merged commit 417feda into apache:master Jan 15, 2020
potiuk added a commit that referenced this pull request Jan 25, 2020
potiuk added a commit that referenced this pull request Jan 26, 2020
kaxil pushed a commit that referenced this pull request Jan 26, 2020
kaxil pushed a commit that referenced this pull request Feb 3, 2020
galuszkak pushed a commit to FlyrInc/apache-airflow that referenced this pull request Mar 5, 2020
kaxil pushed a commit to astronomer/airflow that referenced this pull request Mar 30, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:CLI area:dev-tools area:Scheduler including HA (high availability) scheduler
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants