Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Healthcheck logic #3884

Merged
merged 6 commits into from
Aug 2, 2023
Merged

Conversation

ThomasLaPiana
Copy link
Contributor

@ThomasLaPiana ThomasLaPiana commented Aug 1, 2023

Closes #2517

Description Of Changes

The database health check step was found to be extremely heavy, sometimes spiking response times to ~1 sec when running locally with nothing else hitting the server

After doing some investigation of the worker checks as well, I couldn't find evidence that this was an issue. It only increased processing time by ~6ms. After checking everything I believe that the database was the primary culprit here, but we'll continue to monitor this fix as it is rolled out and verify that conclusion.

Code Changes

  • simplify the database health check

Steps to Confirm

  • open up ipython and run some tests using requests to get those logs populated, verify the numbers look right
  • go to database.py and force the url to be wrong for the healthcheck function. Ensure that the status is 503 and the error is logged

Pre-Merge Checklist

  • All CI Pipelines Succeeded
  • Issue Requirements are Met
  • Update CHANGELOG.md

@ThomasLaPiana ThomasLaPiana self-assigned this Aug 1, 2023
@cypress
Copy link

cypress bot commented Aug 1, 2023

Passing run #3457 ↗︎

0 4 0 0 Flakiness 0

Details:

Merge 9d91ae1 into eccf7b8...
Project: fides Commit: 810fdc0a16 ℹ️
Status: Passed Duration: 00:54 💡
Started: Aug 1, 2023 5:50 PM Ended: Aug 1, 2023 5:51 PM

This comment has been generated by cypress-bot as a result of this project's GitHub integration settings.

@ThomasLaPiana
Copy link
Contributor Author

ThomasLaPiana commented Aug 1, 2023

After refactoring the database check, ~10ms response time

image

@ThomasLaPiana
Copy link
Contributor Author

with a worker enabled:

image

@codecov
Copy link

codecov bot commented Aug 1, 2023

Codecov Report

Patch coverage: 100.00% and no project coverage change.

Comparison is base (8659d74) 86.95% compared to head (9d91ae1) 86.95%.
Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #3884   +/-   ##
=======================================
  Coverage   86.95%   86.95%           
=======================================
  Files         317      317           
  Lines       19398    19393    -5     
  Branches     2494     2493    -1     
=======================================
- Hits        16867    16864    -3     
+ Misses       2103     2102    -1     
+ Partials      428      427    -1     
Files Changed Coverage Δ
src/fides/api/tasks/__init__.py 77.41% <ø> (ø)
src/fides/api/api/v1/endpoints/health.py 86.95% <100.00%> (ø)
src/fides/api/db/database.py 87.87% <100.00%> (+1.96%) ⬆️
src/fides/api/util/connection_util.py 94.07% <100.00%> (ø)

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@ThomasLaPiana ThomasLaPiana marked this pull request as ready for review August 1, 2023 16:00
@ThomasLaPiana
Copy link
Contributor Author

I don't believe any of these test failures are related to these changes

Copy link
Contributor

@SteveDMurphy SteveDMurphy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work @ThomasLaPiana ! Looks good and nice use of the existing db health check in get_db_engine

@ThomasLaPiana ThomasLaPiana merged commit d47236e into main Aug 2, 2023
34 of 38 checks passed
@ThomasLaPiana ThomasLaPiana deleted the ThomasLaPiana-speed-up-health-check branch August 2, 2023 04:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

When Workers are Enabled Webserver Health Check Takes a Long Time to Respond
3 participants