fix: health check waits for tasks to finish before becoming unhealthy #788
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Avoid stopping compute plane due to unhealthy health check when tasks are being executed.
Description
Pollster health check has been changed to become unhealthy when there is no tasks in processing in the pollster. Shut down while not be triggered when health check becomes unhealthy, allowing the current tasks to continue. However, no new tasks will be taken from the queue storage. If the tasks does not access the storages during the time they are not available, it will not have any impact on the tasks. If a storage is missing at this point, the tasks should be set to error.
Testing
Unit tests were added. Unit tests and integration tests are passing.
Impact
No impact on infrastructure. If there are connection loss during the execution of the task and the connection comes back before the end of the task, the task should succeed.
Checklist