Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Attempt to join heartbeat thread during close() #1735

Merged
merged 1 commit into from
Mar 14, 2019

Conversation

dpkp
Copy link
Owner

@dpkp dpkp commented Mar 13, 2019

Fixes #1666 . Underlying issue here is a race on consumer.close() between the client, the connections/sockets, and the heartbeat thread. Although the heartbeat thread is signaled to close, we do not block for it. So when we go on to close the client and its underlying connections, if the heartbeat is still doing work it can cause errors/crashes if it attempts to access the now closed objects (selectors and/or sockets, primarily).

This PR adds a blocking thread join to the heartbeat close. This may cause some additional blocking time on consumer.close() while the heartbeat thread finishes. But it should be small in average case and in the worst case will be no longer than the heartbeat_timeout_ms (though if we timeout the join, race errors may still occur).


This change is Reviewable

@jeffwidman jeffwidman merged commit 921c553 into master Mar 14, 2019
@jeffwidman jeffwidman deleted the join_heartbeat_thread_on_close branch March 14, 2019 01:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

ValueError: I/O operation on closed epoll object with kafka-python >= 1.4.2
2 participants