Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ignore spurious PollerCompletionQueue errors in AsyncioExecutor #6492

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

maffoo
Copy link
Contributor

@maffoo maffoo commented Mar 12, 2024

When using grpc with asyncio from multiple threads, spurious PollerCompletionQueue errors are printed due to multiple event loops listening a socket to be notified of grpc events. More than one event loop may be woken when the completion queue writes a byte to the notification socket, but only one of the loops receives the data and the others raise a BlockingIOError. This doesn't actually cause a problem in the grpc still works with asyncio in multiple threads, but lots of spurious error messages are printed by the default exception handler. (See grpc/grpc#25364 for discussion of the issue.)

This configures the asyncio event loop used for grpc with an exception handler that ignores these BlockingIOErrors from the PollerCompletionQueue so that we don't spam the logs with scary-looking messages when using grpc with asyncio from multiple threads.

@maffoo maffoo requested review from wcourtney, vtomole, cduck, verult and a team as code owners March 12, 2024 05:19
@CirqBot CirqBot added the Size: XS <10 lines changed label Mar 12, 2024
@@ -40,6 +41,13 @@ def __init__(self) -> None:

@staticmethod
async def _main(loop_future: duet.AwaitableFuture) -> None:
def handle_exception(loop, context) -> None:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this function called somewhere?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, forgot to actually set this as the exception handler. Fixed.

@maffoo maffoo requested a review from verult March 12, 2024 17:51
@maffoo
Copy link
Contributor Author

maffoo commented Mar 12, 2024

Note: I filed grpc/grpc#36096 to fix these errors upstream. We'll see what comes of that, but in any case it will be a while before we can rely on that and so I think silencing these errors in our own event loops makes sense.

@github-actions github-actions bot added the Stale label Apr 20, 2024
@github-actions github-actions bot removed the Stale label May 6, 2024
@quantumlib quantumlib deleted a comment Jun 7, 2024
@github-actions github-actions bot added Stale and removed Stale labels Oct 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Size: XS <10 lines changed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants