Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python Exceptions during async batch write are not propagated #7401

Closed
cmnbroad opened this issue Aug 10, 2021 · 1 comment
Closed

Python Exceptions during async batch write are not propagated #7401

cmnbroad opened this issue Aug 10, 2021 · 1 comment
Assignees

Comments

@cmnbroad
Copy link
Collaborator

@kcibul reports that if the CNNScoreVariants python code throws an exception during async batch processing, the GATK tool hangs (specifically, it was happening when GATK was sending a . for a missing annotation, and the python code was trying to interpret that as a number and blowing up).

It looks like this happens because StreamingPythonScriptExecutor::waitForPreviousBatchCompletion waits for the async write thread Future to complete first, before checking the fifo for an ACK/NCK (which is when the exception would be propagated). If the async write thread is blocked because the fifo is full because the python code isn't retrieving data because an exception was thrown, the java side will hang waiting for the Future complete.

The solution is to reverse the order of the waitForPreviousBatchCompletion checking (ack first, then validate that the async write Future completes). There is a branch with a test and a fix for the StreamingPythonExecutor, and a separate branch with a test for CNNScoreVariants that also has the executor fix. I need to verify that the CNNSCoreVariants test actually fails without the fix, and then this can be turned into a PR, which I'll do when I return from vacation.

@droazen
Copy link
Contributor

droazen commented Aug 10, 2021

@kcibul See the potential fix in the branches @cmnbroad linked to above -- it would help us to know whether they resolve your issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants