Using grpcio==1.48.0 causes side effects in Python processes.

[tvalentyn]

What version of gRPC and what language are you using?

1.48.0

What operating system (Linux, Windows,...) and version?

Linux (ubuntu, rodete)

What runtime / compiler are you using (e.g. python version or version of gcc)

Python 3.7

What did you do?

A recent release grpcio==1.48.0 causes a regression, which is manifests through side-effects where calls to subprocess.Popen() can gets stuck (and hang the process).
The reasons are not obvious, likely there some sort of memory corruption is happening.
We are able to reliably repro the issue in one of our tests: apache/beam#22533.

Using git bisect + rebuilding grpc io points to 977ebbe.

Anything else we should know about your project / environment?

@chamikaramj or myself can provide more information about how to reproduce the issue by running our test.

[tvalentyn]

cc: @ctiller

[tvalentyn]

I wonder if we can consider rolling back the release while this is investigated (perhaps by pushing 1.49.0, that is effectively the same as 1.47.0 ?)

[chamikaramj]

Instructions for reproducing using the Apache Beam test: apache/beam#22533 (comment)

[gnossen]

I've yanked 1.48.0 from PyPi. This should resolve the issue in the short term. I will work with @ctiller to ensure that the root cause is found and fixed before we proceed with a patch release.

@tvalentyn Please consider adding a test to your CI that uses our prereleases. This can be done with pip install --pre grpcio. While we have subprocess tests for Python, those tests are still passing. Let me know if you need more any information on how to go about doing this.

[tvalentyn]

Yes, I filed apache/beam#22540 for improving testing.

[gnossen]

Will be resolved by #30572