Bigtable: app is not closing SSL Sockets

[ramonmedeiros]

Environment details

2. OS type and version
   Linux hu3br 4.18.16-300.fc29.x86_64 Add support for namespaces #1 SMP Sat Oct 20 23:24:08 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
3. Python version and virtual environment information: python --version
   Python 3.6.7
4. google-cloud- version: pip show google-<service> or pip freeze
   Name: google-cloud-bigtable
   Version: 0.32.1
   Summary: Google Cloud Bigtable API client library
   Home-page: https://github.com/GoogleCloudPlatform/google-cloud-python
   Author: Google LLC
   Author-email: [email protected]
   License: Apache 2.0

Steps to reproduce

1. My script creates a job in dataflow and follow the progress. For this, I've created a bigtable instance api and a dataflow instance api. They are created, run into a loop, and my script finishes with the above SSL unclosed connections.

Code example

# example

Stack trace

/usr/local/lib/python3.6/dist-packages/google/cloud/bigtable_admin_v2/gapic/bigtable_instance_admin_client.py:170: PendingDeprecationWarning: The `client_config` argument is deprecated.
  PendingDeprecationWarning)
/usr/local/lib/python3.6/dist-packages/google/cloud/bigtable_admin_v2/gapic/bigtable_table_admin_client.py:171: PendingDeprecationWarning: The `client_config` argument is deprecated.
  PendingDeprecationWarning)
/usr/local/lib/python3.6/dist-packages/google/cloud/bigtable_v2/gapic/bigtable_client.py:120: PendingDeprecationWarning: The `client_config` argument is deprecated.
  PendingDeprecationWarning)
/usr/local/lib/python3.6/dist-packages/googleapiclient/_helpers.py:130: ResourceWarning: unclosed <ssl.SSLSocket fd=14, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('172.17.0.2', 54182), raddr=('74.125.140.95', 443)>
  return wrapped(*args, **kwargs)
/usr/local/lib/python3.6/dist-packages/googleapiclient/_helpers.py:130: ResourceWarning: unclosed <ssl.SSLSocket fd=19, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('172.17.0.2', 55778), raddr=('74.125.71.95', 443)>
  return wrapped(*args, **kwargs)
sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=5, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.17.0.2', 54152), raddr=('74.125.140.95', 443)>
sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=14, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('172.17.0.2', 51668), raddr=('74.125.133.95', 443)>
sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=15, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('172.17.0.2', 54188), raddr=('74.125.140.95', 443)>
sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=4, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.17.0.2', 51632), raddr=('74.125.133.95', 443)>
sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=19, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('172.17.0.2', 55780), raddr=('74.125.71.95', 443)>
sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=20, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('172.17.0.2', 55782), raddr=('74.125.71.95', 443)>

Making sure to follow these steps will guarantee the quickest resolution possible.

Thanks!

[tseaver]

@ramonmedeiros You don't show any example code, which makes it hard to see how to reproduce the issue.

My suspicion is that you are creating many instances of bigtable.Client, which is generally not the recommended pattern: you should be able to re-use the client instance across threads / requests, etc.

[sduskis]

It looks like this issue is similar to a stack overflow question: https://stackoverflow.com/questions/54371554/bigtable-grpc-memory-leak-how-to-resolve

@tseaver, is there a way to close the client and its connections?

@googleapis/yoshi-python do we have any examples of using a google-cloud-python library in Cloud Functions as per the Stack Overflow question? Specifically, how would one create a client that's reusable across requests.

[ramonmedeiros]

@tseaver sorry for not providing any code, but I'm quite sure that I'm creating too much instances. I will take a look on this.

[tseaver]

@sduskis PR #6630 added a knob to allow clients to close the gRPC channel manually:

client.table_data_client.transport.channel.close()

[ramonmedeiros]

For debugging purposes, i put a print("NEW_INSTANCE") inside of my virtual environment .venv/lib/python3.6/site-packages/google/cloud/bigtable/client.py:104 and just confirmed that I'm using only one bigtable client instance.

My code is something like:

client = bigtable.Client()
instance = client.instance(instance)

job = dataflow.templates.launch(export_bigtable_template, client)

while job not finished:
   table =  instance.table(my_logging-table)
   table.write(some_logging_stuff)
  
   table.read_rows(finished_job_pattern)

print(job_finished)

Do I need to close connection when doing operations related to write/read rows?

Also, saw that client.instance has a reload method. Is this helpful?

[ramonmedeiros]

I'm closing this issue since it happens when i call googleapiclient.discovery.build:

 /home/ramonm/git/bigtable-backups/.venv/lib/python3.6/site-packages/bigtable_backups/dataflow.py(117)__init__()
-> self.api_service = build(API_COMPONENT, API_VERSION, cache_discovery=False).projects().locations()
(Pdb) n
/home/ramonm/git/bigtable-backups/.venv/lib/python3.6/site-packages/google/auth/_default.py:66: UserWarning: Your application has authenticated using end user credentials from Google Cloud SDK. We recommend that most server applications use service accounts instead. If your application continues to use end user credentials from Cloud SDK, you might receive a "quota exceeded" or "API not enabled" error. For more information about service accounts, see https://cloud.google.com/docs/authentication/.
  warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)
/home/ramonm/git/bigtable-backups/.venv/lib/python3.6/site-packages/googleapiclient/_helpers.py:130: ResourceWarning: unclosed <ssl.SSLSocket fd=13, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('10.43.33.41', 57902), raddr=('173.194.76.95', 443)>
  return wrapped(*args, **kwargs)
> /home/ramonm/git/bigtable-backups/.venv/lib/python3.6/site-packages/bigtable_backups/dataflow.py(120)__init__()