Connection reset by peer when using pygsheets on Google App Engine web app.

[lck3000]

Hi @nithinmurali and dev team,

I think I found a bug, or at least, an opportunity of improvement for the lib.
When deploying and using a little web app that uses pygsheets on Google App Engine, it raises an Error after some transactions: ConnectionResetError: [Errno 104] Connection reset by peer

So, after reading a bit on Google App Engine and Google Cloud docs, I found that they migrated from using httplib2 to requestsas httplib2 is not threadsafe. See this and this. At the end of the topic they mentioned that all google apis should stop using httplib2 and migrate to requests.

Then, I found on the error log that, in fact, pygsheets makes intensive use of the mentioned library. The question is: Is there a possibility to migrate pygsheets to use Requests?

Additional Information:
Digging on the code, I found that pygsheets.authorize and pygsheets.Client do the authorization job with:

from google_auth_httplib2 import AuthorizedHttp
...
class Client(object):
...
def __init__(self, credentials, retries=3, http=None, check=True):
        self.oauth = credentials
        self.logger = logging.getLogger(__name__)

        http = AuthorizedHttp(credentials, http=http) #Here
        data_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), "data")

        self.sheet = SheetAPIWrapper(http, data_path, retries=retries, check=check)
        self.drive = DriveAPIWrapper(http, data_path)

which can be -somehow- replaced by:

from google.auth.transport.requests import AuthorizedSession
from google.oauth2 import service_account  #or another kind of authentication (oauth2, etc)
...
        credentials = service_account.Credentials.from_service_account_file(
                CREDENTIALS,
                scopes=('https://www.googleapis.com/auth/spreadsheets', 
                'https://www.googleapis.com/auth/drive'))
        http = AuthorizedSession(credentials)

because AuthorizedSession is a requests.Session object which has a request method besides predefined methods such as get, post etc.

But, this may imply that every request in the code should be adapted to pass the correct parameters to AuthorizedSession.

Another alternative may be to use the http parameter of pygsheets.authorize and pygsheets.Client to use Requests or google.auth.transport.requests when a object of these ones is passed.

I'll try to reproduce my suggestions and put them on pull requests as candidates but now I'm looking for a workaround. 😢

Finally, sorry if my english is not so good or if there's a typo let, please let me know. 😞
This is the first issue I write on Github ever, so if there's a mistake on the structure of it, please let me know again.

Thanks in advance.

To Reproduce

• Make venv

• Install Flask (pip install flask), Google Apis (pip install --upgrade google-api-python-client), pygsheets

• Point to some spreadsheets.

• Use it on deployed GAE server. (The error is raised when inserting/updating cells and rows.)

Error log:

Traceback (most recent call last):
  File "/env/lib/python3.7/site-packages/flask/app.py", line 2446, in wsgi_app
    response = self.full_dispatch_request()
  File "/env/lib/python3.7/site-packages/flask/app.py", line 1951, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/env/lib/python3.7/site-packages/flask_restful/__init__.py", line 269, in error_router
    return original_handler(e)
  File "/env/lib/python3.7/site-packages/flask/app.py", line 1820, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/env/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/env/lib/python3.7/site-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/env/lib/python3.7/site-packages/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/env/lib/python3.7/site-packages/flask_login/utils.py", line 261, in decorated_view
    return func(*args, **kwargs)
  File "/srv/portal/cartrix/routes.py", line 223, in addCartaEntrada
    CartasEntrada.add(data)
  File "/srv/portal/cartrix/model_pyg.py", line 81, in add
    cls.worksheet.refresh()
  File "/env/lib/python3.7/site-packages/pygsheets/worksheet.py", line 160, in refresh
    jsonsheet = self.client.open_as_json(self.spreadsheet.id)
  File "/env/lib/python3.7/site-packages/pygsheets/client.py", line 179, in open_as_json
    includeGridData=False)
  File "/env/lib/python3.7/site-packages/pygsheets/sheet.py", line 144, in get
    return self._execute_requests(self.service.spreadsheets().get(spreadsheetId=spreadsheet_id, **kwargs))
  File "/env/lib/python3.7/site-packages/pygsheets/sheet.py", line 357, in _execute_requests
    response = request.execute(num_retries=self.retries)
  File "/env/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/env/lib/python3.7/site-packages/googleapiclient/http.py", line 851, in execute
    method=str(self.method), body=self.body, headers=self.headers)
  File "/env/lib/python3.7/site-packages/googleapiclient/http.py", line 165, in _retry_request
    resp, content = http.request(uri, method, *args, **kwargs)
  File "/env/lib/python3.7/site-packages/google_auth_httplib2.py", line 187, in request
    self._request, method, uri, request_headers)
  File "/env/lib/python3.7/site-packages/google/auth/credentials.py", line 122, in before_request
    self.refresh(request)
  File "/env/lib/python3.7/site-packages/google/oauth2/service_account.py", line 322, in refresh
    request, self._token_uri, assertion)
  File "/env/lib/python3.7/site-packages/google/oauth2/_client.py", line 145, in jwt_grant
    response_data = _token_endpoint_request(request, token_uri, body)
  File "/env/lib/python3.7/site-packages/google/oauth2/_client.py", line 106, in _token_endpoint_request
    method='POST', url=token_uri, headers=headers, body=body)
  File "/env/lib/python3.7/site-packages/google_auth_httplib2.py", line 116, in __call__
    url, method=method, body=body, headers=headers, **kwargs)
  File "/env/lib/python3.7/site-packages/httplib2/__init__.py", line 1953, in request
    cachekey,
  File "/env/lib/python3.7/site-packages/httplib2/__init__.py", line 1618, in _request
    conn, request_uri, method, body, headers
  File "/env/lib/python3.7/site-packages/httplib2/__init__.py", line 1556, in _conn_request
    response = conn.getresponse()
  File "/opt/python3.7/lib/python3.7/http/client.py", line 1321, in getresponse
    response.begin()
  File "/opt/python3.7/lib/python3.7/http/client.py", line 296, in begin
    version, status, reason = self._read_status()
  File "/opt/python3.7/lib/python3.7/http/client.py", line 257, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/opt/python3.7/lib/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
  File "/opt/python3.7/lib/python3.7/ssl.py", line 1052, in recv_into
    return self.read(nbytes, buffer)
  File "/opt/python3.7/lib/python3.7/ssl.py", line 911, in read
    return self._sslobj.read(len, buffer)
ConnectionResetError: [Errno 104] Connection reset by peer

System Information

• OS: Windows 10 64 bits.
• Pygsheets version : 2.0.2
• Pygsheets installed from pypi
• Python version: 3.7.2
• Google App Engine standard

[nithinmurali]

will it help if you pass a custom http object like httplib2shim ? you can pass it on authorize method or whlie client init.

[lck3000]

Thx for comment.
Yeah. indeed I'm already doing that (using the httplib2shim monkeypatch) but, from httplib2shim (link):

This library is intended to help existing legacy libraries (and their users) to migrate away from httplib2. It is not intended to be a general purpose replacement for httplib2

and

Unsupported Features

• Arguments to the Http constructor will be accepted, but may not make a difference. For instance, ca_certs will have no effect. Instead, pass a urllib3.Pool instance http = httplib2shim.Http(pool=my_pool).

So, it'd be great to switch to a library that may be sustainable in the long term instead of using a deprecated lib such as httplib2.
I personally think that pygsheets offers several benefits over other libraries, so I wanted to give a little contribution in order to improve it.

Thx again for replying. 😄

[nithinmurali]

Cool, Will add the library change to pipeline.

[TasseDeCafe]

will it help if you pass a custom http object like httplib2shim ? you can pass it on authorize method or whlie client init.

I'm having the same issue, but I don't really understand that answer. Could you please give me more details? Thank you.

[richardARPANET]

@TasseDeCafe

Like this:

pip install httplib2shim

import httplib2shim
from google_auth_httplib2 import AuthorizedHttp

http = httplib2shim.Http()
client = pygsheets.authorize(custom_credentials=credentials, http=http)