Transaction is left open when error happens in Datastore API

[dmho418]

This one is hard to reproduce, but I noticed that when using a transaction with a with statement and an error happens inside the transaction itself (a 500 error for example), it appears that the transaction is left open.

with self.client.transaction():
  # do stuff

# create entities, put, put_multi, etc

Exception:

Traceback (most recent call last):
  File "my_client.py", line 190, in do_stuff
    with self.client.transaction():
  File "site-packages/gcloud/datastore/batch.py", line 275, in __enter__
    self.begin()
  File "site-packages/gcloud/datastore/transaction.py", line 131, in begin
    self._id = self.connection.begin_transaction(self.project)
  File "site-packages/gcloud/datastore/connection.py", line 307, in begin_transaction
    _datastore_pb2.BeginTransactionResponse)
  File "site-packages/gcloud/datastore/connection.py", line 124, in _rpc
    data=request_pb.SerializeToString())
  File "site-packages/gcloud/datastore/connection.py", line 98, in _request
    raise make_exception(headers, error_status.message, use_json=False)
InternalServerError: 500 internal error.

After the exception, any operations to the Datastore using the same client have no effect, and no error is reported.
So my guess is that for some reason the transaction was left open and the changes are never committed.

[tseaver]

Thanks for the report!

[dhermes]

I was able to reproduce this, digging in now.

[dhermes]

After just faking the 500

diff --git a/google/cloud/datastore/transaction.py b/google/cloud/datastore/transaction.py
index 66ef8bb..0318643 100644
--- a/google/cloud/datastore/transaction.py
+++ b/google/cloud/datastore/transaction.py
@@ -128,6 +128,10 @@ class Transaction(Batch):
         :raises: :class:`ValueError` if the transaction has already begun.
         """
         super(Transaction, self).begin()
+        import httplib2
+        from google.cloud.exceptions import make_exception
+        resp = httplib2.Response({'status': 500})
+        raise make_exception(resp, {'error': {'message': 'internal error'}})
         self._id = self.connection.begin_transaction(self.project)

     def rollback(self):

I ran the following:

>>> from google.cloud import datastore
>>> client = datastore.Client()
>>> q = client.query(kind='Foo')
>>> client.connection._datastore_api
<google.cloud.datastore.connection._DatastoreAPIOverGRPC object at 0x7f4997d6c390>
>>> list(q.fetch())
[]
>>> k = client.key('Foo', 1234)
>>> e = datastore.Entity(key=k)
>>> e['food'] = 'noms'
>>> client.put(e)
>>> list(q.fetch())
[<Entity[{'kind': u'Foo', 'id': 1234L}] {u'food': 'noms'}>]
>>> with client.transaction():
...     e['goodbye'] = 123
...     client.put(e)
... 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".tox/py27/local/lib/python2.7/site-packages/google/cloud/datastore/batch.py", line 275, in __enter__
    self.begin()
  File ".tox/py27/local/lib/python2.7/site-packages/google/cloud/datastore/transaction.py", line 134, in begin
    raise make_exception(resp, {'error': {'message': 'internal error'}})
google.cloud.exceptions.InternalServerError: 500 internal error
>>> list(q.fetch())
[<Entity[{'kind': u'Foo', 'id': 1234L}] {u'food': 'noms'}>]
>>> k2 = client.key('Foo', 123456)
>>> e2 = datastore.Entity(key=k2)
>>> e2['other'] = 'brother'
>>> client.put(e2)
>>> 
>>> list(q.fetch())
[<Entity[{'kind': u'Foo', 'id': 1234L}] {u'food': 'noms'}>]
>>>
>>> # Just use a FRESH client
>>> client = datastore.Client()
>>> client.put(e2)
>>> list(q.fetch())
[<Entity[{'kind': u'Foo', 'id': 1234L}] {u'food': 'noms'}>, <Entity[{'kind': u'Foo', 'id': 123456L}] {u'other': 'brother'}>]

The issue is certainly in Batch.__enter__, hope to have it sorted out soon.

[dhermes]

@dmho418 It occurred to me while making the fix #2303 that this error would really only be perceivable if you were in the interpreter or some other situation where the uncaught exception didn't hose your entire application. Are you catching errors and recovering somehow? You may want to look into how much "recovery" is done.

---

@tseaver I wanted to make sure that a failure in __enter__ means __exit__ is never called AND that the failure in __enter__ bubbles up to the user:

>>> class A(object):
...     def __enter__(self):
...         print('__enter__ called')
...         raise ValueError('Bad')
...         return self
...     def __exit__(self, exc_type, exc_val, exc_tb):
...         msg = '__exit__ called: %r, %r, %r' % (exc_type, exc_val, exc_tb)
...         print(msg)
... 
>>> 
>>> a = A()
>>> with a:
...     print('I made it inside the context manager')
... 
__enter__ called
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 4, in __enter__
ValueError: Bad

[dmho418]

@dhermes you got it, I'm catching that exception since I don't want the application to crash. Ideally, the client should remain valid after the exception, right?

[dhermes]

Yeah, #2303 resolves that issue. Thanks for bringing it to our attention.