Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Fix spurious exceptions when client closes conncetion #3925

Merged
merged 2 commits into from
Sep 25, 2018
Merged

Fix spurious exceptions when client closes conncetion #3925

merged 2 commits into from
Sep 25, 2018

Conversation

erikjohnston
Copy link
Member

If a HTTP handler throws an exception while processing a request we
automatically write a JSON error response. If the handler had already
started writing a response twisted throws an exception.

We should check for this case and simple abort the connection if there
was an error after the response had started being written.

@erikjohnston
Fix spurious exceptions when client closes conncetion
b28a7ed 
If a HTTP handler throws an exception while processing a request we
automatically write a JSON error response. If the handler had already
started writing a response twisted throws an exception.

We should check for this case and simple abort the connection if there
was an error after the response had started being written.
@erikjohnston erikjohnston requested a review from a team September 20, 2018 12:51
@erikjohnston
Copy link
Member Author

An example stack trace showing this:

2018-09-20 13:07:08,568 - synapse.http.server - 101 - ERROR - GET-94612 - Failed handle request via <function _async_render_GET at 0x7f2be6eccb18>: <XForwardedForRequest at 0x7f2bd94bdb00 method=u'GET' uri=...
--- <exception caught here> ---
  File "synapse/http/server.py", line 81, in wrapped_request_handler
    
  File "synapse/rest/media/v1/thumbnail_resource.py", line 71, in _async_render_GET
    
  File "synapse/rest/media/v1/thumbnail_resource.py", line 121, in _respond_local_thumbnail
    
  File "synapse/rest/media/v1/_base.py", line 146, in respond_with_responder
    
exceptions.Exception: Consumer asked us to stop producing
2018-09-20 13:07:08,569 - synapse.http.site - 147 - ERROR - GET-94612 - Asynchronous messge handler raised an uncaught exception
Capture point (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/matrix/synapse/synapse/app/media_repository.py", line 181, in <module>
    start(sys.argv[1:])
  File "/home/matrix/synapse/synapse/app/media_repository.py", line 176, in start
    _base.start_worker_reactor("synapse-media-repository", config)
  File "synapse/app/_base.py", line 56, in start_worker_reactor
  File "synapse/app/_base.py", line 115, in start_reactor
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/daemonize.py", line 245, in start
    self.action(*privileged_action_result)
  File "synapse/app/_base.py", line 104, in run
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/base.py", line 1261, in run
    self.mainLoop()
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/base.py", line 1273, in mainLoop
    self.doIteration(t)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/epollreactor.py", line 235, in doPoll
    log.callWithLogger(selectable, _drdw, selectable, fd, event)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/python/log.py", line 103, in callWithLogger
    return callWithContext({"system": lp}, func, *args, **kw)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/python/log.py", line 86, in callWithContext
    return context.call({ILogContext: newCtx}, func, *args, **kw)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/python/context.py", line 122, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/python/context.py", line 85, in callWithContext
    return func(*args,**kw)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/posixbase.py", line 627, in _doReadOrWrite
    self._disconnectSelectable(selectable, why, inRead)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/posixbase.py", line 255, in _disconnectSelectable
    selectable.connectionLost(f)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/tcp.py", line 321, in connectionLost
    abstract.FileDescriptor.connectionLost(self, reason)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/abstract.py", line 205, in connectionLost
    self.producer.stopProducing()
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/web/http.py", line 2510, in stopProducing
    self._requestProducer.stopProducing()
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/_producer_helpers.py", line 125, in stopProducing
    self._producer.stopProducing()
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/protocols/basic.py", line 952, in stopProducing
    Exception("Consumer asked us to stop producing"))
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 501, in errback
    self._startRunCallbacks(fail)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 568, in _startRunCallbacks
    self._runCallbacks()
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1475, in gotResult
    _inlineCallbacks(r, g, status)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1464, in _inlineCallbacks
    status.deferred.errback()
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 501, in errback
    self._startRunCallbacks(fail)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 568, in _startRunCallbacks
    self._runCallbacks()
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1475, in gotResult
    _inlineCallbacks(r, g, status)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1464, in _inlineCallbacks
    status.deferred.errback()
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 501, in errback
    self._startRunCallbacks(fail)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 568, in _startRunCallbacks
    self._runCallbacks()
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1475, in gotResult
    _inlineCallbacks(r, g, status)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1464, in _inlineCallbacks
    status.deferred.errback()
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 501, in errback
    self._startRunCallbacks(fail)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 568, in _startRunCallbacks
    self._runCallbacks()
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1475, in gotResult
    _inlineCallbacks(r, g, status)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1464, in _inlineCallbacks
    status.deferred.errback()
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 501, in errback
    self._startRunCallbacks(fail)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 568, in _startRunCallbacks
    self._runCallbacks()
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1475, in gotResult
    _inlineCallbacks(r, g, status)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/python/failure.py", line 491, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "synapse/http/server.py", line 190, in wrapped_async_request_handler
  File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
Traceback (most recent call last):
  File "synapse/http/site.py", line 144, in processing
  File "synapse/http/server.py", line 190, in wrapped_async_request_handler
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/python/failure.py", line 491, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "synapse/http/server.py", line 111, in wrapped_request_handler
  File "synapse/http/server.py", line 409, in respond_with_json
  File "synapse/http/server.py", line 439, in respond_with_json_bytes
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/web/static.py", line 765, in start
    self.request.registerProducer(self, False)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/web/http.py", line 954, in registerProducer
    "unregistered" % (producer, self.producer))
ValueError: registering producer <twisted.web.static.NoRangeStaticProducer object at 0x7f2bbfe24350> before previous one (<twisted.protocols.basic.FileSender instance at 0x7f2b43315d88>) was unregistered

richvdh
richvdh approved these changes Sep 25, 2018
View reviewed changes
Copy link
Member

@richvdh richvdh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks plausible

@richvdh richvdh merged commit 94f7bef into develop Sep 25, 2018
@richvdh richvdh mentioned this pull request Oct 3, 2018
Closed
@erikjohnston erikjohnston deleted the erikj/fix_producers_unregistered branch December 12, 2018 15:45
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@richvdh richvdh richvdh approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@erikjohnston @richvdh