Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

/state requests can fail with JSON response exceeded max size 104857600 #12876

Closed
richvdh opened this issue May 25, 2022 · 1 comment · Fixed by #12877
Closed

/state requests can fail with JSON response exceeded max size 104857600 #12876

richvdh opened this issue May 25, 2022 · 1 comment · Fixed by #12877
Milestone
Q2 2022 ─ Faster ...

Comments

@richvdh
Copy link
Member

richvdh commented May 25, 2022 •
edited
Loading

the faster joins work uses /federation/v1/state requests (they would rarely be used previously), so this has come to a head

2022-05-25 13:07:49,271 - synapse.http.matrixfederationclient - 245 - WARNING - sync_partial_state_room-0-$k09nja_-OhO47WmvZHM24HZFuz2YHqNyj3wp-YsUmcg-$foEN5-2T3Z_ADBZbTr_avOBmasj9SoMC1LtUpa8Yf7k - {GET-O-525} [matrix.org] JSON response exceeded max size 104857600 - GET matrix://matrix.org/_matrix/federation/v1/state/%21OGEhHVWSdvArJzumhm%3Amatrix.org?event_id=%24foEN5-2T3Z_ADBZbTr_avOBmasj9SoMC1LtUpa8Yf7k
2022-05-25 13:07:49,271 - synapse.handlers.federation_event - 889 - WARNING - sync_partial_state_room-0-$k09nja_-OhO47WmvZHM24HZFuz2YHqNyj3wp-YsUmcg - Error attempting to resolve state at missing prev_events
2022-05-25 13:07:49,271 - synapse.handlers.federation_event - 889 - WARNING - sync_partial_state_room-0-$k09nja_-OhO47WmvZHM24HZFuz2YHqNyj3wp-YsUmcg - Error attempting to resolve state at missing prev_events
Capture point (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/synapse/src/synapse/synapse/app/homeserver.py", line 424, in main
    run(hs)
  File "/home/synapse/src/synapse/synapse/app/homeserver.py", line 410, in run
    logger=logger,
  File "/home/synapse/src/synapse/synapse/app/_base.py", line 187, in start_reactor
    run()
  File "/home/synapse/src/synapse/synapse/app/_base.py", line 169, in run
    run_command()
  File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.7/lib/python3.7/site-packages/twisted/internet/base.py", line 1315, in run
    self.mainLoop()
  File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.7/lib/python3.7/site-packages/twisted/internet/base.py", line 1328, in mainLoop
    reactorBaseSelf.doIteration(t)
  File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.7/lib/python3.7/site-packages/twisted/internet/epollreactor.py", line 244, in doPoll
    log.callWithLogger(selectable, _drdw, selectable, fd, event)
  File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.7/lib/python3.7/site-packages/twisted/python/log.py", line 96, in callWithLogger
    return callWithContext({"system": lp}, func, *args, **kw)
  File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.7/lib/python3.7/site-packages/twisted/python/log.py", line 80, in callWithContext
    return context.call({ILogContext: newCtx}, func, *args, **kw)
  File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.7/lib/python3.7/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.7/lib/python3.7/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
  File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.7/lib/python3.7/site-packages/twisted/internet/posixbase.py", line 683, in _doReadOrWrite
    why = selectable.doRead()
  File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.7/lib/python3.7/site-packages/twisted/internet/tcp.py", line 248, in doRead
    return self._dataReceived(data)
  File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.7/lib/python3.7/site-packages/twisted/internet/tcp.py", line 253, in _dataReceived
    rval = self.protocol.dataReceived(data)
  File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.7/lib/python3.7/site-packages/twisted/internet/endpoints.py", line 147, in dataReceived
    return self._wrappedProtocol.dataReceived(data)
  File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.7/lib/python3.7/site-packages/twisted/protocols/tls.py", line 330, in dataReceived
    self._flushReceiveBIO()
  File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.7/lib/python3.7/site-packages/twisted/protocols/tls.py", line 296, in _flushReceiveBIO
    ProtocolWrapper.dataReceived(self, bytes)
  File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.7/lib/python3.7/site-packages/twisted/protocols/policies.py", line 110, in dataReceived
    self.wrappedProtocol.dataReceived(data)
  File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.7/lib/python3.7/site-packages/twisted/web/_newclient.py", line 1651, in dataReceived
    self._parser.dataReceived(bytes)
  File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.7/lib/python3.7/site-packages/twisted/web/_newclient.py", line 384, in dataReceived
    HTTPParser.dataReceived(self, data)
  File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.7/lib/python3.7/site-packages/twisted/protocols/basic.py", line 555, in dataReceived
    why = self.rawDataReceived(data)
  File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.7/lib/python3.7/site-packages/twisted/web/_newclient.py", line 302, in rawDataReceived
    self.bodyDecoder.dataReceived(data)
  File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.7/lib/python3.7/site-packages/twisted/web/http.py", line 2060, in dataReceived
    goOn = getattr(self, "_dataReceived_" + self.state)()
  File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.7/lib/python3.7/site-packages/twisted/web/http.py", line 2038, in _dataReceived_BODY
    self.dataCallback(chunk)
  File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.7/lib/python3.7/site-packages/twisted/web/_newclient.py", line 1043, in dispatcher
    return func(*args, **kwargs)
  File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.7/lib/python3.7/site-packages/twisted/web/_newclient.py", line 1242, in _bodyDataReceived_CONNECTED
    self._bodyProtocol.dataReceived(data)
  File "/home/synapse/src/synapse/synapse/http/client.py", line 863, in dataReceived
    self.deferred.errback(BodyExceededMaxSize())
  File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.7/lib/python3.7/site-packages/twisted/internet/defer.py", line 700, in errback
    self._startRunCallbacks(fail)
  File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.7/lib/python3.7/site-packages/twisted/internet/defer.py", line 763, in _startRunCallbacks
    self._runCallbacks()
  File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.7/lib/python3.7/site-packages/twisted/internet/defer.py", line 858, in _runCallbacks
    current.result, *args, **kwargs
  File "/home/synapse/src/synapse/synapse/util/async_helpers.py", line 664, in failure_cb
    new_d.errback(val)
  File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.7/lib/python3.7/site-packages/twisted/internet/defer.py", line 700, in errback
    self._startRunCallbacks(fail)
  File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.7/lib/python3.7/site-packages/twisted/internet/defer.py", line 763, in _startRunCallbacks
    self._runCallbacks()
  File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.7/lib/python3.7/site-packages/twisted/internet/defer.py", line 858, in _runCallbacks
    current.result, *args, **kwargs
  File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.7/lib/python3.7/site-packages/twisted/internet/defer.py", line 1750, in gotResult
    current_context.run(_inlineCallbacks, r, gen, status)
  File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.7/lib/python3.7/site-packages/twisted/internet/defer.py", line 1657, in _inlineCallbacks
    cast(Failure, result).throwExceptionIntoGenerator, gen
  File "/home/synapse/poetry-env/matrix-synapse-o-qzlfnx-py3.7/lib/python3.7/site-packages/twisted/python/failure.py", line 514, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/home/synapse/src/synapse/synapse/metrics/background_process_metrics.py", line 247, in run
    desc,
  File "/usr/lib/python3.7/logging/__init__.py", line 1418, in exception
    self.error(msg, *args, exc_info=exc_info, **kwargs)
  File "/usr/lib/python3.7/logging/__init__.py", line 1412, in error
    self._log(ERROR, msg, args, **kwargs)
  File "/usr/lib/python3.7/logging/__init__.py", line 1519, in _log
    self.handle(record)
  File "/usr/lib/python3.7/logging/__init__.py", line 1529, in handle
    self.callHandlers(record)
  File "/usr/lib/python3.7/logging/__init__.py", line 1591, in callHandlers
    hdlr.handle(record)
  File "/usr/lib/python3.7/logging/__init__.py", line 905, in handle
    self.emit(record)
  File "/usr/lib/python3.7/logging/handlers.py", line 1234, in emit
    self.flush()
  File "/usr/lib/python3.7/logging/handlers.py", line 1310, in flush
    self.target.handle(record)
  File "/usr/lib/python3.7/logging/__init__.py", line 905, in handle
    self.emit(record)
  File "/usr/lib/python3.7/logging/handlers.py", line 68, in emit
    if self.shouldRollover(record):
  File "/usr/lib/python3.7/logging/handlers.py", line 184, in shouldRollover
    msg = "%s\n" % self.format(record)
  File "/usr/lib/python3.7/logging/__init__.py", line 880, in format
    return fmt.format(record)
  File "/usr/lib/python3.7/logging/__init__.py", line 627, in format
    record.exc_text = self.formatException(record.exc_info)
  File "/home/synapse/src/synapse/synapse/logging/formatter.py", line 50, in formatException
    traceback.print_stack(tb.tb_frame.f_back, None, sio)
Traceback (most recent call last):
  File "/home/synapse/src/synapse/synapse/http/matrixfederationclient.py", line 234, in _handle_response
    length = await make_deferred_yieldable(d)
synapse.http.client.BodyExceededMaxSize

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/synapse/src/synapse/synapse/handlers/federation_event.py", line 853, in _resolve_state_at_missing_prevs
    dest, room_id, p
  File "/home/synapse/src/synapse/synapse/handlers/federation_event.py", line 970, in _get_state_after_missing_prev_event
    await self._get_state_and_persist(destination, room_id, event_id)
  File "/home/synapse/src/synapse/synapse/handlers/federation_event.py", line 1041, in _get_state_and_persist
    destination, room_id, event_id=event_id, room_version=room_version
  File "/home/synapse/src/synapse/synapse/federation/federation_client.py", line 448, in get_room_state
    event_id,
  File "/home/synapse/src/synapse/synapse/federation/transport/client.py", line 110, in get_room_state
    parser=_StateParser(room_version),
  File "/home/synapse/src/synapse/synapse/http/matrixfederationclient.py", line 1093, in get_json
    max_response_size=max_response_size,
  File "/home/synapse/src/synapse/synapse/http/matrixfederationclient.py", line 247, in _handle_response
    raise RequestSendFailed(e, can_retry=False) from e
synapse.api.errors.RequestSendFailed: Failed to send request: BodyExceededMaxSize:
@richvdh
Copy link
Member Author

richvdh commented May 25, 2022

Since #12013, this could in theory have affected any backfill-style operation.

@richvdh richvdh mentioned this issue May 25, 2022
Merged
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
Q2 2022 ─ Faster joins phase 2: corre...
Development

Successfully merging a pull request may close this issue.

Allow bigger responses to /federation/v1/state matrix-org/synapse
1 participant
@richvdh