Raiden node crashed w/ MatrixRequestError "No known servers"

[nephix]

Problem Definition

1. I've started a raiden node via

$ docker run --rm -it \
    -p 8545:8545 \
    -p 5011:5011 \
    --mount src=/Users/nephix/Library/Ethereum/keystore/,target=/keystore,type=bind \
    raidennetwork/raiden:v0.200.0-rc2 \
    --keystore-path /keystore \
    --network-id 5 \
    --environment-type development \
    --eth-rpc-endpoint http://10.104.6.13:8545 \
    --accept-disclaimer \
    --matrix-server=https://transport04.raiden.network \
    --pathfinding-service-address https://pfs-goerli-with-fee.services-test.raiden.network \
    --api-address "http://0.0.0.0:5011"

2. Did a direct transfer to that instance from the Light Client
3. Received an exception:

Welcome to Raiden, version 0.200.0rc2!
----------------------------------------------------------------------
| This is an Alpha version of experimental open source software      |
| released as a test version under an MIT license and may contain    |
| errors and/or bugs. No guarantee or representations whatsoever is  |
| made regarding its suitability (or its use) for any purpose or     |
| regarding its compliance with any applicable laws and regulations. |
| Use of the software is at your own risk and discretion and by      |
| using the software you acknowledge that you have read this         |
| disclaimer, understand its contents, assume all risk related       |
| thereto and hereby release, waive, discharge and covenant not to   |
| sue Brainbot Labs Establishment or any officers, employees or      |
| affiliates from and for any direct or indirect liability resulting |
| from the use of the software as permissible by applicable laws and |
| regulations.                                                       |
|                                                                    |
| Privacy Warning: Please be aware, that by using the Raiden Client, |
| among others, your Ethereum address, channels, channel deposits,   |
| settlements and the Ethereum address of your channel counterparty  |
| will be stored on the Ethereum chain, i.e. on servers of Ethereum  |
| node operators and ergo are to a certain extent publicly available.|
| The same might also be stored on systems of parties running Raiden |
| nodes connected to the same token network. Data present in the     |
| Ethereum chain is very unlikely to be able to be changed, removed  |
| or deleted from the public arena.                                  |
|                                                                    |
| Also be aware, that data on individual Raiden token transfers will |
| be made available via the Matrix protocol to the recipient,        |
| intermediating nodes of a specific transfer as well as to the      |
| Matrix server operators.                                           |
----------------------------------------------------------------------
2019-12-13 07:16:05.681277 [info     ] Starting Raiden                [raiden.ui.runners] architecture=x86_64 distribution=source python_implementation=CPython python_version=3.7.5 raiden=0.200.0rc2 raiden_db_version=23 system=Linux 64bit 4.9.184-linuxkit
The following accounts were found in your machine:

[  0] - 0xd89F35A9274E4d8Bb95312A1a497431CcBF73478
[  1] - 0x385dB1Ed9509FF184ff9E34a06b2D93daFdBAc0b

Select one of them by index to continue: 0
Enter the password to unlock 0xd89F35A9274E4d8Bb95312A1a497431CcBF73478:
Raiden is running in development mode
Checking if the ethereum node is synchronized
You have chosen the Pathfinding Service at https://pfs-goerli-with-fee.services-test.raiden.network.
Operator: John Doe, running version: 0.5.0, chain_id: 5.
Fees will be paid to 0x94DEe8e391410A9ebbA791B187df2d993212c849. Each request costs 1e-16 RDN.
Message from the Pathfinding Service:
This is your favorite pfs for token network registry 0x62436bbadB629498107278486C2A367FfdFEb2f3
2019-12-13 07:16:17.234770 [info     ] Using Pathfinding Service      [raiden.network.pathfinding] pfs_info=PFSInfo(url='https://pfs-goerli-with-fee.services-test.raiden.network', price=100, chain_id=5, token_network_registry_address=b'bCk\xba\xdbb\x94\x98\x10rxHl*6\x7f\xfd\xfe\xb2\xf3', payment_address=b'\x94\xde\xe8\xe3\x91A\n\x9e\xbb\xa7\x91\xb1\x87\xdf-\x992\x12\xc8I', message='This is your favorite pfs for token network registry 0x62436bbadB629498107278486C2A367FfdFEb2f3', operator='John Doe', version='0.5.0')

You are connected to the 'goerli' network and the DB path is: /root/.raiden/node_d89f35a9/netid_5/network_62436bba/v23_log.db
2019-12-13 07:16:17.799003 [info     ] Using Matrix server            [raiden.network.transport.matrix.utils] average_rtt=0 server_ident=Synapse/1.5.1 server_url=https://transport04.raiden.network
This is the first time Raiden is being used with this address. Processing all the events may take some time. Please wait ...
2019-12-13 07:16:18.375835 [info     ] Synchronizing blockchain events [raiden.blockchain.events] blocks_left=366033
2019-12-13 07:16:47.945972 [info     ] Synchronizing blockchain events [raiden.blockchain.events] blocks_left=304678 blocks_per_second=2074.9615144469662
2019-12-13 07:17:01.817426 [info     ] Synchronizing blockchain events [raiden.blockchain.events] blocks_left=258875 blocks_per_second=3301.96337165763
2019-12-13 07:17:06.817636 [info     ] Synchronizing blockchain events [raiden.blockchain.events] blocks_left=258875 blocks_per_second=0.0
2019-12-13 07:17:11.818077 [info     ] Synchronizing blockchain events [raiden.blockchain.events] blocks_left=258875 blocks_per_second=0.0
2019-12-13 07:17:16.819333 [info     ] Synchronizing blockchain events [raiden.blockchain.events] blocks_left=258875 blocks_per_second=0.0
2019-12-13 07:17:21.819992 [info     ] Synchronizing blockchain events [raiden.blockchain.events] blocks_left=258875 blocks_per_second=0.0
2019-12-13 07:17:26.820230 [info     ] Synchronizing blockchain events [raiden.blockchain.events] blocks_left=258875 blocks_per_second=0.0
2019-12-13 07:17:32.928411 [info     ] Synchronizing blockchain events [raiden.blockchain.events] blocks_left=130644 blocks_per_second=20993.43025194703
2019-12-13 07:17:53.281640 [info     ] Updating channel fees          [raiden.raiden_service] cap_mediation_fees=True channel=CanonicalIdentifier(chain_identifier=5, token_network_address=b'O\xfe0\xa44,\x10\x89\xfe\xd7\x13\x925o\x94\x03r>\x92\\', channel_identifier=59) flat_fee=0 proportional_fee=4000 proportional_imbalance_fee=3000
2019-12-13 07:18:02.995619 [info     ] Raiden started                 [raiden.app] node=0xd89F35A9274E4d8Bb95312A1a497431CcBF73478
The Raiden API RPC server is now running at http://0.0.0.0:5011/.

See the Raiden documentation for all available endpoints at
http://raiden-network.readthedocs.io/en/stable/rest_api.html
WARNING
Your account's RDN balance of 0.0 is below the minimum threshold. Provided that you have either a monitoring service or a path finding service activated, your node is not going to be able to pay those services which may lead to denial of service or loss of funds.
Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 716, in gevent._greenlet.Greenlet.run
  File "/opt/venv/lib/python3.7/site-packages/raiden/network/transport/matrix/transport.py", line 430, in _process_queued_invites
    self._handle_invite(room_id, state)
  File "/opt/venv/lib/python3.7/site-packages/raiden/network/transport/matrix/transport.py", line 773, in _handle_invite
    raise last_ex  # re-raise if couldn't succeed in retries
  File "/opt/venv/lib/python3.7/site-packages/raiden/network/transport/matrix/transport.py", line 763, in _handle_invite
    room = self._client.join_room(room_id)
  File "/opt/venv/lib/python3.7/site-packages/matrix_client/client.py", line 320, in join_room
    response = self.api.join_room(room_id_or_alias)
  File "/opt/venv/lib/python3.7/site-packages/matrix_client/api.py", line 195, in join_room
    return self._send("POST", path)
  File "/opt/venv/lib/python3.7/site-packages/raiden/network/transport/matrix/client.py", line 159, in _send
    return super()._send(method, path, *args, **kwargs)
  File "/opt/venv/lib/python3.7/site-packages/matrix_client/api.py", line 691, in _send
    code=response.status_code, content=response.text
matrix_client.errors.MatrixRequestError: 404: {"errcode":"M_UNKNOWN","error":"No known servers"}
2019-12-13T07:19:45Z <Greenlet "Greenlet-4" at 0x7f586fde8b90: <bound method MatrixTransport._process_queued_invites of <MatrixTransport node:0xd89F35A9274E4d8Bb95312A1a497431CcBF73478 id:1ddeaf38-5077-4d43-9f1c-e6d6d4159c5a>>> failed with MatrixRequestError

2019-12-13 07:19:45.948523 [error    ] Runnable subtask died!         [raiden.utils.runnable] exc=MatrixRequestError('404: {"errcode":"M_UNKNOWN","error":"No known servers"}') running=True subtask=<Greenlet "Greenlet-4" at 0x7f586fde8b90: <bound method MatrixTransport._process_queued_invites of <MatrixTransport node:0xd89F35A9274E4d8Bb95312A1a497431CcBF73478 id:1ddeaf38-5077-4d43-9f1c-e6d6d4159c5a>>> this=<MatrixTransport node:0xd89F35A9274E4d8Bb95312A1a497431CcBF73478 id:1ddeaf38-5077-4d43-9f1c-e6d6d4159c5a>
Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 716, in gevent._greenlet.Greenlet.run
  File "/opt/venv/lib/python3.7/site-packages/raiden/network/transport/matrix/transport.py", line 444, in _run
    self._broadcast_worker()
  File "/opt/venv/lib/python3.7/site-packages/raiden/network/transport/matrix/transport.py", line 618, in _broadcast_worker
    self._broadcast_event.wait(self._config["retry_interval"])
  File "src/gevent/event.py", line 240, in gevent._event.Event.wait
  File "src/gevent/event.py", line 140, in gevent._event._AbstractLinkable._wait
  File "src/gevent/event.py", line 117, in gevent._event._AbstractLinkable._wait_core
  File "src/gevent/event.py", line 119, in gevent._event._AbstractLinkable._wait_core
  File "src/gevent/_greenlet_primitives.py", line 59, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 59, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 63, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/__greenlet_primitives.pxd", line 35, in gevent.__greenlet_primitives._greenlet_switch
matrix_client.errors.MatrixRequestError: 404: {"errcode":"M_UNKNOWN","error":"No known servers"}
2019-12-13T07:19:46Z <Greenlet "MatrixTransport._run node:0xd89F35A9274E4d8Bb95312A1a497431CcBF73478" at 0x7f5875ad2a70: <bound method MatrixTransport._run of <MatrixTransport node:0xd89F35A9274E4d8Bb95312A1a497431CcBF73478 id:1ddeaf38-5077-4d43-9f1c-e6d6d4159c5a>>> failed with MatrixRequestError

2019-12-13 07:19:46.288114 [error    ] Runnable subtask died!         [raiden.utils.runnable] exc=MatrixRequestError('404: {"errcode":"M_UNKNOWN","error":"No known servers"}') running=True subtask=<Greenlet "MatrixTransport._run node:0xd89F35A9274E4d8Bb95312A1a497431CcBF73478" at 0x7f5875ad2a70: <bound method MatrixTransport._run of <MatrixTransport node:0xd89F35A9274E4d8Bb95312A1a497431CcBF73478 id:1ddeaf38-5077-4d43-9f1c-e6d6d4159c5a>>> this=<RaidenService node:0xd89F35A9274E4d8Bb95312A1a497431CcBF73478>
Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 716, in gevent._greenlet.Greenlet.run
  File "/opt/venv/lib/python3.7/site-packages/raiden/raiden_service.py", line 445, in _run
    self.stop_event.wait()
  File "src/gevent/event.py", line 240, in gevent._event.Event.wait
  File "src/gevent/event.py", line 140, in gevent._event._AbstractLinkable._wait
  File "src/gevent/event.py", line 117, in gevent._event._AbstractLinkable._wait_core
  File "src/gevent/event.py", line 119, in gevent._event._AbstractLinkable._wait_core
  File "src/gevent/_greenlet_primitives.py", line 59, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 59, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 63, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/__greenlet_primitives.pxd", line 35, in gevent.__greenlet_primitives._greenlet_switch
matrix_client.errors.MatrixRequestError: 404: {"errcode":"M_UNKNOWN","error":"No known servers"}
2019-12-13T07:19:46Z <Greenlet "RaidenService._run node:0xd89F35A9274E4d8Bb95312A1a497431CcBF73478" at 0x7f5874da6050: <bound method RaidenService._run of <RaidenService node:0xd89F35A9274E4d8Bb95312A1a497431CcBF73478>>> failed with MatrixRequestError

FATAL: An unexpected exception occured. A traceback has been written to /tmp/raiden-exception-2019-12-13T07-19z254tv_0.txt
404: {"errcode":"M_UNKNOWN","error":"No known servers"}

5. Docker container died

Unfortunately, I don't have any other logs.

System Description

Darwin MacBook-Pro.local 19.0.0 Darwin Kernel Version 19.0.0: Thu Oct 17 16:17:15 PDT 2019; root:xnu-6153.41.3~29/RELEASE_X86_64 x86_64
/usr/local/bin/geth
Geth
Version: 1.9.9-stable
Architecture: amd64
Protocol Versions: [64 63]
Go Version: go1.13.4
Operating System: darwin
GOPATH=
GOROOT=/usr/local/Cellar/go/1.13.4/libexec
e45719754f8ff67003ceee8549beda86bfd9eab4

[ulope]

Could be an instance of matrix-org/synapse#1533.

@nephix Can you reproduce this error? Or was it only the one time?

[nephix]

Hi @ulope,

it still seems to occur on my machine in rc2. To reproduce, I only needed to started the docker container of the rc2 image. I didn't do any transactions from the light client.

I also tried to change the matrix server from

https://transport04.raiden.network

to

https://transport01.raiden.network

and the exception didn't occur, but once I switched back to transport04 I got the exception again after a couple of mins idle time (not sure if this actually helps investigating or was just a coincidence)

$ docker run --rm -it \
    -p 8545:8545 \
    -p 5011:5011 \
    --mount src=/Users/nephix/Library/Ethereum/keystore/,target=/keystore,type=bind \
    raidennetwork/raiden:v0.200.0-rc2 \
    --keystore-path /keystore \
    --network-id 5 \
    --environment-type development \
    --eth-rpc-endpoint http://10.104.6.13:8545 \
    --accept-disclaimer \
    --matrix-server=https://transport04.raiden.network \
    --pathfinding-service-address https://pfs-goerli-with-fee.services-test.raiden.network \
    --api-address "http://0.0.0.0:5011"

Welcome to Raiden, version 0.200.0rc2!
----------------------------------------------------------------------
| This is an Alpha version of experimental open source software      |
| released as a test version under an MIT license and may contain    |
| errors and/or bugs. No guarantee or representations whatsoever is  |
| made regarding its suitability (or its use) for any purpose or     |
| regarding its compliance with any applicable laws and regulations. |
| Use of the software is at your own risk and discretion and by      |
| using the software you acknowledge that you have read this         |
| disclaimer, understand its contents, assume all risk related       |
| thereto and hereby release, waive, discharge and covenant not to   |
| sue Brainbot Labs Establishment or any officers, employees or      |
| affiliates from and for any direct or indirect liability resulting |
| from the use of the software as permissible by applicable laws and |
| regulations.                                                       |
|                                                                    |
| Privacy Warning: Please be aware, that by using the Raiden Client, |
| among others, your Ethereum address, channels, channel deposits,   |
| settlements and the Ethereum address of your channel counterparty  |
| will be stored on the Ethereum chain, i.e. on servers of Ethereum  |
| node operators and ergo are to a certain extent publicly available.|
| The same might also be stored on systems of parties running Raiden |
| nodes connected to the same token network. Data present in the     |
| Ethereum chain is very unlikely to be able to be changed, removed  |
| or deleted from the public arena.                                  |
|                                                                    |
| Also be aware, that data on individual Raiden token transfers will |
| be made available via the Matrix protocol to the recipient,        |
| intermediating nodes of a specific transfer as well as to the      |
| Matrix server operators.                                           |
----------------------------------------------------------------------
2020-01-07 07:01:15.713488 [info     ] Starting Raiden                [raiden.ui.runners] architecture=x86_64 distribution=source python_implementation=CPython python_version=3.7.5 raiden=0.200.0rc2 raiden_db_version=23 system=Linux 64bit 4.9.184-linuxkit
The following accounts were found in your machine:

[  0] - 0xd89F35A9274E4d8Bb95312A1a497431CcBF73478
[  1] - 0x385dB1Ed9509FF184ff9E34a06b2D93daFdBAc0b

Select one of them by index to continue: 0
Enter the password to unlock 0xd89F35A9274E4d8Bb95312A1a497431CcBF73478:
Raiden is running in development mode
Checking if the ethereum node is synchronized
You have chosen the Pathfinding Service at https://pfs-goerli-with-fee.services-test.raiden.network.
Operator: John Doe, running version: 0.5.0, chain_id: 5.
Fees will be paid to 0x94DEe8e391410A9ebbA791B187df2d993212c849. Each request costs 1e-16 RDN.
Message from the Pathfinding Service:
This is your favorite pfs for token network registry 0x62436bbadB629498107278486C2A367FfdFEb2f3
2020-01-07 07:01:26.933367 [info     ] Using Pathfinding Service      [raiden.network.pathfinding] pfs_info=PFSInfo(url='https://pfs-goerli-with-fee.services-test.raiden.network', price=100, chain_id=5, token_network_registry_address=b'bCk\xba\xdbb\x94\x98\x10rxHl*6\x7f\xfd\xfe\xb2\xf3', payment_address=b'\x94\xde\xe8\xe3\x91A\n\x9e\xbb\xa7\x91\xb1\x87\xdf-\x992\x12\xc8I', message='This is your favorite pfs for token network registry 0x62436bbadB629498107278486C2A367FfdFEb2f3', operator='John Doe', version='0.5.0')

You are connected to the 'goerli' network and the DB path is: /root/.raiden/node_d89f35a9/netid_5/network_62436bba/v23_log.db
2020-01-07 07:01:27.478705 [info     ] Using Matrix server            [raiden.network.transport.matrix.utils] average_rtt=0 server_ident=Synapse/1.5.1 server_url=https://transport04.raiden.network
This is the first time Raiden is being used with this address. Processing all the events may take some time. Please wait ...
2020-01-07 07:01:28.280410 [info     ] Synchronizing blockchain events [raiden.blockchain.events] blocks_left=509876
2020-01-07 07:02:01.211644 [info     ] Synchronizing blockchain events [raiden.blockchain.events] blocks_left=448521 blocks_per_second=1863.1822321704772
2020-01-07 07:02:18.102047 [info     ] Synchronizing blockchain events [raiden.blockchain.events] blocks_left=402718 blocks_per_second=2711.780154633575
2020-01-07 07:02:23.104019 [info     ] Synchronizing blockchain events [raiden.blockchain.events] blocks_left=402718 blocks_per_second=0.0
2020-01-07 07:02:28.104364 [info     ] Synchronizing blockchain events [raiden.blockchain.events] blocks_left=402718 blocks_per_second=0.0
2020-01-07 07:02:33.104597 [info     ] Synchronizing blockchain events [raiden.blockchain.events] blocks_left=402718 blocks_per_second=0.0
2020-01-07 07:02:38.105116 [info     ] Synchronizing blockchain events [raiden.blockchain.events] blocks_left=402718 blocks_per_second=0.0
2020-01-07 07:02:43.105690 [info     ] Synchronizing blockchain events [raiden.blockchain.events] blocks_left=402718 blocks_per_second=0.0
2020-01-07 07:02:48.683884 [info     ] Synchronizing blockchain events [raiden.blockchain.events] blocks_left=389458 blocks_per_second=2377.113878992713
2020-01-07 07:02:54.492704 [info     ] Synchronizing blockchain events [raiden.blockchain.events] blocks_left=274487 blocks_per_second=19792.542492396024
2020-01-07 07:02:59.506953 [info     ] Synchronizing blockchain events [raiden.blockchain.events] blocks_left=157215 blocks_per_second=23387.78224106455
2020-01-07 07:03:20.145061 [info     ] Updating channel fees          [raiden.raiden_service] cap_mediation_fees=True channel=CanonicalIdentifier(chain_identifier=5, token_network_address=b'O\xfe0\xa44,\x10\x89\xfe\xd7\x13\x925o\x94\x03r>\x92\\', channel_identifier=59) flat_fee=0 proportional_fee=4000 proportional_imbalance_fee=3000
2020-01-07 07:03:32.368215 [info     ] Raiden started                 [raiden.app] node=0xd89F35A9274E4d8Bb95312A1a497431CcBF73478
The Raiden API RPC server is now running at http://0.0.0.0:5011/.

See the Raiden documentation for all available endpoints at
http://raiden-network.readthedocs.io/en/stable/rest_api.html
WARNING
Your account's RDN balance of 0.0 is below the minimum threshold. Provided that you have either a monitoring service or a path finding service activated, your node is not going to be able to pay those services which may lead to denial of service or loss of funds.
Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 716, in gevent._greenlet.Greenlet.run
  File "/opt/venv/lib/python3.7/site-packages/raiden/network/transport/matrix/transport.py", line 430, in _process_queued_invites
    self._handle_invite(room_id, state)
  File "/opt/venv/lib/python3.7/site-packages/raiden/network/transport/matrix/transport.py", line 773, in _handle_invite
    raise last_ex  # re-raise if couldn't succeed in retries
  File "/opt/venv/lib/python3.7/site-packages/raiden/network/transport/matrix/transport.py", line 763, in _handle_invite
    room = self._client.join_room(room_id)
  File "/opt/venv/lib/python3.7/site-packages/matrix_client/client.py", line 320, in join_room
    response = self.api.join_room(room_id_or_alias)
  File "/opt/venv/lib/python3.7/site-packages/matrix_client/api.py", line 195, in join_room
    return self._send("POST", path)
  File "/opt/venv/lib/python3.7/site-packages/raiden/network/transport/matrix/client.py", line 159, in _send
    return super()._send(method, path, *args, **kwargs)
  File "/opt/venv/lib/python3.7/site-packages/matrix_client/api.py", line 691, in _send
    code=response.status_code, content=response.text
matrix_client.errors.MatrixRequestError: 404: {"errcode":"M_UNKNOWN","error":"No known servers"}
2020-01-07T07:05:15Z <Greenlet "Greenlet-4" at 0x7f09151c9cb0: <bound method MatrixTransport._process_queued_invites of <MatrixTransport node:0xd89F35A9274E4d8Bb95312A1a497431CcBF73478 id:31160c90-b29e-4f6d-ac4a-cb0e1f3c9757>>> failed with MatrixRequestError

2020-01-07 07:05:15.376554 [error    ] Runnable subtask died!         [raiden.utils.runnable] exc=MatrixRequestError('404: {"errcode":"M_UNKNOWN","error":"No known servers"}') running=True subtask=<Greenlet "Greenlet-4" at 0x7f09151c9cb0: <bound method MatrixTransport._process_queued_invites of <MatrixTransport node:0xd89F35A9274E4d8Bb95312A1a497431CcBF73478 id:31160c90-b29e-4f6d-ac4a-cb0e1f3c9757>>> this=<MatrixTransport node:0xd89F35A9274E4d8Bb95312A1a497431CcBF73478 id:31160c90-b29e-4f6d-ac4a-cb0e1f3c9757>
Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 716, in gevent._greenlet.Greenlet.run
  File "/opt/venv/lib/python3.7/site-packages/raiden/network/transport/matrix/transport.py", line 444, in _run
    self._broadcast_worker()
  File "/opt/venv/lib/python3.7/site-packages/raiden/network/transport/matrix/transport.py", line 618, in _broadcast_worker
    self._broadcast_event.wait(self._config["retry_interval"])
  File "src/gevent/event.py", line 240, in gevent._event.Event.wait
  File "src/gevent/event.py", line 140, in gevent._event._AbstractLinkable._wait
  File "src/gevent/event.py", line 117, in gevent._event._AbstractLinkable._wait_core
  File "src/gevent/event.py", line 119, in gevent._event._AbstractLinkable._wait_core
  File "src/gevent/_greenlet_primitives.py", line 59, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 59, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 63, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/__greenlet_primitives.pxd", line 35, in gevent.__greenlet_primitives._greenlet_switch
matrix_client.errors.MatrixRequestError: 404: {"errcode":"M_UNKNOWN","error":"No known servers"}
2020-01-07T07:05:15Z <Greenlet "MatrixTransport._run node:0xd89F35A9274E4d8Bb95312A1a497431CcBF73478" at 0x7f0916eda4d0: <bound method MatrixTransport._run of <MatrixTransport node:0xd89F35A9274E4d8Bb95312A1a497431CcBF73478 id:31160c90-b29e-4f6d-ac4a-cb0e1f3c9757>>> failed with MatrixRequestError

2020-01-07 07:05:15.673131 [error    ] Runnable subtask died!         [raiden.utils.runnable] exc=MatrixRequestError('404: {"errcode":"M_UNKNOWN","error":"No known servers"}') running=True subtask=<Greenlet "MatrixTransport._run node:0xd89F35A9274E4d8Bb95312A1a497431CcBF73478" at 0x7f0916eda4d0: <bound method MatrixTransport._run of <MatrixTransport node:0xd89F35A9274E4d8Bb95312A1a497431CcBF73478 id:31160c90-b29e-4f6d-ac4a-cb0e1f3c9757>>> this=<RaidenService node:0xd89F35A9274E4d8Bb95312A1a497431CcBF73478>
Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 716, in gevent._greenlet.Greenlet.run
  File "/opt/venv/lib/python3.7/site-packages/raiden/raiden_service.py", line 445, in _run
    self.stop_event.wait()
  File "src/gevent/event.py", line 240, in gevent._event.Event.wait
  File "src/gevent/event.py", line 140, in gevent._event._AbstractLinkable._wait
  File "src/gevent/event.py", line 117, in gevent._event._AbstractLinkable._wait_core
  File "src/gevent/event.py", line 119, in gevent._event._AbstractLinkable._wait_core
  File "src/gevent/_greenlet_primitives.py", line 59, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 59, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 63, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/__greenlet_primitives.pxd", line 35, in gevent.__greenlet_primitives._greenlet_switch
matrix_client.errors.MatrixRequestError: 404: {"errcode":"M_UNKNOWN","error":"No known servers"}
2020-01-07T07:05:18Z <Greenlet "RaidenService._run node:0xd89F35A9274E4d8Bb95312A1a497431CcBF73478" at 0x7f09161ac050: <bound method RaidenService._run of <RaidenService node:0xd89F35A9274E4d8Bb95312A1a497431CcBF73478>>> failed with MatrixRequestError

FATAL: An unexpected exception occured. A traceback has been written to /tmp/raiden-exception-2020-01-07T07-05r77m0nk7.txt
404: {"errcode":"M_UNKNOWN","error":"No known servers"}

[ulope]

@nephix Does this only occur with one specific keystore?

If yes then from reading the traceback I assume an invite with an invalid server (e.g. not in the federation whitelist) is still stored on the matrix server for your user account.

[nephix]

@ulope all I can say is that I have removed all my keystores, created new ones and the issue does not seem to occur anymore. So your assumption could be right 👍

[ulope]

Ok, thanks.

[ulope]

Issue to track the underlying cause: #5640