addObserver eventually stops observing #751

dmh1974 · 2022-05-19T06:18:11Z

so far i have only seen this happen with js-waku used in a browser (have not tested node)

..and i am currently using the testnet (since prod has been fritzy)

i can reproduce the results here.. usually stops around the 10min mark...

https://gateway.pinata.cloud/ipfs/QmUS84DM9wLhmXnVf1EB41uoGc4HobTPc9W2CMAUWQRMFs?topic=BTCUSDT

dmh1974 · 2022-05-19T06:35:50Z

fyi, backgrounding tabs exacerbates the issue as forgrounding does not appear to gracefully recover the connection

dmh1974 · 2022-05-20T19:59:12Z

fyi, this may be related to PING/PONG frames on the ws connection.

PING alone is not enough the other end must reply with a PONG frame as well.

also note that PING/PONG frames are not required to keep a ws alive unless one end is depending on it.. the other must recognize a ping frame and respond with a pong frame.

https://stackoverflow.com/questions/46111656/must-websockets-have-heartbeats

dmh1974 · 2022-05-21T22:15:31Z

fyi, addObserver is stable when used with node

D4nte · 2022-05-23T04:47:54Z

Thx, will investigate.

dao · 2022-05-23T14:29:15Z

I suspect that this is the same issue we're having in https://railway.xyz - at some point, clients stop receiving fees from relayers and have to refresh the page to get it going again. @D4nte if you have a workaround we can implement on the frontend in the meantime - eg a timer to check if observers are observing and force reinitialize them if not - I'd love to hear it.

dmh1974 · 2022-05-23T20:09:09Z

@dao i have been pondering a workaround myself... one way would be to use the queryHistory function on a timer that keeps track of startdate as a pointer. the nodes also apparently support rpc calls and there are some docs on that, but i havent done a deep dive on it yet.

dmh1974 · 2022-05-29T20:46:51Z

fyi, i discovered this also applies to relay.send when used in a browser. after 10m it too stops working

the issue does not appear to manifest when used in node

D4nte · 2022-05-30T11:12:20Z

@dmh1974 were you using port 443 of the fleet (websockify) or 8000 (nim-websocket)?

I know @dao was using native websocket. It seems that we have a bug in nim-websocket that closes connections after 10min. Hopefully this is the issue.

@jm-clius: can you share the issue # please?

D4nte · 2022-05-30T11:13:07Z

Note: I was surprised by this bug because we used to have the issue with websockify and/or nginx and I resolved it using keep alive.

jm-clius · 2022-05-30T11:47:53Z

Yes, this fix in nim-libp2p addresses the 10 min timeout issue for native websocket connections: vacp2p/nim-libp2p#721 We still need to update the submodule to have an effect in nwaku.

dmh1974 · 2022-05-30T11:54:52Z

i am only specifying a connection for bootstrap when invoking create.. i got it from the fleets json (testnet). im not specifying a port anywhere else after that (im assuming the js-waku library handles such)

dao · 2022-05-30T16:26:19Z

i am only specifying a connection for bootstrap when invoking create.. i got it from the fleets json (testnet). im not specifying a port anywhere else after that (im assuming the js-waku library handles such)

We are doing the very same thing

      logDev(`Creating js-waku client: ${fleet} fleet`);
      const peers = getPredefinedBootstrapNodes(fleet, 2);
      const waku = await Waku.create({
        bootstrap: { peers }
      });

D4nte · 2022-05-31T00:27:09Z

i am only specifying a connection for bootstrap when invoking create.. i got it from the fleets json (testnet). im not specifying a port anywhere else after that (im assuming the js-waku library handles such)

Ok fleets.status.im currently contains the 8000 port so it means it's nim-websocket.
Note that you can work around the issue by replacing 8000 by 443 or simply using the predefined nodes available in js-waku:

https://github.com/status-im/js-waku/blob/6df972694cd9ae948fdaaf4bef16c892b943281d/src/lib/discovery/predefined.ts#L60

Also note that we discourage using fleets.status.im. This is a just a dev tool that can go down at any time.

D4nte · 2022-05-31T00:28:42Z

i am only specifying a connection for bootstrap when invoking create.. i got it from the fleets json (testnet). im not specifying a port anywhere else after that (im assuming the js-waku library handles such)

We are doing the very same thing
      logDev(`Creating js-waku client: ${fleet} fleet`);
      const peers = getPredefinedBootstrapNodes(fleet, 2);
      const waku = await Waku.create({
        bootstrap: { peers }
      });

Ok that's not great because it means you are using websockify (port 443) and getting the timeout. I'll need to investigate.

dmh1974 · 2022-06-01T15:27:41Z

ok gents, happy to report send and receive have been stable in two separate tabs for over 30mins now (also surviving tab backgrounding)

not sure what changed on your end, but keep up the good work! (maybe mem leak reboots?)

current env...

Waku.create({bootstrap: {peers: ["/dns4/node-01.gc-us-central1-a.wakuv2.test.statusim.net/tcp/8000/wss/p2p/16Uiu2HAmJb2e28qLXxT5kZxVUUoJt72EMzNGXB47Rxx5hw3q4YjS",]},

jm-clius · 2022-06-01T16:04:52Z

@dmh1974 excellent news! It was a fix in an underlying library that caused timeouts in websocket connections. Glad to hear it's working as expected.

dmh1974 · 2022-06-02T00:48:41Z

one important note... the connection does not survive 'sleeping'... if i reopen my laptop after some time, the connection does not re-establish gracefully. the page needs refreshing.

therealjmj · 2022-06-07T13:00:09Z

@jm-clius - Fantastic! Is the fix available in js-waku 0.24?

@dmh1974 - In you testing, is there a variable on Waku that we can watch to know "status" - whether connected or disconnected"? We're considering a status indicator that could show on the frontend.

jm-clius · 2022-06-07T13:12:01Z

@jm-clius - Fantastic! Is the fix available in js-waku 0.24?>

The timeout was on nwaku's side (e.g. on the fleets), has been fixed on the latest master and is already running on the wakuv2.test fleet. It will also be included in the upcoming nwaku release (v0.10), scheduled for middle of next week.

dmh1974 · 2022-06-08T00:01:37Z

@jrmeurer i havent.. hopefully that will be handled gracefully by js-waku in the future.

dmh1974 · 2022-06-08T00:03:59Z

@jm-clius i know its all related, but just for completion sake I'll mention on this ticket that even currently with observers stable, there is still a degree of message loss

D4nte · 2022-07-25T11:52:02Z

This might be a fix to this issue: https://github.com/status-im/status-web/pull/288/files

therealjmj · 2022-07-25T13:07:38Z

This might be a fix to this issue: https://github.com/status-im/status-web/pull/288/files

Good potential. Will this change be added to js-waku?

D4nte · 2022-07-25T21:40:49Z

This might be a fix to this issue: https://github.com/status-im/status-web/pull/288/files

Good potential. Will this change be added to js-waku?

I have encouraged it to be pushed directly upstream in the right libp2p dependency which seems to be the right thing to do 🙂

lucille-bellepleure · 2022-10-27T23:11:43Z

Maybe keep this issue open until the problem is resolved - we have been researching why we haz this err and it's useful to see this issue

fryorcraken · 2022-11-01T00:43:23Z

Maybe keep this issue open until the problem is resolved - we have been researching why we haz this err and it's useful to see this issue

Hi @michelleplur, yes the process is to keep issues open until the problem is resolved.

Note that the final solution would be to backport the quoted patch to upstream libp2p dependency.

This is next on the list after the refactoring/doc update is done: #802
I haven't done it before the ESM migration because:

I was hoping that bumping to latest libp2p libs could have solved the issue (it did not)
The proper fix is to patch upstream libp2p lib, no point doing so until we are using the latest libp2p dep, which was blocked on ESM migration as libp2p is ESM only now.

Let me know if you have any question.

fryorcraken · 2022-11-21T00:21:57Z


11:08:38.658 libp2p:websockets dialing /dns4/node-01.ac-cn-hongkong-c.wakuv2.prod.statusim.net/tcp/443/wss/p2p/16Uiu2HAm4v86W3bmT1BiH6oSPzcsSr24iDQpSN5Qa992BCjjwgrD +0ms

...

11:18:27.930 libp2p:dialer token 7 released +9ms common.js:108
11:18:27.931 libp2p:dialer dial failed to 16Uiu2HAm4v86W3bmT1BiH6oSPzcsSr24iDQpSN5Qa992BCjjwgrD +0ms AggregateError: No Promise in Promise.any was resolved
    run dial-request.js:56
    _createPendingDial index.js:208
    dial index.js:98
    openConnection index.js:344
    _autoDial auto-dialler.js:92
    retimer bundle.js:129007
    setTimeout handler*Retimer retimer.js:21
    retimer retimer.js:65
    _autoDial auto-dialler.js:102
    retimer bundle.js:129007
    setTimeout handler*Retimer retimer.js:21
    retimer retimer.js:65
    _autoDial auto-dialler.js:102
    retimer bundle.js:129007
    setTimeout handler*Retimer retimer.js:21
    retimer retimer.js:65
    _autoDial auto-dialler.js:102
    retimer bundle.js:129007
    setTimeout handler*Retimer retimer.js:21
    retimer retimer.js:65
    _autoDial auto-dialler.js:102
    retimer bundle.js:129007
    setTimeout handler*Retimer retimer.js:21
    retimer retimer.js:65
    _autoDial auto-dialler.js:102
    retimer bundle.js:129007
    setTimeout handler*Retimer retimer.js:21
    retimer retimer.js:65
    _autoDial auto-dialler.js:102
    retimer bundle.js:129007
    setTimeout handler*Retimer retimer.js:21
    retimer retimer.js:65
    _autoDial auto-dialler.js:102
common.js:108
11:18:27.931 libp2p:dialer:error Promise.any*run@http://localhost:3004/eth-pm/static/js/bundle.js:164581:28
_createPendingDial@http://localhost:3004/eth-pm/static/js/bundle.js:164874:28
dial@http://localhost:3004/eth-pm/static/js/bundle.js:164764:70
async*openConnection@http://localhost:3004/eth-pm/static/js/bundle.js:165310:55
_autoDial@http://localhost:3004/eth-pm/static/js/bundle.js:164492:51
Async*./node_modules/.pnpm/[email protected]/node_modules/retimer/retimer.js/Retimer/this._timerWrapper@http://localhost:3004/eth-pm/static/js/bundle.js:129007:18
setTimeout handler*Retimer@http://localhost:3004/eth-pm/static/js/bundle.js:129010:19
retimer@http://localhost:3004/eth-pm/static/js/bundle.js:129054:10
_autoDial@http://localhost:3004/eth-pm/static/js/bundle.js:164502:28
Async*./node_modules/.pnpm/[email protected]/node_modules/retimer/retimer.js/Retimer/this._timerWrapper@http://localhost:3004/eth-pm/static/js/bundle.js:129007:18
setTimeout handler*Retimer@http://localhost:3004/eth-pm/static/js/bundle.js:129010:19
retimer@http://localhost:3004/eth-pm/static/js/bundle.js:129054:10
_autoDial@http://localhost:3004/eth-pm/static/js/bundle.js:164502:28
Async*./node_modules/.pnpm/[email protected]/node_modules/retimer/retimer.js/Retimer/this._timerWrapper@http://localhost:3004/eth-pm/static/js/bundle.js:129007:18
setTimeout handler*Retimer@http://localhost:3004/eth-pm/static/js/bundle.js:129010:19
retimer@http://localhost:3004/eth-pm/static/js/bundle.js:129054:10
_autoDial@http://localhost:3004/eth-pm/static/js/bundle.js:164502:28
Async*./node_modules/.pnpm/[email protected]/node_modules/retimer/retimer.js/Retimer/this._timerWrapper@http://localhost:3004/eth-pm/static/js/bundle.js:129007:18
setTimeout handler*Retimer@http://localhost:3004/eth-pm/static/js/bundle.js:129010:19
retimer@http://localhost:3004/eth-pm/static/js/bundle.js:129054:10
_autoDial@http://localhost:3004/eth-pm/static/js/bundle.js:164502:28
Async*./node_modules/.pnpm/[email protected]/node_modules/retimer/retimer.js/Retimer/this._timerWrapper@http://localhost:3004/eth-pm/static/js/bundle.js:129007:18
setTimeout handler*Retimer@http://localhost:3004/eth-pm/static/js/bundle.js:129010:19
retimer@http://localhost:3004/eth-pm/static/js/bundle.js:129054:10
_autoDial@http://localhost:3004/eth-pm/static/js/bundle.js:164502:28
Async*./node_modules/.pnpm/[email protected]/node_modules/retimer/retimer.js/Retimer/this._timerWrapper@http://localhost:3004/eth-pm/static/js/bundle.js:129007:18
setTimeout handler*Retimer@http://localhost:3004/eth-pm/static/js/bundle.js:129010:19
retimer@http://localhost:3004/eth-pm/static/js/bundle.js:129054:10
_autoDial@http://localhost:3004/eth-pm/static/js/bundle.js:164502:28
Async*./node_modules/.pnpm/[email protected]/node_modules/retimer/retimer.js/Retimer/this._timerWrapper@http://localhost:3004/eth-pm/static/js/bundle.js:129007:18
setTimeout handler*Retimer@http://localhost:3004/eth-pm/static/js/bundle.js:129010:19
retimer@http://localhost:3004/eth-pm/static/js/bundle.js:129054:10
_autoDial@http://localhost:3004/eth-pm/static/js/bundle.js:164502:28
 +1ms common.js:108
11:18:27.931 libp2p:connection-manager:auto-dialler:error could not connect to peerStore stored peer +10s AggregateError: No Promise in Promise.any was resolved
    run dial-request.js:56
    _createPendingDial index.js:208
    dial index.js:98
    openConnection index.js:344
    _autoDial auto-dialler.js:92
    retimer bundle.js:129007
    setTimeout handler*Retimer retimer.js:21
    retimer retimer.js:65
    _autoDial auto-dialler.js:102
    retimer bundle.js:129007
    setTimeout handler*Retimer retimer.js:21
    retimer retimer.js:65
    _autoDial auto-dialler.js:102
    retimer bundle.js:129007
    setTimeout handler*Retimer retimer.js:21
    retimer retimer.js:65
    _autoDial auto-dialler.js:102
    retimer bundle.js:129007
    setTimeout handler*Retimer retimer.js:21
    retimer retimer.js:65
    _autoDial auto-dialler.js:102
    retimer bundle.js:129007
    setTimeout handler*Retimer retimer.js:21
    retimer retimer.js:65
    _autoDial auto-dialler.js:102
    retimer bundle.js:129007
    setTimeout handler*Retimer retimer.js:21
    retimer retimer.js:65
    _autoDial auto-dialler.js:102
    retimer bundle.js:129007
    setTimeout handler*Retimer retimer.js:21
    retimer retimer.js:65
    _autoDial auto-dialler.js:102
common.js:108
11:18:37.932 libp2p:peer-store:trace getPeers await read lock +10s common.js:108
11:18:37.932 libp2p:peer-store:trace getPeers got read lock +0ms common.js:108
11:18:37.934 libp2p:peer-store:trace getPeers release read lock +2ms common.js:108
11:18:37.934 libp2p:connection-manager:auto-dialler connecting to a peerStore stored peer 16Uiu2HAm4v86W3bmT1BiH6oSPzcsSr24iDQpSN5Qa992BCjjwgrD +10s common.js:108
11:18:37.934 libp2p:connection-manager dial to 16Uiu2HAm4v86W3bmT1BiH6oSPzcsSr24iDQpSN5Qa992BCjjwgrD +10s common.js:108
11:18:37.934 libp2p:dialer check multiaddrs 16Uiu2HAm4v86W3bmT1BiH6oSPzcsSr24iDQpSN5Qa992BCjjwgrD +10s common.js:108
11:18:37.935 libp2p:dialer creating dial target for 16Uiu2HAm4v86W3bmT1BiH6oSPzcsSr24iDQpSN5Qa992BCjjwgrD +0ms common.js:108
11:18:37.935 libp2p:peer-store:address-book:trace get wait for read lock +10s common.js:108
11:18:37.935 libp2p:peer-store:address-book:trace get got read lock +1ms common.js:108
11:18:37.936 libp2p:peer-store:address-book:trace get release read lock +0ms common.js:108
11:18:37.938 libp2p:dialer 1 tokens request, returning 1, 99 remaining +3ms common.js:108
11:18:37.939 libp2p:websockets dialing /ip4/0.0.0.0/tcp/8000/wss/p2p/16Uiu2HAm4v86W3bmT1BiH6oSPzcsSr24iDQpSN5Qa992BCjjwgrD +10s common.js:108
11:18:37.939 libp2p:websockets dialing 0.0.0.0:8000 +0ms common.js:108
11:18:37.944 Firefox can’t establish a connection to the server at wss://0.0.0.0:8000/p2p/16Uiu2HAm4v86W3bmT1BiH6oSPzcsSr24iDQpSN5Qa992BCjjwgrD. client.js:9
11:18:37.945 libp2p:websockets:error connection error: +10s

Autodial actually happens when the connection is lost after 10min but the wrong multiaddr is used:

11:20:34.350 libp2p:connection-manager dial to 16Uiu2HAm4v86W3bmT1BiH6oSPzcsSr24iDQpSN5Qa992BCjjwgrD +36s [common.js:108](http://localhost:3004/home/fryorcraken/src/waku-org/js-waku-examples/eth-pm/node_modules/.pnpm/[email protected]/node_modules/debug/src/common.js)
11:20:34.351 libp2p:dialer check multiaddrs 16Uiu2HAm4v86W3bmT1BiH6oSPzcsSr24iDQpSN5Qa992BCjjwgrD +10s [common.js:108](http://localhost:3004/home/fryorcraken/src/waku-org/js-waku-examples/eth-pm/node_modules/.pnpm/[email protected]/node_modules/debug/src/common.js)
11:20:34.351 libp2p:dialer creating dial target for 16Uiu2HAm4v86W3bmT1BiH6oSPzcsSr24iDQpSN5Qa992BCjjwgrD +0ms [common.js:108](http://localhost:3004/home/fryorcraken/src/waku-org/js-waku-examples/eth-pm/node_modules/.pnpm/[email protected]/node_modules/debug/src/common.js)
11:20:34.351 libp2p:peer-store:address-book:trace get wait for read lock +36s [common.js:108](http://localhost:3004/home/fryorcraken/src/waku-org/js-waku-examples/eth-pm/node_modules/.pnpm/[email protected]/node_modules/debug/src/common.js)
11:20:34.352 libp2p:peer-store:address-book:trace get got read lock +1ms [common.js:108](http://localhost:3004/home/fryorcraken/src/waku-org/js-waku-examples/eth-pm/node_modules/.pnpm/[email protected]/node_modules/debug/src/common.js)
11:20:34.352 libp2p:peer-store:address-book:trace get release read lock +0ms [common.js:108](http://localhost:3004/home/fryorcraken/src/waku-org/js-waku-examples/eth-pm/node_modules/.pnpm/[email protected]/node_modules/debug/src/common.js)
11:20:34.356 libp2p:dialer 1 tokens request, returning 1, 99 remaining +5ms [common.js:108](http://localhost:3004/home/fryorcraken/src/waku-org/js-waku-examples/eth-pm/node_modules/.pnpm/[email protected]/node_modules/debug/src/common.js)
11:20:34.356 libp2p:websockets dialing /ip4/0.0.0.0/tcp/8000/wss/p2p/16Uiu2HAm4v86W3bmT1BiH6oSPzcsSr24iDQpSN5Qa992BCjjwgrD +36

Need to investigate why the multiaddr is not being stored in the peer store.

fryorcraken · 2022-11-21T03:26:03Z

investigating and reported to libp2p too libp2p/js-libp2p#1484

fryorcraken · 2022-11-21T05:33:03Z

Issue found: libp2p/js-libp2p#1484 (comment) upstream fix is being tested.

fryorcraken · 2022-11-21T06:05:47Z

Upstream fix proposed: libp2p/js-libp2p#1485

Note that latest js-libp2p automatically attempt to reconnect to the remote node when the connection drops. Which means that the workaround https://github.com/status-im/status-web/pull/288/files is not needed any more.

This is done in the form of the autodialer: https://github.com/libp2p/js-libp2p/blob/a3847f2d1725b1c92d5e0ef7bcdf840ea8428a75/src/connection-manager/auto-dialler.ts which only dials when the number of connections is under a threshold.

Unfortunately, an other issue was corrupting the address book after the identify protocol is executed. Which is the upstream fix.

One work around of the latest issue would be to do addressBook.add after identify is done so that the overridden value can be re-instated.

fryorcraken · 2022-11-28T11:52:34Z

Actually, issue in nwaku/nim-libp2p, see waku-org/nwaku#1427

fryorcraken · 2022-12-22T22:23:09Z

To keep in mind: status-im/status-web#320

fryorcraken · 2023-01-06T03:18:44Z

Playing locally with (eth-pm example)

    "@waku/byte-utils": "0.0.2",
    "@waku/core": "^0.0.8",
    "@waku/create": "^0.0.6",
    "@waku/interfaces": "^0.0.6",
    "@waku/message-encryption": "^0.0.7",

By killing a local nwaku node.
libp2p reconnects automatically once the node is back up.
There may be a lack of back off during the dial retry as mentioned in status-im/status-web#320

Next step:

Try to reproduce when connecting to fleet by killing internet connection
Try to reproduce by leaving browser open for 10min

fryorcraken · 2023-01-11T23:49:34Z

I have seen some issue when reconnecting after internet loss.

Noise handshake goes well but identify protocol fails.

waku-org/nwaku#1485 open but will investigate more locally.

fryorcraken · 2023-01-12T03:22:13Z

I have ran the eth-pm example locally (uses relay) for over 10 minutes in Firefox. The example could still send and receiv messages after the lapse time.
There was no disconnection messages in the logs.

It seems that this specific bug is now resolved.

Keeping the issue open until the reconnection problem is solved.

fryorcraken · 2023-01-12T03:34:44Z

No problem either when running in brave from the hosted example https://examples.waku.org/eth-pm keeping the tab open for over 10min.

fryorcraken · 2023-01-17T04:57:46Z

new error reported by @therealjmj is that the timeout happens after 30-40min.
Not sure if it is similar to waku-org/nwaku#1485 or not
First step would be to reproduce locally. waku-org/nwaku#1485 has very specific log output (noise happens but then it fails) so we can understand whether we are facing 1 or 2 issues.

If it's 2 different issues, the 30-40min timeout is more important because it's more generic use case and should prioritized.

If it's the same issue, then let's look at waku-org/nwaku#1485 (comment) and understand how js-waku and nwaku can handle idle connections etc.

danisharora099 · 2023-01-30T06:16:35Z

Not sure if it is related but the inconsistency with relay messages is seen here: waku-org/examples.waku.org#199

danisharora099 · 2023-01-30T07:59:37Z

websocket connection was alive for ~1.4 hours and it can be seen that no messages were sent in the interim and only towards the end (content downloaded)

upon sleeping the laptop, and reopening, that websocket connection was terminated but a new one was immediately opened.

fryorcraken · 2023-01-31T05:14:15Z

As discussed with @danisharora099, this can be closed with guidelines for user that experience it again (enable debug log, share full logs)

danisharora099 · 2023-02-01T10:56:36Z

As discussed with @danisharora099, this can be closed with guidelines for user that experience it again (enable debug log, share full logs)

⬆️

Closing this for now.
If there is an issue again, feel free to reopen this, or create a new, issue with logs. You can enable logs using DEBUG=waku*,libp2p*
A reproducible example along would also help!

D4nte added rqt:platforms bug Something isn't working labels May 26, 2022

fryorcraken added track:production and removed rqt:platforms labels Aug 22, 2022

fryorcraken mentioned this issue Aug 25, 2022

Write RFC for relay ping #562

Closed

3 tasks

fryorcraken added track:maintenance and removed track:production labels Aug 29, 2022

fryorcraken self-assigned this Nov 20, 2022

fryorcraken mentioned this issue Nov 28, 2022

bug: 0.0.0.0 included in listenAddrs of identify message waku-org/nwaku#1427

Closed

felicio mentioned this issue Dec 21, 2022

Light Push and Filter eventually stop working status-im/status-web#322

Open

fryorcraken mentioned this issue Dec 22, 2022

Failed websocket connections are retried instantly status-im/status-web#320

Open

fryorcraken assigned danisharora099 and unassigned fryorcraken Jan 17, 2023

danisharora099 mentioned this issue Jan 30, 2023

bug: inconsistent messages on eth-pm waku-org/examples.waku.org#199

Open

2 tasks

danisharora099 closed this as completed Feb 1, 2023

fryorcraken mentioned this issue Mar 7, 2023

After idle, messages in web app are not sent #185

Closed

addObserver eventually stops observing #751

addObserver eventually stops observing #751

Comments

dmh1974 commented May 19, 2022

dmh1974 commented May 19, 2022

dmh1974 commented May 20, 2022

dmh1974 commented May 21, 2022

D4nte commented May 23, 2022

dao commented May 23, 2022

dmh1974 commented May 23, 2022

dmh1974 commented May 29, 2022

D4nte commented May 30, 2022

D4nte commented May 30, 2022

jm-clius commented May 30, 2022

dmh1974 commented May 30, 2022

dao commented May 30, 2022

D4nte commented May 31, 2022

D4nte commented May 31, 2022

dmh1974 commented Jun 1, 2022

jm-clius commented Jun 1, 2022

dmh1974 commented Jun 2, 2022

therealjmj commented Jun 7, 2022 • edited Loading

jm-clius commented Jun 7, 2022

dmh1974 commented Jun 8, 2022

dmh1974 commented Jun 8, 2022

D4nte commented Jul 25, 2022

therealjmj commented Jul 25, 2022

D4nte commented Jul 25, 2022

lucille-bellepleure commented Oct 27, 2022

fryorcraken commented Nov 1, 2022

fryorcraken commented Nov 21, 2022 • edited Loading

fryorcraken commented Nov 21, 2022

fryorcraken commented Nov 21, 2022

fryorcraken commented Nov 21, 2022

fryorcraken commented Nov 28, 2022

fryorcraken commented Dec 22, 2022

fryorcraken commented Jan 6, 2023 • edited Loading

fryorcraken commented Jan 11, 2023

fryorcraken commented Jan 12, 2023

fryorcraken commented Jan 12, 2023

fryorcraken commented Jan 17, 2023 • edited Loading

danisharora099 commented Jan 30, 2023

danisharora099 commented Jan 30, 2023 • edited Loading

fryorcraken commented Jan 31, 2023

danisharora099 commented Feb 1, 2023 • edited Loading

therealjmj commented Jun 7, 2022 •

edited

Loading

fryorcraken commented Nov 21, 2022 •

edited

Loading

fryorcraken commented Jan 6, 2023 •

edited

Loading

fryorcraken commented Jan 17, 2023 •

edited

Loading

danisharora099 commented Jan 30, 2023 •

edited

Loading

danisharora099 commented Feb 1, 2023 •

edited

Loading