RPC Service Locked Up

[buffalu]

Problem

• RPC service locked up on several v1.14.13-jito RPC servers around 11:17PM CT time, Tuesday Jan 24
• Node continued to replay ontime. Pubsub + geyser continued to work.
• Likely to be batched getMultipleAccounts, getTokenAccountsByOwner

Likely deadlock in RPC Servers or getMA?

Relevant rpc cli args:

• --rpc-threads 8
• --account-index-exclude-key kinXdEcpDQeHPEuQnqmUgtYykqKGVFq6CeVX5iAHJq6
• --account-index-exclude-key TokenkegQfeZyiNwAJbNbGKPFXCWuBvf9Ss623VQ5DA

Proposed Solution

Determine lockup cause and fix

[buffalu]

https://cdn.discordapp.com/attachments/837340113067049050/1067692312420945940/telegram-cloud-photo-size-1-5082650279769385563-y.jpg

https://cdn.discordapp.com/attachments/837340113067049050/1067695770360356924/Screen_Shot_2023-01-25_at_12.41.44_AM.png

[buffalu]

non-potato picture

[steviez]

I'm not super dialed in on this one, but I think there has been some investigation / theory on potential culprit: #24644

RPC service locked up on several v1.14.13-jito RPC servers

Obligatory question comes to mind as to whether the stock software shows the issue. But, given the issue we have + some reports I've heard from others, this is seemingly not an issue that jito commits introduce.

[buffalu]

@mschneider might find this interesting

[mschneider]

So we observed some users of the mango rpc using massively batched gMA and gPA to avoid rate limits, we ended up blocking any form of batching involving these requests, which dramatically improved performance for other users.

Curious: did websocket / pubsub connections that were opened before still transmit data?

[mschneider]

If you were to direct request workload through a reverse proxy you might be able to create a log of the RPC method combination that lead to the issue

[buffalu]

Curious: did websocket / pubsub connections that were opened before still transmit data?

yes and geyser. Everything kept running except rpc.

If you were to direct request workload through a reverse proxy you might be able to create a log of the RPC method combination that lead to the issue

the chart I sent is from our proxy showing the rpc methods that caused it