Rocks DB: io.vertx.core.VertxException: Thread blocked (--Xplugin-rocksdb-high-spec-enabled) #4443
Comments
|
It may be unrelated, but also seeing this error in logs after restarting: |
|
It seems to be related to the memory leak issue fixed in version 22.7.4 released today. Some threads may be blocked for some time by GC threads trying to free up memory. |
|
Thanks! I just noticed the new version and have updated. It seems fixed and
will continue to monitor.
…On Mon, Sep 26, 2022 at 2:28 PM ahamlat ***@***.***> wrote:
It seems to be related to the memory leak issue fixed in version 22.7.4
released today. Some threads may be blocked for some time by GC threads
trying to free up memory.
Could you update your Besu to 22.7.4 version and let us know if you still
have these issues.
—
Reply to this email directly, view it on GitHub
<#4443 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AXS2AHGMYNS5WND5JB5HXSDWAH2PXANCNFSM6AAAAAAQV7ZN4I>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
|
Closing for now - please re-open if we see this behavior later on. |
|
I have the issue with the version 22.7.4: Any idea? |
|
Reports in 22.7.7… reopening for now |
|
Observed on 22.7.6 and .7 without high spec mode enabled. Potentially different underlying cause? |
|
We need to investigate, do you have any feedback from the users regarding these two versions ? |
|
Asked @benjaminion for a thread dump. |
|
I was looking for a heap dump or at least a histogram
|
|
This is occurring again, right after I tried to connect a dApp to Besu via Metamask. I've taken dumps as requested and will figure out a way to get them to @ahamlat. |
|
Interesting, it seems to be related to eth_getLog Json RPC call. I can see 7 executing eth_getLog calls, and 5 of them are executing RocksDB.get method. Do you have more information on these eth_getLog calls ? we can proxy them if there is no personal addresses. The purpose of this is to try to reproduce and understand why these calls are blocking on RocksDB.get method. |
|
My best guess is that connecting metamask to this dApp with my local Besu set as the network provider is the trigger for this. It seems to be repeatable, but I am reluctant to experiment as this is my production staking node. I haven't previously noticed any issues using this Besu instance as my network provider in metamask. For now I have pointed metamask back to Infura 😞 |
|
Thanks Ben, I found this part of the code in their Github, so I guess they are getting logs for large ranges of blocks, and it seems that in Besu, there is no limit to this range. @non-fungible-nelson I think it is important to investigate the performance and reliability of the eth_getLog Json RPC call. We already have some folks working on other Json RPC calls, I don't know if they are also looking into eth_getLog, maybe try to reproduce with a testnet with the same dApp and proxy the calls to get the ranges. I also found this old issue on eth_getLog |
|
I just found in the heapdump fromBlock and toBlock parameters of the eth_getLog call, it is from 0x0 (0 in decimal) to 0xf180de (15827166 in decimal) which represents the whole blocks from genesis to yesterday. There is also a parameter to filter on, but I guess is less efficient with such a number of blocks to filter.
|
|
Hmm interesting @ahamlat - should we limit by default? Or another behavior. Tagging @mark-terry to look at for his RPC investigation. |
|
The default when #4123 suggested adding a configurable maximum search range. This should address what @ahamlat has noted if the call was executed without a It'd be interesting to know whether the Metamask issue is caused by |
|
Added a ticket for a configurable range: #4563 |
|
Now that #4563 is merged can this be closed? |
|
I can confirm that this no longer occurs with 22.10.4 when I visit the dApp that used to reliably trigger it. (Besu is now so slow with that site that it's unusable, but that's a separate issue to this one.) I can't say for sure the underlying issue is resolved but it looks promising. |
|
The issue still happens for me on 22.10.4. If anything, it became even worse - more spam in the logs due to this error. |
|
@ahamlat - i think this warrants more investigation. Users are seeing tons of threads blocked on 22.10.3/4 without using the RPC. |
My node failed to attest and needed a restart to recover using the new high spec enabled flag on v22.7.3 with bonsai trees.
Frequency:
Happened once, stopped attesting completely. Had to restart service to recover.
Versions (Add all that apply)
Additional Information (Add any of the following or anything else that may be relevant)
Server: 26 GB RAM, 18 cores, AMD
###LOGS
The text was updated successfully, but these errors were encountered: