-
Notifications
You must be signed in to change notification settings - Fork 20.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Sync block problem #21825
Comments
Please provide the full message (preferrably as text instead of image), and double-check the version number |
If you’ve just upgraded to the latest release try to run debug.setHead(11234872) or debug.setHead('0xAB6E38') |
Tried this keeps stopping at block 11234899. Any other ideas? |
@peterlayke Make sure you are on the latest geth version, we had this happen too on the same block on one of our older nodes even though no releases have been marked as mandatory since then. |
^ this |
Thanks all, problem was after setting using setHead I had to restart the client. I had just used the command and let it run the first time. Edit: nope stuck in a loop now, node just rewinds blocks constantly now |
After restarting, I can’t get the latest block number
|
Upgraded to 1.9.23-Stable, rolled back blocks. now I get to the below and my node just resets and winds back blocks. { |
https://blog.infura.io/infura-mainnet-outage-post-mortem-2020-11-11/ When can we get this patch publicly? My production workloads are down until there is a resolution. |
My two nodes running geth v1.9.16 suffered from this issue today, I applied the debug.setHead(xxx) trick and restart the geth node, it does not work at all, so I have to upgrade the geth version to the latest v1.9.23, and applied the magic "debug.setHead(xxx)' trick again. restart the geth nodes, Now one node has finished syncing and works again, another node shows weird behavior as following:
The syncing messages are:
I will run it for a while to see if it can finish syncing by itself, most likely its blockchain data integrity is already corrupted. |
My problem is the same as yours. The block number is always 0. |
The bad node with sync.blockNumber == 0 still cannot sync after many hours, there should be a bug somewhere or the blockchain data is already broken. Fortunately I don't have to resync the whole blockchain, making a copy of block data from the good node should work. |
Sorry, my geth nodes are private and only reserved for internal api services. |
Yes, if you have two machines, it's perfectly valid to (when both nodes are shut down) copy the chaindata from one machine to the other. "Sync via scp". |
The good news is: the bad node has healed itself before I give up:
So we must be patient when dealing with eth blockchain ;-) |
Before rsync or scp the good data to overwrite the bad data, please make sure to backup the nodekey file ( geth/nodekey ), it should be unique for each eth node, otherwise you will have twins with the same identity. |
Having this exact problem. |
Confirming that upgrading to Geth-v.1.9.24-stable and then running If anything else goes wrong I'll update here 👇 |
Iam currently upgraded to geth 1.9.24-stable-cc05b050 cause of this issue. Upgraded to M2.SSD without success, even the sync is much faster
Some Warnings i observed related to that:
Tried the setHead here after adding debug to http.api argument:
this result null after several tries and restart the geth service deleted all chaindata and starts syncing again from here:
// Another print after some hours. Some chances this will get synced?
// Current warnings:
// deleted ethash folder and restarted the service. Invalid mix digest is gone so far // After deleting ethash and wait some time, iam NOW finally synced!
|
Afaict this can now be closed? |
I reproduced this issue with reverting my chaindata. After deleting ethash here again, the sync makes progress. So ethash is sometimes corrupted and does not match to the chaindata. |
I had the same problem. Updated to 1.9.24, cleared ethhas & restarted. Seems to work now. |
@Har01d I have having the same issue. and will like to try your suggestions but I am not sure how to run that command. is it geth debug.setHead(11234872) and should I stop geth first before running it? Error: invalid merkle root (remote: 2aab9430b0e4f85bcaf8d8214aa9eecd9382110229d0f5a70e838d83c8fab2fb local: 9ffb7df423de5f7307643d05a9522360064c584547b669331fc94d522430 7a91) WARN [08-24|05:06:43.691] Synchronisation failed, dropping peer peer=48fe5cb55e1e738350754207e2bd14dc093e5847d63cf33307311f9ed5648203 err="retrieved hash chain is in valid: invalid merkle root (remote: 2aab9430b0e4f85bcaf8d8214aa9eecd9382110229d0f5a70e838d83c8fab2fb local: 9ffb7df423de5f7307643d05a9522360064c584547b669331fc94d522430 7a91)" Number: 13066685 |
Sync block is always failing
Error: invalid merkle root (remote: 57cc91ee8b91b956592a27b14386abc2aba723b5f4f9e5d3181ace6b5d3cd433 local: 1f9ee59bfa683a25c7a15b626995a3ad7c58c571b40df96eea31e5c5eed9732d)
##############################
ERROR[11-11|15:08:37.772]
########## BAD BLOCK #########
Chain config: {ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Byzantium: 4370000 Constantinople: 7280000 Petersburg: 7
280000 Istanbul: 9069000, Muir Glacier: 9200000, Engine: ethash}
Number: 11234873
Hash: 0xd307c642087f1e143e0c7c766e47f77af13e496c8267a55b644bfc86b6f184c7
0: cumulative: 56209 gas: 56209 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x413e13facc3c6e287746616a34deec0ba55356beb8f3f28f8cc4e3522c3a76c5 logs
: [0xc0244c00b0] bloom: 00000000000000000000000000040000000002000000000000000000000000000000000000000000000000020000010000000000000000000000000000000000000000000000000000000008000
00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000010000000000000010000001000000000000
2000000080000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
state:
Geth
Version: 1.9.23-stable
Git Commit: 8c2f271
Git Commit Date: 20201015
Architecture: amd64
Protocol Versions: [65 64 63]
Go Version: go1.15.3
Operating System: linux
GOPATH=
GOROOT=go
The text was updated successfully, but these errors were encountered: