Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Invalid PoW verification resulted in an Orphan block on Ethereum Mainnet #21929

Closed
AlexSSD7 opened this issue Nov 30, 2020 · 6 comments
Closed
Assignees
Labels

Comments

@AlexSSD7
Copy link
Contributor

Yesterday, we caught a very odd orphan block. This work package was submitted via eth_submitWork RPC method:

Nonce: 0xc32b010352953f0d
Header Hash: 0xb439887ea92c6838171da6a5bf04d52b8a89f2647d2736aff43a448b1b933273

Mix Digest: 0xb41266b200574a89cb30c8b6839064a47777d216a33b9f6689769639ccbe4d60

------

["0xc32b010352953f0d", "0xb439887ea92c6838171da6a5bf04d52b8a89f2647d2736aff43a448b1b933273", "0xb41266b200574a89cb30c8b6839064a47777d216a33b9f6689769639ccbe4d60"]

Besides validating this PoW solution using our software, I manually checked the work by passing it to the github.com/ethereum/ethash library and github.com/ethereum/go-ethereum/consensus/ethash module (with checking the work with both hashimotoLight and hashimotoFull functions). All of them marked the work package above as valid, having the actual difficulty of 4.5 PH. This should have been a block; however, Geth rejected the work package while printing these logs:

INFO [11-29|17:52:22.078] Commit new mining work                   number=11355065 sealhash="b43988…933273" uncles=0 txs=144   gas=12480527 fees=0.4400674188 elapsed=51.294ms
WARN [11-29|17:52:23.883] Invalid proof-of-work submitted          sealhash="b43988…933273" elapsed="102.806µs" err="invalid mix digest"
INFO [11-29|17:52:25.079] Commit new mining work                   number=11355065 sealhash="1e01a0…b0100e" uncles=0 txs=147   gas=12466945 fees=0.4463130213 elapsed=52.240ms
ERROR[11-29|17:52:25.697] Failed to generate mapped ethash dataset epoch=379 err="mkdir /.ethash: permission denied"
INFO [11-29|17:52:42.772] Commit new mining work                   number=11355065 sealhash="e824f9…bf5f95" uncles=0 txs=147   gas=12466945 fees=0.4564300281 elapsed=1.546s
INFO [11-29|17:52:43.157] Imported new chain segment               blocks=1    txs=137   mgas=12.468   elapsed=273.623ms   mgasps=45.567   number=11355065 hash="5045d4…dc4dcb" dirty=1022.67MiB

System information

Geth version: 1.9.23-stable (via docker image ethereum/client-go)
OS & Version: Linux x86_64
Commit hash: 8c2f271

Linked issues

#21928 (Caught on a different host, but that issue may be related to this one)

@AlexSSD7 AlexSSD7 changed the title Invalid PoW verification resulted in an Orphan block Invalid PoW verification resulted in an Orphan block on Ethereum Mainnet Nov 30, 2020
@karalabe
Copy link
Member

karalabe commented Dec 1, 2020

There was an ethash DAG bugfix included in 1.9.24 #21793. Based on the code I was convinced that it can only trigger in January, but perhaps I missed something? Could you check if it reproduces with 1.9.24?

@karalabe
Copy link
Member

karalabe commented Dec 1, 2020

One odd thing in your logs is err="mkdir /.ethash: permission denied". Did you change file system permissions recently? Or perhaps pointed Geth to a new ethash folder that's not writeable?

@AlexSSD7
Copy link
Contributor Author

AlexSSD7 commented Dec 1, 2020

Thank you for the reply, @karalabe. That DAG generation bug should affect epochs only higher than 385. The block's epoch was 378, and I don't think that that bug affected the DAG generation at all.

And about the permission denied error: we use a very strict security policy on running containers. Only the datadir volume was writeable by the geth process. Also, I didn't really know that it was using the ~/.ethash directory. Since the user that was running geth had no home directory, it defaulted to /.ethash, the root volume directory.

Also, the bug that caused the wrong PoW was never seen before and after the orphan block. I see that we were very "lucky" to catch that bug.

@rjl493456442
Copy link
Member

rjl493456442 commented Dec 3, 2020

Could you please check the submitted sealhash is paired with the pow solution(nonce, mixdigest)? nvm.
I will try to reproduce

@rjl493456442
Copy link
Member

It's verified that the submission is valid. The difficulty can be recovered is 4567006823027140.

@holiman
Copy link
Contributor

holiman commented Nov 3, 2021

Closing this, seems stale and a one-off event

@holiman holiman closed this as completed Nov 3, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

5 participants