Massive mainnet reorganizations of the Ethereum Classic chain after Phoenix hard fork

[q9f]

• OpenEthereum version: v3.0.0-stable-eee06b3f4
• Operating system: Arch Linux
• Installation: built from source
• Fully synchronized: yes
• Network: classic
• Restarted: no

We keep seeing long reorgs on the Ethereum Classic mainnet with the Open Ethereum client only, some are 50 - 90 blocks long. Other clients are not affected.

2020-06-01 13:23:49  Imported #10503645 0x40aa…fb75 (0 txs, 0.00 Mgas, 17 ms, 0.51 KiB)
2020-06-01 13:24:13    20/25 peers   300 MiB chain 136 MiB db 0 bytes queue 26 MiB sync  RPC:  0 conn,    0 req/s, 3458 µs
2020-06-01 13:24:19  Imported #10503646 0x75f8…ac67 (0 txs, 0.00 Mgas, 34 ms, 0.51 KiB)
2020-06-01 13:24:43    20/25 peers   300 MiB chain 136 MiB db 0 bytes queue 26 MiB sync  RPC:  0 conn,    0 req/s, 3458 µs
2020-06-01 13:25:02  eth_accounts is deprecated and will be removed in future versions: Account management is being phased out see #9997 for alternatives.    
2020-06-01 13:25:13    20/25 peers   300 MiB chain 136 MiB db 0 bytes queue 26 MiB sync  RPC:  0 conn,    0 req/s, 3458 µs
2020-06-01 13:25:35  Imported #10503647 0x03c5…ba57 (0 txs, 0.00 Mgas, 27 ms, 0.51 KiB)
2020-06-01 13:25:43    20/25 peers   300 MiB chain 136 MiB db 0 bytes queue 26 MiB sync  RPC:  0 conn,    1 req/s, 3458 µs
2020-06-01 13:26:18    20/25 peers   300 MiB chain 136 MiB db 0 bytes queue 26 MiB sync  RPC:  0 conn,    0 req/s, 3458 µs
2020-06-01 13:26:23  Imported #10503648 0x4201…e932 (20 txs, 7.33 Mgas, 531 ms, 2.79 KiB)
2020-06-01 13:26:48    20/25 peers   300 MiB chain 136 MiB db 0 bytes queue 26 MiB sync  RPC:  0 conn,    0 req/s, 3722 µs
2020-06-01 13:26:50  eth_accounts is deprecated and will be removed in future versions: Account management is being phased out see #9997 for alternatives.    
2020-06-01 13:27:16  Imported #10503649 0x7f57…e4df (18 txs, 7.28 Mgas, 2597 ms, 2.55 KiB)
2020-06-01 13:27:18    20/25 peers   300 MiB chain 136 MiB db 0 bytes queue 26 MiB sync  RPC:  0 conn,    0 req/s, 3722 µs
2020-06-01 13:27:48    20/25 peers   300 MiB chain 136 MiB db 0 bytes queue 26 MiB sync  RPC:  0 conn,    0 req/s, 3722 µs
2020-06-01 13:28:18    20/25 peers   300 MiB chain 136 MiB db 0 bytes queue 26 MiB sync  RPC:  0 conn,    0 req/s, 3722 µs
2020-06-01 13:28:48    20/25 peers   300 MiB chain 136 MiB db 736 bytes queue 26 MiB sync  RPC:  0 conn,    0 req/s, 3722 µs
2020-06-01 13:28:48  Imported #10503650 0x0278…2c74 (13 txs, 7.17 Mgas, 193 ms, 2.00 KiB)
2020-06-01 13:28:50  Imported #10503651 0x5721…9f58 (15 txs, 0.37 Mgas, 300 ms, 2.47 KiB)
2020-06-01 13:29:08  eth_accounts is deprecated and will be removed in future versions: Account management is being phased out see #9997 for alternatives.    
2020-06-01 13:29:18    20/25 peers   300 MiB chain 136 MiB db 0 bytes queue 26 MiB sync  RPC:  0 conn,    1 req/s, 4122 µs
2020-06-01 13:29:33  Imported #10503652 0xb77f…9f5e (3 txs, 0.06 Mgas, 58 ms, 0.84 KiB)
2020-06-01 13:29:48    19/25 peers   300 MiB chain 136 MiB db 0 bytes queue 26 MiB sync  RPC:  0 conn,    0 req/s, 4122 µs
2020-06-01 13:30:23  Syncing #10503652 0xb77f…9f5e     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed  #10503640   17/25 peers   300 MiB chain 136 MiB db 0 bytes queue 26 MiB sync  RPC:  0 conn,    0 req/s, 4122 µs
2020-06-01 13:30:28  Syncing #10503653 0x817c…7b9a     0.20 blk/s    0.2 tx/s    0.0 Mgas/s      0+    0 Qed  #10503626   17/25 peers   300 MiB chain 136 MiB db 0 bytes queue 26 MiB sync  RPC:  0 conn,    0 req/s, 4122 µs
2020-06-01 13:30:33  Syncing #10503654 0x1008…5acf     0.20 blk/s    1.4 tx/s    1.4 Mgas/s      0+    0 Qed  #10503645   19/25 peers   300 MiB chain 136 MiB db 0 bytes queue 26 MiB sync  RPC:  0 conn,    0 req/s, 4122 µs
2020-06-01 13:30:38  Syncing #10503654 0x1008…5acf     0.00 blk/s    0.0 tx/s    0.0 Mgas/s      0+    0 Qed  #10503608   18/25 peers   300 MiB chain 136 MiB db 0 bytes queue 26 MiB sync  RPC:  0 conn,    0 req/s, 4122 µs
2020-06-01 13:30:41  50 -> GetNodeData:   item 0/11 – slow state fetch for hash 0x196ed922b5439ecc69dac4e54b1e918ceab9e5048316bcaf7bbfc99169519fba; took 140.127555ms, total 140.127555ms
2020-06-01 13:30:42  50 -> GetNodeData:   item 0/27 – slow state fetch for hash 0x9f95a6d1a1cc5068c98f2230c9054a6ce1e7c38192af0f81d0b318f938b4e0e0; took 195.815437ms, total 195.815437ms
2020-06-01 13:30:43  Syncing #10503654 0x1008…5acf    12.96 blk/s   44.0 tx/s    1.0 Mgas/s      0+  110 Qed  #10503740   11/25 peers   300 MiB chain 136 MiB db 798 KiB queue 26 MiB sync  RPC:  0 conn,    0 req/s, 4122 µs
2020-06-01 13:30:44  50 -> GetNodeData:   item 21/23 – slow state fetch for hash 0xda342ba17f1a37ec70ae7104948232ed2d8e7e29a8804b52a2a4708bdcd71254; took 204.404712ms, total 368.05115ms
2020-06-01 13:30:44  Reorg to #10503650 0x9e46…3eec (0x1008…5acf 0x817c…7b9a 0xb77f…9f5e 0x5721…9f58 0x0278…2c74 0x7f57…e4df 0x4201…e932 0x03c5…ba57 0x75f8…ac67 0x40aa…fb75 0x59df…1803 0x39d7…9f0e 0x549a…1064 0x8008…fb77 0x2de8…d4b1 0xe32d…71fd 0xcaca…3275 0xf3b0…288d 0xd5ad…3468 0x354b…d426 0x479d…8205 0x2ac4…cd86 
0xa8fc…988c 0xf3f0…8278 0x67a4…129c 0xf4df…973c 0x041d…aa21 0xd0a3…6429 0xc5a0…5471 0x33c3…4519 0x6912…42b9 0x77fa…e7d8 0x88a5…c64c 0x61c7…7a6b 0x8037…d833 0xaaaf…4b67 0xc986…ed40 0x1ab6…10cb 0x13f0…1ff4 0x947f…8f14 0x6c08…9a4c 0x4e16…640e 0x3906…15e8 0xde15…83e1 0x52c6…6fcc 0xcd6f…6fce 0x7345…b8d3 0xd50d…e203 0x678
1…8144 0x5e48…d176 0x256a…b9fd 0x5d57…9518 0x29cb…d242 0xc41a…08b2 0x8a97…9df8 0x2f5b…e68c 0x72cb…d301 0x9b05…370d 0x974e…0458 0xe60b…6184 0x0b5d…8d8a 0xff27…90dc 0xb863…c1a8 0xe048…e222 0x76e8…342e 0x6909…849e 0x4eb3…3389 0x5669…523c 0x5392…8fac 0xab91…3ff8 0x136a…cfe5 0xd327…c106 0x5a8f…e18c 0x9c57…63f2 0x21d3…530
8 0xf59d…3ac5 0x7f75…1798 0xacf0…5744 0xa14b…cd56 0xe126…7998 0xc58c…fe02 0x0f83…558e 0xbfdb…5247 0x511d…6015 0xbad0…31ba 0xfed2…4bee 0x533d…50ee 0x1ad7…184f 0x05f3…5d38 0x02a4…ae10 0xd72a…ab66 0xaaf5…da01 0xc71c…00c9 0x5726…bcbf 0x34bb…780e 0x35c7…2aa3 0x2b44…95e4 0xf6b5…a8d1 0x4767…4fcc 0x0433…4ab6 0x93d5…bab3 0xc
039…4248 0x1559…b820 0xf087…bd45 #10503550 0xfa71…3423 0x3c41…1773 0x40a4…73c4 0x5186…f59c 0x36ec…2690 0xdb43…e2e5 0x2570…2c25 0x47c2…4de8 0xca1c…fdba 0x09f0…6f32 0x0211…9199 0x2c8b…8d27 0xad83…41c2 0x7d1a…a9c8 0x7596…5d2f 0x5003…e698 0xb16b…6f72 0xfa72…9602 0x26e7…bec2 0x91e6…c20d 0x3f17…c93a 0xca57…d1b3 0xec12…0fe
4 0xc448…ae3c 0x08c4…545a 0xfdac…3920 0xf353…6d5e 0xb5c2…e972 0x71db…db7b 0x84bf…5c57 0xb067…b746 0x5726…6684 0x024f…f197 0x0246…9124 0x5732…2639 0xac8a…e52b 0xb390…5746 0x1e65…b8a3 0x2c80…89cd 0x2c3e…e59d 0xa682…7f15 0xb4af…6be0 0x4c79…f33f 0x0a65…97eb 0xe24b…48f3 0x286e…3d0e 0x2d06…943e 0x6d0c…a27f 0x15ee…3dd1 0xc
d03…5382 0xdf7b…6372 0xca96…fbb1 0x939c…43a8 0x07cb…b4b1 0x79d2…f5db 0x86b1…32b1 0xf0a7…e4d7 0x6fba…c7c8 0x13cb…1d38 0x4b86…b6d1 0xf1e3…3d1b 0xda21…5641 0x94fa…2d17 0x4e8d…854f 0x209b…ae7f 0x908c…9c57 0x3327…275e 0xbd6a…53be 0xee8b…13d7 0xb83e…b789 0x74ed…0571 0xe79c…2ee5 0xb215…42a1 0x57ce…77f9 0xc95d…373f 0x1352…d
bd7 0x38fc…6583 0xaa3f…033e 0x1bf7…4e30 0x27dc…ccf5 0x2e07…0749 0xb34b…69a0 0x1dfc…a97c 0x7594…e7df 0x37ad…b145 0x9824…d9d5 0x191b…1ccd 0x7151…810e 0xf36e…a30a 0xf90b…ff06 0xf6c0…16f3 0xda9e…be7b 0xdcd1…3471 0xfdf3…3680 0xc65f…86cd 0x2d7b…42ae 0x7014…1907 0x7966…14c7 0xa1c0…a470 0xfd54…4675)
2020-06-01 13:30:47  Imported #10503741 0x7cc2…411d (3 txs, 0.11 Mgas, 8 ms, 1.05 KiB) + another 5 block(s) containing 13 tx(s)
2020-06-01 13:30:49  eth_accounts is deprecated and will be removed in future versions: Account management is being phased out see #9997 for alternatives.    
2020-06-01 13:30:50  50 -> GetNodeData:   item 7/26 – slow state fetch for hash 0x48ac51e9ce82a462cafa114595a403c7a6bde455ae01860b7baf9630a6a51520; took 244.614273ms, total 524.904671ms
2020-06-01 13:30:51  Imported #10503742 0x9633…7fd0 (3 txs, 0.06 Mgas, 416 ms, 0.86 KiB)
2020-06-01 13:30:54  Imported #10503743 0x0aaf…6b5b (0 txs, 0.00 Mgas, 18 ms, 0.52 KiB)
2020-06-01 13:31:06  Imported #10503744 0x9a52…736a (2 txs, 0.04 Mgas, 35 ms, 0.75 KiB)
2020-06-01 13:31:16  Imported #10503745 0x2b17…c8e5 (0 txs, 0.00 Mgas, 15 ms, 0.53 KiB)
2020-06-01 13:31:18    16/25 peers   301 MiB chain 133 MiB db 0 bytes queue 26 MiB sync  RPC:  0 conn,    0 req/s, 4312 µs
2020-06-01 13:31:27  Imported #10503746 0xbdc2…39a0 (0 txs, 0.00 Mgas, 36 ms, 0.53 KiB)
2020-06-01 13:31:35  Imported #10503747 0xa3ce…8561 (0 txs, 0.00 Mgas, 38 ms, 0.53 KiB)
2020-06-01 13:31:37  Imported #10503748 0x7f0c…196d (0 txs, 0.00 Mgas, 13 ms, 0.52 KiB)
2020-06-01 13:31:48    16/25 peers   301 MiB chain 133 MiB db 0 bytes queue 26 MiB sync  RPC:  0 conn,    0 req/s, 4312 µs
2020-06-01 13:31:49  Imported #10503749 0x9037…e769 (0 txs, 0.00 Mgas, 23 ms, 0.51 KiB)
2020-06-01 13:31:58  50 -> GetNodeData:   item 4/19 – slow state fetch for hash 0x0e47a812aa10b96b2d2df6345e5739616f68705f6996ff381a0b4ca29b1e6864; took 237.298498ms, total 469.596075ms
2020-06-01 13:31:58  50 -> GetNodeData:   item 1/17 – slow state fetch for hash 0x2c93288d54f8d49f0795cdfc16beb6c86c8e595b0edb9ae77be234c0e70ac3e2; took 178.094221ms, total 197.166695ms
2020-06-01 13:31:59  50 -> GetNodeData:   item 10/16 – slow state fetch for hash 0xb04e225fd2972aad45de92bdebb1ece449ed0e404a625b2ee6bbceb05690be04; took 106.411165ms, total 281.528526ms
2020-06-01 13:31:59  Imported #10503751 0xd88d…a245 (0 txs, 0.00 Mgas, 78 ms, 0.51 KiB)
2020-06-01 13:32:03  Imported #10503752 0x05fa…77d6 (0 txs, 0.00 Mgas, 230 ms, 0.52 KiB)
2020-06-01 13:32:04  Imported #10503753 0x8cd9…3d8e (0 txs, 0.00 Mgas, 75 ms, 0.52 KiB)
2020-06-01 13:32:18    17/25 peers   302 MiB chain 133 MiB db 0 bytes queue 26 MiB sync  RPC:  0 conn,    0 req/s, 4312 µs
2020-06-01 13:32:24  Reorg to #10503754 0xe68d…67c8 (0x8cd9…3d8e #10503752 0x05fa…77d6 0x6d3e…5f26)
2020-06-01 13:32:25  Imported #10503754 0xe68d…67c8 (0 txs, 0.00 Mgas, 48 ms, 0.52 KiB) + another 1 block(s) containing 0 tx(s)
2020-06-01 13:32:30  eth_accounts is deprecated and will be removed in future versions: Account management is being phased out see #9997 for alternatives.
2020-06-01 13:32:48    17/25 peers   302 MiB chain 133 MiB db 0 bytes queue 26 MiB sync  RPC:  0 conn,    0 req/s, 3458 µs
2020-06-01 13:32:56  Imported #10503755 0xd06f…e897 (3 txs, 0.06 Mgas, 50 ms, 1.36 KiB)
2020-06-01 13:33:18    17/25 peers   303 MiB chain 133 MiB db 0 bytes queue 26 MiB sync  RPC:  0 conn,    0 req/s, 3458 µs
2020-06-01 13:33:26  Imported #10503756 0x7960…fbbe (0 txs, 0.00 Mgas, 105 ms, 0.52 KiB)
2020-06-01 13:33:27  Imported #10503757 0x7941…105d (15 txs, 0.31 Mgas, 91 ms, 2.18 KiB)
2020-06-01 13:33:41  Imported #10503758 0xde48…bd94 (0 txs, 0.00 Mgas, 63 ms, 0.52 KiB)
2020-06-01 13:33:44  Imported #10503759 0xcff0…9ba7 (14 txs, 0.52 Mgas, 392 ms, 2.57 KiB)
2020-06-01 13:33:47  Imported #10503760 0x2c41…0803 (0 txs, 0.00 Mgas, 87 ms, 0.52 KiB)
2020-06-01 13:33:48    17/25 peers   303 MiB chain 133 MiB db 0 bytes queue 26 MiB sync  RPC:  0 conn,    0 req/s, 3458 µs
2020-06-01 13:33:50  Imported #10503761 0xda77…1db0 (1 txs, 0.02 Mgas, 18 ms, 0.63 KiB)

[q9f]

It's not Ethereum Classic that reorganizes but the Open Ethereum client. The title might be misleading as other clients are not reorganizing.

[sorpaas]

I checked ETC folks' discussions at their Discord, Besu is also suffering some issues after Phoenix hard fork. So looks like the only implementations not affected are Geth-family ones.

@q9f

It's not Ethereum Classic that reorganizes but the Open Ethereum client. Your title is mildly misleading as other clients are not reorganizing.

We welcome discussions here, but please stop provoking others.

[q9f]

Excuse my tone. After rereading my initial bug report, it turns out I was not mentioning that bit and redacted the comment accordingly.

[q9f]

Open Ethereum does not appear peering with the other clients.

curl -s --data '{"method":"parity_netPeers","params":[],"id":1,"jsonrpc":"2.0"}' -H "Content-Type: application/json" -X POST localhost:8545 | jq . | grep '"name": "'
            "name": "OpenEthereum",
            "name": "OpenEthereum",
            "name": "OpenEthereum",
            "name": "Parity-Ethereum",
            "name": "Parity-Ethereum",
            "name": "OpenEthereum",
            "name": "OpenEthereum",
            "name": "Parity-Ethereum",
            "name": "OpenEthereum",
            "name": "Parity-Ethereum",
            "name": "OpenEthereum",
            "name": "Parity-Ethereum",
            "name": "OpenEthereum",
            "name": "OpenEthereum",
            "name": "OpenEthereum",
            "name": "OpenEthereum",
            "name": "OpenEthereum",
            "name": "OpenEthereum",
            "name": "Parity-Ethereum",
            "name": "Parity-Ethereum",
            "name": "OpenEthereum",
            "name": "Parity-Ethereum",
            "name": "OpenEthereum",
            "name": "OpenEthereum",
            "name": "OpenEthereum",
            "name": "Parity-Ethereum",
            "name": "OpenEthereum",
            "name": "Parity-Ethereum",
            "name": "OpenEthereum",
            "name": "Parity-Ethereum",
            "name": "OpenEthereum",
            "name": "Parity-Ethereum",
            "name": "OpenEthereum",
            "name": "Parity-Ethereum",
            "name": "OpenEthereum",
            "name": "Parity-Ethereum",
            "name": "Parity-Ethereum",
            "name": "OpenEthereum",
            "name": "OpenEthereum",
            "name": "OpenEthereum",