[5.0] mapped_private causes fork database issue

[heifner]

Testing with 5.0 commit 42034e3

Syncing with eos mainnet.

start/stop, start
./nodeos --data-dir dd --config-dir mainconfig2 --plugin eosio::prometheus_plugin --plugin eosio::chain_api_plugin --database-map-mode mapped_private

Note forkdb is ahead of state, so maybe state didn't write out.

info  2023-11-08T16:39:39.897 nodeos    main.cpp:155                  operator()           ] nodeos version v5.0.0-rc2 v5.0.0-rc2-42034e3b39203f7400354607bdd377a5b95d49d6-dirty
info  2023-11-08T16:39:39.897 nodeos    main.cpp:62                   log_non_default_opti ] Non-default options: data-dir = dd, config-dir = mainconfig2, plugin = eosio::prometheus_plugin, plugin = eosio::chain_api_plugin, database-map-mode = mapped_private, wasm-runtime = eos-vm, abi-serializer-max-time-ms = 15, chain-state-db-size-mb = 65536, contracts-console = true, http-server-address = 127.0.0.1:8888, access-control-allow-credentials = false, p2p-listen-endpoint = 127.0.0.1:9876, p2p-max-nodes-per-host = 10, state-history-endpoint = 127.0.0.1:8080, verbose-http-errors = true, agent-name = "Replay Test Node", allowed-connection = any, net-threads = 2, max-transaction-time = 100, max-clients = 1024, sync-fetch-span = 1024, producer-name = eosio, enable-stale-production = true, resource-monitor-not-shutdown-on-threshold-exceeded = true, read-only-read-window-time-us = 150000, read-only-write-window-time-us = 50000, plugin = eosio::chain_api_plugin, plugin = eosio::chain_plugin, plugin = eosio::http_plugin, plugin = eosio::producer_plugin, plugin = eosio::producer_api_plugin, plugin = eosio::net_plugin, plugin = eosio::net_api_plugin, p2p-peer-address = eos.seed.eosnation.io:9876, p2p-peer-address = peer1.eosphere.io:9876, p2p-peer-address = peer2.eosphere.io:9876, p2p-peer-address = p2p.genereos.io:9876
error 2023-11-08T16:39:39.897 nodeos    main.cpp:209                  main                 ] 3020000 fork_database_exception: Fork database exception
fork database head (325811706) is inconsistent with state (325811680)
    {"db":325811680,"head":325811706}
    nodeos  controller.cpp:718 init

    {}
    nodeos  chain_plugin.cpp:1158 plugin_startup

The stop looks clean and there is no core.

info  2023-11-08T16:39:38.512 nodeos    net_plugin.cpp:4603           close_all            ] close all 4 connections
info  2023-11-08T16:39:38.512 net-1     net_plugin.cpp:1458           _close               ] ["eosn-eos-seed171:9876 - 47a4e43" - 1 209.249.216.152:9876] closing
info  2023-11-08T16:39:38.512 nodeos    net_plugin.cpp:4419           plugin_shutdown      ] exit shutdown
info  2023-11-08T16:39:38.513 nodeos    producer_plugin.cpp:1406      plugin_shutdown      ] exit shutdown
CHAINBASE: Writing "state" database file, this could take a moment...
CHAINBASE: Writing "state" database file, complete.
info  2023-11-08T16:39:38.718 nodeos    http_plugin.cpp:515           plugin_shutdown      ] exit shutdown
info  2023-11-08T16:39:38.718 nodeos    main.cpp:155                  operator()           ] nodeos version v5.0.0-rc2 v5.0.0-rc2-42034e3b39203f7400354607bdd377a5b95d49d6-dirty
info  2023-11-08T16:39:38.718 nodeos    main.cpp:62                   log_non_default_opti ] Non-default options: data-dir = dd, config-dir = mainconfig2, plugin = eosio::prometheus_plugin, plugin = eosio::chain_api_plugin, database-map-mode = mapped_private, wasm-runtime = eos-vm, abi-serializer-max-time-ms = 15, chain-state-db-size-mb = 65536, contracts-console = true, http-server-address = 127.0.0.1:8888, access-control-allow-credentials = false, p2p-listen-endpoint = 127.0.0.1:9876, p2p-max-nodes-per-host = 10, state-history-endpoint = 127.0.0.1:8080, verbose-http-errors = true, agent-name = "Replay Test Node", allowed-connection = any, net-threads = 2, max-transaction-time = 100, max-clients = 1024, sync-fetch-span = 1024, producer-name = eosio, enable-stale-production = true, resource-monitor-not-shutdown-on-threshold-exceeded = true, read-only-read-window-time-us = 150000, read-only-write-window-time-us = 50000, plugin = eosio::chain_api_plugin, plugin = eosio::chain_plugin, plugin = eosio::http_plugin, plugin = eosio::producer_plugin, plugin = eosio::producer_api_plugin, plugin = eosio::net_plugin, plugin = eosio::net_api_plugin, p2p-peer-address = eos.seed.eosnation.io:9876, p2p-peer-address = peer1.eosphere.io:9876, p2p-peer-address = peer2.eosphere.io:9876, p2p-peer-address = p2p.genereos.io:9876
info  2023-11-08T16:39:38.718 nodeos    main.cpp:233                  main                 ] nodeos successfully exiting

[heifner]

The db_modes_test on branch [GH-1878-mapped_private_test-5.0](https://github.com/AntelopeIO/leap/tree/GH-1878-mapped_private_test-5.0) fails for me but runs fine in ci/cd: https://github.com/AntelopeIO/leap/actions/runs/6802517987

Maybe an issue with WSL2

[matthewdarwin]

We are using mapped_private on debian 11 or 12. No issues so far.

[greg7mdp]

The db_modes_test on branch [GH-1878-mapped_private_test-5.0](https://github.com/AntelopeIO/leap/tree/GH-1878-mapped_private_test-5.0) fails for me but runs fine in ci/cd: https://github.com/AntelopeIO/leap/actions/runs/6802517987

Maybe an issue with WSL2

I'll give the db_modes_test a try under WSL2 probably tomorrow or Friday.

[heifner]

As pointed out by @spoonincode WSL does not have CONFIG_MEM_SOFT_DIRTY enabled:

zcat /proc/config.gz | grep SOFT_DIRTY
CONFIG_HAVE_ARCH_SOFT_DIRTY=y
# CONFIG_MEM_SOFT_DIRTY is not set