Problem: memiavl background snapshot writing don't limit concurrency level

CHANGELOG.md

@@ -77,6 +77,7 @@
 - [#1149](https://github.com/crypto-org-chain/cronos/pull/1149) memiavl support `WorkingHash` api required by `FinalizeBlock`.
 - [#1151](https://github.com/crypto-org-chain/cronos/pull/1151) memiavl `CacheMultiStoreWithVersion` supports `io.Closer`.
 - [#1154](https://github.com/crypto-org-chain/cronos/pull/1154) Remove dependency on cosmos-sdk.
+- [#1171](https://github.com/crypto-org-chain/cronos/pull/1171) Add memiavl background snapshot writing concurrency limit.
 
 *April 13, 2023*
 
@@ -90,21 +91,21 @@
 
 ### Bug Fixes
 
-* [#953](https://github.com/crypto-org-chain/cronos/pull/953) Include third-party bug fixes:
+- [#953](https://github.com/crypto-org-chain/cronos/pull/953) Include third-party bug fixes:
   - update ethermint to include two bug fixes
-    - https://github.com/crypto-org-chain/ethermint/pull/234
-    - https://github.com/crypto-org-chain/ethermint/pull/233
+    - <https://github.com/crypto-org-chain/ethermint/pull/234>
+    - <https://github.com/crypto-org-chain/ethermint/pull/233>
   - update cosmos-sdk to include one bug fix
-    - https://github.com/cosmos/cosmos-sdk/pull/15667
-* [#945](https://github.com/crypto-org-chain/cronos/pull/945) Fix no handler exists for proposal type error when update-client due to wrong ibc route.
+    - <https://github.com/cosmos/cosmos-sdk/pull/15667>
+- [#945](https://github.com/crypto-org-chain/cronos/pull/945) Fix no handler exists for proposal type error when update-client due to wrong ibc route.
 
 *Mar 16, 2023*
 
 ## v1.0.6
 
 ### Bug Fixes
 
-* [#932](https://github.com/crypto-org-chain/cronos/pull/932) Backport multiple json-rpc bug fixes in ethermint ([commits](https://github.com/crypto-org-chain/ethermint/compare/v0.20.8-cronos...v0.20.9-cronos)).
+- [#932](https://github.com/crypto-org-chain/cronos/pull/932) Backport multiple json-rpc bug fixes in ethermint ([commits](https://github.com/crypto-org-chain/ethermint/compare/v0.20.8-cronos...v0.20.9-cronos)).
 
 *Mar 6, 2023*
 
@@ -243,6 +244,7 @@
 ## v0.9.0
 
 ### State Machine Breaking
+
 - [cronos#429](https://github.com/crypto-org-chain/cronos/pull/429) Update ethermint to main, ibc-go to v3.0.0, cosmos sdk to v0.45.4 and gravity to latest, remove v0.7.0 related upgradeHandler.
 - [cronos#532](https://github.com/crypto-org-chain/cronos/pull/532) Add SendtoChain and CancelSendToChain support from evm call.
 - [cronos#600](https://github.com/crypto-org-chain/cronos/pull/600) Implement bidirectional token mapping.
@@ -258,6 +260,7 @@
 - [cronos#639](https://github.com/crypto-org-chain/cronos/pull/639) init and validate-genesis commands don't include experimental modules by default.
 
 ### Improvements
+
 - [cronos#418](https://github.com/crypto-org-chain/cronos/pull/418) Support logs in evm-hooks and return id for SendToEthereum events
 - [cronos#489](https://github.com/crypto-org-chain/cronos/pull/489) Enable jemalloc memory allocator, and update rocksdb src to `v6.29.5`.
 - [cronos#511](https://github.com/crypto-org-chain/cronos/pull/511) Replace ibc-hook with ibc middleware, use ibc-go upstream version.
@@ -313,17 +316,19 @@
 - [cronos#315](https://github.com/crypto-org-chain/cronos/pull/315) Update cosmos-sdk to `v0.45.0`
 
 ### Improvements
+
 - [cronos#210](https://github.com/crypto-org-chain/cronos/pull/210) re-enabling gravity bridge conditionally
 - [cronos#322](https://github.com/crypto-org-chain/cronos/pull/322) Merge min-gas-price change in ethermint: don't check min-gas-price for EVM tx when feemarket enabled.
 - [cronos#345](https://github.com/crypto-org-chain/cronos/pull/345) disable the url query parameter in swagger-ui.
 - [cronos#328](https://github.com/crypto-org-chain/cronos/pull/328) display detail panic information in query result when `--trace` enabled.
 - [cronos#441](https://github.com/crypto-org-chain/cronos/pull/441) Update cosmos-sdk to `v0.45.4`
 
 ### Bug Fixes
+
 - [cronos#287](https://github.com/crypto-org-chain/cronos/pull/287) call upgrade handler before sealing app
 - [cronos#323](https://github.com/crypto-org-chain/cronos/pull/323) Upgrade gravity bridge to v0.3.9 which contain a bugfix on `batchTxExecuted.`
 - [cronos#324](https://github.com/crypto-org-chain/cronos/pull/324) Update to cosmos-sdk `v0.45.1`, which fixes an OOM issue.
-- [cronos#329](https://github.com/crypto-org-chain/cronos/pull/329) Fix panic of eth_call on blocks prior to upgrade. 
+- [cronos#329](https://github.com/crypto-org-chain/cronos/pull/329) Fix panic of eth_call on blocks prior to upgrade.
 - [cronos#340](https://github.com/crypto-org-chain/cronos/pull/340) Update dependencies to include several bug fixes: a) fix subscription deadlock issue in ethermint, b) fix data races `traceContext`.
 - [cronos#370](https://github.com/crypto-org-chain/cronos/pull/370) Update ethermint to fix a websocket bug, add websockets integration tests.
 - [cronos#378](https://github.com/crypto-org-chain/cronos/pull/378) Backport recent ethermint bug fixes: a) fix tx inclusion issue by report correct gasWanted of eth tx, b) Add buffer to eth_gasPrice response to fix client UX, c) Quick fix for eth_feeHistory when reward is nil, d) add returnValue message on tracing.
@@ -343,6 +348,7 @@
 ## v0.6.4
 
 ### Bug Fixes
+
 - [crypto-org-chain/ethermint#19](https://github.com/crypto-org-chain/ethermint/pull/19) revert tharsis#786 because it contains consensus breaking changes
 
 *November 29, 2021*
@@ -360,12 +366,12 @@
 - [tharsis#786](https://github.com/tharsis/ethermint/pull/786) Improve error message of `SendTransaction`/`SendRawTransaction` JSON-RPC APIs.
 - [cronos#222](https://github.com/crypto-org-chain/cronos/pull/222) change solc 0.6.11 to 0.6.8 (from dapp cachix) and update hermes to 0.8.
 
-
 *November 19, 2021*
 
 ## v0.6.2
 
 ### Bug Fixes
+
 - [tharsis#720](https://github.com/tharsis/ethermint/pull/720) traceTransaction fails for succesful tx
 - [tharsis#743](https://github.com/tharsis/ethermint/pull/743) missing debug_tranceBlockByHash RPC method and fix debug_traceBlock*
 - [tharsis#746](https://github.com/tharsis/ethermint/pull/746) set debug based on tracer
@@ -377,16 +383,16 @@
 ## v0.6.1
 
 ### State Machine Breaking
+
 - [cronos#190](https://github.com/crypto-org-chain/cronos/pull/190) upgrade ethermint to v0.7.2 with (#661) and (#689)
 
 ### Bug Fixes
+
 - [cronos#187](https://github.com/crypto-org-chain/cronos/pull/187) multiple denoms can be mapped to same contract
 - [cronos#157](https://github.com/crypto-org-chain/cronos/pull/185) cronos params name has an unnecessary Key prefix
 - [cronos#179](https://github.com/crypto-org-chain/cronos/pull/179) fix denom (symbol) in CRC20Module
 - [cronos#178](https://github.com/crypto-org-chain/cronos/pull/178) version CLI command doesn't output any text
 
-
-
 *October 13, 2021*
 
 ## v0.6.0
@@ -398,16 +404,19 @@ This version removes gravity-bridge from cronos, also includes multiple bug fixe
 - [cronos#171](https://github.com/crypto-org-chain/cronos/pull/171) remove gravity-bridge for mainnet launch
 
 ### Bug Fixes
+
 - [cronos#144](https://github.com/crypto-org-chain/cronos/pull/144) fix events in autodeploy crc20 module contract
 - [gravity-bridge#17](https://github.com/crypto-org-chain/gravity-bridge/pull/17) processEthereumEvent does not persist hooks emitted event
 - [gravity-bridge#20](https://github.com/crypto-org-chain/gravity-bridge/pull/20) fix undeterministic in consensus
 - [cronos#167](https://github.com/crypto-org-chain/cronos/pull/167) upgrade cosmos-sdk to 0.44.2
 
 ### Improvements
+
 - [cronos#162](https://github.com/crypto-org-chain/cronos/pull/162) bump ibc-go to v1.2.1 with hooks support
 - [cronos#169](https://github.com/crypto-org-chain/cronos/pull/169) bump ethermint to v0.7.1 and go-ethereum to v10.1.3-patched which include (CVE-2021-39137) hotfix
 
 *October 4, 2021*
+
 ## v0.5.5
 
 This version fixes various bugs regarding ibc fund transfer and EVM-related in ethermint.
@@ -428,6 +437,7 @@ We also enable swagger doc ui and add the token mapping state in genesis.
 - [cronos#128](https://github.com/crypto-org-chain/cronos/pull/128) add native message to update token mapping
 
 *September 22, 2021*
+
 ## v0.5.4
 
 This version is the same as v0.5.3 with a patched version of ethermint which include a bug fix on the transaction receipts events and on concurrent query.
@@ -438,12 +448,14 @@ This version is the same as v0.5.3 with a patched version of ethermint which inc
 - [cronos#98](https://github.com/crypto-org-chain/cronos/pull/98) node crash under concurrent query
 
 *September 21, 2021*
+
 ## v0.5.3
 
 This version contains several new features, it enables gravity bridge in Cronos and automatic token conversion for bridging tokens to crc20 tokens. It also fixes the decimal conversion issues in the CRO tokens from Crypto.org Chain.
 In addition to that, it also upgrade ethermint to its latest version (v0.5.0.x) which bring several breaking changes (see [changelog](https://github.com/tharsis/ethermint/blob/1a01c6a992c0fb94d70bb1c7127715874cefd057/CHANGELOG.md)).
 
 ### Consensus breaking changes
+
 - [cronos#87](https://github.com/crypto-org-chain/cronos/pull/87) upgrade ethermint to v0.4.2-0.20210920104419-1a01c6a992c0
 
 ### Features
@@ -474,8 +486,6 @@ In addition to that, it also upgrade ethermint to its latest version (v0.5.0.x)
 
 - (ethermint) [tharsis#446](https://github.com/tharsis/ethermint/pull/446) fix chain state export issue
 
-
-
 *August 16, 2021*
 
 ## v0.5.1

go.mod

@@ -53,7 +53,7 @@ require (
 	github.com/HdrHistogram/hdrhistogram-go v1.1.2 // indirect
 	github.com/StackExchange/wmi v0.0.0-20180116203802-5d049714c4a6 // indirect
 	github.com/VictoriaMetrics/fastcache v1.6.0 // indirect
-	github.com/alitto/pond v1.8.2 // indirect
+	github.com/alitto/pond v1.8.3 // indirect
 	github.com/aws/aws-sdk-go v1.44.203 // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
 	github.com/bgentry/go-netrc v0.0.0-20140422174119-9fd32a8b3d3d // indirect

go.sum

@@ -281,8 +281,8 @@ github.com/alecthomas/units v0.0.0-20190717042225-c3de453c63f4/go.mod h1:ybxpYRF
 github.com/alecthomas/units v0.0.0-20190924025748-f65c72e2690d/go.mod h1:rBZYJk541a8SKzHPHnH3zbiI+7dagKZ0cgpgrD7Fyho=
 github.com/alexkohler/prealloc v1.0.0/go.mod h1:VetnK3dIgFBBKmg0YnD9F9x6Icjd+9cvfHR56wJVlKE=
 github.com/alingse/asasalint v0.0.10/go.mod h1:nCaoMhw7a9kSJObvQyVzNTPBDbNpdocqrSP7t/cW5+I=
-github.com/alitto/pond v1.8.2 h1:k0k3GIE7CFLW/kyMJj5DDKLFg1VH09l8skZqg/yJNng=
-github.com/alitto/pond v1.8.2/go.mod h1:CmvIIGd5jKLasGI3D87qDkQxjzChdKMmnXMg3fG6M6Q=
+github.com/alitto/pond v1.8.3 h1:ydIqygCLVPqIX/USe5EaV/aSRXTRXDEI9JwuDdu+/xs=
+github.com/alitto/pond v1.8.3/go.mod h1:CmvIIGd5jKLasGI3D87qDkQxjzChdKMmnXMg3fG6M6Q=
 github.com/allegro/bigcache v1.2.1-0.20190218064605-e24eb225f156/go.mod h1:Cb/ax3seSYIx7SuZdm2G2xzfwmv3TPSk2ucNfQESPXM=
 github.com/allegro/bigcache v1.2.1 h1:hg1sY1raCwic3Vnsvje6TT7/pnZba83LeFck5NrFKSc=
 github.com/andybalholm/brotli v1.0.2/go.mod h1:loMXtMfwqflxFJPmdbJO0a3KNoPuLBgiu3qAvBg8x/Y=

gomod2nix.toml

@@ -66,8 +66,8 @@ schema = 3
     version = "v1.6.0"
     hash = "sha256-u1dkRJ2Y5+hnYlkyMPm14HxKkAv999bjN622nZDjaBo="
   [mod."github.com/alitto/pond"]
-    version = "v1.8.2"
-    hash = "sha256-GbTih6gDKGuCP0V0iEo1DfEePAjm6lBxiUpqmL8hD1I="
+    version = "v1.8.3"
+    hash = "sha256-Iaiw4a8Up0tB2iu8xdmXKHwXBGWBBGnqMGfZYWNc1ng="
   [mod."github.com/armon/go-metrics"]
     version = "v0.4.1"
     hash = "sha256-usxTUHA0QQMdM6sHi2z51nmnEKMbA0qUilxJFpWHlYE="

memiavl/db.go

@@ -11,13 +11,15 @@ import (
 	"sync"
 	"time"
 
+	"github.com/alitto/pond"
 	"github.com/cosmos/iavl"
 	"github.com/tidwall/wal"
 )
 
 const (
-	DefaultSnapshotInterval = 1000
-	LockFileName            = "LOCK"
+	DefaultSnapshotInterval    = 1000
+	LockFileName               = "LOCK"
+	DefaultSnapshotWriterLimit = 4
 )
 
 var errReadOnly = errors.New("db is read-only")
@@ -72,6 +74,8 @@ type DB struct {
 	//   this method is the sole entry point for tree modifications, and there's no concurrency internally
 	//   (the background snapshot rewrite is handled separately), so we don't need locks in the Tree.
 	mtx sync.Mutex
+	// worker goroutine IdleTimeout = 5s
+	snapshotWriterPool *pond.WorkerPool
 }
 
 type Options struct {
@@ -97,6 +101,8 @@ type Options struct {
 	// truncate the versions after the `TargetVersion`, the `TargetVersion` becomes the latest version.
 	// it do nothing if the target version is `0`.
 	LoadForOverwriting bool
+
+	SnapshotWriterLimit int
 }
 
 func (opts Options) Validate() error {
@@ -119,6 +125,10 @@ func (opts *Options) FillDefaults() {
 	if opts.SnapshotInterval == 0 {
 		opts.SnapshotInterval = DefaultSnapshotInterval
 	}
+
+	if opts.SnapshotWriterLimit <= 0 {
+		opts.SnapshotWriterLimit = DefaultSnapshotWriterLimit
+	}
 }
 
 const (
@@ -217,6 +227,8 @@ func Load(dir string, opts Options) (*DB, error) {
 			return nil, fmt.Errorf("fail to prune snapshots: %w", err)
 		}
 	}
+	// create worker pool. recv tasks to write snapshot
+	workerPool := pond.New(opts.SnapshotWriterLimit, opts.SnapshotWriterLimit*10)
 
 	db := &DB{
 		MultiTree:              *mtree,
@@ -229,6 +241,7 @@ func Load(dir string, opts Options) (*DB, error) {
 		snapshotKeepRecent:     opts.SnapshotKeepRecent,
 		snapshotInterval:       opts.SnapshotInterval,
 		triggerStateSyncExport: opts.TriggerStateSyncExport,
+		snapshotWriterPool:     workerPool,
 	}
 
 	if !db.readOnly && db.Version() == 0 && len(opts.InitialStores) > 0 {
@@ -604,10 +617,12 @@ func (db *DB) Copy() *DB {
 
 func (db *DB) copy(cacheSize int) *DB {
 	mtree := db.MultiTree.Copy(cacheSize)
+
 	return &DB{
-		MultiTree: *mtree,
-		logger:    db.logger,
-		dir:       db.dir,
+		MultiTree:          *mtree,
+		logger:             db.logger,
+		dir:                db.dir,
+		snapshotWriterPool: db.snapshotWriterPool,
 	}
 }
 
@@ -623,7 +638,7 @@ func (db *DB) RewriteSnapshot() error {
 	snapshotDir := snapshotName(db.lastCommitInfo.Version)
 	tmpDir := snapshotDir + "-tmp"
 	path := filepath.Join(db.dir, tmpDir)
-	if err := db.MultiTree.WriteSnapshot(path); err != nil {
+	if err := db.MultiTree.WriteSnapshot(path, db.snapshotWriterPool); err != nil {
 		return errors.Join(err, os.RemoveAll(path))
 	}
 	if err := os.Rename(path, filepath.Join(db.dir, snapshotDir)); err != nil {
@@ -801,7 +816,7 @@ func (db *DB) WriteSnapshot(dir string) error {
 	db.mtx.Lock()
 	defer db.mtx.Unlock()
 
-	return db.MultiTree.WriteSnapshot(dir)
+	return db.MultiTree.WriteSnapshot(dir, db.snapshotWriterPool)
 }
 
 func snapshotName(version int64) string {
@@ -901,7 +916,11 @@ func walPath(root string) string {
 func initEmptyDB(dir string, initialVersion uint32) error {
 	tmp := NewEmptyMultiTree(initialVersion, 0)
 	snapshotDir := snapshotName(0)
-	if err := tmp.WriteSnapshot(filepath.Join(dir, snapshotDir)); err != nil {
+	// create tmp worker pool
+	pool := pond.New(DefaultSnapshotWriterLimit, DefaultSnapshotWriterLimit*10)
+	defer pool.Stop()
+
+	if err := tmp.WriteSnapshot(filepath.Join(dir, snapshotDir), pool); err != nil {
 		return err
 	}
 	return updateCurrentSymlink(dir, snapshotDir)

memiavl/export.go

@@ -31,9 +31,10 @@ func NewMultiTreeExporter(dir string, version uint32, supportExportNonSnapshotVe
 	)
 	if supportExportNonSnapshotVersion {
 		db, err = Load(dir, Options{
-			TargetVersion: version,
-			ZeroCopy:      true,
-			ReadOnly:      true,
+			TargetVersion:       version,
+			ZeroCopy:            true,
+			ReadOnly:            true,
+			SnapshotWriterLimit: DefaultSnapshotWriterLimit,
 		})
 		if err != nil {
 			return nil, fmt.Errorf("invalid height: %d, %w", version, err)

memiavl/go.mod

@@ -3,6 +3,7 @@ module github.com/crypto-org-chain/cronos/memiavl
 go 1.20
 
 require (
+	github.com/alitto/pond v1.8.3
 	github.com/cometbft/cometbft-db v0.8.0
 	github.com/confio/ics23/go v0.9.0
 	github.com/cosmos/gogoproto v1.4.10
@@ -14,7 +15,6 @@ require (
 	github.com/tidwall/wal v1.1.7
 	github.com/zbiljic/go-filelock v0.0.0-20170914061330-1dbf7103ab7d
 	golang.org/x/exp v0.0.0-20230711153332-06a737ee72cb
-	golang.org/x/sync v0.1.0
 )
 
 require (

memiavl/go.sum

@@ -49,6 +49,8 @@ github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751/go.mod h1:LOuy
 github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
 github.com/alecthomas/units v0.0.0-20190717042225-c3de453c63f4/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
 github.com/alecthomas/units v0.0.0-20190924025748-f65c72e2690d/go.mod h1:rBZYJk541a8SKzHPHnH3zbiI+7dagKZ0cgpgrD7Fyho=
+github.com/alitto/pond v1.8.3 h1:ydIqygCLVPqIX/USe5EaV/aSRXTRXDEI9JwuDdu+/xs=
+github.com/alitto/pond v1.8.3/go.mod h1:CmvIIGd5jKLasGI3D87qDkQxjzChdKMmnXMg3fG6M6Q=
 github.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6/go.mod h1:grANhF5doyWs3UAsr3K4I6qtAmlQcZDesFNEHPZAzj8=
 github.com/aymerick/raymond v2.0.3-0.20180322193309-b565731e1464+incompatible/go.mod h1:osfaiScAUVup+UC9Nfq76eWqDhXlp+4UYaA8uhTBO6g=
 github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
@@ -532,7 +534,6 @@ golang.org/x/sync v0.0.0-20200625203802-6e8e738ad208/go.mod h1:RxMgew5VJxzue5/jJ
 golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20201207232520-09787c993a3a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.1.0 h1:wsuoTGHzEhffawBOhz5CYhcrV4IdKZbEyZjBMuTp12o=
-golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=

memiavl/mmap.go

@@ -52,11 +52,9 @@ func (m *MmapFile) Data() []byte {
 
 func Mmap(f *os.File) ([]byte, *[mmap.MaxMapSize]byte, error) {
 	fi, err := f.Stat()
-	if err != nil {
+	if err != nil || fi.Size() == 0 {
 		return nil, nil, err
 	}
-	if fi.Size() == 0 {
-		return nil, nil, nil
-	}
+
 	return mmap.Mmap(f, int(fi.Size()))
 }