store: write benchmarks /store and /store/v2 with different load scenarios

[odeke-em]

Summary

We need to write benchmarks that use the store package and can help discern performance differences in backends.

Problem Definition

This feature will allow easy and more educated migrations between backends.

Proposal

Go through the usages of store/* and figure out the best benchmarks that use the package, and how to test them effectively test them out. /cc @cuonglm @robert-zaremba for our Thursday meeting discussion.

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[alexanderbez]

Looking forward to seeing these results!

[robert-zaremba]

The idea is to test the CommitKVStore implementation:

• create test load scenarios with bunch of Has, Get, Set, Delete operations and Commits. Eg: scenario-1: 70% of Set (and bunch of Has, Get, Delete in between). scenario-2: 20% Set and 60% Get....
  • most important scenario: 65% Get, 5% Has and 30% Set
• Try to "rollback" (load previous version)
• Try to create a scenario which will crash the DB.
• analyze the results:
  • what is the most expensive load scenario
  • how commit is impacting the performance

And in the benchmark let's use the tm-db with go-leveldb (the default one) in store/v1 and badger in store/v2, and also rocksdb for both.

Goal: we will run this benchmark on

1. current store (v1) based on 2-3 backends (most important the go-leveldb which is the default one).
2. once store v2 will be ready (based on SMT) we will run the benchmark against RocksDB and Bager.

[adu-web3]

Hi @robert-zaremba, I'm working on this right now.
but I don't quite understand why you ask to test load scenarios with different ratio of Has, Get, Set, Delete operations, given Has and Get operations are read-only and will not affect the load performance after commit.
can you explain this more specifically?

[robert-zaremba]

We want to test store as it's used by msg server. Moreover having different scenarios will let us know more about bottlenecks and performance characteristic.
It's important to do this test on the store level, not SMT. I'm going to update the title of this task.

[roysc]

I've been running @Muggle-Du 's benchmarks and adapted them to use the DB backend by itself. So far it's clear that the cosmos-sdk/db is responsible for a large part of the performance difference.

• Rocksdb performance with optimistic transactions is terrible, and is an outlier, even compared to tm-db using the same Rocksdb version.
• Badgerdb v3 is used in cosmos/db while tm-db uses v2, and there's a small speedup in DB ops, but the store's performance is worse, so we are going to focus on finding out why.

For now I don't see an easy way to improve Rocks and still have transactional semantics, and even without transactions Rocks is also slower than Badger overall, so we'll just stick to Badger for now and look at how to improve the store and algorithms.

[robert-zaremba]

Thanks @roysc for starting looking at it. I propose to create a new issue where we will analyze the roots of inefficiencies and benchmark improvements.

[roysc]

New issue created at #11328