State sync: use native DB export - stream - import

[robert-zaremba]

Summary

Optimize the state snapshot sync by streaming native DB exports.

Problem Definition

Cosmos apps have massive growth, and the state is getting too big to efficiently sync using a naive iterator, where we:

• iterate over key-value pairs and stream the over in batches
• receive and decode key-value pairs and iteratively save them to the store

Proposal

1. Use native DB export
2. Use protobuf streaming functionality for streaming DB export bytes
3. Use native DB import functionality

It should massively improve the performance. The obvious limitation is that only nodes using the same underlying DB can sync. However, our idea is that the nodes in the same network will use the same DB.

Ref: #10194

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[robert-zaremba]

This is not done, why closing?