Change to embedded database system

[ManuGowda]

Description

Replace the postgresql database with an embedded database.

Motivation

• Reduce the complexity of preparation for SDK user
• Optimize database storage size
• RDB features are not required for on-chain processing
• Better read/write performance
• Split of search capability
• Support for cross platform

Tasks

Spike 02

• Create lisk-db package in elements Create lisk-db package in elements #5241
• Create utility to get a range pointer Create utility to get a range pointer #5242

Sprint 19

• Create database instances and save them in one folder Create database instances and save them in one folder #5243
• Update lisk-chain data_access layer Update lisk-chain data_access layer #5257
• Update network module to use node.db Update network module to use node.db #5258
• Update delegate registration to save and use username and address in blockchain.db Update delegate registration to save and use username and address in blockchain.db #5259
• Remove postgresql related code and tests Remove postgresql related code and tests #5260
• Remove rebuild module from the framework Remove rebuild module from the framework #5320
• Logs are not saved under the rootPath Logs are not saved under the rootPath #5329
• Refactor remaining tests in the framework Refactor remaining tests in the framework #5331
• Update database key constant usage to unified format Update database key constant usage to unified format #5345

Sprint 20

• Measure embedded database performance Measure embedded database performance #5261

Additional Information

• Batch write should only happen at one place in the application
• Height/Number in general has to be 0 prepended or converted to buffer
• Throw custom error for non existing records
• Use key type as string and value type as buffer

[pcdinh]

While a server-less embedded database like SQLite or LMDB can make server provisioning much easier, its scalability is limited into a single machine

[shuse2]

@pcdinh For the on-chain processing of the blockchain, data access is always sequential and cannot be multi-processed with our current approach.
Therefore, the scalability of the database does not apply to the SDK

[vitaly-t]

@pcdinh For the on-chain processing of the blockchain, data access is always sequential and cannot be multi-processed with our current approach.
Therefore, the scalability of the database does not apply to the SDK

You don't understand what you are talking about. Scalability isn't just about what happens on the client side. It is about memory+CPU management, plus execution planning by the server to further improve performance. Advanced migrations, etc.

You've just shot yourself in the foot, by replacing PostgreSQL with half-baked file server.