PostgreSQL & ShardLayout

[kobayurii]

As our data size continues to grow, managing the load has become increasingly challenging, leading to several issues:

• Slowing down of contracts due to state provision in contract runtime #150
• View calls are either too slow or return HTTP/502 (Bad Gateway) errors #193
• Performance problems featuring near-vm #236

After extensive discussions and brainstorming, we believe these issues can be mitigated by transitioning to a more traditional relational database management system, specifically PostgreSQL.

In This PR

1. Improved or refactored the database crate to align with the shard layout from nearcore #289
2. Redesigned and designed the missing parts of the PostgreSQL schemas, respecting the shard layout #290

By implemented these changes, we anticipate the following benefits:

1. Increased request handling capacity due to the relational DB's ability to fetch data in batches effectively.
2. The ability to implement the long-standing query.access_key_list method (reference: Redesign data collection and storage for query.view_access_key_list RPC method #187).

[kobayurii]

@race-of-sloths please, include my PR in the Race

[race-of-sloths]

@kobayurii Thank you for your contribution! Your pull request is now a part of the Race of Sloths!
New Sloth joined the Race! Welcome!

Current status: executed

Reviewer	Score
@khorolets	13

Your contribution is much appreciated with a final score of 13!
You have received 137 (130 base + 5% lifetime bonus) Sloth points for this contribution

We would appreciate your feedback or contribution idea that you can submit here

What is the Race of Sloths

Race of Sloths is a friendly competition where you can participate in challenges and compete with other open-source contributors within your normal workflow

For contributors:

• Tag @race-of-sloths inside your pull requests
• Wait for the maintainer to review and score your pull request
• Check out your position in the Leaderboard
• Keep weekly and monthly streaks to reach higher positions
• Boast your contributions with a dynamic picture of your Profile

For maintainers:

• Score pull requests that participate in the Race of Sloths
• Engage contributors with fair scoring and fast responses so they keep their streaks
• Promote the Race to the point where the Race starts promoting you
• Grow the community of your contributors

Feel free to check our website for additional details!

Bot commands

• For contributors
  • Include a PR: @race-of-sloths include to enter the Race with your PR
• For maintainers:
  • Assign points: @race-of-sloths score [1/2/3/5/8/13] to award points based on your assessment.
  • Reject this PR: @race-of-sloths exclude to send this PR back to the drawing board.
  • Exclude repo: @race-of-sloths pause to stop bot activity in this repo until @race-of-sloths unpause command is called

[khorolets]

That's a lot of work! Awesome work! I left comments at all the places that caught my eye. Please, point me to any specific parts you would like my extra attention.

[khorolets]

@race-of-sloths score 13

[race-of-sloths]

🌟 Score recorded!

@khorolets, thank you for scoring this pull request in the Race of Sloths!

[race-of-sloths]

❌ PR inactive

This PR has been inactive for two weeks. It's now marked as stale.

Tip

If you think it's a mistake, you can restart the bot with include command to proceed 🔄

[gilxgil]

hi @kobayurii we're interested in running this for RPC Nodes, is this planned for main?

[kobayurii]

hi @gilxgil, yes, this will be merged into main soon, we are doing final testing and debugging. You can try it out here https://v2.rpc.mainnet.near.org

[kobayurii]

@race-of-sloths please, include my PR in the Race

[race-of-sloths-bot]

@kobayurii Thank you for your contribution! Your pull request is now a part of the Race of Sloths!
Weekly streak is on the road, smart strategy! Secure your streak with another PR!

Current status: waiting for finalization

The pull request is merged, you have 24 hours to finalize your scoring. The scoring ends Thu Oct 10 08:30:36 2024

Reviewer	Score
@khorolets	13

Your contribution is much appreciated with a final score of 13!
You have received 130 Sloth points for this contribution

What is the Race of Sloths

Race of Sloths is a friendly competition where you can participate in challenges and compete with other open-source contributors within your normal workflow

For contributors:

• Tag @race-of-sloths inside your pull requests
• Wait for the maintainer to review and score your pull request
• Check out your position in the Leaderboard
• Keep weekly and monthly streaks to reach higher positions
• Boast your contributions with a dynamic picture of your Profile

For maintainers:

• Score pull requests that participate in the Race of Sloths
• Engage contributors with fair scoring and fast responses so they keep their streaks
• Promote the Race to the point where the Race starts promoting you
• Grow the community of your contributors

Feel free to check our website for additional details!

Bot commands

• For contributors
  • Include a PR: @race-of-sloths include to enter the Race with your PR
• For maintainers:
  • Invite contributor @race-of-sloths invite to invite the contributor to participate in a race or include it, if it's already a runner.
  • Assign points: @race-of-sloths score [1/2/3/5/8/13] to award points based on your assessment.
  • Reject this PR: @race-of-sloths exclude to send this PR back to the drawing board.
  • Exclude repo: @race-of-sloths pause to stop bot activity in this repo until @race-of-sloths unpause command is called

[race-of-sloths-bot]

❓Unknown command

Please check the command and try again