Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: simplify migration script #2674

Merged
merged 4 commits into from
May 7, 2024
Merged

Conversation

Ivansete-status
Copy link
Collaborator

@Ivansete-status Ivansete-status commented May 6, 2024

Description

While running migration scripts we noticed that is better to try to change the table schema directly in the messages table. The reason is that is not possible to keep handle a table backup when the message table occupies 45GiB and the max size allowed in 50GiB

With this PR we try to allow database migration with low disk availability constraints.

We want to avoid errors like:

nim-waku  | ERR 2024-05-06 17:22:42.470+00:00 failed to execute migration statement      topics="waku archive migration" tid=1 file=migrations.nim:81 statement="INSERT INTO messages (\n                        pubsubTopic,\n                        contentTopic,\n                        payload,\n                        version,\n                        timestamp,\n                        id,\n                        messageHash,\n                        storedAt\n                     )\n                SELECT pubsubTopic,\n                       contentTopic,\n                       payload,\n                       version,\n                       timestamp,\n                       id,\n                       messageHash,\n                       storedAt\n                   FROM messages_backup;" error="error in performWriteQuery: error in asyncpool query: error in dbConnQuery calling waitQueryToFinish: error in query: ERROR:  could not extend file \"base/16385/1332673.15\": No space left on device\nHINT:  Check free disk space.\n"
nim-waku  | ERR 2024-05-06 17:22:42.471+00:00 Mounting protocols failed                  tid=1 file=node_factory.nim:433 error="failed to setup archive driver: ArchiveDriver build failed in migration: failed to execute migration statement"
nim-waku  | ERR 2024-05-06 17:22:42.472+00:00 Failed setting up node                     topics="wakunode main" tid=1 file=waku.nim:97 error="Mounting protocols failed: failed to setup archive driver: ArchiveDriver build failed in migration: failed to execute migration statement"
nim-waku  | ERR 2024-05-06 17:22:42.472+00:00 Waku initialization failed                 topics="wakunode main" tid=1 file=wakunode2.nim:139 error="Failed setting up node: Mounting protocols failed: failed to setup archive driver: ArchiveDriver build failed in migration: failed to execute migration statement"

Copy link

github-actions bot commented May 6, 2024

You can find the image built from this PR at

quay.io/wakuorg/nwaku-pr:2674-rln-v1

Built from 98f38f5

Copy link

github-actions bot commented May 6, 2024

You can find the image built from this PR at

quay.io/wakuorg/nwaku-pr:2674-rln-v2

Built from 98f38f5

@Ivansete-status Ivansete-status marked this pull request as ready for review May 6, 2024 18:07
This reverts commit 740bed9.
Copy link
Contributor

@gabrielmer gabrielmer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ufff so much better. Thanks so much! 🤩

Copy link
Contributor

@SionoiS SionoiS left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

The meta field is initialized as null in this case?

@Ivansete-status
Copy link
Collaborator Author

Thanks for the comments guys!
@SionoiS - I just pushed a commit to enforce meta is null by default

1 similar comment
@Ivansete-status
Copy link
Collaborator Author

Thanks for the comments guys!
@SionoiS - I just pushed a commit to enforce meta is null by default

@Ivansete-status Ivansete-status merged commit 91c8573 into master May 7, 2024
8 checks passed
@Ivansete-status Ivansete-status deleted the simplify-migration-script branch May 7, 2024 09:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants