Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

set max_locks_per_transaction to 2160 #34

Merged
merged 1 commit into from
Jun 28, 2024

Conversation

Ivansete-status
Copy link
Contributor

Description

We are using partitions in our postgres DBs. And we have one partition per hour (24 partitions per day.)

The default max_locks_per_transaction value (64) can cause "our of memory" and block issues in the DB because we use to have more than 64 partitions.

With 2160 we aim to avoid that issue for 90 days (2160 == 90*24.) if we consider a time retention policy of 90 days. Nevertheless, we usually have time retention policy of 30 days in our Status fleets, but we are just adding some extra margin.

Issue

PostgreSQL doesn't respond properly during E2E tests - closes waku-org/nwaku#2838

We are using partitions in our postgres DBs. And we have one
partition per hour (24 partitions per day.)

The default max_locks_per_transaction value (64) can cause
"our of memory" and block issues in the DB because we use to
have more than 64 partitions.

With 2160 we aim to avoid that issue for 90 days (2160 == 90*24.)
if we consider a time retention policy of 90 days. Nevertheless,
we usually have time retention policy of 30 days in our Status fleets,
but we are just adding some extra margin.
Copy link
Contributor

@apentori apentori left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@apentori
Copy link
Contributor

On which fleet should we start deploying it ? test ?

@Ivansete-status
Copy link
Contributor Author

On which fleet should we start deploying it ? test ?

The first one should be shards.staging

@apentori apentori self-assigned this Jun 28, 2024
@jakubgs
Copy link
Member

jakubgs commented Jun 28, 2024

Keep in mind the max_locks_per_transaction setting requires a restart:

The shared lock table has space for max_locks_per_transaction objects (e.g., tables) per server process or prepared transaction; hence, no more than this many distinct objects can be locked at any one time. This parameter limits the average number of object locks used by each transaction; individual transactions can lock more objects as long as the locks of all transactions fit in the lock table. This is not the number of rows that can be locked; that value is unlimited. The default, 64, has historically proven sufficient, but you might need to raise this value if you have queries that touch many different tables in a single transaction, e.g., query of a parent table with many children. This parameter can only be set at server start.

https://www.postgresql.org/docs/current/runtime-config-locks.html#GUC-MAX-LOCKS-PER-TRANSACTION

@apentori apentori merged commit 062cb6d into master Jun 28, 2024
2 checks passed
@apentori apentori deleted the increase-max-locks-per-transaction branch June 28, 2024 14:22
@apentori
Copy link
Contributor

Applied to sharding.staging

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

bug: PostgreSQL doesn't respond properly during E2E tests
3 participants