chore(queries): pg_stat_user_tables: skip tables with an AccessExclusiveLock #19
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
#sre |
qfritz
approved these changes
Dec 14, 2023
|
#20 will fix the prometheus-rules test. |
…iveLock Methods computing size of tables or related objects (such as `pg_total_relation_size` or `pg_table_size`) acquires an AccessShareLock. This happens while refreshing a Materialized View or when ALTER'ing a table. A fix has been implemented to estimate (very precisely) the size of materialized views, so the query does not wait while a materialized view is being refreshed. However, this did not fix the issue when a table is being ALTER'ed: the SQL query waits for the AccessExclusiveLock being released, and this can take several minutes. It is an issue because the exporter stops exporting any data while the ALTER is running. To prevent that, we check in `pg_locks` table if an AccessExclusiveLock exists: if it does exist, we skip it: - the exporter will compute size for tables with no AccessExclusiveLock locks - the exporter will not be blocked This is not perfect: there may have a race condition where the Lock is being acquired just before the query is being executed. However, we believe this should prevent the common cases where an ALTER is run for a long time (several minutes), and we consider having the exporter blocked for a single execution acceptable.
…ize of the materialized views without pg_XXX_size Now we skip tables that have an AccessExclusiveLock that prevents pg_XXX_size to be executed, we can rollback the complex optimization and use the simple form with pg_XXX_size for materialized views. Because we use JOIN, if a AccessExclusiveLock is on a table/MatView, the table will be missing from the query, and the line will be missed from the export page. We could use LEFT JOIN: - the line for the table will be kept, with NULL values for size fields. For now, we prefer having missing data rather than NaN number reported.
chtitux
force-pushed
the
fix-access-exclusive-lock-size
branch
from
cf3e1f6
to
b7c175e
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Methods computing size of tables or related objects (such as
pg_total_relation_sizeorpg_table_size) acquires an AccessShareLock. This happens while refreshing a Materialized View or when ALTER'ing a table.A fix has been implemented to estimate (very precisely) the size of materialized views, so the query does not wait while a materialized view is being refreshed. However, this did not fix the issue when a table is being ALTER'ed: the SQL query waits for the AccessExclusiveLock being released, and this can take several minutes. It is an issue because the exporter stops exporting any data while the ALTER is running.
To prevent that, we check in
pg_lockstable if an AccessExclusiveLock exists: if it does exist, we skip it:This is not perfect: there may have a race condition where the Lock is being acquired just before the query is being executed. However, we believe this should prevent the common cases where an ALTER is run for a long time (several minutes), and we consider having the exporter blocked for a single execution acceptable.