Invalidate query compilation cache entries with outdated VIEWs #1960
Conversation
|
⚪
|
|
⚪
|
ydb/core/protos/kqp.proto
Outdated
| optional uint32 OwnerId = 1; | ||
| optional uint32 TableId = 2; | ||
| optional uint64 OwnerId = 1; | ||
| optional uint64 TableId = 2; |
There was a problem hiding this comment.
The change from uint32 to uint64 produces binary compatible protobufs. See the docs of both proto2 and proto3, they have the same line on this topic:
int32, uint32, int64, uint64, and bool are all compatible – this means you can change a field from one of these types to another without breaking forwards- or backwards-compatibility
|
⚪ |
|
⚪ |
|
⚪
|
|
⚪
|
…ery was updated Add views, whose metadata was loaded during the compilation of the query, info to the prepared query proto and use the list to check (with SchemeCache) if the schema version of the view was updated and a recompilation is needed.
jepett0
force-pushed
the
VIEWs.invalidate_query_cache.1
branch
from
e926c59
to
02df54f
Compare
|
⚪ |
|
⚪ |
…re is no PreparedQuery)
|
⚪
|
|
⚪
|
|
⚪
|
|
⚪
|
…latform#1960) In this PR we add the following algorithm for invalidating cache entries for outdated VIEWs: 1. Store path ids and schema versions of the views that were used in the query in the cache entries, so they can be accessed later. 2. Whenever we retrieve a compilation result from cache, send a request for SchemeCache to check if the schema version of the views used in this query (if any) has not changed since we compiled this query. 3. Send a recompilation request if any view is outdated. There are two important things to note about this solution: - We make a SchemeCache request for each repeated query and there is a lot of these in an OLTP-focused database like YDB. However, we have already been sending these request for preliminary (this is not the last check of schema version mismatch (at least for tables)) cache invalidation for tables, so views should not incur an additional performance impact here. - This solution does not guarantee strong consistency for queries using views, because query cache invalidation will not happen instantly after the view definition is updated. The node should get an update from the SchemeCache, which takes some time.
…latform#1960) KIKIMR-21002 In this PR we add the following algorithm for invalidating cache entries for outdated VIEWs: 1. Store path ids and schema versions of the views that were used in the query in the cache entries, so they can be accessed later. 2. Whenever we retrieve a compilation result from cache, send a request for SchemeCache to check if the schema version of the views used in this query (if any) has not changed since we compiled this query. 3. Send a recompilation request if any view is outdated. There are two important things to note about this solution: - We make a SchemeCache request for each repeated query and there is a lot of these in an OLTP-focused database like YDB. However, we have already been sending these request for preliminary (this is not the last check of schema version mismatch (at least for tables)) cache invalidation for tables, so views should not incur an additional performance impact here. - This solution does not guarantee strong consistency for queries using views, because query cache invalidation will not happen instantly after the view definition is updated. The node should get an update from the SchemeCache, which takes some time.
#2479) KIKIMR-21002 In this PR we add the following algorithm for invalidating cache entries for outdated VIEWs: 1. Store path ids and schema versions of the views that were used in the query in the cache entries, so they can be accessed later. 2. Whenever we retrieve a compilation result from cache, send a request for SchemeCache to check if the schema version of the views used in this query (if any) has not changed since we compiled this query. 3. Send a recompilation request if any view is outdated. There are two important things to note about this solution: - We make a SchemeCache request for each repeated query and there is a lot of these in an OLTP-focused database like YDB. However, we have already been sending these request for preliminary (this is not the last check of schema version mismatch (at least for tables)) cache invalidation for tables, so views should not incur an additional performance impact here. - This solution does not guarantee strong consistency for queries using views, because query cache invalidation will not happen instantly after the view definition is updated. The node should get an update from the SchemeCache, which takes some time.
KIKIMR-21002
YDB caches query compilation results on the server side for efficiency. (For small queries compilation can take up to 100x times more time than the execution.) We search the cache entry by the text of the query. One can miss the cache just by adding a comment to the query. (Well, not anymore, because we have added cache by AST recently. Compilation takes much more time than AST building, so it makes sense.) The compilation result caching could theoretically cause problems even for the basic:
We could change the definition of this table in a separate session like this:
and expect the select from this table to produce wrong results, because of the query cache, which will not notice the change in the definition of the table, because it is not apparent in the text of the query. However, there are some special mitigation mechanisms implemented for tables, which were missing for views up until this PR.
In this PR we add the following algorithm for invalidating cache entries for outdated VIEWs:
There are two important things to note about this solution: