forked from JanusGraph/janusgraph
-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add possibility to manual db-cache invalidation
Related to JanusGraph#3155 Signed-off-by: Oleksandr Porunov <[email protected]>
- Loading branch information
Showing
9 changed files
with
688 additions
and
19 deletions.
There are no files selected for viewing
339 changes: 339 additions & 0 deletions
339
janusgraph-backend-testutils/src/main/java/org/janusgraph/graphdb/JanusGraphTest.java
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
192 changes: 192 additions & 0 deletions
192
...ph-core/src/main/java/org/janusgraph/graphdb/database/cache/CacheInvalidationService.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,192 @@ | ||
// Copyright 2022 JanusGraph Authors | ||
// | ||
// Licensed under the Apache License, Version 2.0 (the "License"); | ||
// you may not use this file except in compliance with the License. | ||
// You may obtain a copy of the License at | ||
// | ||
// http://www.apache.org/licenses/LICENSE-2.0 | ||
// | ||
// Unless required by applicable law or agreed to in writing, software | ||
// distributed under the License is distributed on an "AS IS" BASIS, | ||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
// See the License for the specific language governing permissions and | ||
// limitations under the License. | ||
|
||
package org.janusgraph.graphdb.database.cache; | ||
|
||
import org.janusgraph.core.log.Change; | ||
import org.janusgraph.diskstorage.StaticBuffer; | ||
|
||
/** | ||
* Cache invalidation service for manual JanusGraph database-level cache invalidation. | ||
* Use with great care because improper invalidation may result in stale data left in db-cache. | ||
* This service wraps two different caches called as `edgeStore` and `indexStore` which form a | ||
* single database-level cache (can be enabled via `cache.db-cache` configuration property). | ||
* When db-cache is disabled this service doesn't make any effective changes because the cache is | ||
* fully disabled in that case. | ||
* <p> | ||
* This class provides method for data invalidation for both `edgeStore` cache and `indexStore` cache | ||
* but invalidating entries in one cache doesn't invalidate entries in another cache. | ||
* Thus, for proper invalidation you need to invalidate necessary keys for both `edgeStore` and `indexStore`. | ||
* <p> | ||
* EdgeStore accepts keys where a key is an encoded vertex id. It's usually easy to invalidate EdgeStore because | ||
* it doesn't require you to know more information to form a key except knowing a vertex id. | ||
* <p> | ||
* IndexStore accepts keys where a key is an encoded IndexUpdate key. To form IndexUpdate key you need to know | ||
* the next information: vertex id, updated property name, previous property value and / or new property value. | ||
* Thus forming IndexUpdate key to invalidate IndexStore may be more complicated. | ||
* <p> | ||
* EdgeStore caches properties and edges for vertices. | ||
* IndexStore caches results for queries which use indices. | ||
* <p> | ||
* See JavaDoc on methods to learn how to properly form a `key` to invalidate cache in `edgeStore` or `indexStore`. | ||
*/ | ||
public interface CacheInvalidationService { | ||
|
||
/** | ||
* Marks specific vertex as expired in `edgeStore` cache. | ||
* It will make sure that any retrieved properties and edges associated with this vertex will be invalidated in vertex cache. | ||
* <p> | ||
* Warning! This doesn't invalidate `indexStore` cache. Thus, any queries which are using indices may still return | ||
* stale data. See {@link #markKeyAsExpiredInIndexStore(StaticBuffer)} JavaDoc to learn how to invalidate data for | ||
* `indexStore`. | ||
* | ||
* @param vertexId vertex id to expire in `edgeStore` cache | ||
*/ | ||
void markVertexAsExpiredInEdgeStore(Long vertexId); | ||
|
||
/** | ||
* Marks specific key as expired in `edgeStore` cache. | ||
* It will make sure that any retrieved properties and edges associated with this key will be invalidated in vertex cache. | ||
* <p> | ||
* Warning! This doesn't invalidate `indexStore` cache. Thus, any queries which are using indices may still return | ||
* stale data. See {@link #markKeyAsExpiredInIndexStore(StaticBuffer)} JavaDoc to learn how to invalidate data for | ||
* `indexStore`. | ||
* <p> | ||
* {@link org.janusgraph.graphdb.idmanagement.IDManager#getKey(long)} can be used to form a `key` from vertex id. | ||
* Alternatively, a method {@link #markVertexAsExpiredInEdgeStore(Long)} can be used which converts vertex id into | ||
* the `key` before passing the key to this method. | ||
* <p> | ||
* In case vertices invalidation is needed by processing transaction logs via {@link org.janusgraph.core.log.ChangeState} | ||
* then the method {@link org.janusgraph.core.log.ChangeState#getVertices(Change)} can be used to retrieve all | ||
* changed vertices and passing their ids to {@link #markVertexAsExpiredInEdgeStore(Long)}. | ||
* | ||
* @param key key to expire in `edgeStore` cache | ||
*/ | ||
void markKeyAsExpiredInEdgeStore(StaticBuffer key); | ||
|
||
/** | ||
* Marks specific key as expired in `indexStore` cache. | ||
* It will make sure that any retrieved data associated with this key will be invalidated in index cache. | ||
* <p> | ||
* Warning! This doesn't invalidate `edgeStore` cache. Thus, trying to return properties or edges for the vertex | ||
* may still return stale data. See {@link #markKeyAsExpiredInEdgeStore(StaticBuffer)} JavaDoc to learn how to invalidate | ||
* data for `edgeStore`. | ||
* <p> | ||
* To form a `key` for invalidation it is needed to know vertex id, updated property name, previous value (if there was any), | ||
* new value (if there is any) from which IndexUpdate key can be formed. | ||
* <p> | ||
* Usually this information can be found in a retrieved mutation logs which are passed via {@link org.janusgraph.core.log.ChangeState} | ||
* (described in `Transaction Log` documentation of JanusGraph). In case `indexStore` invalidation should triggered | ||
* from processing transaction logs via `ChangeState` then it can be done using an example like below: | ||
* | ||
* <pre> | ||
* IndexSerializer indexSerializer = (StandardJanusGraph) graph.getIndexSerializer(); | ||
* CacheInvalidationService invalidationService = graph.getDBCacheInvalidationService(); | ||
* changeState.getRelations(Change.ANY).forEach(janusGraphRelation -> { | ||
* if(janusGraphRelation.isProperty() && janusGraphRelation instanceof InternalRelation){ | ||
* JanusGraphVertexProperty property = (JanusGraphVertexProperty) janusGraphRelation; | ||
* if(property.element() instanceof InternalVertex){ | ||
* Collection<IndexUpdate> indexUpdates = indexSerializer.getIndexUpdates((InternalVertex) property.element(), | ||
* Collections.singleton((InternalRelation) property)); | ||
* | ||
* for(IndexUpdate indexUpdate : indexUpdates){ | ||
* StaticBuffer keyToInvalidate = (StaticBuffer) indexUpdate.getKey(); | ||
* invalidationService.markKeyAsExpiredInIndexStore(keyToInvalidate); | ||
* } | ||
* | ||
* invalidationService.forceClearExpiredKeysInIndexStoreCache(); | ||
* invalidationService.forceInvalidateVertexInEdgeStoreCache(property.element().longId()); | ||
* } | ||
* } | ||
* }); | ||
* </pre> | ||
* | ||
* It is also possible to trigger `indexStore` invalidation by forming vertex and a property yourself. The example | ||
* below can be used as a reference. | ||
* | ||
* <pre> | ||
* public void invalidateUpdatedVertexProperty(StandardJanusGraph graph, long vertexIdUpdated, String propertyNameUpdated, Object previousPropertyValue, Object newPropertyValue){ | ||
* JanusGraphTransaction tx = graph.newTransaction(); | ||
* JanusGraphManagement graphMgmt = graph.openManagement(); | ||
* PropertyKey propertyKey = graphMgmt.getPropertyKey(propertyNameUpdated); | ||
* CacheVertex cacheVertex = new CacheVertex((StandardJanusGraphTx) tx, vertexIdUpdated, ElementLifeCycle.Loaded); | ||
* StandardVertexProperty propertyPreviousVal = new StandardVertexProperty(propertyKey.longId(), propertyKey, cacheVertex, previousPropertyValue, ElementLifeCycle.Removed); | ||
* StandardVertexProperty propertyNewVal = new StandardVertexProperty(propertyKey.longId(), propertyKey, cacheVertex, newPropertyValue, ElementLifeCycle.New); | ||
* IndexSerializer indexSerializer = graph.getIndexSerializer(); | ||
* | ||
* Collection<IndexUpdate> indexUpdates = indexSerializer.getIndexUpdates(cacheVertex, Arrays.asList(propertyPreviousVal, propertyNewVal)); | ||
* CacheInvalidationService invalidationService = graph.getDBCacheInvalidationService(); | ||
* | ||
* for(IndexUpdate indexUpdate : indexUpdates){ | ||
* StaticBuffer keyToInvalidate = (StaticBuffer) indexUpdate.getKey(); | ||
* invalidationService.markKeyAsExpiredInIndexStore(keyToInvalidate); | ||
* } | ||
* | ||
* invalidationService.forceClearExpiredKeysInIndexStoreCache(); | ||
* invalidationService.forceInvalidateVertexInEdgeStoreCache(vertexIdUpdated); | ||
* | ||
* graphMgmt.rollback(); | ||
* tx.rollback(); | ||
* } | ||
* </pre> | ||
* | ||
* @param key key to expire in `indexStore` cache | ||
*/ | ||
void markKeyAsExpiredInIndexStore(StaticBuffer key); | ||
|
||
/** | ||
* Instead of waiting for a probabilistic invalidation it triggers all cached queries scan and invalidation in `edgeStore`. | ||
* This will remove any cached expired data. | ||
*/ | ||
void forceClearExpiredKeysInEdgeStoreCache(); | ||
|
||
/** | ||
* Instead of waiting for a probabilistic invalidation it triggers all cached queries scan and invalidation in `indexStore`. | ||
* This will remove any cached expired data. | ||
*/ | ||
void forceClearExpiredKeysInIndexStoreCache(); | ||
|
||
/** | ||
* Marks a vertex as expired in `edgeStore` cache ({@link #markVertexAsExpiredInEdgeStore(Long)}) and triggers force | ||
* clear of expired cache (i.e. {@link #forceClearExpiredKeysInEdgeStoreCache()}) | ||
* | ||
* @param vertexId vertex id to invalidate in `edgeStore` cache | ||
*/ | ||
void forceInvalidateVertexInEdgeStoreCache(Long vertexId); | ||
|
||
/** | ||
* Marks vertices as expired in `edgeStore` cache ({@link #markVertexAsExpiredInEdgeStore(Long)}) and triggers force | ||
* clear of expired cache (i.e. {@link #forceClearExpiredKeysInEdgeStoreCache()}) | ||
* | ||
* @param vertexIds vertex ids to invalidate in `edgeStore` cache | ||
*/ | ||
void forceInvalidateVerticesInEdgeStoreCache(Iterable<Long> vertexIds); | ||
|
||
/** | ||
* Clears `edgeStore` cache fully | ||
*/ | ||
void clearEdgeStoreCache(); | ||
|
||
/** | ||
* Clears `indexStore` cache fully | ||
*/ | ||
void clearIndexStoreCache(); | ||
|
||
/** | ||
* Clears both `edgeStore` cache and `indexStore` cache fully. | ||
* It is the same as calling {@link #clearEdgeStoreCache()} and {@link #clearIndexStoreCache()} | ||
*/ | ||
void clearDBCache(); | ||
|
||
} |
Oops, something went wrong.
8d4cdbf
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Benchmark
org.janusgraph.MgmtOlapJobBenchmark.runRemoveIndex
114.09884923454544
ms/op114.90347956363637
ms/op0.99
org.janusgraph.JanusGraphSpeedBenchmark.basicAddAndDelete
18324.320372602982
ms/op16912.67303256719
ms/op1.08
org.janusgraph.GraphCentricQueryBenchmark.getVertices
2141.921422949411
ms/op2244.533148796801
ms/op0.95
org.janusgraph.MgmtOlapJobBenchmark.runReindex
429.1107533446154
ms/op403.35688975634366
ms/op1.06
org.janusgraph.JanusGraphSpeedBenchmark.basicCount
303.09688068754866
ms/op369.41181090112127
ms/op0.82
org.janusgraph.CQLMultiQueryBenchmark.getNeighborNames
59808.19720566667
ms/op46939.17807455001
ms/op1.27
org.janusgraph.CQLMultiQueryBenchmark.getNames
58117.66215536666
ms/op48136.52566508333
ms/op1.21
This comment was automatically generated by workflow using github-action-benchmark.