Refactor Datastore integration #40
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Previously, the DeletedBlobIndex stored the ids of soft-deleted blobs in the root namespace in Google Cloud Datastore. After this change set, the ids are stored in a new hierarchy that is friendlier for a) multiple google cloud blobstore instances in the same project and b) will support us using Datastore for other blobstore metadata. Also included is further expansion of the HealthCheck - it can now indicate that a google cloud blobstore has a high number of soft-deleted blobs in need of compaction.
The previous metrics store inherited functionality to integrate with BlobStore soft-quotas. The new metrics store does not inherit this, so this change set merges that functionality into the BlobStore implementation. This changeset also includes Datastore transactions around the read and write operations.
Also made the flush frequency a constant for better visibility.
|
PR opened here for visibility, proceeding with the merge as there are continued efforts that will build on this as we prep for the 3.18 NXRM release. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request is a significant refactor in how this plugin manages Blobstore metadata:
What's included:
DeletedBlobIndexnow nests its entries in a new key hierarchy. This addresses a design flaw in previous releases that offered no separation of data between multiple Google Cloud Blobstore instances in one Repository Manager deployment.ShardedCounterMetricsStoreis a new way to calculate the total size and count of all blobs in the blobstore. It replaces themetrics-[nodeid].propertiesfile approach.Blobstore#remove()previously was a no-op; it now will delete all blobstore metadata in datastore when invoked. It still does not delete the Storage bucket nor any files within the bucket.There is a lot of change in this set; I've included an ADR to cover most of it, I recommend starting there.