CBG-3703 create a basic bucket to bucket XDCR implementation [HAS DEPENDENCY] #29

torcolvin · 2024-03-28T00:03:52Z

uses new common elements from sg-bucket
absorbs DataStoreName and GetCollectionID into DataStore to avoid duplication
The implementation only copies documents with xattrs, and excludes specially named documents. It does not implement
_vv, _mou, or _sync handling.
test is duplicated in sync_gateway xdcr package with the code matching the common interface

Includes #30 which can be merged independently

- uses new common elements from sg-bucket - absorbs DataStoreName and GetCollectionID into DataStore to avoid duplication - The implementation only copies documents with xattrs, and excludes specially named documents. It does not implement _vv, _mou, or _sync handling.

gregns1

Generally looks good I think! Just couple of Q's in comments. Will let Adam have final say too.

xdcr_test.go

…is implemented

adamcfraser

A few questions/clarifications.

xdcr.go

adamcfraser · 2024-03-28T18:01:34Z

xdcr.go

+		fromBucket:              fromBucket,
+		toBucket:                toBucket,
+		replicationID:           fmt.Sprintf("%s-%s", fromBucket.GetName(), toBucket.GetName()),
+		fromBucketCollectionIDs: map[uint32]sgbucket.DataStoreName{},


What's the expected handling if there are collections defined in fromBucket that aren't present in toBucket? Should we be checking for this now instead of logging a warning for every mutation that arrives for a non-existent collection?

Oh, I see below that we're only supporting the default collection at this point? (we're not passing any scopes args to the DCP feed). Are you thinking of adding that later?

This was an oversight and I've added tests for the non default collection.

adamcfraser · 2024-03-28T18:06:54Z

xdcr.go

+
+// getFromBucketCollectionName returns the collection name for a given collection ID in the from bucket.
+func (r *XDCR) getFromBucketCollectionName(collectionID uint32) (sgbucket.DataStoreName, error) {
+	dsName, ok := r.fromBucketCollectionIDs[collectionID]


Since we need to know the set of fromBucket collections when we start the DCP feed anyway, I think populating fromBucketCollectionIDs on start instead of lazily will simplify things and avoid potential races accessing fromBucketCollectionIDs.

I made a precondition of this that the DataStores are computed when XDCR.Start is called, and that there have to be matching collection names.

We could relax this in a future PR but I don't think this is necessary for testing at this time.

adamcfraser · 2024-03-28T18:11:38Z

xdcr.go

+
+		switch event.Opcode {
+		case sgbucket.FeedOpDeletion, sgbucket.FeedOpMutation:
+			if strings.HasPrefix(docID, sgbucket.SyncDocPrefix) && !strings.HasPrefix(docID, sgbucket.Att2Prefix) {


It would be nice to abstract this as a key filter function that's defined on the replication (i.e. XDCR.keyFilterFunc), and set that filter to a pre-defined function when the replication is created with sgbucket.XDCRMobileOn. That would keep the general XDCR implementation a bit more generic and make it clear what's happening when XDCRMobileOn is set.

xdcr.go

adamcfraser · 2024-03-28T18:15:56Z

xdcr.go

+				return true
+			}
+
+			toCollection, ok := toDataStore.(*Collection)


It's unexpected to me that we're doing this check after we've already done toDataStore.Get. Can we just get the collection in the first place on line 107?

I changed this code to avoid casts by implementing fixes in sgbucket to consolidate these interfaces into DataStore. This code exists in a separate PR but is included in this PR.

adamcfraser · 2024-03-28T18:17:41Z

xdcr.go

+	return nil
+}
+
+// writeDoc writes a document to the target datastore. This will not return an error on a CAS mismatch, but will return error on other types of write.


I'm not sure what "other types of write" means in this comment. Also, setWithMeta still needs to do a CAS check to ensure the document hasn't been mutated on the target between the time we evaluated LWW and the time we write, right?

adamcfraser · 2024-03-28T18:21:03Z

xdcr.go

+// writeDoc writes a document to the target datastore. This will not return an error on a CAS mismatch, but will return error on other types of write.
+func writeDoc(ctx context.Context, collection *Collection, originalCas uint64, event sgbucket.FeedEvent) error {
+	if event.Opcode == sgbucket.FeedOpDeletion {
+		_, err := collection.Remove(string(event.Key), originalCas)


I think XDCR does delWithMeta to ensure that the delete mutation on the target ends up with the same CAS as the delete mutation on the source - does collection.Remove() do the same thing?

Good catch, fixed.

adamcfraser · 2024-03-28T18:23:31Z

xdcr.go

+
+	err := collection.SetWithMeta(ctx, string(event.Key), originalCas, event.Cas, event.Expiry, xattrs, body, event.DataType)
+
+	if !collection.IsError(err, sgbucket.KeyNotFoundError) {


Why do we want to ignore a KeyNotFound error here?

Good catch.

adamcfraser · 2024-03-28T18:26:01Z

xdcr.go

+
+	}
+
+	err := collection.SetWithMeta(ctx, string(event.Key), originalCas, event.Cas, event.Expiry, xattrs, body, event.DataType)


Are we preserving system (_sync) xattrs on the target here? Or is that a pending enhancement?

All xattrs are preserved. A future enhancement will be to handle _vv and _mou xattrs.

torcolvin · 2024-04-09T14:22:00Z

Dropping this PR to implement this entirely within sync gateway to avoid three repo PRs.

torcolvin mentioned this pull request Mar 28, 2024

CBG-3764 update base implementation of XDCR in rosmar couchbase/sync_gateway#6746

Closed

6 tasks

torcolvin requested review from gregns1 and adamcfraser March 28, 2024 00:55

torcolvin assigned gregns1 and adamcfraser Mar 28, 2024

gregns1 reviewed Mar 28, 2024

View reviewed changes

xdcr_test.go Outdated Show resolved Hide resolved

xdcr_test.go Show resolved Hide resolved

torcolvin added 2 commits March 28, 2024 11:16

Remove unused code

9f9e245

Make sure that only XDCR mobile on is allowed since that is all that …

b885e35

…is implemented

adamcfraser requested changes Mar 28, 2024

View reviewed changes

adamcfraser assigned torcolvin and unassigned adamcfraser and gregns1 Mar 28, 2024

torcolvin added 2 commits March 31, 2024 20:04

remove comments

4798974

review comments

bfa0cf1

torcolvin changed the title ~~[DO NOT MERGE] CBG-3764 create a basic bucket to bucket XDCR implementation~~ CBG-3703 create a basic bucket to bucket XDCR implementation [HAS DEPENDENCY] Apr 1, 2024

torcolvin added 3 commits April 1, 2024 11:29

update go.mod

0d5cc3a

Merged MutationFeedStore2, Collection into sgbucket.DataStore

249fc06

Remove print statements

ce4cab0

torcolvin assigned adamcfraser and unassigned torcolvin Apr 2, 2024

torcolvin closed this Apr 2, 2024

torcolvin deleted the CBG-3764 branch April 2, 2024 14:30

torcolvin restored the CBG-3764 branch April 2, 2024 14:30

torcolvin reopened this Apr 2, 2024

torcolvin added 2 commits April 4, 2024 21:40

Merge remote-tracking branch 'origin/main' into CBG-3764

4bf8f47

update go.mod

8c60c5d

torcolvin requested a review from adamcfraser April 5, 2024 01:42

torcolvin closed this Apr 9, 2024

torcolvin deleted the CBG-3764 branch April 9, 2024 14:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CBG-3703 create a basic bucket to bucket XDCR implementation [HAS DEPENDENCY] #29

CBG-3703 create a basic bucket to bucket XDCR implementation [HAS DEPENDENCY] #29

torcolvin commented Mar 28, 2024 •

edited

Loading

gregns1 left a comment

adamcfraser left a comment

adamcfraser Mar 28, 2024

adamcfraser Mar 28, 2024

torcolvin Apr 1, 2024

adamcfraser Mar 28, 2024

torcolvin Apr 1, 2024

adamcfraser Mar 28, 2024

adamcfraser Mar 28, 2024

torcolvin Apr 1, 2024

adamcfraser Mar 28, 2024

adamcfraser Mar 28, 2024

torcolvin Apr 1, 2024

adamcfraser Mar 28, 2024

torcolvin Apr 1, 2024

adamcfraser Mar 28, 2024

torcolvin Apr 1, 2024

torcolvin commented Apr 9, 2024


		err := collection.SetWithMeta(ctx, string(event.Key), originalCas, event.Cas, event.Expiry, xattrs, body, event.DataType)

		if !collection.IsError(err, sgbucket.KeyNotFoundError) {

CBG-3703 create a basic bucket to bucket XDCR implementation [HAS DEPENDENCY] #29

CBG-3703 create a basic bucket to bucket XDCR implementation [HAS DEPENDENCY] #29

Conversation

torcolvin commented Mar 28, 2024 • edited Loading

gregns1 left a comment

Choose a reason for hiding this comment

adamcfraser left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

torcolvin commented Apr 9, 2024

torcolvin commented Mar 28, 2024 •

edited

Loading