Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change Feed Processor: Adds support for EPK leases #2013

Merged
merged 48 commits into from
Dec 14, 2020
Merged

Conversation

ealsur
Copy link
Member

@ealsur ealsur commented Nov 17, 2020

Description

This PR adds support for leases that represent a range of partition values and the wiring into the internal FeedProcessor to execute the Change Feed read based on the type of lease acquired by the Host.

The default lease schema is maintained as the current one (identified as Version 1) that maps 1 lease to 1 physical partition for backward compatibility with existing lease stores and V2 CFP.

When an event that affects the current partition (such as a partition Merge) happens, the current lease is migrated to the new schema.

This new schema versioning also prepares CFP to support decoupling the number of partitions from the number of leases, which will eventually let users define a higher degree of parallelization.

Lease schema difference

The current default (coming from V2 CFP) schema looks like:

{
    "id": "<some value>",
    "version": 0,
    "LeaseToken": "<partition id>",
    "Owner": "<instance name>",
    "ContinuationToken": "<continuation>",
    "properties": {},
    "timestamp": "<timestamp>"
}

The new range based lease's schema is:

{
    "id": "<some value>",
    "version": 1,
    "LeaseToken": "<range representation>",
    "FeedRange": {
        "Range": {
            "min": "<initial partition key value of the range>",
            "max": "<final partition key value of the range>"
        }
    },
    "Owner": "<instance name>",
    "ContinuationToken": "<continuation>",
    "properties": {},
    "timestamp": "<timestamp>"
}

Code changes details

  • CFP code previously was using a ReadFeed on the partition key ranges to list partitions, instead of relying on the SDKs caches. We are now wiring down the caches to leverage them instead of doing independent calls.
  • PartitionSynchronizer now has independent handling based on lease type/version, as the handling is now dependent on what does the lease represent (full partition or a range)
  • FeedProcessor uses ChangeFeedPartitionKeyResultSetIteratorCore which now can set the EPK filter headers for leases that represent a range.
  • Adding new sets of UTs to cover existing classes and new classes

Type of change

  • New feature (non-breaking change which adds functionality)

j82w
j82w previously approved these changes Dec 14, 2020
@ealsur ealsur merged commit 2674a02 into master Dec 14, 2020
@ealsur ealsur deleted the users/ealsur/mergecfp branch December 14, 2020 21:59
@ghost
Copy link

ghost commented Dec 15, 2021

Closing due to in-activity, pease feel free to re-open.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants