Improved Cross-Region deployment features

[shenli]

Description

Deploy a cluster across multiple geo regions. Schedule the data to the AZ/region near the workload, like Google Spanner.

We need to reduce query latency by reducing cross-region data access. The applications deployed in a certain region may only read/write the data related to that region. So for better performance, the basic idea is to put the related data near the application (data locality). Placement Rules is a powerful tool provided by PD. We can use it to place all(or at least the majority) replicas of a table/partition in a specific region. This could mitigate the cross-region network round-trip for both read and write operations.

Besides the data access, another thing we should consider is the transaction timestamp allocation (TSO). For now, it's allocated by the PD leader only. So for the transactions that occur not in the same region with the PD leader, there is still one cross-region round-trip to get the transaction timestamp. We need to figure out a way to allocate transaction timestamps from PD followers. In this way, we can mitigate the cross-region network round-trip for TSO allocation.

Task List

Note: Each of the following big tasks is composed of lots of small tasks, please click on the task URL to get more detail of each big task.

• P0: Define the placement of data by SQL statements #18030 Config Placement Rules of Data by SQL Statements (Partition Table)
• P0: Support Region/Global Linearizability Consistency #18253 Support Regional/Global Linearizability Consistency
  • P0: Local/Global Transaction Txn Allocator in Cross-DC Deployment Roadmap tikv/pd#2759 Support Regional and Global TSO Allocator in Cross-Region Deployment
  • P0: Local/Global Transaction in Cross-DC Deployment Tasklist #20448 Support Regional and Global Transaction in Cross-Region Deployment
• P1: Read local replica to reduce read latency in cross region scenario #18033 Read Local Replica to Reduce Latency in Cross-Region Deployment
  • P0: Support timestamp bounded read-only transactions on data replicas Tasklist #21094 Support timestamp bounded read-only transactions on data replicas

Technical Design Docs

• docs: Add RFC for "Defining placement rules in SQL" #18199 Config the placement of data through SQL
• Local/Global Transaction in Cross-Region Deployment (Chinese)
• Propoasl: Follower to serve local(stale) read (In Chinese)

Progress Tracking

• Define the placement of data by SQL statements("Geo Partition") https://github.com/pingcap/tidb/projects/49
• Weekly Report: Cross-Region Deployment & GEO Partition, which contains the progress of the whole project:
  • Config Placement Rules by SQL
  • Support Regional and Global TSO Allocator in Cross-Region Deployment
  • Support Regional and Global Transaction in Cross-Region Deployment
  • Read Local Replica to Reduce Latency in Cross-Region Deployment

Join Us

• Slack: #sig-scheduling in the TiDB-Community Slack Workspace

[zz-jason]

Some questions about Geo-Partition:

• We planned to support geo-partition on the partition table. But at present, TiDB only supports hash and range partition. Is it enough for applications to map data into the different partitions and further map specific partitions to different Region/Availability Zone(AZ)?

• If the applications require to join two tables, one is the geo-partitioned table, another is a normal table. Or if the applications just need to query another non-partition table. In these scenarios, We may need to figure out a way to replicate the normal table and place each replication to each Region/AZ to save the query latency.

[zz-jason]

In order to improve the query performance on a Region or AZ, we'd better improve the partition pruning algorithm to prune un-related partition as many as possible. IMO #17474 and #18016 are also important in cross-region workloads.