[5.0] Online import

[kennytm]

Description

Lightning currently requires the cluster to be offline and empty because

1. SST import violates ACID
2. Import mode trades space amplification for write speed
3. Region-split works better on empty ranges
4. Cluster performance is reduced.

We may support online import based on pingcap/br#87.

Category

• Feature
• Performance

Value

Value description

This is needed for data-hub and data middle platform cases. A lot of cases, users have not only real-time data but also third party data, to import and analyze together with transactional data. This might as well be merged together with batch scenario and speed them up all together.

Value score

• (TBD) / 5

Workload estimation

• 30 person-day

Time

GanttStart: YYYY-MM-DD
GanttDue: YYYY-MM-DD

[IANTHEREAL]

discussed with @ilovesoup and after a collection of scenarios, there are no specific scenarios that require this feature, so we can lower our priority to P2 and prioritize it later when we encounter a specific scenario.

also /cc @kennytm @overvenus @scsldb