Support heterogeneous design

[Yisaer]

Description

Currently, the spec for the TidbCluster describes a group of PD/TiKV/TiDB instances.
For each component, the spec is the same for all of the Pods.
This design is easy to use and also covers the most cases for using the TiDB cluster.

However, as a distributed database with clear and multiple layers, we may need to deploy the component with different configurations to meet the different requirements.

For example, as the SQL layer, the TiDB component could be composed of multiple instances with different resource requests and configurations to handle the different workloads (AP/TP query).
For the storage layer, the TiKV component could be composed of multiple instances with different store labels which determine the data distribution among different instances.
And this is also useful for the auto-scaling case, where we may want to scale out TiDBs/TiKVs with different resources or configurations from the original TidbCluster.

To implement the above proposal, we can:

• Make the spec.pd, spec.tidb, and spec.tikv optional in the TidbCluster CR, e.g. change them to pointer.
• Add cluster reference in the TidbCluster CR so that the components can start with the pdAddress in the cluster reference.

In this case, we can:

• Deploy TidbCluster CR with spec.tikv + spec.pdAddress to deploy a TiKV cluster with different configurations and join the existing TiDB Cluster
• Deploy TidbCluster CR with spec.tidb + spec.pdAddress to deploy a TiDB cluster with different configurations and join the existing TiDB Cluster
• Actually all of the supported components in the TidbCluster CR, e.g. TiFlash can be deployed with different configurations with minor changes.

Category

Feature

Value

Support heterogeneous structured components of TiDB Cluster on Kubernetes

TODO lists

• P0 features

  • Make the spec.pd, spec.tidb, and spec.tikv optional in the TidbCluster CR, e.g. change them to pointer. Make the spec.pd, spec.tidb, and spec.tikv optional in the TidbCluster CR #2971
  • Add TidbCluster Reference in the TidbCluster CR Add TidbCluster Reference in the TidbCluster CR #2972
  • TidbCluster controller update for the spec.pd, spec.tidb, and spec.tikv change. TidbCluster controller update for the spec.pd, spec.tidb, and spec.tikv change #2973
  • TidbCluster controller update for the change in Add TidbCluster Reference in the TidbCluster CR #2972, so that the components can start with the pdAddress in the cluster reference. (Focus on TiKV and TiDB first) TidbCluster controller update for cluster reference change #2974

• P1 features

  • The status of each TidbCluster should only include the components managed by this TidbCluster and based on this, the failureMembers/failureStores will only include the Pods managed by this TidbCluster so that failover function can work normally Status sync update for heterogeneous deployment #2975
  • TidbCluster controller update for the change in Add TidbCluster Reference in the TidbCluster CR #2972, so that the components can start with the pdAddress in the cluster reference. (Focus on TiFlash) TidbCluster controller update for cluster reference change - TiFlash #3038
  • TLS support for heterogeneous components TLS support for heterogeneous components #3061
  • Monitoring for heterogeneous components Monitoring for heterogeneous components #3062
  • Doc update Doc for heterogeneous deployment docs-tidb-operator#707

Workload Estimation (P0 features)

30

Time

DevProgress: 100%
QAProgress: 100%
DocProgress: 100%

GanttStart: 2020-07-20
GanttDue: 2020-09-30

Documentions

Project

[DanielZhangQD]

We decide to reuse the existing TidbCluster CR to implement the heterogeneous deployment.

Record the original solution here:

Deprecated

Description

Currently, the spec for the TidbCluster describes a group of PD/TiKV/TiDB instances. For each kind of components,
their specs are all the same. This design is easy to use and also covers the most cases for using the tidb cluster.

However, as a distributed database with clear and multilayers, the components in each layer could be different from
each other to meet the different requirements.

For example, as the SQL layer, tidb components could be composed of multiple instances with different resource requests
and configurations to handle the different workloads (AP/TP query). For the storage layer, tikv components could be
composed of multiple instances with different store labels which decided the data distribution as a whole storage system.

Add 2 new CRD as TiKVGroup and TiDBGroup which represents a group of TiDB and TiKV instances. As a whole system, one
TiDB Cluster can be composed of one TidbCluster and multiple TiKVGroup and TiDBGroup. Here is the basic example:

For TiKVGroup:

type TiKVGroup struct {
	Spec   TiKVGroupSpec
	Status TiKVGroupStatus
}

type TiKVGroupSpec struct {
    TiKVSpec
    ClusterName string
}

type TiKVGroupStatus struct {
    TiKVStatus
}

Category

Cloud TiDB

Value

Support heterogeneous structured TidbCluster on Kubernetes

TODO lists

• P0 features (support TiKVGroup Controller)

  • Support TiKVGroup/TiDBGroup API design
  • Support Controller For TiKVGroup
  • Support Upgrading For TiKVGroup
  • Support Scaling For TiKVGroup

• P1 features (support TiDBGroup Controller / TiKVGroup Failover)

  • Support Controller For TiDBGroup
  • Support Upgrading For TiDBGroup
  • Support Scaling For TiDBGroup
  • Support Failover for TiDBGroup Pods
  • Support Failover for TiKVGroup Pods
  • Support setting Timezone for TiKVGroup & TiDBGroup Pods

• P3 features

  • Support Mixed Structured Design for TiKVGroup / TiDBGroup

Documentions

document

[DanielZhangQD]

@mikechengwei You can take this issue if it's OK.
Thanks for your contribution!

[mikechengwei]

/assign

[DanielZhangQD]

@cofyc Please also help review the solution here #2240 (comment)

[cofyc]

can you make an update to the proposal here ?
https://github.com/pingcap/tidb-operator/blob/master/docs/design-proposals/2020-06-12-heterogeneous-design-for-tidb-cluister.md

[DanielZhangQD]

OK.

[cofyc]

SGTM

[uglyengineer]

This issue is a sub-item of pingcap/tidb#18374, no need to be directly related in Longterm board.

[DanielZhangQD]

@mikechengwei Thanks a lot for your contribution!