From e74c80560a4c093723fc82de3ee6ce42b10bd4c6 Mon Sep 17 00:00:00 2001 From: Aolin Date: Fri, 6 Sep 2024 16:10:20 +0800 Subject: [PATCH 01/45] releases: add release notes for v8.4.0 --- releases/release-8.4.0.md | 194 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 194 insertions(+) create mode 100644 releases/release-8.4.0.md diff --git a/releases/release-8.4.0.md b/releases/release-8.4.0.md new file mode 100644 index 000000000000..2aad2e03b8b7 --- /dev/null +++ b/releases/release-8.4.0.md @@ -0,0 +1,194 @@ +--- +title: TiDB 8.4.0 Release Notes +summary: Learn about the new features, compatibility changes, improvements, and bug fixes in TiDB 8.4.0. +--- + +# TiDB 8.4.0 Release Notes + +Release date: xx xx, 2024 + +TiDB version: 8.4.0 + +Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with-tidb) + +8.4.0 introduces the following key features and improvements: + +## Feature details + +### Scalability + +* Feature summary [#issue-number](issue-link) @[pr-auorthor-id](author-link) + + Feature descriptions (including what the feature is, why it is valuable for users, and how to use this feature generally) + + For more information, see [documentation](doc-link). + +### Performance + +* Feature summary [#issue-number](issue-link) @[pr-auorthor-id](author-link) + + Feature descriptions (including what the feature is, why it is valuable for users, and how to use this feature generally) + + For more information, see [documentation](doc-link). + +### Reliability + +* Feature summary [#issue-number](issue-link) @[pr-auorthor-id](author-link) + + Feature descriptions (including what the feature is, why it is valuable for users, and how to use this feature generally) + + For more information, see [documentation](doc-link). + +### Availability + +* Feature summary [#issue-number](issue-link) @[pr-auorthor-id](author-link) + + Feature descriptions (including what the feature is, why it is valuable for users, and how to use this feature generally) + + For more information, see [documentation](doc-link). + +### SQL + +* Feature summary [#issue-number](issue-link) @[pr-auorthor-id](author-link) + + Feature descriptions (including what the feature is, why it is valuable for users, and how to use this feature generally) + + For more information, see [documentation](doc-link). + +### DB operations + +* Feature summary [#issue-number](issue-link) @[pr-auorthor-id](author-link) + + Feature descriptions (including what the feature is, why it is valuable for users, and how to use this feature generally) + + For more information, see [documentation](doc-link). + +### Observability + +* Feature summary [#issue-number](issue-link) @[pr-auorthor-id](author-link) + + Feature descriptions (including what the feature is, why it is valuable for users, and how to use this feature generally) + + For more information, see [documentation](doc-link). + +### Security + +* Feature summary [#issue-number](issue-link) @[pr-auorthor-id](author-link) + + Feature descriptions (including what the feature is, why it is valuable for users, and how to use this feature generally) + + For more information, see [documentation](doc-link). + +### Data migration + +* Feature summary [#issue-number](issue-link) @[pr-auorthor-id](author-link) + + Feature descriptions (including what the feature is, why it is valuable for users, and how to use this feature generally) + + For more information, see [documentation](doc-link). + +## Compatibility changes + +> **Note:** +> +> This section provides compatibility changes you need to know when you upgrade from v8.3.0 to the current version (v8.4.0). If you are upgrading from v8.2.0 or earlier versions to the current version, you might also need to check the compatibility changes introduced in intermediate versions. + +### Behavior changes + +* Behavior change + +### System variables + +| Variable name | Change type | Description | +|--------|------------------------------|------| +| | | | +| | | | +| | | | +| | | | + +### Configuration file parameters + +| Configuration file | Configuration parameter | Change type | Description | +| -------- | -------- | -------- | -------- | +| | | | | +| | | | | +| | | | | +| | | | | + +### System tables + +## Offline package changes + +## Deprecated features + +* The following features are deprecated starting from v8.4.0: + + * Deprecated feature + +* The following features are planned for deprecation in future versions: + + * TiDB introduces the system variable [`tidb_enable_auto_analyze_priority_queue`](/system-variables.md#tidb_enable_auto_analyze_priority_queue-new-in-v800), which controls whether priority queues are enabled to optimize the ordering of tasks that automatically collect statistics. In future releases, the priority queue will be the only way to order tasks for automatically collecting statistics, so this system variable will be deprecated. + * TiDB introduces the system variable [`tidb_enable_async_merge_global_stats`](/system-variables.md#tidb_enable_async_merge_global_stats-new-in-v750) in v7.5.0. You can use it to set TiDB to use asynchronous merging of partition statistics to avoid OOM issues. In future releases, partition statistics will be merged asynchronously, so this system variable will be deprecated. + * It is planned to redesign [the automatic evolution of execution plan bindings](/sql-plan-management.md#baseline-evolution) in subsequent releases, and the related variables and behavior will change. + * In v8.0.0, TiDB introduces the [`tidb_enable_parallel_hashagg_spill`](/system-variables.md#tidb_enable_parallel_hashagg_spill-new-in-v800) system variable to control whether TiDB supports disk spill for the concurrent HashAgg algorithm. In future versions, the [`tidb_enable_parallel_hashagg_spill`](/system-variables.md#tidb_enable_parallel_hashagg_spill-new-in-v800) system variable will be deprecated. + * The TiDB Lightning parameter [`conflict.max-record-rows`](/tidb-lightning/tidb-lightning-configuration.md#tidb-lightning-task) is planned for deprecation in a future release and will be subsequently removed. This parameter will be replaced by [`conflict.threshold`](/tidb-lightning/tidb-lightning-configuration.md#tidb-lightning-task), which means that the maximum number of conflicting records is consistent with the maximum number of conflicting records that can be tolerated in a single import task. + +* The following features are planned for removal in future versions: + + * Starting from v8.0.0, TiDB Lightning deprecates the [old version of conflict detection](/tidb-lightning/tidb-lightning-physical-import-mode-usage.md#the-old-version-of-conflict-detection-deprecated-in-v800) strategy for the physical import mode, and enables you to control the conflict detection strategy for both logical and physical import modes via the [`conflict.strategy`](/tidb-lightning/tidb-lightning-configuration.md#tidb-lightning-task) parameter. The [`duplicate-resolution`](/tidb-lightning/tidb-lightning-configuration.md) parameter for the old version of conflict detection will be removed in a future release. + +## Improvements + ++ TiDB + ++ TiKV + ++ PD + ++ TiFlash + ++ Tools + + + Backup & Restore (BR) + + + TiCDC + + + TiDB Data Migration (DM) + + + TiDB Lightning + + + Dumpling + + + TiUP + + + TiDB Binlog + +## Bug fixes + ++ TiDB + ++ TiKV + ++ PD + ++ TiFlash + ++ Tools + + + Backup & Restore (BR) + + + TiCDC + + + TiDB Data Migration (DM) + + + TiDB Lightning + + + Dumpling + + + TiUP + + + TiDB Binlog + +## Contributors + +We would like to thank the following contributors from the TiDB community: From 248c152a64054a902d828be9d762ac1f8837f683 Mon Sep 17 00:00:00 2001 From: Ben Meadowcroft Date: Tue, 17 Sep 2024 08:47:31 -0700 Subject: [PATCH 02/45] Update release-8.4.0.md Added release note content for various items --- releases/release-8.4.0.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/releases/release-8.4.0.md b/releases/release-8.4.0.md index 2aad2e03b8b7..e3b43f2e5f6c 100644 --- a/releases/release-8.4.0.md +++ b/releases/release-8.4.0.md @@ -57,6 +57,18 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- ### DB operations +* PITR adds client-side log backup data encryption support (experimental) [#issue-number](issue-link) @[Tristan1900](https://github.com/Tristan1900) + + Previously only the data from a snapshot based backup could be encrypted (on the client side) with a data key provided by the user. With this feature, log backups may now also be encrypted, ensuring that the confidentiality of information within the backup data is secured. + + For more information, see [documentation](doc-link). + +* BR reduces requires storage permissions for restores [#55870](https://github.com/pingcap/tidb/issues/55870) @[Leavrth](https://github.com/Leavrth) + + Previously, when BR was restoring data, checkpoint information about the progress of the restore was recorded in the location hosting the backup data. These restore checkpoints enabled restoration to be quickly resumed if it was interrupted. With this feature, the restore checkpoints are now stored in the target TiDB cluster. This means that BR only requires read access to the backup dataset location for restores. + + For more information, see [documentation](doc-link). + * Feature summary [#issue-number](issue-link) @[pr-auorthor-id](author-link) Feature descriptions (including what the feature is, why it is valuable for users, and how to use this feature generally) @@ -73,6 +85,12 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- ### Security +* TiDB BR Supports AWS IMDSv2 [#16443](https://github.com/tikv/tikv/issues/16443) @[pingyu](https://github.com/pingyu) + + TiDB BR now supports AWS's Instance Metadata Service Version 2 (IMDSv2) when deployed on AWS EC2. This enables users to configure the newer session-oriented method on their EC2 instances, and for BR to be able to successfully use the IAM Role associated with the instance to access AWS S3 with the appropriate privileges. + + For more information, see [documentation](/backup-and-restore-storages#authentication). + * Feature summary [#issue-number](issue-link) @[pr-auorthor-id](author-link) Feature descriptions (including what the feature is, why it is valuable for users, and how to use this feature generally) From d05fef6969b53b1502dcba1111f0437c8963cd65 Mon Sep 17 00:00:00 2001 From: Aolin Date: Wed, 18 Sep 2024 15:46:59 +0800 Subject: [PATCH 03/45] update client-side log backup, BR reduces requires storage permissions for restores, BR Supports AWS IMDSv2 Signed-off-by: Aolin --- releases/release-8.4.0.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/releases/release-8.4.0.md b/releases/release-8.4.0.md index e3b43f2e5f6c..bf7b0f2ef3a6 100644 --- a/releases/release-8.4.0.md +++ b/releases/release-8.4.0.md @@ -57,13 +57,13 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- ### DB operations -* PITR adds client-side log backup data encryption support (experimental) [#issue-number](issue-link) @[Tristan1900](https://github.com/Tristan1900) +* PITR adds client-side log backup data encryption support (experimental) [55834](https://github.com/pingcap/tidb/issues/55834) @[Tristan1900](https://github.com/Tristan1900) **tw@qiancai** Previously only the data from a snapshot based backup could be encrypted (on the client side) with a data key provided by the user. With this feature, log backups may now also be encrypted, ensuring that the confidentiality of information within the backup data is secured. For more information, see [documentation](doc-link). -* BR reduces requires storage permissions for restores [#55870](https://github.com/pingcap/tidb/issues/55870) @[Leavrth](https://github.com/Leavrth) +* BR reduces requires storage permissions for restores [#55870](https://github.com/pingcap/tidb/issues/55870) @[Leavrth](https://github.com/Leavrth) **tw@Oreoxmt** Previously, when BR was restoring data, checkpoint information about the progress of the restore was recorded in the location hosting the backup data. These restore checkpoints enabled restoration to be quickly resumed if it was interrupted. With this feature, the restore checkpoints are now stored in the target TiDB cluster. This means that BR only requires read access to the backup dataset location for restores. @@ -85,7 +85,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- ### Security -* TiDB BR Supports AWS IMDSv2 [#16443](https://github.com/tikv/tikv/issues/16443) @[pingyu](https://github.com/pingyu) +* TiDB BR Supports AWS IMDSv2 [#16443](https://github.com/tikv/tikv/issues/16443) @[pingyu](https://github.com/pingyu) **tw@hfxsd** TiDB BR now supports AWS's Instance Metadata Service Version 2 (IMDSv2) when deployed on AWS EC2. This enables users to configure the newer session-oriented method on their EC2 instances, and for BR to be able to successfully use the IAM Role associated with the instance to access AWS S3 with the appropriate privileges. From e6766c37388ca8644be896d2006722b33acbf728 Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Fri, 20 Sep 2024 11:01:34 +0800 Subject: [PATCH 04/45] Update releases/release-8.4.0.md --- releases/release-8.4.0.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/releases/release-8.4.0.md b/releases/release-8.4.0.md index bf7b0f2ef3a6..feb572364889 100644 --- a/releases/release-8.4.0.md +++ b/releases/release-8.4.0.md @@ -31,6 +31,24 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- For more information, see [documentation](doc-link). +* The performance of batch user creation and password changes has been improved by hundreds of times [#55604](https://github.com/pingcap/tidb/pull/55604) @[wjhuang2016](https://github.com/wjhuang2016) **tw@hfxsd** + + In SaaS scenarios, you might need to batch-create a large number of users, rotate passwords periodically, and complete these tasks within a specific time window. Starting from v8.4.0, the performance of batch user creation and password rotation has been significantly improved. Additionally, you can further enhance performance by increasing concurrency through a higher number of session connections, which greatly reduces execution time for these operations. + +* Partitioned tables support global indexes (GA) [#45133](https://github.com/pingcap/tidb/issues/45133) @[mjonss](https://github.com/mjonss) @[Defined2014](https://github.com/Defined2014) @[jiyfhust](https://github.com/jiyfhust) @[L-maple](https://github.com/L-maple) + + In previous versions of partitioned tables, some limitations exist because global indexes are not supported. For example, the unique key must use every column in the table's partitioning expression. If the query condition does not use the partitioning key, the query will scan all partitions, resulting in poor performance. Starting from v7.6.0, the system variable [`tidb_enable_global_index`](/system-variables.md#tidb_enable_global_index-new-in-v760) is introduced to enable the global index feature. But this feature was under development at that time and it is not recommended to enable it. + + Starting with v8.3.0, the global index feature is released as an experimental feature. You can explicitly create a global index for a partitioned table with the keyword `Global` to remove the restriction that the unique key must use every column in the table's partitioning expression, to meet flexible business needs. Global indexes also enhance the performance of queries that do not include partition keys. + + In v8.4.0, this feature becomes generally available (GA). + + For more information, see [documentation](/partitioned-table.md#global-indexes). + +* Optimize query performance for cached tables in some scenarios [#43249](https://github.com/pingcap/tidb/issues/43249) @[tiancaiamao](https://github.com/tiancaiamao) **tw@hfxsd** + + Optimize query performance on cached tables by up to 5. 4 times when using `IndexLookup` to execute `SELECT ... LIMIT 1` with `IndexLookup`. Improve the performance of `IndexLookupReader` in full table scan and primary key query scenarios. + ### Reliability * Feature summary [#issue-number](issue-link) @[pr-auorthor-id](author-link) From fd382637063fec325b8c39730ec42bd08127afe7 Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Fri, 20 Sep 2024 13:54:12 +0800 Subject: [PATCH 05/45] Update releases/release-8.4.0.md --- releases/release-8.4.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-8.4.0.md b/releases/release-8.4.0.md index feb572364889..fdc10ce9f43c 100644 --- a/releases/release-8.4.0.md +++ b/releases/release-8.4.0.md @@ -47,7 +47,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- * Optimize query performance for cached tables in some scenarios [#43249](https://github.com/pingcap/tidb/issues/43249) @[tiancaiamao](https://github.com/tiancaiamao) **tw@hfxsd** - Optimize query performance on cached tables by up to 5. 4 times when using `IndexLookup` to execute `SELECT ... LIMIT 1` with `IndexLookup`. Improve the performance of `IndexLookupReader` in full table scan and primary key query scenarios. + Optimize the query performance of cached tables by up to 5.4 times when using `IndexLookup` to execute `SELECT ... LIMIT 1` with `IndexLookup`. Improve the performance of `IndexLookupReader` in full table scan and primary key query scenarios. ### Reliability From d433b0c344106a1d0b752d918f06f9847ce8e1b0 Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Fri, 20 Sep 2024 15:05:46 +0800 Subject: [PATCH 06/45] Apply suggestions from code review --- releases/release-8.4.0.md | 35 ++++++++++++++++++++++++++++++----- 1 file changed, 30 insertions(+), 5 deletions(-) diff --git a/releases/release-8.4.0.md b/releases/release-8.4.0.md index fdc10ce9f43c..14378819fc83 100644 --- a/releases/release-8.4.0.md +++ b/releases/release-8.4.0.md @@ -57,6 +57,25 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- For more information, see [documentation](doc-link). +* Support runaway queries to switch resource groups [#54434](https://github.com/pingcap/tidb/issues/54434) @[JmPotato](https://github.com/JmPotato) **tw@hfxsd** + + In TiDB v8.4.0, you can redirect runaway queries to a specific resource group. If the `COOLDOWN` mechanism fails to lower resource consumption, you can create a [resource group](/tidb-resource-control.md#create-a-resource-group) and set the `SWITCH_GROUP` parameter to move identified runaway queries to this group. Meanwhile, subsequent queries within the same session will continue to execute in the original resource group. By switching resource groups, you can more precisely manage resource usage and better control the impact of runaway queries. + + For more information, see [documentation](/tidb-resource-control.md#query_limit-parameters). + +* The system variable `tidb_scatter_region` supports the cluster-level Region scattering strategy [#55184](https://github.com/pingcap/tidb/issues/55184) @[D3Hunter](https://github.com/D3Hunter) **tw@hfxsd** + + In previous versions, the system variable `tidb_scatter_region` can only be enabled or disabled. When enabled, it applies a table-level scattering strategy during batch table creation. However, when creating hundreds of thousands of tables in a batch, this approach results in a concentration of regions on a few TiKV nodes, causing out-of-memory (OOM) issues on those nodes. + + To address this, starting from v8.4.0, `tidb_scatter_region` is changed to a string type. It now supports a cluster-level scattering strategy, helping scatter regions more evenly and preventing OOM problems on TiKV nodes. + + For more information, see [documentation](/system-variables.md#tidb_scatter_region). + +* Support setting resource caps for background tasks of resource control [#56019](https://github.com/pingcap/tidb/issues/56019) @[glorv](https://github.com/glorv) **tw@hfxsd** + + TiDB resource control can identify and lower the priority of background tasks. In certain scenarios, you might want to limit the resource consumption of these tasks, even when resources are available. Starting from v8.4.0, you can use the `UTILIZATION_LIMIT` parameter to set a maximum percentage of resources that a background task can consume. Each node will ensure that the resource usage of all background tasks stays within this limit. This feature enables precise control over resource consumption for background tasks, enhancing cluster stability. + + For more information, see [documentation](/tidb-resource-control.md#manage-background-tasks). ### Availability * Feature summary [#issue-number](issue-link) @[pr-auorthor-id](author-link) @@ -95,17 +114,23 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- ### Observability -* Feature summary [#issue-number](issue-link) @[pr-auorthor-id](author-link) +* Display the CPU time of TiDB and TiKV in the system table [#55542](https://github.com/pingcap/tidb/issues/55542) @[yibin87](https://github.com/yibin87) **tw@hfxsd** - Feature descriptions (including what the feature is, why it is valuable for users, and how to use this feature generally) + The [Top SQL page](/dashboard/top-sql.md) of [TiDB Dashboard](/dashboard/dashboard-intro.md) displays SQL statements with high CPU consumption. Starting from v8.4.0, TiDB includes CPU time consumption data in the system table, alongside other session or SQL metrics, allowing you to easily monitor high CPU usage from multiple perspectives. This information is especially useful in identifying the root cause of issues such as CPU spikes or hotspots in cluster read/write operations. - For more information, see [documentation](doc-link). + - [STATEMENTS_SUMMARY](/statement-summary-tables.md) adds `AVG_TIDB_CPU_TIME` and `AVG_TIKV_CPU_TIME` to show the average CPU time consumed by individual SQL statements historically. + - [INFORMATION_SCHEMA.PROCESSLIST](/information-schema/information-schema-processlist.md) adds `TIDB_CPU` and `TIKV_CPU` to display the cumulative CPU consumption of currently executing SQL statements in a session. + - The [slow Log](/analyze-slow-queries.md) adds the `Tidb_cpu_time` and `Tikv_cpu_time` fields to show the CPU time of captured SQL statements. + + By default, TiKV CPU time is displayed. Collecting TiDB CPU time introduces an additional overhead (about 8%), so TiDB CPU time is only displayed as the actual value when the [Top SQL feature](https://github.com/dashboard/top-sql.md) is enabled; otherwise, it will always display as `0`. + + For more information, see [documentation](/information-schema/information-schema-processlist.md) and [documentation](information-schema/information-schema-slow-query.md). ### Security -* TiDB BR Supports AWS IMDSv2 [#16443](https://github.com/tikv/tikv/issues/16443) @[pingyu](https://github.com/pingyu) **tw@hfxsd** +* BR supports AWS IMDSv2 [#16443](https://github.com/tikv/tikv/issues/16443) @[pingyu](https://github.com/pingyu) **tw@hfxsd** - TiDB BR now supports AWS's Instance Metadata Service Version 2 (IMDSv2) when deployed on AWS EC2. This enables users to configure the newer session-oriented method on their EC2 instances, and for BR to be able to successfully use the IAM Role associated with the instance to access AWS S3 with the appropriate privileges. + BR now supports AWS Instance Metadata Service Version 2 (IMDSv2) when deployed on AWS EC2. This allows you to configure the newer session-oriented method on EC2 instances, enabling BR to successfully use the instance's associated IAM role to access AWS S3 with the appropriate privileges. For more information, see [documentation](/backup-and-restore-storages#authentication). From 95b7f2e14c482584eb3072c8e4cd204ac6bb7f7d Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Mon, 23 Sep 2024 11:13:28 +0800 Subject: [PATCH 07/45] Apply suggestions from code review --- releases/release-8.4.0.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/releases/release-8.4.0.md b/releases/release-8.4.0.md index 14378819fc83..f3cb6ef50a66 100644 --- a/releases/release-8.4.0.md +++ b/releases/release-8.4.0.md @@ -41,7 +41,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- Starting with v8.3.0, the global index feature is released as an experimental feature. You can explicitly create a global index for a partitioned table with the keyword `Global` to remove the restriction that the unique key must use every column in the table's partitioning expression, to meet flexible business needs. Global indexes also enhance the performance of queries that do not include partition keys. - In v8.4.0, this feature becomes generally available (GA). + In v8.4.0, this feature becomes generally available (GA). You can directly use the keyword `GLOBAL` to create a global index, instead of setting the system variable [`tidb_enable_global_index`](/system-variables.md#tidb_enable_global_index-new-in-v760) to enable the global index feature. For more information, see [documentation](/partitioned-table.md#global-indexes). @@ -202,8 +202,11 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- + TiDB -+ TiKV +- Optimize MEMDB implementation to reduce write latency in transactions and TiDB CPU usage [#55287](https://github.com/pingcap/tidb/issues/55287) @[you06](https://github.com/you06) **tw@hfxsd** ++ TiKV +- Increase the default value of Region from 96 MiB to 256 MiB to avoid the extra overhead caused by too many Regions [#17309](https://github.com/tikv/tikv/issues/17309) [LykxSassinator](https://github.com/LykxSassinator) **tw@hfxsd** +- Introduce a new `spill-dir` configuration in Raft Engine to support multi-disk storage for Raft logs. When the disk containing the home directory (`dir`) runs out of space, Raft Engine automatically writes new logs to `spill-dir`, ensuring continuous operation. [LykxSassinator](https://github.com/LykxSassinator) **tw@hfxsd** + PD + TiFlash From c38c9e4b96cadb6773ff6ec2c6d80aaae83190ac Mon Sep 17 00:00:00 2001 From: Aolin Date: Mon, 23 Sep 2024 16:41:44 +0800 Subject: [PATCH 08/45] update TiProxy traffic replay Signed-off-by: Aolin --- releases/release-8.4.0.md | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/releases/release-8.4.0.md b/releases/release-8.4.0.md index f3cb6ef50a66..5c0008e7ace8 100644 --- a/releases/release-8.4.0.md +++ b/releases/release-8.4.0.md @@ -76,13 +76,23 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- TiDB resource control can identify and lower the priority of background tasks. In certain scenarios, you might want to limit the resource consumption of these tasks, even when resources are available. Starting from v8.4.0, you can use the `UTILIZATION_LIMIT` parameter to set a maximum percentage of resources that a background task can consume. Each node will ensure that the resource usage of all background tasks stays within this limit. This feature enables precise control over resource consumption for background tasks, enhancing cluster stability. For more information, see [documentation](/tidb-resource-control.md#manage-background-tasks). + ### Availability -* Feature summary [#issue-number](issue-link) @[pr-auorthor-id](author-link) +* TiProxy supports traffic replay (experimental) [#642](https://github.com/pingcap/tiproxy/issues/642) @[djshow832](https://github.com/djshow832) **tw@Oreoxmt** - Feature descriptions (including what the feature is, why it is valuable for users, and how to use this feature generally) + Starting from TiProxy v1.3.0, you can use TiProxy to capture access traffic in a TiDB production cluster and replay it in a test cluster at a specified rate. This feature enables you to reproduce actual workloads from the production cluster in a test environment, verifying SQL statement execution results and performance. - For more information, see [documentation](doc-link). + Traffic replay is suitable for the following scenarios: + + - Validate TiDB version upgrades + - Assess change impact + - Validate performance before TiDB scaling + - Test performance limits + + You can use `tiproxyctrl` to connect to the TiProxy instance and perform traffic capture and replay. + + For more information, see [documentation](/tiproxy/tiproxy-traffic-replay.md). ### SQL From 842af141d0022c8d2b0493748e91704d2157da05 Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Mon, 23 Sep 2024 17:15:07 +0800 Subject: [PATCH 09/45] Update releases/release-8.4.0.md --- releases/release-8.4.0.md | 77 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 77 insertions(+) diff --git a/releases/release-8.4.0.md b/releases/release-8.4.0.md index 5c0008e7ace8..9260b5823962 100644 --- a/releases/release-8.4.0.md +++ b/releases/release-8.4.0.md @@ -13,6 +13,83 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- 8.4.0 introduces the following key features and improvements: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
CategoryFeature/EnhancementDescription
Scalability and Performance 执行计划缓存在实例内共享（实验特性） 实例级执行计划缓存支持在内存中缓存更多的执行计划，消除 SQL 编译时所消耗的时间，从而减少 SQL 的运行时间，提升 OLTP 系统的性能和吞吐。同时，也能更好的控制内存占用，提升数据库稳定性。
Global indexes for partitioned tables (GA)Global indexes can effectively improve the efficiency of retrieving non-partitioned columns, and remove the restriction that a unique key must contain the partition key. This feature extends the usage scenarios of TiDB partitioned tables and avoids some of the application modification work that might be required for data migration.
TiDB 并行获取 TSO在高并发场景下，并行获取 TSO 能够有效降低等待获取 TSO 的时间，提升集群的吞吐。
Improve the execution efficiency of administrative SQL statementsIn some SaaS systems, there is a need to create users in batch and rotate passwords regularly. TiDB enhances the performance of creating and modifying database users, ensuring these operations can be completed within the desired time window.
Improve query performance for cached tablesImprove query performance for index scanning on cached tables, with improvements of up to 5.4 times in some scenarios. For high-speed queries on small tables, cached tables can significantly enhance overall performance.
Reliability and Availability Support more triggers for runaway queries and provide the ability to switch resource groupsRunaway Queries offer an effective way to mitigate the impact of unexpected SQL performance issues on systems. The new version introduces PROCESSED_KEYS and RU as identifying conditions, allowing identified queries to be placed into a specified resource group for more precise identification and control of runaway queries.
Support setting resource usage caps for background tasks for resource control By setting a percentage cap on background tasks of resource control, you can manage their resource consumption based on the needs of different business systems. This ensures background tasks consume minimal resources, maintaining the service quality of online operations.
TiProxy 流量捕捉和回放在做集群升级、迁移、部署变化等重要变更之前，通过捕捉真实负载来验证目标集群的性能，确保变更的成功。
统计信息收集自适应并发度自动统计信息收集会根据节点规模和硬件规格自动决定采集并发度，提升统计信息收集效率，减少手工调优，保证集群性能稳定。
SQL 外键成为正式功能支持 MySQL 兼容的外键约束，维护数据一致性，进一步提升了 TiDB 对 MySQL 的兼容能力。
向量搜索功能（实验特性）加速向量搜索的性能，适用于检索增强生成（RAG）、语义搜索、推荐系统等应用类型。把 TiDB 应用场景扩展到 AI 和 大语言模型（LLM）领域。
数据库管理和可观测性 持久化内存表到 Workload Repository（实验特性） 持久化内存表中的运行指标和状态信息，是观测性的重要增强，能极大提升过往问题诊断和追溯的效率，并为未来的自动化运维，提供了数据集支持。 围绕 Workload Repository 构建报告、诊断、推荐一体化的能力，会成为未来提升 TiDB 易用性的重要组成。
Display TiKV and TiDB CPU times in memory tablesCPU times are now integrated into a system table and displayed alongside other session or SQL metrics, allowing for easier observation of operations with high CPU consumption from multiple perspectives, improving diagnostic efficiency. This is particularly useful for diagnosing instances with CPU spikes or read/write hotspots in the cluster.
Support backing up TiKV instances with IMDSv2 service enabledAWS EC2 now uses IMDSv2 as the default metadata service. TiDB supports data backups from TiKV instances with IMDSv2 enabled, enhancing your ability to run TiDB clusters in public cloud environments.
安全 备份数据加密成为正式功能 加密数据库备份是一种增强数据安全性的重要措施，既可以保护数据备份中敏感信息，又有助于合规，确保数据在存储和传输中的安全。
+ ## Feature details ### Scalability From 2ada23079dbe70d0b4c291183ebb6deeb625933c Mon Sep 17 00:00:00 2001 From: Aolin Date: Mon, 23 Sep 2024 17:34:16 +0800 Subject: [PATCH 10/45] add Instance-level execution plan cache Signed-off-by: Aolin --- releases/release-8.4.0.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/releases/release-8.4.0.md b/releases/release-8.4.0.md index 9260b5823962..2e3b376e990d 100644 --- a/releases/release-8.4.0.md +++ b/releases/release-8.4.0.md @@ -112,6 +112,17 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- In SaaS scenarios, you might need to batch-create a large number of users, rotate passwords periodically, and complete these tasks within a specific time window. Starting from v8.4.0, the performance of batch user creation and password rotation has been significantly improved. Additionally, you can further enhance performance by increasing concurrency through a higher number of session connections, which greatly reduces execution time for these operations. +* Instance-level execution plan cache (experimental) [#54057](https://github.com/pingcap/tidb/issues/54057) @[qw4990](https://github.com/qw4990) **tw@Oreoxmt** + + TiDB v8.4.0 introduces instance-level execution plan cache as an experimental feature. This feature allows all sessions within the same TiDB instance to share the execution plan cache, significantly reducing TiDB latency, improving cluster throughput, decreasing the likelihood of execution plan fluctuations, and maintaining stable cluster performance. Compared with session-level execution plan cache, instance-level execution plan cache offers the following advantages: + + - Eliminates redundancy, caching more execution plans with the same memory consumption. + - Allocates a fixed-size memory on the instance, limiting memory usage more effectively. + + In v8.4.0, instance-level execution plan cache only supports caching query execution plans and is disabled by default. You can enable this feature using [`tidb_enable_instance_plan_cache`](/system-variables.md#tidb_enable_instance_plan_cache-new-in-v840) and set its maximum memory usage using [`tidb_instance_plan_cache_max_size`](/system-variables.md#tidb_instance_plan_cache_max_size-new-in-v840). Before enabling this feature, disable [Prepared execution plan cache](/sql-prepared-plan-cache.md) and [Non-prepared execution plan cache](/sql-non-prepared-plan-cache.md). + + For more information, see [documentation](/system-variables.md#tidb_enable_instance_plan_cache-new-in-v840). + * Partitioned tables support global indexes (GA) [#45133](https://github.com/pingcap/tidb/issues/45133) @[mjonss](https://github.com/mjonss) @[Defined2014](https://github.com/Defined2014) @[jiyfhust](https://github.com/jiyfhust) @[L-maple](https://github.com/L-maple) In previous versions of partitioned tables, some limitations exist because global indexes are not supported. For example, the unique key must use every column in the table's partitioning expression. If the query condition does not use the partitioning key, the query will scan all partitions, resulting in poor performance. Starting from v7.6.0, the system variable [`tidb_enable_global_index`](/system-variables.md#tidb_enable_global_index-new-in-v760) is introduced to enable the global index feature. But this feature was under development at that time and it is not recommended to enable it. From 31cb7fa706ac46aa2140ee55993c38357a8c4846 Mon Sep 17 00:00:00 2001 From: Ben Meadowcroft Date: Mon, 23 Sep 2024 16:35:36 -0700 Subject: [PATCH 11/45] Update release-8.4.0.md Added additional TiCDC items. Removal of TiDB Binlog --- releases/release-8.4.0.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/releases/release-8.4.0.md b/releases/release-8.4.0.md index 2e3b376e990d..0dc3f0f77f5d 100644 --- a/releases/release-8.4.0.md +++ b/releases/release-8.4.0.md @@ -246,6 +246,18 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- For more information, see [documentation](doc-link). +* TiCDC claim check nows supports raw value format [#11396](https://github.com/pingcap/tiflow/issues/11396) @[3AceShowHand](https://github.com/3AceShowHand) + + When TiCDC used the claim check capability to handle large messages it included both the Key and the Value are encoded and stored in the external storage system. With the new raw value option, TiCDC can now be configured to store the value format only in the external storage system, using the protocol encoding. + + For more information, see [documentation](ticdc-sink-to-kafka.md#send-large-messages-to-external-storage). + +* TiCDC introduces new row checksum to verify old values after Add and Drop Column operations [#10969](https://github.com/pingcap/tiflow/issues/10969) @[3AceShowHand](https://github.com/3AceShowHand) + + Starting from v8.4.0, TiDB and TiCDC introduce Checksum V2 to address issues with Checksum V1 in verifying old values in Update or Delete events after Add Column or Drop Column operations. For new clusters created in v8.4.0 or later, or clusters upgraded to v8.4.0, TiDB uses Checksum V2 by default when single-row data checksum verification is enabled. TiCDC supports handling both Checksum V1 and V2. This change only affects TiDB and TiCDC internal implementation and does not impact checksum calculation methods for downstream Kafka consumers. + + For more information, see [documentation](/ticdc-integrity-check.md). + ## Compatibility changes > **Note:** @@ -296,6 +308,10 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- * Starting from v8.0.0, TiDB Lightning deprecates the [old version of conflict detection](/tidb-lightning/tidb-lightning-physical-import-mode-usage.md#the-old-version-of-conflict-detection-deprecated-in-v800) strategy for the physical import mode, and enables you to control the conflict detection strategy for both logical and physical import modes via the [`conflict.strategy`](/tidb-lightning/tidb-lightning-configuration.md#tidb-lightning-task) parameter. The [`duplicate-resolution`](/tidb-lightning/tidb-lightning-configuration.md) parameter for the old version of conflict detection will be removed in a future release. +* The following features are removed in this version: + + * TiDB Binlog replication is now removed from this version. Starting from v8.3.0, TiDB Binlog was fully deprecated. For incremental data replication, use [TiCDC](/ticdc-overview.md) instead. For point-in-time recovery (PITR), use [PITR](/br-pitr-guide.md). + ## Improvements + TiDB From f0bbf31e40bac3db11d51baa13b4dc1c582f64df Mon Sep 17 00:00:00 2001 From: Aolin Date: Tue, 24 Sep 2024 15:02:00 +0800 Subject: [PATCH 12/45] update TiCDC and removal of TiDB Binlog Signed-off-by: Aolin --- releases/release-8.4.0.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/releases/release-8.4.0.md b/releases/release-8.4.0.md index 0dc3f0f77f5d..2122a9314f37 100644 --- a/releases/release-8.4.0.md +++ b/releases/release-8.4.0.md @@ -246,13 +246,13 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- For more information, see [documentation](doc-link). -* TiCDC claim check nows supports raw value format [#11396](https://github.com/pingcap/tiflow/issues/11396) @[3AceShowHand](https://github.com/3AceShowHand) +* TiCDC claim check nows supports raw value format [#11396](https://github.com/pingcap/tiflow/issues/11396) @[3AceShowHand](https://github.com/3AceShowHand) **tw@Oreoxmt** When TiCDC used the claim check capability to handle large messages it included both the Key and the Value are encoded and stored in the external storage system. With the new raw value option, TiCDC can now be configured to store the value format only in the external storage system, using the protocol encoding. For more information, see [documentation](ticdc-sink-to-kafka.md#send-large-messages-to-external-storage). -* TiCDC introduces new row checksum to verify old values after Add and Drop Column operations [#10969](https://github.com/pingcap/tiflow/issues/10969) @[3AceShowHand](https://github.com/3AceShowHand) +* TiCDC introduces new row checksum to verify old values after Add and Drop Column operations [#10969](https://github.com/pingcap/tiflow/issues/10969) @[3AceShowHand](https://github.com/3AceShowHand) **tw@Oreoxmt** Starting from v8.4.0, TiDB and TiCDC introduce Checksum V2 to address issues with Checksum V1 in verifying old values in Update or Delete events after Add Column or Drop Column operations. For new clusters created in v8.4.0 or later, or clusters upgraded to v8.4.0, TiDB uses Checksum V2 by default when single-row data checksum verification is enabled. TiCDC supports handling both Checksum V1 and V2. This change only affects TiDB and TiCDC internal implementation and does not impact checksum calculation methods for downstream Kafka consumers. @@ -292,6 +292,10 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- ## Deprecated features +* The following features are removed starting from v8.4.0: + + * TiDB Binlog replication is now removed from this version. Starting from v8.3.0, TiDB Binlog was fully deprecated. For incremental data replication, use [TiCDC](/ticdc-overview.md) instead. For point-in-time recovery (PITR), use [PITR](/br-pitr-guide.md). **tw@lilin90** + * The following features are deprecated starting from v8.4.0: * Deprecated feature @@ -308,10 +312,6 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- * Starting from v8.0.0, TiDB Lightning deprecates the [old version of conflict detection](/tidb-lightning/tidb-lightning-physical-import-mode-usage.md#the-old-version-of-conflict-detection-deprecated-in-v800) strategy for the physical import mode, and enables you to control the conflict detection strategy for both logical and physical import modes via the [`conflict.strategy`](/tidb-lightning/tidb-lightning-configuration.md#tidb-lightning-task) parameter. The [`duplicate-resolution`](/tidb-lightning/tidb-lightning-configuration.md) parameter for the old version of conflict detection will be removed in a future release. -* The following features are removed in this version: - - * TiDB Binlog replication is now removed from this version. Starting from v8.3.0, TiDB Binlog was fully deprecated. For incremental data replication, use [TiCDC](/ticdc-overview.md) instead. For point-in-time recovery (PITR), use [PITR](/br-pitr-guide.md). - ## Improvements + TiDB From 6707391ea8c9ba1f24318723437a52d39d12c851 Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Wed, 25 Sep 2024 09:26:51 +0800 Subject: [PATCH 13/45] Apply suggestions from code review Co-authored-by: Mattias Jonsson --- releases/release-8.4.0.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/releases/release-8.4.0.md b/releases/release-8.4.0.md index 2122a9314f37..bc1a0bde4ac5 100644 --- a/releases/release-8.4.0.md +++ b/releases/release-8.4.0.md @@ -123,13 +123,13 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- For more information, see [documentation](/system-variables.md#tidb_enable_instance_plan_cache-new-in-v840). -* Partitioned tables support global indexes (GA) [#45133](https://github.com/pingcap/tidb/issues/45133) @[mjonss](https://github.com/mjonss) @[Defined2014](https://github.com/Defined2014) @[jiyfhust](https://github.com/jiyfhust) @[L-maple](https://github.com/L-maple) +* Partitioned tables support global indexes (GA) [#45133](https://github.com/pingcap/tidb/issues/45133) @[mjonss](https://github.com/mjonss) @[Defined2014](https://github.com/Defined2014) @[jiyfhust](https://github.com/jiyfhust) @[L-maple](https://github.com/L-maple) **tw@hfxsd** In previous versions of partitioned tables, some limitations exist because global indexes are not supported. For example, the unique key must use every column in the table's partitioning expression. If the query condition does not use the partitioning key, the query will scan all partitions, resulting in poor performance. Starting from v7.6.0, the system variable [`tidb_enable_global_index`](/system-variables.md#tidb_enable_global_index-new-in-v760) is introduced to enable the global index feature. But this feature was under development at that time and it is not recommended to enable it. Starting with v8.3.0, the global index feature is released as an experimental feature. You can explicitly create a global index for a partitioned table with the keyword `Global` to remove the restriction that the unique key must use every column in the table's partitioning expression, to meet flexible business needs. Global indexes also enhance the performance of queries that do not include partition keys. - In v8.4.0, this feature becomes generally available (GA). You can directly use the keyword `GLOBAL` to create a global index, instead of setting the system variable [`tidb_enable_global_index`](/system-variables.md#tidb_enable_global_index-new-in-v760) to enable the global index feature. + In v8.4.0, this feature becomes generally available (GA). You must use the keyword `GLOBAL` to create a global index, instead of setting the system variable [`tidb_enable_global_index`](/system-variables.md#tidb_enable_global_index-new-in-v760) to enable the global index feature. From v8.4.0 this system variable is deprecated and is always `ON`. For more information, see [documentation](/partitioned-table.md#global-indexes). @@ -272,9 +272,9 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- | Variable name | Change type | Description | |--------|------------------------------|------| -| | | | -| | | | -| | | | +| tidb_enable_table_partition | Deprecated | Always set to `ON`. Table partitioning has been GA since v5.1 and this experimental flag, if set to `OFF`, would cause create table with partitioning, just parse and ignore the partitioning clause. | +| tidb_enable_list_partition | Deprecated | Always set to ON. List partitioning has been GA since v6.1 | +| tidb_enable_global_index | Deprecated | Always set to `ON`. Global Index for partitioned tables is GA since v8.4.0, and needs explicit `GLOBAL` IndexOption to be used, so this system variable is no longer needed. | | | | | ### Configuration file parameters From d2469910d479e5343855737cad8524affb22343e Mon Sep 17 00:00:00 2001 From: lilin90 Date: Wed, 25 Sep 2024 11:09:17 +0800 Subject: [PATCH 14/45] Update feature highlights --- releases/release-8.4.0.md | 14 +++++--------- 1 file changed, 5 insertions(+), 9 deletions(-) diff --git a/releases/release-8.4.0.md b/releases/release-8.4.0.md index bc1a0bde4ac5..4be223491ceb 100644 --- a/releases/release-8.4.0.md +++ b/releases/release-8.4.0.md @@ -45,8 +45,8 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- Reliability and Availability - Support more triggers for runaway queries and provide the ability to switch resource groups - Runaway Queries offer an effective way to mitigate the impact of unexpected SQL performance issues on systems. The new version introduces PROCESSED_KEYS and RU as identifying conditions, allowing identified queries to be placed into a specified resource group for more precise identification and control of runaway queries. + Support more triggers for runaway queries, and support switching resource groups + Runaway Queries offer an effective way to mitigate the impact of unexpected SQL performance issues on systems. TiDB v8.4.0 introduces the number of keys processed by the Coprocessor (PROCESSED_KEYS) and request units (RU) as identifying conditions, and puts identified queries into the specified resource group for more precise identification and control of runaway queries. Support setting resource usage caps for background tasks for resource control @@ -62,19 +62,15 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- SQL - 外键成为正式功能 - 支持 MySQL 兼容的外键约束，维护数据一致性，进一步提升了 TiDB 对 MySQL 的兼容能力。 + Foreign keys (GA) + Support MySQL-compatible foreign key constraints to maintain data consistency and further enhance TiDB's compatibility with MySQL. 向量搜索功能（实验特性） 加速向量搜索的性能，适用于检索增强生成（RAG）、语义搜索、推荐系统等应用类型。把 TiDB 应用场景扩展到 AI 和 大语言模型（LLM）领域。 - 数据库管理和可观测性 - 持久化内存表到 Workload Repository（实验特性） - 持久化内存表中的运行指标和状态信息，是观测性的重要增强，能极大提升过往问题诊断和追溯的效率，并为未来的自动化运维，提供了数据集支持。 围绕 Workload Repository 构建报告、诊断、推荐一体化的能力，会成为未来提升 TiDB 易用性的重要组成。 - - + DB Operations and Observability Display TiKV and TiDB CPU times in memory tables CPU times are now integrated into a system table and displayed alongside other session or SQL metrics, allowing for easier observation of operations with high CPU consumption from multiple perspectives, improving diagnostic efficiency. This is particularly useful for diagnosing instances with CPU spikes or read/write hotspots in the cluster. From e5331eca3d78a1df5e26a070f4676d93dbf285d4 Mon Sep 17 00:00:00 2001 From: lilin90 Date: Wed, 25 Sep 2024 11:48:29 +0800 Subject: [PATCH 15/45] Update release-8.4.0.md --- releases/release-8.4.0.md | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/releases/release-8.4.0.md b/releases/release-8.4.0.md index 4be223491ceb..97083756839e 100644 --- a/releases/release-8.4.0.md +++ b/releases/release-8.4.0.md @@ -119,6 +119,12 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- For more information, see [documentation](/system-variables.md#tidb_enable_instance_plan_cache-new-in-v840). +* TiDB Lightning 的逻辑导入支持 prepare 接口 [#54850](https://github.com/pingcap/tidb/issues/54850) @[dbsid](https://github.com/dbsid) @[qw4990](https://github.com/qw4990) **tw@lilin90** + + 通过开启配置项 `logical-import-prep-stmt`，TiDB Lightning 逻辑导入产生的 SQL 语句将通过 prepare 接口执行，可提升 SQL 执行效率，并有更大机会命中执行计划缓存，提升逻辑导入的速度。 + + 更多信息，请参考[用户文档](tidb-lightning/tidb-lightning-configuration.md)。 + * Partitioned tables support global indexes (GA) [#45133](https://github.com/pingcap/tidb/issues/45133) @[mjonss](https://github.com/mjonss) @[Defined2014](https://github.com/Defined2014) @[jiyfhust](https://github.com/jiyfhust) @[L-maple](https://github.com/L-maple) **tw@hfxsd** In previous versions of partitioned tables, some limitations exist because global indexes are not supported. For example, the unique key must use every column in the table's partitioning expression. If the query condition does not use the partitioning key, the query will scan all partitions, resulting in poor performance. Starting from v7.6.0, the system variable [`tidb_enable_global_index`](/system-variables.md#tidb_enable_global_index-new-in-v760) is introduced to enable the global index feature. But this feature was under development at that time and it is not recommended to enable it. @@ -141,6 +147,16 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- For more information, see [documentation](doc-link). +* 超出预期的查询 (Runaway Queries) 新增处理行数 和 RU 作为阈值 [#54434](https://github.com/pingcap/tidb/issues/54434) @[HuSharp](https://github.com/HuSharp) **tw@lilin90** + + 从 v8.4.0 开始， TiDB 可以依据处理行数 (`PROCESSED_KEYS`) 和 Request Unit (`RU`) 定义超出预期的查询。和执行时间 (`EXEC_ELAPSED`) 相比，新增阈值能够更准确地定义查询的资源消耗，避免整体性能下降时发生识别偏差。 + + 支持同时设置多个条件，满足任意条件即识别为 `Runaway Queries`。 + + 可以观测 [Statement Summary Tables](/statement-summary-tables.md) 中的几个对应字段 (`RESOURCE_GROUP`、`MAX_REQUEST_UNIT_WRITE`、`MAX_REQUEST_UNIT_READ`、`MAX_PROCESSED_KEYS`)，根据历史执行情况决定条件值的大小。 + + 更多信息，请参考[用户文档](/tidb-resource-control.md#管理资源消耗超出预期的查询-runaway-queries)。 + * Support runaway queries to switch resource groups [#54434](https://github.com/pingcap/tidb/issues/54434) @[JmPotato](https://github.com/JmPotato) **tw@hfxsd** In TiDB v8.4.0, you can redirect runaway queries to a specific resource group. If the `COOLDOWN` mechanism fails to lower resource consumption, you can create a [resource group](/tidb-resource-control.md#create-a-resource-group) and set the `SWITCH_GROUP` parameter to move identified runaway queries to this group. Meanwhile, subsequent queries within the same session will continue to execute in the original resource group. By switching resource groups, you can more precisely manage resource usage and better control the impact of runaway queries. @@ -161,6 +177,13 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- For more information, see [documentation](/tidb-resource-control.md#manage-background-tasks). +* 优化资源组资源分配策略 [#50831](https://github.com/pingcap/tidb/issues/50831) @[nolouch](https://github.com/nolouch) **tw@lilin90** + + TiDB 在 v8.4.0 部分调整了资源分配策略，更好地满足用户对资源管控的预期。 + + - 控制大查询在运行时的资源分配，避免超出资源组限额。配合 Runaway Queries 的 `COOLDOWN` 动作，识别并降低大查询并发度，降低瞬时资源消耗。 + - 调整默认的优先级调度策略。当不同优先级的任务同时运行时，高优先级的任务获得更多资源。 + ### Availability * TiProxy supports traffic replay (experimental) [#642](https://github.com/pingcap/tiproxy/issues/642) @[djshow832](https://github.com/djshow832) **tw@Oreoxmt** @@ -186,6 +209,22 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- For more information, see [documentation](doc-link). +* TiDB 外键约束检查功能成为正式功能 (GA) [#55861](https://github.com/pingcap/tidb/issues/55861) @[GitHub ID-TBD]() **tw@lilin90** + + 从 v6.6.0 开始，TiDB 支持通过系统变量 [`foreign_key_checks`](/system-variables.md#foreign_key_checks) 做外键约束检查，但一直为实验特性。v8.4.0 对外键特性在更多场景做了覆盖测试，提升了稳定性和性能，因此从 v8.4.0 开始，外键功能成为正式功能 (GA)。 + + 更多信息，请参考[用户文档](/foreign-key.md)。 + +* 支持字符集 `gb18030` 和排序规则 `gb18030_bin` 和 `gb18030_chinese_ci` [#17470](https://github.com/tikv/tikv/issues/17470) [#55791](https://github.com/pingcap/tidb/issues/55791) @[cbcwestwolf](https://github.com/cbcwestwolf) **tw@lilin90** + + 从 v8.4.0 开始，TiDB 支持 `gb18030` 字符集，以确保 TiDB 能够更好地处理中文相关的数据存储和查询需求。该字符集是一个广泛用于中文字符编码的标准。 + + 从 v8.4.0 开始，TiDB 支持 `gb18030_bin` 和 `gb18030_chinese_ci` 排序规则。`gb18030_bin` 提供了基于二进制的精准排序，而 `gb18030_chinese_ci` 则支持大小写不敏感的通用排序规则。这两种排序规则使得对 `gb18030` 编码文本的排序和比较更加灵活高效。 + + 通过支持 `gb18030` 字符集及其排序规则，TiDB v8.4.0 增强了与中文应用场景的兼容性，特别是在涉及多种语言和字符编码的场景下，可以更方便地进行字符集的选择和操作，提升了数据库的使用体验。 + + 更多信息，请参考[用户文档](/character-set-gb18030.md)。 + ### DB operations * PITR adds client-side log backup data encryption support (experimental) [55834](https://github.com/pingcap/tidb/issues/55834) @[Tristan1900](https://github.com/Tristan1900) **tw@qiancai** From 55fcebc746e6f955c309bd8467f46b5fe4f59dba Mon Sep 17 00:00:00 2001 From: lilin90 Date: Wed, 25 Sep 2024 14:33:07 +0800 Subject: [PATCH 16/45] Update release-8.4.0.md --- releases/release-8.4.0.md | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/releases/release-8.4.0.md b/releases/release-8.4.0.md index 97083756839e..f3d3472fc0eb 100644 --- a/releases/release-8.4.0.md +++ b/releases/release-8.4.0.md @@ -209,7 +209,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- For more information, see [documentation](doc-link). -* TiDB 外键约束检查功能成为正式功能 (GA) [#55861](https://github.com/pingcap/tidb/issues/55861) @[GitHub ID-TBD]() **tw@lilin90** +* TiDB 外键约束检查功能成为正式功能 (GA) [#55861](https://github.com/pingcap/tidb/issues/55861) @[YangKeao](https://github.com/YangKeao) **tw@lilin90** 从 v6.6.0 开始，TiDB 支持通过系统变量 [`foreign_key_checks`](/system-variables.md#foreign_key_checks) 做外键约束检查，但一直为实验特性。v8.4.0 对外键特性在更多场景做了覆盖测试，提升了稳定性和性能，因此从 v8.4.0 开始，外键功能成为正式功能 (GA)。 @@ -259,6 +259,12 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- For more information, see [documentation](/information-schema/information-schema-processlist.md) and [documentation](information-schema/information-schema-slow-query.md). +* TOP SQL 支持按表或数据库维度查看聚合结果 [#55540](https://github.com/pingcap/tidb/issues/55540) @[nolouch](https://github.com/nolouch) **tw@lilin90** + + 在 v8.4.0 之前，[TOP SQL](/dashboard/top-sql.md) 以 SQL 为单位来聚合 CPU 时间。如果 CPU 时间不是由少数几个 SQL 贡献，按 SQL 聚合并不能有效发现问题。从 v8.4.0 开始，你可以选择 **By TABLE** 或者 **By DB** 聚合 CPU 时间。在多系统融合的场景下，新的聚合方式能够更有效地识别来自某个特定系统的负载变化，提升问题诊断的效率。 + + 更多信息，请参考[用户文档](/dashboard/top-sql.md)。 + ### Security * BR supports AWS IMDSv2 [#16443](https://github.com/tikv/tikv/issues/16443) @[pingyu](https://github.com/pingyu) **tw@hfxsd** @@ -339,7 +345,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- * TiDB introduces the system variable [`tidb_enable_auto_analyze_priority_queue`](/system-variables.md#tidb_enable_auto_analyze_priority_queue-new-in-v800), which controls whether priority queues are enabled to optimize the ordering of tasks that automatically collect statistics. In future releases, the priority queue will be the only way to order tasks for automatically collecting statistics, so this system variable will be deprecated. * TiDB introduces the system variable [`tidb_enable_async_merge_global_stats`](/system-variables.md#tidb_enable_async_merge_global_stats-new-in-v750) in v7.5.0. You can use it to set TiDB to use asynchronous merging of partition statistics to avoid OOM issues. In future releases, partition statistics will be merged asynchronously, so this system variable will be deprecated. - * It is planned to redesign [the automatic evolution of execution plan bindings](/sql-plan-management.md#baseline-evolution) in subsequent releases, and the related variables and behavior will change. + * It is planned to redesign [the automatic evolution of execution plan bindings](/sql-plan-management.md#baseline-evolution) in subsequent releases, and the related variables and behavior will change. * In v8.0.0, TiDB introduces the [`tidb_enable_parallel_hashagg_spill`](/system-variables.md#tidb_enable_parallel_hashagg_spill-new-in-v800) system variable to control whether TiDB supports disk spill for the concurrent HashAgg algorithm. In future versions, the [`tidb_enable_parallel_hashagg_spill`](/system-variables.md#tidb_enable_parallel_hashagg_spill-new-in-v800) system variable will be deprecated. * The TiDB Lightning parameter [`conflict.max-record-rows`](/tidb-lightning/tidb-lightning-configuration.md#tidb-lightning-task) is planned for deprecation in a future release and will be subsequently removed. This parameter will be replaced by [`conflict.threshold`](/tidb-lightning/tidb-lightning-configuration.md#tidb-lightning-task), which means that the maximum number of conflicting records is consistent with the maximum number of conflicting records that can be tolerated in a single import task. @@ -351,11 +357,14 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- + TiDB -- Optimize MEMDB implementation to reduce write latency in transactions and TiDB CPU usage [#55287](https://github.com/pingcap/tidb/issues/55287) @[you06](https://github.com/you06) **tw@hfxsd** + - Optimize MEMDB implementation to reduce write latency in transactions and TiDB CPU usage [#55287](https://github.com/pingcap/tidb/issues/55287) @[you06](https://github.com/you06) **tw@hfxsd** + - 为日志表 [`mysql.tidb_runaway_queries`](/mysql-schema/mysql-schema.md#runaway-queries-相关系统表) 增加写入控制，降低大量并发写入引发的开销 [#54434](https://github.com/pingcap/tidb/issues/54434) @[HuSharp](https://github.com/HuSharp) **tw@lilin90** + TiKV -- Increase the default value of Region from 96 MiB to 256 MiB to avoid the extra overhead caused by too many Regions [#17309](https://github.com/tikv/tikv/issues/17309) [LykxSassinator](https://github.com/LykxSassinator) **tw@hfxsd** -- Introduce a new `spill-dir` configuration in Raft Engine to support multi-disk storage for Raft logs. When the disk containing the home directory (`dir`) runs out of space, Raft Engine automatically writes new logs to `spill-dir`, ensuring continuous operation. [LykxSassinator](https://github.com/LykxSassinator) **tw@hfxsd** + + - Increase the default value of Region from 96 MiB to 256 MiB to avoid the extra overhead caused by too many Regions [#17309](https://github.com/tikv/tikv/issues/17309) [LykxSassinator](https://github.com/LykxSassinator) **tw@hfxsd** + - Introduce a new `spill-dir` configuration in Raft Engine to support multi-disk storage for Raft logs. When the disk containing the home directory (`dir`) runs out of space, Raft Engine automatically writes new logs to `spill-dir`, ensuring continuous operation. [LykxSassinator](https://github.com/LykxSassinator) **tw@hfxsd** + + PD + TiFlash From c9afc5f099813eabb3828972e75d4bd9cfef3358 Mon Sep 17 00:00:00 2001 From: Aolin Date: Wed, 25 Sep 2024 15:28:14 +0800 Subject: [PATCH 17/45] update BR reduces storage permission for restores, TiCDC Claim-Check, and TiCDC Checksum V2 Signed-off-by: Aolin --- releases/release-8.4.0.md | 28 +++++++++------------------- 1 file changed, 9 insertions(+), 19 deletions(-) diff --git a/releases/release-8.4.0.md b/releases/release-8.4.0.md index f3d3472fc0eb..6d74bb0f8b85 100644 --- a/releases/release-8.4.0.md +++ b/releases/release-8.4.0.md @@ -233,17 +233,11 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- For more information, see [documentation](doc-link). -* BR reduces requires storage permissions for restores [#55870](https://github.com/pingcap/tidb/issues/55870) @[Leavrth](https://github.com/Leavrth) **tw@Oreoxmt** +* BR reduces privileges when restoring backup data in a cloud storage system [#55870](https://github.com/pingcap/tidb/issues/55870) @[Leavrth](https://github.com/Leavrth) **tw@Oreoxmt** - Previously, when BR was restoring data, checkpoint information about the progress of the restore was recorded in the location hosting the backup data. These restore checkpoints enabled restoration to be quickly resumed if it was interrupted. With this feature, the restore checkpoints are now stored in the target TiDB cluster. This means that BR only requires read access to the backup dataset location for restores. + Before v8.4.0, BR stores checkpoint information about restore progress in the backup data location during restore. These checkpoints enable quick resumption of interrupted restores. Starting from v8.4.0, BR stores restore checkpoint information in the target TiDB cluster. This means that BR only requires read access to the backup directories. - For more information, see [documentation](doc-link). - -* Feature summary [#issue-number](issue-link) @[pr-auorthor-id](author-link) - - Feature descriptions (including what the feature is, why it is valuable for users, and how to use this feature generally) - - For more information, see [documentation](doc-link). + For more information, see [documentation](/br/backup-and-restore-storages.md#authentication). ### Observability @@ -281,23 +275,19 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- ### Data migration -* Feature summary [#issue-number](issue-link) @[pr-auorthor-id](author-link) - - Feature descriptions (including what the feature is, why it is valuable for users, and how to use this feature generally) - - For more information, see [documentation](doc-link). +* TiCDC Claim-Check supports sending only the `value` field of Kafka messages to external storage [#11396](https://github.com/pingcap/tiflow/issues/11396) @[3AceShowHand](https://github.com/3AceShowHand) **tw@Oreoxmt** -* TiCDC claim check nows supports raw value format [#11396](https://github.com/pingcap/tiflow/issues/11396) @[3AceShowHand](https://github.com/3AceShowHand) **tw@Oreoxmt** + Before v8.4.0, when using the Claim-Check feature to handle large messages (by setting `large-message-handle-option` to `claim-check`), TiCDC encodes and stores both the `key` and `value` fields in the external storage system. - When TiCDC used the claim check capability to handle large messages it included both the Key and the Value are encoded and stored in the external storage system. With the new raw value option, TiCDC can now be configured to store the value format only in the external storage system, using the protocol encoding. + Starting from v8.4.0, TiCDC supports sending only the `value` field of Kafka messages to external storage. This feature is only applicable to non-Open Protocol protocols. You can control this feature by setting the `claim-check-raw-value` parameter. - For more information, see [documentation](ticdc-sink-to-kafka.md#send-large-messages-to-external-storage). + For more information, see [documentation](/ticdc/ticdc-sink-to-kafka.md#send-the-value-field-to-external-storage-only). -* TiCDC introduces new row checksum to verify old values after Add and Drop Column operations [#10969](https://github.com/pingcap/tiflow/issues/10969) @[3AceShowHand](https://github.com/3AceShowHand) **tw@Oreoxmt** +* TiCDC introduces Checksum V2 to verify old values after Add Column or Drop Column operations [#10969](https://github.com/pingcap/tiflow/issues/10969) @[3AceShowHand](https://github.com/3AceShowHand) **tw@Oreoxmt** Starting from v8.4.0, TiDB and TiCDC introduce Checksum V2 to address issues with Checksum V1 in verifying old values in Update or Delete events after Add Column or Drop Column operations. For new clusters created in v8.4.0 or later, or clusters upgraded to v8.4.0, TiDB uses Checksum V2 by default when single-row data checksum verification is enabled. TiCDC supports handling both Checksum V1 and V2. This change only affects TiDB and TiCDC internal implementation and does not impact checksum calculation methods for downstream Kafka consumers. - For more information, see [documentation](/ticdc-integrity-check.md). + For more information, see [documentation](/ticdc/ticdc-integrity-check.md). ## Compatibility changes From 1f997f438adedda759cced395a6c6d158f9dfed2 Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Wed, 25 Sep 2024 15:38:04 +0800 Subject: [PATCH 18/45] Apply suggestions from code review --- releases/release-8.4.0.md | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/releases/release-8.4.0.md b/releases/release-8.4.0.md index 6d74bb0f8b85..359311d6b423 100644 --- a/releases/release-8.4.0.md +++ b/releases/release-8.4.0.md @@ -104,6 +104,25 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- For more information, see [documentation](doc-link). +* 增加获取 TSO 的 RPC 模式，降低获取 TSO 的延迟 [#54960](https://github.com/pingcap/tidb/issues/54960) @[MyonKeminta](https://github.com/MyonKeminta) **tw@qiancai** + + TiDB 在向 PD 请求 TSO 时，会将一段时间内的请求汇总起来并以同步的方式进行批处理，以减少 RPC (Remote Procedure Call) 请求数量从而降低 PD 负载。对于延迟敏感的场景，这种模式的性能并不理想。在 v8.4.0 中，TiDB 新增 TSO 请求的异步批处理模式，并提供不同的并发能力。异步模式可以降低获取 TSO 的延迟，但可能会增加 PD 的负载。你可以通过 [tidb_tso_client_rpc_mode](/system-variables.md#tidb_tso_client_rpc_mode-从-v840-版本开始引入) 变量设定获取 TSO 的 RPC 模式。 + + 更多信息，请参考[用户文档](/system-variables.md#tidb_tso_client_rpc_mode-从-v840-版本开始引入)。 + +* 优化 TiDB Hash Join 算子的执行效率（实验特性） [#55153](https://github.com/pingcap/tidb/issues/55153) [#53127](https://github.com/pingcap/tidb/issues/53127) @[windtalker](https://github.com/windtalker) @[xzhangxian1008](https://github.com/xzhangxian1008) @[XuHuaiyu](https://github.com/XuHuaiyu) @[wshwsh12](https://github.com/wshwsh12) **tw@qiancai** + + 在 v8.4.0 中，TiDB 对 Hash Join 算子的实现方法进行了优化，以提升其执行效率。目前，优化后的 Hash Join 实现方法为实验特性，仅对 Inner Join 和 Outer Join 类型的 Hash Join 生效，且默认关闭。你可以将变量 [tidb_hash_join_version](/system-variables.md#tidb_hash_join_version-从-v840-版本开始引入) 设置为 `optimized` 开启该优化实现方法。开启后，TiDB 在执行 Inner Join 和 Outer Join 类型的 Hash Join 时，将使用优化后的实现方法。 + + 更多信息，请参考[用户文档](/system-variables.md#tidb_hash_join_version-从-v840-版本开始引入)。 + +* 支持下推以下字符串函数到 TiKV [#17529](https://github.com/tikv/tikv/issues/17529) @[gengliqi](https://github.com/gengliqi) **tw@qiancai** + + * `DATE_ADD()` + * `DATE_SUB()` + + 更多信息，请参考[用户文档](/functions-and-operators/expressions-pushed-down.md)。 + * The performance of batch user creation and password changes has been improved by hundreds of times [#55604](https://github.com/pingcap/tidb/pull/55604) @[wjhuang2016](https://github.com/wjhuang2016) **tw@hfxsd** In SaaS scenarios, you might need to batch-create a large number of users, rotate passwords periodically, and complete these tasks within a specific time window. Starting from v8.4.0, the performance of batch user creation and password rotation has been significantly improved. Additionally, you can further enhance performance by increasing concurrency through a higher number of session connections, which greatly reduces execution time for these operations. @@ -209,6 +228,18 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- For more information, see [documentation](doc-link). +* 支持向量搜索功能（实验特性） [#54245](https://github.com/pingcap/tidb/issues/54245) [#9032](https://github.com/pingcap/tiflash/issues/9032) @[breezewish](https://github.com/breezewish) @[Lloyd-Pottiger](https://github.com/Lloyd-Pottiger) @[EricZequan](https://github.com/EricZequan) @[zimulala](https://github.com/zimulala) @[JaySon-Huang](https://github.com/JaySon-Huang) **tw@qiancai** + + 向量搜索是一种基于数据语义的搜索方法，可以提供更相关的搜索结果。作为 AI 和大语言模型（LLM）的核心功能之一，向量搜索可用于检索增强生成（Retrieval-Augmented Generation, RAG）、语义搜索、推荐系统等多种场景。 + + 从 v8.4.0 开始，TiDB 支持[向量数据类型](vector-search-data-types.md)和[向量搜索索引](vector-search-index.md)，具备强大的向量搜索能力。TiDB 的向量数据类型最多可支持 16383 维度，并支持多种[距离函数](/vector-search-functions-and-operators.md#向量函数)，包括 L2 距离（欧式距离）、余弦距离、负内积和 L1 距离（曼哈顿距离）。 + + 在使用时，你只需要创建包含向量数据类型的表，并插入向量数据，即可执行向量搜索查询，也可进行向量数据与传统关系数据的混合查询。此外，你可以创建并利用向量搜索索引来提升向量搜索的性能。 + + 需要注意的是，TiDB 的向量搜索索引依赖于 TiFlash。因此，在使用向量搜索索引之前，需要确保 TiDB 集群中已部署 TiFlash 节点。 + + 更多信息，请参考[用户文档](/vector-search-overview.md)。 + * TiDB 外键约束检查功能成为正式功能 (GA) [#55861](https://github.com/pingcap/tidb/issues/55861) @[YangKeao](https://github.com/YangKeao) **tw@lilin90** 从 v6.6.0 开始，TiDB 支持通过系统变量 [`foreign_key_checks`](/system-variables.md#foreign_key_checks) 做外键约束检查，但一直为实验特性。v8.4.0 对外键特性在更多场景做了覆盖测试，提升了稳定性和性能，因此从 v8.4.0 开始，外键功能成为正式功能 (GA)。 @@ -348,6 +379,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- + TiDB - Optimize MEMDB implementation to reduce write latency in transactions and TiDB CPU usage [#55287](https://github.com/pingcap/tidb/issues/55287) @[you06](https://github.com/you06) **tw@hfxsd** + - 优化系统变量 `tidb_dml_type` 为 `"bulk"` 时 DML 语句的执行性能 [#50215](https://github.com/pingcap/tidb/issues/50215) @[ekexium](https://github.com/ekexium) **tw@qiancai** - 为日志表 [`mysql.tidb_runaway_queries`](/mysql-schema/mysql-schema.md#runaway-queries-相关系统表) 增加写入控制，降低大量并发写入引发的开销 [#54434](https://github.com/pingcap/tidb/issues/54434) @[HuSharp](https://github.com/HuSharp) **tw@lilin90** + TiKV @@ -357,6 +389,8 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- + PD + - 在 TiDB Lightning 数据导入过程中，允许 PD 进行 `evict-leader` 调度，避免阻断 TiKV 节点的 offline 进程 [#issue号](链接) @[okJiang](https://github.com/okJiang) **tw@qiancai** + + TiFlash + Tools From 07b9178524628a088a730cdd2aa65a47a2704a9e Mon Sep 17 00:00:00 2001 From: Aolin Date: Wed, 25 Sep 2024 16:06:20 +0800 Subject: [PATCH 19/45] add link for highlights and update improvements Signed-off-by: Aolin --- releases/release-8.4.0.md | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/releases/release-8.4.0.md b/releases/release-8.4.0.md index 359311d6b423..a489a2694124 100644 --- a/releases/release-8.4.0.md +++ b/releases/release-8.4.0.md @@ -24,8 +24,8 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- Scalability and Performance - 执行计划缓存在实例内共享（实验特性） - 实例级执行计划缓存支持在内存中缓存更多的执行计划，消除 SQL 编译时所消耗的时间，从而减少 SQL 的运行时间，提升 OLTP 系统的性能和吞吐。同时，也能更好的控制内存占用，提升数据库稳定性。 + Global indexes for partitioned tables (GA) @@ -53,12 +53,12 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- By setting a percentage cap on background tasks of resource control, you can manage their resource consumption based on the needs of different business systems. This ensures background tasks consume minimal resources, maintaining the service quality of online operations. - TiProxy 流量捕捉和回放 - 在做集群升级、迁移、部署变化等重要变更之前，通过捕捉真实负载来验证目标集群的性能，确保变更的成功。 + Adaptive concurrency for statistics collection + Automatic statistics collection determines the collection concurrency based on node scale and hardware specifications. This improves statistics collection efficiency, reduces manual tuning, and ensures stable cluster performance. SQL @@ -380,11 +380,17 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- - Optimize MEMDB implementation to reduce write latency in transactions and TiDB CPU usage [#55287](https://github.com/pingcap/tidb/issues/55287) @[you06](https://github.com/you06) **tw@hfxsd** - 优化系统变量 `tidb_dml_type` 为 `"bulk"` 时 DML 语句的执行性能 [#50215](https://github.com/pingcap/tidb/issues/50215) @[ekexium](https://github.com/ekexium) **tw@qiancai** + - Support using [Optimizer Fix Control 47400](/optimizer-fix-controls.md#47400-new-in-v840) to control whether the optimizer limits the minimum value estimated for `estRows` to `1`, which is consistent with databases such as Oracle and DB2 [#47400](https://github.com/pingcap/tidb/issues/47400) @[terry1purcell](https://github.com/terry1purcell) **tw@Oreoxmt** - 为日志表 [`mysql.tidb_runaway_queries`](/mysql-schema/mysql-schema.md#runaway-queries-相关系统表) 增加写入控制，降低大量并发写入引发的开销 [#54434](https://github.com/pingcap/tidb/issues/54434) @[HuSharp](https://github.com/HuSharp) **tw@lilin90** + - Spport Index Join by default when the inner table has `Selection` or `Projection` operators on it [#issue号](链接) @[winoros](https://github.com/winoros) **tw@Oreoxmt** + - Reduce the number of column details fetched from TiKV for `DELETE` operations in certain scenarios, lowering the resource overhead of these operations [#38911](https://github.com/pingcap/tidb/issues/38911) @[winoros](https://github.com/winoros) **tw@Oreoxmt** + - Improve the efficiency of the priority queue for automatic statistics collection tasks [#49972](https://github.com/pingcap/tidb/issues/49972) @[Rustin170506](https://github.com/Rustin170506) **tw@Oreoxmt** + - Improve automatic statistics collection by determining the collection concurrency based on node scale and hardware specifications [#issue号](链接) @[hawkingrei](https://github.com/hawkingrei) **tw@Oreoxmt** + TiKV - Increase the default value of Region from 96 MiB to 256 MiB to avoid the extra overhead caused by too many Regions [#17309](https://github.com/tikv/tikv/issues/17309) [LykxSassinator](https://github.com/LykxSassinator) **tw@hfxsd** + - Support setting memory usage limits for in-memory pessimistic locks in a Region or TiKV instance. To prevent CPU/IO overhead caused by pessimistic locks spilling to disk during write hotspots, you can increase the memory limit by modifying the configuration items [#17542](https://github.com/tikv/tikv/issues/17542) @[cfzjywxk](https://github.com/cfzjywxk) **tw@Oreoxmt** - Introduce a new `spill-dir` configuration in Raft Engine to support multi-disk storage for Raft logs. When the disk containing the home directory (`dir`) runs out of space, Raft Engine automatically writes new logs to `spill-dir`, ensuring continuous operation. [LykxSassinator](https://github.com/LykxSassinator) **tw@hfxsd** + PD @@ -397,6 +403,8 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- + Backup & Restore (BR) + - Disable full data restoration to a non-empty cluster using the `RESTORE` SQL statement by default [#55087](https://github.com/pingcap/tidb/issues/55087) @[BornChanger](https://github.com/BornChanger) **tw@Oreoxmt** + + TiCDC + TiDB Data Migration (DM) From aeb29e48f5841ccdbe216f41ff23496e89278bd1 Mon Sep 17 00:00:00 2001 From: Aolin Date: Wed, 25 Sep 2024 16:19:38 +0800 Subject: [PATCH 20/45] fix HTML table Signed-off-by: Aolin --- releases/release-8.4.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-8.4.0.md b/releases/release-8.4.0.md index a489a2694124..6b5fb34c270e 100644 --- a/releases/release-8.4.0.md +++ b/releases/release-8.4.0.md @@ -24,7 +24,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- Scalability and Performance - Instance-level execution plan cache (experimental) Instance-level execution plan cache allows all sessions within the same TiDB instance to share the execution plan cache. It stores more execution plans in memory, eliminating SQL compilation time. This reduces SQL execution time, improves OLTP system performance and throughput, and provides better control over memory usage, enhancing database stability. From 6a313f99ea6e34566a889b0d6cb0bf19b69a9bb1 Mon Sep 17 00:00:00 2001 From: Aolin Date: Wed, 25 Sep 2024 16:28:13 +0800 Subject: [PATCH 21/45] fix HTML table Signed-off-by: Aolin --- releases/release-8.4.0.md | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/releases/release-8.4.0.md b/releases/release-8.4.0.md index 6b5fb34c270e..5d011ee1410b 100644 --- a/releases/release-8.4.0.md +++ b/releases/release-8.4.0.md @@ -24,64 +24,64 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- Scalability and Performance - Instance-level execution plan cache (experimental) + Instance-level execution plan cache (experimental)**tw@Oreoxmt 1569** Instance-level execution plan cache allows all sessions within the same TiDB instance to share the execution plan cache. It stores more execution plans in memory, eliminating SQL compilation time. This reduces SQL execution time, improves OLTP system performance and throughput, and provides better control over memory usage, enhancing database stability. - Global indexes for partitioned tables (GA) + Global indexes for partitioned tables (GA)**tw@hfxsd 1961** Global indexes can effectively improve the efficiency of retrieving non-partitioned columns, and remove the restriction that a unique key must contain the partition key. This feature extends the usage scenarios of TiDB partitioned tables and avoids some of the application modification work that might be required for data migration. - TiDB 并行获取 TSO + TiDB 并行获取 TSO**tw@qiancai 1893** 在高并发场景下，并行获取 TSO 能够有效降低等待获取 TSO 的时间，提升集群的吞吐。 - Improve the execution efficiency of administrative SQL statements + Improve the execution efficiency of administrative SQL statements**tw@hfxsd 1941** In some SaaS systems, there is a need to create users in batch and rotate passwords regularly. TiDB enhances the performance of creating and modifying database users, ensuring these operations can be completed within the desired time window. - Improve query performance for cached tables + Improve query performance for cached tables**tw@hfxsd 1965** Improve query performance for index scanning on cached tables, with improvements of up to 5.4 times in some scenarios. For high-speed queries on small tables, cached tables can significantly enhance overall performance. Reliability and Availability - Support more triggers for runaway queries, and support switching resource groups - Runaway Queries offer an effective way to mitigate the impact of unexpected SQL performance issues on systems. TiDB v8.4.0 introduces the number of keys processed by the Coprocessor (PROCESSED_KEYS) and request units (RU) as identifying conditions, and puts identified queries into the specified resource group for more precise identification and control of runaway queries. + Support more triggers for runaway queries, and support switching resource groups**tw@hfxsd 1832 tw@lilin90 1800** + Runaway Queries offer an effective way to mitigate the impact of unexpected SQL performance issues on systems. TiDB v8.4.0 introduces the number of keys processed by the Coprocessor (PROCESSED_KEYS) and request units (RU) as identifying conditions, and puts identified queries into the specified resource group for more precise identification and control of runaway queries. - Support setting resource usage caps for background tasks for resource control + Support setting resource usage caps for background tasks for resource control **tw@hfxsd 1909** By setting a percentage cap on background tasks of resource control, you can manage their resource consumption based on the needs of different business systems. This ensures background tasks consume minimal resources, maintaining the service quality of online operations. - TiProxy supports traffic capture and replay (experimental)**tw@Oreoxmt 1942** Use TiProxy to capture real workloads from TiDB production clusters before major operations like cluster upgrades, migrations, or deployment changes. Replay these workloads on target test clusters to validate performance and ensure successful changes. - Adaptive concurrency for statistics collection + Adaptive concurrency for statistics collection**tw@Oreoxmt 1739** Automatic statistics collection determines the collection concurrency based on node scale and hardware specifications. This improves statistics collection efficiency, reduces manual tuning, and ensures stable cluster performance. SQL - Foreign keys (GA) + Foreign keys (GA)**tw@lilin90 1894** Support MySQL-compatible foreign key constraints to maintain data consistency and further enhance TiDB's compatibility with MySQL. - 向量搜索功能（实验特性） + 向量搜索功能（实验特性）**tw@qiancai 1898** 加速向量搜索的性能，适用于检索增强生成（RAG）、语义搜索、推荐系统等应用类型。把 TiDB 应用场景扩展到 AI 和 大语言模型（LLM）领域。 DB Operations and Observability - Display TiKV and TiDB CPU times in memory tables + Display TiKV and TiDB CPU times in memory tables**tw@hfxsd 1877** CPU times are now integrated into a system table and displayed alongside other session or SQL metrics, allowing for easier observation of operations with high CPU consumption from multiple perspectives, improving diagnostic efficiency. This is particularly useful for diagnosing instances with CPU spikes or read/write hotspots in the cluster. - Support backing up TiKV instances with IMDSv2 service enabled + Support backing up TiKV instances with IMDSv2 service enabled**tw@hfxsd 1945** AWS EC2 now uses IMDSv2 as the default metadata service. TiDB supports data backups from TiKV instances with IMDSv2 enabled, enhancing your ability to run TiDB clusters in public cloud environments. 安全 - 备份数据加密成为正式功能 - 加密数据库备份是一种增强数据安全性的重要措施，既可以保护数据备份中敏感信息，又有助于合规，确保数据在存储和传输中的安全。 + 备份数据加密成为正式功能**tw@qiancai 1920** + 加密数据库备份是一种增强数据安全性的重要措施，既可以保护数据备份中敏感信息，又有助于合规，确保数据在存储和传输中的安全。 From 2bea17077188cf4991cb77f5fe3ffefa991d0969 Mon Sep 17 00:00:00 2001 From: lilin90 Date: Wed, 25 Sep 2024 16:38:20 +0800 Subject: [PATCH 22/45] Add more descriptions --- releases/release-8.4.0.md | 60 ++++++++++++++++++--------------------- 1 file changed, 27 insertions(+), 33 deletions(-) diff --git a/releases/release-8.4.0.md b/releases/release-8.4.0.md index 5d011ee1410b..58759e37c849 100644 --- a/releases/release-8.4.0.md +++ b/releases/release-8.4.0.md @@ -138,11 +138,11 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- For more information, see [documentation](/system-variables.md#tidb_enable_instance_plan_cache-new-in-v840). -* TiDB Lightning 的逻辑导入支持 prepare 接口 [#54850](https://github.com/pingcap/tidb/issues/54850) @[dbsid](https://github.com/dbsid) @[qw4990](https://github.com/qw4990) **tw@lilin90** +* TiDB Lightning's logical import mode supports prepared statements and client statement cache [#54850](https://github.com/pingcap/tidb/issues/54850) @[dbsid](https://github.com/dbsid) **tw@lilin90** - 通过开启配置项 `logical-import-prep-stmt`，TiDB Lightning 逻辑导入产生的 SQL 语句将通过 prepare 接口执行，可提升 SQL 执行效率，并有更大机会命中执行计划缓存，提升逻辑导入的速度。 + By enabling the `logical-import-prep-stmt` configuration item, the SQL statements generated by TiDB Lightning's logical import mode will use prepared statements and client statement cache. This reduces the cost of TiDB SQL parsing and compilation, improves SQL execution efficiency, and increases the likelihood of hitting the execution plan cache, thereby speeding up logical import. - 更多信息，请参考[用户文档](tidb-lightning/tidb-lightning-configuration.md)。 + For more information, see [documentation](/tidb-lightning/tidb-lightning-configuration.md). * Partitioned tables support global indexes (GA) [#45133](https://github.com/pingcap/tidb/issues/45133) @[mjonss](https://github.com/mjonss) @[Defined2014](https://github.com/Defined2014) @[jiyfhust](https://github.com/jiyfhust) @[L-maple](https://github.com/L-maple) **tw@hfxsd** @@ -166,15 +166,15 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- For more information, see [documentation](doc-link). -* 超出预期的查询 (Runaway Queries) 新增处理行数 和 RU 作为阈值 [#54434](https://github.com/pingcap/tidb/issues/54434) @[HuSharp](https://github.com/HuSharp) **tw@lilin90** +* Runaway queries support the number of processed keys and request units as thresholds [#54434](https://github.com/pingcap/tidb/issues/54434) @[HuSharp](https://github.com/HuSharp) **tw@lilin90** - 从 v8.4.0 开始， TiDB 可以依据处理行数 (`PROCESSED_KEYS`) 和 Request Unit (`RU`) 定义超出预期的查询。和执行时间 (`EXEC_ELAPSED`) 相比，新增阈值能够更准确地定义查询的资源消耗，避免整体性能下降时发生识别偏差。 + Starting from v8.4.0, TiDB can identify runaway queries based on the number of processed keys (`PROCESSED_KEYS`) and request units (`RU`). Compared with execution time (`EXEC_ELAPSED`), these new thresholds more accurately define the resource consumption of queries, avoiding identification bias when overall performance decreases. - 支持同时设置多个条件，满足任意条件即识别为 `Runaway Queries`。 + You can set multiple conditions simultaneously, and a query is identified as a runaway query if any condition is met. - 可以观测 [Statement Summary Tables](/statement-summary-tables.md) 中的几个对应字段 (`RESOURCE_GROUP`、`MAX_REQUEST_UNIT_WRITE`、`MAX_REQUEST_UNIT_READ`、`MAX_PROCESSED_KEYS`)，根据历史执行情况决定条件值的大小。 + You can observe the corresponding fields (`RESOURCE_GROUP`, `MAX_REQUEST_UNIT_WRITE`, `MAX_REQUEST_UNIT_READ`, `MAX_PROCESSED_KEYS`) in the [Statement Summary Tables](/statement-summary-tables.md) to determine the condition values based on historical execution. - 更多信息，请参考[用户文档](/tidb-resource-control.md#管理资源消耗超出预期的查询-runaway-queries)。 + For more information, see [documentation](/tidb-resource-control.md#manage-queries-that-consume-more-resources-than-expected-runaway-queries). * Support runaway queries to switch resource groups [#54434](https://github.com/pingcap/tidb/issues/54434) @[JmPotato](https://github.com/JmPotato) **tw@hfxsd** @@ -183,25 +183,25 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- For more information, see [documentation](/tidb-resource-control.md#query_limit-parameters). * The system variable `tidb_scatter_region` supports the cluster-level Region scattering strategy [#55184](https://github.com/pingcap/tidb/issues/55184) @[D3Hunter](https://github.com/D3Hunter) **tw@hfxsd** - + In previous versions, the system variable `tidb_scatter_region` can only be enabled or disabled. When enabled, it applies a table-level scattering strategy during batch table creation. However, when creating hundreds of thousands of tables in a batch, this approach results in a concentration of regions on a few TiKV nodes, causing out-of-memory (OOM) issues on those nodes. To address this, starting from v8.4.0, `tidb_scatter_region` is changed to a string type. It now supports a cluster-level scattering strategy, helping scatter regions more evenly and preventing OOM problems on TiKV nodes. - + For more information, see [documentation](/system-variables.md#tidb_scatter_region). * Support setting resource caps for background tasks of resource control [#56019](https://github.com/pingcap/tidb/issues/56019) @[glorv](https://github.com/glorv) **tw@hfxsd** - + TiDB resource control can identify and lower the priority of background tasks. In certain scenarios, you might want to limit the resource consumption of these tasks, even when resources are available. Starting from v8.4.0, you can use the `UTILIZATION_LIMIT` parameter to set a maximum percentage of resources that a background task can consume. Each node will ensure that the resource usage of all background tasks stays within this limit. This feature enables precise control over resource consumption for background tasks, enhancing cluster stability. For more information, see [documentation](/tidb-resource-control.md#manage-background-tasks). -* 优化资源组资源分配策略 [#50831](https://github.com/pingcap/tidb/issues/50831) @[nolouch](https://github.com/nolouch) **tw@lilin90** +* Optimize the resource allocation strategy of resource groups [#50831](https://github.com/pingcap/tidb/issues/50831) @[nolouch](https://github.com/nolouch) **tw@lilin90** - TiDB 在 v8.4.0 部分调整了资源分配策略，更好地满足用户对资源管控的预期。 + TiDB improves the resource allocation strategy in v8.4.0 to better meet user expectations for resource management. - - 控制大查询在运行时的资源分配，避免超出资源组限额。配合 Runaway Queries 的 `COOLDOWN` 动作，识别并降低大查询并发度，降低瞬时资源消耗。 - - 调整默认的优先级调度策略。当不同优先级的任务同时运行时，高优先级的任务获得更多资源。 + - Controlling the resource allocation of large queries at runtime to avoid exceeding the resource group limit, combined with runaway queries `COOLDOWN`. This can help identify and reduce the concurrency of large queries, and reduce instantaneous resource consumption. + - Adjusting the default priority scheduling strategy. When tasks of different priorities run simultaneously, high-priority tasks receive more resources. ### Availability @@ -222,12 +222,6 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- ### SQL -* Feature summary [#issue-number](issue-link) @[pr-auorthor-id](author-link) - - Feature descriptions (including what the feature is, why it is valuable for users, and how to use this feature generally) - - For more information, see [documentation](doc-link). - * 支持向量搜索功能（实验特性） [#54245](https://github.com/pingcap/tidb/issues/54245) [#9032](https://github.com/pingcap/tiflash/issues/9032) @[breezewish](https://github.com/breezewish) @[Lloyd-Pottiger](https://github.com/Lloyd-Pottiger) @[EricZequan](https://github.com/EricZequan) @[zimulala](https://github.com/zimulala) @[JaySon-Huang](https://github.com/JaySon-Huang) **tw@qiancai** 向量搜索是一种基于数据语义的搜索方法，可以提供更相关的搜索结果。作为 AI 和大语言模型（LLM）的核心功能之一，向量搜索可用于检索增强生成（Retrieval-Augmented Generation, RAG）、语义搜索、推荐系统等多种场景。 @@ -240,21 +234,21 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- 更多信息，请参考[用户文档](/vector-search-overview.md)。 -* TiDB 外键约束检查功能成为正式功能 (GA) [#55861](https://github.com/pingcap/tidb/issues/55861) @[YangKeao](https://github.com/YangKeao) **tw@lilin90** +* The TiDB foreign key feature becomes generally available (GA) [#55861](https://github.com/pingcap/tidb/issues/55861) @[YangKeao](https://github.com/YangKeao) **tw@lilin90** - 从 v6.6.0 开始，TiDB 支持通过系统变量 [`foreign_key_checks`](/system-variables.md#foreign_key_checks) 做外键约束检查，但一直为实验特性。v8.4.0 对外键特性在更多场景做了覆盖测试，提升了稳定性和性能，因此从 v8.4.0 开始，外键功能成为正式功能 (GA)。 + Starting from v6.6.0, TiDB supports foreign key constraints using the system variable [`foreign_key_checks`](/system-variables.md#foreign_key_checks), but it has been an experimental feature. In v8.4.0, the foreign key feature has been extensively tested in more scenarios to improve stability and performance. Therefore, starting from v8.4.0, the foreign key feature becomes generally available (GA). - 更多信息，请参考[用户文档](/foreign-key.md)。 + For more information, see [documentation](/foreign-key.md). -* 支持字符集 `gb18030` 和排序规则 `gb18030_bin` 和 `gb18030_chinese_ci` [#17470](https://github.com/tikv/tikv/issues/17470) [#55791](https://github.com/pingcap/tidb/issues/55791) @[cbcwestwolf](https://github.com/cbcwestwolf) **tw@lilin90** +* Support the `gb18030` character set and the `gb18030_bin` and `gb18030_chinese_ci` collations [#17470](https://github.com/tikv/tikv/issues/17470) [#55791](https://github.com/pingcap/tidb/issues/55791) @[cbcwestwolf](https://github.com/cbcwestwolf) **tw@lilin90** - 从 v8.4.0 开始，TiDB 支持 `gb18030` 字符集，以确保 TiDB 能够更好地处理中文相关的数据存储和查询需求。该字符集是一个广泛用于中文字符编码的标准。 + Starting from v8.4.0, TiDB supports the `gb18030` character set to ensure that TiDB can better handle Chinese-related data storage and query requirements. This character set is a standard widely used for Chinese character encoding. - 从 v8.4.0 开始，TiDB 支持 `gb18030_bin` 和 `gb18030_chinese_ci` 排序规则。`gb18030_bin` 提供了基于二进制的精准排序，而 `gb18030_chinese_ci` 则支持大小写不敏感的通用排序规则。这两种排序规则使得对 `gb18030` 编码文本的排序和比较更加灵活高效。 + Starting from v8.4.0, TiDB supports the `gb18030_bin` and `gb18030_chinese_ci` collations. `gb18030_bin` provides precise binary sorting, while `gb18030_chinese_ci` supports case-insensitive general sorting. These two collations make sorting and comparison of `gb18030` encoded text more flexible and efficient. - 通过支持 `gb18030` 字符集及其排序规则，TiDB v8.4.0 增强了与中文应用场景的兼容性，特别是在涉及多种语言和字符编码的场景下，可以更方便地进行字符集的选择和操作，提升了数据库的使用体验。 + By supporting the `gb18030` character set and its collations, TiDB v8.4.0 enhances compatibility with Chinese scenarios. In scenarios involving multiple languages and character encodings, you can select and operate on character sets with better user experience. - 更多信息，请参考[用户文档](/character-set-gb18030.md)。 + For more information, see [documentation](/character-set-gb18030.md). ### DB operations @@ -284,11 +278,11 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- For more information, see [documentation](/information-schema/information-schema-processlist.md) and [documentation](information-schema/information-schema-slow-query.md). -* TOP SQL 支持按表或数据库维度查看聚合结果 [#55540](https://github.com/pingcap/tidb/issues/55540) @[nolouch](https://github.com/nolouch) **tw@lilin90** +* Top SQL supports viewing aggregated results by table or database [#55540](https://github.com/pingcap/tidb/issues/55540) @[nolouch](https://github.com/nolouch) **tw@lilin90** - 在 v8.4.0 之前，[TOP SQL](/dashboard/top-sql.md) 以 SQL 为单位来聚合 CPU 时间。如果 CPU 时间不是由少数几个 SQL 贡献，按 SQL 聚合并不能有效发现问题。从 v8.4.0 开始，你可以选择 **By TABLE** 或者 **By DB** 聚合 CPU 时间。在多系统融合的场景下，新的聚合方式能够更有效地识别来自某个特定系统的负载变化，提升问题诊断的效率。 + Before v8.4.0, [Top SQL](/dashboard/top-sql.md) aggregates CPU time by SQL. If CPU time is not consumed by a few SQL statements, aggregation by SQL cannot effectively identify issues. Starting from v8.4.0, you can choose to aggregate CPU time **By TABLE** or **By DB**. In scenarios with multiple systems, the new aggregation method can more effectively identify load changes from a specific system, improving diagnostic efficiency. - 更多信息，请参考[用户文档](/dashboard/top-sql.md)。 + For more information, see [documentation](/dashboard/top-sql.md). ### Security @@ -381,7 +375,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- - Optimize MEMDB implementation to reduce write latency in transactions and TiDB CPU usage [#55287](https://github.com/pingcap/tidb/issues/55287) @[you06](https://github.com/you06) **tw@hfxsd** - 优化系统变量 `tidb_dml_type` 为 `"bulk"` 时 DML 语句的执行性能 [#50215](https://github.com/pingcap/tidb/issues/50215) @[ekexium](https://github.com/ekexium) **tw@qiancai** - Support using [Optimizer Fix Control 47400](/optimizer-fix-controls.md#47400-new-in-v840) to control whether the optimizer limits the minimum value estimated for `estRows` to `1`, which is consistent with databases such as Oracle and DB2 [#47400](https://github.com/pingcap/tidb/issues/47400) @[terry1purcell](https://github.com/terry1purcell) **tw@Oreoxmt** - - 为日志表 [`mysql.tidb_runaway_queries`](/mysql-schema/mysql-schema.md#runaway-queries-相关系统表) 增加写入控制，降低大量并发写入引发的开销 [#54434](https://github.com/pingcap/tidb/issues/54434) @[HuSharp](https://github.com/HuSharp) **tw@lilin90** + - Add write control to the [`mysql.tidb_runaway_queries`](/mysql-schema/mysql-schema.md#system-tables-related-to-runaway-queries) log table to reduce overhead caused by a large number of concurrent writes [#54434](https://github.com/pingcap/tidb/issues/54434) @[HuSharp](https://github.com/HuSharp) **tw@lilin90** - Spport Index Join by default when the inner table has `Selection` or `Projection` operators on it [#issue号](链接) @[winoros](https://github.com/winoros) **tw@Oreoxmt** - Reduce the number of column details fetched from TiKV for `DELETE` operations in certain scenarios, lowering the resource overhead of these operations [#38911](https://github.com/pingcap/tidb/issues/38911) @[winoros](https://github.com/winoros) **tw@Oreoxmt** - Improve the efficiency of the priority queue for automatic statistics collection tasks [#49972](https://github.com/pingcap/tidb/issues/49972) @[Rustin170506](https://github.com/Rustin170506) **tw@Oreoxmt** From c7bc219568e6536da2a4dd0d1969a591cf8a4fdb Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Thu, 26 Sep 2024 09:45:01 +0800 Subject: [PATCH 23/45] add translation of new features --- releases/release-8.4.0.md | 52 ++++++++++++++++++++------------------- 1 file changed, 27 insertions(+), 25 deletions(-) diff --git a/releases/release-8.4.0.md b/releases/release-8.4.0.md index 58759e37c849..72ad36037ffd 100644 --- a/releases/release-8.4.0.md +++ b/releases/release-8.4.0.md @@ -104,24 +104,26 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- For more information, see [documentation](doc-link). -* 增加获取 TSO 的 RPC 模式，降低获取 TSO 的延迟 [#54960](https://github.com/pingcap/tidb/issues/54960) @[MyonKeminta](https://github.com/MyonKeminta) **tw@qiancai** +* Add RPC mode for obtaining TSO, reducing TSO retrieval latency [#54960](https://github.com/pingcap/tidb/issues/54960) @[MyonKeminta](https://github.com/MyonKeminta) **tw@qiancai** - TiDB 在向 PD 请求 TSO 时，会将一段时间内的请求汇总起来并以同步的方式进行批处理，以减少 RPC (Remote Procedure Call) 请求数量从而降低 PD 负载。对于延迟敏感的场景，这种模式的性能并不理想。在 v8.4.0 中，TiDB 新增 TSO 请求的异步批处理模式，并提供不同的并发能力。异步模式可以降低获取 TSO 的延迟，但可能会增加 PD 的负载。你可以通过 [tidb_tso_client_rpc_mode](/system-variables.md#tidb_tso_client_rpc_mode-从-v840-版本开始引入) 变量设定获取 TSO 的 RPC 模式。 + When TiDB requests TSO from PD, it consolidates multiple requests over a period of time and processes them in synchronous batches to decrease the number of RPC (Remote Procedure Call) requests and reduce PD workload. However, the performance of this synchronous batching mode is not ideal in latency-sensitive scenarios. - 更多信息，请参考[用户文档](/system-variables.md#tidb_tso_client_rpc_mode-从-v840-版本开始引入)。 + In v8.4.0, TiDB introduces an asynchronous batching mode for TSO requests, which provides various levels of concurrency. This asynchronous mode can reduce the latency for obtaining a TSO but might increase PD workload. To set the RPC mode for obtaining TSO, you can configure the [tidb_tso_client_rpc_mode](/system-variables.md#tidb_tso_client_rpc_mode-new-in-v840) system variable. -* 优化 TiDB Hash Join 算子的执行效率（实验特性） [#55153](https://github.com/pingcap/tidb/issues/55153) [#53127](https://github.com/pingcap/tidb/issues/53127) @[windtalker](https://github.com/windtalker) @[xzhangxian1008](https://github.com/xzhangxian1008) @[XuHuaiyu](https://github.com/XuHuaiyu) @[wshwsh12](https://github.com/wshwsh12) **tw@qiancai** + For more information, see [documentation](/system-variables.md#tidb_tso_client_rpc_mode-new-in-v840). - 在 v8.4.0 中，TiDB 对 Hash Join 算子的实现方法进行了优化，以提升其执行效率。目前，优化后的 Hash Join 实现方法为实验特性，仅对 Inner Join 和 Outer Join 类型的 Hash Join 生效，且默认关闭。你可以将变量 [tidb_hash_join_version](/system-variables.md#tidb_hash_join_version-从-v840-版本开始引入) 设置为 `optimized` 开启该优化实现方法。开启后，TiDB 在执行 Inner Join 和 Outer Join 类型的 Hash Join 时，将使用优化后的实现方法。 +* Optimize the execution efficiency of the hash join operator for TiDB (experimental) [#55153](https://github.com/pingcap/tidb/issues/55153) [#53127](https://github.com/pingcap/tidb/issues/53127) @[windtalker](https://github.com/windtalker) @[xzhangxian1008](https://github.com/xzhangxian1008) @[XuHuaiyu](https://github.com/XuHuaiyu) @[wshwsh12](https://github.com/wshwsh12) **tw@qiancai** - 更多信息，请参考[用户文档](/system-variables.md#tidb_hash_join_version-从-v840-版本开始引入)。 + In v8.4.0, TiDB introduces an optimized implementation of the hash join operator to improve its execution efficiency. Currently, this optimized implementation is experimental, disabled by default, and applies only to inner join and outer join operations. To enable this optimized method, configure the [tidb_hash_join_version](/system-variables.md#tidb_hash_join_version-new-in-v840) system variable to `optimized`. -* 支持下推以下字符串函数到 TiKV [#17529](https://github.com/tikv/tikv/issues/17529) @[gengliqi](https://github.com/gengliqi) **tw@qiancai** + For more information, see [documentation](/system-variables.md#tidb_hash_join_version-new-in-v840). + +* Support pushing down the following string functions to TiKV [#17529](https://github.com/tikv/tikv/issues/17529) @[gengliqi](https://github.com/gengliqi) **tw@qiancai** * `DATE_ADD()` * `DATE_SUB()` - 更多信息，请参考[用户文档](/functions-and-operators/expressions-pushed-down.md)。 + For more information, see [documentation](/functions-and-operators/expressions-pushed-down.md). * The performance of batch user creation and password changes has been improved by hundreds of times [#55604](https://github.com/pingcap/tidb/pull/55604) @[wjhuang2016](https://github.com/wjhuang2016) **tw@hfxsd** @@ -222,17 +224,17 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- ### SQL -* 支持向量搜索功能（实验特性） [#54245](https://github.com/pingcap/tidb/issues/54245) [#9032](https://github.com/pingcap/tiflash/issues/9032) @[breezewish](https://github.com/breezewish) @[Lloyd-Pottiger](https://github.com/Lloyd-Pottiger) @[EricZequan](https://github.com/EricZequan) @[zimulala](https://github.com/zimulala) @[JaySon-Huang](https://github.com/JaySon-Huang) **tw@qiancai** +* Support vector search (experimental) [#54245](https://github.com/pingcap/tidb/issues/54245) [#9032](https://github.com/pingcap/tiflash/issues/9032) @[breezewish](https://github.com/breezewish) @[Lloyd-Pottiger](https://github.com/Lloyd-Pottiger) @[EricZequan](https://github.com/EricZequan) @[zimulala](https://github.com/zimulala) @[JaySon-Huang](https://github.com/JaySon-Huang) **tw@qiancai** - 向量搜索是一种基于数据语义的搜索方法，可以提供更相关的搜索结果。作为 AI 和大语言模型（LLM）的核心功能之一，向量搜索可用于检索增强生成（Retrieval-Augmented Generation, RAG）、语义搜索、推荐系统等多种场景。 + Vector search is a search method based on data semantics, which provides more relevant search results. As one of the core functions of AI and large language models (LLMs), vector search can be used in various scenarios such as Retrieval-Augmented Generation (RAG), semantic search, and recommendation systems. - 从 v8.4.0 开始，TiDB 支持[向量数据类型](vector-search-data-types.md)和[向量搜索索引](vector-search-index.md)，具备强大的向量搜索能力。TiDB 的向量数据类型最多可支持 16383 维度，并支持多种[距离函数](/vector-search-functions-and-operators.md#向量函数)，包括 L2 距离（欧式距离）、余弦距离、负内积和 L1 距离（曼哈顿距离）。 + Starting from v8.4.0, TiDB supports [vector data types](vector-search-data-types.md) and [vector search indexes](vector-search-index.md), offering powerful vector search capabilities. TiDB vector data types support up to 16,383 dimensions and support various [distance functions](/vector-search-functions-and-operators.md#vector-functions), including L2 distance (Euclidean distance), cosine distance, negative inner product, and L1 distance (Manhattan distance). - 在使用时，你只需要创建包含向量数据类型的表，并插入向量数据，即可执行向量搜索查询，也可进行向量数据与传统关系数据的混合查询。此外，你可以创建并利用向量搜索索引来提升向量搜索的性能。 + To start vector search, you only need to create a table with vector data types, insert vector data, and then perform a query of vector data. You can also perform mixed queries of vector data and traditional relational data. - 需要注意的是，TiDB 的向量搜索索引依赖于 TiFlash。因此，在使用向量搜索索引之前，需要确保 TiDB 集群中已部署 TiFlash 节点。 + To enhance the performance of vector search, you can create and use [vector search indexes](vector-search-index.md). Note that TiDB vector search indexes rely on TiFlash. Therefore, before using vector search indexes, make sure that TiFlash nodes are deployed in your TiDB cluster. - 更多信息，请参考[用户文档](/vector-search-overview.md)。 + For more information, see [documentation](/vector-search-overview.md). * The TiDB foreign key feature becomes generally available (GA) [#55861](https://github.com/pingcap/tidb/issues/55861) @[YangKeao](https://github.com/YangKeao) **tw@lilin90** @@ -372,14 +374,14 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- + TiDB - - Optimize MEMDB implementation to reduce write latency in transactions and TiDB CPU usage [#55287](https://github.com/pingcap/tidb/issues/55287) @[you06](https://github.com/you06) **tw@hfxsd** - - 优化系统变量 `tidb_dml_type` 为 `"bulk"` 时 DML 语句的执行性能 [#50215](https://github.com/pingcap/tidb/issues/50215) @[ekexium](https://github.com/ekexium) **tw@qiancai** - - Support using [Optimizer Fix Control 47400](/optimizer-fix-controls.md#47400-new-in-v840) to control whether the optimizer limits the minimum value estimated for `estRows` to `1`, which is consistent with databases such as Oracle and DB2 [#47400](https://github.com/pingcap/tidb/issues/47400) @[terry1purcell](https://github.com/terry1purcell) **tw@Oreoxmt** - - Add write control to the [`mysql.tidb_runaway_queries`](/mysql-schema/mysql-schema.md#system-tables-related-to-runaway-queries) log table to reduce overhead caused by a large number of concurrent writes [#54434](https://github.com/pingcap/tidb/issues/54434) @[HuSharp](https://github.com/HuSharp) **tw@lilin90** - - Spport Index Join by default when the inner table has `Selection` or `Projection` operators on it [#issue号](链接) @[winoros](https://github.com/winoros) **tw@Oreoxmt** - - Reduce the number of column details fetched from TiKV for `DELETE` operations in certain scenarios, lowering the resource overhead of these operations [#38911](https://github.com/pingcap/tidb/issues/38911) @[winoros](https://github.com/winoros) **tw@Oreoxmt** - - Improve the efficiency of the priority queue for automatic statistics collection tasks [#49972](https://github.com/pingcap/tidb/issues/49972) @[Rustin170506](https://github.com/Rustin170506) **tw@Oreoxmt** - - Improve automatic statistics collection by determining the collection concurrency based on node scale and hardware specifications [#issue号](链接) @[hawkingrei](https://github.com/hawkingrei) **tw@Oreoxmt** + - Optimize MEMDB implementation to reduce write latency in transactions and TiDB CPU usage [#55287](https://github.com/pingcap/tidb/issues/55287) @[you06](https://github.com/you06) **tw@hfxsd** + - Optimize the execution performance of DML statements when the system variable `tidb_dml_type` is set to `"bulk"` [#50215](https://github.com/pingcap/tidb/issues/50215) @[ekexium](https://github.com/ekexium) **tw@qiancai** + - Support using [Optimizer Fix Control 47400](/optimizer-fix-controls.md#47400-new-in-v840) to control whether the optimizer limits the minimum value estimated for `estRows` to `1`, which is consistent with databases such as Oracle and DB2 [#47400](https://github.com/pingcap/tidb/issues/47400) @[terry1purcell](https://github.com/terry1purcell) **tw@Oreoxmt** + - Add write control to the [`mysql.tidb_runaway_queries`](/mysql-schema/mysql-schema.md#system-tables-related-to-runaway-queries) log table to reduce overhead caused by a large number of concurrent writes [#54434](https://github.com/pingcap/tidb/issues/54434) @[HuSharp](https://github.com/HuSharp) **tw@lilin90** + - Spport Index Join by default when the inner table has `Selection` or `Projection` operators on it [#issue号](链接) @[winoros](https://github.com/winoros) **tw@Oreoxmt** + - Reduce the number of column details fetched from TiKV for `DELETE` operations in certain scenarios, lowering the resource overhead of these operations [#38911](https://github.com/pingcap/tidb/issues/38911) @[winoros](https://github.com/winoros) **tw@Oreoxmt** + - Improve the efficiency of the priority queue for automatic statistics collection tasks [#49972](https://github.com/pingcap/tidb/issues/49972) @[Rustin170506](https://github.com/Rustin170506) **tw@Oreoxmt** + - Improve automatic statistics collection by determining the collection concurrency based on node scale and hardware specifications [#issue号](链接) @[hawkingrei](https://github.com/hawkingrei) **tw@Oreoxmt** + TiKV @@ -389,15 +391,15 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- + PD - - 在 TiDB Lightning 数据导入过程中，允许 PD 进行 `evict-leader` 调度，避免阻断 TiKV 节点的 offline 进程 [#issue号](链接) @[okJiang](https://github.com/okJiang) **tw@qiancai** - + - Support graceful offline of TiKV nodes during data import by TiDB Lightning [#7853](https://github.com/tikv/pd/issues/7853) @[okJiang](https://github.com/okJiang) **tw@qiancai** + TiFlash + Tools + Backup & Restore (BR) - - Disable full data restoration to a non-empty cluster using the `RESTORE` SQL statement by default [#55087](https://github.com/pingcap/tidb/issues/55087) @[BornChanger](https://github.com/BornChanger) **tw@Oreoxmt** + - Disable splitting Regions by table to improve restore speed when restoring data to a cluster where the `split-table` and `split-region-on-table` configuration items are `false` (default value) [#53532](https://github.com/pingcap/tidb/issues/53532) @[Leavrth](https://github.com/Leavrth) **tw@qiancai** + - Disable full data restoration to a non-empty cluster using the `RESTORE` SQL statement by default [#55087](https://github.com/pingcap/tidb/issues/55087) @[BornChanger](https://github.com/BornChanger) **tw@Oreoxmt** + TiCDC From d7e16a3b48f66f01671687476c61011c567a679d Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Thu, 26 Sep 2024 11:15:51 +0800 Subject: [PATCH 24/45] add translation for BR highlights --- releases/release-8.4.0.md | 24 ++++++++++++++---------- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/releases/release-8.4.0.md b/releases/release-8.4.0.md index 72ad36037ffd..eb06cb10730a 100644 --- a/releases/release-8.4.0.md +++ b/releases/release-8.4.0.md @@ -32,8 +32,8 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- Global indexes can effectively improve the efficiency of retrieving non-partitioned columns, and remove the restriction that a unique key must contain the partition key. This feature extends the usage scenarios of TiDB partitioned tables and avoids some of the application modification work that might be required for data migration. - TiDB 并行获取 TSO**tw@qiancai 1893** - 在高并发场景下，并行获取 TSO 能够有效降低等待获取 TSO 的时间，提升集群的吞吐。 + Concurrent TSO retrieval for TiDB**tw@qiancai 1893** + In high-concurrency scenarios, you can use this feature to reduce the wait time for obtaining TSO and improve the cluster throughput. Improve the execution efficiency of administrative SQL statements**tw@hfxsd 1941** @@ -66,8 +66,8 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- Support MySQL-compatible foreign key constraints to maintain data consistency and further enhance TiDB's compatibility with MySQL. - 向量搜索功能（实验特性）**tw@qiancai 1898** - 加速向量搜索的性能，适用于检索增强生成（RAG）、语义搜索、推荐系统等应用类型。把 TiDB 应用场景扩展到 AI 和 大语言模型（LLM）领域。 + Vector search (experimental)**tw@qiancai 1898** + Vector search is a search method based on data semantics, which provides more relevant search results. As one of the core functions of AI and large language models (LLMs), vector search can be used in various scenarios such as Retrieval-Augmented Generation (RAG), semantic search, and recommendation systems. DB Operations and Observability @@ -79,9 +79,9 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- AWS EC2 now uses IMDSv2 as the default metadata service. TiDB supports data backups from TiKV instances with IMDSv2 enabled, enhancing your ability to run TiDB clusters in public cloud environments. - 安全 - 备份数据加密成为正式功能**tw@qiancai 1920** - 加密数据库备份是一种增强数据安全性的重要措施，既可以保护数据备份中敏感信息，又有助于合规，确保数据在存储和传输中的安全。 + Security + Client-side encryption of log backups**tw@qiancai 1920** + Before uploading a log backup to your backup storage, you can encrypt the backup data to ensure its security during storage and transmission. @@ -254,11 +254,15 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- ### DB operations -* PITR adds client-side log backup data encryption support (experimental) [55834](https://github.com/pingcap/tidb/issues/55834) @[Tristan1900](https://github.com/Tristan1900) **tw@qiancai** +* BR supports client-side encryption of log backups (experimental) [55834](https://github.com/pingcap/tidb/issues/55834) @[Tristan1900](https://github.com/Tristan1900) **tw@qiancai** - Previously only the data from a snapshot based backup could be encrypted (on the client side) with a data key provided by the user. With this feature, log backups may now also be encrypted, ensuring that the confidentiality of information within the backup data is secured. + In earlier TiDB versions, only snapshot backups can be encrypted on the client side. Starting from v8.4.0, log backups can also be encrypted on the client side. Before uploading a log backup to your backup storage, you can encrypt the backup data to ensure its security via one of the following methods: - For more information, see [documentation](doc-link). + - Encrypt using a custom fixed key + - Encrypt using the main key from the local disk + - Encrypt using the main key from a Key Management Service (KMS) + + For more information, see [documentation](/br/br-pitr-manual.md#encrypt-log-backup-data). * BR reduces privileges when restoring backup data in a cloud storage system [#55870](https://github.com/pingcap/tidb/issues/55870) @[Leavrth](https://github.com/Leavrth) **tw@Oreoxmt** From 6be2f6d7507a29f008f718c133762664396cd1bd Mon Sep 17 00:00:00 2001 From: Lilian Lee Date: Thu, 26 Sep 2024 11:23:36 +0800 Subject: [PATCH 25/45] Update TiDB Binlog removal --- releases/release-8.4.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-8.4.0.md b/releases/release-8.4.0.md index eb06cb10730a..8875153e0398 100644 --- a/releases/release-8.4.0.md +++ b/releases/release-8.4.0.md @@ -356,7 +356,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- * The following features are removed starting from v8.4.0: - * TiDB Binlog replication is now removed from this version. Starting from v8.3.0, TiDB Binlog was fully deprecated. For incremental data replication, use [TiCDC](/ticdc-overview.md) instead. For point-in-time recovery (PITR), use [PITR](/br-pitr-guide.md). **tw@lilin90** + * In v8.4.0, [TiDB Binlog](https://docs.pingcap.com/tidb/v8.3/tidb-binlog-overview) is removed. Starting from v8.3.0, TiDB Binlog is fully deprecated. For incremental data replication, use [TiCDC](/ticdc-overview.md) instead. For point-in-time recovery (PITR), use [PITR](/br-pitr-guide.md). Before you upgrade your TiDB cluster to v8.4.0 or later versions, be sure to switch to TiCDC and PITR. **tw@lilin90** * The following features are deprecated starting from v8.4.0: From 0793443970d89013180d1322f4a4d73f050f26fb Mon Sep 17 00:00:00 2001 From: Aolin Date: Thu, 26 Sep 2024 15:13:24 +0800 Subject: [PATCH 26/45] add GitHub issue link for automatic statistics collection --- releases/release-8.4.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-8.4.0.md b/releases/release-8.4.0.md index 8875153e0398..15e05d0abd3a 100644 --- a/releases/release-8.4.0.md +++ b/releases/release-8.4.0.md @@ -385,7 +385,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- - Spport Index Join by default when the inner table has `Selection` or `Projection` operators on it [#issue号](链接) @[winoros](https://github.com/winoros) **tw@Oreoxmt** - Reduce the number of column details fetched from TiKV for `DELETE` operations in certain scenarios, lowering the resource overhead of these operations [#38911](https://github.com/pingcap/tidb/issues/38911) @[winoros](https://github.com/winoros) **tw@Oreoxmt** - Improve the efficiency of the priority queue for automatic statistics collection tasks [#49972](https://github.com/pingcap/tidb/issues/49972) @[Rustin170506](https://github.com/Rustin170506) **tw@Oreoxmt** - - Improve automatic statistics collection by determining the collection concurrency based on node scale and hardware specifications [#issue号](链接) @[hawkingrei](https://github.com/hawkingrei) **tw@Oreoxmt** + - Improve automatic statistics collection by determining the collection concurrency based on node scale and hardware specifications [#53460](https://github.com/pingcap/tidb/issues/53460) @[hawkingrei](https://github.com/hawkingrei) **tw@Oreoxmt** + TiKV From 858595fd95b03ba4c3b62725c93591c1c6a2f60a Mon Sep 17 00:00:00 2001 From: Lilian Lee Date: Thu, 26 Sep 2024 15:32:21 +0800 Subject: [PATCH 27/45] Remove foreign key GA --- releases/release-8.4.0.md | 12 +----------- 1 file changed, 1 insertion(+), 11 deletions(-) diff --git a/releases/release-8.4.0.md b/releases/release-8.4.0.md index 15e05d0abd3a..4ad3300540b8 100644 --- a/releases/release-8.4.0.md +++ b/releases/release-8.4.0.md @@ -61,11 +61,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- Automatic statistics collection determines the collection concurrency based on node scale and hardware specifications. This improves statistics collection efficiency, reduces manual tuning, and ensures stable cluster performance. - SQL - Foreign keys (GA)**tw@lilin90 1894** - Support MySQL-compatible foreign key constraints to maintain data consistency and further enhance TiDB's compatibility with MySQL. - - + SQL Vector search (experimental)**tw@qiancai 1898** Vector search is a search method based on data semantics, which provides more relevant search results. As one of the core functions of AI and large language models (LLMs), vector search can be used in various scenarios such as Retrieval-Augmented Generation (RAG), semantic search, and recommendation systems. @@ -236,12 +232,6 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- For more information, see [documentation](/vector-search-overview.md). -* The TiDB foreign key feature becomes generally available (GA) [#55861](https://github.com/pingcap/tidb/issues/55861) @[YangKeao](https://github.com/YangKeao) **tw@lilin90** - - Starting from v6.6.0, TiDB supports foreign key constraints using the system variable [`foreign_key_checks`](/system-variables.md#foreign_key_checks), but it has been an experimental feature. In v8.4.0, the foreign key feature has been extensively tested in more scenarios to improve stability and performance. Therefore, starting from v8.4.0, the foreign key feature becomes generally available (GA). - - For more information, see [documentation](/foreign-key.md). - * Support the `gb18030` character set and the `gb18030_bin` and `gb18030_chinese_ci` collations [#17470](https://github.com/tikv/tikv/issues/17470) [#55791](https://github.com/pingcap/tidb/issues/55791) @[cbcwestwolf](https://github.com/cbcwestwolf) **tw@lilin90** Starting from v8.4.0, TiDB supports the `gb18030` character set to ensure that TiDB can better handle Chinese-related data storage and query requirements. This character set is a standard widely used for Chinese character encoding. From bcf3e8661872966c89f2dc33f831c987c2e67c3c Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Thu, 26 Sep 2024 16:49:34 +0800 Subject: [PATCH 28/45] wording updates --- releases/release-8.4.0.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/releases/release-8.4.0.md b/releases/release-8.4.0.md index 4ad3300540b8..2602fa083883 100644 --- a/releases/release-8.4.0.md +++ b/releases/release-8.4.0.md @@ -76,8 +76,8 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- Security - Client-side encryption of log backups**tw@qiancai 1920** - Before uploading a log backup to your backup storage, you can encrypt the backup data to ensure its security during storage and transmission. + Client-side encryption of log backup data**tw@qiancai 1920** + Before uploading log backup data to your backup storage, you can encrypt the backup data to ensure its security during storage and transmission. @@ -110,7 +110,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- * Optimize the execution efficiency of the hash join operator for TiDB (experimental) [#55153](https://github.com/pingcap/tidb/issues/55153) [#53127](https://github.com/pingcap/tidb/issues/53127) @[windtalker](https://github.com/windtalker) @[xzhangxian1008](https://github.com/xzhangxian1008) @[XuHuaiyu](https://github.com/XuHuaiyu) @[wshwsh12](https://github.com/wshwsh12) **tw@qiancai** - In v8.4.0, TiDB introduces an optimized implementation of the hash join operator to improve its execution efficiency. Currently, this optimized implementation is experimental, disabled by default, and applies only to inner join and outer join operations. To enable this optimized method, configure the [tidb_hash_join_version](/system-variables.md#tidb_hash_join_version-new-in-v840) system variable to `optimized`. + In v8.4.0, TiDB introduces an optimized implementation of the hash join operator to improve its execution efficiency. Currently, this optimized implementation of hash join applies only to inner join and outer join operations and is disabled by default. To enable this optimized implementation, configure the [tidb_hash_join_version](/system-variables.md#tidb_hash_join_version-new-in-v840) system variable to `optimized`. For more information, see [documentation](/system-variables.md#tidb_hash_join_version-new-in-v840). @@ -244,13 +244,13 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- ### DB operations -* BR supports client-side encryption of log backups (experimental) [55834](https://github.com/pingcap/tidb/issues/55834) @[Tristan1900](https://github.com/Tristan1900) **tw@qiancai** +* BR supports client-side encryption of log backup data (experimental) [55834](https://github.com/pingcap/tidb/issues/55834) @[Tristan1900](https://github.com/Tristan1900) **tw@qiancai** - In earlier TiDB versions, only snapshot backups can be encrypted on the client side. Starting from v8.4.0, log backups can also be encrypted on the client side. Before uploading a log backup to your backup storage, you can encrypt the backup data to ensure its security via one of the following methods: + In earlier TiDB versions, only snapshot backup data can be encrypted on the client side. Starting from v8.4.0, log backup data can also be encrypted on the client side. Before uploading log backup data to your backup storage, you can encrypt the backup data to ensure its security via one of the following methods: - Encrypt using a custom fixed key - - Encrypt using the main key from the local disk - - Encrypt using the main key from a Key Management Service (KMS) + - Encrypt using a master key stored on a local disk + - Encrypt using a master key managed by a Key Management Service (KMS) For more information, see [documentation](/br/br-pitr-manual.md#encrypt-log-backup-data). From 3675a62759e4f4a270702b1265cc18cd605ec55b Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Thu, 26 Sep 2024 23:05:35 +0800 Subject: [PATCH 29/45] Apply suggestions from code review --- releases/release-8.4.0.md | 8 -------- 1 file changed, 8 deletions(-) diff --git a/releases/release-8.4.0.md b/releases/release-8.4.0.md index 2602fa083883..d2e5216f1592 100644 --- a/releases/release-8.4.0.md +++ b/releases/release-8.4.0.md @@ -35,10 +35,6 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- Concurrent TSO retrieval for TiDB**tw@qiancai 1893** In high-concurrency scenarios, you can use this feature to reduce the wait time for obtaining TSO and improve the cluster throughput. - - Improve the execution efficiency of administrative SQL statements**tw@hfxsd 1941** - In some SaaS systems, there is a need to create users in batch and rotate passwords regularly. TiDB enhances the performance of creating and modifying database users, ensuring these operations can be completed within the desired time window. - Improve query performance for cached tables**tw@hfxsd 1965** Improve query performance for index scanning on cached tables, with improvements of up to 5.4 times in some scenarios. For high-speed queries on small tables, cached tables can significantly enhance overall performance. @@ -121,10 +117,6 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- For more information, see [documentation](/functions-and-operators/expressions-pushed-down.md). -* The performance of batch user creation and password changes has been improved by hundreds of times [#55604](https://github.com/pingcap/tidb/pull/55604) @[wjhuang2016](https://github.com/wjhuang2016) **tw@hfxsd** - - In SaaS scenarios, you might need to batch-create a large number of users, rotate passwords periodically, and complete these tasks within a specific time window. Starting from v8.4.0, the performance of batch user creation and password rotation has been significantly improved. Additionally, you can further enhance performance by increasing concurrency through a higher number of session connections, which greatly reduces execution time for these operations. - * Instance-level execution plan cache (experimental) [#54057](https://github.com/pingcap/tidb/issues/54057) @[qw4990](https://github.com/qw4990) **tw@Oreoxmt** TiDB v8.4.0 introduces instance-level execution plan cache as an experimental feature. This feature allows all sessions within the same TiDB instance to share the execution plan cache, significantly reducing TiDB latency, improving cluster throughput, decreasing the likelihood of execution plan fluctuations, and maintaining stable cluster performance. Compared with session-level execution plan cache, instance-level execution plan cache offers the following advantages: From a7cb9014a6d8ba089143b8871bf2c11c39861ebe Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Fri, 27 Sep 2024 11:09:06 +0800 Subject: [PATCH 30/45] Apply suggestions from code review Co-authored-by: Lilian Lee --- releases/release-8.4.0.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/releases/release-8.4.0.md b/releases/release-8.4.0.md index d2e5216f1592..ae007638cf92 100644 --- a/releases/release-8.4.0.md +++ b/releases/release-8.4.0.md @@ -29,7 +29,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- Global indexes for partitioned tables (GA)**tw@hfxsd 1961** - Global indexes can effectively improve the efficiency of retrieving non-partitioned columns, and remove the restriction that a unique key must contain the partition key. This feature extends the usage scenarios of TiDB partitioned tables and avoids some of the application modification work that might be required for data migration. + Global indexes can effectively improve the efficiency of retrieving non-partitioned columns, and remove the restriction that a unique key must contain the partition key. This feature extends the usage scenarios of TiDB partitioned tables, and avoids some of the application modification work required for data migration. Concurrent TSO retrieval for TiDB**tw@qiancai 1893** @@ -37,7 +37,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- Improve query performance for cached tables**tw@hfxsd 1965** - Improve query performance for index scanning on cached tables, with improvements of up to 5.4 times in some scenarios. For high-speed queries on small tables, cached tables can significantly enhance overall performance. + Improve query performance for index scanning on cached tables, with improvements of up to 5.4 times in some scenarios. For high-speed queries on small tables, using cached tables can significantly enhance overall performance. Reliability and Availability @@ -45,8 +45,8 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- Runaway Queries offer an effective way to mitigate the impact of unexpected SQL performance issues on systems. TiDB v8.4.0 introduces the number of keys processed by the Coprocessor (PROCESSED_KEYS) and request units (RU) as identifying conditions, and puts identified queries into the specified resource group for more precise identification and control of runaway queries. - Support setting resource usage caps for background tasks for resource control **tw@hfxsd 1909** - By setting a percentage cap on background tasks of resource control, you can manage their resource consumption based on the needs of different business systems. This ensures background tasks consume minimal resources, maintaining the service quality of online operations. + Support setting the maximum limit on resource usage for background tasks of resource control **tw@hfxsd 1909** + By setting a maximum percentage limit on background tasks of resource control, you can control their resource consumption based on the needs of different application systems. This keeps background task consumption at a low level and ensures the quality of online services. TiProxy supports traffic capture and replay (experimental)**tw@Oreoxmt 1942** @@ -64,11 +64,11 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- DB Operations and Observability Display TiKV and TiDB CPU times in memory tables**tw@hfxsd 1877** - CPU times are now integrated into a system table and displayed alongside other session or SQL metrics, allowing for easier observation of operations with high CPU consumption from multiple perspectives, improving diagnostic efficiency. This is particularly useful for diagnosing instances with CPU spikes or read/write hotspots in the cluster. + The CPU time is now integrated into a system table, displayed alongside other metrics for sessions or SQL, letting you observe high CPU consumption operations from multiple perspectives, and improves diagnostic efficiency. This is especially useful for diagnosing scenarios such as CPU spikes in instances or read/write hotspots in clusters. Support backing up TiKV instances with IMDSv2 service enabled**tw@hfxsd 1945** - AWS EC2 now uses IMDSv2 as the default metadata service. TiDB supports data backups from TiKV instances with IMDSv2 enabled, enhancing your ability to run TiDB clusters in public cloud environments. + AWS EC2 now uses IMDSv2 as the default metadata service. TiDB supports backing up data from TiKV instances that have IMDSv2 enabled, helping you run TiDB clusters more effectively in public cloud services. Security From ee4b8653451779d62fdd010140303c33e000db71 Mon Sep 17 00:00:00 2001 From: Lilian Lee Date: Fri, 27 Sep 2024 14:00:47 +0800 Subject: [PATCH 31/45] Update wording Co-authored-by: xixirangrang --- releases/release-8.4.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-8.4.0.md b/releases/release-8.4.0.md index ae007638cf92..82ac7713b944 100644 --- a/releases/release-8.4.0.md +++ b/releases/release-8.4.0.md @@ -130,7 +130,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- * TiDB Lightning's logical import mode supports prepared statements and client statement cache [#54850](https://github.com/pingcap/tidb/issues/54850) @[dbsid](https://github.com/dbsid) **tw@lilin90** - By enabling the `logical-import-prep-stmt` configuration item, the SQL statements generated by TiDB Lightning's logical import mode will use prepared statements and client statement cache. This reduces the cost of TiDB SQL parsing and compilation, improves SQL execution efficiency, and increases the likelihood of hitting the execution plan cache, thereby speeding up logical import. + By enabling the `logical-import-prep-stmt` configuration item, the SQL statements executed in TiDB Lightning's logical import mode will use prepared statements and client statement cache. This reduces the cost of TiDB SQL parsing and compilation, improves SQL execution efficiency, and increases the likelihood of hitting the execution plan cache, thereby speeding up logical import. For more information, see [documentation](/tidb-lightning/tidb-lightning-configuration.md). From 424e94117429a6542fa99f413efccd53eb5c405a Mon Sep 17 00:00:00 2001 From: Lilian Lee Date: Fri, 27 Sep 2024 14:55:56 +0800 Subject: [PATCH 32/45] Apply suggestions from code review --- releases/release-8.4.0.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/releases/release-8.4.0.md b/releases/release-8.4.0.md index 82ac7713b944..ccdf3af6dd51 100644 --- a/releases/release-8.4.0.md +++ b/releases/release-8.4.0.md @@ -136,17 +136,17 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- * Partitioned tables support global indexes (GA) [#45133](https://github.com/pingcap/tidb/issues/45133) @[mjonss](https://github.com/mjonss) @[Defined2014](https://github.com/Defined2014) @[jiyfhust](https://github.com/jiyfhust) @[L-maple](https://github.com/L-maple) **tw@hfxsd** - In previous versions of partitioned tables, some limitations exist because global indexes are not supported. For example, the unique key must use every column in the table's partitioning expression. If the query condition does not use the partitioning key, the query will scan all partitions, resulting in poor performance. Starting from v7.6.0, the system variable [`tidb_enable_global_index`](/system-variables.md#tidb_enable_global_index-new-in-v760) is introduced to enable the global index feature. But this feature was under development at that time and it is not recommended to enable it. + In early TiDB versions, the partitioned table has some limitations because it does not support global indexes. For example, the unique key must use every column in the table's partition expression. If the query condition does not use the partition key, the query will scan all partitions, resulting in poor performance. Starting from v7.6.0, the system variable [`tidb_enable_global_index`](/system-variables.md#tidb_enable_global_index-new-in-v760) is introduced to enable the global index feature. But this feature was under development at that time and it is not recommended to enable it. - Starting with v8.3.0, the global index feature is released as an experimental feature. You can explicitly create a global index for a partitioned table with the keyword `Global` to remove the restriction that the unique key must use every column in the table's partitioning expression, to meet flexible business needs. Global indexes also enhance the performance of queries that do not include partition keys. + Starting from v8.3.0, the global index feature is released as an experimental feature. You can explicitly create a global index for a partitioned table with the `GLOBAL` keyword. This removes the restriction that a unique key in a partitioned table must include all columns used in the partition expression, allowing for more flexible application requirements. Additionally, global indexes also improve the performance of queries based on non-partitioned columns. In v8.4.0, this feature becomes generally available (GA). You must use the keyword `GLOBAL` to create a global index, instead of setting the system variable [`tidb_enable_global_index`](/system-variables.md#tidb_enable_global_index-new-in-v760) to enable the global index feature. From v8.4.0 this system variable is deprecated and is always `ON`. For more information, see [documentation](/partitioned-table.md#global-indexes). -* Optimize query performance for cached tables in some scenarios [#43249](https://github.com/pingcap/tidb/issues/43249) @[tiancaiamao](https://github.com/tiancaiamao) **tw@hfxsd** +* Improve query performance for cached tables in some scenarios [#43249](https://github.com/pingcap/tidb/issues/43249) @[tiancaiamao](https://github.com/tiancaiamao) **tw@hfxsd** - Optimize the query performance of cached tables by up to 5.4 times when using `IndexLookup` to execute `SELECT ... LIMIT 1` with `IndexLookup`. Improve the performance of `IndexLookupReader` in full table scan and primary key query scenarios. + In v8.4.0, TiDB improves the query performance of cached tables by up to 5.4 times when executing `SELECT ... LIMIT 1` with `IndexLookup`. In addition, TiDB improves the performance of `IndexLookupReader` in full table scan and primary key query scenarios. ### Reliability @@ -166,21 +166,21 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- For more information, see [documentation](/tidb-resource-control.md#manage-queries-that-consume-more-resources-than-expected-runaway-queries). -* Support runaway queries to switch resource groups [#54434](https://github.com/pingcap/tidb/issues/54434) @[JmPotato](https://github.com/JmPotato) **tw@hfxsd** +* Support switching resource groups for runaway queries [#54434](https://github.com/pingcap/tidb/issues/54434) @[JmPotato](https://github.com/JmPotato) **tw@hfxsd** - In TiDB v8.4.0, you can redirect runaway queries to a specific resource group. If the `COOLDOWN` mechanism fails to lower resource consumption, you can create a [resource group](/tidb-resource-control.md#create-a-resource-group) and set the `SWITCH_GROUP` parameter to move identified runaway queries to this group. Meanwhile, subsequent queries within the same session will continue to execute in the original resource group. By switching resource groups, you can more precisely manage resource usage and better control the impact of runaway queries. + Starting from TiDB v8.4.0, you can switch the resource group of runaway queries to a specific one. If the `COOLDOWN` mechanism fails to lower resource consumption, you can create a [resource group](/tidb-resource-control.md#create-a-resource-group) and set the `SWITCH_GROUP` parameter to move identified runaway queries to this group. Meanwhile, subsequent queries within the same session will continue to execute in the original resource group. By switching resource groups, you can manage resource usage more precisely, and control the resource consumption more strictly. For more information, see [documentation](/tidb-resource-control.md#query_limit-parameters). -* The system variable `tidb_scatter_region` supports the cluster-level Region scattering strategy [#55184](https://github.com/pingcap/tidb/issues/55184) @[D3Hunter](https://github.com/D3Hunter) **tw@hfxsd** +* Support setting the cluster-level Region scattering strategy using the `tidb_scatter_region` system variable [#55184](https://github.com/pingcap/tidb/issues/55184) @[D3Hunter](https://github.com/D3Hunter) **tw@hfxsd** - In previous versions, the system variable `tidb_scatter_region` can only be enabled or disabled. When enabled, it applies a table-level scattering strategy during batch table creation. However, when creating hundreds of thousands of tables in a batch, this approach results in a concentration of regions on a few TiKV nodes, causing out-of-memory (OOM) issues on those nodes. + Before v8.4.0, the `tidb_scatter_region` system variable can only be enabled or disabled. When it is enabled, TiDB applies a table-level scattering strategy during batch table creation. However, when creating hundreds of thousands of tables in a batch, this strategy results in a concentration of Regions in a few TiKV nodes, causing OOM (Out of Memory) issues in those nodes. - To address this, starting from v8.4.0, `tidb_scatter_region` is changed to a string type. It now supports a cluster-level scattering strategy, helping scatter regions more evenly and preventing OOM problems on TiKV nodes. + Starting from v8.4.0, `tidb_scatter_region` is changed to the string type. It now supports a cluster-level scattering strategy, which can help avoid TiKV OOM issues in the preceding scenario. For more information, see [documentation](/system-variables.md#tidb_scatter_region). -* Support setting resource caps for background tasks of resource control [#56019](https://github.com/pingcap/tidb/issues/56019) @[glorv](https://github.com/glorv) **tw@hfxsd** +* Support setting the maximum limit on resource usage for background tasks of resource control [#56019](https://github.com/pingcap/tidb/issues/56019) @[glorv](https://github.com/glorv) **tw@hfxsd** TiDB resource control can identify and lower the priority of background tasks. In certain scenarios, you might want to limit the resource consumption of these tasks, even when resources are available. Starting from v8.4.0, you can use the `UTILIZATION_LIMIT` parameter to set a maximum percentage of resources that a background task can consume. Each node will ensure that the resource usage of all background tasks stays within this limit. This feature enables precise control over resource consumption for background tasks, enhancing cluster stability. From 10a6495afd3276b69030b7e3c1719c0781a2f642 Mon Sep 17 00:00:00 2001 From: Lilian Lee Date: Fri, 27 Sep 2024 14:56:35 +0800 Subject: [PATCH 33/45] Apply suggestions from code review Co-authored-by: xixirangrang --- releases/release-8.4.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-8.4.0.md b/releases/release-8.4.0.md index ccdf3af6dd51..8da1c944d385 100644 --- a/releases/release-8.4.0.md +++ b/releases/release-8.4.0.md @@ -266,7 +266,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- For more information, see [documentation](/information-schema/information-schema-processlist.md) and [documentation](information-schema/information-schema-slow-query.md). -* Top SQL supports viewing aggregated results by table or database [#55540](https://github.com/pingcap/tidb/issues/55540) @[nolouch](https://github.com/nolouch) **tw@lilin90** +* Top SQL supports viewing aggregated CPU time results by table or database [#55540](https://github.com/pingcap/tidb/issues/55540) @[nolouch](https://github.com/nolouch) **tw@lilin90** Before v8.4.0, [Top SQL](/dashboard/top-sql.md) aggregates CPU time by SQL. If CPU time is not consumed by a few SQL statements, aggregation by SQL cannot effectively identify issues. Starting from v8.4.0, you can choose to aggregate CPU time **By TABLE** or **By DB**. In scenarios with multiple systems, the new aggregation method can more effectively identify load changes from a specific system, improving diagnostic efficiency. From cdcd68268614ffc25d7aa36b572dcd301a9afc51 Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Fri, 27 Sep 2024 17:08:29 +0800 Subject: [PATCH 34/45] Update releases/release-8.4.0.md --- releases/release-8.4.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-8.4.0.md b/releases/release-8.4.0.md index 8da1c944d385..c01dc1327fcc 100644 --- a/releases/release-8.4.0.md +++ b/releases/release-8.4.0.md @@ -212,7 +212,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- ### SQL -* Support vector search (experimental) [#54245](https://github.com/pingcap/tidb/issues/54245) [#9032](https://github.com/pingcap/tiflash/issues/9032) @[breezewish](https://github.com/breezewish) @[Lloyd-Pottiger](https://github.com/Lloyd-Pottiger) @[EricZequan](https://github.com/EricZequan) @[zimulala](https://github.com/zimulala) @[JaySon-Huang](https://github.com/JaySon-Huang) **tw@qiancai** +* Support vector search (experimental) [#54245](https://github.com/pingcap/tidb/issues/54245) [#9032](https://github.com/pingcap/tiflash/issues/9032) @[breezewish](https://github.com/breezewish) @[Lloyd-Pottiger](https://github.com/Lloyd-Pottiger) @[EricZequan](https://github.com/EricZequan) @[zimulala](https://github.com/zimulala) @[JaySon-Huang](https://github.com/JaySon-Huang) @[winoros](https://github.com/winoros) **tw@qiancai** Vector search is a search method based on data semantics, which provides more relevant search results. As one of the core functions of AI and large language models (LLMs), vector search can be used in various scenarios such as Retrieval-Augmented Generation (RAG), semantic search, and recommendation systems. From cac1db3d1484914e2084f803957cd8c9a00d2067 Mon Sep 17 00:00:00 2001 From: lilin90 Date: Fri, 27 Sep 2024 18:22:00 +0800 Subject: [PATCH 35/45] Refine descriptions --- releases/release-8.4.0.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/releases/release-8.4.0.md b/releases/release-8.4.0.md index c01dc1327fcc..e1537d41c3aa 100644 --- a/releases/release-8.4.0.md +++ b/releases/release-8.4.0.md @@ -182,7 +182,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- * Support setting the maximum limit on resource usage for background tasks of resource control [#56019](https://github.com/pingcap/tidb/issues/56019) @[glorv](https://github.com/glorv) **tw@hfxsd** - TiDB resource control can identify and lower the priority of background tasks. In certain scenarios, you might want to limit the resource consumption of these tasks, even when resources are available. Starting from v8.4.0, you can use the `UTILIZATION_LIMIT` parameter to set a maximum percentage of resources that a background task can consume. Each node will ensure that the resource usage of all background tasks stays within this limit. This feature enables precise control over resource consumption for background tasks, enhancing cluster stability. + TiDB resource control can identify and lower the priority of background tasks. In certain scenarios, you might want to limit the resource consumption of background tasks, even when resources are available. Starting from v8.4.0, you can use the `UTILIZATION_LIMIT` parameter to set the maximum percentage of resources that background tasks can consume. Each node will keep the resource usage of all background tasks below this percentage. This feature enables precise control over resource consumption for background tasks, further enhancing cluster stability. For more information, see [documentation](/tidb-resource-control.md#manage-background-tasks). @@ -254,15 +254,15 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- ### Observability -* Display the CPU time of TiDB and TiKV in the system table [#55542](https://github.com/pingcap/tidb/issues/55542) @[yibin87](https://github.com/yibin87) **tw@hfxsd** +* Display the CPU time consumed by TiDB and TiKV in the system table [#55542](https://github.com/pingcap/tidb/issues/55542) @[yibin87](https://github.com/yibin87) **tw@hfxsd** - The [Top SQL page](/dashboard/top-sql.md) of [TiDB Dashboard](/dashboard/dashboard-intro.md) displays SQL statements with high CPU consumption. Starting from v8.4.0, TiDB includes CPU time consumption data in the system table, alongside other session or SQL metrics, allowing you to easily monitor high CPU usage from multiple perspectives. This information is especially useful in identifying the root cause of issues such as CPU spikes or hotspots in cluster read/write operations. + The [Top SQL page](/dashboard/top-sql.md) of [TiDB Dashboard](/dashboard/dashboard-intro.md) displays SQL statements with high CPU consumption. Starting from v8.4.0, TiDB adds CPU time consumption information to the system table, presented alongside other metrics for sessions or SQL, making it easier to observe high CPU consumption operations from multiple perspectives. This information can help you quickly identify the causes of issues in scenarios like instance CPU spikes or read/write hotspots in clusters. - - [STATEMENTS_SUMMARY](/statement-summary-tables.md) adds `AVG_TIDB_CPU_TIME` and `AVG_TIKV_CPU_TIME` to show the average CPU time consumed by individual SQL statements historically. - - [INFORMATION_SCHEMA.PROCESSLIST](/information-schema/information-schema-processlist.md) adds `TIDB_CPU` and `TIKV_CPU` to display the cumulative CPU consumption of currently executing SQL statements in a session. - - The [slow Log](/analyze-slow-queries.md) adds the `Tidb_cpu_time` and `Tikv_cpu_time` fields to show the CPU time of captured SQL statements. + - The [statement summary tables](/statement-summary-tables.md) add `AVG_TIDB_CPU_TIME` and `AVG_TIKV_CPU_TIME`, showing the average CPU time consumed by individual SQL statements historically. + - The [INFORMATION_SCHEMA.PROCESSLIST](/information-schema/information-schema-processlist.md) table adds `TIDB_CPU` and `TIKV_CPU`, showing the cumulative CPU consumption of the SQL statements currently being executed in a session. + - The [slow query Log](/analyze-slow-queries.md) adds the `Tidb_cpu_time` and `Tikv_cpu_time` fields, showing the CPU time of captured SQL statements. - By default, TiKV CPU time is displayed. Collecting TiDB CPU time introduces an additional overhead (about 8%), so TiDB CPU time is only displayed as the actual value when the [Top SQL feature](https://github.com/dashboard/top-sql.md) is enabled; otherwise, it will always display as `0`. + By default, the CPU time consumed by TiKV is displayed. Collecting the CPU time consumed by TiDB brings additional overhead (about 8%), so the CPU time consumed by TiDB only shows the actual value when [Top SQL](https://github.com/dashboard/top-sql.md) is enabled; otherwise, it always show as `0`. For more information, see [documentation](/information-schema/information-schema-processlist.md) and [documentation](information-schema/information-schema-slow-query.md). @@ -276,7 +276,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- * BR supports AWS IMDSv2 [#16443](https://github.com/tikv/tikv/issues/16443) @[pingyu](https://github.com/pingyu) **tw@hfxsd** - BR now supports AWS Instance Metadata Service Version 2 (IMDSv2) when deployed on AWS EC2. This allows you to configure the newer session-oriented method on EC2 instances, enabling BR to successfully use the instance's associated IAM role to access AWS S3 with the appropriate privileges. + When deploying TiDB on Amazon EC2, BR supports AWS Instance Metadata Service Version 2 (IMDSv2). You can configure your EC2 instance to allow BR to use the IAM role associated with the instance for appropriate permissions to access Amazon S3. For more information, see [documentation](/backup-and-restore-storages#authentication). @@ -373,7 +373,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- - Increase the default value of Region from 96 MiB to 256 MiB to avoid the extra overhead caused by too many Regions [#17309](https://github.com/tikv/tikv/issues/17309) [LykxSassinator](https://github.com/LykxSassinator) **tw@hfxsd** - Support setting memory usage limits for in-memory pessimistic locks in a Region or TiKV instance. To prevent CPU/IO overhead caused by pessimistic locks spilling to disk during write hotspots, you can increase the memory limit by modifying the configuration items [#17542](https://github.com/tikv/tikv/issues/17542) @[cfzjywxk](https://github.com/cfzjywxk) **tw@Oreoxmt** - - Introduce a new `spill-dir` configuration in Raft Engine to support multi-disk storage for Raft logs. When the disk containing the home directory (`dir`) runs out of space, Raft Engine automatically writes new logs to `spill-dir`, ensuring continuous operation. [LykxSassinator](https://github.com/LykxSassinator) **tw@hfxsd** + - Introduce a new `spill-dir` configuration item in Raft Engine, supporting multi-disk storage for Raft logs; when the disk where the home directory (`dir`) is located runs out of space, the Raft Engine automatically writes new logs to `spill-dir`, ensuring continuous operation of the system [#17356](https://github.com/tikv/tikv/issues/17356) [LykxSassinator](https://github.com/LykxSassinator) **tw@hfxsd** + PD From 21d5e7aabb86b3bbaa9149b01ffb2a36ea9864e4 Mon Sep 17 00:00:00 2001 From: Lilian Lee Date: Fri, 27 Sep 2024 18:38:18 +0800 Subject: [PATCH 36/45] Update rowspan to fix format --- releases/release-8.4.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-8.4.0.md b/releases/release-8.4.0.md index e1537d41c3aa..560ec256b873 100644 --- a/releases/release-8.4.0.md +++ b/releases/release-8.4.0.md @@ -23,7 +23,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- - Scalability and Performance + Scalability and Performance Instance-level execution plan cache (experimental)**tw@Oreoxmt 1569** Instance-level execution plan cache allows all sessions within the same TiDB instance to share the execution plan cache. It stores more execution plans in memory, eliminating SQL compilation time. This reduces SQL execution time, improves OLTP system performance and throughput, and provides better control over memory usage, enhancing database stability. From d16fab5bf777ca1f209c33ba63e82d3a9a5633d6 Mon Sep 17 00:00:00 2001 From: Lilian Lee Date: Fri, 27 Sep 2024 18:51:03 +0800 Subject: [PATCH 37/45] Add one highlight --- releases/release-8.4.0.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/releases/release-8.4.0.md b/releases/release-8.4.0.md index 560ec256b873..6a4be304b871 100644 --- a/releases/release-8.4.0.md +++ b/releases/release-8.4.0.md @@ -62,10 +62,14 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- Vector search is a search method based on data semantics, which provides more relevant search results. As one of the core functions of AI and large language models (LLMs), vector search can be used in various scenarios such as Retrieval-Augmented Generation (RAG), semantic search, and recommendation systems. - DB Operations and Observability + DB Operations and Observability Display TiKV and TiDB CPU times in memory tables**tw@hfxsd 1877** The CPU time is now integrated into a system table, displayed alongside other metrics for sessions or SQL, letting you observe high CPU consumption operations from multiple perspectives, and improves diagnostic efficiency. This is especially useful for diagnosing scenarios such as CPU spikes in instances or read/write hotspots in clusters. + + Support viewing aggregated TiKV CPU time by table or database **tw@lilin90 1878** + When hotspot issues are not caused by individual SQL statements, using the aggregated CPU time by table or database level in TOP SQL can help you quickly identify the tables or applications responsible for the hotspots, significantly improving the efficiency of diagnosing hotspot and CPU consumption issues. + Support backing up TiKV instances with IMDSv2 service enabled**tw@hfxsd 1945** AWS EC2 now uses IMDSv2 as the default metadata service. TiDB supports backing up data from TiKV instances that have IMDSv2 enabled, helping you run TiDB clusters more effectively in public cloud services. From 951671115bbf6155e3da310658c0be6e7ad8efe2 Mon Sep 17 00:00:00 2001 From: Aolin Date: Sun, 29 Sep 2024 12:30:32 +0800 Subject: [PATCH 38/45] add a contributor for vector search --- releases/release-8.4.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-8.4.0.md b/releases/release-8.4.0.md index 6a4be304b871..14cf1a349940 100644 --- a/releases/release-8.4.0.md +++ b/releases/release-8.4.0.md @@ -216,7 +216,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- ### SQL -* Support vector search (experimental) [#54245](https://github.com/pingcap/tidb/issues/54245) [#9032](https://github.com/pingcap/tiflash/issues/9032) @[breezewish](https://github.com/breezewish) @[Lloyd-Pottiger](https://github.com/Lloyd-Pottiger) @[EricZequan](https://github.com/EricZequan) @[zimulala](https://github.com/zimulala) @[JaySon-Huang](https://github.com/JaySon-Huang) @[winoros](https://github.com/winoros) **tw@qiancai** +* Support vector search (experimental) [#54245](https://github.com/pingcap/tidb/issues/54245) [#9032](https://github.com/pingcap/tiflash/issues/9032) @[breezewish](https://github.com/breezewish) @[Lloyd-Pottiger](https://github.com/Lloyd-Pottiger) @[EricZequan](https://github.com/EricZequan) @[zimulala](https://github.com/zimulala) @[JaySon-Huang](https://github.com/JaySon-Huang) @[winoros](https://github.com/winoros) @[wk989898](https://github.com/wk989898) **tw@qiancai** Vector search is a search method based on data semantics, which provides more relevant search results. As one of the core functions of AI and large language models (LLMs), vector search can be used in various scenarios such as Retrieval-Augmented Generation (RAG), semantic search, and recommendation systems. From afd91e45c8d6dcfe32f1909acb27cd6d19ee418b Mon Sep 17 00:00:00 2001 From: Aolin Date: Sun, 29 Sep 2024 16:05:15 +0800 Subject: [PATCH 39/45] align with Chinese Signed-off-by: Aolin --- releases/release-8.4.0.md | 20 +++++++++----------- 1 file changed, 9 insertions(+), 11 deletions(-) diff --git a/releases/release-8.4.0.md b/releases/release-8.4.0.md index 14cf1a349940..96827d50bd90 100644 --- a/releases/release-8.4.0.md +++ b/releases/release-8.4.0.md @@ -25,7 +25,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- Scalability and Performance Instance-level execution plan cache (experimental)**tw@Oreoxmt 1569** - Instance-level execution plan cache allows all sessions within the same TiDB instance to share the execution plan cache. It stores more execution plans in memory, eliminating SQL compilation time. This reduces SQL execution time, improves OLTP system performance and throughput, and provides better control over memory usage, enhancing database stability. + Instance-level execution plan cache allows all sessions within the same TiDB instance to share the execution plan cache. This feature reduces SQL compilation time by caching more execution plans in memory, decreasing overall SQL execution time. It improves OLTP system performance and throughput while providing better control over memory usage and enhancing database stability. Global indexes for partitioned tables (GA)**tw@hfxsd 1961** @@ -121,9 +121,9 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- For more information, see [documentation](/functions-and-operators/expressions-pushed-down.md). -* Instance-level execution plan cache (experimental) [#54057](https://github.com/pingcap/tidb/issues/54057) @[qw4990](https://github.com/qw4990) **tw@Oreoxmt** +* Support instance-level execution plan cache (experimental) [#54057](https://github.com/pingcap/tidb/issues/54057) @[qw4990](https://github.com/qw4990) **tw@Oreoxmt** - TiDB v8.4.0 introduces instance-level execution plan cache as an experimental feature. This feature allows all sessions within the same TiDB instance to share the execution plan cache, significantly reducing TiDB latency, improving cluster throughput, decreasing the likelihood of execution plan fluctuations, and maintaining stable cluster performance. Compared with session-level execution plan cache, instance-level execution plan cache offers the following advantages: + Instance-level execution plan cache allows all sessions within the same TiDB instance to share the execution plan cache. This feature significantly reduces TiDB query response time, increases cluster throughput, decreases the likelihood of execution plan mutations, and maintains stable cluster performance. Compared with session-level execution plan cache, instance-level execution plan cache offers the following advantages: - Eliminates redundancy, caching more execution plans with the same memory consumption. - Allocates a fixed-size memory on the instance, limiting memory usage more effectively. @@ -201,7 +201,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- * TiProxy supports traffic replay (experimental) [#642](https://github.com/pingcap/tiproxy/issues/642) @[djshow832](https://github.com/djshow832) **tw@Oreoxmt** - Starting from TiProxy v1.3.0, you can use TiProxy to capture access traffic in a TiDB production cluster and replay it in a test cluster at a specified rate. This feature enables you to reproduce actual workloads from the production cluster in a test environment, verifying SQL statement execution results and performance. + Starting from TiProxy v1.3.0, you can use `tiproxyctrl` to connect to the TiProxy instance and capture access traffic in a TiDB production cluster and replay it in a test cluster at a specified rate. This feature enables you to reproduce actual workloads from the production cluster in a test environment, verifying SQL statement execution results and performance. Traffic replay is suitable for the following scenarios: @@ -210,8 +210,6 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- - Validate performance before TiDB scaling - Test performance limits - You can use `tiproxyctrl` to connect to the TiProxy instance and perform traffic capture and replay. - For more information, see [documentation](/tiproxy/tiproxy-traffic-replay.md). ### SQL @@ -252,7 +250,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- * BR reduces privileges when restoring backup data in a cloud storage system [#55870](https://github.com/pingcap/tidb/issues/55870) @[Leavrth](https://github.com/Leavrth) **tw@Oreoxmt** - Before v8.4.0, BR stores checkpoint information about restore progress in the backup data location during restore. These checkpoints enable quick resumption of interrupted restores. Starting from v8.4.0, BR stores restore checkpoint information in the target TiDB cluster. This means that BR only requires read access to the backup directories. + Before v8.4.0, BR writes checkpoint information about restore progress to the backup storage system during restore. These checkpoints enable quick resumption of interrupted restores. Starting from v8.4.0, BR writes restore checkpoint information to the target TiDB cluster. This means that BR only requires read access to the backup directories during restore. For more information, see [documentation](/br/backup-and-restore-storages.md#authentication). @@ -294,7 +292,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- * TiCDC Claim-Check supports sending only the `value` field of Kafka messages to external storage [#11396](https://github.com/pingcap/tiflow/issues/11396) @[3AceShowHand](https://github.com/3AceShowHand) **tw@Oreoxmt** - Before v8.4.0, when using the Claim-Check feature to handle large messages (by setting `large-message-handle-option` to `claim-check`), TiCDC encodes and stores both the `key` and `value` fields in the external storage system. + Before v8.4.0, when the Claim-Check feature is enabled (by setting `large-message-handle-option` to `claim-check`), TiCDC encodes and stores both the `key` and `value` fields in the external storage system when handling large messages. Starting from v8.4.0, TiCDC supports sending only the `value` field of Kafka messages to external storage. This feature is only applicable to non-Open Protocol protocols. You can control this feature by setting the `claim-check-raw-value` parameter. @@ -302,7 +300,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- * TiCDC introduces Checksum V2 to verify old values after Add Column or Drop Column operations [#10969](https://github.com/pingcap/tiflow/issues/10969) @[3AceShowHand](https://github.com/3AceShowHand) **tw@Oreoxmt** - Starting from v8.4.0, TiDB and TiCDC introduce Checksum V2 to address issues with Checksum V1 in verifying old values in Update or Delete events after Add Column or Drop Column operations. For new clusters created in v8.4.0 or later, or clusters upgraded to v8.4.0, TiDB uses Checksum V2 by default when single-row data checksum verification is enabled. TiCDC supports handling both Checksum V1 and V2. This change only affects TiDB and TiCDC internal implementation and does not impact checksum calculation methods for downstream Kafka consumers. + Starting from v8.4.0, TiDB and TiCDC introduce Checksum V2 to address issues of Checksum V1 in verifying old values in Update or Delete events after `ADD COLUMN` or `DROP COLUMN` operations. For clusters created in v8.4.0 or later, or clusters upgraded to v8.4.0, TiDB uses Checksum V2 by default when single-row data checksum verification is enabled. TiCDC supports handling both Checksum V1 and V2. This change only affects TiDB and TiCDC internal implementation and does not affect checksum calculation methods for downstream Kafka consumers. For more information, see [documentation](/ticdc/ticdc-integrity-check.md). @@ -368,7 +366,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- - Optimize the execution performance of DML statements when the system variable `tidb_dml_type` is set to `"bulk"` [#50215](https://github.com/pingcap/tidb/issues/50215) @[ekexium](https://github.com/ekexium) **tw@qiancai** - Support using [Optimizer Fix Control 47400](/optimizer-fix-controls.md#47400-new-in-v840) to control whether the optimizer limits the minimum value estimated for `estRows` to `1`, which is consistent with databases such as Oracle and DB2 [#47400](https://github.com/pingcap/tidb/issues/47400) @[terry1purcell](https://github.com/terry1purcell) **tw@Oreoxmt** - Add write control to the [`mysql.tidb_runaway_queries`](/mysql-schema/mysql-schema.md#system-tables-related-to-runaway-queries) log table to reduce overhead caused by a large number of concurrent writes [#54434](https://github.com/pingcap/tidb/issues/54434) @[HuSharp](https://github.com/HuSharp) **tw@lilin90** - - Spport Index Join by default when the inner table has `Selection` or `Projection` operators on it [#issue号](链接) @[winoros](https://github.com/winoros) **tw@Oreoxmt** + - Spport Index Join by default when the inner table has `Selection`, `Projection` or `Aggregation` operators on it [#47233](https://github.com/pingcap/tidb/issues/47233) @[winoros](https://github.com/winoros) **tw@Oreoxmt** - Reduce the number of column details fetched from TiKV for `DELETE` operations in certain scenarios, lowering the resource overhead of these operations [#38911](https://github.com/pingcap/tidb/issues/38911) @[winoros](https://github.com/winoros) **tw@Oreoxmt** - Improve the efficiency of the priority queue for automatic statistics collection tasks [#49972](https://github.com/pingcap/tidb/issues/49972) @[Rustin170506](https://github.com/Rustin170506) **tw@Oreoxmt** - Improve automatic statistics collection by determining the collection concurrency based on node scale and hardware specifications [#53460](https://github.com/pingcap/tidb/issues/53460) @[hawkingrei](https://github.com/hawkingrei) **tw@Oreoxmt** @@ -376,7 +374,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- + TiKV - Increase the default value of Region from 96 MiB to 256 MiB to avoid the extra overhead caused by too many Regions [#17309](https://github.com/tikv/tikv/issues/17309) [LykxSassinator](https://github.com/LykxSassinator) **tw@hfxsd** - - Support setting memory usage limits for in-memory pessimistic locks in a Region or TiKV instance. To prevent CPU/IO overhead caused by pessimistic locks spilling to disk during write hotspots, you can increase the memory limit by modifying the configuration items [#17542](https://github.com/tikv/tikv/issues/17542) @[cfzjywxk](https://github.com/cfzjywxk) **tw@Oreoxmt** + - Support setting memory usage limits for in-memory pessimistic locks in a Region or TiKV instance. When hot write scenarios cause a large number of pessimistic locks, you can increase the memory limit by modifying the configuration. This helps avoid CPU and I/O overhead caused by pessimistic locks being written to disk. [#17542](https://github.com/tikv/tikv/issues/17542) @[cfzjywxk](https://github.com/cfzjywxk) **tw@Oreoxmt** - Introduce a new `spill-dir` configuration item in Raft Engine, supporting multi-disk storage for Raft logs; when the disk where the home directory (`dir`) is located runs out of space, the Raft Engine automatically writes new logs to `spill-dir`, ensuring continuous operation of the system [#17356](https://github.com/tikv/tikv/issues/17356) [LykxSassinator](https://github.com/LykxSassinator) **tw@hfxsd** + PD From 3f64d282a7527ff63cc02a19fb6fcd3f2f329ec9 Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Sun, 29 Sep 2024 17:04:52 +0800 Subject: [PATCH 40/45] Update releases/release-8.4.0.md --- releases/release-8.4.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-8.4.0.md b/releases/release-8.4.0.md index 96827d50bd90..90e0119b875f 100644 --- a/releases/release-8.4.0.md +++ b/releases/release-8.4.0.md @@ -110,7 +110,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- * Optimize the execution efficiency of the hash join operator for TiDB (experimental) [#55153](https://github.com/pingcap/tidb/issues/55153) [#53127](https://github.com/pingcap/tidb/issues/53127) @[windtalker](https://github.com/windtalker) @[xzhangxian1008](https://github.com/xzhangxian1008) @[XuHuaiyu](https://github.com/XuHuaiyu) @[wshwsh12](https://github.com/wshwsh12) **tw@qiancai** - In v8.4.0, TiDB introduces an optimized implementation of the hash join operator to improve its execution efficiency. Currently, this optimized implementation of hash join applies only to inner join and outer join operations and is disabled by default. To enable this optimized implementation, configure the [tidb_hash_join_version](/system-variables.md#tidb_hash_join_version-new-in-v840) system variable to `optimized`. + In v8.4.0, TiDB introduces an optimized version of the hash join operator to improve its execution efficiency. Currently, the optimized version of the hash join applies only to inner join and outer join operations and is disabled by default. To enable this optimized version, configure the [tidb_hash_join_version](/system-variables.md#tidb_hash_join_version-new-in-v840) system variable to `optimized`. For more information, see [documentation](/system-variables.md#tidb_hash_join_version-new-in-v840). From 7675436db25c56eede101178700d62cbefb855b6 Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Sun, 29 Sep 2024 18:11:52 +0800 Subject: [PATCH 41/45] Apply suggestions from code review --- releases/release-8.4.0.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/releases/release-8.4.0.md b/releases/release-8.4.0.md index 90e0119b875f..6bd1f9fa43b9 100644 --- a/releases/release-8.4.0.md +++ b/releases/release-8.4.0.md @@ -58,7 +58,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- SQL - Vector search (experimental)**tw@qiancai 1898** + Vector search (experimental) **tw@qiancai 1898** Vector search is a search method based on data semantics, which provides more relevant search results. As one of the core functions of AI and large language models (LLMs), vector search can be used in various scenarios such as Retrieval-Augmented Generation (RAG), semantic search, and recommendation systems. @@ -76,7 +76,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- Security - Client-side encryption of log backup data**tw@qiancai 1920** + Client-side encryption of log backup data (experimental) **tw@qiancai 1920** Before uploading log backup data to your backup storage, you can encrypt the backup data to ensure its security during storage and transmission. @@ -110,14 +110,16 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- * Optimize the execution efficiency of the hash join operator for TiDB (experimental) [#55153](https://github.com/pingcap/tidb/issues/55153) [#53127](https://github.com/pingcap/tidb/issues/53127) @[windtalker](https://github.com/windtalker) @[xzhangxian1008](https://github.com/xzhangxian1008) @[XuHuaiyu](https://github.com/XuHuaiyu) @[wshwsh12](https://github.com/wshwsh12) **tw@qiancai** - In v8.4.0, TiDB introduces an optimized version of the hash join operator to improve its execution efficiency. Currently, the optimized version of the hash join applies only to inner join and outer join operations and is disabled by default. To enable this optimized version, configure the [tidb_hash_join_version](/system-variables.md#tidb_hash_join_version-new-in-v840) system variable to `optimized`. + In v8.4.0, TiDB introduces an optimized version of the hash join operator to improve its execution efficiency. Currently, the optimized version of the hash join applies only to inner join and outer join operations and is disabled by default. To enable this optimized version, configure the [`tidb_hash_join_version`](/system-variables.md#tidb_hash_join_version-new-in-v840) system variable to `optimized`. For more information, see [documentation](/system-variables.md#tidb_hash_join_version-new-in-v840). -* Support pushing down the following string functions to TiKV [#17529](https://github.com/tikv/tikv/issues/17529) @[gengliqi](https://github.com/gengliqi) **tw@qiancai** +* Support pushing down the following date functions to TiKV [#17529](https://github.com/tikv/tikv/issues/17529) @[gengliqi](https://github.com/gengliqi) **tw@qiancai** * `DATE_ADD()` * `DATE_SUB()` + * `ADDDATE()` + * `SUBDATE()` For more information, see [documentation](/functions-and-operators/expressions-pushed-down.md). @@ -222,7 +224,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- To start vector search, you only need to create a table with vector data types, insert vector data, and then perform a query of vector data. You can also perform mixed queries of vector data and traditional relational data. - To enhance the performance of vector search, you can create and use [vector search indexes](vector-search-index.md). Note that TiDB vector search indexes rely on TiFlash. Therefore, before using vector search indexes, make sure that TiFlash nodes are deployed in your TiDB cluster. + To enhance the performance of vector search, you can create and use [vector search indexes](vector-search-index.md). Note that TiDB vector search indexes rely on TiFlash. Before using vector search indexes, make sure that TiFlash nodes are deployed in your TiDB cluster. For more information, see [documentation](/vector-search-overview.md). From 3b5e9ecb7ca234c950fedff9061b8fe72d77ba38 Mon Sep 17 00:00:00 2001 From: Aolin Date: Sun, 29 Sep 2024 19:01:04 +0800 Subject: [PATCH 42/45] Apply suggestions from code review Co-authored-by: Grace Cai --- releases/release-8.4.0.md | 27 +++++++++++++-------------- 1 file changed, 13 insertions(+), 14 deletions(-) diff --git a/releases/release-8.4.0.md b/releases/release-8.4.0.md index 6bd1f9fa43b9..915f06ef7774 100644 --- a/releases/release-8.4.0.md +++ b/releases/release-8.4.0.md @@ -25,7 +25,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- Scalability and Performance Instance-level execution plan cache (experimental)**tw@Oreoxmt 1569** - Instance-level execution plan cache allows all sessions within the same TiDB instance to share the execution plan cache. This feature reduces SQL compilation time by caching more execution plans in memory, decreasing overall SQL execution time. It improves OLTP system performance and throughput while providing better control over memory usage and enhancing database stability. + Instance-level execution plan cache allows all sessions within the same TiDB instance to share the execution plan cache. This feature reduces SQL compilation time by caching more execution plans in memory, decreasing overall SQL execution time. It improves OLTP performance and throughput while providing better control over memory usage and enhancing database stability. Global indexes for partitioned tables (GA)**tw@hfxsd 1961** @@ -50,7 +50,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- TiProxy supports traffic capture and replay (experimental)**tw@Oreoxmt 1942** - Use TiProxy to capture real workloads from TiDB production clusters before major operations like cluster upgrades, migrations, or deployment changes. Replay these workloads on target test clusters to validate performance and ensure successful changes. + Use TiProxy to capture real workloads from TiDB production clusters before major operations such as cluster upgrades, migrations, or deployment changes. Replay these workloads on target test clusters to validate performance and ensure successful changes. Adaptive concurrency for statistics collection**tw@Oreoxmt 1739** @@ -125,7 +125,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- * Support instance-level execution plan cache (experimental) [#54057](https://github.com/pingcap/tidb/issues/54057) @[qw4990](https://github.com/qw4990) **tw@Oreoxmt** - Instance-level execution plan cache allows all sessions within the same TiDB instance to share the execution plan cache. This feature significantly reduces TiDB query response time, increases cluster throughput, decreases the likelihood of execution plan mutations, and maintains stable cluster performance. Compared with session-level execution plan cache, instance-level execution plan cache offers the following advantages: + Instance-level execution plan cache allows all sessions within the same TiDB instance to share the execution plan cache. This feature significantly reduces TiDB query response time, increases cluster throughput, decreases the possibility of execution plan mutations, and maintains stable cluster performance. Compared with session-level execution plan cache, instance-level execution plan cache offers the following advantages: - Eliminates redundancy, caching more execution plans with the same memory consumption. - Allocates a fixed-size memory on the instance, limiting memory usage more effectively. @@ -203,13 +203,13 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- * TiProxy supports traffic replay (experimental) [#642](https://github.com/pingcap/tiproxy/issues/642) @[djshow832](https://github.com/djshow832) **tw@Oreoxmt** - Starting from TiProxy v1.3.0, you can use `tiproxyctrl` to connect to the TiProxy instance and capture access traffic in a TiDB production cluster and replay it in a test cluster at a specified rate. This feature enables you to reproduce actual workloads from the production cluster in a test environment, verifying SQL statement execution results and performance. + Starting from TiProxy v1.3.0, you can use `tiproxyctrl` to connect to the TiProxy instance, capture access traffic in a TiDB production cluster, and replay it in a test cluster at a specified rate. This feature enables you to reproduce actual workloads from the production cluster in a test environment, verifying SQL statement execution results and performance. - Traffic replay is suitable for the following scenarios: + Traffic replay is useful in the following scenarios: - Validate TiDB version upgrades - Assess change impact - - Validate performance before TiDB scaling + - Validate performance before scaling TiDB - Test performance limits For more information, see [documentation](/tiproxy/tiproxy-traffic-replay.md). @@ -250,9 +250,9 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- For more information, see [documentation](/br/br-pitr-manual.md#encrypt-log-backup-data). -* BR reduces privileges when restoring backup data in a cloud storage system [#55870](https://github.com/pingcap/tidb/issues/55870) @[Leavrth](https://github.com/Leavrth) **tw@Oreoxmt** +* BR requires fewer privileges when restoring backup data in a cloud storage system [#55870](https://github.com/pingcap/tidb/issues/55870) @[Leavrth](https://github.com/Leavrth) **tw@Oreoxmt** - Before v8.4.0, BR writes checkpoint information about restore progress to the backup storage system during restore. These checkpoints enable quick resumption of interrupted restores. Starting from v8.4.0, BR writes restore checkpoint information to the target TiDB cluster. This means that BR only requires read access to the backup directories during restore. + Before v8.4.0, BR writes checkpoint information about the restore progress to the backup storage system during restore. These checkpoints enable quick resumption of interrupted restores. Starting from v8.4.0, BR writes restore checkpoint information to the target TiDB cluster instead. This means that BR only requires read access to the backup directories during restore. For more information, see [documentation](/br/backup-and-restore-storages.md#authentication). @@ -302,7 +302,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- * TiCDC introduces Checksum V2 to verify old values after Add Column or Drop Column operations [#10969](https://github.com/pingcap/tiflow/issues/10969) @[3AceShowHand](https://github.com/3AceShowHand) **tw@Oreoxmt** - Starting from v8.4.0, TiDB and TiCDC introduce Checksum V2 to address issues of Checksum V1 in verifying old values in Update or Delete events after `ADD COLUMN` or `DROP COLUMN` operations. For clusters created in v8.4.0 or later, or clusters upgraded to v8.4.0, TiDB uses Checksum V2 by default when single-row data checksum verification is enabled. TiCDC supports handling both Checksum V1 and V2. This change only affects TiDB and TiCDC internal implementation and does not affect checksum calculation methods for downstream Kafka consumers. + Starting from v8.4.0, TiDB and TiCDC introduce the Checksum V2 algorithm to address issues of Checksum V1 in verifying old values in Update or Delete events after `ADD COLUMN` or `DROP COLUMN` operations. For clusters created in v8.4.0 or later, or clusters upgraded to v8.4.0, TiDB uses Checksum V2 by default when single-row data checksum verification is enabled. TiCDC supports handling both Checksum V1 and V2. This change only affects TiDB and TiCDC internal implementation and does not affect checksum calculation methods for downstream Kafka consumers. For more information, see [documentation](/ticdc/ticdc-integrity-check.md). @@ -368,16 +368,15 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- - Optimize the execution performance of DML statements when the system variable `tidb_dml_type` is set to `"bulk"` [#50215](https://github.com/pingcap/tidb/issues/50215) @[ekexium](https://github.com/ekexium) **tw@qiancai** - Support using [Optimizer Fix Control 47400](/optimizer-fix-controls.md#47400-new-in-v840) to control whether the optimizer limits the minimum value estimated for `estRows` to `1`, which is consistent with databases such as Oracle and DB2 [#47400](https://github.com/pingcap/tidb/issues/47400) @[terry1purcell](https://github.com/terry1purcell) **tw@Oreoxmt** - Add write control to the [`mysql.tidb_runaway_queries`](/mysql-schema/mysql-schema.md#system-tables-related-to-runaway-queries) log table to reduce overhead caused by a large number of concurrent writes [#54434](https://github.com/pingcap/tidb/issues/54434) @[HuSharp](https://github.com/HuSharp) **tw@lilin90** - - Spport Index Join by default when the inner table has `Selection`, `Projection` or `Aggregation` operators on it [#47233](https://github.com/pingcap/tidb/issues/47233) @[winoros](https://github.com/winoros) **tw@Oreoxmt** + - Support Index Join by default when the inner table has `Selection`, `Projection`, or `Aggregation` operators on it [#47233](https://github.com/pingcap/tidb/issues/47233) @[winoros](https://github.com/winoros) **tw@Oreoxmt** - Reduce the number of column details fetched from TiKV for `DELETE` operations in certain scenarios, lowering the resource overhead of these operations [#38911](https://github.com/pingcap/tidb/issues/38911) @[winoros](https://github.com/winoros) **tw@Oreoxmt** - - Improve the efficiency of the priority queue for automatic statistics collection tasks [#49972](https://github.com/pingcap/tidb/issues/49972) @[Rustin170506](https://github.com/Rustin170506) **tw@Oreoxmt** - Improve automatic statistics collection by determining the collection concurrency based on node scale and hardware specifications [#53460](https://github.com/pingcap/tidb/issues/53460) @[hawkingrei](https://github.com/hawkingrei) **tw@Oreoxmt** + TiKV - - Increase the default value of Region from 96 MiB to 256 MiB to avoid the extra overhead caused by too many Regions [#17309](https://github.com/tikv/tikv/issues/17309) [LykxSassinator](https://github.com/LykxSassinator) **tw@hfxsd** - - Support setting memory usage limits for in-memory pessimistic locks in a Region or TiKV instance. When hot write scenarios cause a large number of pessimistic locks, you can increase the memory limit by modifying the configuration. This helps avoid CPU and I/O overhead caused by pessimistic locks being written to disk. [#17542](https://github.com/tikv/tikv/issues/17542) @[cfzjywxk](https://github.com/cfzjywxk) **tw@Oreoxmt** - - Introduce a new `spill-dir` configuration item in Raft Engine, supporting multi-disk storage for Raft logs; when the disk where the home directory (`dir`) is located runs out of space, the Raft Engine automatically writes new logs to `spill-dir`, ensuring continuous operation of the system [#17356](https://github.com/tikv/tikv/issues/17356) [LykxSassinator](https://github.com/LykxSassinator) **tw@hfxsd** + - Increase the default value of Region from 96 MiB to 256 MiB to avoid the extra overhead caused by too many Regions [#17309](https://github.com/tikv/tikv/issues/17309) [LykxSassinator](https://github.com/LykxSassinator) **tw@hfxsd** + - Support setting memory usage limits for in-memory pessimistic locks in a Region or TiKV instance. When hot write scenarios cause a large number of pessimistic locks, you can increase the memory limits via configuration. This helps avoid CPU and I/O overhead caused by pessimistic locks being written to disk. [#17542](https://github.com/tikv/tikv/issues/17542) @[cfzjywxk](https://github.com/cfzjywxk) **tw@Oreoxmt** + - Introduce a new `spill-dir` configuration item in Raft Engine, supporting multi-disk storage for Raft logs; when the disk where the home directory (`dir`) is located runs out of space, the Raft Engine automatically writes new logs to `spill-dir`, ensuring continuous operation of the system [#17356](https://github.com/tikv/tikv/issues/17356) [LykxSassinator](https://github.com/LykxSassinator) **tw@hfxsd** + PD From d60d449b929595715e59badfca1129204ef5c9d6 Mon Sep 17 00:00:00 2001 From: Aolin Date: Mon, 30 Sep 2024 13:14:32 +0800 Subject: [PATCH 43/45] fix Checksum V2 --- releases/release-8.4.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-8.4.0.md b/releases/release-8.4.0.md index 915f06ef7774..82a0eecfe8ad 100644 --- a/releases/release-8.4.0.md +++ b/releases/release-8.4.0.md @@ -300,7 +300,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- For more information, see [documentation](/ticdc/ticdc-sink-to-kafka.md#send-the-value-field-to-external-storage-only). -* TiCDC introduces Checksum V2 to verify old values after Add Column or Drop Column operations [#10969](https://github.com/pingcap/tiflow/issues/10969) @[3AceShowHand](https://github.com/3AceShowHand) **tw@Oreoxmt** +* TiCDC introduces Checksum V2 to verify old values in Update or Delete events [#10969](https://github.com/pingcap/tiflow/issues/10969) @[3AceShowHand](https://github.com/3AceShowHand) **tw@Oreoxmt** Starting from v8.4.0, TiDB and TiCDC introduce the Checksum V2 algorithm to address issues of Checksum V1 in verifying old values in Update or Delete events after `ADD COLUMN` or `DROP COLUMN` operations. For clusters created in v8.4.0 or later, or clusters upgraded to v8.4.0, TiDB uses Checksum V2 by default when single-row data checksum verification is enabled. TiCDC supports handling both Checksum V1 and V2. This change only affects TiDB and TiCDC internal implementation and does not affect checksum calculation methods for downstream Kafka consumers. From e1d212985d06a65772700b77cc4feeacaa62a7d3 Mon Sep 17 00:00:00 2001 From: Aolin Date: Mon, 30 Sep 2024 13:38:38 +0800 Subject: [PATCH 44/45] Apply suggestions from code review --- releases/release-8.4.0.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/releases/release-8.4.0.md b/releases/release-8.4.0.md index 82a0eecfe8ad..bc2ba935275e 100644 --- a/releases/release-8.4.0.md +++ b/releases/release-8.4.0.md @@ -53,8 +53,8 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- Use TiProxy to capture real workloads from TiDB production clusters before major operations such as cluster upgrades, migrations, or deployment changes. Replay these workloads on target test clusters to validate performance and ensure successful changes. - Adaptive concurrency for statistics collection**tw@Oreoxmt 1739** - Automatic statistics collection determines the collection concurrency based on node scale and hardware specifications. This improves statistics collection efficiency, reduces manual tuning, and ensures stable cluster performance. + Concurrent automatic statistics collection**tw@Oreoxmt 1739** + You can set the concurrency within a single automatic statistics collection task using the system variable tidb_auto_analyze_concurrency. TiDB automatically determines the concurrency of scanning tasks based on node scale and hardware specifications. This improves statistics collection efficiency by fully utilizing system resources, reduces manual tuning, and ensures stable cluster performance. SQL @@ -370,7 +370,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- - Add write control to the [`mysql.tidb_runaway_queries`](/mysql-schema/mysql-schema.md#system-tables-related-to-runaway-queries) log table to reduce overhead caused by a large number of concurrent writes [#54434](https://github.com/pingcap/tidb/issues/54434) @[HuSharp](https://github.com/HuSharp) **tw@lilin90** - Support Index Join by default when the inner table has `Selection`, `Projection`, or `Aggregation` operators on it [#47233](https://github.com/pingcap/tidb/issues/47233) @[winoros](https://github.com/winoros) **tw@Oreoxmt** - Reduce the number of column details fetched from TiKV for `DELETE` operations in certain scenarios, lowering the resource overhead of these operations [#38911](https://github.com/pingcap/tidb/issues/38911) @[winoros](https://github.com/winoros) **tw@Oreoxmt** - - Improve automatic statistics collection by determining the collection concurrency based on node scale and hardware specifications [#53460](https://github.com/pingcap/tidb/issues/53460) @[hawkingrei](https://github.com/hawkingrei) **tw@Oreoxmt** + - Support setting the concurrency within a single automatic statistics collection task using the system variable `tidb_auto_analyze_concurrency` [#53460](https://github.com/pingcap/tidb/issues/53460) @[hawkingrei](https://github.com/hawkingrei) **tw@Oreoxmt** + TiKV From 3653bd5a70a063877b40f7109b4a01ea8209c4d8 Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Mon, 30 Sep 2024 14:56:30 +0800 Subject: [PATCH 45/45] Apply suggestions from code review Co-authored-by: Aolin --- releases/release-8.4.0.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/releases/release-8.4.0.md b/releases/release-8.4.0.md index bc2ba935275e..d6a303c0afe3 100644 --- a/releases/release-8.4.0.md +++ b/releases/release-8.4.0.md @@ -220,7 +220,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- Vector search is a search method based on data semantics, which provides more relevant search results. As one of the core functions of AI and large language models (LLMs), vector search can be used in various scenarios such as Retrieval-Augmented Generation (RAG), semantic search, and recommendation systems. - Starting from v8.4.0, TiDB supports [vector data types](vector-search-data-types.md) and [vector search indexes](vector-search-index.md), offering powerful vector search capabilities. TiDB vector data types support up to 16,383 dimensions and support various [distance functions](/vector-search-functions-and-operators.md#vector-functions), including L2 distance (Euclidean distance), cosine distance, negative inner product, and L1 distance (Manhattan distance). + Starting from v8.4.0, TiDB supports [vector data types](/vector-search-data-types.md) and [vector search indexes](/vector-search-index.md), offering powerful vector search capabilities. TiDB vector data types support up to 16,383 dimensions and support various [distance functions](/vector-search-functions-and-operators.md#vector-functions), including L2 distance (Euclidean distance), cosine distance, negative inner product, and L1 distance (Manhattan distance). To start vector search, you only need to create a table with vector data types, insert vector data, and then perform a query of vector data. You can also perform mixed queries of vector data and traditional relational data. @@ -240,7 +240,7 @@ Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.4/quick-start-with- ### DB operations -* BR supports client-side encryption of log backup data (experimental) [55834](https://github.com/pingcap/tidb/issues/55834) @[Tristan1900](https://github.com/Tristan1900) **tw@qiancai** +* BR supports client-side encryption of log backup data (experimental) [#55834](https://github.com/pingcap/tidb/issues/55834) @[Tristan1900](https://github.com/Tristan1900) **tw@qiancai** In earlier TiDB versions, only snapshot backup data can be encrypted on the client side. Starting from v8.4.0, log backup data can also be encrypted on the client side. Before uploading log backup data to your backup storage, you can encrypt the backup data to ensure its security via one of the following methods: