TiDB vector search doc #18502
Open
TiDB vector search doc #18502
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: “EricZequan” <[email protected]>
ti-chi-bot
bot
added
the
first-time-contributor
|
|
ti-chi-bot
bot
added
missing-translation-status
Signed-off-by: “EricZequan” <[email protected]>
qiancai
added
v8.4
ti-chi-bot
bot
removed
the
missing-translation-status
|
/cc @breezewish |
Signed-off-by: “EricZequan” <[email protected]>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Signed-off-by: “EricZequan” <[email protected]>
breezewish
reviewed
Sep 2, 2024
Signed-off-by: “EricZequan” <[email protected]>
breezewish
reviewed
Sep 3, 2024
Signed-off-by: “EricZequan” <[email protected]>
breezewish
reviewed
Sep 3, 2024
breezewish
reviewed
Sep 3, 2024
breezewish
reviewed
Sep 3, 2024
breezewish
reviewed
Sep 3, 2024
breezewish
reviewed
Sep 3, 2024
breezewish
reviewed
Sep 3, 2024
breezewish
reviewed
Sep 3, 2024
breezewish
reviewed
Sep 3, 2024
breezewish
reviewed
Sep 3, 2024
breezewish
reviewed
Sep 3, 2024
hfxsd
reviewed
Sep 24, 2024
hfxsd
reviewed
Sep 24, 2024
hfxsd
reviewed
Sep 24, 2024
hfxsd
reviewed
Sep 24, 2024
hfxsd
reviewed
Sep 24, 2024
hfxsd
reviewed
Sep 24, 2024
hfxsd
reviewed
Sep 24, 2024
hfxsd
reviewed
Sep 24, 2024
Signed-off-by: “EricZequan” <[email protected]>
Signed-off-by: “EricZequan” <[email protected]>
Signed-off-by: JaySon-Huang <[email protected]>
Comment on lines
+122
to
+124
| ## 从 v7.x 升级至 v8.4 或以上版本 | ||
|
|
||
| 从 v8.4 开始，为了支持[向量搜索功能](/vector-search-index.md)，TiFlash 底层存储格式发生改动。因此，升级 TiFlash 到 v8.4 或以上版本后，不支持原地降级到之前的版本。 |
There was a problem hiding this comment.
FYI, I've pushed a commit about the tiflash upgrade notice
Signed-off-by: JaySon-Huang <[email protected]>
Signed-off-by: JaySon-Huang <[email protected]>
Comment on lines
+63
to
+65
| + 向量类型数据同步 | ||
|
|
||
| - DM 不支持 MySQL 9.0 的向量数据类型同步。 |
There was a problem hiding this comment.
Add limitation about DM
| - 创建[向量搜索索引](/vector-search-index.md)时的限制，参照[向量搜索索引 - 使用限制](/vector-search-index.md#使用限制)。 | ||
| - 向量数据类型不支持存储双精度浮点数（该功能计划在未来的版本中支持）。当向 TiDB 中的向量列插入或存储双精度浮点数时，TiDB 会将这些双精度浮点数自动转换为单精度浮点数。 | ||
| - 确保使用 BR v8.4.0 及以上版本进行备份与恢复。不支持将带有向量数据类型的恢复至 v8.4.0 之前的 TiDB 集群。 | ||
| - DM 不支持 MySQL 9.0 的向量数据类型同步。 |
| @@ -120,6 +120,7 @@ TiDB 支持将数据备份到 Amazon S3、Google Cloud Storage (GCS)、Azure Blo | |||
| | 全局临时表 | | 确保使用 BR v5.3.0 及以上版本进行备份和恢复，否则会导致全局临时表的表定义错误。 | | |||
| | TiDB Lightning 物理导入模式| |上游数据库使用 TiDB Lightning 物理导入模式导入的数据，无法作为数据日志备份下来。推荐在数据导入后执行一次全量备份，细节参考[上游数据库使用 TiDB Lightning 物理导入模式导入数据的恢复](/faq/backup-and-restore-faq.md#上游数据库使用-tidb-lightning-物理导入模式导入数据时为什么无法使用日志备份功能)。| | |||
| | TiCDC | | BR v8.2.0 及以上版本：如果在恢复的目标集群有 [CheckpointTS](/ticdc/ticdc-architecture.md#checkpointts) 早于 BackupTS 的 Changefeed，BR 会拒绝执行恢复。BR v8.2.0 之前的版本：如果在恢复的目标集群有任何活跃的 TiCDC Changefeed，BR 会拒绝执行恢复。 | | |||
| | 向量搜索 | | 确保使用 BR v8.4.0 及以上版本进行备份与恢复。不支持将带有向量数据类型的恢复至 v8.4.0 之前的 TiDB 集群。 | | |||
There was a problem hiding this comment.
Add limitation about BR
| - 向量数据中不支持 `NaN`、`Infinity` 和 `-Infinity` 浮点数。 | ||
| - 创建[向量搜索索引](/vector-search-index.md)时的限制，参照[向量搜索索引 - 使用限制](/vector-search-index.md#使用限制)。 | ||
| - 向量数据类型不支持存储双精度浮点数（该功能计划在未来的版本中支持）。当向 TiDB 中的向量列插入或存储双精度浮点数时，TiDB 会将这些双精度浮点数自动转换为单精度浮点数。 | ||
| - 确保使用 BR v8.4.0 及以上版本进行备份与恢复。不支持将带有向量数据类型的恢复至 v8.4.0 之前的 TiDB 集群。 |
16 tasks
Signed-off-by: JaySon-Huang <[email protected]>
hfxsd
reviewed
Sep 30, 2024
Signed-off-by: “EricZequan” <[email protected]>
|
@EricZequan: The following test failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
hfxsd
reviewed
Sep 30, 2024
|
|
||
| ALTER TABLE foo ADD VECTOR INDEX idx_name ((VEC_COSINE_DISTANCE(data))) USING HNSW; | ||
| ``` | ||
|
|
There was a problem hiding this comment.
这注意建议删掉，在特性改为 GA 时很容易把此处漏改。
hfxsd
reviewed
Sep 30, 2024
|
|
||
| 更多信息，请参阅 [`ALTER TABLE ... COMPACT`](/sql-statements/sql-statement-alter-table-compact.md)。 | ||
|
|
||
| 此外，你也可以通过 `ADMIN SHOW DDL JOBS;` 查看 DDL 任务的执行进度，观察其 `row count`。不过这种方式并不准确，`row count` 的值是从 `TIFLASH_INDEXES` 里的 `rows_stable_indexed` 获取的。此方式也可作为你查看索引构建进度的一种参考方式。 |
There was a problem hiding this comment.
Suggested change
| 此外，你也可以通过 `ADMIN SHOW DDL JOBS;` 查看 DDL 任务的执行进度，观察其 `row count`。不过这种方式并不准确，`row count` 的值是从 `TIFLASH_INDEXES` 里的 `rows_stable_indexed` 获取的。此方式也可作为你查看索引构建进度的一种参考方式。 | |
| 此外，你也可以通过 `ADMIN SHOW DDL JOBS;` 查看 DDL 任务的执行进度，观察其 `row count`。不过这种方式并不准确，`row count` 的值是从 `TIFLASH_INDEXES` 里的 `rows_stable_indexed` 获取的。你也可以使用此方式查看索引构建进度。 |
qiancai
reviewed
Sep 30, 2024
Comment on lines
+156
to
+159
| - `CAST(... AS VECTOR)`: 将字符串类型转换为向量类型 | ||
| - `CAST(... AS CHAR)`: 将向量类型转换为字符串类型 | ||
| - `VEC_FROM_TEXT`: 将字符串类型转换为向量类型 | ||
| - `VEC_AS_TEXT`: 将向量类型转换为字符串类型 |
There was a problem hiding this comment.
Suggested change
| - `CAST(... AS VECTOR)`: 将字符串类型转换为向量类型 | |
| - `CAST(... AS CHAR)`: 将向量类型转换为字符串类型 | |
| - `VEC_FROM_TEXT`: 将字符串类型转换为向量类型 | |
| - `VEC_AS_TEXT`: 将向量类型转换为字符串类型 | |
| - `CAST(... AS VECTOR)`：将字符串类型转换为向量类型 | |
| - `CAST(... AS CHAR)`：将向量类型转换为字符串类型 | |
| - `VEC_FROM_TEXT`：将字符串类型转换为向量类型 | |
| - `VEC_AS_TEXT`：将向量类型转换为字符串类型 |
| - 不支持在同一列上创建多个使用了相同距离函数的向量搜索索引。 | ||
| - 不支持删除具有向量搜索索引的列，也不支持在同一个 SQL 语句中创建多个索引。 | ||
| - 不支持修改带有向量索引的列的类型（有损变更，即修改了列数据）。 | ||
| - 不支持将向量搜索索引[设置为不可见](/sql-statements/sql-statement-alter-index.md)。 |
There was a problem hiding this comment.
Suggested change
| - 不支持将向量搜索索引[设置为不可见](/sql-statements/sql-statement-alter-index.md)。 | |
| - 不支持将向量搜索索引[设置为不可见](/sql-statements/sql-statement-alter-index.md)（该功能计划在未来的版本中支持）。 | |
| - 不支持在分区表中添加向量索引（该功能计划在未来的版本中支持）。 |
| > 向量搜索目前为实验特性，不建议在生产环境中使用。该功能可能会在未事先通知的情况下发生变化或删除。如果发现 bug，请在 GitHub 上提 [issue](https://github.com/pingcap/tidb/issues) 反馈。 | ||
|
|
||
| - 向量最大支持 16383 维。 | ||
| - 向量数据中不支持 `NaN`、`Infinity` 和 `-Infinity` 浮点数。 |
There was a problem hiding this comment.
Suggested change
| - 向量数据中不支持 `NaN`、`Infinity` 和 `-Infinity` 浮点数。 | |
| - 向量数据中不支持 `NaN`、`Infinity` 和 `-Infinity` 浮点数。 | |
| - 向量列不能作为主键或者主键的一部分 | |
| - 向量列不能作为唯一索引或者唯一索引的一部分 | |
| - 向量列不能作为分区键或者分区键的一部分 |
| - 向量搜索索引只能基于单一的向量列创建，不能与其他列（如整数列或字符串列）组合形成复合索引。 | ||
| - 创建和使用搜索向量索引时需要指定距离函数。目前只支持余弦距离函数 `VEC_COSINE_DISTANCE()` 和 L2 距离函数 `VEC_L2_DISTANCE()`。 | ||
| - 不支持在同一列上创建多个使用了相同距离函数的向量搜索索引。 | ||
| - 不支持删除具有向量搜索索引的列，也不支持在同一个 SQL 语句中创建多个索引。 |
There was a problem hiding this comment.
Suggested change
| - 不支持删除具有向量搜索索引的列，也不支持在同一个 SQL 语句中创建多个索引。 | |
| - 不支持直接删除具有向量搜索索引的列。可以通过先删除列上的向量搜索索引，再删除列的方式完成删除。 | |
| - 不支持在同一个 SQL 语句中创建多个索引。 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
First-time contributors' checklist
What is changed, added or deleted? (Required)
Which TiDB version(s) do your changes apply to? (Required)
Tips for choosing the affected version(s):
By default, CHOOSE MASTER ONLY so your changes will be applied to the next TiDB major or minor releases. If your PR involves a product feature behavior change or a compatibility change, CHOOSE THE AFFECTED RELEASE BRANCH(ES) AND MASTER.
For details, see tips for choosing the affected versions (in Chinese).
What is the related PR or file link(s)?
Do your changes match any of the following descriptions?