*: support vector search #9486
Merged
*: support vector search #9486
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ref #9032 Signed-off-by: Lloyd-Pottiger <[email protected]> Co-authored-by: JaySon-Huang <[email protected]> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ref #9032 Co-authored-by: JaySon-Huang <[email protected]> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ref #9032 storage: Use mmap to view vector index Signed-off-by: Lloyd-Pottiger <[email protected]> Co-authored-by: JaySon-Huang <[email protected]>
ref #9032 storage: Add vector search metrics Signed-off-by: Wish <[email protected]> Co-authored-by: Wenxuan <[email protected]>
ref #9032 *: use SimSIMD for vectors Signed-off-by: Lloyd-Pottiger <[email protected]>
ref #9032 storage: Add system.dt_local_indexes Signed-off-by: Lloyd-Pottiger <[email protected]>
ref #9032 DMFile: Support modify DMFile meta --------- Signed-off-by: Wish <[email protected]> Signed-off-by: Lloyd-Pottiger <[email protected]> Co-authored-by: Wenxuan <[email protected]> Co-authored-by: JaySon <[email protected]>
ref #9032 storage: Force evict when downloading vector index files Signed-off-by: Wish <[email protected]> Signed-off-by: Lloyd-Pottiger <[email protected]> Co-authored-by: Wenxuan <[email protected]>
ref #9032 storage: add local indexer scheduler Signed-off-by: Lloyd-Pottiger <[email protected]> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ref #9032 storage: Support adding vector index in background Signed-off-by: Lloyd-Pottiger <[email protected]>
close #9032 storage: Abort vector index building as soon as possible Signed-off-by: Wish <[email protected]> Signed-off-by: Lloyd-Pottiger <[email protected]> Co-authored-by: Wenxuan <[email protected]> Co-authored-by: Lloyd-Pottiger <[email protected]>
ref #9032 ddl: Support parsing VectorIndex defined in IndexInfo Co-authored-by: JaySon <[email protected]>
ref #9032 storage: support the HTTP API of sync table schema Signed-off-by: Lloyd-Pottiger <[email protected]> Co-authored-by: Lynn <[email protected]> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ref #9032 storage: cache PK column in memory Signed-off-by: Lloyd-Pottiger <[email protected]> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
…yncTableSchema (#9451) ref #9032 *: support vector index and adding/dropping vector index when doing syncTableSchema Signed-off-by: Lloyd-Pottiger <[email protected]> Co-authored-by: JaySon <[email protected]>
ref #9032 ddl: Adapt with the latest vector index def
ref #9032 Storage: Support multiple vec indexes on the same column Signed-off-by: Lloyd-Pottiger <[email protected]> Co-authored-by: JaySon <[email protected]> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
…is dropped (#9475) ### What problem does this PR solve? Issue Number: ref #9032 Problem Summary: ### What is changed and how it works? Pick tidbcloud/tiflash-cse#283 and tidbcloud/tiflash-cse#300 * Unify the logic of `generateLocalIndexInfos` and `initLocalIndexInfos` * Print 1 logging for the vector index added/dropped/existing in one table instead. This can avoid the flood of logging when tiflash restart with lots of table with vector index defined * Support drop the vector index defined on ColumnInfo after the Column has been dropped in TiDB * Add more ut in the DeltaMergeStore read level * vector search fallback when top_k = max uint32 ```commit-message ``` ### Check List Tests <!-- At least one of them must be included. --> - [ ] Unit test - [ ] Integration test - [ ] Manual test (add detailed scripts or steps below) - [ ] No code Side effects - [ ] Performance regression: Consumes more CPU - [ ] Performance regression: Consumes more Memory - [ ] Breaking backward compatibility Documentation - [ ] Affects user behaviors - [ ] Contains syntax changes - [ ] Contains variable changes - [ ] Contains experimental features - [ ] Changes MySQL compatibility ### Release note <!-- bugfix or new feature needs a release note --> ```release-note None ``` --------- Signed-off-by: Lloyd-Pottiger <[email protected]> Co-authored-by: JaySon <[email protected]> Co-authored-by: jinhelin <[email protected]>
ti-chi-bot
bot
added
size/XXL
|
PR needs rebase. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
ti-chi-bot
bot
added
the
do-not-merge/work-in-progress
|
PR needs rebase. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
) close #9485 vector: Fix ColumnArray does not work well with CHBlockChunkCodec Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Storage: Add error message when fail to build local index
storage: remove vector_index in column level
ti-chi-bot
bot
removed
the
do-not-merge/work-in-progress
JaySon-Huang
removed
the
needs-rebase
There was a problem hiding this comment.
temporary disable clang-tidy for this PR because too many file changes and it takes too long
ti-chi-bot
bot
added
approved
needs-1-more-lgtm
zanmato1984
approved these changes
Sep 30, 2024
ti-chi-bot
bot
added
lgtm
and removed
needs-1-more-lgtm
[LGTM Timeline notifier]Timeline:
|
zimulala
approved these changes
Sep 30, 2024
|
@zimulala: adding LGTM is restricted to approvers and reviewers in OWNERS files. In response to this: Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: JaySon-Huang, zanmato1984, zimulala The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
This was referenced Sep 30, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #9032
Problem Summary:
What is changed and how it works?
Check List
Tests
Side effects
Documentation
Release note