Storage: Support multiple vec indexes on the same column

[Lloyd-Pottiger]

What problem does this PR solve?

Issue Number: ref #9032

Problem Summary:

What is changed and how it works?

• For query, add index_id in ANNQueryInfo (DMFileWithVectorIndexBlockInputStream.cpp)
  • If the index_id > 0, then try to query with the vector index defined by index_id
  • If the index_id <= 0, then try to query with the vector index defined in column
• For write, turn ColumnStat.vector_index from std::optional<dtpb::VectorIndexFileProps> vector_index to std::vector<dtpb::VectorIndexFileProps>
  • DMFileIndexWriter.cpp supports saving multiple vector index with different index_id
  • Save the size of index into each dtpb::VectorIndexFileProps instead of ColumnStat.index_bytes
  • The vector index filename is defined as "idx_${index_id}.vector"
• Ensure the atomic between reading vector index from DMFile and bumping meta version of DMFile

Storage: Support multiple vec indexes on the same column

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

None

[JinheLin]

If the index_id > 0, then try to query with the vector index defined by index_id
If the index_id <= 0, then try to query with the vector index defined in column

When would index_id <= 0? Why not make index_id always greater than 0?

[JaySon-Huang]

If the index_id > 0, then try to query with the vector index defined by index_id
If the index_id <= 0, then try to query with the vector index defined in column

When would index_id <= 0? Why not make index_id always greater than 0?

In the beta implementation Vector Search Index(Beta), we support adding vector index by comment. That kind of vector index is created along with column_id and no index_id. For backward compatibility, we define the index_id==0 for those indexes. And in the cse branch, those indexes still workable for a period of time.

[Lloyd-Pottiger]

If the index_id > 0, then try to query with the vector index defined by index_id
If the index_id <= 0, then try to query with the vector index defined in column

When would index_id <= 0? Why not make index_id always greater than 0?

https://github.com/pingcap/tidb/blob/9dff38ba98405422cb0eb15993f385efe9068b47/pkg/meta/model/index.go#L26 Follow tidb.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: JaySon-Huang, JinheLin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [JaySon-Huang,JinheLin]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2024-09-26 07:01:14.514491868 +0000 UTC m=+1722144.254915806: ☑️ agreed by JaySon-Huang.
• 2024-09-26 07:56:53.747144389 +0000 UTC m=+1725483.487568331: ☑️ agreed by JinheLin.