ref #9032

Signed-off-by: Lloyd-Pottiger <[email protected]>

Co-authored-by: JaySon-Huang <[email protected]>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

ref #9032

ref #9032

Co-authored-by: JaySon-Huang <[email protected]>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

ref #9032

storage: Use mmap to view vector index

Signed-off-by: Lloyd-Pottiger <[email protected]>

Co-authored-by: JaySon-Huang <[email protected]>

ref #9032

storage: Add vector search metrics

Signed-off-by: Wish <[email protected]>

Co-authored-by: Wenxuan <[email protected]>

ref #9032

*: use SimSIMD for vectors

Signed-off-by: Lloyd-Pottiger <[email protected]>

ref #9032

storage: Add system.dt_local_indexes

Signed-off-by: Lloyd-Pottiger <[email protected]>

…or-index

ref #9032

DMFile: Support modify DMFile meta

---------

Signed-off-by: Wish <[email protected]>
Signed-off-by: Lloyd-Pottiger <[email protected]>
Co-authored-by: Wenxuan <[email protected]>
Co-authored-by: JaySon <[email protected]>

ref #9032

storage: Force evict when downloading vector index files

Signed-off-by: Wish <[email protected]>
Signed-off-by: Lloyd-Pottiger <[email protected]>

Co-authored-by: Wenxuan <[email protected]>

ref #9032

storage: add local indexer scheduler

Signed-off-by: Lloyd-Pottiger <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

ref #9032

storage: Support adding vector index in background

Signed-off-by: Lloyd-Pottiger <[email protected]>

close #9032

storage: Abort vector index building as soon as possible

Signed-off-by: Wish <[email protected]>
Signed-off-by: Lloyd-Pottiger <[email protected]>

Co-authored-by: Wenxuan <[email protected]>
Co-authored-by: Lloyd-Pottiger <[email protected]>

ref #9032

ddl: Support parsing VectorIndex defined in IndexInfo

Co-authored-by: JaySon <[email protected]>

ref #9032

storage: support the HTTP API of sync table schema

Signed-off-by: Lloyd-Pottiger <[email protected]>

Co-authored-by: Lynn <[email protected]>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

ref #9032

storage: cache PK column in memory

Signed-off-by: Lloyd-Pottiger <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

…yncTableSchema (#9451)

ref #9032

*: support vector index and adding/dropping vector index when doing syncTableSchema

Signed-off-by: Lloyd-Pottiger <[email protected]>

Co-authored-by: JaySon <[email protected]>

ref #9032

ddl: Adapt with the latest vector index def

ref #9032

Storage: Support multiple vec indexes on the same column

Signed-off-by: Lloyd-Pottiger <[email protected]>

Co-authored-by: JaySon <[email protected]>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

…is dropped (#9475)

### What problem does this PR solve?

Issue Number: ref #9032

Problem Summary:

### What is changed and how it works?

Pick tidbcloud/tiflash-cse#283 and
tidbcloud/tiflash-cse#300

* Unify the logic of `generateLocalIndexInfos` and `initLocalIndexInfos`
* Print 1 logging for the vector index added/dropped/existing in one
table instead. This can avoid the flood of logging when tiflash restart
with lots of table with vector index defined
* Support drop the vector index defined on ColumnInfo after the Column
has been dropped in TiDB
* Add more ut in the DeltaMergeStore read level
* vector search fallback when top_k = max uint32

```commit-message

```

### Check List

Tests <!-- At least one of them must be included. -->

- [ ] Unit test
- [ ] Integration test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No code

Side effects

- [ ] Performance regression: Consumes more CPU
- [ ] Performance regression: Consumes more Memory
- [ ] Breaking backward compatibility

Documentation

- [ ] Affects user behaviors
- [ ] Contains syntax changes
- [ ] Contains variable changes
- [ ] Contains experimental features
- [ ] Changes MySQL compatibility

### Release note

<!-- bugfix or new feature needs a release note -->

```release-note
None
```

---------

Signed-off-by: Lloyd-Pottiger <[email protected]>
Co-authored-by: JaySon <[email protected]>
Co-authored-by: jinhelin <[email protected]>

)

close #9485

vector: Fix ColumnArray does not work well with CHBlockChunkCodec

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

Storage: Add error message when fail to build local index

ref #9032

…oat32), Nullable(Array(Float32))" (#9490)

ref #9032

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>

storage: remove vector_index in column level