storage: Abort vector index building as soon as possible

[Lloyd-Pottiger]

What problem does this PR solve?

Issue Number: close #9032

Problem Summary:

What is changed and how it works?

Pick https://github.com/tidbcloud/tiflash-cse/pull/232, https://github.com/tidbcloud/tiflash-cse/pull/240 and https://github.com/tidbcloud/tiflash-cse/pull/270

• Use a lower efSearch to possibly improve performance and trade off with recall. Performance can be improved when searching top K where K < 64.
• Abort vector index building as soon as possible. This could be helpful during the process of loading a lot of vector data.

TODO: In this PR, index build is cancelled after segment stable is invalid, e.g. after applyMergeDelta. Maybe we can do it as soon as possible, e.g. when prepareMergeDelta?

Benchmark @ M1 Pro:

efSearch=64, TopK=10: 107826 ns
efSearch=16, TopK=10: 63858 ns

storage: Abort vector index building as soon as possible

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)

Run VectorDBBench, and we can see that during the data loading process actually all vector index build will be cancelled:

[2024/07/07 10:47:52.290 +08:00] [DEBUG] [Segment.cpp:1244] ["MergeDelta - Begin apply"] [source="keyspace=1 table_id=96 segment_id=1 epoch=3"] [thread_id=41]
[2024/07/07 10:47:52.291 +08:00] [DEBUG] [Segment.cpp:1277] ["MergeDelta - Finish apply, old_me=<segment_id=1 epoch=3 range=[-9223372036854775808,9223372036854775807) next_segment_id=0 delta_rows=94500 delta_bytes=3591000 delta_deletes=0 stable_file=dmf_4(v=0) stable_rows=220500 stable_bytes=684873000 dmf_rows=220500 dmf_bytes=684873000 dmf_packs=27> new_me=<segment_id=1 epoch=4 range=[-9223372036854775808,9223372036854775807) next_segment_id=0 delta_rows=10500 delta_bytes=399000 delta_deletes=0 stable_file=dmf_5(v=0) stable_rows=304500 stable_bytes=945777000 dmf_rows=304500 dmf_bytes=945777000 dmf_packs=38>"] [source="keyspace=1 table_id=96 segment_id=1 epoch=3"] [thread_id=41]
[2024/07/07 10:47:52.291 +08:00] [DEBUG] [Segment.h:585] ["Abandon segment, segment=<segment_id=1 epoch=3 stable=dmf_4(v=0) range=[-9223372036854775808,9223372036854775807)>"] [source="keyspace=1 table_id=96 segment_id=1 epoch=3"] [thread_id=41]
[2024/07/07 10:47:52.291 +08:00] [INFO] [DeltaMergeStore_InternalSegment.cpp:909] ["MergeDelta - Finish, delta is merged, old_segment=<segment_id=1 epoch=3 range=[-9223372036854775808,9223372036854775807) abandoned=true next_segment_id=0 delta_rows=94500 delta_bytes=3591000 delta_deletes=0 stable_file=dmf_4(v=0) stable_rows=220500 stable_bytes=684873000 dmf_rows=220500 dmf_bytes=684873000 dmf_packs=27> new_segment=<segment_id=1 epoch=4 range=[-9223372036854775808,9223372036854775807) next_segment_id=0 delta_rows=10500 delta_bytes=399000 delta_deletes=0 stable_file=dmf_5(v=0) stable_rows=304500 stable_bytes=945777000 dmf_rows=304500 dmf_bytes=945777000 dmf_packs=38>"] [source="keyspace=1 table_id=96"] [thread_id=41]
...
...
[2024/07/07 10:47:52.478 +08:00] [DEBUG] [DMFileV3IncrementWriter.cpp:125] ["Abandon increment write, local_path=/Users/breezewish/.tiup/data/serverless/tiflash-0/main_data/data/ks_1_t_96/stable/dmf_4 file_names={}"] [thread_id=36]
[2024/07/07 10:47:52.481 +08:00] [INFO] [DeltaMergeStore_InternalSegment.cpp:739] ["EnsureStableIndex - Build index aborted because DMFile is no longer valid, dm_files=dmf_4(v=0) source_segment=<segment_id=1 epoch=3 stable=dmf_4(v=0) range=[-9223372036854775808,9223372036854775807)>"] [source="keyspace=1 table_id=96"] [thread_id=36]

• No code

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

None

[JaySon-Huang]

LGTM

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: breezewish, JaySon-Huang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [JaySon-Huang,breezewish]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2024-09-20 06:19:23.700022594 +0000 UTC m=+1201233.440446534: ☑️ agreed by JaySon-Huang.
• 2024-09-20 06:36:56.355795748 +0000 UTC m=+1202286.096219672: ☑️ agreed by breezewish.