storage: Support adding vector index in background #9411
Conversation
ti-chi-bot
bot
added
release-note-none
Signed-off-by: Wish <[email protected]> Signed-off-by: Lloyd-Pottiger <[email protected]> Co-authored-by: Wish <[email protected]> Co-authored-by: JaySon-Huang <[email protected]>
Lloyd-Pottiger
force-pushed
the
add-index-background
branch
from
f9422af
to
707e91c
Compare
| } | ||
| else if (global_context->getSharedContextDisagg()->isDisaggregatedStorageMode()) | ||
| { | ||
| global_context->initializeGlobalLocalIndexerScheduler( |
There was a problem hiding this comment.
initialize the LocalIndexerScheduler introduce by #9383, all vector index build task will be schedule by this scheduler.
There was a problem hiding this comment.
I checked that PR, and I can not figure out why it is * 8 / 10
There was a problem hiding this comment.
// There is no compute task in write node.
// Set the pool size to 80% of logical cores and 60% of memory
// to take full advantage of the resources and avoid blocking other tasks like writes and compactions.
Add some comments.
Signed-off-by: Lloyd-Pottiger <[email protected]>
| { | ||
| RUNTIME_CHECK(segment != nullptr); | ||
|
|
||
| // TODO(local index): There could be some indexes are built while some indexes is not yet built after DDL |
There was a problem hiding this comment.
There could be some indexes are being built?
There was a problem hiding this comment.
Just ignore it, will remove it in a later PR.
| for (const auto & seg_id : segment_ids) | ||
| { | ||
| auto segment = id_to_segment[seg_id]; | ||
| auto new_segment = segmentUpdateMeta(lock, dm_context, segment, new_dmfiles); |
There was a problem hiding this comment.
In segmentUpdateMeta, we will acquire a lock for using Lock = std::unique_lock<std::recursive_mutex>; of a DeltaValueSpace for a Segment, while we are holding also a unique_lock of DM.
I wonder if the lock is too grain? AFAIK, the segment lock is only used in replaceStableMetaVersion. So I wonder if we could do replaceStableMetaVersion without DM lock?
There was a problem hiding this comment.
Accquiring DM lock to make sure segment will not be abandoned by other thread.
Signed-off-by: Lloyd-Pottiger <[email protected]>
ti-chi-bot
bot
added
needs-1-more-lgtm
Signed-off-by: Lloyd-Pottiger <[email protected]>
dbms/src/Storages/DeltaMerge/DeltaMergeStore_InternalSegment.cpp
Outdated
Show resolved
Hide resolved
dbms/src/Storages/DeltaMerge/DeltaMergeStore_InternalSegment.cpp
Outdated
Show resolved
Hide resolved
dbms/src/Storages/DeltaMerge/DeltaMergeStore_InternalSegment.cpp
Outdated
Show resolved
Hide resolved
| @@ -1470,8 +1465,11 @@ SegmentPtr Segment::replaceStableMetaVersion( | |||
|
|
|||
| wbs.writeAll(); | |||
|
|
|||
| LOG_DEBUG(log, "ReplaceStableMetaVersion - Finish, new_stable_files={}", new_stable_files_str()); | |||
|
|
|||
| LOG_DEBUG( | |||
There was a problem hiding this comment.
We will print info log in DeltaMergeStore::segmentUpdateMeta
Signed-off-by: Lloyd-Pottiger <[email protected]>
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: breezewish, JaySon-Huang The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
[LGTM Timeline notifier]Timeline:
|
What problem does this PR solve?
Issue Number: ref #9032
Problem Summary:
What is changed and how it works?
Check List
Tests
Side effects
STORAGE_FORMAT_V7Documentation
Release note