storage: add local indexer scheduler

[Lloyd-Pottiger]

What problem does this PR solve?

Issue Number: ref #9032

Problem Summary:

What is changed and how it works?

Extract LocalIndexerScheduler from https://github.com/tidbcloud/tiflash-cse/pull/203 to reduce the difficulty of code review. Meanwhile, LocalIndexerScheduler::Task::dmfile_ids is changed to LocalIndexerScheduler::Task::file_ids, which means we use page ids to identify each task, so we can add task for ColumnFileTiny later.

storage: add local indexer scheduler

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

None

[breezewish]

LocalIndexerScheduler::Task::dmfile_ids is changed to LocalIndexerScheduler::Task::page_ids

Maybe need some better design (or names), since after logical split the left and right segment will hold the same dmfile_id but different page_id.

[Lloyd-Pottiger]

LocalIndexerScheduler::Task::dmfile_ids is changed to LocalIndexerScheduler::Task::page_ids

Maybe need some better design (or names), since after logical split the left and right segment will hold the same dmfile_id but different page_id.

@breezewish rename to

    // The file id of the DMFile.
    struct DMFileID
    {
        explicit DMFileID(PageIdU64 id_)
            : id(id_)
        {}
        PageIdU64 id;
    };
    // The page id of the ColumnFileTiny.
    struct ColumnFileTinyID
    {
        explicit ColumnFileTinyID(PageIdU64 id_)
            : id(id_)
        {}
        PageIdU64 id;
    };
    using FileID = std::variant<DMFileID, ColumnFileTinyID>;

[SeaRise]

It looks like internal_task will not be shared, how about using unique_ptr ?

[Lloyd-Pottiger]

But internal_task should be copy-able

[JaySon-Huang]

LGTM

[CalvinNeo]

lgtm

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: CalvinNeo, JaySon-Huang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [CalvinNeo,JaySon-Huang]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2024-09-04 08:51:37.697110388 +0000 UTC m=+434422.215163310: ☑️ agreed by JaySon-Huang.
• 2024-09-04 11:27:39.341103479 +0000 UTC m=+443783.859156401: ☑️ agreed by CalvinNeo.

[JaySon-Huang]

/test pull-integration-test