Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DDL: Add index acceleration demo draft pr #35110

Closed
wants to merge 1 commit into from

Conversation

Benjamin2037
Copy link
Collaborator

create sysvar TiDBFastDDL (#3)

  • Change TiDBFastDDL to Global scope.
  • TiDBFastDDL Variable usage
    1, At beginning of doReorgWorkForCreateIndex, use global variable TiDBFastDDL to detemine whether enable lightning backfill process. After set up lightning, then set
    reorgInfo.IsLightningEnabled to true to show the lightning add index is uded for this DDL job.
    2, Later, check reorgInfo.IsLightningEnabled to control the DDL job level lightning's process.
    Complete lightning env initlize procdeure (Improve Documentation: various corrections of the English usage i… #4)
  1. Get max open file limition for lightning, default 1024
  2. Set light sorted path:
    2.1 If sysVar DataDir is not a start with / data dir path, then set path to /tmp/lightning
    2.2 otherwise set path to DataDir + "lightning"
    Check whether the lightning sorted path is exist and a dir, if yes then keep use it.
    otherwise create new one.
    3, Set the lightning dir quota, default 10G
    Add one unit test
    uniform lightning related errmessage text to lightning_error file. (fix CurrentTimestamp check error for different timezones #5)
    complete memory track module work.
    refacter memmory manager to resource manager for expand to control concurrent
    base available cpu core number.
    Finsih integrate with lightning concurrency process logic.
    Add log for lightning processing.
    refactor some code`
    complete memory track module work. (tools: add .travis.yml #7)
    complete user cancel and exception part logic implement
  • complete memory track module work.
  • refacter memmory manager to resource manager for expand to control concurrent
    base available cpu core number.

Finsih integrate with lightning concurrency process logic.

  • Add log for lightning processing.
    refactor some code`
    restore logic first stage
    adjust metric of ddl process progress value for lightning solution
    refine import cycle
    set up disk quota
    refine code and add some ut and ft.
    Add config paramemter TiDBlightningSortPath for setting sort parth for add index.
    fix mem reclaim problems
    Add log infromation
    combine the optimizes:
    1, prundecode
    2, json expression
    Add some log information

What problem does this PR solve?

Issue Number: close #xxx

Problem Summary:

What is changed and how it works?

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

create sysvar TiDBFastDDL (#3)
* Change TiDBFastDDL to Global scope.
* TiDBFastDDL Variable usage
1, At beginning of doReorgWorkForCreateIndex, use global variable TiDBFastDDL to detemine whether enable lightning backfill process. After set up lightning, then set
reorgInfo.IsLightningEnabled to true to show the lightning add index is uded for this DDL job.
2, Later, check reorgInfo.IsLightningEnabled to control the DDL job level lightning's process.
Complete lightning env initlize procdeure (#4)
1. Get max open file limition for lightning, default 1024
2. Set light sorted path:
   2.1 If sysVar DataDir is not a start with / data dir path, then set path to /tmp/lightning
   2.2 otherwise set path to DataDir + "lightning"
   Check whether the lightning sorted path is exist and a dir, if yes then keep use it.
   otherwise create new one.
3, Set the lightning dir quota, default 10G
Add one unit test
uniform lightning related errmessage text to lightning_error file. (#5)
complete memory track module work.
refacter memmory manager to resource manager for expand to control concurrent
base available  cpu core number.
Finsih integrate with lightning concurrency process logic.
Add log for lightning processing.
refactor some code`
complete memory track module work. (#7)
complete user cancel and exception part logic implement
* complete memory track module work.
* refacter memmory manager to resource manager for expand to control concurrent
base available  cpu core number.

Finsih integrate with lightning concurrency process logic.

* Add log for lightning processing.
refactor some code`
restore logic first stage
adjust metric of ddl process progress value for lightning solution
refine import cycle
set up disk quota
refine code and add some ut and ft.
Add config paramemter TiDBlightningSortPath for setting sort parth for add index.
fix mem reclaim problems
Add log infromation
combine the optimizes:
1, prundecode
2, json expression
Add some log information
@ti-chi-bot
Copy link
Member

[REVIEW NOTIFICATION]

This pull request has not been approved.

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added do-not-merge/needs-linked-issue release-note-none Denotes a PR that doesn't merit a release note. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jun 1, 2022
@sre-bot
Copy link
Contributor

sre-bot commented Jun 1, 2022

Please follow PR Title Format:

  • pkg [, pkg2, pkg3]: what is changed

Or if the count of mainly changed packages are more than 3, use

  • *: what is changed

After you have format title, you can leave a comment /run-check_title to recheck it

@Benjamin2037 Benjamin2037 changed the title fix 'import cycle' for package tidb/ddl. DDL: Add index acceleration demo draft pr Jun 1, 2022
@ti-chi-bot
Copy link
Member

[FORMAT CHECKER NOTIFICATION]

Notice: To remove the do-not-merge/needs-linked-issue label, please provide the linked issue number on one line in the PR body, for example: Issue Number: close #123 or Issue Number: ref #456.

📖 For more info, you can check the "Contribute Code" section in the development guide.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
do-not-merge/needs-linked-issue do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants