Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support throttling vstreamer copy table work on source tablets #9923

Merged
merged 17 commits into from
Apr 4, 2022
Merged

Support throttling vstreamer copy table work on source tablets #9923

merged 17 commits into from
Apr 4, 2022

Commits on Mar 29, 2022

  1. Support throttling vstream work on source tablets 

    This provides mechanisms to limit the impact of large vreplication
workflows on production source tablets.

It supports throttling based on the InnoDB history list length and
the replication lag seen.

Signed-off-by: Matt Lord <[email protected]>
    @mattlord
    mattlord committed Mar 29, 2022
    Configuration menu
    Copy the full SHA
    4e13f14 View commit details
    Browse the repository at this point in the history

  2. Add unit and e2e test for copy table throttling 

    Signed-off-by: Matt Lord <[email protected]>
    @mattlord
    mattlord committed Mar 29, 2022
    Configuration menu
    Copy the full SHA
    e1c069a View commit details
    Browse the repository at this point in the history

  3. Use shorter internal name for max replica lag seconds 

    Signed-off-by: Matt Lord <[email protected]>
    @mattlord
    mattlord committed Mar 29, 2022
    Configuration menu
    Copy the full SHA
    e38349e View commit details
    Browse the repository at this point in the history

  4. Use consistent case for json vars in config test 

    Signed-off-by: Matt Lord <[email protected]>
    @mattlord
    mattlord committed Mar 29, 2022
    Configuration menu
    Copy the full SHA
    be73d02 View commit details
    Browse the repository at this point in the history

  5. Add MySQL specific context to flags 

    Signed-off-by: Matt Lord <[email protected]>
    @mattlord
    mattlord committed Mar 29, 2022
    Configuration menu
    Copy the full SHA
    df0d9dd View commit details
    Browse the repository at this point in the history

  6. Add copy table row streamer stats 

    Signed-off-by: Matt Lord <[email protected]>
    @mattlord
    mattlord committed Mar 29, 2022
    Configuration menu
    Copy the full SHA
    3a30068 View commit details
    Browse the repository at this point in the history

  7. Rearrange things to wait for MySQL in initial row stream and copy pha… 

    …se cycles

Signed-off-by: Matt Lord <[email protected]>
    @mattlord
    mattlord committed Mar 29, 2022
    Configuration menu
    Copy the full SHA
    69566f8 View commit details
    Browse the repository at this point in the history

  8. Correct & improve on e2e and unit tests 

    Signed-off-by: Matt Lord <[email protected]>
    @mattlord
    mattlord committed Mar 29, 2022
    Configuration menu
    Copy the full SHA
    6c8655f View commit details
    Browse the repository at this point in the history

  9. Update test config 

    I spent way too much time trying to get both tests to work within
the same go test run... giving up for now.

Signed-off-by: Matt Lord <[email protected]>
    @mattlord
    mattlord committed Mar 29, 2022
    Configuration menu
    Copy the full SHA
    c579047 View commit details
    Browse the repository at this point in the history

  10. Adding additional stats work 

    Signed-off-by: Matt Lord <[email protected]>
    @mattlord
    mattlord committed Mar 29, 2022
    Configuration menu
    Copy the full SHA
    ac8564a View commit details
    Browse the repository at this point in the history

  11. Add ability to modify the RowStreamerConfig at runtime via /debug/env 

    Signed-off-by: Matt Lord <[email protected]>
    @mattlord
    mattlord committed Mar 29, 2022
    Configuration menu
    Copy the full SHA
    0095dbc View commit details
    Browse the repository at this point in the history

  12. Add current RowStreamerConfig values to /debug/vars 

    Signed-off-by: Matt Lord <[email protected]>
    @mattlord
    mattlord committed Mar 29, 2022
    Configuration menu
    Copy the full SHA
    9de8182 View commit details
    Browse the repository at this point in the history

Commits on Mar 30, 2022

  1. Add waitingForMySQL stats to vreplication phase timings 

    Signed-off-by: Matt Lord <[email protected]>
    @mattlord
    mattlord committed Mar 30, 2022
    Configuration menu
    Copy the full SHA
    4fca26b View commit details
    Browse the repository at this point in the history

  2. Track waitForMySQL by table on vstreamer side 

    Signed-off-by: Matt Lord <[email protected]>
    @mattlord
    mattlord committed Mar 30, 2022
    Configuration menu
    Copy the full SHA
    07bd5c9 View commit details
    Browse the repository at this point in the history

  3. Move row streamer config export to vstreamer engine 

    And make it a gauge so that it's always showing the current value
as we allow this to be changed in the running process via /debug/env.

Signed-off-by: Matt Lord <[email protected]>
    @mattlord
    mattlord committed Mar 30, 2022
    Configuration menu
    Copy the full SHA
    4ba3209 View commit details
    Browse the repository at this point in the history

Commits on Mar 31, 2022

  1. Minor changes after self review 

    Signed-off-by: Matt Lord <[email protected]>
    @mattlord
    mattlord committed Mar 31, 2022
    Configuration menu
    Copy the full SHA
    e60a3ce View commit details
    Browse the repository at this point in the history

  2. Shorten/generalize InnoDB row history test func names 

    Signed-off-by: Matt Lord <[email protected]>
    @mattlord
    mattlord committed Mar 31, 2022
    Configuration menu
    Copy the full SHA
    ac3a3ac View commit details
    Browse the repository at this point in the history