Skip to content

Commit

Permalink
slack-vitess-r14.0.5: remove legacy sharding, add MySQL 8.x backup …
Browse files Browse the repository at this point in the history
…test + compressors (#119)

* Delete all legacy sharding related code (vitessio#10278)

* Delete all legacy sharding related code

Signed-off-by: Matt Lord <[email protected]>

* Move used until functions from initialsharding to cluster

Signed-off-by: Matt Lord <[email protected]>

* Remove vtctl commands

Signed-off-by: Matt Lord <[email protected]>

* Kill vtworker and SetKeyspaceServedFrom cmd

Signed-off-by: Matt Lord <[email protected]>

* WaitForDrain related stragglers

Signed-off-by: Matt Lord <[email protected]>

* Legacy local straggler workflow

Signed-off-by: Matt Lord <[email protected]>

* Get rid of SetKeyspaceShardingInfo & wait for drain stragglers

Signed-off-by: Matt Lord <[email protected]>

* Remove vtworker stragglers

Signed-off-by: Matt Lord <[email protected]>

* Update throttlerservice protobuf

Signed-off-by: Matt Lord <[email protected]>

* Rename test 24, add hashicorp vault test to it (now mysql_server_vault)

Signed-off-by: Matt Lord <[email protected]>

* Remove last mentiones of legacy sharding in vtctl

Signed-off-by: Matt Lord <[email protected]>

* remove binlog_use_v3_resharding_mode

Signed-off-by: Matt Lord <[email protected]>

* Address review comments

Signed-off-by: Matt Lord <[email protected]>

* Address review comments

Signed-off-by: Matt Lord <[email protected]>

* Correct vtgate help output

Signed-off-by: Matt Lord <[email protected]>

* go fmt

Signed-off-by: Tim Vaillancourt <[email protected]>

* Remove v2 resharding fields (vitessio#10409)

* cleanup: remove sharding_column_name and sharding_column_type

Signed-off-by: Arvind Murty <[email protected]>

* cleanup: remove sharding_column_name and sharding_column_type

Signed-off-by: Arvind Murty <[email protected]>

* cleanup: remove sharding_column_name and sharding_column_type

Signed-off-by: Arvind Murty <[email protected]>

* cleanup: remove sharding_column_name and sharding_column_type

Signed-off-by: Arvind Murty <[email protected]>

* generate vtadmin files

Signed-off-by: Arvind Murty <[email protected]>

* cleanup: remove sharding_column_name and sharding_column_type from vtadmin

Signed-off-by: Arvind Murty <[email protected]>

* Merge from main

Signed-off-by: Arvind Murty <[email protected]>

* Fix bad merge conflict resolution

Signed-off-by: Tim Vaillancourt <[email protected]>

* Fix missing 'sharding' import

Signed-off-by: Tim Vaillancourt <[email protected]>

* Fix bad conflict resolution in go/test/endtoend/cellalias/cell_alias_test.go

Signed-off-by: Tim Vaillancourt <[email protected]>

* Backup/Restore: add support for external compressors and decompressors (vitessio#10558)

* change to support an external decompressor

Signed-off-by: Renan Rangel <[email protected]>
Signed-off-by: Rameez Sajwani <[email protected]>

* add external compressor support + builtin additional compressors

Signed-off-by: Renan Rangel <[email protected]>
Signed-off-by: Rameez Sajwani <[email protected]>

* wrap external compressor/decompressor

Signed-off-by: Renan Rangel <[email protected]>
Signed-off-by: Rameez Sajwani <[email protected]>

* go mod tidy + comments

Signed-off-by: Renan Rangel <[email protected]>
Signed-off-by: Rameez Sajwani <[email protected]>

* add copyright notices

Signed-off-by: Renan Rangel <[email protected]>
Signed-off-by: Rameez Sajwani <[email protected]>

* add support for builtin engine

Signed-off-by: Renan Rangel <[email protected]>
Signed-off-by: Rameez Sajwani <[email protected]>

* Adding test case for buckup compression

Signed-off-by: Rameez Sajwani <[email protected]>

* Fixing unit test and run mod tidy

Signed-off-by: Rameez Sajwani <[email protected]>

* Removing unwanted unit tests

Signed-off-by: Rameez Sajwani <[email protected]>

* Increase timeout of backup tests

Signed-off-by: Rameez Sajwani <[email protected]>

* fixing linter errors

Signed-off-by: Rameez Sajwani <[email protected]>

* Change test logic to accomodate running selective tests

Signed-off-by: Rameez Sajwani <[email protected]>

* removing lint warning

Signed-off-by: Rameez Sajwani <[email protected]>

* fixing test failure

Signed-off-by: Rameez Sajwani <[email protected]>

* Removing un-necessary test

Signed-off-by: Rameez Sajwani <[email protected]>

* Fixing code review feeback

Signed-off-by: Rameez Sajwani <[email protected]>

* Change builtinEngine to consider 'auto' decompressor

Signed-off-by: Rameez Sajwani <[email protected]>

* fixing Upgrade/Downgrade test

Signed-off-by: Rameez Sajwani <[email protected]>

* Fix type & add summary under release notes

Signed-off-by: Rameez Sajwani <[email protected]>

* Fixing typos in summary

Signed-off-by: Rameez Sajwani <[email protected]>

* Fixing flag name typos

Signed-off-by: Rameez Sajwani <[email protected]>

Co-authored-by: Renan Rangel <[email protected]>
Co-authored-by: Renan Rangel <[email protected]>

* Add MySQL 8 Support to Backup Tests (vitessio#10691)

* Add support for MySQL 8.0 in backup tests

Signed-off-by: Matt Lord <[email protected]>

* Add 8.0 workflow

Signed-off-by: Matt Lord <[email protected]>

* whitespace

Signed-off-by: Matt Lord <[email protected]>

* Use vtctldclient SetKeyspaceDurabilityPolicy to manage semi-sync

This needed to be done after the shard was setup in order to satisfy
the semantic assumptions related to semi-sync in the tests.

Signed-off-by: Matt Lord <[email protected]>

* Remove extraneous changes

Signed-off-by: Matt Lord <[email protected]>

* We need lz4 for TestXtrabackupStreamWithlz4Compression

Signed-off-by: Matt Lord <[email protected]>

* Try using Percona Repo for MySQL 8 to align mysqld and xtrabackup versions

Signed-off-by: Matt Lord <[email protected]>

* Specify stream type everywhere

Signed-off-by: Matt Lord <[email protected]>

* Remove repeated server install

Signed-off-by: Matt Lord <[email protected]>

* Moar...

Signed-off-by: Matt Lord <[email protected]>

* Move vtctlbackup test to 8.0

Signed-off-by: Matt Lord <[email protected]>

* Rename vtbackup test and move to MySQL 8

Signed-off-by: Matt Lord <[email protected]>

* Split the xbstream tests so the workflow doesn't time out

Otherwise it was going over the 10min limit and getting
killed.

Signed-off-by: Matt Lord <[email protected]>

* Use MySQL 8 compat method for setting passwords

Signed-off-by: Matt Lord <[email protected]>

* Test increasing timeout at another level

Signed-off-by: Matt Lord <[email protected]>

* Don't use the init passwords file with 8.0

mysqlctl doesn't start...

This is likely due to the change in initialization behavior with
MySQL 8.0 as it goes though two phases and you can't simply
start up mysqld and pass it data, it has to initialize and restart
first.

Signed-off-by: Matt Lord <[email protected]>

* Fix incorrect password update statement for vt_repl user

Signed-off-by: Rohit Nayak <[email protected]>

* Bump timeouts for 8.0 backup tests

Signed-off-by: Matt Lord <[email protected]>

* Bump it more :(

Signed-off-by: Matt Lord <[email protected]>

* Increase backup/restore timeout in backup_utils

Signed-off-by: Matt Lord <[email protected]>

* Apply new 8.0 template everywhere

Signed-off-by: Matt Lord <[email protected]>

* Fix bugs around how the compression flags were getting passed

Signed-off-by: Matt Lord <[email protected]>

* Use 45m timeout for the workflow, 30m for the run.

Signed-off-by: Matt Lord <[email protected]>

* These changes were no longer needed so limiting diff.

Signed-off-by: Matt Lord <[email protected]>

* Explicitly skip new linter check

Signed-off-by: Matt Lord <[email protected]>

Co-authored-by: Rohit Nayak <[email protected]>

* Fix test file merge issues

Signed-off-by: Tim Vaillancourt <[email protected]>

* Go 1.18.7 to fix error

Signed-off-by: Tim Vaillancourt <[email protected]>

* Fix lint

Signed-off-by: Tim Vaillancourt <[email protected]>

* Fix vttablet.txt

Signed-off-by: Tim Vaillancourt <[email protected]>

* Remove sharding_column_name and sharding_column_type from vtctld2 (vitessio#10459)

* Remove sharding_column_name and sharding_column_type

Signed-off-by: notfelineit <[email protected]>

* Run make web_build to compile production files

Signed-off-by: notfelineit <[email protected]>

* Remove vars from test

Signed-off-by: notfelineit <[email protected]>

* Remove sharding column type select box test from web test

Signed-off-by: notfelineit <[email protected]>

---------

Signed-off-by: Matt Lord <[email protected]>
Signed-off-by: Tim Vaillancourt <[email protected]>
Signed-off-by: Arvind Murty <[email protected]>
Signed-off-by: notfelineit <[email protected]>
Co-authored-by: Matt Lord <[email protected]>
Co-authored-by: Arvind Murty <[email protected]>
Co-authored-by: Rameez Sajwani <[email protected]>
Co-authored-by: Renan Rangel <[email protected]>
Co-authored-by: Renan Rangel <[email protected]>
Co-authored-by: Rohit Nayak <[email protected]>
Co-authored-by: Frances Thai <[email protected]>
  • Loading branch information
8 people authored Aug 30, 2023
1 parent 772645c commit 0a2ef83
Show file tree
Hide file tree
Showing 333 changed files with 4,605 additions and 34,577 deletions.
112 changes: 0 additions & 112 deletions .github/workflows/cluster_endtoend_19.yml

This file was deleted.

110 changes: 57 additions & 53 deletions .github/workflows/cluster_endtoend_mysql80.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,29 +6,23 @@ concurrency:
group: format('{0}-{1}', ${{ github.ref }}, 'Cluster (mysql80)')
cancel-in-progress: true

env:
LAUNCHABLE_ORGANIZATION: "vitess"
LAUNCHABLE_WORKSPACE: "vitess-app"
GITHUB_PR_HEAD_SHA: "${{ github.event.pull_request.head.sha }}"

jobs:
build:
name: Run endtoend tests on Cluster (mysql80)
runs-on:
group: vitess-ubuntu20
timeout-minutes: 45

steps:
- name: Check if workflow needs to be skipped
id: skip-workflow
run: |
skip='false'
if [[ "${{github.event.pull_request}}" == "" ]] && [[ "${{github.ref}}" != "refs/heads/main" ]] && [[ ! "${{github.ref}}" =~ ^refs/heads/slack-vitess-r[0-9]+\.[0-9]\.[0-9]+$ ]] && [[ ! "${{github.ref}}" =~ "refs/tags/.*" ]]; then
skip='true'
fi
echo Skip ${skip}
echo "skip-workflow=${skip}" >> $GITHUB_OUTPUT
- name: Check out code
if: steps.skip-workflow.outputs.skip-workflow == 'false'
uses: actions/checkout@v3
uses: actions/checkout@v2

- name: Check for changes in relevant files
if: steps.skip-workflow.outputs.skip-workflow == 'false'
uses: frouioui/paths-filter@main
id: changes
with:
Expand All @@ -39,74 +33,84 @@ jobs:
- 'test.go'
- 'Makefile'
- 'build.env'
- 'go.sum'
- 'go.mod'
- 'go.[sumod]'
- 'proto/*.proto'
- 'tools/**'
- 'config/**'
- 'bootstrap.sh'
- '.github/workflows/cluster_endtoend_mysql80.yml'
- '.github/workflows/**'
- name: Set up Go
if: steps.skip-workflow.outputs.skip-workflow == 'false' && steps.changes.outputs.end_to_end == 'true'
uses: actions/setup-go@v3
if: steps.changes.outputs.end_to_end == 'true'
uses: actions/setup-go@v2
with:
go-version: 1.18.7

- name: Set up python
if: steps.skip-workflow.outputs.skip-workflow == 'false' && steps.changes.outputs.end_to_end == 'true'
uses: actions/setup-python@v4
if: steps.changes.outputs.end_to_end == 'true'
uses: actions/setup-python@v2

- name: Tune the OS
if: steps.skip-workflow.outputs.skip-workflow == 'false' && steps.changes.outputs.end_to_end == 'true'
if: steps.changes.outputs.end_to_end == 'true'
run: |
# Limit local port range to not use ports that overlap with server side
# ports that we listen on.
echo '1024 65535' | sudo tee -a /proc/sys/net/ipv4/ip_local_port_range
# Increase the asynchronous non-blocking I/O. More information at https://dev.mysql.com/doc/refman/5.7/en/innodb-parameters.html#sysvar_innodb_use_native_aio
echo "fs.aio-max-nr = 1048576" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p /etc/sysctl.conf
- name: Get dependencies
if: steps.skip-workflow.outputs.skip-workflow == 'false' && steps.changes.outputs.end_to_end == 'true'
if: steps.changes.outputs.end_to_end == 'true'
run: |
# Get key to latest MySQL repo
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 467B942D3A79BD29
# Setup MySQL 8.0
wget -c https://dev.mysql.com/get/mysql-apt-config_0.8.20-1_all.deb
echo mysql-apt-config mysql-apt-config/select-server select mysql-8.0 | sudo debconf-set-selections
sudo DEBIAN_FRONTEND="noninteractive" dpkg -i mysql-apt-config*
# Setup Percona Server for MySQL 8.0
sudo apt-get update
# Install everything else we need, and configure
sudo apt-get install -y mysql-server mysql-client make unzip g++ etcd curl git wget eatmydata xz-utils
sudo apt-get install -y lsb-release gnupg2 curl
wget https://repo.percona.com/apt/percona-release_latest.$(lsb_release -sc)_all.deb
sudo DEBIAN_FRONTEND="noninteractive" dpkg -i percona-release_latest.$(lsb_release -sc)_all.deb
sudo percona-release setup ps80
sudo apt-get update
sudo service etcd stop
echo 'mysql version: '
sudo mysql --version
# Install everything else we need, and configure
sudo apt-get install -y percona-server-server percona-server-client make unzip g++ etcd git wget eatmydata xz-utils
sudo service mysql stop
sudo service etcd stop
sudo ln -s /etc/apparmor.d/usr.sbin.mysqld /etc/apparmor.d/disable/
sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.mysqld
go mod download
# install JUnit report formatter
go install github.com/vitessio/go-junit-report@HEAD
- name: Setup launchable dependencies
if: steps.changes.outputs.end_to_end == 'true'
run: |
# Get Launchable CLI installed. If you can, make it a part of the builder image to speed things up
pip3 install --user launchable~=1.0 > /dev/null
# verify that launchable setup is all correct.
launchable verify || true
# Tell Launchable about the build you are producing and testing
launchable record build --name "$GITHUB_RUN_ID" --source .
- name: Run cluster endtoend test
if: steps.skip-workflow.outputs.skip-workflow == 'false' && steps.changes.outputs.end_to_end == 'true'
uses: nick-fields/retry@v2
with:
timeout_minutes: 45
max_attempts: 3
retry_on: error
command: |
# We set the VTDATAROOT to the /tmp folder to reduce the file path of mysql.sock file
# which musn't be more than 107 characters long.
export VTDATAROOT="/tmp/"
source build.env
set -x
# run the tests however you normally do, then produce a JUnit XML file
eatmydata -- go run test.go -docker=false -follow -shard mysql80
if: steps.changes.outputs.end_to_end == 'true'
timeout-minutes: 30
run: |
# We set the VTDATAROOT to the /tmp folder to reduce the file path of mysql.sock file
# which musn't be more than 107 characters long.
export VTDATAROOT="/tmp/"
source build.env
set -x
# run the tests however you normally do, then produce a JUnit XML file
eatmydata -- go run test.go -docker=false -follow -shard mysql80 | tee -a output.txt | go-junit-report -set-exit-code > report.xml
- name: Print test output and Record test result in launchable
if: steps.changes.outputs.end_to_end == 'true' && always()
run: |
# send recorded tests to launchable
launchable record tests --build "$GITHUB_RUN_ID" go-test . || true
# print test output
cat output.txt
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
# DO NOT MODIFY: THIS FILE IS GENERATED USING "make generate_ci_workflows"

name: Cluster (24)
name: Cluster (mysql_server_vault)
on: [push, pull_request]
concurrency:
group: format('{0}-{1}', ${{ github.ref }}, 'Cluster (24)')
group: format('{0}-{1}', ${{ github.ref }}, 'Cluster (mysql_server_vault)')
cancel-in-progress: true

jobs:
build:
name: Run endtoend tests on Cluster (24)
name: Run endtoend tests on Cluster (mysql_server_vault)
runs-on:
group: vitess-ubuntu20

Expand Down Expand Up @@ -113,5 +113,14 @@ jobs:
set -x
# run the tests however you normally do, then produce a JUnit XML file
eatmydata -- go run test.go -docker=false -follow -shard 24
# run the tests however you normally do, then produce a JUnit XML file
eatmydata -- go run test.go -docker=false -follow -shard mysql_server_vault | tee -a output.txt | go-junit-report -set-exit-code > report.xml

- name: Print test output and Record test result in launchable
if: steps.changes.outputs.end_to_end == 'true' && always()
run: |
# send recorded tests to launchable
launchable record tests --build "$GITHUB_RUN_ID" go-test . || true
# print test output
cat output.txt
Loading

0 comments on commit 0a2ef83

Please sign in to comment.