Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

vtgate table schema tracking to use GetSchema rpc #13544

Merged
merged 7 commits into from
Jul 24, 2023
Merged

vtgate table schema tracking to use GetSchema rpc #13544

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
9ce0ecd
vtgate table schema tracking to use GetSchema rpc
harshit-gangal Jul 18, 2023
4a18c38
updated test to reflect the new way of getting table schema
harshit-gangal Jul 19, 2023
d909515
removed unauthorized user test for schema tracking as the api is changed
harshit-gangal Jul 20, 2023
3dcaa20
Merge remote-tracking branch 'upstream/main' into schema-tracking-upg…
harshit-gangal Jul 20, 2023
1099c06
schema_change_signal_user is deprecated
harshit-gangal Jul 24, 2023
39c6684
addressed review comments
harshit-gangal Jul 24, 2023
7fd0bae
Merge remote-tracking branch 'upstream/main' into schema-tracking-upg…
harshit-gangal Jul 24, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions changelog/18.0/18.0.0/summary.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,10 @@ Throttler related `vttablet` flags:
- `--throttle_check_as_check_self` is deprecated and will be removed in `v19.0`
- `--throttler-config-via-topo` is deprecated after asummed `true` in `v17.0`. It will be removed in a future version.

VTGate flag:

- `--schema_change_signal_user` is deprecated and will be removed in `v19.0`

#### <a id="deleted-v3"/>Deleted `v3` planner

The `Gen4` planner has been the default planner since Vitess 14. The `v3` planner was deprecated in Vitess 15 and has now been removed in this release.
Expand Down
1 change: 0 additions & 1 deletion go/flags/endtoend/vtgate.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -148,7 +148,6 @@ Usage of vtgate:
--remote_operation_timeout duration time to wait for a remote operation (default 15s)
--retry-count int retry count (default 2)
--schema_change_signal Enable the schema tracker; requires queryserver-config-schema-change-signal to be enabled on the underlying vttablets for this to work (default true)
--schema_change_signal_user string User to be used to send down query to vttablet to retrieve schema changes
--security_policy string the name of a registered security policy to use for controlling access to URLs - empty means allow all for anyone (built-in policies: deny-all, read-only)
--service_map strings comma separated list of services to enable (or disable if prefixed with '-') Example: grpc-queryservice
--sql-max-length-errors int truncate queries in error logs to the given length (default unlimited)
Expand Down
34 changes: 0 additions & 34 deletions go/mysql/schema.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,24 +51,6 @@ FROM (
) _inner
GROUP BY table_name, column_name, ordinal_position, character_set_name, collation_name, data_type, column_key
HAVING COUNT(*) = 1
`

// DetectSchemaChangeOnlyBaseTable query detects if there is any schema change from previous copy excluding view tables.
DetectSchemaChangeOnlyBaseTable = `
SELECT DISTINCT table_name
FROM (
SELECT table_name, column_name, ordinal_position, character_set_name, collation_name, data_type, column_key
FROM information_schema.columns
WHERE table_schema = database() and table_name in (select table_name from information_schema.tables where table_schema = database() and table_type = 'BASE TABLE')

UNION ALL

SELECT table_name, column_name, ordinal_position, character_set_name, collation_name, data_type, column_key
FROM %s.schemacopy
WHERE table_schema = database()
) _inner
GROUP BY table_name, column_name, ordinal_position, character_set_name, collation_name, data_type, column_key
HAVING COUNT(*) = 1
`

// ClearSchemaCopy query clears the schemacopy table.
Expand All @@ -80,22 +62,6 @@ select table_schema, table_name, column_name, ordinal_position, character_set_na
from information_schema.columns
where table_schema = database()`

// fetchColumns are the columns we fetch
fetchColumns = "table_name, column_name, data_type, collation_name"

// FetchUpdatedTables queries fetches all information about updated tables
FetchUpdatedTables = `select ` + fetchColumns + `
from %s.schemacopy
where table_schema = database() and
table_name in ::tableNames
order by table_name, ordinal_position`

// FetchTables queries fetches all information about tables
FetchTables = `select ` + fetchColumns + `
from %s.schemacopy
where table_schema = database()
order by table_name, ordinal_position`

// GetColumnNamesQueryPatternForTable is used for mocking queries in unit tests
GetColumnNamesQueryPatternForTable = `SELECT COLUMN_NAME.*TABLE_NAME.*%s.*`
)
Expand Down
9 changes: 2 additions & 7 deletions go/test/endtoend/vtgate/schematracker/restarttablet/schema_restart_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,6 @@ var (
hostname = "localhost"
keyspaceName = "ks"
cell = "zone1"
signalInterval = 1
sqlSchema = `
create table vt_user (
id bigint,
Expand Down Expand Up @@ -78,8 +77,7 @@ func TestMain(m *testing.M) {
}

// List of users authorized to execute vschema ddl operations
clusterInstance.VtGateExtraArgs = []string{"--schema_change_signal"}

clusterInstance.VtGateExtraArgs = append(clusterInstance.VtGateExtraArgs, "--schema_change_signal")
// Start keyspace
keyspace := &cluster.Keyspace{
Name: keyspaceName,
Expand All @@ -91,10 +89,7 @@ func TestMain(m *testing.M) {

// restart the tablet so that the schema.Engine gets a chance to start with existing schema
tablet := clusterInstance.Keyspaces[0].Shards[0].PrimaryTablet()
tablet.VttabletProcess.ExtraArgs = []string{
"--queryserver-config-schema-change-signal",
fmt.Sprintf("--queryserver-config-schema-change-signal-interval=%d", signalInterval),
}
tablet.VttabletProcess.ExtraArgs = append(tablet.VttabletProcess.ExtraArgs, "--queryserver-config-schema-change-signal")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are no longer using the signal interval is that expected? If yes, we should remove the definition of signalInterval on line 45.

if err := tablet.RestartOnlyTablet(); err != nil {
return 1
}
Expand Down
20 changes: 6 additions & 14 deletions go/test/endtoend/vtgate/schematracker/sharded/st_sharded_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,15 +26,12 @@ import (
"time"

"github.com/stretchr/testify/assert"

"vitess.io/vitess/go/test/endtoend/utils"
"vitess.io/vitess/go/vt/sidecardb"
"vitess.io/vitess/go/vt/vtgate/planbuilder"

"github.com/stretchr/testify/require"

"vitess.io/vitess/go/mysql"
"vitess.io/vitess/go/test/endtoend/cluster"
"vitess.io/vitess/go/test/endtoend/utils"
"vitess.io/vitess/go/vt/sidecardb"
)

var (
Expand Down Expand Up @@ -86,15 +83,10 @@ func TestMain(m *testing.M) {
VSchema: VSchema,
SidecarDBName: sidecarDBName,
}
clusterInstance.VtGateExtraArgs = []string{"--schema_change_signal",
"--vschema_ddl_authorized_users", "%",
"--schema_change_signal_user", "userData1"}
clusterInstance.VtGatePlannerVersion = planbuilder.Gen4
clusterInstance.VtTabletExtraArgs = []string{"--queryserver-config-schema-change-signal",
"--queryserver-config-schema-change-signal-interval", "0.1",
"--queryserver-config-strict-table-acl",
"--queryserver-config-acl-exempt-acl", "userData1",
"--table-acl-config", "dummy.json"}
clusterInstance.VtGateExtraArgs = append(clusterInstance.VtGateExtraArgs,
"--schema_change_signal",
"--vschema_ddl_authorized_users", "%")
clusterInstance.VtTabletExtraArgs = append(clusterInstance.VtTabletExtraArgs, "--queryserver-config-schema-change-signal")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some acl flags have also been dropped. Is that intentional?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not required anymore. I will add a deprecation note for it.


if vtgateVer >= 16 && vttabletVer >= 16 {
clusterInstance.VtGateExtraArgs = append(clusterInstance.VtGateExtraArgs, "--enable-views")
Expand Down
4 changes: 2 additions & 2 deletions go/test/endtoend/vtgate/schematracker/sharded_prs/st_sharded_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,8 +145,8 @@ func TestMain(m *testing.M) {
sidecarDBName = sidecardb.DefaultName
}

clusterInstance.VtGateExtraArgs = []string{"--schema_change_signal", "--schema_change_signal_user", "userData1"}
clusterInstance.VtTabletExtraArgs = []string{"--queryserver-config-schema-change-signal", "--queryserver-config-schema-change-signal-interval", "5", "--queryserver-config-strict-table-acl", "--queryserver-config-acl-exempt-acl", "userData1", "--table-acl-config", "dummy.json"}
clusterInstance.VtGateExtraArgs = append(clusterInstance.VtGateExtraArgs, "--schema_change_signal")
clusterInstance.VtTabletExtraArgs = append(clusterInstance.VtTabletExtraArgs, "--queryserver-config-schema-change-signal")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same acl question

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#13544 (comment)


// Start topo server
err = clusterInstance.StartTopo()
Expand Down
5 changes: 0 additions & 5 deletions go/test/endtoend/vtgate/schematracker/unauthorized/schema.sql
View file
Open in desktop

This file was deleted.

130 changes: 0 additions & 130 deletions go/test/endtoend/vtgate/schematracker/unauthorized/unauthorized_test.go
View file
Open in desktop

This file was deleted.

18 changes: 0 additions & 18 deletions go/test/endtoend/vtgate/schematracker/unauthorized/vschema.json
View file
Open in desktop

This file was deleted.

2 changes: 1 addition & 1 deletion go/test/endtoend/vtgate/schematracker/unsharded/st_unsharded_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@ func TestMain(m *testing.M) {
SchemaSQL: sqlSchema,
SidecarDBName: sidecarDBName,
}
clusterInstance.VtTabletExtraArgs = []string{"--queryserver-config-schema-change-signal", "--queryserver-config-schema-change-signal-interval", "0.1"}
clusterInstance.VtTabletExtraArgs = []string{"--queryserver-config-schema-change-signal"}
err = clusterInstance.StartUnshardedKeyspace(*keyspace, 0, false)
if err != nil {
return 1
Expand Down
Loading
Loading