Add OnlineDDL show support #13738

ajm188 · 2023-08-07T17:47:23Z

Description

Adds OnlineDDL show support to vtctldclient and a corresponding RPC.

Also adds the ability to (mostly) marshal a go structure into a sqltypes.Result object.

Example output: vtctldclient --server "localhost:15999" OnlineDDL show commerce --json

{
  "migrations": [
    {
      "uuid": "5a1682cc_3381_11ee_9fad_2e0cac6950de",
      "keyspace": "commerce",
      "shard": "0",
      "schema": "vt_commerce",
      "table": "customer",
      "migration_statement": "alter table customer rename column last_name to lname",
      "strategy": 0,
      "options": "",
      "added_at": {
        "seconds": "1691234115",
        "nanoseconds": 0
      },
      "requested_at": {
        "seconds": "1691234115",
        "nanoseconds": 0
      },
      "ready_at": null,
      "started_at": {
        "seconds": "1691234116",
        "nanoseconds": 0
      },
      "liveness_timestamp": {
        "seconds": "1691234122",
        "nanoseconds": 0
      },
      "completed_at": {
        "seconds": "1691234123",
        "nanoseconds": 203770000
      },
      "cleaned_up_at": {
        "seconds": "1691320929",
        "nanoseconds": 0
      },
      "status": 6,
      "log_path": "",
      "artifacts": "",
      "retries": "0",
      "tablet": {
        "cell": "zone1",
        "uid": 101
      },
      "tablet_failure": false,
      "progress": 100,
      "migration_context": "vtctl:5a14a2fe-3381-11ee-9fad-2e0cac6950de",
      "ddl_action": "alter",
      "message": "",
      "eta_seconds": "0",
      "rows_copied": "0",
      "table_rows": "0",
      "added_unique_keys": 0,
      "removed_unique_keys": 0,
      "log_file": "",
      "artifact_retention": {
        "seconds": "86400",
        "nanos": 0
      },
      "postpone_completion": false,
      "removed_unique_key_names": "",
      "dropped_no_default_column_names": "",
      "expanded_column_names": "",
      "revertible_notes": "",
      "allow_concurrent": false,
      "reverted_uuid": "",
      "is_view": false,
      "ready_to_complete": true,
      "vitess_liveness_indicator": "1691234119",
      "user_throttle_ratio": 0,
      "special_plan": "",
      "last_throttled_at": null,
      "component_throttled": "",
      "cancelled_at": null,
      "postpone_launch": false,
      "stage": "re-enabling writes",
      "cutover_attempts": 1,
      "is_immediate_operation": false,
      "reviewed_at": {
        "seconds": "1691234116",
        "nanoseconds": 0
      },
      "ready_to_complete_at": {
        "seconds": "1691234120",
        "nanoseconds": 0
      }
    }
  ]
}

Example output: $ vtctldclient --server "localhost:15999" OnlineDDL show commerce

+--------------------------------------+----------+-------+--------------+-------------+--------------------------------+----------+---------+---------------------+---------------------+-----------------+---------------------+---------------------+---------------------+-------------------+----------+----------+-------------------------------------------------------------+---------+------------------+----------------+----------+--------------------------------------------+------------+--------------------------------------------------------------+-------------+-------------+------------+-------------------+---------------------+----------+----------------------------+---------------------+--------------------------+---------------------------------+-----------------------+------------------+------------------+---------------+---------+-------------------+---------------------------+---------------------+--------------+--------------------------+---------------------+---------------------+-----------------+--------------------+------------------+------------------------+---------------------+-----------------------------+
|            migration_uuid            | keyspace | shard | mysql_schema | mysql_table |      migration_statement       | strategy | options |   added_timestamp   | requested_timestamp | ready_timestamp |  started_timestamp  | liveness_timestamp  | completed_timestamp | cleanup_timestamp |  status  | log_path |                          artifacts                          | retries |      tablet      | tablet_failure | progress |             migration_context              | ddl_action |                           message                            | eta_seconds | rows_copied | table_rows | added_unique_keys | removed_unique_keys | log_file | artifact_retention_seconds | postpone_completion | removed_unique_key_names | dropped_no_default_column_names | expanded_column_names | revertible_notes | allow_concurrent | reverted_uuid | is_view | ready_to_complete | vitess_liveness_indicator | user_throttle_ratio | special_plan | last_throttled_timestamp | component_throttled | cancelled_timestamp | postpone_launch |       stage        | cutover_attempts | is_immediate_operation | reviewed_timestamp  | ready_to_complete_timestamp |
+--------------------------------------+----------+-------+--------------+-------------+--------------------------------+----------+---------+---------------------+---------------------+-----------------+---------------------+---------------------+---------------------+-------------------+----------+----------+-------------------------------------------------------------+---------+------------------+----------------+----------+--------------------------------------------+------------+--------------------------------------------------------------+-------------+-------------+------------+-------------------+---------------------+----------+----------------------------+---------------------+--------------------------+---------------------------------+-----------------------+------------------+------------------+---------------+---------+-------------------+---------------------------+---------------------+--------------+--------------------------+---------------------+---------------------+-----------------+--------------------+------------------+------------------------+---------------------+-----------------------------+
| b3944b78_3b55_11ee_9603_2e0cac6950de | commerce |     0 | vt_commerce  | customer    | alter table customer rename    | vitess   |         | 2023-08-15 06:22:56 | 2023-08-15 06:22:57 |                 | 2023-08-15 06:22:58 | 2023-08-15 06:22:58 |                     |                   | failed   |          | _b3944b78_3b55_11ee_9603_2e0cac6950de_20230815062257_vrepl, |       0 | zone1-0000000101 |                |        0 | vtctl:b391c95c-3b55-11ee-9603-2e0cac6950de | alter      | Unknown column 'last_name' in                                |          -1 |           0 |          0 |                 0 |                   0 |          |                      86400 |                     |                          |                                 |                       |                  |                  |               |         |                   |                         0 |                   0 |              |                          |                     |                     |                 |                    |                0 |                        | 2023-08-15 06:22:58 |                             |
|                                      |          |       |              |             | column last_name to lname      |          |         |                     |                     |                 |                     |                     |                     |                   |          |          |                                                             |         |                  |                |          |                                            |            | '_b3944b78_3b55_11ee_9603_2e0cac6950de_20230815062257_vrepl' |             |             |            |                   |                     |          |                            |                     |                          |                                 |                       |                  |                  |               |         |                   |                           |                     |              |                          |                     |                     |                 |                    |                  |                        |                     |                             |
|                                      |          |       |              |             |                                |          |         |                     |                     |                 |                     |                     |                     |                   |          |          |                                                             |         |                  |                |          |                                            |            | (errno 1054) (sqlstate 42S22) during query: ALTER TABLE      |             |             |            |                   |                     |          |                            |                     |                          |                                 |                       |                  |                  |               |         |                   |                           |                     |              |                          |                     |                     |                 |                    |                  |                        |                     |                             |
|                                      |          |       |              |             |                                |          |         |                     |                     |                 |                     |                     |                     |                   |          |          |                                                             |         |                  |                |          |                                            |            | `_b3944b78_3b55_11ee_9603_2e0cac6950de_20230815062257_vrepl` |             |             |            |                   |                     |          |                            |                     |                          |                                 |                       |                  |                  |               |         |                   |                           |                     |              |                          |                     |                     |                 |                    |                  |                        |                     |                             |
|                                      |          |       |              |             |                                |          |         |                     |                     |                 |                     |                     |                     |                   |          |          |                                                             |         |                  |                |          |                                            |            | RENAME COLUMN `last_name` TO `lname`, ALGORITHM = copy       |             |             |            |                   |                     |          |                            |                     |                          |                                 |                       |                  |                  |               |         |                   |                           |                     |              |                          |                     |                     |                 |                    |                  |                        |                     |                             |
| dba5a774_3b55_11ee_9603_2e0cac6950de | commerce |     0 | vt_commerce  | customer    | alter table customer rename    | vitess   |         | 2023-08-15 06:24:03 | 2023-08-15 06:24:04 |                 | 2023-08-15 06:24:05 | 2023-08-15 06:24:11 | 2023-08-15 06:24:11 |                   | complete |          | _dba5a774_3b55_11ee_9603_2e0cac6950de_20230815062404_vrepl, |       0 | zone1-0000000101 |                |      100 | vtctl:dba393da-3b55-11ee-9603-2e0cac6950de | alter      |                                                              |           0 |           0 |          0 |                 0 |                   0 |          |                      86400 |                     |                          |                                 |                       |                  |                  |               |         |                   |                1692095048 |                   0 |              |                          |                     |                     |                 | re-enabling writes |                1 |                        | 2023-08-15 06:24:05 | 2023-08-15 06:24:09         |
|                                      |          |       |              |             | column last_name to lname      |          |         |                     |                     |                 |                     |                     |                     |                   |          |          |                                                             |         |                  |                |          |                                            |            |                                                              |             |             |            |                   |                     |          |                            |                     |                          |                                 |                       |                  |                  |               |         |                   |                           |                     |              |                          |                     |                     |                 |                    |                  |                        |                     |                             |
+--------------------------------------+----------+-------+--------------+-------------+--------------------------------+----------+---------+---------------------+---------------------+-----------------+---------------------+---------------------+---------------------+-------------------+----------+----------+-------------------------------------------------------------+---------+------------------+----------------+----------+--------------------------------------------+------------+--------------------------------------------------------------+-------------+-------------+------------+-------------------+---------------------+----------+----------------------------+---------------------+--------------------------+---------------------------------+-----------------------+------------------+------------------+---------------+---------+-------------------+---------------------------+---------------------+--------------+--------------------------+---------------------+---------------------+-----------------+--------------------+------------------+------------------------+---------------------+-----------------------------+

Related Issue(s)

Fixes: Migrate OnlineDDL Show #13714

Checklist

"Backport to:" labels have been added if this change should be back-ported
Tests were added or are not required
Did the new or modified tests pass consistently locally and on the CI
Documentation was added or is not required

vitess-bot · 2023-08-07T17:47:26Z

Signed-off-by: Andrew Mason <[email protected]>

ajm188 · 2023-08-07T17:48:46Z

go/cmd/vtctldclient/command/onlineddl.go

+
+var (
+	OnlineDDL = &cobra.Command{
+		Use:                   "OnlineDDL <cmd> <keyspace> [args]",


unfortunately cobra won't support a positional argument before the subcommand name (or at least i couldn't see an easy/obvious way)

ajm188 · 2023-08-07T17:49:36Z

go/cmd/vtctldclient/command/onlineddl.go

+		}
+		fmt.Printf("%s\n", data)
+	default:
+		// TODO: support tabular/textual format


I want to at least best-effort match the format that a mysql query provides, which we have currently lost in the new client

go/cmd/vtctldclient/command/onlineddl.go

ajm188 · 2023-08-07T17:50:34Z

go/sqltypes/value.go

+// ToFloat32 first parses the value as MySQL would return it as a float64, then
+// downcasts it to a float32. It may overflow if the value exceeds 32 bits.
+func (v Value) ToFloat32() (float32, error) {


it would be a bit clunkier for me to do this inline in grpcvtctldserver, but this is also very much not a safe cast to do in general so i'm happy to move it there and grumble about the clunkiness :)

No strong opinions. If we keep this here, though, people may start using the function without realizing it loses precision. So perhaps better off moved to vtctld

actually, using float32(.AsFloat64()) works just as well for me

ajm188 · 2023-08-07T17:51:01Z

go/vt/vtctl/grpcvtctldserver/query.go

+		return nil, err
+	}
+
+	// liveness_indicator row[13]


this is still missing as i'm not sure what it is/is for

vitess_liveness_indicator is a column used internally for keeping track of vreplication "liveness", so this is only relevant for vreplication migrations, ie ALTER TABLE with online/vitess strategy.

vreplication liveness is based on the last update received by vreplication: either a processed binlog entry, or a heartbeat, or a new read transaction. It basically means vreplication is ongoing and not stuck.

The column is pretty much internal because it is then used to set liveness_timestamp. We can choose to export it or to hide it, at your discretion.

Let's include it in the proto and server response, and we can decide later to include it in whether the client actually shows it to the user. But at least it will be in the protobuf definition so folks with more advanced needs can pry into it (and, it sounds like it's not going anywhere so we're not going to need to deprecate/remove from the proto message anytime soon)

ajm188 · 2023-08-07T17:51:28Z

go/vt/vtctl/grpcvtctldserver/query.go

+	}
+
+	sm.MigrationContext = row[23].ToString()
+	// ddl_action row[24]


also still missing because i'm debating whether it's worth defining an enum for or just dealing with the raw string

IMHO raw string is fine.

ajm188 · 2023-08-07T17:52:22Z

go/vt/vtctl/grpcvtctldserver/query.go

+	switch s[len(s)-1] {
+	case '0', '1', '2', '3', '4', '5', '6', '7', '8', '9':
+		s += defaultUnit


this could use some actual code commentary, but, essentially: "if the string ends with a digit, it's missing a duration unit, so stick on our default"

go/cmd/vtctldclient/command/onlineddl.go

ajm188 · 2023-08-08T10:15:11Z

go/vt/vtctl/grpcvtctldserver/query.go

+}
+
+func rowToSchemaMigration(row sqltypes.Row) (sm *vtctldatapb.SchemaMigration, err error) {
+	row = row[1:] // We're off by 1 because of the extra Tablet field added to the grouped query result


we're also not currently doing anything with that extra Tablet field, which is a net-loss from the old output (i'm not necessarily saying that's a bad thing, just noting it so everyone is aware of this. we can put it back, or leave it off, or mystery 3rd option)

Not sure if we should use the Tablet field, as it just happens to be the PRIMARY of the shard, but does not otherwise have any specific meaning to the results.

Signed-off-by: Andrew Mason <[email protected]>

shlomi-noach

Please see inline comments for suggestions. Otherwise looks good!

shlomi-noach · 2023-08-08T14:58:21Z

go/sqltypes/value.go

+// ToFloat32 first parses the value as MySQL would return it as a float64, then
+// downcasts it to a float32. It may overflow if the value exceeds 32 bits.
+func (v Value) ToFloat32() (float32, error) {


No strong opinions. If we keep this here, though, people may start using the function without realizing it loses precision. So perhaps better off moved to vtctld

shlomi-noach · 2023-08-08T15:00:44Z

go/vt/vtctl/grpcvtctldserver/query.go

+)
+
+// 2023-08-05 07:15:16
+const mysqlTimestampLayout = "2006-01-02 15:04:05"


Better reuse sqltypes.TimestampFormat from go/sqltypes/type.go. See also TimestampFormatPrecision3 and TimestampFormatPrecision6.

shlomi-noach · 2023-08-08T15:03:42Z

go/vt/vtctl/grpcvtctldserver/query.go

+}
+
+func rowToSchemaMigration(row sqltypes.Row) (sm *vtctldatapb.SchemaMigration, err error) {
+	row = row[1:] // We're off by 1 because of the extra Tablet field added to the grouped query result


Not sure if we should use the Tablet field, as it just happens to be the PRIMARY of the shard, but does not otherwise have any specific meaning to the results.

shlomi-noach · 2023-08-08T15:10:50Z

go/vt/vtctl/grpcvtctldserver/query.go

+		return nil, err
+	}
+
+	return sm, nil


Let's switch away from row[42].ToInt64() to row.AsInt64("column_name", <defaultvalue>). To do that, use rows.Named() which give you a slice of decorated row objects. See for example:

vitess/go/vt/vttablet/onlineddl/executor.go

Lines 3496 to 3503 in 30d90fb

for _, row := range r.Named().Rows {

uuid := row["migration_uuid"].ToString()

onlineDDL, migrationRow, err := e.readMigration(ctx, uuid)

if err != nil {

return countRunnning, cancellable, err

}

postponeCompletion := row.AsBool("postpone_completion", false)

elapsedSeconds := row.AsInt64("elapsed_seconds", 0)

The reason we should make the columns named rather than use their indexes, is that nothing forces me to keep table columns at their current order. I may, and will, add new columns in the middle, or rearrange their order. So we can't rely on the column positioning.

All that assuming, of course, you're running a SELECT *. But if you SELECT this_column, that_column and want to rely on that order, feel free to ignore the comment.

no this is way better! i'm new to Named() this is awesome

Signed-off-by: Andrew Mason <[email protected]>

shlomi-noach

Changes look awesome!

Signed-off-by: Andrew Mason <[email protected]>

ajm188 · 2023-08-15T11:48:52Z

Latest SQL format output example:

$ vtctldclient --server "localhost:15999" OnlineDDL show commerce
+--------------------------------------+----------+-------+--------------+-------------+--------------------------------+----------+---------+---------------------+---------------------+-----------------+---------------------+---------------------+---------------------+-------------------+----------+----------+-------------------------------------------------------------+---------+------------------+----------------+----------+--------------------------------------------+------------+--------------------------------------------------------------+-------------+-------------+------------+-------------------+---------------------+----------+----------------------------+---------------------+--------------------------+---------------------------------+-----------------------+------------------+------------------+---------------+---------+-------------------+---------------------------+---------------------+--------------+--------------------------+---------------------+---------------------+-----------------+--------------------+------------------+------------------------+---------------------+-----------------------------+
|            migration_uuid            | keyspace | shard | mysql_schema | mysql_table |      migration_statement       | strategy | options |   added_timestamp   | requested_timestamp | ready_timestamp |  started_timestamp  | liveness_timestamp  | completed_timestamp | cleanup_timestamp |  status  | log_path |                          artifacts                          | retries |      tablet      | tablet_failure | progress |             migration_context              | ddl_action |                           message                            | eta_seconds | rows_copied | table_rows | added_unique_keys | removed_unique_keys | log_file | artifact_retention_seconds | postpone_completion | removed_unique_key_names | dropped_no_default_column_names | expanded_column_names | revertible_notes | allow_concurrent | reverted_uuid | is_view | ready_to_complete | vitess_liveness_indicator | user_throttle_ratio | special_plan | last_throttled_timestamp | component_throttled | cancelled_timestamp | postpone_launch |       stage        | cutover_attempts | is_immediate_operation | reviewed_timestamp  | ready_to_complete_timestamp |
+--------------------------------------+----------+-------+--------------+-------------+--------------------------------+----------+---------+---------------------+---------------------+-----------------+---------------------+---------------------+---------------------+-------------------+----------+----------+-------------------------------------------------------------+---------+------------------+----------------+----------+--------------------------------------------+------------+--------------------------------------------------------------+-------------+-------------+------------+-------------------+---------------------+----------+----------------------------+---------------------+--------------------------+---------------------------------+-----------------------+------------------+------------------+---------------+---------+-------------------+---------------------------+---------------------+--------------+--------------------------+---------------------+---------------------+-----------------+--------------------+------------------+------------------------+---------------------+-----------------------------+
| b3944b78_3b55_11ee_9603_2e0cac6950de | commerce |     0 | vt_commerce  | customer    | alter table customer rename    | vitess   |         | 2023-08-15 06:22:56 | 2023-08-15 06:22:57 |                 | 2023-08-15 06:22:58 | 2023-08-15 06:22:58 |                     |                   | failed   |          | _b3944b78_3b55_11ee_9603_2e0cac6950de_20230815062257_vrepl, |       0 | zone1-0000000101 |                |        0 | vtctl:b391c95c-3b55-11ee-9603-2e0cac6950de | alter      | Unknown column 'last_name' in                                |          -1 |           0 |          0 |                 0 |                   0 |          |                      86400 |                     |                          |                                 |                       |                  |                  |               |         |                   |                         0 |                   0 |              |                          |                     |                     |                 |                    |                0 |                        | 2023-08-15 06:22:58 |                             |
|                                      |          |       |              |             | column last_name to lname      |          |         |                     |                     |                 |                     |                     |                     |                   |          |          |                                                             |         |                  |                |          |                                            |            | '_b3944b78_3b55_11ee_9603_2e0cac6950de_20230815062257_vrepl' |             |             |            |                   |                     |          |                            |                     |                          |                                 |                       |                  |                  |               |         |                   |                           |                     |              |                          |                     |                     |                 |                    |                  |                        |                     |                             |
|                                      |          |       |              |             |                                |          |         |                     |                     |                 |                     |                     |                     |                   |          |          |                                                             |         |                  |                |          |                                            |            | (errno 1054) (sqlstate 42S22) during query: ALTER TABLE      |             |             |            |                   |                     |          |                            |                     |                          |                                 |                       |                  |                  |               |         |                   |                           |                     |              |                          |                     |                     |                 |                    |                  |                        |                     |                             |
|                                      |          |       |              |             |                                |          |         |                     |                     |                 |                     |                     |                     |                   |          |          |                                                             |         |                  |                |          |                                            |            | `_b3944b78_3b55_11ee_9603_2e0cac6950de_20230815062257_vrepl` |             |             |            |                   |                     |          |                            |                     |                          |                                 |                       |                  |                  |               |         |                   |                           |                     |              |                          |                     |                     |                 |                    |                  |                        |                     |                             |
|                                      |          |       |              |             |                                |          |         |                     |                     |                 |                     |                     |                     |                   |          |          |                                                             |         |                  |                |          |                                            |            | RENAME COLUMN `last_name` TO `lname`, ALGORITHM = copy       |             |             |            |                   |                     |          |                            |                     |                          |                                 |                       |                  |                  |               |         |                   |                           |                     |              |                          |                     |                     |                 |                    |                  |                        |                     |                             |
| dba5a774_3b55_11ee_9603_2e0cac6950de | commerce |     0 | vt_commerce  | customer    | alter table customer rename    | vitess   |         | 2023-08-15 06:24:03 | 2023-08-15 06:24:04 |                 | 2023-08-15 06:24:05 | 2023-08-15 06:24:11 | 2023-08-15 06:24:11 |                   | complete |          | _dba5a774_3b55_11ee_9603_2e0cac6950de_20230815062404_vrepl, |       0 | zone1-0000000101 |                |      100 | vtctl:dba393da-3b55-11ee-9603-2e0cac6950de | alter      |                                                              |           0 |           0 |          0 |                 0 |                   0 |          |                      86400 |                     |                          |                                 |                       |                  |                  |               |         |                   |                1692095048 |                   0 |              |                          |                     |                     |                 | re-enabling writes |                1 |                        | 2023-08-15 06:24:05 | 2023-08-15 06:24:09         |
|                                      |          |       |              |             | column last_name to lname      |          |         |                     |                     |                 |                     |                     |                     |                   |          |          |                                                             |         |                  |                |          |                                            |            |                                                              |             |             |            |                   |                     |          |                            |                     |                          |                                 |                       |                  |                  |               |         |                   |                           |                     |              |                          |                     |                     |                 |                    |                  |                        |                     |                             |
+--------------------------------------+----------+-------+--------------+-------------+--------------------------------+----------+---------+---------------------+---------------------+-----------------+---------------------+---------------------+---------------------+-------------------+----------+----------+-------------------------------------------------------------+---------+------------------+----------------+----------+--------------------------------------------+------------+--------------------------------------------------------------+-------------+-------------+------------+-------------------+---------------------+----------+----------------------------+---------------------+--------------------------+---------------------------------+-----------------------+------------------+------------------+---------------+---------+-------------------+---------------------------+---------------------+--------------+--------------------------+---------------------+---------------------+-----------------+--------------------+------------------+------------------------+---------------------+-----------------------------+

bcc4dac...73c485a is the marshalling stuff. i can extract that into a separate PR if folks would like, but it is only used here

Signed-off-by: Andrew Mason <[email protected]>

shlomi-noach · 2023-08-16T10:36:30Z

I love the marshaling! I'll definitely use it elsewhere (and would have used it in the past). It's fine to keep it as part of this PR.

mattlord

Nice work on this! I only had minor comments/questions/suggestions so will approve and you can resolve those as you see fit.

How were you planning to manage the release summary? Once we're (hopefully) done with the OnlineDDL and VReplication work, we could add one (both) or two (individual) entries in the summary.md file about support for these being added in vtctldclient.

mattlord · 2023-08-17T14:29:48Z

go/sqltypes/marshal.go

+	"vitess.io/vitess/go/protoutil"
+	"vitess.io/vitess/go/vt/vterrors"
+
+	querypb "vitess.io/vitess/go/vt/proto/query"
+	"vitess.io/vitess/go/vt/proto/vtrpc"
+	"vitess.io/vitess/go/vt/proto/vttime"


I'm surprised this made it past you! 😄 It's a nit, I know, but we could use:

"vitess.io/vitess/go/vt/proto/vttime" "vitess.io/vitess/go/protoutil" "vitess.io/vitess/go/vt/vterrors" querypb "vitess.io/vitess/go/vt/proto/query" vtrpcpb "vitess.io/vitess/go/vt/proto/vtrpc"

dang it! i blame vscode and me moving files around too much :P

oh, actually, vttime is proto, so just changing the vtrpc to be named with pb (weirdly we don't do that anywhere with vttime 🤷 )

Yeah, it's fine either way. I don't know why our standard use of vttime is non-standard :D

go/sqltypes/marshal.go

mattlord · 2023-08-17T14:32:45Z

go/sqltypes/marshal.go

go/vt/vtctl/schematools/marshal.go

go/vt/vtctl/schematools/schematools.go

mattlord · 2023-08-17T15:49:49Z

go/vt/vtctl/schematools/schematools.go

+		// TODO (andrew): Remove special handling for lower/uppercase and
+		// gh-ost=>ghost/pt-osc=>ptosc support. (file issue for this).


Is there an issue already? Before we forget 🙂

mattlord · 2023-08-17T15:50:27Z

go/vt/vtctl/schematools/schematools.go

+	if upperName != name {
+		// TODO (andrew): Remove special handling for lower/uppercase and
+		// gh-ost=>ghost/pt-osc=>ptosc support. (file issue for this).
+		log.Warningf("detected legacy strategy name syntax; parsed %q as %q. this will break in the next version.", upperName, name)


This should probably be more prominent, no? IMO this would also warrant notes in the release summary.

i actually think we maybe don't want to warn at all, and instead accept both (since to truly deprecate this we'd also have to change the strings that vttablet uses to write to the table)

Just offering my take, people should use gh-ost and pt-osc and we don't need to cover for spelling mistakes or upper/lower.

Yep, I'm removing these warnings now. My question now though is, should we update SchemaMigrationStrategyName to re-dash the GHOST and PTOSC enums? In other words, should this pass?

assert.Equal(t, "gh-ost", SchemaMigrationStrategyName(vtctldatapb.SchemaMigrationStrategy_GHOST) assert.Equal(t, "pt-osc", SchemaMigrationStrategyName(vtctldatapb.SchemaMigrationStrategy_PTOSC)

Yes, I think this should pass. gh-ost is the official name, ghost never was. And we choose the same notation for pt-osc (FYI: we will deprecate support for pt-osc sometime soon).

go/vt/vtctl/schematools/schematools.go

Signed-off-by: Andrew Mason <[email protected]>

ajm188 · 2023-08-17T22:48:10Z

How were you planning to manage the release summary? Once we're (hopefully) done with the OnlineDDL and VReplication work, we could add one (both) or two (individual) entries in the summary.md file about support for these being added in vtctldclient.

I think let's do a "new in vtctld server/client section" with bullets for each. How does that sound to you?

Signed-off-by: Andrew Mason <[email protected]>

mattlord · 2023-08-18T17:21:41Z

How were you planning to manage the release summary? Once we're (hopefully) done with the OnlineDDL and VReplication work, we could add one (both) or two (individual) entries in the summary.md file about support for these being added in vtctldclient.

I think let's do a "new in vtctld server/client section" with bullets for each. How does that sound to you?

Sure, sounds good.

Signed-off-by: Andrew Mason <[email protected]>

* fix conflicts * merge origin v18.0.0 * Back to dev mode after v18.0.0 (vitessio#14475) * Release of v18.0.0 (vitessio#14406) * [release-18.0] VReplication: Handle multiple streams in UpdateVReplicationWorkflow RPC (vitessio#14447) (vitessio#14468) * [release-18.0] VDiff: "show all" should only report vdiffs for the specified keyspace and workflow (vitessio#14442) (vitessio#14466) * [release-18.0] TestStreamMigrateMainflow: fix panic in test (vitessio#14425) * [release-18.0] vtgate: Allow more errors for the warning check (vitessio#14421) (vitessio#14423) * [release-18.0] servenv: Remove double close() logic (vitessio#14457) (vitessio#14459) * [release-18.0] viper: register dynamic config with both disk and live (vitessio#14453) (vitessio#14455) * [release-18.0] vtgate/engine: Fix race condition in join logic (vitessio#14435) (vitessio#14441) * tx_throttler: remove topo watchers metric (vitessio#14444) * [release-18.0] VDiff tablet selection: pick non-serving tablets in Reshard workflows (vitessio#14413) (vitessio#14418) * Code freeze of release-18.0 (vitessio#14405) * [release-18.0] Online DDL: lint DDL strategy flags (vitessio#14373) (vitessio#14399) * [release-18.0] tuple: serialized form (vitessio#14392) (vitessio#14394) * [release-18.0] schemadiff: fix missing `DROP CONSTRAINT` in duplicate/redundant constraints scenario. (vitessio#14387) (vitessio#14391) * [release-18.0] Performance Fixes for Vitess 18 (vitessio#14383) (vitessio#14393) * [release-18.0] Vtctld SwitchReads: fix bug where writes were also being switched as part of switching reads when all traffic was switched using SwitchTraffic (vitessio#14360) (vitessio#14379) * [release-18.0] VDiff: wait for shard streams of one table diff to complete for before starting that of the next table (vitessio#14345) (vitessio#14382) * [release-18.0] [Docker] Fix VTadmin build (vitessio#14363) (vitessio#14378) * [release-18.0] Fix backup on s3 like storage (vitessio#14311) (vitessio#14362) * [release-18.0] use aggregation engine over distinct engine when overlapping order by (vitessio#14359) (vitessio#14361) * [release-18.0] Automatic approval of `vitess-bot` clean backports (vitessio#14352) (vitessio#14357) * [release-18.0] evalengine: Misc bugs (vitessio#14351) (vitessio#14354) * [release-18.0] OnlineDDL: reduce vrepl_stress workload in forks (vitessio#14302) (vitessio#14349) * [release-18.0] VReplication: Add --all-cells flag to create sub-commands (vitessio#14341) (vitessio#14343) * [release-18.0] Incremental backup: fix race condition in reading 'mysqlbinlog' output (vitessio#14330) (vitessio#14335) * [release-18.0] release notes: edit summary for consistency (vitessio#14319) (vitessio#14320) * [release-18.0] Bump @babel/traverse from 7.21.4 to 7.23.2 in /web/vtadmin (vitessio#14304) (vitessio#14308) * [release-18.0] Rename Foreign Key enum values in VSchema and drop `FK_` prefix (vitessio#14274) (vitessio#14299) * [release-18.0] VReplication: error on vtctldclient commands w/o tablet types (vitessio#14294) (vitessio#14298) * [release-18.0] Make vtctldclient mount command more standard (vitessio#14281) (vitessio#14283) * [release-18.0] VReplication: Add traffic state to vtctldclient workflow status output (vitessio#14280) (vitessio#14282) * [release-18.0] fix: analyze statement parsing and planning (vitessio#14268) (vitessio#14275) * [release-18.0] Bypass cobra completion commands so they still function (vitessio#14217) (vitessio#14234) * [release-18.0] Bump golang.org/x/net from 0.14.0 to 0.17.0 (vitessio#14260) (vitessio#14264) * Add vtctldclient info to the 18.0 summary (vitessio#14259) * [release-18.0] Bump postcss from 8.4.21 to 8.4.31 in /web/vtadmin (vitessio#14173) (vitessio#14258) * [release-18.0] Bump github.com/cyphar/filepath-securejoin from 0.2.3 to 0.2.4 (vitessio#14239) (vitessio#14253) * [release-18.0] Tablet throttler: fix race condition by removing goroutine call (vitessio#14179) (vitessio#14198) * [release-18.0] fix: insert with negative value (vitessio#14244) (vitessio#14247) * [release-18.0] Fix anonymous paths in cobra code-gen (vitessio#14185) (vitessio#14238) * [release-18.0] Throttler: set timeouts on gRPC communication and on topo communication (vitessio#14165) (vitessio#14167) * [release-18.0] Move all examples to vtctldclient (vitessio#14226) (vitessio#14241) * [release-18.0] VReplication: Add missing info to vtctldclient workflow SHOW output (vitessio#14225) (vitessio#14240) * [release-18.0] Upgrade the Golang version to `go1.21.3` (vitessio#14230) * [release-18.0] Optimize the GetWorkflows RPC (vitessio#14212) (vitessio#14233) * [Release 18.0]: Online DDL: timeouts for all gRPC calls (vitessio#14182) (vitessio#14189) * [release-18.0] Migrate Materialize command to vtctldclient (vitessio#14184) (vitessio#14214) * [Release 18.0] Backport of #17174 (vitessio#14210) * [release-18.0] Upgrade the Golang version to `go1.21.2` (vitessio#14195) * [release-18.0] Migrate CreateLookupVindex and ExternalizeVindex to vtctldclient (vitessio#14086) (vitessio#14183) * Back to dev mode after `v18.0.0-rc1` release (vitessio#14169) * Release of v18.0.0-rc1 (vitessio#14136) * [release-18.0] docker: add dedicated vtorc container (vitessio#14126) (vitessio#14148) * [release-18.0] gen4: Support explicit column aliases on derived tables (vitessio#14129) (vitessio#14156) * Code freeze of release-18.0 (vitessio#14131) * VTGate FK stress tests suite: improvements (vitessio#14098) * DDL execution to commit open transaction (vitessio#14110) * Summary changes for foreign keys (vitessio#14112) * Move subqueries to use the operator model (vitessio#13750) * servenv: Allow for explicit bind address (vitessio#13188) * vtorc: add detected_problems counter (vitessio#13967) * fix bad copy-paste in zkctld docgen (vitessio#14123) * VDiff: Cleanup the controller for a VDiff before deleting it (vitessio#14107) * Remove deprecated flags before `v18.0.0` (vitessio#14071) * Remove FOSSA Test from CI until we can do it in a secure way (vitessio#14119) * go/cmd/vtbackup: wait for plugins to finish initializing (vitessio#14113) * bugfix: change column name and type to json (vitessio#14093) * `vtctld`/`vtorc`: improve reparenting stats (vitessio#13723) * E2E Fuzzing testing for foreign keys (vitessio#13980) * Backup/restore: provision and restore a tablet with point-in-time recovery flags (vitessio#13964) * anonymize homedirs in generated docs (vitessio#14101) * VDiff: properly split cell values in record when using TabletPicker (vitessio#14099) * miscellaneous cobras (vitessio#14069) * Reduce network pressure on multi row insert (vitessio#14064) * [cli] cobra zookeeper (vitessio#14094) * switch casing in onlineddl subcommand help text (vitessio#14091) * Fix Fk verification and update queries to accommodate for bindVariables being NULL (vitessio#14061) * actually test vtcombo (vitessio#14095) * VDiff: Migrate client command to vtctldclient (vitessio#13976) * ci: pool-related test flakyness (vitessio#14076) * Improve the rewriter to simplify more queries (vitessio#14059) * update docgen to embed commit ID in autogenerated doc frontmatter (vitessio#14056) * [CLI] cobra lots of things (vitessio#14007) * Add VSchema DDL support for dropping sequence and auto increment (vitessio#13882) * json: Fix quoting JSON keys (vitessio#14066) * Flakes: Address TestMigrate Failures (vitessio#12866) * [cli] migrate mysqlctl and mysqlctld to cobra (vitessio#13946) * evalengine: Mark UUID() function as non-constant (vitessio#14051) * Fix cascading Delete failure while using Prepared statements (vitessio#14048) * remove query_analyzer binary and release (vitessio#14055) * go/vt/mysqlctl: instrument s3 upload time (vitessio#12500) * Add session flag for stream execute grpc api (vitessio#14046) * Endtoend: stress tests for VTGate FOREIGN KEY support (vitessio#13799) * metrics: change vtbackup_duration_by_phase to binary-valued vtbackup_phase (vitessio#12973) * TableGC: support DROP VIEW (vitessio#14020) * Support arbitrary ZooKeeper config lines (vitessio#13829) * Bump protobufjs from 7.2.3 to 7.2.5 in /web/vtadmin (vitessio#13833) * Bump tough-cookie and @cypress/request in /vitess-mixin/e2e (vitessio#13768) * Remove excessive logging in transactions (vitessio#14021) * Fix bug in `fileNameFromPosition` test helper (vitessio#13778) * MoveTables Cancel: drop denied tables on target when dropping source/target tables (vitessio#14008) * java: update to latest dependencies for grpc and protobuf (vitessio#13996) * OnlineDDL: cleanup cancelled migration artifacts; support `--retain-artifacts=<duration>` DDL strategy flag (vitessio#14029) * moved timeout test to different package (vitessio#14028) * fix: cost to include subshard opcode (vitessio#14023) * VReplication VPlayer: set foreign_key_checks on initialization (vitessio#14013) * go/cmd/vtbackup: report replication status metrics during catch-up phase (vitessio#13995) * Fix the `SELECT` query we run on the child table to verify that update is allowed on a RESTRICT constraint (vitessio#13991) * fix data race in join engine primitive olap streaming mode execution (vitessio#14012) * test: added test to check binlogs to contain the cascade events (vitessio#13970) * Fix `TestLeftJoinUsingUnsharded` and remove instability when running E2E locally (vitessio#13973) * Enable failures in `tools/e2e_test_race.sh` and fix races (vitessio#13654) * Improve release process documentation (vitessio#14000) * Make `Static Code Checks Etc` fail if the `./changelog` folder is out-of-date (vitessio#14003) * Fix foreign key plan tests expectation (vitessio#13997) * Flakes: Add recently added 'select rows_copied' query to ignore list (vitessio#13993) * Foreign key cascade: retain "for update" lock on select query plans (vitessio#13985) * Fix `NOT IN` expression used in the SET NULL for a child table on an update (vitessio#13988) * consolidate docs (vitessio#13959) * VDiff: correct handling of default source and target cells (vitessio#13969) * [cobra] vtgate and vttablet (vitessio#13943) * handle large number of predicates without timing out (vitessio#13979) * Fix missing deprecated flags in `vttablet` and `vtgate` (vitessio#13975) * wrangler,workflow/workflow: materialize from intersecting source shards based on primary vindexes (vitessio#13782) * Cache v3 (vitessio#13939) * Implement Reshard in vtctldclient (vitessio#13792) * Disallow Insert with Duplicate key update and Replace Into queries on foreign key column, set locks on fk queries (vitessio#13953) * End to end testing suite for foreign keys (vitessio#13870) * copy over existing vreplication rows copied to local counter if resuming from another tablet (vitessio#13949) * vtctldclient OnlineDDL: support `throttle`, `unthrottle` (vitessio#13916) * Change internal vindex type recommendation for integrals to xxhash (vitessio#13956) * VTOrc converts a tablet to DRAINED type if it detects errant GTIDs on it (vitessio#13873) * Fix `ApplySchema --batch-size` with ` --allow-zero-in-date` (vitessio#13951) * Tablet throttler: empty list of probes on non-leader (vitessio#13926) * VReplication: Handle SQL NULL and JSON 'null' correctly for JSON columns (vitessio#13944) * [vtctld] more cobra binaries (vitessio#13930) * Merge branch 'planetscale-fk-verify-update-planning' * [main] Upgrade the Golang version to `go1.21.1` (vitessio#13933) * update fk error messages * cascade with new value should work * small refactor - removed unused code * return verify error based on type * store the verification type * update plan output test * small refactor * feat: add code to verify valdity of ON UPDATE RESTRICT foreign keys wherever required * Rewrite `USING` to `ON` condition for joins (vitessio#13931) * test: fix tests to reflect the recent changes * feat: add code to verify update with cascade run with foreign key checks 0 and validate all the foreign keys on vtgate * add where condition to fk verify query * changed parent verification query and accordingly change the engine primitive for foreign key constraint verification * foreign key verify operator and logical * refactor: add comment explaining map in input * add support for foreign key constraint verify on update * refactor: move DML logic to sql_builder.go (vitessio#13920) * OnlineDDL: fix nil 'completed_timestamp' for cancelled migrations (vitessio#13928) * Silence 'CheckThrottler' gRPC calls (vitessio#13925) * migrate vtorc to use cobra commands (vitessio#13917) * MoveTables: allow copying all tables in a single atomic copy phase cycle (vitessio#13137) * proto: Faster clone (vitessio#13914) * Properly support ignore_nulls in CreateLookupVindex (vitessio#13913) * [staticcheck] Last few staticchecks! (vitessio#13909) * icuregex: Update to ICU 73 (vitessio#13912) * gen4: Fast aggregations (vitessio#13904) * Use correct syntax in test (vitessio#13907) * Misc Local Install improvements. (vitessio#13446) * Consolidate helper functions for working with proto3 time messages (vitessio#13905) * Add vtsql flags to vtadmin (vitessio#13674) * MoveTables: add flag to specify that routing rules should not be created when a movetables workflow is created (vitessio#13895) * [staticcheck] miscellaneous tidying (vitessio#13892) * [misc] tidy imports (vitessio#13885) * Remove duplicate ACL check in tabletserver handleHTTPConsolidations (vitessio#13876) * vtexplain: Fix passing through context for cleanup (vitessio#13900) * [staticcheck] Cleanup deprecations (vitessio#13898) * inputs method to return additional information about the input primitive (vitessio#13883) * vttablet: do not notify `vtgate` about internal tables (vitessio#13897) * Skip launchable if the Pull Request is marked as a Draft (vitessio#13886) * vtctldclient: support OnlineDDL `complete`, `launch` commands (vitessio#13896) * collations: implement collation dumping as a docker image (vitessio#13879) * sqlparser: Tablespace option is case sensitive (vitessio#13884) * Clean up deprecated slice header usage and unused code (vitessio#13880) * [misc] Delete more unused functions, tidy up dupe imports (vitessio#13878) * collations: Refactor to separate basic collation information from data (vitessio#13868) * [wrangler] cleanup unused functions (vitessio#13867) * Fix setup order to avoid races (vitessio#13871) * Add Foreign key verify constraint engine primitive (vitessio#13848) * Foreign key cascade planning for DELETE and UPDATE queries (vitessio#13823) * Fix merge conflict with new tests (vitessio#13869) * vtctldclient OnlineDDL CANCEL (vitessio#13860) * Add leak checking for vtgate tests (vitessio#13835) * Fix for "text type with an unknown/unsupported collation cannot be hashed" error (vitessio#13852) * Go 1.21 cleanups (vitessio#13862) * VTGate Buffering: Use a more accurate heuristic for determining if we're doing a reshard (vitessio#13856) * docker/bootstrap: remove --no-cache flag (vitessio#13785) * Flakes: Improve reliability of vreplication_copy_parallel test (vitessio#13857) * [main] Upgrade the Golang version to `go1.21.0` (vitessio#13853) * Fix regular expression issue in Golang Upgrade and remove `release-14.0` from target branch (vitessio#13846) * Migrates most workflows to 4 and 16 cores Large GitHub-Hosted-Runners (vitessio#13845) * [onlineddl] retry and cleanup (vitessio#13830) * Bump word-wrap from 1.2.3 to 1.2.4 in /web/vtadmin (vitessio#13569) * Bump tough-cookie from 4.1.2 to 4.1.3 in /web/vtadmin (vitessio#13767) * Skip VTAdmin build in Docker tests (vitessio#13836) * Flakes: Synchronize access to logErrStacks in vterrors (vitessio#13827) * Flakes: VReplication unit tests: reduce goroutine leakage (vitessio#13824) * Remove explicit usage of etcd v2 (api and storage) (vitessio#13791) * Add OnlineDDL show support (vitessio#13738) * VReplication: Improve MoveTables Create Error Handling (vitessio#13737) * Add 2 new metrics with tablet type labels (vitessio#13521) * Add 2 more durability policies that allow RDONLY tablets to send semi-sync ACKs (vitessio#13698) * CI: Misc test improvements to limit failures with various runners (vitessio#13825) * Flakes: empty vtdataroot before starting a new vreplication e2e test (vitessio#13803) * Fix `BackupShard` to get its options from its own flags (vitessio#13813) * Flakes: skip flaky check that ETA for a VReplication VDiff2 Progress command is in the future. (vitessio#13804) * Copy release notes for v17.0.2 and v16.0.4 (vitessio#13811) * More union merging (vitessio#13743) * Add Foreign key Cascade engine primitive (vitessio#13802) * Add support for tuple as value type (vitessio#13800) * Foreign Keys: `UPDATE` planning (vitessio#13762) * Improving random query generation for endtoend testing (vitessio#13460) * Flakes: Delete VTDATAROOT files in reparent test teardown within CI (vitessio#13793) * vtgate: fix race condition iterating tables and views from schema tracker (vitessio#13673) * Fixing `backup_pitr` flaky tests via wait-for loop on topo reads (vitessio#13781) * Do not drain tablet in incremental backup (vitessio#13773) * Address vttablet memory usage with backups to Azure Blob Service (vitessio#13770) * Run auto golang upgrade only on vitessio/vitess (vitessio#13766) * Flakes: remove non-determinism from vtctldclient MoveTables unit test (vitessio#13765) * Minor --initialize-target-sequences followups (vitessio#13758) * CI: fix onlineddl_scheduler flakiness (vitessio#13754) * VReplication: Initialize Sequence Tables Used By Tables Being Moved (vitessio#13656) * Bump docker images to `bullseye` (vitessio#13664) * Use NodeJS v18 in VTAdmin Dockerfile (vitessio#13751) * Refactor Expression and Statement Simplifier (vitessio#13636) * Foreign Keys: `DELETE` planning (vitessio#13746) * build: Allow passing in custom -ldflags (vitessio#13748) * Point in time recovery: fix cross-tablet GTID evaluation (vitessio#13555) * Cache info schema table info (vitessio#13724) * BackupShard: support incremental backup (vitessio#13522) * OnlineDDL: support @@migration_context in vtgate session. Use if non-empty (vitessio#13675) * schemadiff: add time measure test for massive schema load and diff (vitessio#13697) * Vtgate: pass 'SHOW VITESS_MIGRATIONS' to tablet's query executor (vitessio#13726) * Move UNION planning to the operators (vitessio#13450) * Fix vtcombo DBDDL plugin race condition (vitessio#13117) * Foreign Keys: `INSERT` planning (vitessio#13676) * go/vt/vitessdriver: implement driver.{Connector,DriverContext} (vitessio#13704) * Backup: safe compressor/decompressor closure (vitessio#13668) * sqlparser: Track if original default value is a literal (vitessio#13730) * Solve RevertMigration.Comment read/write concurrency issue (vitessio#13700) * `ApplySchema`: support `--batch-size` flag in 'direct' strategy (vitessio#13693) * Fix closed channel `panic` in Online DDL cutover (vitessio#13729) * mysql: Refactor dependencies (vitessio#13688) * vtgate tablet gateway buffering: don't shutdown if not initialized (vitessio#13695) * [OnlineDDL] add label so break works as intended (vitessio#13691) * fastparse: Fix bug in overflow detection (vitessio#13702) * Refactor code to remove `evalengine` as a dependency of `VTOrc` (vitessio#13642) * `txthrottler`: remove `txThrottlerConfig` struct, rely on `tabletenv` (vitessio#13624) * [vtctldclient] flags need to be defined to be deprecated (vitessio#13681) * Add dry-run/monitoring-only mode for TxThrottler (vitessio#13604) * Enhancing VTGate buffering for MoveTables and Shard by Shard Migration (vitessio#13507) * Improvements to PRS (vitessio#13623) * Errant GTID Metrics Refactor (vitessio#13670) * Improve logging and renaming PrimaryTermStartTimestamp in vttablets (vitessio#13625) * Fix a couple of logs in VTOrc (vitessio#13667) * Throttler: exempt apps via `UpdateThrottlerConfig --throttle-app-exempt` (vitessio#13666) * Tablet throttler: inter-checks via gRPC (vitessio#13514) * Reroute 'ALTER VITESS_MIGRATION ... THROTTLE ...' through topo (vitessio#13511) * Build foreign key definition in schema tracker (vitessio#13657) * evalengine: Fix JSON weight string computation (vitessio#13669) * Throttler: verify deprecated flags are still allowed (vitessio#13615) * Backup & Restore: vtctldclient to support PITR flags (vitessio#13513) * `UpdateThrottlerConfig --unthrottle-app ...` (vitessio#13494) * tx throttler: healthcheck all cells if `--tx-throttler-healthcheck-cells` is undefined (vitessio#12477) * evalengine: Improve weight string support (vitessio#13658) * mysqlctl: Reduce logging for running commands (vitessio#13659) * Add v15.0.4, v16.0.3, and v17.0.1 changelogs (vitessio#13661) * [viper WatchConfig] platform-specific write to ensure callback fires exactly once (vitessio#13627) * go/mysql: switch to new API for x/exp/slices.SortFunc (vitessio#13644) * Per workload TxThrottler metrics (vitessio#13526) * VReplication: Make Source Tablet Selection More Robust (vitessio#13582) * Update known issues in `v16.x` and `v17.0.0` (vitessio#13618) * Augment VTOrc to also store the shard records and use it to better judge Primary recoveries (vitessio#13587) * icuregex: Lazy load ICU data into memory (vitessio#13640) * Ensure to call `servenv.Init` when needed (vitessio#13638) * GetSchema: limit concurrent operations (vitessio#13617) * check keyspace snapshot time if none specified for backup restores (vitessio#13557) * vtgate buffering logic: remove the deprecated healthcheck based implementation (vitessio#13584) * Fix type comparisons for Nullsafe* functions (vitessio#13605) * Reintroduce `TestReadOutdatedInstanceKeys` with debugging information (vitessio#13562) * mysqlctl: Remove noisy log line (vitessio#13599) * Vtctldclient MoveTables (vitessio#13015) * txthrottler: add metrics for topoWatcher and healthCheckStreamer (vitessio#13153) * Reduce usages of old horizon planning fallback (vitessio#13595) * Throttler: reintroduce deprecated flags so that deprecation actually works (vitessio#13597) * Incremental backup & recovery: restore-to-timestamp (vitessio#13270) * Fix potential panics due to "Fail in goroutine after test completed" (vitessio#13596) * ignore all error for views in engine reload (vitessio#13590) * vtgate table schema tracking to use GetSchema rpc (vitessio#13544) * MoveTables sequence e2e tests: change terminology to use basic vs simple everywhere for partial movetables workflows (vitessio#13435) * Skip VTAdmin build in more places (vitessio#13588) * Fix show character set (vitessio#13565) * Remove unused chromedriver (vitessio#13573) * fix TestQueryTimeoutWithTables flaky test (vitessio#13579) * CI: Fix make build related issues (vitessio#13583) * docker/mini: remove refs to orc configs (vitessio#13495) * stats: use *time.Ticker instead of time.After() (vitessio#13492) * VReplication: Ensure ROW events are sent within a transaction (vitessio#13547) * Add a `keyspace` configuration in the `vschema` for foreign key mode (vitessio#13553) * vreplication: Move to use collations package (vitessio#13566) * Flaky tests: Fix race in memory topo (vitessio#13559) * Fix flaky vtgate test TestInconsistentStateDetectedBuffering (vitessio#13560) * Flaky tests: Fix wrangler tests (vitessio#13568) * Optimize `make build` in `test.go` and in CI (vitessio#13567) * Unset the PREFIX environment variable when building VTAdmin (vitessio#13554) * vtctldclient: Add missing new backup option (vitessio#13543) * Skip flaky test `TestReadOutdatedInstanceKeys` (vitessio#13561) * Fix a number of encoding issues when evaluating expressions with the evalengine (vitessio#13509) * Deflake `TestPlannedReparentShardPromoteReplicaFail` (vitessio#13548) * [vipersync] deflake TestWatchConfig (vitessio#13545) * vtgate v3 planner removal (vitessio#13458) * Fix flakiness in VTOrc tests (vitessio#13489) * Replace deprecated `github.com/golang/mock` with `go.uber.org/mock` (vitessio#13512) * Fix dependencies in docker build script (vitessio#13520) * fix docgen for subcommands (vitessio#13518) * Merge pull request vitessio#13515 from planetscale/partial-movetables-traffic-status * Correct unit test * docker/k8s: add bookworm builds (vitessio#13436) * ignore ongoing backfill vindex from routing selection (vitessio#13505) * Adjust function names for improved clarity * More fixes related to partial traffic handling * Ignore unrelated shards in partial movetables workflow status * Better handling of vreplication setState() failure (vitessio#13488) * flags: Remove hardcoded runner paths (vitessio#13482) * viperutil: Remove potential cross site reflecting issue (vitessio#13483) * skip flaky test (vitessio#13501) * feat: remove --disable_active_reparents flag in vttablet-up.sh (vitessio#13504) * fix: error.as method usage to send pointer to the reference type expected. (vitessio#13496) * Fix `TestGatewayBufferingWhileReparenting` flakiness (vitessio#13469) * ApplySchema: deprecate '--allow_long_unavailability' flag (vitessio#10717) * Incremental backup: accept GTID position without 'MySQL56/' flavor prefix (vitessio#13474) * Deprecating and removing tablet throttler CLI flags and tests (vitessio#13246) * Online DDL: improved row estimation via ANALYE TABLE with --analyze-table strategy flag (vitessio#13352) * Tablet throttler: throttled app configuration via `vtctl UpdateThrottlerConfig` (vitessio#13351) * added no-commit-collection option to launchable record build command (vitessio#13490) * Fix remote VersionString API (vitessio#13484) * Fix logging by omitting the host and port in `SetReadOnly` (vitessio#13470) * `vtctl OnlineDDL`: complete command set (vitessio#12963) * Random selection of keyspace based on available tablet (vitessio#13359) * Improve and Fix Distinct Aggregation planner (vitessio#13466) * Fix ubi8.arm64.mysql80 build package mirrorserver error (vitessio#13431) * backup: Allow for upgrade safe backups (vitessio#13449) * Merge pull request vitessio#13468 from timvaillancourt/examples-compose-fix-consul-tag * Update a number of dependencies (vitessio#13031) * Enable Tcp keep alive and provide keep alive period setting (vitessio#13434) * rm mistaken commit * Update docker-compose.beginners.yml too * compose: fix `consul:latest` error * Add support for kill statement (vitessio#13371) * feat: remove excessive logging (vitessio#13459) * schema.Reload(): ignore column reading errors for views only, error for tables (vitessio#13442) * [CI] deflake viper sync tests (vitessio#13185) * mysql: introduce icuregex package (vitessio#13391) * Fix `Fakemysqldaemon` to store the host and port after `SetReplicationSource` call (vitessio#13439) * Examples: only terminate vtadmin if it was started (vitessio#13433) * [main] Upgrade-Downgrade Fix: Schema-initialization stuck on semi-sync ACKs while upgrading (vitessio#13411) (vitessio#13440) * Move more horizon planning to the operators (vitessio#13412) * Refactor backup_pitr into two distinct CI tests: builtin vs Xtrabackup (vitessio#13395) * Improve VTOrc logging statements, now that we have alias as a field (vitessio#13428) * vtorc: Cleanup more unused code (vitessio#13354) * VReplication Workflows: make sequence tables follow routing rules (vitessio#13238) * Ignore error while reading table data in Schema.Engine reload (vitessio#13421) * Improve time taken to run the examples by optimizing `vtadmin` build (vitessio#13262) * vtctl,vindexes: logs warnings and export stat for unknown vindex params (vitessio#13322) * Add end-of-life documentation + re-organize internal documentation (vitessio#13401) * Adding random query generation for endtoend testing of the Gen4 planner (vitessio#13260) * Aggregation engine refactor (vitessio#13378) * Deflake `TestQueryTimeoutWithDual` test (vitessio#13405) * Optimize release notes generation to use GitHub Milestones (vitessio#13398) * feat: don't run any reparent commands if the host is empty (vitessio#13396) * `vttestserver`: persist vschema changes in `--persistent_mode` (vitessio#13065) * Tablet throttler: only start watching SrvKeyspace once it's confirmed to exist (vitessio#13384) * Support views in BaseShowTablesWithSizes for MySQL 8.0 (vitessio#13394) * Fix incorrect output in release scripts (vitessio#13385) * Adds support for ANY_VALUE (vitessio#13342) * BaseShowTablesWithSizes: optimize MySQL 8.0 query (vitessio#13375) * Forward port of release notes changes from v17.0.0 GA (vitessio#13370) * feat: add timestamp to vtorc debug page (vitessio#13379) * Prevent resetting replication every time we set replication source (vitessio#13377) * Local example 101: idempotent on existing clusters (vitessio#13373) * governance: clean up language, link steering doc from first occurrence instead of from a random occurrence (vitessio#13337) * Add group_concat aggregation support (vitessio#13331) * Improve VTOrc failure detection to be able to better handle dead primary failures (vitessio#13190) * Improve lock action string (vitessio#13355) * Add RestorePosition and RestoredBackupTime as metrics to vttablet (vitessio#13339) * update link for reparenting guide (vitessio#13350) * Handle inconsistent state error in query buffering (vitessio#13333) * mysqlctl: Move more to use built in MySQL client (vitessio#13338) * VTOrc: Update the primary key for all the tables from `hostname, port` to `alias` (vitessio#13243) * docker/k8s: Cleanup done TODO (vitessio#13347) * bug: don't always wrap aggregation in coalesce (vitessio#13348) * Vttablet schema tracking: Fix _vt.schema_version corruption (vitessio#13045) * bugfixes: collection of fixes to bugs found while fuzzing (vitessio#13332) * Remove CI endtoend test for VReplication copy throttling (vitessio#13343) * Use sqlparser for all dynamic query building in VDiff2 (vitessio#13319) * txthrottler: verify config at vttablet startup, consolidate funcs (vitessio#13115) * mysqlctl: Use DBA connection for schema operations (vitessio#13178) * Bug fix: SQL queries erroring with message `unknown aggregation random` (vitessio#13330) * Support complex aggregation in Gen4's Operators (vitessio#13326) * [VTAdmin] Upgrade to use node 18.16.0 (vitessio#13288) * Cleanup unused Dockerfile entries (vitessio#13327) * Add metric for showing the errant GTIDs in VTOrc (vitessio#13281) * vtgate planner: HAVING in the new operator horizon planner (vitessio#13289) * vtgr: Remove deprecated vtgr (vitessio#13308) * mysqlctl: Correctly encode database and table names (vitessio#13312) * sqlparser: Add support for TIMESTAMPADD (vitessio#13314) * Refactor and add a comment to schema initialisation code (vitessio#13309) * Release notes for 17.0.0-rc2 (vitessio#13306) * remove os.Exit (vitessio#13310) * k8stopo: Remove the deprecated Kubernetes topo (vitessio#13303) * vindexes: return unknown params (vitessio#12951) * Operator planner refactor (vitessio#13294) * Deprecate VTGR (vitessio#13301) * k8stopo: Include deprecation warning (vitessio#13299) * [main] Upgrade the Golang version to `go1.20.5` (vitessio#13256) * evalengine: implement date/time math (vitessio#13274) * increase size of reparent_journal columns (vitessio#13287) * VReplication: Fix VDiff2 DeleteByUUID Query (vitessio#13255) * Miscellaneous code modifications based on observations made while doing a code walkthrough (vitessio#12873) * Set the number of threads for release notes generation with a flag (vitessio#13273) * Fix and Make aggregation planner handle aggregation functions better (vitessio#13228) * fix: ShardedRouting clone to clone slice of reference correctly (vitessio#13265) * Remove viper warnings from local examples (vitessio#13234) * Add flag to VTOrc to enable/disable its ability to run ERS (vitessio#13259) * Remove `out.txt` and add `release-17.0` to go upgrade automation (vitessio#13261) * build(deps-dev): bump vite from 4.2.1 to 4.2.3 in /web/vtadmin (vitessio#13240) * Copy v17.0.0-rc changelog to main (vitessio#13248) * fix: GetField to use existing session for query (vitessio#13219) * Fix flakiness in `TestDeadPrimaryRecoversImmediately` (vitessio#13232) * Augmenting the `GetSchema` RPC to also work for `Table` and `All` type of input (vitessio#13197) * Incremental backup and point in time recovery for XtraBackup (vitessio#13156) * VReplication: More intelligently manage vschema table entries on unsharded targets (vitessio#13220) * Use $hostname in vtadmin script as other scripts do (vitessio#13231) * Tablet throttler: throttler-config-via-topo defaults 'true', deprecation message for old flags (vitessio#13130) * [ci] add generator for templated flag testdata (vitessio#13150) * schemadiff: validating case-sensitive view names (vitessio#13208) * Add security audit report (vitessio#13221) * gentler warning message on config-not-found (vitessio#13215) * Bump the vitess version on main (vitessio#13212) * Handle DISTINCT with the new operators (vitessio#13201) * Gen4: move insert planner to gen4 (vitessio#12934)

* cherry pick of 13738 * resolved conflict Signed-off-by: Shlomi Noach <[email protected]> --------- Signed-off-by: Shlomi Noach <[email protected]> Co-authored-by: Shlomi Noach <[email protected]>

ajm188 added Type: Feature Component: Cluster management Component: Online DDL Online DDL (vitess/native/gh-ost/pt-osc) labels Aug 7, 2023

vitess-bot bot added NeedsDescriptionUpdate The description is not clear or comprehensive enough, and needs work NeedsIssue A linked issue is missing for this Pull Request NeedsWebsiteDocsUpdate What it says labels Aug 7, 2023

Add OnlineDDL show support

4da799d

Signed-off-by: Andrew Mason <[email protected]>

ajm188 force-pushed the andrew/vtctldserver-onlineddl-show branch from 525acef to 4da799d Compare August 7, 2023 17:48

ajm188 commented Aug 7, 2023

View reviewed changes

github-actions bot added this to the v18.0.0 milestone Aug 7, 2023

ajm188 commented Aug 8, 2023

View reviewed changes

go/cmd/vtctldclient/command/onlineddl.go Outdated Show resolved Hide resolved

ajm188 commented Aug 8, 2023

View reviewed changes

Andrew Mason added 2 commits August 8, 2023 06:23

finish defining SchemaMigrations protobuf message

b2bf58f

Signed-off-by: Andrew Mason <[email protected]>

commentary

2a11c41

Signed-off-by: Andrew Mason <[email protected]>

ajm188 marked this pull request as ready for review August 8, 2023 14:47

ajm188 requested review from deepthi, harshit-gangal, mattlord, notfelineit, rohit-nayak-ps and vmg as code owners August 8, 2023 14:47

ajm188 mentioned this pull request Aug 8, 2023

[vtctldserver] OnlineDDL protobuf message introduction #12195

Closed

3 tasks

shlomi-noach approved these changes Aug 8, 2023

View reviewed changes

Andrew Mason added 2 commits August 8, 2023 18:53

named row, so nice

d67a012

Signed-off-by: Andrew Mason <[email protected]>

unit test row conversion

3bdfb6f

Signed-off-by: Andrew Mason <[email protected]>

shlomi-noach approved these changes Aug 9, 2023

View reviewed changes

Andrew Mason added 2 commits August 9, 2023 10:21

minor tweaks

3a68dbc

Signed-off-by: Andrew Mason <[email protected]>

unit tests

98f6fcf

Signed-off-by: Andrew Mason <[email protected]>

Andrew Mason added 2 commits August 15, 2023 07:44

support marshalling multiple migrations into one result

73c485a

Signed-off-by: Andrew Mason <[email protected]>

add SQL table format

8ab6696

Signed-off-by: Andrew Mason <[email protected]>

ajm188 changed the title ~~[WIP] Add OnlineDDL show support~~ Add OnlineDDL show support Aug 15, 2023

Andrew Mason added 3 commits August 15, 2023 10:00

Merge branch 'main' into andrew/vtctldserver-onlineddl-show

3addca0

Signed-off-by: Andrew Mason <[email protected]>

show takes at least one arg

d37da34

Signed-off-by: Andrew Mason <[email protected]>

update flag testdata

495ee03

Signed-off-by: Andrew Mason <[email protected]>

mattlord removed NeedsDescriptionUpdate The description is not clear or comprehensive enough, and needs work NeedsIssue A linked issue is missing for this Pull Request labels Aug 16, 2023

mattlord approved these changes Aug 17, 2023

View reviewed changes

Andrew Mason added 3 commits August 17, 2023 18:39

PR feedback (nitpicks and wordings)

56271d2

Signed-off-by: Andrew Mason <[email protected]>

PR feedback (TODO comments)

ef39e58

Signed-off-by: Andrew Mason <[email protected]>

concurrent fetches

9472588

Signed-off-by: Andrew Mason <[email protected]>

Andrew Mason added 2 commits August 17, 2023 18:49

remove unsafe cast

0677d9c

Signed-off-by: Andrew Mason <[email protected]>

nit

22d6c13

Signed-off-by: Andrew Mason <[email protected]>

Andrew Mason added 3 commits August 20, 2023 06:26

remove deprecation warnings that are unneeded

370aa09

Signed-off-by: Andrew Mason <[email protected]>

standard ghost/ptosc names

8e58aec

Signed-off-by: Andrew Mason <[email protected]>

fix marshal to handle null_type

0785e2c

Signed-off-by: Andrew Mason <[email protected]>

This was referenced Aug 21, 2023

Release Summary for new vtctldclient vrep/onlineddl commands #13826

Closed

website docs for onlineddl show vitessio/website#1565

Merged

ajm188 removed the NeedsWebsiteDocsUpdate What it says label Aug 21, 2023

ajm188 merged commit a97f468 into vitessio:main Aug 21, 2023
117 of 118 checks passed

ajm188 deleted the andrew/vtctldserver-onlineddl-show branch August 21, 2023 16:42

	for _, row := range r.Named().Rows {
	uuid := row["migration_uuid"].ToString()
	onlineDDL, migrationRow, err := e.readMigration(ctx, uuid)
	if err != nil {
	return countRunnning, cancellable, err
	}
	postponeCompletion := row.AsBool("postpone_completion", false)
	elapsedSeconds := row.AsInt64("elapsed_seconds", 0)

		// TODO (andrew): Remove special handling for lower/uppercase and
		// gh-ost=>ghost/pt-osc=>ptosc support. (file issue for this).

Add OnlineDDL show support #13738

Add OnlineDDL show support #13738

Conversation

ajm188 commented Aug 7, 2023 • edited by mattlord Loading

Description

Related Issue(s)

Checklist

vitess-bot bot commented Aug 7, 2023

Review Checklist

General

Tests

Documentation

New flags

If a workflow is added or modified:

Backward compatibility

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shlomi-noach left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shlomi-noach left a comment

Choose a reason for hiding this comment

ajm188 commented Aug 15, 2023 • edited Loading

shlomi-noach commented Aug 16, 2023

mattlord left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ajm188 commented Aug 17, 2023

mattlord commented Aug 18, 2023

ajm188 commented Aug 7, 2023 •

edited by mattlord

Loading

ajm188 commented Aug 15, 2023 •

edited

Loading