VReplication: Support automatically replacing auto_inc cols with sequences

changelog/21.0/21.0.0/summary.md

@@ -19,12 +19,13 @@
     - **[Reference Table Materialization](#reference-table-materialization)**
     - **[New VEXPLAIN Modes: TRACE and KEYS](#new-vexplain-modes)**
     - **[Errant GTID Detection on Vttablets](#errant-gtid-vttablet)**
+    - **[Automatically Replace MySQL auto_increment Clauses with Vitess Sequences](#auto-replace-mysql-autoinc-with-seq)**
 
-## <a id="major-changes"/>Major Changes
+## <a id="major-changes"/>Major Changes</a>
 
-### <a id="deprecations-and-deletions"/>Deprecations and Deletions
+### <a id="deprecations-and-deletions"/>Deprecations and Deletions</a>
 
-#### <a id="metric-deletion"/>Deletion of deprecated metrics
+#### <a id="metric-deletion"/>Deletion of deprecated metrics</a>
 
 The following metrics that were deprecated in the previous release, have now been deleted.
 
@@ -43,12 +44,12 @@ The following metrics that were deprecated in the previous release, have now bee
 |          `planned_reparent_counts`           |      
 |      `reparent_shard_operation_timings`      |  
 
-#### <a id="vttablet-flags"/>VTTablet Flags
+#### <a id="vttablet-flags"/>VTTablet Flags</a>
 
 - `queryserver-enable-settings-pool` flag, added in v15, has been on by default since v17.
   It is now deprecated and will be removed in a future release.
 
-#### <a id="deprecations-metrics"/>Metrics
+#### <a id="deprecations-metrics"/>Metrics</a>
 
 The following metrics are now deprecated, if provided please use their replacement.
 
@@ -74,7 +75,7 @@ $ vtctldclient --server :15999 MoveTables --target-keyspace customer --workflow
 
 Mirror rules can be inspected with `GetMirrorRules`.
 
-### <a id="new-vtgate-shutdown-behavior"/>New VTGate Shutdown Behavior
+### <a id="new-vtgate-shutdown-behavior"/>New VTGate Shutdown Behavior</a>
 
 We added a new option to affect the VTGate shutdown process in v21 by using a connection drain timeout rather than the
 older activity drain timeout.
@@ -87,7 +88,7 @@ This new behavior can be enabled by specifying the new `--mysql-server-drain-ont
 
 See more information about this change by [reading its RFC](https://github.com/vitessio/vitess/issues/15971).
 
-### <a id="tablet-throttler"/>Tablet Throttler: Multi-Metric support
+### <a id="tablet-throttler"/>Tablet Throttler: Multi-Metric support</a>
 
 Up till `v20`, the tablet throttler would only monitor and use a single metric. That would be replication lag, by
 default, or could be the result of a custom query. `v21` introduces a major redesign where the throttler monitors and
@@ -114,20 +115,20 @@ Explicit app to metric assignments will override the catch-all configuration.
 Metrics are assigned a default _scope_, which could be `self` (isolated to the tablet) or `shard` (max, aka _worst_
 value among shard tablets). It is further possible to require a different scope for each metric.
 
-### <a id="allow-cross-cell"/>Allow Cross Cell Promotion in PRS
+### <a id="allow-cross-cell"/>Allow Cross Cell Promotion in PRS</a>
 
 Up until now if the users wanted to promote a replica in a different cell than the current primary
 using `PlannedReparentShard`, they had to specify the new primary with the `--new-primary` flag.
 
 We have now added a new flag `--allow-cross-cell-promotion` that lets `PlannedReparentShard` choose a primary in a
 different cell even if no new primary is provided explicitly.
 
-### <a id="recursive-cte"/>Experimental support for recursive CTEs
+### <a id="recursive-cte"/>Experimental support for recursive CTEs</a>
 
 We have added experimental support for recursive CTEs in Vitess. We are marking it as experimental because it is not yet
 fully tested and may have some limitations. We are looking for feedback from the community to improve this feature.
 
-### <a id="tablet-balancer"/>VTGate Tablet Balancer
+### <a id="tablet-balancer"/>VTGate Tablet Balancer</a>
 
 When a VTGate routes a query and has multiple available tablets for a given shard / tablet type (e.g. REPLICA), the
 current default behavior routes the query with local cell affinity and round robin policy. The VTGate Tablet Balancer
@@ -139,7 +140,7 @@ and `--balancer-keyspaces`.
 
 See [RFC for details](https://github.com/vitessio/vitess/issues/12241).
 
-### <a id="query-timeout"/>Query Timeout Override
+### <a id="query-timeout"/>Query Timeout Override</a>
 
 VTGate sends an authoritative query timeout to VTTablet when the `QUERY_TIMEOUT_MS` comment directive,
 `query_timeout` session system variable, or `query-timeout` flag is set.
@@ -155,26 +156,27 @@ A query can also be set to have no timeout by using the `QUERY_TIMEOUT_MS` comme
 Example usage:
 `select /*vt+ QUERY_TIMEOUT_MS=30 */ col from tbl`
 
-### <a id="new-backup-engine"/>New Backup Engine (EXPERIMENTAL)
+### <a id="new-backup-engine"/>New Backup Engine (EXPERIMENTAL)</a>
 
 We are introducing a backup engine supporting logical backups starting on v21 to support use cases that require something else besides physical backups. This is experimental and is based on the 
 [MySQL Shell](https://dev.mysql.com/doc/mysql-shell/8.0/en/).
 
 The new engine is enabled by using `--backup_engine_implementation=mysqlshell`. There are other options that are required, so [check the docs](https://vitess.io/docs/21.0/user-guides/operating-vitess/backup-and-restore/creating-a-backup/) on which options are required and how to use it.
 
-### <a id="dynamic-vreplication-configuration"/>Dynamic VReplication Configuration
+### <a id="dynamic-vreplication-configuration"/>Dynamic VReplication Configuration</a>
 
 Currently many of the configuration options for VReplication Workflows are vttablet flags. This means that any change
 requires restarts of vttablets. We now allow these to be overridden while creating a workflow or dynamically once
 the workflow is in progress. See https://github.com/vitessio/vitess/pull/16583 for details.
 
-### <a id="reference-table-materialization"/>Reference Table Materialization
+### <a id="reference-table-materialization"/>Reference Table Materialization</a>
 
 There is a new option in [`Materialize` workflows](https://vitess.io/docs/reference/vreplication/materialize/) to keep 
 a synced copy of [reference or lookup tables](https://vitess.io/docs/reference/vreplication/reference_tables/) 
 (countries, states, zip_codes, etc) from an unsharded keyspace, which holds the source of truth for the reference 
 table, to all shards in a sharded keyspace.
-### <a id="new-vexplain-modes"/>New VEXPLAIN Modes: TRACE and KEYS
+
+### <a id="new-vexplain-modes"/>New VEXPLAIN Modes: TRACE and KEYS</a>
 
 #### VEXPLAIN TRACE
 
@@ -199,10 +201,34 @@ KEYS mode analyzes the query structure without executing it, providing JSON outp
 
 These new VEXPLAIN modes enhance Vitess's query analysis capabilities, allowing for more informed decisions about sharding strategies and query optimization.
 
-### <a id="errant-gtid-vttablet"/>Errant GTID Detection on Vttablets
+### <a id="errant-gtid-vttablet"/>Errant GTID Detection on Vttablets</a>
 
 Vttablets now run an errant GTID detection logic before they join the replication stream. So, if a replica has an errant GTID, it will
 not start replicating from the primary. It will fail the call the set its replication source because of the errant GTID. This prevents us 
 from running into situations from which recovery is very hard.
 
 For users running with the vitess operator on kubernetes, this change means that the replicas with errant GTIDs will have broken replication and will report as unready. The users will need to manually clean up these errant replica tablets.
+
+### <a id="auto-replace-mysql-autoinc-with-seq"/>Automatically Replace MySQL auto_increment Clauses with Vitess Sequences</a>
+
+When migrating tables from an unsharded keyspace to a sharded one using the [VReplication `MoveTables` command](https://vitess.io/docs/reference/vreplication/movetables/), we now support
+options for how to handle auto incrementing values for the table(s) being moved. This is important as now that these tables will be sharded in the target keyspace you cannot rely on the
+[MySQL `auto_increment` clauses](https://dev.mysql.com/doc/refman/en/example-auto-increment.html) — as the values will need to be unique across all shards in the sharded target keyspace —
+and you would instead need to rely on [Vitess Sequences](https://vitess.io/docs/reference/features/vitess-sequences/) to achieve the equivalent behavior. In https://github.com/vitessio/vitess/pull/15679
+we added the ability (enabled by default) to remove those MySQL `auto_increment` clauses when moving tables from an unsharded keyspace to a sharded one. And in https://github.com/vitessio/vitess/pull/13656
+we added the ability to automatically initialize the sequences used in the target keyspace so that they can be used without any additional work and picking up right where the MySQL `auto_increment`
+values left off on the source keyspace. This new feature, added in https://github.com/vitessio/vitess/pull/16860, builds on all of that work to support replacing those MySQL `auto_increment`
+clauses with [Vitess Sequences](https://vitess.io/docs/reference/features/vitess-sequences/), performing all of the setup work automatically as part of the workflow.
+
+Please note that we have deprecated the [`--remove-sharded-auto-increment` boolean flag](https://vitess.io/docs/20.0/reference/programs/vtctldclient/vtctldclient_movetables/vtctldclient_movetables_create/)
+(added in https://github.com/vitessio/vitess/pull/15679) and it has been replaced with the new [`--sharded-auto-increment-handling` flag](https://vitess.io/docs/21.0/reference/programs/vtctldclient/vtctldclient_movetables/vtctldclient_movetables_create/)
+which takes a string where the valid values today are `LEAVE` (leave the `auto_increment` clauses in place), `REMOVE` (remove the clauses), and `REPLACE` (replace them with sequences). The
+`--remove-sharded-auto-increment[=true]` behavior in v21 is equal to the new `--sharded-auto-increment-handling=remove` behavior in v21+. To use the new support for automatically replacing those
+MySQL `auto_increment` clauses with Vitess Sequences, you would utilize these two
+[`MoveTables create` flags](https://vitess.io/docs/21.0/reference/programs/vtctldclient/vtctldclient_movetables/vtctldclient_movetables_create/): `--sharded-auto-increment-handling=replace --global-keyspace=foo`
+where `foo` is an unsharded keyspace that can be used for sequences, [reference tables](https://vitess.io/docs/21.0/reference/vreplication/reference_tables/), and other "global" resources. That keyspace is
+where the sequence tables will be created as needed as part of the replacement work. Then, when switching the application traffic to the target keyspace you would specify the
+[`--initialize-target-sequences` flag for the `MoveTables switchtraffic` command](https://vitess.io/docs/reference/programs/vtctldclient/vtctldclient_movetables/vtctldclient_movetables_switchtraffic/).
+It's at this point where the sequence tables will be created, if needed, and the starting value used will be initialized based on the current maximum value used for the table in the unsharded source
+keyspace. This allows us to replace the MySQL feature with the equivalent Vitess one in a way that is entirely transparent to the application and its users — they can continue to elide values for the
+column where auto incrementing values are desired on `INSERT`, and there will be no visible difference in the behavior resulting from the traffic switch.

go/cmd/vtctldclient/command/vreplication/common/utils.go

@@ -287,7 +287,7 @@ func AddCommonSwitchTrafficFlags(cmd *cobra.Command, initializeTargetSequences b
 	cmd.Flags().BoolVar(&SwitchTrafficOptions.DryRun, "dry-run", false, "Print the actions that would be taken and report any known errors that would have occurred.")
 	cmd.Flags().BoolVar(&SwitchTrafficOptions.Force, "force", false, "Force the traffic switch even if some potentially non-critical actions cannot be performed; for example the tablet refresh fails on some tablets in the keyspace. WARNING: this should be used with extreme caution and only in emergency situations!")
 	if initializeTargetSequences {
-		cmd.Flags().BoolVar(&SwitchTrafficOptions.InitializeTargetSequences, "initialize-target-sequences", false, "When moving tables from an unsharded keyspace to a sharded keyspace, initialize any sequences that are being used on the target when switching writes.")
+		cmd.Flags().BoolVar(&SwitchTrafficOptions.InitializeTargetSequences, "initialize-target-sequences", false, "When moving tables from an unsharded keyspace to a sharded keyspace, initialize any sequences that are being used on the target when switching writes. If the sequence table is not found, and the sequence table reference was fully qualified OR a value was specified for --global-keyspace, then we will attempt to create the sequence table in that keyspace.")
 	}
 }
 

go/cmd/vtctldclient/command/vreplication/movetables/create.go

@@ -40,6 +40,8 @@ var (
 		NoRoutingRules      bool
 		AtomicCopy          bool
 		WorkflowOptions     vtctldatapb.WorkflowOptions
+		// This maps to a WorkflowOptions.ShardedAutoIncrementHandling ENUM value.
+		ShardedAutoIncrementHandlingStr string
 	}{}
 
 	// create makes a MoveTablesCreate gRPC call to a vtctld.
@@ -87,6 +89,20 @@ var (
 				return fmt.Errorf("cannot specify both --tenant-id (i.e. a multi-tenant migration) and --source-shards (i.e. a shard-by-shard migration)")
 			}
 
+			// createOptions.ShardedAutoIncrementHandlingStr is the CLI flag value
+			// provided and we need to map that to the ENUM value for
+			// createOptions.WorkflowOptions.ShardedAutoIncrementHandling which
+			// gets saved in the _vt.vreplication record's options column.
+			createOptions.ShardedAutoIncrementHandlingStr = strings.ToUpper(createOptions.ShardedAutoIncrementHandlingStr)
+			val, ok := vtctldatapb.ShardedAutoIncrementHandling_value[createOptions.ShardedAutoIncrementHandlingStr]
+			if !ok {
+				return fmt.Errorf("invalid value provided for --sharded-auto-increment-handling, valid values are: %s", shardedAutoIncHandlingStrOptions)
+			}
+			createOptions.WorkflowOptions.ShardedAutoIncrementHandling = vtctldatapb.ShardedAutoIncrementHandling(val)
+			if val == int32(vtctldatapb.ShardedAutoIncrementHandling_REPLACE) && createOptions.WorkflowOptions.GlobalKeyspace == "" {
+				fmt.Println("WARNING: no global-keyspace value provided so all sequence table references not fully qualified must be created manually before switching traffic")
+			}
+
 			return nil
 		},
 		RunE: commandCreate,

go/cmd/vtctldclient/command/vreplication/movetables/movetables.go

@@ -17,10 +17,15 @@ limitations under the License.
 package movetables
 
 import (
+	"fmt"
+	"strings"
+
 	"github.com/spf13/cobra"
 
 	"vitess.io/vitess/go/cmd/vtctldclient/command/vreplication/common"
 	"vitess.io/vitess/go/vt/topo/topoproto"
+
+	vtctldatapb "vitess.io/vitess/go/vt/proto/vtctldata"
 )
 
 var (
@@ -32,6 +37,7 @@ var (
 		Aliases:               []string{"movetables"},
 		Args:                  cobra.ExactArgs(1),
 	}
+	shardedAutoIncHandlingStrOptions string
 )
 
 func registerCommands(root *cobra.Command) {
@@ -49,8 +55,16 @@ func registerCommands(root *cobra.Command) {
 	create.Flags().BoolVar(&createOptions.NoRoutingRules, "no-routing-rules", false, "(Advanced) Do not create routing rules while creating the workflow. See the reference documentation for limitations if you use this flag.")
 	create.Flags().BoolVar(&createOptions.AtomicCopy, "atomic-copy", false, "(EXPERIMENTAL) A single copy phase is run for all tables from the source. Use this, for example, if your source keyspace has tables which use foreign key constraints.")
 	create.Flags().StringVar(&createOptions.WorkflowOptions.TenantId, "tenant-id", "", "(EXPERIMENTAL: Multi-tenant migrations only) The tenant ID to use for the MoveTables workflow into a multi-tenant keyspace.")
-	create.Flags().BoolVar(&createOptions.WorkflowOptions.StripShardedAutoIncrement, "remove-sharded-auto-increment", true, "If moving the table(s) to a sharded keyspace, remove any auto_increment clauses when copying the schema to the target as sharded keyspaces should rely on either user/application generated values or Vitess sequences to ensure uniqueness.")
 	create.Flags().StringSliceVar(&createOptions.WorkflowOptions.Shards, "shards", nil, "(EXPERIMENTAL: Multi-tenant migrations only) Specify that vreplication streams should only be created on this subset of target shards. Warning: you should first ensure that all rows on the source route to the specified subset of target shards using your VIndex of choice or you could lose data during the migration.")
+	create.Flags().StringVar(&createOptions.WorkflowOptions.GlobalKeyspace, "global-keyspace", "", "If specified, then attempt to create any global resources here such as sequence tables needed to replace auto_increment table clauses that are removed due to --sharded-auto-increment-handling=REPLACE. The value must be an unsharded keyspace that already exists.")
+	create.Flags().StringVar(&createOptions.ShardedAutoIncrementHandlingStr, "sharded-auto-increment-handling", vtctldatapb.ShardedAutoIncrementHandling_REMOVE.String(),
+		fmt.Sprintf("If moving the table(s) to a sharded keyspace, remove any MySQL auto_increment clauses when copying the schema to the target as sharded keyspaces should rely on either user/application generated values or Vitess sequences to ensure uniqueness. If REPLACE is specified then they are automatically replaced by Vitess sequence definitions. (options are: %s)",
+			shardedAutoIncHandlingStrOptions))
+	// This flag was deprecated in v21 so can be removed in v22+.
+	create.Flags().StringVar(&createOptions.ShardedAutoIncrementHandlingStr, "remove-sharded-auto-increment", vtctldatapb.ShardedAutoIncrementHandling_REMOVE.String(),
+		fmt.Sprintf("If moving the table(s) to a sharded keyspace, remove any MySQL auto_increment clauses when copying the schema to the target as sharded keyspaces should rely on either user/application generated values or Vitess sequences to ensure uniqueness. If REPLACE is specified then they are automatically replaced by Vitess sequence definitions. (options are: %s)",
+			shardedAutoIncHandlingStrOptions))
+	create.Flags().MarkDeprecated("remove-sharded-auto-increment", "please use --sharded-auto-increment-handling instead.")
 	base.AddCommand(create)
 
 	opts := &common.SubCommandsOpts{
@@ -100,4 +114,17 @@ func registerCommands(root *cobra.Command) {
 
 func init() {
 	common.RegisterCommandHandler("MoveTables", registerCommands)
+
+	sb := strings.Builder{}
+	strvals := make([]string, len(vtctldatapb.ShardedAutoIncrementHandling_name))
+	for enumval, strval := range vtctldatapb.ShardedAutoIncrementHandling_name {
+		strvals[enumval] = strval
+	}
+	for i, v := range strvals {
+		if i > 0 {
+			sb.WriteByte(',')
+		}
+		sb.WriteString(v)
+	}
+	shardedAutoIncHandlingStrOptions = sb.String()
 }

go/test/endtoend/vreplication/vreplication_vtctldclient_cli_test.go

@@ -110,8 +110,8 @@ func testMoveTablesFlags1(t *testing.T, mt *iMoveTables, sourceKeyspace, targetK
 	createFlags := []string{"--auto-start=false", "--defer-secondary-keys=false", "--stop-after-copy",
 		"--no-routing-rules", "--on-ddl", "STOP", "--exclude-tables", "customer2",
 		"--tablet-types", "primary,rdonly", "--tablet-types-in-preference-order=true",
-		"--all-cells",
-		"--config-overrides", mapToCSV(overrides),
+		"--all-cells", "--config-overrides", mapToCSV(overrides),
+		"--sharded-auto-increment-handling=REPLACE", fmt.Sprintf("--global-keyspace=%s", sourceKeyspace),
 	}
 	completeFlags := []string{"--keep-routing-rules", "--keep-data"}
 	switchFlags := []string{}
@@ -591,6 +591,10 @@ func validateMoveTablesWorkflow(t *testing.T, workflows []*vtctldatapb.Workflow)
 	require.Equalf(t, 1, len(bls.Filter.Rules), "Rules are %+v", bls.Filter.Rules) // only customer, customer2 should be excluded
 	require.Equal(t, binlogdatapb.OnDDLAction_STOP, bls.OnDdl)
 	require.True(t, bls.StopAfterCopy)
+
+	// Validate the sharded-auto-increment-handling related value handling.
+	require.Equal(t, vtctldatapb.ShardedAutoIncrementHandling_REPLACE, vtctldatapb.ShardedAutoIncrementHandling(wf.Options.ShardedAutoIncrementHandling))
+	require.Equal(t, wf.Source.Keyspace, wf.Options.GlobalKeyspace)
 }
 
 // Test that routing rules can be applied using the vtctldclient CLI for all types of routing rules.