You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
For the copy phase, a rowstreamer (a specific type of vstreamer) streams the rows resulting from a query used to get a consistent snapshot of the table at that logical point in time (a GTID set which matches the consistent snapshot of the table). For the running/replicating phase a standard vstreamer streams filtered binary log events from the source tablet to the target tablet(s).
Because the rowstreamer sends the results of a query, for ENUM and SET columns it sends the string value that it gets back from MySQL. Because the vstreamer sends filtered binary log events, for ENUM and SET columns it sends the integer based value as that's what is in the binary log events.
This has been (partially) addressed in the past on the consumer side. For VReplication workflows the consumers are vcopier for the rowstreamer and vplayer for the vstreamer and the vplayer started doing the integer to string mapping in #15349 (this was done as it caused problems for Vitess OnlineDDL when the schema change was shuffling the order of elements in the ENUM around). In VStream consumers such as the Debezium Vitess connector and the PlanetScale Airbyte connector this mapping was done when processing the incoming VEvents (debezium connector PR, airbyte connector PR).
Rather than pushing this work on to each current and future consumer, we should unify the behavior in the VReplicationvstreamer and its subtypes such as rowstreamer.
Reproduction Steps
Setup the test env:
git checkout main && make build
pushd examples/local
./101_initial_cluster.sh
mysql < ../common/insert_commerce_data.sql
vtctldclient ApplySchema --ddl-strategy=direct --sql "alter table customer add size enum ('small', 'medium', 'large') default 'medium'" commerce
Modify the vstream_client example to stream everything from the customer table in the unsharded commerce keyspace:
In another Terminal, start a vtgate VStream of the customer table. It will start with the copy phase and then continue on in the running/replicating phase:
cd ${VTROOT}/examples/local
go run vstream_client.go
Generate some more data now that the vstream is past the copy phase and in the running/replicating phase:
mysql commerce
# ... perform inserts, updates, deletes against the customer table
Overview of the Issue
In
VReplication
workflows and theVStream
API there is a copy phase (initial snapshot copy) and a running/replicating phase. You can read more about the lifecycle here.For the copy phase, a
rowstreamer
(a specific type ofvstreamer
) streams the rows resulting from a query used to get a consistent snapshot of the table at that logical point in time (a GTID set which matches the consistent snapshot of the table). For the running/replicating phase a standardvstreamer
streams filtered binary log events from the source tablet to the target tablet(s).Because the
rowstreamer
sends the results of a query, forENUM
andSET
columns it sends the string value that it gets back from MySQL. Because thevstreamer
sends filtered binary log events, forENUM
andSET
columns it sends the integer based value as that's what is in the binary log events.This has been (partially) addressed in the past on the consumer side. For
VReplication
workflows the consumers arevcopier
for therowstreamer
andvplayer
for thevstreamer
and thevplayer
started doing the integer to string mapping in #15349 (this was done as it caused problems for Vitess OnlineDDL when the schema change was shuffling the order of elements in the ENUM around). InVStream
consumers such as the Debezium Vitess connector and the PlanetScale Airbyte connector this mapping was done when processing the incomingVEvents
(debezium connector PR, airbyte connector PR).Rather than pushing this work on to each current and future consumer, we should unify the behavior in the
VReplication
vstreamer
and its subtypes such asrowstreamer
.Reproduction Steps
Setup the test env:
Modify the
vstream_client
example to stream everything from thecustomer
table in the unshardedcommerce
keyspace:In another Terminal, start a vtgate VStream of the
customer
table. It will start with the copy phase and then continue on in the running/replicating phase:Generate some more data now that the vstream is past the copy phase and in the running/replicating phase:
Clean up:
Sample results:
Binary Version
Operating System and Environment details
Log Fragments
No response
The text was updated successfully, but these errors were encountered: