Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(debeizum): upgrade version of debeizum from 2.4.2 to 2.6.2 #17504

Merged
merged 5 commits into from
Jul 26, 2024

Conversation

KeXiangWang
Copy link
Contributor

@KeXiangWang KeXiangWang commented Jun 28, 2024

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

Due to some fixes/fetures requirements, tentatively upgrade the dependent version of debeizum from 2.4.2.Final to 2.6.2.Final.

  1. https://issues.redhat.com/browse/DBZ-7933?jql=component%20%3D%20sqlserver-connector%20AND%20text%20~%20%22time%22 It's a time precision issue from Debezium. The debezium developers fix it in 2.7.0 (PR). So I was planning to upgrade to 2.7.0.Final directly, which was released on June 27. After testing, it still contains some weird bugs in mysql-cdc support. THe error log is here.
  2. In feat(cdc): support sql server cdc #17429, I found that the current 2.4.2.Final does not support retrieve schema history from upstream sql server automatically. While after 2.6.0, recovery snapshot mode is added to support recovery from empty schema history storage. snapshot.mode. If we keep using 2.4.2, we need to find a way to store the schema history, which is non-trivial, making the JVM stateful.

Checklist

  • I have written necessary rustdoc comments
  • I have added necessary unit tests and integration tests
  • I have added test labels as necessary. See details.
  • I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
  • My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
  • All checks passed in ./risedev check (or alias, ./risedev c)
  • My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)
  • My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

  • My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

@KeXiangWang KeXiangWang added ci/run-backwards-compat-tests Run backwards compatibility tests in your PR. and removed type/feature labels Jun 28, 2024
@KeXiangWang KeXiangWang force-pushed the wkx/update-debeizum-to-2.7.1 branch 2 times, most recently from 15605b0 to cefbc89 Compare June 30, 2024 23:55
@tabVersion
Copy link
Contributor

Waiting for @StrikeW 's review.

@StrikeW StrikeW changed the title feat(debeizum): upgrade the dependent version of debeizum from 2.4.2.Final… feat(debeizum): upgrade version of debeizum from 2.4.2 to 2.6.2 Jul 1, 2024
@StrikeW
Copy link
Contributor

StrikeW commented Jul 5, 2024

Usually a new version of Debezium will introduce some breaking changes to existing connectors, could you analyze the breaking changes in their release changelog and find out those related to the connectors we have supported? I suggest you can list those changes in the PR description.

java/pom.xml Show resolved Hide resolved
@KeXiangWang
Copy link
Contributor Author

KeXiangWang commented Jul 25, 2024

No directly related break change is found.

Try to list related break changes/fixes related to MySQL, Postgres and MongoDB here:

2.6 (source):

  • Upgrade PostgreSQL driver to 42.6.1 DBZ-7571

Fixes

  • Postgres: Potential data loss on connector restart DBZ-7816
  • The MySQL Debezium connector is not doing the snapshot after the reset. DBZ-7743
  • MongoDb connector doesn’t work with Load Balanced cluster DBZ-7744
  • Apply MySQL binlog client version 0.29.1 - bugfix: read long value when deserializing gtid transaction’s length DBZ-7757
  • PostgreSQL connector doesn’t restart properly if database if not reachable DBZ-6236
  • Fix MySQL 8 event timestamp resolution logic error where fallback to seconds occurs erroneously for non-GTID events DBZ-7500
  • PostgreSQL ad-hoc blocking snapshots fail when snapshot mode is "never" DBZ-7311

2.5 (source):
Fixes

  • Debezium MySQL could not parse certain grant privileges. DBZ-7277
  • MongoDB table/colelction snapshot notification contain incorrect offsets DBZ-7252
  • Support logical decoding from Postgres 16 stand-bys DBZ-7181
  • Failed to authenticate to the MySQL database after snapshot DBZ-7132
  • Debezium-ddl-parser crashes on parsing MySQL DDL statement (specific UNION) DBZ-7140
  • Fix DebeziumMySqlConnectorResource not using the new MySQL adatper structure to support different MySQL flavors DBZ-7179
  • Parsing MySQL indexes for JSON field fails, when casting is used with types double and float DBZ-7189
  • Debezium crashes on parsing MySQL DDL statement (specific INSERT) DBZ-7119

Copy link
Contributor

@StrikeW StrikeW left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rest lgtm. To confirm whether we need to modify json parser to parsebool to int for mysql.

@KeXiangWang KeXiangWang added this pull request to the merge queue Jul 26, 2024
Merged via the queue into main with commit a8ef5a7 Jul 26, 2024
30 of 31 checks passed
@KeXiangWang KeXiangWang deleted the wkx/update-debeizum-to-2.7.1 branch July 26, 2024 19:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
breaking-change ci/run-backwards-compat-tests Run backwards compatibility tests in your PR. type/feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants