Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

mysql source : implement support for snapshot of new tables in cdc mode #16954

Merged
merged 14 commits into from
Sep 26, 2022
Merged

mysql source : implement support for snapshot of new tables in cdc mode #16954

merged 14 commits into from
Sep 26, 2022

Conversation

subodh1810
Copy link
Contributor

Issue : #15471

@subodh1810 subodh1810 self-assigned this Sep 21, 2022
@subodh1810 subodh1810 requested a review from a team as a code owner September 21, 2022 10:59
@github-actions github-actions bot added the area/connectors Connector related issues label Sep 21, 2022
@github-actions
Copy link
Contributor

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

  • source-oracle-strict-encrypt
  • source-mysql
  • source-db2-strict-encrypt
  • source-clickhouse-strict-encrypt
  • source-scaffold-java-jdbc
  • source-alloydb-strict-encrypt
  • source-postgres-strict-encrypt
  • source-snowflake
  • source-tidb
  • source-cockroachdb
  • source-oracle
  • source-clickhouse
  • source-alloydb
  • source-postgres
  • source-cockroachdb-strict-encrypt
  • source-db2
  • source-mysql-strict-encrypt
  • source-mssql
  • source-bigquery
  • source-mssql-strict-encrypt
  • source-redshift

@subodh1810 subodh1810 linked an issue Sep 21, 2022 that may be closed by this pull request
Store database catalog for Debezium #15471
Closed
@subodh1810
Copy link
Contributor Author

subodh1810 commented Sep 21, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/source-mysql

🕑 connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/3098272032

@subodh1810
Copy link
Contributor Author

subodh1810 commented Sep 21, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3098273807
✅ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3098273807
No Python unittests run

Build Passed

Test summary info:

All Passed

edgao
edgao reviewed Sep 21, 2022
View reviewed changes
Copy link
Contributor

@edgao edgao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

couple questions around expected behavior. LGTM pending those comments

...es/debezium-v1-9-2/src/testFixtures/java/io/airbyte/integrations/debezium/CdcSourceTest.java Show resolved Hide resolved
...ectors/source-jdbc/src/main/java/io/airbyte/integrations/source/jdbc/AbstractJdbcSource.java Show resolved Hide resolved
.../connectors/source-mysql/src/main/java/io/airbyte/integrations/source/mysql/MySqlSource.java Outdated Show resolved Hide resolved
...tors/source-mysql/src/test/java/io/airbyte/integrations/source/mysql/CdcMysqlSourceTest.java Show resolved Hide resolved
@github-actions
Copy link
Contributor

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

  • source-snowflake
  • source-mysql
  • source-alloydb-strict-encrypt
  • source-postgres-strict-encrypt
  • source-cockroachdb-strict-encrypt
  • source-scaffold-java-jdbc
  • source-clickhouse-strict-encrypt
  • source-oracle-strict-encrypt
  • source-alloydb
  • source-redshift
  • source-clickhouse
  • source-db2
  • source-db2-strict-encrypt
  • source-tidb
  • source-oracle
  • source-bigquery
  • source-mssql
  • source-mysql-strict-encrypt
  • source-mssql-strict-encrypt
  • source-postgres
  • source-cockroachdb

@subodh1810
Copy link
Contributor Author

subodh1810 commented Sep 22, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/source-mysql

🕑 connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/3105933835
❌ connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/3105933835
🐛 https://gradle.com/s/tkjeo5vjrw77o

Build Failed

Test summary info:

Could not find result summary

@subodh1810
Copy link
Contributor Author

subodh1810 commented Sep 22, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3105933768
✅ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3105933768
No Python unittests run

Build Passed

Test summary info:

All Passed

@github-actions
Copy link
Contributor

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

  • source-postgres
  • source-oracle
  • source-mssql
  • source-snowflake
  • source-postgres-strict-encrypt
  • source-db2-strict-encrypt
  • source-cockroachdb
  • source-oracle-strict-encrypt
  • source-db2
  • source-mssql-strict-encrypt
  • source-mysql-strict-encrypt
  • source-clickhouse
  • source-redshift
  • source-bigquery
  • source-tidb
  • source-alloydb-strict-encrypt
  • source-alloydb
  • source-clickhouse-strict-encrypt
  • source-mysql
  • source-cockroachdb-strict-encrypt
  • source-scaffold-java-jdbc

edgao
edgao approved these changes Sep 22, 2022
View reviewed changes
Copy link
Contributor

@edgao edgao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@subodh1810
Copy link
Contributor Author

subodh1810 commented Sep 22, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/source-mysql

🕑 connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/3106498637
❌ connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/3106498637
🐛 https://gradle.com/s/4wreqsiws6ark

Build Failed

Test summary info:

Could not find result summary

@github-actions
Copy link
Contributor

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

  • source-mssql-strict-encrypt
  • source-alloydb
  • source-snowflake
  • source-mysql-strict-encrypt
  • source-clickhouse-strict-encrypt
  • source-mssql
  • source-postgres-strict-encrypt
  • source-clickhouse
  • source-tidb
  • source-oracle
  • source-scaffold-java-jdbc
  • source-db2-strict-encrypt
  • source-oracle-strict-encrypt
  • source-redshift
  • source-cockroachdb
  • source-alloydb-strict-encrypt
  • source-postgres
  • source-db2
  • source-mysql
  • source-bigquery
  • source-cockroachdb-strict-encrypt

@github-actions
Copy link
Contributor

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

  • source-redshift
  • source-mysql-strict-encrypt
  • source-db2-strict-encrypt
  • source-mssql
  • source-snowflake
  • source-bigquery
  • source-scaffold-java-jdbc
  • source-cockroachdb-strict-encrypt
  • source-alloydb-strict-encrypt
  • source-tidb
  • source-mysql
  • source-db2
  • source-postgres-strict-encrypt
  • source-oracle-strict-encrypt
  • source-postgres
  • source-mssql-strict-encrypt
  • source-clickhouse
  • source-clickhouse-strict-encrypt
  • source-oracle
  • source-alloydb
  • source-cockroachdb

@subodh1810
Copy link
Contributor Author

subodh1810 commented Sep 23, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/source-mysql

🕑 connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/3111363123
❌ connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/3111363123
🐛 https://gradle.com/s/guolda6wluhpy

Build Failed

Test summary info:

Could not find result summary

@github-actions
Copy link
Contributor

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

  • source-db2
  • source-mysql-strict-encrypt
  • source-alloydb-strict-encrypt
  • source-postgres
  • source-bigquery
  • source-scaffold-java-jdbc
  • source-cockroachdb-strict-encrypt
  • source-oracle
  • source-mssql
  • source-mysql
  • source-db2-strict-encrypt
  • source-clickhouse
  • source-oracle-strict-encrypt
  • source-postgres-strict-encrypt
  • source-clickhouse-strict-encrypt
  • source-alloydb
  • source-mssql-strict-encrypt
  • source-cockroachdb
  • source-redshift
  • source-snowflake
  • source-tidb

@subodh1810
Copy link
Contributor Author

subodh1810 commented Sep 23, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/source-mysql

🕑 connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/3111737858
❌ connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/3111737858
🐛 https://gradle.com/s/wkrpfn543nyhu

Build Failed

Test summary info:

Could not find result summary

@subodh1810
Copy link
Contributor Author

subodh1810 commented Sep 26, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/source-mysql

🕑 connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/3127309584
✅ connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/3127309584
No Python unittests run

Build Passed

Test summary info:

All Passed

@github-actions
Copy link
Contributor

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

  • source-redshift
  • source-cockroachdb-strict-encrypt
  • source-mysql-strict-encrypt
  • source-oracle
  • source-alloydb
  • source-bigquery
  • source-db2-strict-encrypt
  • source-postgres
  • source-alloydb-strict-encrypt
  • source-mysql
  • source-scaffold-java-jdbc
  • source-snowflake
  • source-clickhouse
  • source-cockroachdb
  • source-tidb
  • source-clickhouse-strict-encrypt
  • source-mssql-strict-encrypt
  • source-oracle-strict-encrypt
  • source-mssql
  • source-db2
  • source-postgres-strict-encrypt

@github-actions github-actions bot added the area/documentation Improvements or additions to documentation label Sep 26, 2022
@github-actions
Copy link
Contributor

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

  • source-alloydb-strict-encrypt
  • source-clickhouse-strict-encrypt
  • source-scaffold-java-jdbc
  • source-tidb
  • source-oracle
  • source-alloydb
  • source-postgres
  • source-db2
  • source-redshift
  • source-bigquery
  • source-cockroachdb-strict-encrypt
  • source-mssql
  • source-mssql-strict-encrypt
  • source-mysql
  • source-clickhouse
  • source-snowflake
  • source-cockroachdb
  • source-postgres-strict-encrypt
  • source-db2-strict-encrypt
  • source-oracle-strict-encrypt
  • source-mysql-strict-encrypt

@subodh1810
Copy link
Contributor Author

subodh1810 commented Sep 26, 2022 •
edited by github-actions bot
Loading

/publish connector=connectors/source-mysql

🕑 Publishing the following connectors:
connectors/source-mysql
https://github.com/airbytehq/airbyte/actions/runs/3129273991


Connector Did it publish? Were definitions generated?
connectors/source-mysql ✅ ✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

@edgao
Copy link
Contributor

edgao commented Sep 26, 2022 •
edited by github-actions bot
Loading

/publish connector=connectors/source-mysql-strict-encrypt

🕑 Publishing the following connectors:
connectors/source-mysql-strict-encrypt
https://github.com/airbytehq/airbyte/actions/runs/3129427384


Connector Did it publish? Were definitions generated?
connectors/source-mysql-strict-encrypt ✅ ❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

rodireich
rodireich reviewed Sep 26, 2022
View reviewed changes
.../connectors/source-mysql/src/main/java/io/airbyte/integrations/source/mysql/MySqlSource.java
fetcher,

final Supplier<AutoCloseableIterator<AirbyteMessage>> incrementalIteratorSupplier = () -> handler.getIncrementalIterators(catalog,
new MySqlCdcSavedInfoFetcher(cdcState.orElse(null)),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The case of orElse is expected on a first time sync?

@subodh1810 subodh1810 merged commit 9974608 into master Sep 26, 2022
@subodh1810 subodh1810 deleted the snapshot-of-new-tables-cdc-mysql branch September 26, 2022 18:44
This was referenced Sep 27, 2022
Closed
Merged
robbinhan pushed a commit to robbinhan/airbyte that referenced this pull request Sep 29, 2022
@subodh1810 @octavia-squidington-iii @robbinhan
mysql source : implement support for snapshot of new tables in cdc mo…
459b8dd 
…de (airbytehq#16954)

* mysql source : implement support for snapshot of new tables in cdc mode

* undo unwanted changes

* add more assertions

* format

* fix build

* fix build

* revert acceptance test changes

* bump version

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <[email protected]>
jhammarstedt pushed a commit to jhammarstedt/airbyte that referenced this pull request Oct 31, 2022
@subodh1810 @octavia-squidington-iii @jhammarstedt
mysql source : implement support for snapshot of new tables in cdc mo…
aca29fb 
…de (airbytehq#16954)

* mysql source : implement support for snapshot of new tables in cdc mode

* undo unwanted changes

* add more assertions

* format

* fix build

* fix build

* revert acceptance test changes

* bump version

* auto-bump connector version [ci skip]

Co-authored-by: Octavia Squidington III <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rodireich rodireich rodireich approved these changes

@grishick grishick grishick approved these changes

@edgao edgao edgao approved these changes

@akashkulk akashkulk akashkulk approved these changes

Assignees

@subodh1810 subodh1810

Labels
area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/source/jdbc connectors/source/mysql connectors/source/mysql-strict-encrypt connectors/source/postgres
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Store database catalog for Debezium
6 participants
@subodh1810 @edgao @rodireich @grishick @akashkulk @octavia-squidington-iii