Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[INLONG-7908][Sort] PostgreSQL connector supports parallel read #8664

Merged
merged 18 commits into from
Sep 15, 2023
Merged

[INLONG-7908][Sort] PostgreSQL connector supports parallel read #8664

merged 18 commits into from
Sep 15, 2023

Conversation

kuansix
Copy link
Contributor

@kuansix kuansix commented Aug 8, 2023 •
edited
Loading

Prepare a Pull Request

Motivation

Enable incremental snapshot (support parallel read) and adapt metrics for PostgreSQL CDC.

Modifications

1. package org.apache.inlong.sort.cdc.postgres.source

  • The core modification is implemented, refer to MySQL/Oracle's incremental snapshotting implementation.
  • The feature can be enabled by setting scan.incremental.snapshot.enabled=true.

2.package io.debezium.connector.postgresql

  • Made some modifications to getting around the limitation.

3.package org.apache.inlong.sort.cdc.base

  • Add a new field List schemaList and make it compatible with PostgreSQL and Debezium’s terminology.
  • Add parameters related to incremental snapshot.
  • Add parameters related to metrics.
  • Adapt incremental snapshot and metrics.

Documentation

Documentation

@dockerzhang dockerzhang requested review from gong and EMsnap August 8, 2023 07:13
@dockerzhang
Copy link
Contributor

@kuansix, please add more description for this PR.

@EMsnap
Copy link
Contributor

EMsnap commented Aug 9, 2023

Please check the ut and compile thanks

@healchow healchow changed the title [INLONG-7908][Sort] Postgres connector supports parallel read [INLONG-7908][Sort] PostgreSQL connector supports parallel read Aug 9, 2023
@kuansix
Copy link
Contributor Author

kuansix commented Aug 13, 2023

@kuansix, please add more description for this PR.

done

@kuansix
Copy link
Contributor Author

kuansix commented Aug 14, 2023

Please check the ut and compile thanks

done

EMsnap
EMsnap reviewed Aug 15, 2023
View reviewed changes
Copy link
Contributor

@EMsnap EMsnap left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a test on parallel read and I guess some of the classes can reuse flink-cdc rather than copying them

@kuansix
Copy link
Contributor Author

kuansix commented Sep 13, 2023 •
edited
Loading

Please add a test on parallel read and I guess some of the classes can reuse flink-cdc rather than copying them

1.add a test (done)
2.we have changed many class in postgres.source, and it will conflict if we use flink-cdc all. just like mysql oracle ...

@dockerzhang dockerzhang merged commit a63b1d9 into apache:master Sep 15, 2023
9 checks passed
liaosunny123 pushed a commit to liaosunny123/inlong that referenced this pull request Oct 11, 2023
@kuansix @e-mhui @liaosunny123
[INLONG-7908][Sort] PostgreSQL connector supports parallel read (apac…
60f9c09 
…he#8664)

Co-authored-by: e-mhui <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@EMsnap EMsnap EMsnap left review comments

@gong gong gong approved these changes

@dockerzhang dockerzhang dockerzhang approved these changes

@yunqingmoswu yunqingmoswu yunqingmoswu approved these changes

@e-mhui e-mhui e-mhui approved these changes

Assignees
No one assigned
Labels
component/sort
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Feature][Sort] PostgreSQL connector supports parallel read
6 participants
@kuansix @dockerzhang @EMsnap @gong @yunqingmoswu @e-mhui