Added Debezium Source for MS SQL Server #12256

dlg99 · 2021-09-30T17:30:53Z

Motivation

Add Debezium Source for MS SQL Server

Modifications

Added the source, integration tests, and updated docs.

Verifying this change

Make sure that the change passes the CI checks.
- Added integration tests

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

Dependencies (does it add or upgrade a dependency): YES

debezium-connector-sqlserver (Apache 2.0 licensed)
which brings in mssql-jdbc (MIT licensed, https://github.com/microsoft/mssql-jdbc)

The integration test uses mcr.microsoft.com/mssql/server:2019-latest container.
See https://hub.docker.com/_/microsoft-mssql-server
EULA: https://go.microsoft.com/fwlink/?linkid=857698

You may install and use copies of the software on any device,
including third party shared devices, to design, develop, test and demonstrate your programs.
You may not use the software on a device or server in a production environment.

The public API: (yes / no)
The schema: (yes / no / don't know)
The default values of configurations: (yes / no)
The wire protocol: (yes / no)
The rest endpoints: (yes / no)
The admin cli options: (yes / no)
Anything that affects deployment: (yes / no / don't know)

Documentation

Check the box below and label this PR (if you have committer privilege).

Need to update docs?

doc

(If this PR contains doc changes)

eolivelli

Looks good to me.

I have one question before merging this patch:
what about the new third party dependencies introduced by Debezium connector ?
is there some MSSQL driver ? does it use jTDS ?
what about the LICENSE ?
can you add some explanation in the Description of the PR (it would be also very good to write it in the commit message, for further reference)

.../java/org/apache/pulsar/tests/integration/io/sources/debezium/DebeziumMsSqlSourceTester.java

dlg99 · 2021-10-01T16:08:03Z

@eolivelli I updated PR description with the dependencies details.

eolivelli

Lgtm

site2/docs/io-connectors.md

site2/docs/io-debezium-source.md

Anonymitaet · 2021-10-13T00:45:52Z

site2/docs/io-debezium-source.md

+### Configuration
+
+Debezium [requires](https://debezium.io/documentation/reference/1.5/connectors/sqlserver.html#sqlserver-overview) SQL Server with CDC enabled.
+Steps outlined in the [documentation](https://debezium.io/documentation/reference/1.5/connectors/sqlserver.html#setting-up-sqlserver) and used in the [integration test](https://github.com/apache/pulsar/blob/master/tests/integration/src/test/java/org/apache/pulsar/tests/integration/src/test/java/org/apache/pulsar/tests/integration/io/sources/debezium/DebeziumMsSqlSourceTester.java).


The link of integration test (https://github.com/apache/pulsar/blob/master/tests/integration/src/test/java/org/apache/pulsar/tests/integration/src/test/java/org/apache/pulsar/tests/integration/io/sources/debezium/DebeziumMsSqlSourceTester.java) is invalid

that's because it is not merged into master yet

Anonymitaet · 2021-10-13T00:47:12Z

site2/docs/io-debezium-source.md

+
+Debezium [requires](https://debezium.io/documentation/reference/1.5/connectors/sqlserver.html#sqlserver-overview) SQL Server with CDC enabled.
+Steps outlined in the [documentation](https://debezium.io/documentation/reference/1.5/connectors/sqlserver.html#setting-up-sqlserver) and used in the [integration test](https://github.com/apache/pulsar/blob/master/tests/integration/src/test/java/org/apache/pulsar/tests/integration/src/test/java/org/apache/pulsar/tests/integration/io/sources/debezium/DebeziumMsSqlSourceTester.java).
+Please refer to the [documentation for Microsoft SQL Server](https://docs.microsoft.com/en-us/sql/relational-databases/track-changes/enable-and-disable-change-data-capture-sql-server) as needed.


For more information, see Enable and disable change data capture in Microsoft SQL Server.

site2/docs/io-connectors.md

site2/docs/io-debezium-source.md

Co-authored-by: Anonymitaet <[email protected]>

dlg99 · 2021-10-13T01:00:37Z

@Anonymitaet I updated the doc

Anonymitaet

Approve from doc perspective.

* up/master: (26 commits) [pulsar-admin] Allow setting --forward-source-message-property to false when updating a pulsar function (apache#12128) [website][upgrade]feat: docs migration - Development (apache#12320) Update delete inactive topic configuration documentation (apache#12350) [PIP 95][Issue 12040][broker] Multiple bind addresses for Pulsar protocol (apache#12056) Added Debezium Source for MS SQL Server (apache#12256) Fix: flaky oracle tests (apache#12306) [C++] Use URL encoded content type for OAuth 2.0 authentication (apache#12341) [C++] Handle OAuth 2.0 exceptional cases gracefully (apache#12335) feat(cli): add restart command to pulsar-daemon (apache#12279) [client-tools] Remove redundant initial value (apache#12296) Make AuthenticationTokenTest to run on windows (apache#12329) [offload] fix FileSystemManagedLedgerOffloader can not cleanup outdated ledger data (apache#12309) [Doc]--Update contents for Pulsar adaptor for Apache Spark (apache#12338) [PIP 95][Issue 12040][broker] Improved multi-listener in standalone mode (apache#12066) [website][upgrade]feat: docs migration - Cookbooks (apache#12319) [testclient] Make --payload-file take effect in PerformanceClient (apache#12187) [website][upgrade]feat: docs migration - adaptor (apache#12318) [pulsar-client] Add partition-change api for producer/consumer interceptors (apache#12287) [Transaction]Fix lowWaterMark of TopicTransactionBuffer (apache#12312) [pulsar-admin] New option takes precedence over deprecated option (apache#12260) ... # Conflicts: # site2/website-next/docusaurus.config.js # site2/website-next/versions.json

* up/master: (37 commits) re-enabling integration tests for Sinks (apache#12307) [PIP 95][Issue 12040][web] Topic lookup with listener header (apache#12072) Fix the master CI broken with update dispatch rate block issue (apache#12360) Fix message being ignored when the non-persistent topic reader reconnect. (apache#12348) Fix log format. (apache#12346) [website][upgrade]feat: docs migration - version-2.7.2 Concepts and Architecture (apache#12354) [website][upgrade] feat: full docs migration for version 2.8.0 (apache#12359) [website][upgrade]feat: dynamic replace version info before build (apache#12337) Fix flaky tests: ElasticSearchClientTests (apache#12347) Use asyncCloseCursorLedger to replace cursorLedger.asyncClose method in the ManagedCursorImpl.VoidCallback#operationComplete (apache#12113) fix-npe-ZkBookieRackAffinityMapping (apache#11947) [pulsar-admin] Allow setting --forward-source-message-property to false when updating a pulsar function (apache#12128) [website][upgrade]feat: docs migration - Development (apache#12320) Update delete inactive topic configuration documentation (apache#12350) [PIP 95][Issue 12040][broker] Multiple bind addresses for Pulsar protocol (apache#12056) Added Debezium Source for MS SQL Server (apache#12256) Fix: flaky oracle tests (apache#12306) [C++] Use URL encoded content type for OAuth 2.0 authentication (apache#12341) [C++] Handle OAuth 2.0 exceptional cases gracefully (apache#12335) feat(cli): add restart command to pulsar-daemon (apache#12279) ... # Conflicts: # site2/website-next/docusaurus.config.js # site2/website-next/versioned_sidebars/version-2.7.2-sidebars.json # site2/website-next/versions.json

(cherry picked from commit 5c28686)

Added Deebzium Source for MS SQL Server

12cace7

dlg99 changed the title ~~Added Deebzium Source for MS SQL Server~~ Added Debezium Source for MS SQL Server Sep 30, 2021

Added docs

ec1e65f

eolivelli reviewed Oct 1, 2021

View reviewed changes

.../java/org/apache/pulsar/tests/integration/io/sources/debezium/DebeziumMsSqlSourceTester.java Show resolved Hide resolved

eolivelli approved these changes Oct 1, 2021

View reviewed changes

merlimat added this to the 2.10.0 milestone Oct 12, 2021

merlimat added the type/feature The PR added a new feature or issue requested a new feature label Oct 12, 2021

Anonymitaet reviewed Oct 13, 2021

View reviewed changes

Anonymitaet added the doc Your PR contains doc changes, no matter whether the changes are in markdown or code files. label Oct 13, 2021

dlg99 commented Oct 13, 2021

View reviewed changes

site2/docs/io-connectors.md Outdated Show resolved Hide resolved

dlg99 commented Oct 13, 2021

View reviewed changes

site2/docs/io-debezium-source.md Outdated Show resolved Hide resolved

Apply suggestions from code review

02945ee

Co-authored-by: Anonymitaet <[email protected]>

Anonymitaet approved these changes Oct 13, 2021

View reviewed changes

merlimat approved these changes Oct 13, 2021

View reviewed changes

eolivelli modified the milestones: 2.10.0, 2.9.0 Oct 13, 2021

eolivelli merged commit 5c28686 into apache:master Oct 13, 2021

dlg99 deleted the mssql_debezium branch October 14, 2021 23:31

dlg99 added a commit to dlg99/pulsar that referenced this pull request Mar 7, 2022

Added Debezium Source for MS SQL Server (apache#12256)

9549650

(cherry picked from commit 5c28686)

dlg99 added a commit to dlg99/pulsar that referenced this pull request Mar 7, 2022

Added Debezium Source for MS SQL Server (apache#12256)

67e7ce5

(cherry picked from commit 5c28686)

eolivelli pushed a commit to datastax/pulsar that referenced this pull request Mar 8, 2022

Added Debezium Source for MS SQL Server (apache#12256)

0e98fbd

(cherry picked from commit 5c28686)

eolivelli pushed a commit to datastax/pulsar that referenced this pull request Mar 8, 2022

Added Debezium Source for MS SQL Server (apache#12256)

e9897ea

(cherry picked from commit 5c28686)

bharanic-dev pushed a commit to bharanic-dev/pulsar that referenced this pull request Mar 18, 2022

Added Debezium Source for MS SQL Server (apache#12256)

e878ff8

This was referenced Nov 10, 2022

[SECURITY] Hardcode the password in debezium-{}-config.yaml #14324

Closed

PulsarIO [Debezium] - Support MS SqlServer #6957

Closed

[Issue 6957][pulsar-io-debezium] Support CDC Connector for SqlServer #7042

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added Debezium Source for MS SQL Server #12256

Added Debezium Source for MS SQL Server #12256

dlg99 commented Sep 30, 2021 •

edited

Loading

eolivelli left a comment

dlg99 commented Oct 1, 2021

eolivelli left a comment

Anonymitaet Oct 13, 2021

dlg99 Oct 13, 2021

Anonymitaet Oct 13, 2021

dlg99 commented Oct 13, 2021

Anonymitaet left a comment

Added Debezium Source for MS SQL Server #12256

Added Debezium Source for MS SQL Server #12256

Conversation

dlg99 commented Sep 30, 2021 • edited Loading

Motivation

Modifications

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

eolivelli left a comment

Choose a reason for hiding this comment

dlg99 commented Oct 1, 2021

eolivelli left a comment

Choose a reason for hiding this comment

Anonymitaet Oct 13, 2021

Choose a reason for hiding this comment

dlg99 Oct 13, 2021

Choose a reason for hiding this comment

Anonymitaet Oct 13, 2021

Choose a reason for hiding this comment

dlg99 commented Oct 13, 2021

Anonymitaet left a comment

Choose a reason for hiding this comment

dlg99 commented Sep 30, 2021 •

edited

Loading