Source MSSQL: ability to add new table and reset specific table without resetting entire workflow

[sivankumar86]

Tell us about the problem you're trying to solve

We have an usecase that we need to reset specific table in cdc sync job and add new table however, it required to reset all the rest of the tables which was set up in the job.

Describe the solution you’d like

• Have an option to reset specific table
• only run initial load/rest for the new tables which are added to the set up

Describe the alternative you’ve considered or used

• Created multiple job/connections with small number of tables.
• create a new connection for new tables
  Example,
  we have 100 tables and I have created 20 connections for sync job hence we need reset only 5 tables for any changes

Additional context

[grishick]

per-stream state has been implemented and it should be possible to add this feature now to MySQL Source following what we did for Postgres Source

[tuliren]

This issue depends on #15248.

[tuliren]

TODOs:

• Non CDC: emit per-stream state
• CDC
  • Wait for Store database catalog for Debezium #15471.
  • Emit per-stream state (copy the work from MySQL to MSSQL).
• Testing

[tuliren]

Hey team! Please add your planning poker estimate with ZenHub @ryankfu @subodh1810 @edgao @rodireich

[grishick]

@misteryeo I am adding this to MySQL To Beta epic, because this is part of "feature completeness". Essentially, this is "support per stream state in MySQL Source"

[misteryeo]

@grishick We removed it because this is specifically for MSSQL. I've created a separate one here that Subodh's already started working on: #15725 :) I'll let you read this and remove!

[grishick]

@grishick We removed it because this is specifically for MSSQL. I've created a separate one here that Subodh's already started working on: #15725 :) I'll let you read this and remove!

I must have gone blind last night and read MySQL instead of MS SQL :)

[prateekmukhedkar]

Available with per-stream functionality in the latest version of MS-SQL connector