Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sink/(ticdc): Add csv encoder into codec #6942

Merged
merged 23 commits into from
Oct 10, 2022
Merged

sink/(ticdc): Add csv encoder into codec #6942

merged 23 commits into from
Oct 10, 2022

Conversation

zhaoxinyu
Copy link
Contributor

@zhaoxinyu zhaoxinyu commented Aug 31, 2022 •
edited
Loading

What problem does this PR solve?

Issue Number: ref #6797

What is changed and how it works?

Add a feature-rich csv encoder in codec package.
This encoder will convert cdc row changed events into following format:

"I","employee","hr",433305438660591626,101,"Smith","Bob","2014-06-04","New York"
"U","employee","hr",433305438660591627,101,"Smith","Bob","2015-10-08","Los Angeles"
"D","employee","hr",433305438660591629,101,"Smith","Bob","2017-03-13","Dallas"

The meaning of each csv column is:

  • Col1: The operation-type indicator: I, D, U; I means INSERT, U means UPDATE, D means DELETE.
  • Col2: Table name, the name of the source table.
  • Col3: Schema name, the name of the source schema.
  • Col4: Commit TS, the commit-ts of the source txn. The exist of this column can be configured.
  • Col5-n: one or more columns that represent the data to be changed.

The csv encoder has the following basic rules:

  • If the column is a string, quote it if necessary and then output; otherwise directly output it.
  • If quote is not empty and quote mark is contained in the string column, then replace the quote mark in column with two quote marks.
  • If quote is empty and the string column contains special characters like '\r', '\n', '\' or delimiter, then escape those characters.

Check List

Tests

  • Unit test
  • Integration test

Questions

Will it cause performance regression or break compatibility?

No

Do you need to update user documentation, design documentation or monitoring documentation?

Yes

Release note 

None

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Aug 31, 2022 •
edited
Loading

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • 3AceShowHand
  • asddongmen

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note-none Denotes a PR that doesn't merit a release note. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Aug 31, 2022
@zhaoxinyu zhaoxinyu changed the title [WIP]sink/(ticdc): Add csv encoder into codec sink/(ticdc): Add csv encoder into codec Sep 2, 2022
@ti-chi-bot ti-chi-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 2, 2022
@zhaoxinyu
Copy link
Contributor Author

/cc @asddongmen
/cc @hi-rustin
/cc @3AceShowHand

@ti-chi-bot ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 2, 2022
@Rustin170506
Copy link
Member

Please ping me when this PR is ready!

Rustin170506
Rustin170506 reviewed Sep 8, 2022
View reviewed changes
cdc/sink/codec/csv/csv_message.go

func (c *csvMessage) encode() []byte {
strBuilder := new(strings.Builder)
c.formatValue(c.opType, strBuilder)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

formatValue actually appends the value into the builder. So I think the name may be confusing.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you give some advise about this function name here?

cdc/sink/codec/csv/csv_message.go Show resolved Hide resolved
cdc/sink/codec/csv/csv_message.go Show resolved Hide resolved
cdc/sink/codec/csv/csv_message.go
strBuilder.WriteRune(rune(ch))

if isDelimiterStart {
for k := 1; k < len(c.csvConfig.Delimiter); k++ {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

cdc/sink/codec/csv/csv_message.go
}

func (c *csvMessage) formatValue(value any, strBuilder *strings.Builder) {
defer func() {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

nongfushanquan
nongfushanquan reviewed Sep 13, 2022
View reviewed changes
cdc/sink/codec/csv/csv_message.go
newRecord: true,
}
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you add some examples about how to handle different strings

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's covered in unit tests.

@ti-chi-bot ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 17, 2022
@zhaoxinyu
Copy link
Contributor Author

/run-verify

@codecov-commenter
Copy link

codecov-commenter commented Sep 27, 2022 •
edited
Loading

Codecov Report

Merging #6942 (01e8b1f) into master (d8fff7a) will increase coverage by 0.0578%.
The diff coverage is 75.0936%.

Additional details and impacted files
Flag Coverage Δ
cdc 67.2560% <78.2051%> (+0.0404%) ⬆️
dm 51.8601% <ø> (+0.0226%) ⬆️
engine 60.9259% <70.7207%> (+0.0678%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

@@               Coverage Diff                @@
##             master      #6942        +/-   ##
================================================
+ Coverage   59.6043%   59.6621%   +0.0578%     
================================================
  Files           790        792         +2     
  Lines         91089      91403       +314     
================================================
+ Hits          54293      54533       +240     
- Misses        32051      32109        +58     
- Partials       4745       4761        +16

@ti-chi-bot ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 1, 2022
@ti-chi-bot ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 9, 2022
@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Oct 9, 2022
@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Oct 10, 2022
@Rustin170506
Copy link
Member

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 7f41d6f

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Oct 10, 2022
@ti-chi-bot
Copy link
Member

@zhaoxinyu: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot ti-chi-bot merged commit 13e6f48 into pingcap:master Oct 10, 2022
3AceShowHand pushed a commit to 3AceShowHand/tiflow that referenced this pull request Oct 10, 2022
@zhaoxinyu @3AceShowHand
sink/(ticdc): Add csv encoder into codec (pingcap#6942)
40be734 
ref pingcap#6797
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nongfushanquan nongfushanquan nongfushanquan left review comments

@3AceShowHand 3AceShowHand 3AceShowHand approved these changes

@asddongmen asddongmen asddongmen approved these changes

@Rustin170506 Rustin170506 Rustin170506 approved these changes

Assignees
No one assigned
Labels
release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@zhaoxinyu @ti-chi-bot @Rustin170506 @codecov-commenter @3AceShowHand @asddongmen @nongfushanquan