Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

import: add skip-header-row parameter for csv #41070

Closed
wants to merge 2 commits into from
Closed

import: add skip-header-row parameter for csv #41070

wants to merge 2 commits into from

Conversation

dsdashun
Copy link
Contributor

@dsdashun dsdashun commented Feb 5, 2023

What problem does this PR solve?

Issue Number: ref #40839

Problem Summary:

What is changed and how it works?

  • Introduce a new configuration parameter, skip-header-row, to allow skipping the first row of each CSV file
  • If csv.header is set to true, it implies that skip-header-row is also true
  • Enhance the feature to support skipping multiple rows for all data types, not just CSV, by making each chunk have a configurable skip rows
  • Enhance splitting large CSV chunks into smaller chunks with some unnecessary chunks dropped as well as proper row skipping for each accepted chunk

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

Introduce a new configuration parameter in Lightning to skip the first row in each CSV file.

@ti-chi-bot
Copy link
Member

[REVIEW NOTIFICATION]

This pull request has not been approved.

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Feb 5, 2023
@dsdashun
Copy link
Contributor Author

dsdashun commented Feb 6, 2023

/run-integration-br-test

@dsdashun
Copy link
Contributor Author

dsdashun commented Feb 6, 2023

/run-integration-br-test

lance6716
lance6716 reviewed Feb 6, 2023
View reviewed changes
Copy link
Contributor

@lance6716 lance6716 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure we should make skipNRows as a functionality of parser, so LOAD DATA can reuse it

br/pkg/lightning/mydump/region.go
if remainingSkipRows > 0 {
iterPos, _ := parser.Pos()
for remainingSkipRows > 0 && iterPos < endOffset {
if err := parser.ReadRow(); err != nil {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In #40852 I use parser.ReadUntilTerminator because the skipped row may contain bad syntax

br/pkg/lightning/mydump/region.go
PrevRowIDMax: prevRowIdxMax,
RowIDMax: rowIDMax,
Columns: columns,
SkipFirstNRows: rowsToSkip,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changing startOffset is enough? no need to add SkipFirstNRows

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've considered this problem. We need the SkipFirstNRows, because for some data format like parquet, it is not stored row by row. In this situation, we cannot just specify a offset to skip first N rows.

@lance6716
Copy link
Contributor

I'm not sure we should make skipNRows as a functionality of parser, so LOAD DATA can reuse it

@gozssky PTAL

@dsdashun
Copy link
Contributor Author

dsdashun commented Feb 7, 2023

Close the PR because of the discussion in #40839 (comment)

@dsdashun dsdashun closed this Feb 7, 2023
@dsdashun dsdashun deleted the fix-40839 branch February 7, 2023 06:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lance6716 lance6716 lance6716 left review comments

@sleepymole sleepymole Awaiting requested review from sleepymole

@okJiang okJiang Awaiting requested review from okJiang

@lichunzhu lichunzhu Awaiting requested review from lichunzhu

Assignees
No one assigned
Labels
release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@dsdashun @ti-chi-bot @lance6716