Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lightning failed to log encoding error #44321

Closed
lance6716 opened this issue Jun 1, 2023 · 0 comments · Fixed by #45241
Closed

lightning failed to log encoding error #44321

lance6716 opened this issue Jun 1, 2023 · 0 comments · Fixed by #45241
Assignees
Labels
affects-6.5 component/lightning This issue is related to Lightning of TiDB. severity/moderate type/bug The issue is confirmed as a bug.

Comments

@lance6716
Copy link
Contributor

func (e *BaseKVEncoder) LogKVConvertFailed(row []types.Datum, j int, colInfo *model.ColumnInfo, err error) error {
var original types.Datum
if 0 <= j && j < len(row) {
original = row[j]
row = row[j : j+1]
}
e.logger.Error("kv convert failed",
zap.Array("original", RowArrayMarshaller(row)),

When the source file has ill-formated row, the row is very large so it can't be logged. We can't locate the bad row

2023-06-01 10:55:27.887067902 +0800 CST m=+35.508861459 write error: write length 20297970322 exceeds maximum file size 536870912

image

@lance6716 lance6716 added the type/bug The issue is confirmed as a bug. label Jun 1, 2023
@lance6716 lance6716 changed the title lightning failed to record encoding error lightning failed to log encoding error Jun 1, 2023
@seiya-annie seiya-annie added severity/moderate component/lightning This issue is related to Lightning of TiDB. labels Jun 1, 2023
@lance6716 lance6716 self-assigned this Jun 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affects-6.5 component/lightning This issue is related to Lightning of TiDB. severity/moderate type/bug The issue is confirmed as a bug.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants