Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TiCDC can't recover after TiDB of upstream OOM #8561

Closed
WangLe1321 opened this issue Mar 17, 2023 · 3 comments · Fixed by tikv/tikv#14465
Closed

TiCDC can't recover after TiDB of upstream OOM #8561

WangLe1321 opened this issue Mar 17, 2023 · 3 comments · Fixed by tikv/tikv#14465
Assignees
@overvenus
Labels
affects-6.0 affects-6.1 affects-6.2 affects-6.3 affects-6.4 affects-6.5 affects-6.6 affects-7.0 area/ticdc Issues or PRs related to TiCDC. severity/major type/bug The issue is confirmed as a bug.

Comments

@WangLe1321
Copy link

What did you do?

TiDB of upstream has too high qps and connections, causing TIDB OOM.
metrics of TiDB:
https://clinic.pingcap.com/grafana/d/000000013tidbcloudv6_1_x/tidb-runtime-cluster-clinic?orgId=1&var-tidb_cluster=1379661944640134080&var-instance=All&var-store=All&from=1678407999000&to=1678415199000&var-tidb_version=v6.1.x&var-org=1372813089196917631&refresh=15m&var-org_type=cloud
https://clinic.pingcap.com/grafana/d/2bbdcd7d4d88a2dtidbcloudv6_1_x/tidb-summary-cluster-clinic?orgId=1&var-tidb_cluster=1379661944640134080&var-instance=All&var-store=All&from=1678407999000&to=1678415199000&var-tidb_version=v6.1.x&var-org=1372813089196917631&refresh=15m&var-org_type=cloud
https://clinic.pingcap.com/portal/#/orgs/1372813089196917631/clusters/1379661944640134080/report/5/fed8c3b6-fe7a-48f7-80e1-100dacf18938

after scale up TiDB nodes, TiDB recovered.

What did you expect to see?

After TiDB recovered, cdc should reduce lag.

What did you see instead?

cdc lag is still high.

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

(paste TiDB cluster version here)

Upstream TiKV version (execute tikv-server --version):

(paste TiKV version here)

TiCDC version (execute cdc version):

(paste TiCDC version here)

https://clinic.pingcap.com/portal/#/orgs/1372813089196917631/clusters/1379661944640134080

cluster version is v6.1.3
@WangLe1321 WangLe1321 added area/ticdc Issues or PRs related to TiCDC. type/bug The issue is confirmed as a bug. labels Mar 17, 2023
@fubinzh
Copy link

fubinzh commented Mar 17, 2023

The issue can be work around per #4516 (comment)

@fubinzh
Copy link

fubinzh commented Mar 17, 2023

/severity major

@nongfushanquan
Copy link
Contributor

/assign @overvenus

@overvenus overvenus mentioned this issue Mar 27, 2023
Merged
ti-chi-bot added a commit to tikv/tikv that referenced this issue Apr 7, 2023
@overvenus @ti-chi-bot
cdc: batch send resolved ts exponentially to speed up TiCDC resolve l…
83ce091 
…ock (#14465)

close pingcap/tiflow#8561, ref #11993

cdc: batch send resolved ts exponentially to speed up TiCDC resolve lock

Signed-off-by: Neil Shen <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>
@ti-chi-bot ti-chi-bot mentioned this issue May 23, 2023
Merged
ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this issue May 23, 2023
@overvenus @ti-chi-bot
This is an automated cherry-pick of tikv#14465
4704551 
close pingcap/tiflow#8561, ref tikv#11993

Signed-off-by: ti-chi-bot <[email protected]>
overvenus added a commit to ti-chi-bot/tikv that referenced this issue May 23, 2023
@overvenus @ti-chi-bot
cdc: batch send resolved ts exponentially to speed up TiCDC resolve l…
03fd8e3 
…ock (tikv#14465)

close pingcap/tiflow#8561, ref tikv#11993

Signed-off-by: Neil Shen <[email protected]>

Co-authored-by: Ti Chi Robot <[email protected]>
Signed-off-by: Neil Shen <[email protected]>
ti-chi-bot bot pushed a commit to tikv/tikv that referenced this issue May 23, 2023
@ti-chi-bot @overvenus @Rustin170506
cdc: batch send resolved ts exponentially to speed up TiCDC resolve l…
49a5c96 
…ock (#14465) (#14796)

close pingcap/tiflow#8561, ref #11993, ref #14465

cdc: batch send resolved ts exponentially to speed up TiCDC resolve lock

Signed-off-by: Neil Shen <[email protected]>

Co-authored-by: Neil Shen <[email protected]>
Co-authored-by: 二手掉包工程师 <[email protected]>
Co-authored-by: Ti Chi Robot <[email protected]>
@nongfushanquan nongfushanquan mentioned this issue Jun 9, 2023
Merged
17 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@overvenus overvenus

Labels
affects-6.0 affects-6.1 affects-6.2 affects-6.3 affects-6.4 affects-6.5 affects-6.6 affects-7.0 area/ticdc Issues or PRs related to TiCDC. severity/major type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

cdc: batch send resolved ts exponentially to speed up TiCDC resolve lock overvenus/tikv
6 participants
@overvenus @fubinzh @VelocityLight @WangLe1321 @ti-chi-bot @nongfushanquan