[CDCSDK] GetConsistentChanges fails on colocated tables having indexes #23809
Assignees
Labels
area/cdcsdk
CDC SDK
jira-originated
kind/bug
This issue is a bug
priority/highest
Highest priority issue
Comments
yugabyte-ci
added
area/cdcsdk
yugabyte-ci
added
priority/highest
Sumukh-Phalgaonkar
added a commit
that referenced
this issue
Sep 11, 2024
…colocated DBs Summary: If an index is created on a colocated table, the index table also resides on the same colocated tablet. If GetChanges is called on such a tablet, then we may encounter intents corresponding to the index table. Currently for the gRPC model of consumption cdc_service sends such records for index tables and the connector filters them. However for logical replication, the GetChanges call fails when it encounters the intents corresponding to index tables. This is because for logical replication, we try to find the replica identity corresponding to the index table in stream metadata. However since index tables are not part of CDC streams, this operation fails causing GetChanges to return with non-ok status. In this diff we introduce filtering mechanism by which we filter out the intents corresponding to the index and other non-eligible tables. A record from colocated tablet is shipped from cdc_service if it belongs to a tablet in qualified tables list and does not belong to a table in unqualified tables list. Jira: DB-12710 Test Plan: ./yb_build.sh --cxx-test integration-tests_cdcsdk_consumption_consistent_changes-test --gtest_filter CDCSDKConsumptionConsistentChangesTest.TestColocationWithIndexes Reviewers: skumar, siddharth.shah, asrinivasan Reviewed By: skumar Subscribers: steve.varnau, svc_phabricator, ycdcxcluster Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D37929
Sumukh-Phalgaonkar
added a commit
that referenced
this issue
Sep 12, 2024
… index tables in colocated DBs Summary: ####Backport Description No merge conflicts were encountered. ####Original Description If an index is created on a colocated table, the index table also resides on the same colocated tablet. If GetChanges is called on such a tablet, then we may encounter intents corresponding to the index table. Currently for the gRPC model of consumption cdc_service sends such records for index tables and the connector filters them. However for logical replication, the GetChanges call fails when it encounters the intents corresponding to index tables. This is because for logical replication, we try to find the replica identity corresponding to the index table in stream metadata. However since index tables are not part of CDC streams, this operation fails causing GetChanges to return with non-ok status. In this diff we introduce filtering mechanism by which we filter out the intents corresponding to the index and other non-eligible tables. A record from colocated tablet is shipped from cdc_service if it belongs to a tablet in qualified tables list and does not belong to a table in unqualified tables list. Jira: DB-12710 Original commit: 361a99a / D37929 Test Plan: ./yb_build.sh --cxx-test integration-tests_cdcsdk_consumption_consistent_changes-test --gtest_filter CDCSDKConsumptionConsistentChangesTest.TestColocationWithIndexes Reviewers: skumar, siddharth.shah, asrinivasan Reviewed By: skumar Subscribers: ycdcxcluster, svc_phabricator, steve.varnau Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D37987
jasonyb
pushed a commit
that referenced
this issue
Sep 12, 2024
Summary: aec9a66 [doc][xcluster] Truncate limitation (#23833) fe8890d [PLAT-13984] Change default metric graph points count from 100 to 250 + make runtime configurable. 7d9b57b [#23869] YSQL: Fix one type of ddl atomicity stress test flakiness 7c1bca8 [PLAT-14052][PLAT-15237] :Add advanced date-time option,Restrict CP for K8s afce6ad [PLAT-14158] Support multiple transactional xCluster configs per universe on YBA UI 33342b3 [PLAT-15101] Add runtime config to turn on/off rollN for k8s 361a99a [#23809] CDCSDK: Filter out records corresponding to index tables in colocated DBs 8bbdf66 [docs] changed date (#23885) ee639f4 [#22104,#23506] YSQL: auto analyze service collects mutation counts from DDL and skips indexes 3dbf6da Delete architecture/design/multi-region-xcluster-async-replication.md e013578 [#23864] DocDB: Move cluster_uuid to server common flags 2b6a2d3 [PLAT-15062][PLAT-15071] Support DB scoped on UI and display schema change mode information 8260075 [PLAT-15079] Treat dropped on target tables as unconfigured but preselected f5169ca DocDB: Follow redirects for callhome, and fix URL eb61ef6 [PLAT-15228] Update package installation command for YBA e791c40 [#18822] YSQL: Add serialization/deserialization mechanism for update optimization metadata e69d8cb [doc] Backups and DDLs (#23840) e72ae64 [PLAT-14552]filter support bundle core files by date 8796c83 [PLAT-15274]: BackFill Pitr params for DR config and add not-null constraints in DB. dc9cc67 Fixed NPM test:ci build error 2e5ebef [PLAT-15287]: Add PITR Column and Restore Window Information to Backup List d053e45 [#238989]yugabyted: Node doesn't join using `--join` flag da4da45 [#23399] DocDB: Fix StopActiveTxnsPriorTo from using local variables in the call back Test Plan: Jenkins: rebase: pg15-cherrypicks Reviewers: jason, tfoucher Differential Revision: https://phorge.dev.yugabyte.com/D38008
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Jira Link: DB-12710
The text was updated successfully, but these errors were encountered: