chore: remove shadow write of new sip 68 dataset models #21986

eschutho · 2022-10-31T23:34:12Z

SUMMARY

We're seeing some sqlalchemy errors related to the dual writes of the sip68 models, so I'm removing the dual writing so that we can decide whether to move forward again and do another backfill migration or start over smaller with one model at a time. We also haven't been writing permissions to the models, so we would need to run a migration for that in the future regardless of whether we stopped dual writing or not, so it seemed to make sense to take a step back and stop the errors for now.

I also fixed a test on dataset deletion due to sqlalchemy not having access to the database relationship in the session. The fix was to eagerly load the database with the dataset on lookup. I found other logs of this happening for this command, and fixed it here for the tests to pass. There was a query in the after update hook that I removed that was re-loading the database into the session as an unexpected side-effect.

TESTING INSTRUCTIONS

Normal dataset crud operations should work as normal.

ADDITIONAL INFORMATION

Has associated issue:
Required feature flags:
Changes UI
Includes DB Migration (follow approval process in SIP-59)
- Migration is atomic, supports rollback & is backwards-compatible
- Confirm DB migration upgrade and downgrade tested
- Runtime estimates and downtime expectations provided
Introduces new feature or API
Removes existing feature or API

codecov · 2022-11-09T17:59:52Z

Codecov Report

Merging #21986 (0373446) into master (33fc578) will decrease coverage by 0.04%.
The diff coverage is 77.91%.

❗ Current head 0373446 differs from pull request most recent head d2523c8. Consider uploading reports for the commit d2523c8 to get more accurate results

@@            Coverage Diff             @@
##           master   #21986      +/-   ##
==========================================
- Coverage   66.95%   66.91%   -0.05%     
==========================================
  Files        1807     1818      +11     
  Lines       69222    69471     +249     
  Branches     7403     7496      +93     
==========================================
+ Hits        46351    46484     +133     
- Misses      20966    21049      +83     
- Partials     1905     1938      +33

Flag	Coverage Δ
hive	`52.50% <40.94%> (-0.42%)`	⬇️
mysql	`78.14% <79.13%> (-0.23%)`	⬇️
postgres	`?`
presto	`52.40% <40.55%> (-0.43%)`	⬇️
python	`81.19% <87.40%> (-0.30%)`	⬇️
sqlite	`?`
unit	`50.73% <59.84%> (-0.36%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...ackages/superset-ui-chart-controls/src/fixtures.ts	`100.00% <ø> (ø)`
...d/packages/superset-ui-chart-controls/src/index.ts	`100.00% <ø> (ø)`
...perset-ui-chart-controls/src/sections/sections.tsx	`71.42% <0.00%> (-16.08%)`	⬇️
...chart-controls/src/shared-controls/dndControls.tsx	`58.33% <ø> (ø)`
...d/packages/superset-ui-chart-controls/src/types.ts	`100.00% <ø> (ø)`
...kages/superset-ui-core/src/query/types/Operator.ts	`100.00% <ø> (ø)`
...packages/superset-ui-core/src/query/types/Query.ts	`100.00% <ø> (ø)`
...set-ui-core/src/ui-overrides/ExtensionsRegistry.ts	`100.00% <ø> (ø)`
.../legacy-plugin-chart-histogram/src/controlPanel.ts	`40.00% <0.00%> (ø)`
...lugin-chart-handlebars/src/plugin/controlPanel.tsx	`50.00% <ø> (ø)`
... and 137 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

ktmud

@betodealmeida definitely is more qualified of reviewing this than me, but I agree the dual-write is probably not ideal if we aren't going to resume working on the new models any time soon.

If it's something you are allow to share, can you elaborate what were the errors you saw?

ktmud · 2022-11-13T08:46:11Z

superset/datasets/dao.py

@@ -35,6 +37,26 @@ class DatasetDAO(BaseDAO):  # pylint: disable=too-many-public-methods
    model_cls = SqlaTable
    base_filter = DatasourceFilter

+    @classmethod
+    def find_by_ids(cls, model_ids: Union[List[str], List[int]]) -> List[SqlaTable]:


Is this related?

Looks unrelated... maybe this was supposed to be in another branch?

Sorry I thought I had mentioned it in the PR description, but yes tests were failing on dataset deletion due to sqlalchemy not having access to the database relationship in the session. I found other logs of this happening for this command, and fixed it here for the tests to pass. There was a query in the after update hook that I removed that was re-loading the database into the session as an unexpected side-effect.

betodealmeida

betodealmeida · 2022-11-14T17:08:10Z

superset/datasets/dao.py

@@ -35,6 +37,26 @@ class DatasetDAO(BaseDAO):  # pylint: disable=too-many-public-methods
    model_cls = SqlaTable
    base_filter = DatasourceFilter

+    @classmethod
+    def find_by_ids(cls, model_ids: Union[List[str], List[int]]) -> List[SqlaTable]:


Looks unrelated... maybe this was supposed to be in another branch?

john-bodley · 2022-11-14T21:49:20Z

@betodealmeida and @eschutho what’s the status of SIP-68? Are there blockers for preventing us from reading from the new datamodel tables?

eschutho · 2022-11-15T02:02:32Z

@john-bodley after this change, the new datasets/models and the sqlaTable/columns, etc will be out of sync. Prior to this change, some of the permissions may have been out of sync as well. We're going to try this again, but maybe with a model by model approach so that we can maybe knock out some of the new models quicker and then do the dataset change last, which will be the most complex. We just haven't had much time to contribute to wrapping this up unfortunately.

(cherry picked from commit 86d52fc)

pull-request-size bot added the size/XL label Oct 31, 2022

eschutho force-pushed the elizabeth/revert-SIP68-dual-write branch from 648f708 to a0008fe Compare October 31, 2022 23:44

pull-request-size bot added size/L and removed size/XL labels Oct 31, 2022

eschutho force-pushed the elizabeth/revert-SIP68-dual-write branch 2 times, most recently from 839ea7e to 92a8d28 Compare October 31, 2022 23:52

eschutho changed the title ~~chore: remove shadow write of new sip 68 dataset models~~ DRAFT: chore: remove shadow write of new sip 68 dataset models Nov 1, 2022

eschutho force-pushed the elizabeth/revert-SIP68-dual-write branch 2 times, most recently from de7fb3c to ee16eca Compare November 2, 2022 18:59

pull-request-size bot added size/XL and removed size/L labels Nov 2, 2022

eschutho force-pushed the elizabeth/revert-SIP68-dual-write branch from ee16eca to 880f9d3 Compare November 4, 2022 23:59

pull-request-size bot added size/XXL and removed size/XL labels Nov 4, 2022

eschutho force-pushed the elizabeth/revert-SIP68-dual-write branch from 880f9d3 to 3384e12 Compare November 9, 2022 17:51

remove shadow write of new sip 68 dataset models

d2523c8

eschutho force-pushed the elizabeth/revert-SIP68-dual-write branch from 3384e12 to d2523c8 Compare November 10, 2022 18:28

eschutho changed the title ~~DRAFT: chore: remove shadow write of new sip 68 dataset models~~ chore: remove shadow write of new sip 68 dataset models Nov 10, 2022

eschutho requested review from hughhhh, betodealmeida and ktmud November 10, 2022 22:34

ktmud approved these changes Nov 13, 2022

View reviewed changes

betodealmeida approved these changes Nov 14, 2022

View reviewed changes

eschutho merged commit 86d52fc into apache:master Nov 14, 2022

eschutho mentioned this pull request Nov 15, 2022

fix: add back database lookup from sip 68 revert #22129

Merged

9 tasks

AnushaErrabelli pushed a commit to preset-io/superset that referenced this pull request Nov 15, 2022

chore: remove shadow write of new sip 68 dataset models (apache#21986)

be88027

(cherry picked from commit 86d52fc)

mistercrunch added the 🚢 2.1.3 label Feb 18, 2024

mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 2.1.0 and removed 🚢 2.1.3 labels Mar 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: remove shadow write of new sip 68 dataset models #21986

chore: remove shadow write of new sip 68 dataset models #21986

eschutho commented Oct 31, 2022 •

edited

Loading

codecov bot commented Nov 9, 2022 •

edited

Loading

ktmud left a comment •

edited

Loading

ktmud Nov 13, 2022

betodealmeida Nov 14, 2022

eschutho Nov 14, 2022 •

edited

Loading

betodealmeida left a comment

betodealmeida Nov 14, 2022

john-bodley commented Nov 14, 2022

eschutho commented Nov 15, 2022

chore: remove shadow write of new sip 68 dataset models #21986

chore: remove shadow write of new sip 68 dataset models #21986

Conversation

eschutho commented Oct 31, 2022 • edited Loading

SUMMARY

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

codecov bot commented Nov 9, 2022 • edited Loading

Codecov Report

ktmud left a comment • edited Loading

Choose a reason for hiding this comment

ktmud Nov 13, 2022

Choose a reason for hiding this comment

betodealmeida Nov 14, 2022

Choose a reason for hiding this comment

eschutho Nov 14, 2022 • edited Loading

Choose a reason for hiding this comment

betodealmeida left a comment

Choose a reason for hiding this comment

betodealmeida Nov 14, 2022

Choose a reason for hiding this comment

john-bodley commented Nov 14, 2022

eschutho commented Nov 15, 2022

eschutho commented Oct 31, 2022 •

edited

Loading

codecov bot commented Nov 9, 2022 •

edited

Loading

ktmud left a comment •

edited

Loading

eschutho Nov 14, 2022 •

edited

Loading