Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(migration): handle permalink edge cases correctly #23980

Merged
merged 3 commits into from
May 9, 2023

Conversation

villebro
Copy link
Member

@villebro villebro commented May 9, 2023

SUMMARY

This is a continuation of #23888 and addresses two issues that were identified after this PR was merged:

  • On MySQL, large key-value entries that had been persisted prior to fix(migration): Ensure key_value LargeBinary is encoded as a MEDIUMBLOB as opposed to BLOB for MySQL #20385 and exceeded the column size would cause the migration to fail, as the pickled value was truncated. This checks for truncation errors during deserialization, and replaces the value with an empty dict to ensure that the value is deserializable in the future.
  • Deserializing Explore permalinks failed, as one of the values in the dict was a string Enum that hadn't been cast to a pure str (apparently my devenv didn't have a single Explore permalink when I ran the migration 🙁). For this reason we add a check to find_class to make sure superset.utils.core.DatasourceType is a valid class. Note that this problem will now go away, as the JSON codec ensures that the Enum becomes a simple string in the future. In addition and to be more explicit, we cast datasourceType in the Explore payload of the create command to a str which is the correct type based on the ExplorePermalinkValue type:
    class ExplorePermalinkValue(TypedDict):
    chartId: Optional[int]
    # either datasetId or datasourceId is required
    # TODO: deprecated - datasetId is deprecated
    # and should be removed in next major release
    datasetId: Optional[int]
    datasourceId: Optional[int]
    datasourceType: str
    datasource: str
    state: ExplorePermalinkState

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

  1. downgrade your devenv to 7e67aecbf3f1, checkout master prior to f1fa1a733d247ec4793c17b69c4eb40c631c5080 and create an Explore permalink
  2. checkout latest master branch, run superset db upgrade and notice the error
  3. checkout this PR, rerun superset db upgrade and see the migration complete
  4. To validate that downgrading and re-upgrading works, run `superset db downgrade 7e67aecbf3f1 && superset db upgrade"

ADDITIONAL INFORMATION

  • Has associated issue:
  • Required feature flags:
  • Changes UI
  • Includes DB Migration (follow approval process in SIP-59)
    • Migration is atomic, supports rollback & is backwards-compatible
    • Confirm DB migration upgrade and downgrade tested
    • Runtime estimates and downtime expectations provided
  • Introduces new feature or API
  • Removes existing feature or API

@villebro villebro requested a review from a team as a code owner May 9, 2023 07:21
@villebro villebro added the 2.1.1 label May 9, 2023
@codecov
Copy link

codecov bot commented May 9, 2023

Codecov Report

Merging #23980 (02a3fd1) into master (d96b72d) will decrease coverage by 0.68%.
The diff coverage is n/a.

❗ Current head 02a3fd1 differs from pull request most recent head 8e9f9f5. Consider uploading reports for the commit 8e9f9f5 to get more accurate results

@@            Coverage Diff             @@
##           master   #23980      +/-   ##
==========================================
- Coverage   68.18%   67.51%   -0.68%     
==========================================
  Files        1941     1941              
  Lines       75287    75287              
  Branches     8177     8177              
==========================================
- Hits        51332    50827     -505     
- Misses      21860    22365     +505     
  Partials     2095     2095              
Flag Coverage Δ
hive 53.17% <ø> (ø)
postgres ?
presto 53.09% <ø> (ø)
python 81.32% <ø> (-1.39%) ⬇️
sqlite 77.50% <ø> (ø)
unit 52.98% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
superset/explore/permalink/commands/create.py 90.90% <ø> (ø)

... and 38 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Copy link
Member

@michael-s-molina michael-s-molina left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the fix @villebro. I tested the migration here at Airbnb and it executed successfully.

@villebro
Copy link
Member Author

villebro commented May 9, 2023

Thank you for the fix @villebro. I tested the migration here at Airbnb and it executed successfully.

Awesome, thanks for testing @michael-s-molina !

@villebro villebro merged commit 7a41170 into apache:master May 9, 2023
@villebro villebro deleted the villebro/fix-unpickling branch May 9, 2023 12:28
sadpandajoe pushed a commit to preset-io/superset that referenced this pull request May 18, 2023
@sadpandajoe
Copy link
Member

🏷️ preset:2023.19

@eschutho eschutho added the v2.1 label Jul 11, 2023
@mistercrunch mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 3.0.0 labels Mar 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels size/S 🚢 3.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants