Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

chore: Migrate warm up cache endpoint to api v1 #23853

Merged
merged 40 commits into from
Jun 20, 2023
Merged

chore: Migrate warm up cache endpoint to api v1 #23853

merged 40 commits into from
Jun 20, 2023

Conversation

jfrag1
Copy link
Member

@jfrag1 jfrag1 commented Apr 27, 2023 •
edited
Loading

SUMMARY

This PR deprecates the superset/warm_up_cache endpoint and creates the api/v1/charts/warm_up_cache and api/v1/dataset/warm_up_cache endpoints to replace it.

The new endpoint has the following key differences:

  • Follows command pattern
  • uses PUT instead of GET
  • accepts a json body validated by marshmallow instead of query params
  • added openapi spec

Also improves test coverage

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

  • Has associated issue:
  • Required feature flags:
  • Changes UI
  • Includes DB Migration (follow approval process in SIP-59)
    • Migration is atomic, supports rollback & is backwards-compatible
    • Confirm DB migration upgrade and downgrade tested
    • Runtime estimates and downtime expectations provided
  • Introduces new feature or API
  • Removes existing feature or API

@codecov
Copy link

codecov bot commented Apr 27, 2023 •
edited
Loading

Codecov Report

Merging #23853 (079ff1b) into master (3e76736) will increase coverage by 0.10%.
The diff coverage is 87.57%.

❗ Current head 079ff1b differs from pull request most recent head fcb89ad. Consider uploading reports for the commit fcb89ad to get more accurate results

@@            Coverage Diff             @@
##           master   #23853      +/-   ##
==========================================
+ Coverage   68.91%   69.01%   +0.10%     
==========================================
  Files        1899     1901       +2     
  Lines       73843    73969     +126     
  Branches     8119     8119              
==========================================
+ Hits        50892    51053     +161     
+ Misses      20840    20805      -35     
  Partials     2111     2111
Flag Coverage Δ
hive 53.90% <46.58%> (+<0.01%) ⬆️
mysql 79.40% <87.57%> (+0.17%) ⬆️
postgres 79.49% <87.57%> (+0.17%) ⬆️
presto 53.82% <46.58%> (+<0.01%) ⬆️
python 83.47% <87.57%> (+0.15%) ⬆️
sqlite 77.99% <87.57%> (+0.17%) ⬆️
unit 54.59% <42.23%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
superset/tasks/cache.py 59.34% <48.64%> (-1.81%) ⬇️
superset/charts/commands/warm_up_cache.py 97.50% <97.50%> (ø)
superset/charts/api.py 87.78% <100.00%> (+0.70%) ⬆️
superset/charts/commands/exceptions.py 94.00% <100.00%> (+0.38%) ⬆️
superset/charts/schemas.py 99.38% <100.00%> (+0.01%) ⬆️
superset/datasets/api.py 88.65% <100.00%> (+0.70%) ⬆️
superset/datasets/commands/exceptions.py 94.36% <100.00%> (+0.24%) ⬆️
superset/datasets/commands/warm_up_cache.py 100.00% <100.00%> (ø)
superset/datasets/schemas.py 97.64% <100.00%> (+0.16%) ⬆️

... and 4 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

dpgaspar
dpgaspar reviewed May 3, 2023
View reviewed changes
superset/cachekeys/api.py Outdated Show resolved Hide resolved
superset/cachekeys/api.py Outdated Show resolved Hide resolved
superset/cachekeys/api.py Outdated Show resolved Hide resolved
superset/cachekeys/api.py Outdated Show resolved Hide resolved
superset/cachekeys/commands/exceptions.py Outdated Show resolved Hide resolved
superset/cachekeys/api.py Outdated Show resolved Hide resolved
tests/integration_tests/cachekeys/api_tests.py Outdated Show resolved Hide resolved
@pull-request-size pull-request-size bot added size/XL and removed size/L labels May 5, 2023
dpgaspar
dpgaspar reviewed May 8, 2023
View reviewed changes
Copy link
Member

@dpgaspar dpgaspar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.
Can you add the following line on UPDATING.md

- [22809](https://github.com/apache/superset/pull/23853): Migrated endpoint `/superset/warm_up_cache` to `/api/v1/cachekey/warm_up_cache/`. Corresponding permissions are `can warm_up_cache on Superset` to `can warm_up_cache on cachekey`. Make sure you add/replace the necessary permissions on any custom roles you may have.

superset/cachekeys/commands/warm_up_cache.py Outdated Show resolved Hide resolved
@jfrag1
Copy link
Member Author

jfrag1 commented May 18, 2023

An alternative could be to place it on api/v1/chart and/or api/v1/dataset

I would model these endpoints under the resources they refer to. I feels really weird for me treating cache warm up as a first-level resource.

Moved to api/v1/chart

dpgaspar
dpgaspar reviewed May 26, 2023
View reviewed changes
tests/integration_tests/charts/api_tests.py Outdated
"table_name": "energy_usage",
"db_name": get_example_database().database_name,
},
)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems weird to be warming up cache for a dataset on /api/v1/chart/.... The idea would be to create a new endpoint on dataset's and perhaps use the same Command, or split it your call. the PUT payload would be restricted to it's resource

dpgaspar
dpgaspar approved these changes Jun 14, 2023
View reviewed changes
Copy link
Member

@dpgaspar dpgaspar left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, I'll defer to @john-bodley since he has more context here

superset/tasks/cache.py
req = request.Request(url, headers=headers)
baseurl = "{WEBDRIVER_BASEURL}".format(**app.config)
url = f"{baseurl}api/v1/chart/warm_up_cache"
logger.info("Fetching %s with payload %s", url, data)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: checked that data contains the get_payload that only contains chart_id and dashboard_id so it's probably safe to log, but I would just remove it from the logs, it can also generate a huge log entry

@dpgaspar dpgaspar merged commit 5af298e into apache:master Jun 20, 2023
@dpgaspar dpgaspar deleted the jack/migrate-warm-up-cache-to-api-v1 branch June 20, 2023 11:08
@mapledan mapledan mentioned this pull request Nov 16, 2023
Closed
9 tasks
@mistercrunch mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 3.0.0 labels Mar 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dpgaspar dpgaspar dpgaspar approved these changes

@john-bodley john-bodley john-bodley approved these changes

@villebro villebro Awaiting requested review from villebro

@michael-s-molina michael-s-molina Awaiting requested review from michael-s-molina

Assignees
No one assigned
Labels
🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels size/XL 🚢 3.0.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@jfrag1 @dpgaspar @michael-s-molina @john-bodley @mistercrunch