feat(tags): Export and Import Functionality for Superset Dashboards and Charts #30833

asher-lab · 2024-11-04T16:58:33Z

SUMMARY

This PR is related to Discussion: #30629 by Spens

Motivation

We have adopted the tagging functionality and it is working quite well for us for dashboards and and charts. Our use case and process requires us to build dashboards on a development server, export the dashboards, commit the YAMLs to source control, and finally import the ZIPs to our production server. We would like the tags we assign on the development server to be captured in the export so they are replicated to our production server.

Proposed Change

UI elements are not affected by this change. All changes are related to updated logic/processing of the existing export and import functionality.

Only tags of type "custom" are exported. Other tags that exist but are not exported are "type" and "owner".

Export Logic
When an object (Dashboard/Chart) is exported, a new parameter is added to the exported YAML file. The parameter is "tags" and the value is an array of tag names. Here is an example:

slice_name: Vaccine Candidates per Approach & Stage
description: null
tags:
  - CovidTag
  - Items that start with the letter V
  - Superset Example Data
certified_by: null

...
Because tags have additional information (for example, a description), a new file is needed to hold this information. The file is called "tags.yaml" and lives in the top level of the zip file. For example:

dashboard_export_20241017T025146.zip
  charts (folder)
  dashboards (folder)
  databases (folder)
  datasets (folder)
  metadata.yaml (file)
  tags.yaml (file)

The format of the tags.yaml file is an array of tags

tags:
- tag_name: CovidTag
  description: Everything related to Covid
- tag_name: Items that start with the letter V
  description: Just for fun, tag Vanilla, Vikings, and anything that starts with V
- tag_name: Superset Example Data
  description: Native example data included with Superset

Import Logic
When an object is imported, the code checks to see if any tags included in the import file.
If yes, the tag is created if it does not already exist then the tag is assigned to the imported object. If the tag already exists, it is just assigned with no changes to the tag.

Details

A tag can only be created if the user doing the import has permission to create tags. Otherwise the tagging is ignored.
A created tag picks up the current time as the "created_on" and "changed_on" properties for the tag
A created tag picks up the current user as the "created_by_fk" and "changed_by_fk"
A created tag picks up the description from the tags.yaml file described above
If a tag already exists, the description is not updated (for consistency with other export/import behavior - for example, if you import a dashboard and the chart included in the zip already exists, Superset does not update that existing chart)
Existing Superset behavior: When importing objects that already exist, the user is prompted whether it is ok to overwrite. If no, the import process aborts. If yes, the old object is replaced with the new object.

To be consistent with this behavior, when importing an object and specifying "overwrite", old tags that are no longer specified with the object are removed, new tags that didn't originally exist are created, old tags that still remain in the new set of tags are retained (descriptions are not updated for example)

It is important to note that in this proposal, imported tags are not added to existing tags
This is intentional because for the use case of trying to mirror the settings and properties from one system to another, you must have the ability to remove tags from the target system. Otherwise, for example, if you rename a tag, the new name would get added and the old name would stick around which is not the desired behavior.

Thank you to anyone taking the time to read and comment!

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

Has associated issue:
Required feature flags:
Changes UI
Includes DB Migration (follow approval process in SIP-59)
- Migration is atomic, supports rollback & is backwards-compatible
- Confirm DB migration upgrade and downgrade tested
- Runtime estimates and downtime expectations provided
Introduces new feature or API
Removes existing feature or API

…Charts

DR-4503: Added export Tags functionality for Superset Dashboards and …

…Charts

DR-4503: Added Import Tags functionality for Superset Dashboards and Charts

DR-4503: Fix Export Tag to ignore owner:

remove docker .env custom SUPERSET_SECRET_KEY

Superset remove unwanted files

docker/pythonpath_dev/superset_config.py

superset/commands/chart/export.py

michael-s-molina · 2024-11-04T18:33:03Z

Thank you for the PR @asher-lab. Could you add unit tests for the new feature with the feature flag on and off?

Add superset logic to handle disable tagging system

asher-lab · 2024-11-05T11:30:48Z

Good day, @michael-s-molina. We've added logic to handle cases when the Tagging System is disabled in Superset. When disabled, the tags.yaml file and any associated tags within charts.yaml and dashboard.yaml will not be exported, and the same applies to the import operation.

Could you provide guidance on creating a unit test for this new feature? I have limited experience with writing unit tests, so any help would be appreciated.

Thank you!

michael-s-molina · 2024-11-05T12:15:48Z

Could you provide guidance on creating a unit test for this new feature? I have limited experience with writing unit tests, so any help would be appreciated.

Hi @asher-lab. You can check this file for examples.

Add Unit Testing for Tag Export and Import in Superset

asher-lab · 2024-11-06T17:21:05Z

Hi @michael-s-molina . Added unit tests for the new feature with the feature flag TAGGING_SYSTEM on and off. Thank you.

codecov · 2024-11-06T17:41:51Z

Codecov Report

Attention: Patch coverage is 49.28571% with 71 lines in your changes missing coverage. Please review.

Project coverage is 65.35%. Comparing base (76d897e) to head (99fb8f1).
Report is 950 commits behind head on master.

Files with missing lines	Patch %	Lines
superset/commands/tag/export.py	35.84%	34 Missing ⚠️
superset/commands/chart/export.py	30.76%	9 Missing ⚠️
...perset/commands/dashboard/importers/v1/__init__.py	27.27%	8 Missing ⚠️
superset/commands/dashboard/export.py	30.00%	7 Missing ⚠️
superset/commands/importers/v1/utils.py	82.50%	7 Missing ⚠️
superset/commands/chart/importers/v1/__init__.py	44.44%	5 Missing ⚠️
superset/commands/importers/v1/__init__.py	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #30833      +/-   ##
==========================================
+ Coverage   60.48%   65.35%   +4.86%     
==========================================
  Files        1931      537    -1394     
  Lines       76236    39053   -37183     
  Branches     8568        0    -8568     
==========================================
- Hits        46114    25522   -20592     
+ Misses      28017    13531   -14486     
+ Partials     2105        0    -2105

Flag	Coverage Δ
hive	`48.79% <28.57%> (-0.37%)`	⬇️
javascript	`?`
presto	`53.25% <28.57%> (-0.55%)`	⬇️
python	`65.35% <49.28%> (+1.86%)`	⬆️
unit	`60.84% <49.28%> (+3.22%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

asher-lab · 2024-11-06T17:55:48Z

I seen a bunch of tests: 10 are failing and 23 are successful. Could you provide what tests that I need to look at?

Vitor-Avila · 2024-11-06T23:15:13Z

superset/charts/schemas.py

@@ -1550,7 +1550,7 @@ class ImportV1ChartSchema(Schema):
    dataset_uuid = fields.UUID(required=True)
    is_managed_externally = fields.Boolean(allow_none=True, dump_default=False)
    external_url = fields.String(allow_none=True)
-
+    tag = fields.List(fields.String(), allow_none=True)


Could we have it named as tags, since it's a list?

Vitor-Avila · 2024-11-06T23:19:47Z

superset/commands/chart/export.py

+                model.tags if hasattr(model, "tags") else []
+            )
+            # Filter out any tags that contain "type:" in their name
+            payload["tag"] = [tag.name for tag in tags if not any(x in tag.name for x in ["type:", "owner:"])]


Since users can create tags with those strings, could we filter for type, instead? something like:

payload["tags"] = [tag.name for tag in tags if tag.type == TagType.custom]

Vitor-Avila · 2024-11-07T00:50:49Z

superset/dashboards/schemas.py

@@ -453,7 +453,7 @@ class ImportV1DashboardSchema(Schema):
    certified_by = fields.String(allow_none=True)
    certification_details = fields.String(allow_none=True)
    published = fields.Boolean(allow_none=True)
-
+    tag = fields.List(fields.String(), allow_none=True)


Vitor-Avila · 2024-11-07T01:23:44Z

superset/commands/dashboard/export.py

+            tags = (
+                model.tags if hasattr(model, "tags") else []
+            )
+            payload["tag"] = [tag.name for tag in tags if not any(prefix in tag.name for prefix in ["type:", "owner:"])]


Vitor-Avila · 2024-11-07T01:25:24Z

Thank you so much @asher-lab! This is super cool and very useful 😍 🙌 I left a few comments, and I plan to give it another pass Tomorrow.

Fix integration test superset, change to "tags" and update tag filter using TagType apache#30869

asher-lab · 2024-11-07T14:32:21Z

Hi @Vitor-Avila , we have fixed the integration test issue, updated it to use "tags" instead of "tag" and update filter using TagType. Thank you!

asher-lab · 2024-11-07T15:15:39Z

Hi @Vitor-Avila and @michael-s-molina, if these are merged in, when is the expected date it be released?

rusackas · 2024-11-08T05:07:50Z

Running CI 🤞 This would definitely be part of 5.0, which needs a new estimated date. It will have missed the boat for 4.1.0, but could make a 4.2.0, though it's unclear if/when that would happen. It would not make a 4.1.1 release since that would only include fixes. Feel free to join us on slack in the release strategy or release announcement channels, if you want to get involved with the process.

asher-lab · 2024-11-12T14:38:57Z

Currently fixing the tests that are failing. Will be making a commit in order to fix those. Thank you. I realized that we need to work with pre-commit checks and integration tests...

* Added self.contents in database and datasets for import * Add self.content in query for import * Run precommit and fix some issues in superset * Add ExportTagsCommand * Fix ImportExamplesCommand arguments to make Optional * Remove should_export_tags parameter to solve pre-commit error * Fix ExportDashboard command calling ExportTagsCommandfrom ExportChartsCommand * Remove protected access for ExportTagsCommand _export * Fix some Pylint issues * Ensure Feature Flag for TAGGING_SYSTEM is turned off --------- Co-authored-by: Asher Manangan <[email protected]>

asher-lab · 2024-11-13T11:43:31Z

Good morning team, @rusackas @Vitor-Avila and @michael-s-molina. Could I get another run for this one please. I did run it on my own fork and all tests have passed. 🤞

Vitor-Avila · 2024-11-13T13:02:13Z

hey @asher-lab,
Sorry for the delay on my end. Approved the workflows and hoping to check it today

Asher Manangan and others added 16 commits October 25, 2024 19:49

DR-4503: Added export Tags functionality for Superset Dashboards and …

9f34dd8

…Charts

Merge pull request #1 from asher-lab/DR-4503

a067536

DR-4503: Added export Tags functionality for Superset Dashboards and …

Merge branch 'apache:master' into master

0978578

Merge branch 'apache:master' into master

8b3fb08

DR-4503: Added Import Tags functionality for Superset Dashboards and …

9c289bd

…Charts

DR-4503: Exclude owner tag in Superset export

aaf4c2a

Merge pull request #2 from asher-lab/DR-4503-import

2ba437d

DR-4503: Added Import Tags functionality for Superset Dashboards and Charts

DR-4503: Fix Export Tag to ignore owner:

3225c3a

Merge pull request #3 from asher-lab/exportTagLogic

fa2bbe2

DR-4503: Fix Export Tag to ignore owner:

Merge branch 'apache:master' into master

1a32ca8

Merge branch 'apache:master' into master

9c3f0c5

Merge branch 'apache:master' into master

c797866

Revert docker env secret string

2d2d2b0

Merge pull request #4 from asher-lab/master-1

d4e294f

remove docker .env custom SUPERSET_SECRET_KEY

Superset remove unwanted files

f96aa88

Merge pull request #5 from asher-lab/master-1

df3d310

Superset remove unwanted files

pull-request-size bot added the size/L label Nov 4, 2024

dosubot bot added dashboard:export Related to exporting dashboards dashboard:import Related to importing dashboards labels Nov 4, 2024

michael-s-molina requested review from betodealmeida, eschutho and hughhhh November 4, 2024 18:27

michael-s-molina reviewed Nov 4, 2024

View reviewed changes

docker/pythonpath_dev/superset_config.py Outdated Show resolved Hide resolved

superset/commands/chart/export.py Outdated Show resolved Hide resolved

Asher Manangan and others added 2 commits November 5, 2024 19:24

Add superset logic to handle disable tagging system

3ec09bc

Merge pull request #6 from asher-lab/DisableTaggingSystem

dbc1955

Add superset logic to handle disable tagging system

michael-s-molina requested a review from Vitor-Avila November 5, 2024 12:15

Add Unit Testing for Tag Export and Import in Superset

ab9ed0f

Merge pull request #7 from asher-lab/AddUnitTesting

8dea188

Add Unit Testing for Tag Export and Import in Superset

Vitor-Avila reviewed Nov 6, 2024

View reviewed changes

Vitor-Avila reviewed Nov 7, 2024

View reviewed changes

Asher Manangan and others added 3 commits November 7, 2024 19:33

Fix Superset Integration for test_export_chart_command_no_related

d5074c2

Change to tags and upfate filter by TagType

27b3bc5

Merge pull request #8 from asher-lab/FixIntegrationTestSuperset

99fb8f1

Fix integration test superset, change to "tags" and update tag filter using TagType apache#30869

Merge branch 'apache:master' into master

62bfcd2

asher-lab mentioned this pull request Nov 13, 2024

Fix test pipeline asher-lab/superset#12

Merged

pull-request-size bot added size/XL and removed size/L labels Nov 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(tags): Export and Import Functionality for Superset Dashboards and Charts #30833

feat(tags): Export and Import Functionality for Superset Dashboards and Charts #30833

asher-lab commented Nov 4, 2024

michael-s-molina commented Nov 4, 2024

asher-lab commented Nov 5, 2024

michael-s-molina commented Nov 5, 2024

asher-lab commented Nov 6, 2024

codecov bot commented Nov 6, 2024 •

edited

Loading

asher-lab commented Nov 6, 2024

Vitor-Avila Nov 6, 2024

Vitor-Avila Nov 6, 2024

Vitor-Avila Nov 7, 2024

Vitor-Avila Nov 7, 2024

Vitor-Avila commented Nov 7, 2024

asher-lab commented Nov 7, 2024 •

edited

Loading

asher-lab commented Nov 7, 2024

rusackas commented Nov 8, 2024 •

edited

Loading

asher-lab commented Nov 12, 2024 •

edited

Loading

asher-lab commented Nov 13, 2024

Vitor-Avila commented Nov 13, 2024

feat(tags): Export and Import Functionality for Superset Dashboards and Charts #30833

Are you sure you want to change the base?

feat(tags): Export and Import Functionality for Superset Dashboards and Charts #30833

Conversation

asher-lab commented Nov 4, 2024

SUMMARY

Motivation

Proposed Change

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

michael-s-molina commented Nov 4, 2024

asher-lab commented Nov 5, 2024

michael-s-molina commented Nov 5, 2024

asher-lab commented Nov 6, 2024

codecov bot commented Nov 6, 2024 • edited Loading

Codecov Report

asher-lab commented Nov 6, 2024

Vitor-Avila Nov 6, 2024

Choose a reason for hiding this comment

Vitor-Avila Nov 6, 2024

Choose a reason for hiding this comment

Vitor-Avila Nov 7, 2024

Choose a reason for hiding this comment

Vitor-Avila Nov 7, 2024

Choose a reason for hiding this comment

Vitor-Avila commented Nov 7, 2024

asher-lab commented Nov 7, 2024 • edited Loading

asher-lab commented Nov 7, 2024

rusackas commented Nov 8, 2024 • edited Loading

asher-lab commented Nov 12, 2024 • edited Loading

asher-lab commented Nov 13, 2024

Vitor-Avila commented Nov 13, 2024

codecov bot commented Nov 6, 2024 •

edited

Loading

asher-lab commented Nov 7, 2024 •

edited

Loading

rusackas commented Nov 8, 2024 •

edited

Loading

asher-lab commented Nov 12, 2024 •

edited

Loading