chore: isolate examples database by default #25003

dpgaspar · 2023-08-16T13:57:29Z

SUMMARY

This PR places the examples database on it's own database, previously the examples database tables were being created on Superset's metadata database by default.
Setting the examples database on it's own database will stop giving users access to Superset's metadata database through SQLLab avoiding several possible attack vectors.

By default without using docker-compose, the examples database will be set on it's own SQLite db. Using docker-compose the postgres container will create a specific user for the examples db and the db itself.
Removed the default that set's the SQLALCHEMY_EXAMPLES_URI to be equal to SQLALCHEMY_DATABASE_URI when not set.

Before:

After:

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

Has associated issue:
Required feature flags:
Changes UI
Includes DB Migration (follow approval process in SIP-59)
- Migration is atomic, supports rollback & is backwards-compatible
- Confirm DB migration upgrade and downgrade tested
- Runtime estimates and downtime expectations provided
Introduces new feature or API
Removes existing feature or API

john-bodley · 2023-08-16T16:57:43Z

/testenv up

github-actions · 2023-08-16T16:58:07Z

@john-bodley Ephemeral environment creation is currently limited to committers.

betodealmeida

Looks good, i just left a question about #23399. I think it could be good to have the examples DB with DML and file uploads enabled, for a better initial experience with Superset.

betodealmeida · 2023-08-16T17:15:02Z

superset/utils/database.py

-        or current_app.config["SQLALCHEMY_DATABASE_URI"]
-    )
-    return get_or_create_db("examples", db_uri)
+    return get_or_create_db("examples", current_app.config["SQLALCHEMY_EXAMPLES_URI"])


Regarding #23399, should we have the examples DB have DML on and with file uploads enabled?

left a comment on #23399, no strong opinions about allowing DML/upload on the examples db, yet I do see this as a readonly db

(cherry picked from commit 269c992)

sfirke · 2023-08-18T13:59:28Z

Someone in Slack reported that they are trying to follow the docker-compose setup instructions and are getting the error message:

The environment variable EXAMPLES_USER was missing, abort...

I was surprised to see zero hits for EXAMPLES_USER in Superset GitHub issues, so went looking to see if there was a recent PR that related to this variable, and found this one. Do folks think this could have broken the default deployment in that way?

michael-s-molina · 2023-08-18T18:37:01Z

@dpgaspar @sfirke @betodealmeida The problem is because when running with docker-compose-non-dev, it's using the docker/.env file which does not have the examples variables. My question is, for non-dev environments should we have the examples database? If yes, then the fix is just adding these variables to docker/.env. If not, then we need to skip the lookup for them in pythonpath_dev/superset_config.py. What do you think?

sfirke · 2023-08-18T20:26:55Z

I don't feel strongly but think it would be nice to offer admins the option of examples or not. So add the necessary values to the .env file. Not sure if the examples should default to enabled or not for that deployment.

misiekofski · 2023-08-21T10:29:06Z

Just tried to go with docs and got the same issue when running : docker-compose -f docker-compose-non-dev.yml up

superset_worker_beat    |     EXAMPLES_USER = get_env_variable("EXAMPLES_USER")
superset_worker_beat    |   File "/app/docker/pythonpath_dev/superset_config.py", line 44, in get_env_variable
superset_worker_beat    |     raise OSError(error_msg)
superset_worker_beat    | OSError: The environment variable EXAMPLES_USER was missing, abort...

sfirke · 2023-08-21T13:47:51Z

Did this PR from @sebastianliebscher and merged 2 hours ago by @michael-s-molina address this problem?

michael-s-molina · 2023-08-22T16:15:39Z

@sfirke @misiekofski #25055 fixes the reported error.

eschutho · 2023-10-02T22:29:14Z

superset/config.py

@@ -1452,7 +1452,7 @@ def EMAIL_HEADER_MUTATOR(  # pylint: disable=invalid-name,unused-argument

 # URI to database storing the example data, points to
 # SQLALCHEMY_DATABASE_URI by default if set to `None`
-SQLALCHEMY_EXAMPLES_URI = None
+SQLALCHEMY_EXAMPLES_URI = "sqlite:///" + os.path.join(DATA_DIR, "examples.db")


@dpgaspar are we supporting sqlite as a valid examples db in production?

same as before since our default for the metadata db is also sqlite https://github.com/apache/superset/blob/master/superset/config.py#L187
The difference is before this PR both schemas were on the same sqlite db

We have an approved SIP to drop support for SQL Lite. I've added a ticket to the 4.0 board to ensure we execute this in the next major release.

We do discourage people to use sqlite, though, so I'm wondering if we should put something in UPDATING.md that encourage people to change this value to a different db type?

chore: isolate examples database by default

c7a3bc7

pull-request-size bot added the size/M label Aug 16, 2023

dpgaspar added 2 commits August 16, 2023 15:08

fix

4be7d7e

fix lint

3b1b6b3

dpgaspar requested review from villebro, john-bodley, betodealmeida and eschutho August 16, 2023 15:48

john-bodley added the review:checkpoint Last PR reviewed during the daily review standup label Aug 16, 2023

betodealmeida approved these changes Aug 16, 2023

View reviewed changes

dpgaspar merged commit 269c992 into apache:master Aug 17, 2023
35 checks passed

dpgaspar deleted the chore/isolate-examples-db branch August 17, 2023 10:50

michael-s-molina added the v3.0 Label added by the release manager to track PRs to be included in the 3.0 branch label Aug 17, 2023

john-bodley removed the review:checkpoint Last PR reviewed during the daily review standup label Aug 17, 2023

michael-s-molina pushed a commit that referenced this pull request Aug 17, 2023

chore: isolate examples database by default (#25003)

c2c5f23

(cherry picked from commit 269c992)

michael-s-molina mentioned this pull request Aug 22, 2023

fix: docker-compose non-dev #25055

Merged

9 tasks

eschutho added v2.1 2.1.2 labels Oct 2, 2023

eschutho reviewed Oct 2, 2023

View reviewed changes

sebastianliebscher mentioned this pull request Oct 11, 2023

[WIP] feat(docker): enable single curl of docker-compose.yml #25546

Draft

9 tasks

eschutho removed v2.1 2.1.2 labels Oct 20, 2023

y1ong mentioned this pull request Nov 28, 2023

Apache Superset 默认示例数据库权限提升漏洞 y1ong/blog-timeline#320

Open

john-bodley mentioned this pull request Jan 5, 2024

chore: Reenable SQLite tests which leverage foreign key constraints et al. #24605

Merged

9 tasks

tsndqst mentioned this pull request Feb 1, 2024

Superset 3.0 - Example charts throw SQLite Error - no such table: main.FCC 2018 Survey #25825

Open

mistercrunch added 🍒 3.0.0 🍒 3.0.1 🍒 3.0.2 🍒 3.0.3 🍒 3.0.4 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 3.1.0 labels Mar 8, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: isolate examples database by default #25003

chore: isolate examples database by default #25003

dpgaspar commented Aug 16, 2023 •

edited

Loading

john-bodley commented Aug 16, 2023

github-actions bot commented Aug 16, 2023

betodealmeida left a comment

betodealmeida Aug 16, 2023

dpgaspar Aug 17, 2023

sfirke commented Aug 18, 2023 •

edited

Loading

michael-s-molina commented Aug 18, 2023

sfirke commented Aug 18, 2023

misiekofski commented Aug 21, 2023

sfirke commented Aug 21, 2023

michael-s-molina commented Aug 22, 2023

eschutho Oct 2, 2023

dpgaspar Oct 3, 2023

michael-s-molina Oct 3, 2023

eschutho Oct 20, 2023

chore: isolate examples database by default #25003

chore: isolate examples database by default #25003

Conversation

dpgaspar commented Aug 16, 2023 • edited Loading

SUMMARY

Before:

After:

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

john-bodley commented Aug 16, 2023

github-actions bot commented Aug 16, 2023

betodealmeida left a comment

Choose a reason for hiding this comment

betodealmeida Aug 16, 2023

Choose a reason for hiding this comment

dpgaspar Aug 17, 2023

Choose a reason for hiding this comment

sfirke commented Aug 18, 2023 • edited Loading

michael-s-molina commented Aug 18, 2023

sfirke commented Aug 18, 2023

misiekofski commented Aug 21, 2023

sfirke commented Aug 21, 2023

michael-s-molina commented Aug 22, 2023

eschutho Oct 2, 2023

Choose a reason for hiding this comment

dpgaspar Oct 3, 2023

Choose a reason for hiding this comment

michael-s-molina Oct 3, 2023

Choose a reason for hiding this comment

eschutho Oct 20, 2023

Choose a reason for hiding this comment

dpgaspar commented Aug 16, 2023 •

edited

Loading

sfirke commented Aug 18, 2023 •

edited

Loading