chore: switching out ConnectorRegistry references for DatasourceDAO #20149

hughhhh · 2022-05-20T19:35:33Z

SUMMARY

Switching out all references of ConnectorRegistry to DatasourceDAO. In the PR, we are focusing on just removing the references and moving towards allowing other Datasources to power charts.

One issue is the circular dependencies that are blocking us from fully removing the logic to allow the app to run. So i've moved that code out the ConnectorRegistry for now and will prioritize a bigger refactor to stop the import issues in superset moving forward.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

Has associated issue:
Required feature flags:
Changes UI
Includes DB Migration (follow approval process in SIP-59)
- Migration is atomic, supports rollback & is backwards-compatible
- Confirm DB migration upgrade and downgrade tested
- Runtime estimates and downtime expectations provided
Introduces new feature or API
Removes existing feature or API

…uperset into connreg-to-datadao_2

…reg-to-datadao_2

hughhhh · 2022-05-20T19:40:59Z

superset/views/core.py

@@ -282,7 +286,7 @@ def request_access(self) -> FlaskResponse:
        datasource_id = request.args.get("datasource_id")
        datasource_type = request.args.get("datasource_type")
        if datasource_id and datasource_type:
-            ds_class = ConnectorRegistry.sources.get(datasource_type)
+            ds_class = DatasourceDAO.sources.get(datasource_type)


use DatasourceType(datasource_type) here

DatasourceDAO.sources should be able to take either a string or the DatasourceType instance, so I think either works here.

of course you'll get an error either way if the datasource_type is not valid.

hughhhh · 2022-05-20T19:41:40Z

superset/views/core.py

@@ -2535,7 +2539,7 @@ def fetch_datasource_metadata(self) -> FlaskResponse:  # pylint: disable=no-self
        """

        datasource_id, datasource_type = request.args["datasourceKey"].split("__")
-        datasource = ConnectorRegistry.get_datasource(
+        datasource = DatasourceDAO.get_datasource(
            datasource_type,


use DatasourceType(datasource.type)

hughhhh · 2022-05-20T19:41:58Z

superset/views/datasource/views.py

@@ -74,7 +75,7 @@ def save(self) -> FlaskResponse:
        datasource_id = datasource_dict.get("id")
        datasource_type = datasource_dict.get("type")
        database_id = datasource_dict["database"].get("id")
-        orm_datasource = ConnectorRegistry.get_datasource(
+        orm_datasource = DatasourceDAO.get_datasource(
            datasource_type, datasource_id, db.session


use DatasourceType(datasource.type)

hughhhh · 2022-05-20T19:42:44Z

tests/integration_tests/query_context_tests.py

@@ -132,7 +132,7 @@ def test_query_cache_key_changes_when_datasource_is_updated(self):
        cache_key_original = query_context.query_cache_key(query_object)

        # make temporary change and revert it to refresh the changed_on property
-        datasource = ConnectorRegistry.get_datasource(
+        datasource = DatasourceDAO.get_datasource(
            datasource_type=payload["datasource"]["type"],


use DatasourceType(datasource.type)

hughhhh · 2022-05-20T19:42:55Z

tests/integration_tests/query_context_tests.py

@@ -156,7 +156,7 @@ def test_query_cache_key_changes_when_metric_is_updated(self):
        payload = get_query_context("birth_names")

        # make temporary change and revert it to refresh the changed_on property
-        datasource = ConnectorRegistry.get_datasource(
+        datasource = DatasourceDAO.get_datasource(


use DatasourceType(datasource.type)

hughhhh · 2022-05-20T19:53:07Z

superset/models/dashboard.py

@@ -407,16 +408,19 @@ def export_dashboards(  # pylint: disable=too-many-locals
                    id_ = target.get("datasetId")
                    if id_ is None:
                        continue
-                    datasource = ConnectorRegistry.get_datasource_by_id(session, id_)
+                    # todo(phillip): get datasource type for this method


remove this comment

hughhhh · 2022-05-20T19:53:42Z

superset/models/slice.py

+        # pylint: disable=import-outside-toplevel
+        from superset.datasource.dao import DatasourceDAO
+
+        return DatasourceDAO.sources[self.datasource_type]


use DatasourceType(datasource_type)

hughhhh · 2022-05-20T19:57:43Z

superset/cachekeys/api.py

@@ -83,13 +83,14 @@ def invalidate(self) -> Response:
            return self.response_400(message=str(error))
        datasource_uids = set(datasources.get("datasource_uids", []))
        for ds in datasources.get("datasources", []):
-            ds_obj = ConnectorRegistry.get_datasource_by_name(
+            ds_obj = DatasourceDAO.get_datasource_by_name(
                session=db.session,
                datasource_type=ds.get("datasource_type"),


update DatasourceType(datasource_type)

hughhhh · 2022-06-06T13:45:28Z

I left a comment about moving sqlatable specific lookups to the model or a sqlatable dao

Decided to move all the functions into the model

AAfghahi · 2022-06-06T14:16:01Z

superset/explore/form_data/commands/update.py

@@ -75,7 +76,7 @@ def run(self) -> Optional[str]:
                new_state: TemporaryExploreState = {
                    "owner": owner,
                    "datasource_id": datasource_id,
-                    "datasource_type": datasource_type,
+                    "datasource_type": DatasourceType(datasource_type),


why is this one different from the above one?

are you referencing this class definition?
https://github.com/apache/superset/pull/20149/files/b678a8ba981f7687c17bc68b4f529209bde2d961#diff-6029cd3077bd4123d18311935a0fef83a580ccf22f4a98b6085ed780fd7257e9R27

No I am referring to this: https://github.com/apache/superset/pull/20149/files/b678a8ba981f7687c17bc68b4f529209bde2d961#diff-6029cd3077bd4123d18311935a0fef83a580ccf22f4a98b6085ed780fd7257e9R27

THis one has DatasourceType(datasource_type) the version above has DatasourceType

eschutho · 2022-06-06T23:34:24Z

superset/datasource/dao.py

+    def get_datasource(
+        cls, session: Session, datasource_type: DatasourceType, datasource_id: int
+    ) -> Datasource:
+        if datasource_type not in cls.sources:


I think we're going to want to accept the string value as well here.
I don't think "table" in cls.sources is going to work for example.

I'm currently casting all the values going into the get_datasource function like this:

DatasourceDAO.get_datasource( db.session, DatasourceType(datasource["type"]), int(datasource["id"]) )

So moving forward i think we should push for the enums since it's easy to cast via just wrapping it in a DatasourceType() constructor. If we allow string we'd have to add logic to handle/report whenever the user sending us invalid values.

I'm okay with putting the Union[str, DatasourceType] but don't think it takes much work to just wrap the string with the DatasourceType constructor before passing it into the function

you could also just wrap source = cls.sources[datasource_type] in a try/catch block. I just think there's too much room for error here across the application with people who may not know that they can't pass in a string. This should work if datasource_type is either a string or the enum key.

eschutho · 2022-06-06T23:39:01Z

superset/security/manager.py

-            set
-        )
+        session = self.get_session
+        all_datasources = SqlaTable.get_all_sqlatables_datasources(session)


nit, but any reason why you wouldn't name this method get_all_datasources?

eschutho

most are small questions, but if datasource_type not in cls.sources: could cause problems if a string is passed in. I personally think we could allow both types, but we could prob go either way. It's just going to be difficult to enforce with typing.

eschutho · 2022-06-13T17:42:07Z

superset/examples/helpers.py


 BASE_URL = "https://github.com/apache-superset/examples-data/blob/master/"

 misc_dash_slices: Set[str] = set()  # slices assembled in a 'Misc Chart' dashboard


 def get_table_connector_registry() -> Any:
-    return ConnectorRegistry.sources["table"]
+    return DatasourceDAO.sources[DatasourceType.TABLE]


can we use the SqlaTable model directly since this is the only option here?

hughhhh · 2022-06-13T18:14:34Z

superset/datasource/dao.py

+        cls, session: Session, datasource_type: DatasourceType, datasource_id: int
+    ) -> Datasource:
+        if datasource_type not in cls.sources:
+            raise DatasourceTypeNotSupportedError()


notes: set this exception to return 422

hughhhh · 2022-06-13T18:15:33Z

superset/datasource/dao.py

+
+    @classmethod
+    def get_datasource(
+        cls, session: Session, datasource_type: DatasourceType, datasource_id: int


go with a Union[str, DatasourceType] and add test to verify works both and will throw 422

hughhhh · 2022-06-14T15:03:31Z

Moving here #20380

pkdotson and others added 14 commits May 18, 2022 15:11

rename and move dao file

0fbde4a

Update dao.py

148f400

add cachekey

c63d575

Merge branch 'connreg-to-datadao_2' of https://github.com/preset-io/s…

139e8f4

…uperset into connreg-to-datadao_2

Update __init__.py

bb79bc6

change reference in query context test

8119f7a

add utils ref

7804433

more ref changes

7ea4d1e

add helpers

e691320

add todo in dashboard.py

d0a694a

add cachekey

c92443a

circular import error in dar.py

606a285

push rest of refs

f0d308f

fix linting

0e44849

pull-request-size bot added the size/L label May 20, 2022

Merge branch 'master' of https://github.com/apache/superset into conn…

ea7ee06

…reg-to-datadao_2

hughhhh mentioned this pull request May 20, 2022

test(do not merge): rename and move dao file #20122

Closed

9 tasks

hughhhh changed the title ~~Connreg to datadao 2~~ chore: switching out ConnectorRegistry references for DatasourceDAO May 20, 2022

hughhhh commented May 20, 2022

View reviewed changes

fix more linting

6439dfa

hughhhh commented May 20, 2022

View reviewed changes

hughhhh force-pushed the connreg-to-datadao_2 branch 3 times, most recently from c9cd671 to 6a85df3 Compare May 20, 2022 21:04

hughhhh added 2 commits June 5, 2022 15:54

fixed merge conflicts

5dff58f

big reafctor

95c5eb2

hughhhh force-pushed the connreg-to-datadao_2 branch from 66cdb1c to 95c5eb2 Compare June 5, 2022 16:50

pull-request-size bot added size/XL and removed size/L labels Jun 5, 2022

hughhhh force-pushed the connreg-to-datadao_2 branch 5 times, most recently from 821da1c to ac2d8aa Compare June 5, 2022 19:04

take value

e08c4b8

hughhhh force-pushed the connreg-to-datadao_2 branch 2 times, most recently from efb667e to 051c26c Compare June 5, 2022 23:09

fix

ec17f51

hughhhh force-pushed the connreg-to-datadao_2 branch from 051c26c to ec17f51 Compare June 5, 2022 23:24

hughhhh requested a review from eschutho June 5, 2022 23:36

test to see if removing value works

19de44a

delete connectregistry

b678a8b

AAfghahi approved these changes Jun 6, 2022

View reviewed changes

AAfghahi reviewed Jun 6, 2022

View reviewed changes

eschutho reviewed Jun 6, 2022

View reviewed changes

eschutho requested changes Jun 6, 2022

View reviewed changes

address comments

eb0da19

eschutho reviewed Jun 13, 2022

View reviewed changes

hughhhh commented Jun 13, 2022

View reviewed changes

hughhhh closed this Jun 14, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: switching out ConnectorRegistry references for DatasourceDAO #20149

chore: switching out ConnectorRegistry references for DatasourceDAO #20149

hughhhh commented May 20, 2022 •

edited

Loading

hughhhh May 20, 2022

eschutho Jun 6, 2022

eschutho Jun 6, 2022

hughhhh May 20, 2022

hughhhh May 20, 2022

hughhhh May 20, 2022

hughhhh May 20, 2022

hughhhh May 20, 2022

hughhhh May 20, 2022

hughhhh May 20, 2022

hughhhh commented Jun 6, 2022

AAfghahi Jun 6, 2022

hughhhh Jun 6, 2022

AAfghahi Jun 7, 2022

eschutho Jun 6, 2022

hughhhh Jun 7, 2022 •

edited

Loading

eschutho Jun 7, 2022

eschutho Jun 6, 2022

eschutho left a comment

eschutho Jun 13, 2022

hughhhh Jun 13, 2022

hughhhh Jun 13, 2022

hughhhh commented Jun 14, 2022

chore: switching out ConnectorRegistry references for DatasourceDAO #20149

chore: switching out ConnectorRegistry references for DatasourceDAO #20149

Conversation

hughhhh commented May 20, 2022 • edited Loading

SUMMARY

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hughhhh commented Jun 6, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hughhhh Jun 7, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eschutho left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hughhhh commented Jun 14, 2022

hughhhh commented May 20, 2022 •

edited

Loading

hughhhh Jun 7, 2022 •

edited

Loading