Create a Saas Connector from a Template [#814] #1076

galvana · 2022-08-12T23:14:05Z

Purpose

Rather than making five or so separate API requests, it would be useful to be able to create all the resources for a SaasConnector in a single request.

Adds an endpoint to be able to create all the resources for a SaaS Connector in one place.

This also starts the creation of a registry for SaaS Connectors that other parts of the application can eventually use as a source of truth. This registry defines templates that we use to create the necessary resources.

Changes

Starts creating a registry for SaaSConnectors. This file describes the templates for SaasConnectors which consist of a config file, a dataset file, and an icon. Some icons are added here, but icon management will be explored further in a followup ticket.
Adds a single endpoint that creates a saas connector from a template.
- Creates a saas ConnectionConfig with a saas config attached without saving.
- The saas_config replaces every instance of instance_fides_key with the instance_key specified by the user.
- We then use this ConnectionConfig/SaasConfig to validate the supplied secrets.
- The ConnectionConfig is then saved.
- We create a separate DatasetConfig that links to the ConnectionConfig we just created. Every instance of instance_fides_key is also replaced in this json.
- The endpoint currently returns the finished dataset only, but we can modify the response to be whatever we need.

Out of scope

Updating connectors on server startup Update Saas Configs and Datasets on System Startup #1098
Icon management (Unticketed; I think @galvana's idea was to use the templates as the source of truth for icons too)
Other changes in the codebase to use the registry as the source of truth for saas connectors
- available connectors
- available datasets

Checklist

Ticket

Fixes #814

…mplates # Conflicts: # data/saas/dataset/mailchimp_dataset.yml # src/fidesops/main.py # src/fidesops/ops/api/v1/endpoints/saas_config_endpoints.py

…key already exists.

…te in the template endpoint. Don't save ConnectionConfig until secrets are validated.

…s and datasets with instance_fides_key. - Fix datadog yaml so it can be included in the saas connector registry. There was an error in how the saas config was formatted.

…to have brackets around the "instance_fides_key" to indicate these will be replaced. Update the fides_key definition to allow "<instance_fides_key>" with brackets specifically to pass validation.

…crets that should be supplied for a given connector. Use the saas config type instead of the fides key for the model title. Add test verifying that fides key /instance key validation works as expected.

…mplates

- Add new endpoint to postman collection - Add drafts doc. - Update old response body in docs for connection types.

pattisdr · 2022-08-16T22:53:50Z

data/saas/config/datadog_config.yml

@@ -1,59 +1,59 @@
 saas_config:
-  - fides_key: datadog_connector_example


There was an error in the datadog yamls which showed up when we validate the saas connector templates that are defined in the registry on startup.

pattisdr · 2022-08-16T22:55:26Z

src/fidesops/ops/schemas/shared_schemas.py

+        if value == "<instance_fides_key>":
+            return value


Added so we can specify the fides_keys in the saas connector templates as <instance_fides_key>, which will be replaced with the name the user actually wants when a resource is created from a template. Without this, the files fail validation since angle brackets can't be in a FidesKey.

@galvana wanted to follow existing conventions in the saas config files that have angle brackets around fields that are meant to be replaced.

can we add a small comment above line 14, like "this is needed to support SaaS connector templates"

pattisdr · 2022-08-16T22:56:47Z

src/fidesops/ops/api/v1/endpoints/saas_config_endpoints.py

+        connection_config: ConnectionConfig = (
+            create_connection_config_from_template_no_save(
+                db, connector_template, template_values
+            )
+        )
+    except KeyOrNameAlreadyExists as exc:
+        raise HTTPException(
+            status_code=HTTP_400_BAD_REQUEST,
+            detail=exc.args[0],
+        )
+
+    connection_config.secrets = validate_secrets(
+        template_values.secrets, connection_config
+    ).dict()
+    connection_config.save(db=db)  # Not persisted to db until secrets are validated


We create the ConnectionConfig with the ConnectionConfig.saas_config attached first without saving. Creating the ConnectionConfig lets us use existing logic in validate_secrets to dynamically figure out which secret types are expected for this particular types of saas config. Not saving this means we don't have to delete the connectionconfig if validation fails.

If this passes, we add the connection config secrets and then save the resource.

pattisdr · 2022-08-16T23:00:10Z

src/fidesops/ops/api/v1/endpoints/saas_config_endpoints.py

+    dataset_config: DatasetConfig = create_dataset_config_from_template(
+        db, connection_config, connector_template, template_values
+    )
+


The ConnectionConfig needs to be saved to the database before we create a DatasetConfig. I should be catching all of the known instances that would cause DatasetConfig creation to fail, but it's always possible something happens here. Should I add something that removes the created ConnectionConfig if DatasetConfig creation fails?

Probably.

OK I've added this functionality -

pattisdr · 2022-08-16T23:01:03Z

src/fidesops/ops/schemas/connection_configuration/connection_secrets_saas.py

        model: Type[SaaSSchema] = create_model(
-            f"{self.saas_config.fides_key}_schema",
+            f"{self.saas_config.type}_schema",


This was added because we now have the saas_config templates fides_keys as <instance_fides_key> which would cause validation to fail when we're trying to build schemas.

pattisdr · 2022-08-16T23:03:58Z

src/fidesops/ops/api/v1/endpoints/saas_config_endpoints.py

+        db, connection_config, connector_template, template_values
+    )
+
+    return dataset_config.dataset


I'm currently just returning the finished dataset as confirmation the process is complete, but is there something more useful here? What is the ideal response body? Main thing, we need to make sure we don't return ConnectionConfig secrets here.

Unsure, I think it depends on how front-end wishes to use / display info? This should be fine for now though.

pattisdr · 2022-08-17T11:40:37Z

forgot to run unsafe checks, I assume these will fail, I'll need to update the instance_fides_key as part of the tests

…in the saas fixtures.

…onConfig.

…es_key.

eastandwestwind

Overall looking good, with small comments, doing some manual testing now.

eastandwestwind · 2022-08-17T15:22:16Z

saas_connector_registry.toml

@@ -0,0 +1,64 @@
+[adobe_campaign]


I wonder if this file might be better located in our data/saas dir? what do you think?

eastandwestwind · 2022-08-17T15:25:56Z

src/fidesops/ops/api/v1/endpoints/saas_config_endpoints.py

+    if not connector_template:
+        raise HTTPException(
+            status_code=HTTP_404_NOT_FOUND,
+            detail=f"SaaS connector type '{saas_connector_type}' is not registered.",


I think I clearer message would be something like "SaaS connector type '{saas_connector_type}' is not yet available in Fidesops. For a list of available SaaS connectors, refer to xyz endpoint"

eastandwestwind · 2022-08-17T15:29:08Z

src/fidesops/ops/api/v1/endpoints/saas_config_endpoints.py

+    ).dict()
+    connection_config.save(db=db)  # Not persisted to db until secrets are validated
+
+    try:


can we throw a friendly or at least unique message if not all required fields are provided?

I can't think of a specific error to catch if it makes it to this point.

I added this as a final catch-all in case the database was down or we had some other unexpected failure, so I can remove the connection_config we just created.

If it makes it here, the connection_config exists, the instance_key is the correct format and has been verified to not exist already, and the dataset has been verified to be of the proper form.

ohh I see, I'm thinking more about what if secrets aren't valid, or the needed secrets aren't provided? I'll test this

I see, the secrets are validated just above with connection_config.secrets = validate_secrets( template_values.secrets, connection_config ).dict().

I'm reusing the same code we wrote for the connection config secrets endpoint! Note this endpoint doesn't attempt to run a test of the secrets, it just saves them.

my unit tests on this endpoint show a lot of the error cases that are handled!

eastandwestwind · 2022-08-17T16:04:41Z

src/fidesops/ops/api/v1/endpoints/saas_config_endpoints.py

+        db, connection_config, connector_template, template_values
+    )
+
+    return dataset_config.dataset


Unsure, I think it depends on how front-end wishes to use / display info? This should be fine for now though.

eastandwestwind · 2022-08-17T16:05:53Z

src/fidesops/ops/schemas/shared_schemas.py

+        if value == "<instance_fides_key>":
+            return value


can we add a small comment above line 14, like "this is needed to support SaaS connector templates"

…ser-* data categories after the fideslang update, so they would show up if the user picked a "user" data category.

pattisdr · 2022-08-17T16:10:53Z

Remaining failing hubspot test is unrelated to this work. It was noticed in the fideslang work https://github.com/ethyca/fidesops/runs/7792050618?check_suite_focus=true, trying a fix now.

eastandwestwind · 2022-08-17T16:56:23Z

Also heads up that #1059 might be merged first, which includes a postman collection change.

eastandwestwind · 2022-08-17T17:06:50Z

@pattisdr Manually tested and it looks good. I was able to confirm the dataset returned for the new endpoint.

Also, I think the response error is specific enough when one of the required secrets fields isn't present. e.g.

{
    "detail": [
        {
            "loc": [
                "api_key"
            ],
            "msg": "field required",
            "type": "value_error.missing"
        },
        {
            "loc": [
                "__root__"
            ],
            "msg": "mailchimp_schema must be supplied all of: [domain, username, api_key].",
            "type": "value_error"
        }
    ]
}

pattisdr · 2022-08-17T17:07:38Z

ok great, getting the branch up to date, thanks for the review @eastandwestwind

…mplates # Conflicts: # CHANGELOG.md # docs/fidesops/docs/postman/Fidesops.postman_collection.json

eastandwestwind · 2022-08-17T17:44:47Z

Hubspot failing test is a pre-existing issue, and has been discussed here- #890 (comment)

Going ahead and merging this ticket

pattisdr · 2022-08-17T17:46:13Z

It also seems like some of the failure is related to fideslang, in that some of the data categories changed on the hubspot dataset, which causes more fields to be returned under various policies.

* Starting point for SaaS connector templates * Fix imports from restructuring. * Get happy path working for instantiate connector from template endpoint. * Remove updating connector instances for now - out of scope. * Test nonexistent templates, secrets validation, instance key / fides key already exists. * Create DatasetConfigs and ConnectionConfigs instead of create_or_update in the template endpoint. Don't save ConnectionConfig until secrets are validated. * Add the other saas connectors to the registry and update their configs and datasets with instance_fides_key. - Fix datadog yaml so it can be included in the saas connector registry. There was an error in how the saas config was formatted. * Update the fides_keys in the existing saas configs and dataset yamls to have brackets around the "instance_fides_key" to indicate these will be replaced. Update the fides_key definition to allow "<instance_fides_key>" with brackets specifically to pass validation. * Fix a side effect on a separate endpoint that returns the types of secrets that should be supplied for a given connector. Use the saas config type instead of the fides key for the model title. Add test verifying that fides key /instance key validation works as expected. * - Update CHANGELOG - Add new endpoint to postman collection - Add drafts doc. - Update old response body in docs for connection types. * Replace the <instance_fides_key> with a properly formatted fides_key in the saas fixtures. * If DatasetConfig creation fails, delete the recently created ConnectionConfig. * Address some of the saas integration tests where I've changed the fides_key. * Fix typos. * Fix typo. * Fix unrelated bug where hubspot dataset has new datacategories with user-* data categories after the fideslang update, so they would show up if the user picked a "user" data category. * Respond to CR. Co-authored-by: Dawn Pattison <[email protected]>

Starting point for SaaS connector templates

51c33cb

galvana linked an issue Aug 12, 2022 that may be closed by this pull request

SaaS Connector Templates #814

Closed

pattisdr added 11 commits August 15, 2022 08:54

Merge remote-tracking branch 'ethyca/main' into 814-saas-connector-te…

407b702

…mplates # Conflicts: # data/saas/dataset/mailchimp_dataset.yml # src/fidesops/main.py # src/fidesops/ops/api/v1/endpoints/saas_config_endpoints.py

Fix imports from restructuring.

36c98a0

Get happy path working for instantiate connector from template endpoint.

e69c94a

Remove updating connector instances for now - out of scope.

5f37820

Test nonexistent templates, secrets validation, instance key / fides …

2f10957

…key already exists.

Create DatasetConfigs and ConnectionConfigs instead of create_or_upda…

ea306d4

…te in the template endpoint. Don't save ConnectionConfig until secrets are validated.

Add the other saas connectors to the registry and update their config…

c50f59f

…s and datasets with instance_fides_key. - Fix datadog yaml so it can be included in the saas connector registry. There was an error in how the saas config was formatted.

Update the fides_keys in the existing saas configs and dataset yamls …

7d59388

…to have brackets around the "instance_fides_key" to indicate these will be replaced. Update the fides_key definition to allow "<instance_fides_key>" with brackets specifically to pass validation.

Fix a side effect on a separate endpoint that returns the types of se…

70d6404

…crets that should be supplied for a given connector. Use the saas config type instead of the fides key for the model title. Add test verifying that fides key /instance key validation works as expected.

Merge remote-tracking branch 'ethyca/main' into 814-saas-connector-te…

f30c701

…mplates

- Update CHANGELOG

6239ccc

- Add new endpoint to postman collection - Add drafts doc. - Update old response body in docs for connection types.

pattisdr changed the title ~~Starting point for SaaS connector templates~~ Create a Saas Connector from a Template [#814] Aug 16, 2022

pattisdr reviewed Aug 16, 2022

View reviewed changes

pattisdr marked this pull request as ready for review August 16, 2022 23:04

pattisdr added the run unsafe ci checks Triggers running of unsafe CI checks label Aug 17, 2022

Replace the <instance_fides_key> with a properly formatted fides_key …

8acd9a0

…in the saas fixtures.

pattisdr added run unsafe ci checks Triggers running of unsafe CI checks and removed run unsafe ci checks Triggers running of unsafe CI checks labels Aug 17, 2022

pattisdr added 2 commits August 17, 2022 09:41

If DatasetConfig creation fails, delete the recently created Connecti…

96c4990

…onConfig.

Address some of the saas integration tests where I've changed the fid…

d8f7d21

…es_key.

pattisdr added run unsafe ci checks Triggers running of unsafe CI checks and removed run unsafe ci checks Triggers running of unsafe CI checks labels Aug 17, 2022

seanpreston assigned eastandwestwind Aug 17, 2022

Fix typos.

23cde59

pattisdr added run unsafe ci checks Triggers running of unsafe CI checks and removed run unsafe ci checks Triggers running of unsafe CI checks labels Aug 17, 2022

Fix typo.

33c85b2

pattisdr added run unsafe ci checks Triggers running of unsafe CI checks and removed run unsafe ci checks Triggers running of unsafe CI checks labels Aug 17, 2022

eastandwestwind reviewed Aug 17, 2022

View reviewed changes

Fix unrelated bug where hubspot dataset has new datacategories with u…

a4b2dd7

…ser-* data categories after the fideslang update, so they would show up if the user picked a "user" data category.

pattisdr added run unsafe ci checks Triggers running of unsafe CI checks and removed run unsafe ci checks Triggers running of unsafe CI checks labels Aug 17, 2022

Respond to CR.

cec0d02

Merge remote-tracking branch 'ethyca/main' into 814-saas-connector-te…

e604ac2

…mplates # Conflicts: # CHANGELOG.md # docs/fidesops/docs/postman/Fidesops.postman_collection.json

pattisdr added run unsafe ci checks Triggers running of unsafe CI checks and removed run unsafe ci checks Triggers running of unsafe CI checks labels Aug 17, 2022

eastandwestwind approved these changes Aug 17, 2022

View reviewed changes

eastandwestwind merged commit 80c8266 into main Aug 17, 2022

eastandwestwind deleted the 814-saas-connector-templates branch August 17, 2022 17:44

pattisdr mentioned this pull request Aug 17, 2022

Update Response Body for Saas Connection Instantiate [#814] #1105

Merged

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create a Saas Connector from a Template [#814] #1076

Create a Saas Connector from a Template [#814] #1076

galvana commented Aug 12, 2022 •

edited by pattisdr

Loading

pattisdr Aug 16, 2022

pattisdr Aug 16, 2022

eastandwestwind Aug 17, 2022

pattisdr Aug 17, 2022

pattisdr Aug 16, 2022 •

edited

Loading

pattisdr Aug 16, 2022

pattisdr Aug 17, 2022

pattisdr Aug 16, 2022

pattisdr Aug 16, 2022 •

edited

Loading

eastandwestwind Aug 17, 2022

pattisdr commented Aug 17, 2022

eastandwestwind left a comment

eastandwestwind Aug 17, 2022

eastandwestwind Aug 17, 2022

eastandwestwind Aug 17, 2022

pattisdr Aug 17, 2022

eastandwestwind Aug 17, 2022

pattisdr Aug 17, 2022 •

edited

Loading

pattisdr Aug 17, 2022 •

edited

Loading

eastandwestwind Aug 17, 2022

eastandwestwind Aug 17, 2022

pattisdr commented Aug 17, 2022

eastandwestwind commented Aug 17, 2022

eastandwestwind commented Aug 17, 2022

pattisdr commented Aug 17, 2022

eastandwestwind commented Aug 17, 2022

pattisdr commented Aug 17, 2022

		@@ -1,59 +1,59 @@
		saas_config:
		- fides_key: datadog_connector_example

Create a Saas Connector from a Template [#814] #1076

Create a Saas Connector from a Template [#814] #1076

Conversation

galvana commented Aug 12, 2022 • edited by pattisdr Loading

Purpose

Changes

Out of scope

Checklist

Ticket

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pattisdr Aug 16, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pattisdr Aug 16, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pattisdr commented Aug 17, 2022

eastandwestwind left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pattisdr Aug 17, 2022 • edited Loading

Choose a reason for hiding this comment

pattisdr Aug 17, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pattisdr commented Aug 17, 2022

eastandwestwind commented Aug 17, 2022

eastandwestwind commented Aug 17, 2022

pattisdr commented Aug 17, 2022

eastandwestwind commented Aug 17, 2022

pattisdr commented Aug 17, 2022

galvana commented Aug 12, 2022 •

edited by pattisdr

Loading

pattisdr Aug 16, 2022 •

edited

Loading

pattisdr Aug 16, 2022 •

edited

Loading

pattisdr Aug 17, 2022 •

edited

Loading

pattisdr Aug 17, 2022 •

edited

Loading