Skip to content
This repository has been archived by the owner on Nov 30, 2022. It is now read-only.

Create a Saas Connector from a Template [#814] #1076

Merged
merged 20 commits into from
Aug 17, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
51c33cb
Starting point for SaaS connector templates
galvana Aug 7, 2022
407b702
Merge remote-tracking branch 'ethyca/main' into 814-saas-connector-te…
pattisdr Aug 15, 2022
36c98a0
Fix imports from restructuring.
pattisdr Aug 15, 2022
e69c94a
Get happy path working for instantiate connector from template endpoint.
pattisdr Aug 15, 2022
5f37820
Remove updating connector instances for now - out of scope.
pattisdr Aug 16, 2022
2f10957
Test nonexistent templates, secrets validation, instance key / fides …
pattisdr Aug 16, 2022
ea306d4
Create DatasetConfigs and ConnectionConfigs instead of create_or_upda…
pattisdr Aug 16, 2022
c50f59f
Add the other saas connectors to the registry and update their config…
pattisdr Aug 16, 2022
7d59388
Update the fides_keys in the existing saas configs and dataset yamls …
pattisdr Aug 16, 2022
70d6404
Fix a side effect on a separate endpoint that returns the types of se…
pattisdr Aug 16, 2022
f30c701
Merge remote-tracking branch 'ethyca/main' into 814-saas-connector-te…
pattisdr Aug 16, 2022
6239ccc
- Update CHANGELOG
pattisdr Aug 16, 2022
8acd9a0
Replace the <instance_fides_key> with a properly formatted fides_key …
pattisdr Aug 17, 2022
96c4990
If DatasetConfig creation fails, delete the recently created Connecti…
pattisdr Aug 17, 2022
d8f7d21
Address some of the saas integration tests where I've changed the fid…
pattisdr Aug 17, 2022
23cde59
Fix typos.
pattisdr Aug 17, 2022
33c85b2
Fix typo.
pattisdr Aug 17, 2022
a4b2dd7
Fix unrelated bug where hubspot dataset has new datacategories with u…
pattisdr Aug 17, 2022
cec0d02
Respond to CR.
pattisdr Aug 17, 2022
e604ac2
Merge remote-tracking branch 'ethyca/main' into 814-saas-connector-te…
pattisdr Aug 17, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ The types of changes are:
* Access support for Datadog Logs [#1060](https://github.com/ethyca/fidesops/pull/1060)
* Access and erasure support for Logi ID [#1074](https://github.com/ethyca/fidesops/pull/1074)
* Adds infra for email config and dispatch [#1059](https://github.com/ethyca/fidesops/pull/1059)
* Add an endpoint that allows you to create a Saas connector and all supporting resources with a single request [#1076](https://github.com/ethyca/fidesops/pull/1076)

### Developer Experience

Expand Down
2 changes: 1 addition & 1 deletion data/saas/config/adobe_campaign_config.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
saas_config:
fides_key: adobe_campaign_connector_example
fides_key: <instance_fides_key>
name: Adobe Campaign SaaS Config
type: adobe_campaign
description: A schema representing the Adobe Campaign connector for Fidesops
Expand Down
6 changes: 3 additions & 3 deletions data/saas/config/auth0_config.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
saas_config:
fides_key: auth0_connector_example
fides_key: <instance_fides_key>
name: Auth0 SaaS Config
type: auth0
description: A sample schema representing the Auth0 connector for Fidesops
Expand Down Expand Up @@ -46,7 +46,7 @@ saas_config:
param_values:
- name: user_id
references:
- dataset: auth0_connector_example
- dataset: <instance_fides_key>
field: users.user_id
direction: from
- name: user_logs
Expand All @@ -57,6 +57,6 @@ saas_config:
param_values:
- name: user_id
references:
- dataset: auth0_connector_example
- dataset: <instance_fides_key>
field: users.user_id
direction: from
108 changes: 54 additions & 54 deletions data/saas/config/datadog_config.yml
Original file line number Diff line number Diff line change
@@ -1,59 +1,59 @@
saas_config:
- fides_key: datadog_connector_example
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There was an error in the datadog yamls which showed up when we validate the saas connector templates that are defined in the registry on startup.

name: Datadog SaaS Config
type: datadog
description: A sample schema representing the Datadog connector for Fidesops
version: 0.0.1
fides_key: <instance_fides_key>
name: Datadog SaaS Config
type: datadog
description: A sample schema representing the Datadog connector for Fidesops
version: 0.0.1

connector_params:
- name: domain
- name: api_key
- name: app_key
- name: page_size
connector_params:
- name: domain
- name: api_key
- name: app_key
- name: page_size

client_config:
protocol: https
host: <domain>
client_config:
protocol: https
host: <domain>

test_request:
method: GET
path: /api/v2/logs/events
headers:
- name: DD-APPLICATION-KEY
value: <app_key>
- name: DD-API-KEY
value: <api_key>
test_request:
method: GET
path: /api/v2/logs/events
headers:
- name: DD-APPLICATION-KEY
value: <app_key>
- name: DD-API-KEY
value: <api_key>

endpoints:
- name: events
requests:
read:
method: GET
path: /api/v2/logs/events
headers:
- name: DD-APPLICATION-KEY
value: <app_key>
- name: DD-API-KEY
value: <api_key>
query_params:
- name: filter[query]
value: <email>
- name: filter[from]
value: 0
- name: filter[to]
value: now
- name: page[limit]
value: <page_size>
param_values:
- name: app_key
connector_param: app_key
- name: api_key
connector_param: api_key
- name: email
identity: email
data_path: data
pagination:
strategy: link
configuration:
source: body
path: links.next
endpoints:
- name: events
requests:
read:
method: GET
path: /api/v2/logs/events
headers:
- name: DD-APPLICATION-KEY
value: <app_key>
- name: DD-API-KEY
value: <api_key>
query_params:
- name: filter[query]
value: <email>
- name: filter[from]
value: 0
- name: filter[to]
value: now
- name: page[limit]
value: <page_size>
param_values:
- name: app_key
connector_param: app_key
- name: api_key
connector_param: api_key
- name: email
identity: email
data_path: data
pagination:
strategy: link
configuration:
source: body
path: links.next
6 changes: 3 additions & 3 deletions data/saas/config/hubspot_config.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
saas_config:
fides_key: hubspot_connector_example
fides_key: <instance_fides_key>
name: Hubspot SaaS Config
type: hubspot
description: A sample schema representing the Hubspot connector for Fidesops
Expand Down Expand Up @@ -64,7 +64,7 @@ saas_config:
param_values:
- name: contactId
references:
- dataset: hubspot_connector_example
- dataset: <instance_fides_key>
field: contacts.id
direction: from
- name: owners
Expand Down Expand Up @@ -113,7 +113,7 @@ saas_config:
identity: email
- name: subscriptionId
references:
- dataset: hubspot_connector_example
- dataset: <instance_fides_key>
field: subscription_preferences.id
direction: from
postprocessors:
Expand Down
4 changes: 2 additions & 2 deletions data/saas/config/logi_id_config.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
saas_config:
fides_key: logi_id_connector_example
fides_key: <instance_fides_key>
name: Logi ID SaaS Config
type: logi_id
description: A sample schema representing the Logi ID connector for Fidesops
Expand Down Expand Up @@ -44,7 +44,7 @@ saas_config:
param_values:
- name: user_id
references:
- dataset: logi_id_connector_example
- dataset: <instance_fides_key>
field: users.id
direction: from
- name: user_claims
Expand Down
8 changes: 4 additions & 4 deletions data/saas/config/mailchimp_config.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
saas_config:
fides_key: mailchimp_connector_example
fides_key: <instance_fides_key>
name: Mailchimp SaaS Config
type: mailchimp
description: A sample schema representing the Mailchimp connector for Fidesops
Expand Down Expand Up @@ -32,7 +32,7 @@ saas_config:
param_values:
- name: conversation_id
references:
- dataset: mailchimp_connector_example
- dataset: <instance_fides_key>
field: conversations.id
direction: from
data_path: conversation_messages
Expand Down Expand Up @@ -80,12 +80,12 @@ saas_config:
param_values:
- name: list_id
references:
- dataset: mailchimp_connector_example
- dataset: <instance_fides_key>
field: member.list_id
direction: from
- name: subscriber_hash
references:
- dataset: mailchimp_connector_example
- dataset: <instance_fides_key>
field: member.id
direction: from
body: |
Expand Down
2 changes: 1 addition & 1 deletion data/saas/config/outreach_config.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
saas_config:
fides_key: outreach_connector_example
fides_key: <instance_fides_key>
name: Outreach Example Config
type: outreach
description: A sample schema representing the Outreach connector for Fidesops
Expand Down
24 changes: 12 additions & 12 deletions data/saas/config/salesforce_config.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
saas_config:
fides_key: salesforce_connector_example
fides_key: <instance_fides_key>
name: Salesforce SaaS Config
type: salesforce
description: A sample schema representing the Salesforce connector for Fidesops
Expand Down Expand Up @@ -87,7 +87,7 @@ saas_config:
param_values:
- name: contact_id
references:
- dataset: salesforce_connector_example
- dataset: <instance_fides_key>
field: contact_list.Id
direction: from
update:
Expand All @@ -100,7 +100,7 @@ saas_config:
param_values:
- name: contact_id
references:
- dataset: salesforce_connector_example
- dataset: <instance_fides_key>
field: contacts.Id
direction: from
- name: case_list
Expand All @@ -114,7 +114,7 @@ saas_config:
param_values:
- name: contact_id
references:
- dataset: salesforce_connector_example
- dataset: <instance_fides_key>
field: contact_list.Id
direction: from
data_path: records
Expand All @@ -126,7 +126,7 @@ saas_config:
param_values:
- name: case_id
references:
- dataset: salesforce_connector_example
- dataset: <instance_fides_key>
field: case_list.Id
direction: from
update:
Expand All @@ -139,7 +139,7 @@ saas_config:
param_values:
- name: case_id
references:
- dataset: salesforce_connector_example
- dataset: <instance_fides_key>
field: cases.Id
direction: from
- name: lead_list
Expand All @@ -162,7 +162,7 @@ saas_config:
param_values:
- name: lead_id
references:
- dataset: salesforce_connector_example
- dataset: <instance_fides_key>
field: lead_list.Id
direction: from
update:
Expand All @@ -175,7 +175,7 @@ saas_config:
param_values:
- name: lead_id
references:
- dataset: salesforce_connector_example
- dataset: <instance_fides_key>
field: leads.Id
direction: from
- name: accounts
Expand All @@ -186,7 +186,7 @@ saas_config:
param_values:
- name: account_id
references:
- dataset: salesforce_connector_example
- dataset: <instance_fides_key>
field: contacts.AccountId
update:
method: PATCH
Expand All @@ -198,7 +198,7 @@ saas_config:
param_values:
- name: account_id
references:
- dataset: salesforce_connector_example
- dataset: <instance_fides_key>
field: accounts.Id
direction: from
- name: campaign_member_list
Expand All @@ -221,7 +221,7 @@ saas_config:
param_values:
- name: campaign_member_id
references:
- dataset: salesforce_connector_example
- dataset: <instance_fides_key>
field: campaign_member_list.Id
direction: from
update:
Expand All @@ -234,6 +234,6 @@ saas_config:
param_values:
- name: campaign_member_id
references:
- dataset: salesforce_connector_example
- dataset: <instance_fides_key>
field: campaign_members.Id
direction: from
8 changes: 4 additions & 4 deletions data/saas/config/segment_config.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
saas_config:
fides_key: segment_connector_example
fides_key: <instance_fides_key>
name: Segment SaaS Config
type: segment
description: A sample schema representing the Segment connector for Fidesops
Expand Down Expand Up @@ -55,7 +55,7 @@ saas_config:
connector_param: namespace_id
- name: segment_id
references:
- dataset: segment_connector_example
- dataset: <instance_fides_key>
field: segment_user.segment_id
direction: from
data_path: data
Expand Down Expand Up @@ -84,7 +84,7 @@ saas_config:
connector_param: namespace_id
- name: segment_id
references:
- dataset: segment_connector_example
- dataset: <instance_fides_key>
field: segment_user.segment_id
direction: from
data_path: traits
Expand All @@ -110,7 +110,7 @@ saas_config:
connector_param: namespace_id
- name: segment_id
references:
- dataset: segment_connector_example
- dataset: <instance_fides_key>
field: segment_user.segment_id
direction: from
data_path: data
Expand Down
4 changes: 2 additions & 2 deletions data/saas/config/sendgrid_config.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
saas_config:
fides_key: sendgrid_connector_example
fides_key: <instance_fides_key>
name: Sendgrid SaaS Config
type: sendgrid
description: A sample schema representing the Sendgrid connector for Fidesops
Expand Down Expand Up @@ -41,6 +41,6 @@ saas_config:
param_values:
- name: contact_id
references:
- dataset: sendgrid_connector_example
- dataset: <instance_fides_key>
field: contacts.id
direction: from
Loading