Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

1746 fides connector #1861

Merged
merged 16 commits into from
Nov 29, 2022
Merged

1746 fides connector #1861

merged 16 commits into from
Nov 29, 2022

Conversation

adamsachs
Copy link
Contributor

@adamsachs adamsachs commented Nov 28, 2022

Closes #1746
Closes #1747

Code Changes

  • New fides connector/connection type that forwards a privacy request to "child" Fides instances whose details are provided via connector config secrets
    • the new authenticated privacy request endpoint is invoked on the child, meaning that the fides connector config needs to have valid credentials (username and password) for the child server
    • DB migration for the new fides connection type is also included
  • Also provides a helper function to filter out datasets that belong to a Fides connector connection config. We use this helper in the request runner to identify these datasets, because they will require some further "special-cased" post-processing in the graph aggregation/result upload (that will be covered in a subsequent PR)
  • Automated tests to cover the above functionality, including some integration tests that interact with a running Fides server
  • Included some nox and docker options to spin up a test environment locally that mimics a parent/child setup, with two running fides instances. I've used this environment to manually verify end-to-end functionality

Steps to Confirm

  • nox -s dev -- child remote_debug
    • the remote_debug flag is optional but can be useful here
  • docker exec -ti fides-fides-1 /bin/bash and then python scripts/load_fides_child_examples.py to configure the fides connector config on the "parent" fides
  • configure any valid connector on the "child" fides via API (it should be accessible at http://0.0.0.0:8081 from a host machine)
  • submit a privacy request against the "parent" fides and confirm that results are output to whatever storage location is configured on the policy
    • e.g.
curl --location --request POST 'http://0.0.0.0:8080/api/v1/privacy-request' \
--header 'Content-Type: application/json' \
--data-raw '[
    {
        "identity": {"email": "[email protected]"},
        "policy_key": "default_access_policy"
    }
]'

Pre-Merge Checklist

  • All CI Pipelines Succeeded
  • Documentation Updated:
    • documentation complete, or draft/outline provided (tag docs-team to complete/review on this branch)
    • documentation issue created (tag docs-team to complete issue separately)
  • Issue Requirements are Met
  • Relevant Follow-Up Issues Created
  • Update CHANGELOG.md

Description Of Changes

Initial implementation of Fides connector functionality. Does not include result aggregation functionality, as that will be covered in a subsequent PR.

Clean up for mypy and linting
This utility will be used in order to perform post processing on Fides connector
results in the graph traversal aggregation.
Use timeouts instead of retries config parameter.
Update our config parameters for proper typing and defaulting with new function call.
@adamsachs adamsachs marked this pull request as ready for review November 29, 2022 15:08
@adamsachs adamsachs requested a review from a team November 29, 2022 15:08
@adamsachs adamsachs self-assigned this Nov 29, 2022
Copy link
Contributor

@sanders41 sanders41 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Just a couple minor questions/suggestions.

…tor access request

Add some fixtures as this requires a sample policy with a local storage destination
so that the request can complete successfully.
Copy link
Contributor

@sanders41 sanders41 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add ability to test Fides Connector Create Fides Connector
2 participants