Enable Manual Webhooks in Request Execution [#1228] #1285

pattisdr · 2022-09-09T02:41:15Z

👉 Note that new scopes have been added here.

Purpose

Enable the new "Manual Webhook" Connector. In short, if manual webhooks are configured, the privacy request will be in a state of requires_input until data has been supplied for each webhook. Once the privacy request is resumed, the graph will run as usual and any manually uploaded data will be directly added to the final data package. The data is not used as part of the graph and is not filtered by data category. All uploaded data is passed to the user as-is.

Changes

Add four new API endpoints
- Get all access manual webhooks GET "/access_manual_webhook"
- Upload data for a manual webhook on a given privacy request: PATCH /privacy-request/{privacy_request_id}/access_manual_webhook/{connection_key}
  - There is no webhook key since there's only one AccessManualWebhook per ConnectionConfig of type manual_webhook so the connection_key is for the ConnectionConfig.
- Resume privacy request in requires_input state once all webhook data is uploaded: POST "/privacy-request/{privacy_request_id}/resume_from_requires_input"
- Get manually uploaded data for a given webhook GET /privacy-request/{privacy_request_id}/access_manual_webhook/{connection_key}
Update privacy request execution to accommodate new manual webhooks. Once all data has been uploaded, proceed with request execution, and then take the manual data and add it directly to the final upload package.

Checklist

Ticket

Fixes #1228

…lar privacy request.

…a manual webhook for a given privacy request with "requires_input" status.

…ebhook. - Add new scopes for the endpoints to upload/view manual data for webhooks. - Enforce that at least one field is added when defining a manual webhook, and add a fallback if no fields were defined.

…tus once all input has been added. None of the fields are required, but the a key for each manual webhook still needs to exist in the cache to proceed. As part of request execution check if data has been uploaded (data can be empty) for all manual webhooks. If True, we can proceed with request execution, otherwise, we put the PrivacyRequest in "requires_input" status and exits. Also adds the manual data uploaded directly to the packet we upload to the user at the very end.

…vacy request when getting its status.

…webhook_execution # Conflicts: # CHANGELOG.md

pattisdr · 2022-09-09T15:04:19Z

src/fidesops/ops/service/privacy_request/request_runner_service.py

+        manual_data, proceed = get_access_manual_webhook_inputs(
+            session, privacy_request
+        )
+        if not proceed:
+            return


This is the primary change of this PR - this puts a PrivacyRequest in requires_input status if we don't have every manual webhook confirmed. If we do have confirmation from all our manual webhooks, we retrieve any manually uploaded data, and this is passed onto the upload_access_results function and data is combined with the automatically retrieved data at the end.

pattisdr · 2022-09-09T15:05:46Z

src/fidesops/ops/service/privacy_request/request_runner_service.py

+def get_access_manual_webhook_inputs(
+    db: Session, privacy_request: PrivacyRequest
+) -> Tuple[Dict[str, List[Dict[str, Optional[Any]]]], bool]:
+    """Retrieves manually uploaded data for all AccessManualWebhooks and formats in a way
+    to match automatically retrieved data. Also returns if execution should proceed.
+
+    This data will be uploaded to the user as-is, without filtering.
+    """
+    manual_inputs: Dict[str, List[Dict[str, Optional[Any]]]] = {}
+
+    try:
+        for manual_webhook in AccessManualWebhook.get_enabled(db):
+            manual_inputs[manual_webhook.connection_config.key] = [
+                privacy_request.get_manual_webhook_input(manual_webhook)
+            ]
+    except (MissingManualWebhookData, ValidationError) as exc:
+        logger.info(exc)
+        privacy_request.status = PrivacyRequestStatus.requires_input
+        privacy_request.save(db)
+        return manual_inputs, False
+
+    return manual_inputs, True


Normally the access results have keys in the format "dataset:collection", so here I am using the manual_webhook connectionconfig key instead, since there is no "dataset" or "collection" defined for the webhook results. I think an overlap with existing keys is unlikely since they are mashups of dataset and collection names.

As an aside, since ConnectionConfigs and AccessManualWebhooks have a 1:1 relationship, I am not giving a separate identifier to the AccessManualWebhook, and using its ConnectionConfig key everywhere. So the question is, do we want them to be separate? We could store a separate AccessManualWebhook key and use that here, and on the UI it could be surfaced connection_identifier, like we use for saas configs. I didn't think we needed this extra identifier, but I can see an opposing argument.

Discussed with Sean, seems okay for now to have this 1:1 mapping and use the connection_config key as the identifier for the entire webhook.

src/fidesops/ops/service/privacy_request/request_runner_service.py

…ly has erasure rules.

…webhook_execution # Conflicts: # CHANGELOG.md

src/fidesops/ops/api/v1/endpoints/privacy_request_endpoints.py

eastandwestwind · 2022-09-12T19:18:49Z

src/fidesops/ops/api/v1/scope_registry.py

 )
 PRIVACY_REQUEST_REVIEW = "privacy-request:review"
+PRIVACY_REQUEST_UPLOAD_DATA = "privacy-request:upload_data"
+PRIVACY_REQUEST_VIEW_DATA = "privacy-request:view_data"


do we also need to add new scopes in fideslib at this point?

@sanders41 are we keeping fideslib scopes synced with fidesops scopes with the fides unification work, or is it okay that this is the most current list?

I have been trying to keep them in sync, but it won't break if they get out of sync for now.

src/fidesops/ops/models/manual_webhook.py

src/fidesops/ops/schemas/manual_webhook_schemas.py

src/fidesops/ops/service/privacy_request/request_runner_service.py

tests/ops/service/privacy_request/request_runner_service_test.py

eastandwestwind · 2022-09-12T19:52:00Z

Nice work on this @pattisdr ! Learning some new things around pydantic models and types. Just a couple Qs to address.

Tagging @conceptualshark also to review docs.

…webhook_execution # Conflicts: # CHANGELOG.md # src/fidesops/ops/service/privacy_request/request_runner_service.py # tests/ops/service/privacy_request/request_runner_service_test.py

pattisdr · 2022-09-12T21:52:37Z

Thanks for your comments @eastandwestwind, back to you!

eastandwestwind

Appreciate all the detail on these changes, @pattisdr !

eastandwestwind · 2022-09-12T22:16:43Z

src/fidesops/ops/service/privacy_request/request_runner_service.py

+class ManualWebhookResults(BaseSchema):
+    """Represents manual webhook data retrieved from the cache and whether privacy request execution should continue"""
+
+    manual_data: Dict[str, List[Dict[str, Optional[Any]]]]


conceptualshark

This looks great! Thank you for the work on the docs!

pattisdr · 2022-09-13T14:14:16Z

Thanks for taking a look @conceptualshark!

* Add a method to cache data supplied for a manual webhook on a particular privacy request. * Add an endpoint to retrieve all enabled access manual webhooks. * Add an endpoint for uploading manual data corresponding to fields in a manual webhook for a given privacy request with "requires_input" status. * Add an endpoint to view data manually uploaded for an access manual webhook. - Add new scopes for the endpoints to upload/view manual data for webhooks. - Enforce that at least one field is added when defining a manual webhook, and add a fallback if no fields were defined. * Add an endpoint to resume a privacy request from "requires_input" status once all input has been added. None of the fields are required, but the a key for each manual webhook still needs to exist in the cache to proceed. As part of request execution check if data has been uploaded (data can be empty) for all manual webhooks. If True, we can proceed with request execution, otherwise, we put the PrivacyRequest in "requires_input" status and exits. Also adds the manual data uploaded directly to the packet we upload to the user at the very end. * Update postman collection. * Fix request_id query param in existing postman request. * Include additional details about how to resume a "requires_input" privacy request when getting its status. * Add docs and update changelog. * Upload new ERD diagram. * Don't put a privacy request in requires_input state if this policy only has erasure rules. * Respond to CR! * Update manual_webhooks.md

pattisdr added 8 commits September 8, 2022 14:34

Add a method to cache data supplied for a manual webhook on a particu…

68a87ff

…lar privacy request.

Add an endpoint to retrieve all enabled access manual webhooks.

f6feeb2

Add an endpoint for uploading manual data corresponding to fields in …

76a7f54

…a manual webhook for a given privacy request with "requires_input" status.

Add an endpoint to view data manually uploaded for an access manual w…

8f7d7e1

…ebhook. - Add new scopes for the endpoints to upload/view manual data for webhooks. - Enforce that at least one field is added when defining a manual webhook, and add a fallback if no fields were defined.

Update postman collection.

0353e87

Fix request_id query param in existing postman request.

fe57d37

Include additional details about how to resume a "requires_input" pri…

2ea2130

…vacy request when getting its status.

pattisdr changed the title ~~Enable Manual Webhook Request Execution [#1228]~~ Enable Manual Webhooks in Request Execution [#1228] Sep 9, 2022

pattisdr added 3 commits September 8, 2022 22:43

Add docs and update changelog.

b87f4b5

Merge remote-tracking branch 'ethyca/main' into fidesops_1228_manual_…

f818633

…webhook_execution # Conflicts: # CHANGELOG.md

Upload new ERD diagram.

dd1ddc1

pattisdr marked this pull request as ready for review September 9, 2022 14:48

pattisdr commented Sep 9, 2022

View reviewed changes

src/fidesops/ops/service/privacy_request/request_runner_service.py Outdated Show resolved Hide resolved

pattisdr added 2 commits September 12, 2022 08:07

Don't put a privacy request in requires_input state if this policy on…

e610e5c

…ly has erasure rules.

Merge remote-tracking branch 'ethyca/main' into fidesops_1228_manual_…

2d73b51

…webhook_execution # Conflicts: # CHANGELOG.md

seanpreston assigned eastandwestwind Sep 12, 2022

eastandwestwind reviewed Sep 12, 2022

View reviewed changes

eastandwestwind requested a review from conceptualshark September 12, 2022 19:52

pattisdr added 2 commits September 12, 2022 15:28

Merge remote-tracking branch 'ethyca/main' into fidesops_1228_manual_…

03d0643

…webhook_execution # Conflicts: # CHANGELOG.md # src/fidesops/ops/service/privacy_request/request_runner_service.py # tests/ops/service/privacy_request/request_runner_service_test.py

Respond to CR!

77b64d9

eastandwestwind approved these changes Sep 12, 2022

View reviewed changes

conceptualshark approved these changes Sep 13, 2022

View reviewed changes

Update manual_webhooks.md

68284d4

eastandwestwind merged commit df4834a into main Sep 13, 2022

eastandwestwind deleted the fidesops_1228_manual_webhook_execution branch September 13, 2022 15:07

This was referenced Sep 13, 2022

Handling Invalid Tokens [#1257] ethyca/fideslib#71

Merged

Manual Webhook Followup: UI scopes / Delete [#1228] #1308

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable Manual Webhooks in Request Execution [#1228] #1285

Enable Manual Webhooks in Request Execution [#1228] #1285

pattisdr commented Sep 9, 2022 •

edited

Loading

pattisdr Sep 9, 2022

pattisdr Sep 9, 2022

pattisdr Sep 9, 2022

eastandwestwind Sep 12, 2022

pattisdr Sep 12, 2022

sanders41 Sep 12, 2022

eastandwestwind commented Sep 12, 2022

pattisdr commented Sep 12, 2022

eastandwestwind left a comment

eastandwestwind Sep 12, 2022

conceptualshark left a comment

pattisdr commented Sep 13, 2022

Enable Manual Webhooks in Request Execution [#1228] #1285

Enable Manual Webhooks in Request Execution [#1228] #1285

Conversation

pattisdr commented Sep 9, 2022 • edited Loading

Purpose

Changes

Checklist

Ticket

pattisdr Sep 9, 2022

Choose a reason for hiding this comment

pattisdr Sep 9, 2022

Choose a reason for hiding this comment

pattisdr Sep 9, 2022

Choose a reason for hiding this comment

eastandwestwind Sep 12, 2022

Choose a reason for hiding this comment

pattisdr Sep 12, 2022

Choose a reason for hiding this comment

sanders41 Sep 12, 2022

Choose a reason for hiding this comment

eastandwestwind commented Sep 12, 2022

pattisdr commented Sep 12, 2022

eastandwestwind left a comment

Choose a reason for hiding this comment

eastandwestwind Sep 12, 2022

Choose a reason for hiding this comment

conceptualshark left a comment

Choose a reason for hiding this comment

pattisdr commented Sep 13, 2022

pattisdr commented Sep 9, 2022 •

edited

Loading