Enable Manual Webhooks in Request Execution [#1228]

CHANGELOG.md

@@ -42,6 +42,7 @@ The types of changes are:
 * Added support for one-to-many relationships for param_values in SaaS configs [#1253](https://github.com/ethyca/fidesops/pull/1253)
 * Added erasure endpoints for Shopify connector [#1289](https://github.com/ethyca/fidesops/pull/1289)
 * Adds ability to send email notification upon privacy request completion [#1282](https://github.com/ethyca/fidesops/pull/1282)
+* Enable new manual webhooks in privacy request execution [#1285](https://github.com/ethyca/fidesops/pull/1285)
 
 ### Docs
 

docs/fidesops/docs/guides/manual_webhooks.md

@@ -0,0 +1,105 @@
+# Manual Webhooks
+
+Manual webhooks are a simple way for data to be manually uploaded for an access request. Erasure requests are not supported at this time.
+They differ from the more complex [manual connection configs](datasets.md#Configure-a-manual-Dataset) that integrate directly with the graph.
+Manual webhooks gather data *outside* of the graph as a first step, and are more similar to [policy_webhooks](policy_webhooks.md).
+
+
+If you have manual webhooks defined, privacy request execution will exit early and remain in a state of `requires_input`.
+Once data has been manually uploaded for all the manual webhooks, then the privacy request can be resumed. Data uploaded 
+for manual webhooks is passed on directly to the data subject alongside the data package. It is
+not filtered on data category.  Any manual data uploaded is passed on as-is.
+
+
+
+## Configuration
+
+### Create a connection config of type `manual_webhook`
+
+
+```json title="<code>POST api/v1/connection</code>"
+[
+    {"name": "Manual Webhook ConnectionConfig",
+    "key": "manual_webhook_key",
+    "connection_type": "manual_webhook",
+    "access": "read"
+    }
+]
+```
+
+| Field | Description |
+|----|----|
+| `key` | *Optional.* A unique key used to manage your connection config. This is auto-generated from `name` if left blank. Accepted values are alphanumeric, `_`, and `.`. |
+| `name` | A unique user-friendly name for your connection config. This key will also be used to identity the manual webhook|
+| `connection_type` | Should be `manual_webhook` for the resource described here. |
+| `access` | One of `read` or `write` |
+
+
+### Define the fields expected for your `manual_webhook`
+
+Submit a list of fields that will need to be manually uploaded.
+
+
+```json title="<code>PATCH api/v1/connection/{{manual_webhook_key}}/access_manual_webhook</code>"
+{
+    "fields": [
+        {"pii_field": "First Name", "dsr_package_label": "first_name"},
+        {"pii_field": "Last Name", "dsr_package_label": "last_name"},
+        {"pii_field": "Phone Number", "dsr_package_label": null},
+        {"pii_field": "Height", "dsr_package_label": "height"}
+    ]
+}
+```
+
+| Field | Description |
+|----|----|
+| `fields` | *Required.* A list of field mappings with `pii_field` and `dsr_package_label` keys. The `pii_field` is the label fidesops will display when it solicits manual input, and the `dsr_package_label` is the identifier fidesops will use when it uploads the data to the data subject.  If no `dsr_package_label` is supplied, it will be created from the `pii_field`.
+
+
+### Upload manual webhook data for a given privacy request
+
+Privacy request execution will exit early with a status of `requires_input` if we're missing data for `manual_webhooks`.
+A request will need to be made for each manual_webhook to upload the requested data before request execution can proceed.
+
+Note that the fields here are dynamic and should match the fields specified on the manual webhook. All fields are optional.
+If no data exists, an empty dictionary should be uploaded. Fidesops treats this upload as confirmation that the
+system was searched for data related to the data subject.
+
+```json title="<code>PATCH /privacy-request/{{privacy_request_id}}/access_manual_webhook/{{manual_webhook_key}}</code>"
+{
+    "first_name": "Jane",
+    "last_name": "Customer"
+}
+```
+
+### Resume Privacy Request Execution
+
+Once a PrivacyRequest with `requires_input` has had all of its manual data uploaded, prompt the privacy request to resume.
+
+```json title="<code>POST /privacy-request/{{privacy_request_id}}/resume_from_requires_input</code>"
+```
+
+#### Example Upload
+
+In this example, we visited one postgres collection automatically and retrieved Jane's `name`, `email`, and `id`.
+Her `first_name` and `last_name` were manually uploaded as part of the `manual_webhook_key` Manual Webhook
+and directly included here.
+
+```json
+
+{
+  "postgres_example:customer": [
+    {
+      "name": "Jane Customer",
+      "email": "[email protected]",
+      "id": 1
+    }
+  ],
+  "manual_webhook_key": [
+    {
+      "first_name": "Jane",
+      "last_name": "Customer"
+    }
+  ]
+}
+```

docs/fidesops/docs/postman/Fidesops.postman_collection.json

@@ -786,7 +786,7 @@
 						"method": "GET",
 						"header": [],
 						"url": {
-							"raw": "{{host}}/privacy-request/?id={{privacy_request_id}}&verbose=True",
+							"raw": "{{host}}/privacy-request/?request_id={{privacy_request_id}}&verbose=True",
 							"host": [
 								"{{host}}"
 							],
@@ -796,7 +796,7 @@
 							],
 							"query": [
 								{
-									"key": "id",
+									"key": "request_id",
 									"value": "{{privacy_request_id}}"
 								},
 								{
@@ -940,6 +940,35 @@
 						}
 					},
 					"response": []
+				},
+				{
+					"name": "Restart failed node",
+					"request": {
+						"auth": {
+							"type": "bearer",
+							"bearer": [
+								{
+									"key": "token",
+									"value": "{{client_token}}",
+									"type": "string"
+								}
+							]
+						},
+						"method": "POST",
+						"header": [],
+						"url": {
+							"raw": "{{host}}/privacy-request/{{privacy_request_id}}/retry",
+							"host": [
+								"{{host}}"
+							],
+							"path": [
+								"privacy-request",
+								"{{privacy_request_id}}",
+								"retry"
+							]
+						}
+					},
+					"response": []
 				}
 			]
 		},
@@ -2678,6 +2707,133 @@
 						}
 					},
 					"response": []
+				},
+				{
+					"name": "Get all Access Manual Webhooks",
+					"request": {
+						"auth": {
+							"type": "bearer",
+							"bearer": [
+								{
+									"key": "token",
+									"value": "{{client_token}}",
+									"type": "string"
+								}
+							]
+						},
+						"method": "GET",
+						"header": [],
+						"url": {
+							"raw": "{{host}}/connection/{{manual_webhook_key}}/access_manual_webhook",
+							"host": [
+								"{{host}}"
+							],
+							"path": [
+								"connection",
+								"{{manual_webhook_key}}",
+								"access_manual_webhook"
+							]
+						}
+					},
+					"response": []
+				},
+				{
+					"name": "Upload Data for Access Manual Webhook",
+					"request": {
+						"auth": {
+							"type": "bearer",
+							"bearer": [
+								{
+									"key": "token",
+									"value": "{{client_token}}",
+									"type": "string"
+								}
+							]
+						},
+						"method": "PATCH",
+						"header": [],
+						"body": {
+							"mode": "raw",
+							"raw": "{\n    \"first_name\": \"Jane\",\n    \"last_name\": \"Customer\"\n}",
+							"options": {
+								"raw": {
+									"language": "json"
+								}
+							}
+						},
+						"url": {
+							"raw": "{{host}}/privacy-request/{{privacy_request_id}}/access_manual_webhook/{{manual_webhook_key}}",
+							"host": [
+								"{{host}}"
+							],
+							"path": [
+								"privacy-request",
+								"{{privacy_request_id}}",
+								"access_manual_webhook",
+								"{{manual_webhook_key}}"
+							]
+						}
+					},
+					"response": []
+				},
+				{
+					"name": "View Manually Uploaded Data",
+					"request": {
+						"auth": {
+							"type": "bearer",
+							"bearer": [
+								{
+									"key": "token",
+									"value": "{{client_token}}",
+									"type": "string"
+								}
+							]
+						},
+						"method": "GET",
+						"header": [],
+						"url": {
+							"raw": "{{host}}/privacy-request/{{privacy_request_id}}/access_manual_webhook/{{manual_webhook_key}}",
+							"host": [
+								"{{host}}"
+							],
+							"path": [
+								"privacy-request",
+								"{{privacy_request_id}}",
+								"access_manual_webhook",
+								"{{manual_webhook_key}}"
+							]
+						}
+					},
+					"response": []
+				},
+				{
+					"name": "Resume Privacy Request after Manual Input",
+					"request": {
+						"auth": {
+							"type": "bearer",
+							"bearer": [
+								{
+									"key": "token",
+									"value": "{{client_token}}",
+									"type": "string"
+								}
+							]
+						},
+						"method": "POST",
+						"header": [],
+						"url": {
+							"raw": "{{host}}/privacy-request/{{privacy_request_id}}/resume_from_requires_input",
+							"host": [
+								"{{host}}"
+							],
+							"path": [
+								"privacy-request",
+								"{{privacy_request_id}}",
+								"resume_from_requires_input"
+							]
+						}
+					},
+					"response": []
 				}
 			]
 		},
@@ -4169,40 +4325,6 @@
 				}
 			]
 		},
-		{
-			"name": "Restart from Failure",
-			"item": [
-				{
-					"name": "Restart failed node",
-					"request": {
-						"auth": {
-							"type": "bearer",
-							"bearer": [
-								{
-									"key": "token",
-									"value": "{{client_token}}",
-									"type": "string"
-								}
-							]
-						},
-						"method": "POST",
-						"header": [],
-						"url": {
-							"raw": "{{host}}/privacy-request/{{privacy_request_id}}/retry",
-							"host": [
-								"{{host}}"
-							],
-							"path": [
-								"privacy-request",
-								"{{privacy_request_id}}",
-								"retry"
-							]
-						}
-					},
-					"response": []
-				}
-			]
-		},
 		{
 			"name": "ConnectionType",
 			"item": [

docs/fidesops/mkdocs.yml

@@ -28,6 +28,7 @@ nav:
       - Preview Query Execution: guides/query_execution.md
       - Data Rights Protocol: guides/data_rights_protocol.md
       - Configure Automatic Emails: guides/email_communications.md
+      - Configure Manual Webhooks: guides/manual_webhooks.md
   - SaaS Connectors:
       - Connect to SaaS Applications: saas_connectors/saas_connectors.md
       - SaaS Configuration: saas_connectors/saas_config.md

src/fidesops/ops/api/v1/endpoints/manual_webhook_endpoints.py

@@ -1,5 +1,5 @@
 import logging
-from typing import Optional
+from typing import Optional, Sequence
 
 from fastapi import Depends, Security
 from fastapi.encoders import jsonable_encoder
@@ -21,7 +21,11 @@
     WEBHOOK_DELETE,
     WEBHOOK_READ,
 )
-from fidesops.ops.api.v1.urn_registry import ACCESS_MANUAL_WEBHOOK, V1_URL_PREFIX
+from fidesops.ops.api.v1.urn_registry import (
+    ACCESS_MANUAL_WEBHOOK,
+    ACCESS_MANUAL_WEBHOOKS,
+    V1_URL_PREFIX,
+)
 from fidesops.ops.models.connectionconfig import ConnectionConfig, ConnectionType
 from fidesops.ops.models.manual_webhook import AccessManualWebhook
 from fidesops.ops.schemas.manual_webhook_schemas import (
@@ -178,3 +182,21 @@ def delete_access_manual_webhook(
         "Deleted access manual webhook for connection config '%s'",
         connection_config.key,
     )
+
+
+@router.get(
+    ACCESS_MANUAL_WEBHOOKS,
+    status_code=HTTP_200_OK,
+    dependencies=[Security(verify_oauth_client, scopes=[WEBHOOK_READ])],
+    response_model=Sequence[AccessManualWebhookResponse],
+)
+def get_access_manual_webhooks(
+    db: Session = Depends(deps.get_db),
+) -> Sequence[AccessManualWebhookResponse]:
+    """
+    Get all enabled Access Manual Webhooks
+    """
+    logger.info(
+        "Retrieving all enabled access manual webhooks",
+    )
+    return AccessManualWebhook.get_enabled(db)