Backend Consent Reporting #3095

pattisdr · 2023-04-18T03:38:17Z

❗ Contains migration; check downrev before merge
Closes #2836
Closes #2837

Code Changes

Index Privacy Preference History / Current Privacy Preference "created" fields as this will be a common filter. Missed this earlier because this field is defined on the base model.
Two new endpoints/new scopes for consent reporting: historical and current preferences
Updating PrivacyPreferenceHistory.affected_system_status at the connector level as we execute privacy requests for consent reporting as well as secondary user ids - see updates in saas connector and consent email connector

Steps to Confirm

In Postman collection, get a privacy_notice_history_id Privacy Notices > Create Privacy Notice endpoint
In Postman collection, save Privacy preferences with:
- See Privacy Preferences > "Create Verification Code for Consent Request" endpoint to get a consent_request_id
- See Privacy Preferences > "Verify Code and Save Privacy Preferences" to create a PrivacyPreferenceHistory and `CurrentPrivacyPreference record
Make requests to GET {{host}}/historical-privacy-preferences and GET {{host}}/current-privacy-preferences to see the results of the two endpoints

Pre-Merge Checklist

All CI Pipelines Succeeded
Documentation:
- documentation complete, PR opened in fidesdocs
- documentation issue created in fidesdocs
Issue Requirements are Met
Relevant Follow-Up Issues Created
Update CHANGELOG.md
For API changes, the Postman collection has been updated

Description Of Changes

Start tracking which systems were affected on each Privacy Preference History record for consent reporting. Start tracking a subset of the relevant secondary user ids (like browser identifiers) on the Privacy Preference History record instead of all identifiers. Only relevant user ides should be in this report. Surface endpoints for consent reporting.

Historical Preferences

GET {{host}}/historical-privacy-preferences

{
    "items": [
        {
            "id": "pri_30c36d54-b494-4c64-8a81-a330da52106e",
            "privacy_request_id": "pri_5b56727d-d27e-4a27-9ea4-4f75bbbf07c8",
            "user_id": "[email protected]",
            "secondary_user_ids": {"ga_client_id": "UA-XSDFRSDFSDFTEST"},
            "request_timestamp": "2023-04-18T03:49:32.077520+00:00",
            "request_origin": "privacy_center",
            "request_status": "complete",
            "request_type": "consent",
            "approver_id": "[email protected]",
            "privacy_notice_history_id": "pri_223d72b4-5355-42ae-94e5-076f3f6c2141",
            "preference": "opt_out",
            "user_geography": "us_ca",
            "relevant_systems": ["mailchimp_transactional", "google_analytics"],
            "affected_system_status": {"mailchimp_transactional": "skipped", "google_analytics": "complete"},
            "url_recorded": "example.com/privacy_center",
            "user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_2) AppleWebKit/324.42 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/425.24"
        }
    ],
    "total": 1,
    "page": 1,
    "size": 50
}

Current Preferences

GET {{{host}}/current-privacy-preferences

{
    "items": [
        {
            "id": "cur_f2e98f01-da72-4665-8705-d326e39a73cc",
            "preference": "opt_out",
            "privacy_notice_history_id": "pri_223d72b4-5355-42ae-94e5-076f3f6c2141",
            "privacy_preference_history_id": "pri_30c36d54-b494-4c64-8a81-a330da52106e",
            "provided_identity_id": "pro_26bc257c-ad13-48fa-80c8-70fb4991f564",
            "created_at": "2023-04-18T03:49:32.134876+00:00"
        }
    ],
    "total": 1,
    "page": 1,
    "size": 50
}

…erences across all users, and another to surface the most current privacy preferences (with a subset of fields for now) - Stop setting secondary_user_ids on privacy request create: instead, dictate to consent connectors to add only relevant identities that we intend to persist to the connector - On both saas and email consent connectors, update Privacy Preference History Records with "affected_system_status" and "secondary_user_ids" where applicable. If the connector isn't relevant to the privacy preference or we don't have the information to send the request, add a "skipped" status, otherwise, update with pending/completed/errored. Only add secondary user ids where our goal is to propagate those ids.

cypress · 2023-04-18T03:54:19Z

Passing run #1445 ↗︎

0	3	0	0	0
⚠️ You've recorded test results over your free plan limit. Upgrade your plan to view test results.

Details:

Merge `4226bde` into `a603f3d`...
Project: fides	Commit: `121e866fbe ℹ️`
Status: Passed	Duration: 00:37 💡
Started: Apr 19, 2023 10:27 PM	Ended: Apr 19, 2023 10:28 PM

_{This comment has been generated by cypress-bot as a result of this
project's GitHub integration settings.}

src/fides/api/ops/api/v1/endpoints/consent_request_endpoints.py

src/fides/api/ops/api/v1/endpoints/privacy_preference_endpoints.py

src/fides/api/ops/util/consent_util.py

src/fides/api/ops/task/graph_task.py

src/fides/api/ctl/migrations/versions/3ac5aef6fe45_index_privacy_preference_created.py

src/fides/api/ops/service/connectors/consent_email_connector.py

pattisdr · 2023-04-18T04:21:57Z

@adamsachs requesting your review because I think you're the most familiar with these new models.
@SteveDMurphy tagging you for awareness because I'm tracking some specific information here for consent reporting for "saas" consent connectors and "email" consent connectors and assume this will be relevant for dynamo db.
@seanpreston tagging you here as you just did the endpoints for the old style Consent records, and this is the new style version of that.

src/fides/api/ops/api/v1/endpoints/privacy_preference_endpoints.py

src/fides/api/ops/models/privacy_preference.py

adamsachs

generally this looks good to me @pattisdr so i'm going to give a tentative approval! i've got some relatively minor nits and questions, but i'm not certain any of them need to be addressed.

maintaining the PrivacyPreferenceHistory.affected_system_status state definitely feels like the most complex piece here. i think you've done a good job here of keeping things relatively organized and traceable given how complex that requirement is, especially with the different possibilities between consent propagating through saas and email connectors.

on a broader, separate note - not anything to address here - but the fact that you've needed to take so much care in covering those edge cases across different code paths does make me feel like we're getting to the point where we may need a broader refactor to streamline some of the consent propagation processing. i think this will only be more necessary as we work to support consent propagation in the DB connectors codepaths - and maybe that will be a natural point for a bit broader of a refactor.

i'd like to get your POV on this. to me it's starting to feel like with how things are structured now, we need to account for many different edge cases across many different code paths when considering updates/improvements to the consent workflow. i think what you've done on this PR makes sense, but as the use cases and requirements evolve, i'm getting the sense that it may not be sustainable. the consent pieces almost feel sprinkled throughout the DSR execution workflow, in a certain sense.

(i hope this goes without saying, but that's not in any way meant to be a knock on how you've built things here or in previous increments - we (mainly you :)) have rapidly built an impressive amount of support for complex requirements in basically no time. i just want to voice a forward-looking impression i have to see if/how it aligns with your perspective)

src/fides/api/ops/api/v1/endpoints/privacy_preference_endpoints.py

src/fides/api/ops/schemas/privacy_preference.py

src/fides/api/ops/task/graph_task.py

src/fides/api/ops/util/consent_util.py

src/fides/api/ops/task/graph_task.py

- Share validate_start_and_end_filters between privacy request, consent request, and privacy preference endpoints. - Index updated_at additionally on CurrentPrivacyPreference and update this report to filter on updated at. - Pass in the session off of task resources through to the PrivacyPreference methods that save affected systems and secondary user ids to follow typical patterns. - Remove comma after catching SkippingConsentPropagation exception

# Conflicts: # CHANGELOG.md

codecov · 2023-04-19T21:06:48Z

Codecov Report

Patch coverage: 99.40% and project coverage change: +0.15 🎉

Comparison is base (a603f3d) 87.39% compared to head (4226bde) 87.54%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3095      +/-   ##
==========================================
+ Coverage   87.39%   87.54%   +0.15%     
==========================================
  Files         306      307       +1     
  Lines       17600    17734     +134     
  Branches     2271     2288      +17     
==========================================
+ Hits        15381    15525     +144     
+ Misses       1801     1793       -8     
+ Partials      418      416       -2

Impacted Files	Coverage Δ
.../service/privacy_request/request_runner_service.py	`77.65% <ø> (ø)`
.../ops/service/connectors/consent_email_connector.py	`97.14% <90.90%> (-0.76%)`	⬇️
.../ops/api/v1/endpoints/consent_request_endpoints.py	`86.48% <100.00%> (+2.27%)`	⬆️
...s/api/v1/endpoints/privacy_preference_endpoints.py	`100.00% <100.00%> (ø)`
.../ops/api/v1/endpoints/privacy_request_endpoints.py	`90.36% <100.00%> (+0.48%)`	⬆️
src/fides/api/ops/api/v1/endpoints/utils.py	`100.00% <100.00%> (ø)`
src/fides/api/ops/api/v1/scope_registry.py	`100.00% <100.00%> (ø)`
src/fides/api/ops/api/v1/urn_registry.py	`100.00% <100.00%> (ø)`
src/fides/api/ops/models/connectionconfig.py	`96.03% <100.00%> (+0.20%)`	⬆️
src/fides/api/ops/models/privacy_preference.py	`100.00% <100.00%> (ø)`
... and 5 more

... and 1 file with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

pattisdr · 2023-04-19T21:07:59Z

@adamsachs I'm with you, the current path is concerning especially given that we have a consent database connector coming in, where this will surely need additional special handling.

Big picture for each consent connector we need to:

Get relevant privacy preferences
Mark irrelevant preferences as skipped
Mark relevant preferences with status of privacy request - pending/failed/complete
Add identifiers used if relevant preferences exist.

The main trickiness here was that the consent email send happens in a completely separate process where we batch all preferences collected throughout the week and send a single email. That process required hooking in to a lot of different locations. For now, I've tried to share methods between the saas and consent email connectors and added multiple levels of tests.

…mes in saas connector tests as well).

Update postman collection and changelog.

a7c78d7

pattisdr commented Apr 18, 2023

View reviewed changes

src/fides/api/ctl/migrations/versions/3ac5aef6fe45_index_privacy_preference_created.py Outdated Show resolved Hide resolved

pattisdr commented Apr 18, 2023

View reviewed changes

src/fides/api/ops/service/connectors/consent_email_connector.py Outdated Show resolved Hide resolved

pattisdr commented Apr 18, 2023

View reviewed changes

src/fides/api/ops/service/connectors/consent_email_connector.py Outdated Show resolved Hide resolved

pattisdr requested review from adamsachs, SteveDMurphy and seanpreston April 18, 2023 04:20

pattisdr added the run unsafe ci checks Runs fides-related CI checks that require sensitive credentials label Apr 18, 2023

pattisdr commented Apr 18, 2023

View reviewed changes

src/fides/api/ops/api/v1/endpoints/privacy_preference_endpoints.py Outdated Show resolved Hide resolved

pattisdr marked this pull request as ready for review April 18, 2023 04:25

seanpreston reviewed Apr 18, 2023

View reviewed changes

src/fides/api/ops/models/privacy_preference.py Outdated Show resolved Hide resolved

seanpreston reviewed Apr 18, 2023

View reviewed changes

src/fides/api/ops/models/privacy_preference.py Show resolved Hide resolved

adamsachs approved these changes Apr 19, 2023

View reviewed changes

pattisdr added 2 commits April 19, 2023 15:35

Merge branch 'main' into fides_2836_consent_reporting_endpoints

1afa1d7

# Conflicts: # CHANGELOG.md

Add further testing for code cov (a lot of this is tested multiple ti…

4226bde

…mes in saas connector tests as well).

pattisdr merged commit 45e7ab4 into main Apr 19, 2023

pattisdr deleted the fides_2836_consent_reporting_endpoints branch April 19, 2023 23:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Backend Consent Reporting #3095

Backend Consent Reporting #3095

pattisdr commented Apr 18, 2023 •

edited

Loading

cypress bot commented Apr 18, 2023 •

edited

Loading

pattisdr commented Apr 18, 2023

adamsachs left a comment •

edited

Loading

codecov bot commented Apr 19, 2023 •

edited

Loading

pattisdr commented Apr 19, 2023

Backend Consent Reporting #3095

Backend Consent Reporting #3095

Conversation

pattisdr commented Apr 18, 2023 • edited Loading

Code Changes

Steps to Confirm

Pre-Merge Checklist

Description Of Changes

Historical Preferences

Current Preferences

cypress bot commented Apr 18, 2023 • edited Loading

Passing run #1445 ↗︎

pattisdr commented Apr 18, 2023

adamsachs left a comment • edited Loading

Choose a reason for hiding this comment

codecov bot commented Apr 19, 2023 • edited Loading

Codecov Report

pattisdr commented Apr 19, 2023

pattisdr commented Apr 18, 2023 •

edited

Loading

cypress bot commented Apr 18, 2023 •

edited

Loading

adamsachs left a comment •

edited

Loading

codecov bot commented Apr 19, 2023 •

edited

Loading