Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bigquery Row-Level Deletes #5293

Open
wants to merge 11 commits into
base: main
Choose a base branch
from
Open

Bigquery Row-Level Deletes #5293

wants to merge 11 commits into from

Conversation

pattisdr
Copy link
Contributor

@pattisdr pattisdr commented Sep 17, 2024 •
edited
Loading

Closes #2744

Description Of Changes

Adds initial support for collection-level masking overrides via Bigquery.

Specifying a delete masking strategy override at the collection level on a bigquery dataset yaml will cause matching rows to be deleted instead of updated when running an erasure request. The collection-level override takes precedence over the policy override. Again this just takes affect for Bigquery only as a first iteration

...
   collections:
      - name: address
        fides_meta:
          masking_strategy_override:
            strategy: delete

Code Changes

  • Changes scoped to BigQuery - if the collection contains a delete masking override, generate a delete statement instead of an update statement -

Caution

  • Collections need FK's to facilitate deletes (and updates for that matter)
  • Erasure order is unchanged. If you require that one node be erased before another, use erase_after functionality
  • masking_strategy_overrrides at the collection level are ignored for all but bigquery integrations

Steps to Confirm

Pre-Merge Checklist

  • All CI Pipelines Succeeded
  • Documentation:
    • documentation complete, PR opened in fidesdocs
    • documentation issue created in fidesdocs
    • if there are any new client scopes created as part of the pull request, remember to update public-facing documentation that references our scope registry
  • Issue Requirements are Met
  • Relevant Follow-Up Issues Created
  • Update CHANGELOG.md
  • For API changes, the Postman collection has been updated
  • If there are any database migrations:
    • Ensure that your downrev is up to date with the latest revision on main
    • Ensure that your downgrade() migration is correct and works
      • If a downgrade migration is not possible for this change, please call this out in the PR description!

@vercel Vercel
Copy link

vercel bot commented Sep 17, 2024 •
edited
Loading

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment
Name Status Preview Comments Updated (UTC)
fides-plus-nightly ⬜️ Ignored (Inspect) Visit Preview Sep 21, 2024 9:00am

@pattisdr pattisdr mentioned this pull request Sep 17, 2024
Closed
13 tasks
@cypress Cypress
Copy link

cypress bot commented Sep 17, 2024 •
edited
Loading

fides    Run #10045

Run Properties:  status check passed Passed #10045  •  git commit 5231c441ba ℹ️: Merge 2bc6a90e37a17badb20063b500e4bb9eb1d0c70e into b4e014f2001d57284659a686651e...
Project fides
Branch Review refs/pull/5293/merge
Run status status check passed Passed #10045
Run duration 00m 38s
Commit git commit 5231c441ba ℹ️: Merge 2bc6a90e37a17badb20063b500e4bb9eb1d0c70e into b4e014f2001d57284659a686651e...
Committer Dawn Pattison
View all properties for this run ↗︎
Test results
Tests that failed  Failures 0
Tests that were flaky  Flaky 0
Tests that did not run due to a developer annotating a test with .skip  Pending 0
Tests that did not run due to a failure in a mocha hook  Skipped 0
Tests that passed  Passing 4
⚠️ You've recorded test results over your free plan limit.
Upgrade your plan to view test results.
View all changes introduced in this branch ↗︎

@pattisdr
Add end-to-end testing of row-level deletes as a masking strategy for…
9fa25f2 
… bigquery

- Seed existing bigquery table with employee record
- Updated employee dataset reference direction
- Confirm employee record exists before running DSR and confirm it's been removed afterwards
@pattisdr
Add query config tests for bigquery around the update or delete state…
0f60402 
…ment built.
@pattisdr
Improve docstrings
4973996
@pattisdr pattisdr marked this pull request as ready for review September 19, 2024 22:08
@pattisdr pattisdr added the run unsafe ci checks Runs fideops-related CI checks that require sensitive credentials label Sep 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
run unsafe ci checks Runs fideops-related CI checks that require sensitive credentials
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@pattisdr