Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PROD-2316: database exclude list #5080

Conversation

thingscouldbeworse
Copy link
Contributor

@thingscouldbeworse thingscouldbeworse commented Jul 11, 2024

DB prereq for #PROD-2316

Description Of Changes

BE adjustments to allow excluding databases in a monitor along with including them

Code Changes

  • add an excluded_databases property to the MonitorConfig model

Steps to Confirm

  • list any manual steps for reviewers to confirm the changes

Pre-Merge Checklist

  • All CI Pipelines Succeeded
  • Documentation:
    • documentation complete, PR opened in fidesdocs
    • documentation issue created in fidesdocs
    • if there are any new client scopes created as part of the pull request, remember to update public-facing documentation that references our scope registry
  • Issue Requirements are Met
  • Relevant Follow-Up Issues Created
  • Update CHANGELOG.md
  • For API changes, the Postman collection has been updated
  • If there are any database migrations:
    • Ensure that your downrev is up to date with the latest revision on main
    • Ensure that your downgrade() migration is correct and works
      • If a downgrade migration is not possible for this change, please call this out in the PR description!

Copy link

vercel bot commented Jul 11, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment
Name Status Preview Comments Updated (UTC)
fides-plus-nightly ⬜️ Ignored (Inspect) Visit Preview Jul 17, 2024 6:29pm

Copy link

cypress bot commented Jul 12, 2024

Passing run #8976 ↗︎

0 4 0 0 Flakiness 0
⚠️ You've recorded test results over your free plan limit.
Upgrade your plan to view test results.

Details:

Merge e5d32d7 into f390fc9...
Project: fides Commit: 368384237f ℹ️
Status: Passed Duration: 00:34 💡
Started: Jul 17, 2024 6:40 PM Ended: Jul 17, 2024 6:40 PM

Review all test suite changes for PR #5080 ↗︎

Copy link

codecov bot commented Jul 12, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 86.48%. Comparing base (f390fc9) to head (e5d32d7).
Report is 2 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #5080      +/-   ##
==========================================
+ Coverage   86.47%   86.48%   +0.01%     
==========================================
  Files         357      357              
  Lines       22271    22280       +9     
  Branches     2944     2945       +1     
==========================================
+ Hits        19258    19269      +11     
+ Misses       2498     2496       -2     
  Partials      515      515              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@jpople jpople mentioned this pull request Jul 17, 2024
10 tasks
Copy link
Contributor

@erosselli erosselli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work 🚀 I left some questions and comments :)

src/fides/api/models/detection_discovery.py Show resolved Hide resolved
Comment on lines 156 to 158
"""Check that the list of databases included and excluded has no intersection"""
include = data.get("databases", [])
exclude = data.get("excluded_databases", [])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it make sense to have configured both databases AND excluded_databases? I'd think this would be an either you explicitly monitor specific DBs, or you explicitly "skip" specific DBs, but not both?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no, and I misread that slack thread. Here is the fix: ca17da8

raise ValueError(
"The lists of included and excluded databases must have no overlap."
)
return True
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we return a boolean here if we don't use the return value? also I think either we raise the exception in the invalid configuration case and not return anything, or we return True when OK and False when not. I think I prefer the exception tbh

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I buy that, fixed here: 3cffff6

assert mc.databases == ["db1", "db2"]
assert mc.excluded_databases == ["db3"]

def test_update_monitor_config_fails_with_conflicting_dbs(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice work on the tests 😎

Copy link
Contributor

@erosselli erosselli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved! Left two very minor suggestions

Comment on lines 146 to 149
"""
Override the base class `create` to validate database include/exclude
and derive the `execution_trigger` dict field
"""
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"""
Override the base class `create` to validate database include/exclude
and derive the `execution_trigger` dict field
"""
"""
Override the base class `update` to validate database include/exclude
and derive the `execution_trigger` dict field
"""

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice catch; e5d32d7

self, db: Session, create_monitor_config, connection_config: ConnectionConfig
) -> None:
""" """
with pytest.raises(Exception):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: can we check specifically for ValueError here rather than exception?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you, nice catch: e5d32d7

@thingscouldbeworse thingscouldbeworse merged commit e4a1cfa into main Jul 17, 2024
51 checks passed
@thingscouldbeworse thingscouldbeworse deleted the PROD-2316_support-excluded-databases-field-on-monitor-config branch July 17, 2024 19:06
Copy link

cypress bot commented Jul 17, 2024

Passing run #8979 ↗︎

0 4 0 0 Flakiness 0
⚠️ You've recorded test results over your free plan limit.
Upgrade your plan to view test results.

Details:

PROD-2316: database exclude list (#5080)
Project: fides Commit: e4a1cfa7ad
Status: Passed Duration: 00:37 💡
Started: Jul 17, 2024 7:17 PM Ended: Jul 17, 2024 7:17 PM

Review all test suite changes for PR #5080 ↗︎

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants