Merge unified fides into main #1245
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Bumps [pydash](https://github.com/dgilland/pydash) from 5.0.2 to 5.1.0. - [Release notes](https://github.com/dgilland/pydash/releases) - [Changelog](https://github.com/dgilland/pydash/blob/develop/CHANGELOG.rst) - [Commits](dgilland/pydash@v5.0.2...v5.1.0) --- updated-dependencies: - dependency-name: pydash dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Updates the requirements on [boto3](https://github.com/boto/boto3) to permit the latest version. - [Release notes](https://github.com/boto/boto3/releases) - [Changelog](https://github.com/boto/boto3/blob/develop/CHANGELOG.rst) - [Commits](boto/boto3@1.18.14...1.24.34) --- updated-dependencies: - dependency-name: boto3 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [dask](https://github.com/dask/dask) from 2022.6.1 to 2022.7.0. - [Release notes](https://github.com/dask/dask/releases) - [Changelog](https://github.com/dask/dask/blob/main/docs/release-procedure.md) - [Commits](dask/dask@2022.6.1...2022.7.0) --- updated-dependencies: - dependency-name: dask dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…changed for config variables. (#921)
* Correct build arg variable name * Update CHANGELOG Co-authored-by: Paul Sanders <[email protected]>
* Update python docker base image from slim-buster to slim-bullseye * Update CHANGELOG * Remove ipython from dev-requirements.txt Co-authored-by: Paul Sanders <[email protected]> Co-authored-by: Sean Preston <[email protected]>
Updates the requirements on [boto3](https://github.com/boto/boto3) to permit the latest version. - [Release notes](https://github.com/boto/boto3/releases) - [Changelog](https://github.com/boto/boto3/blob/develop/CHANGELOG.rst) - [Commits](boto/boto3@1.24.34...1.24.36) --- updated-dependencies: - dependency-name: boto3 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [faker](https://github.com/joke2k/faker) from 13.15.0 to 13.15.1. - [Release notes](https://github.com/joke2k/faker/releases) - [Changelog](https://github.com/joke2k/faker/blob/master/CHANGELOG.md) - [Commits](joke2k/faker@v13.15.0...v13.15.1) --- updated-dependencies: - dependency-name: faker dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [types-ujson](https://github.com/python/typeshed) from 5.2.0 to 5.4.0. - [Release notes](https://github.com/python/typeshed/releases) - [Commits](https://github.com/python/typeshed/commits) --- updated-dependencies: - dependency-name: types-ujson dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* add db vs saas to connection type api * adds changelog line * mypy * fix test * format * more tests * formatting * adds system type query param * format * adjust test saas search * correct query param
* Create noxfile.py * update the dockerfile with more stages * add GitPython as a dev requirement (used by nox builds) * add the noxfiles, all docker builds work * fix the "make compose-build" command * remove the worker docker stage due to it being redundant with prod * create a unified compose file for integrations * update the password prompt to be accurate * copy/pasta run_infrastructure into the noxfiles dir so it can be called directly via python * add create_user and seed_test_data to the nox utils * docs commands work * get the generic dev command working * add db commands to utils * clean up mypy configuration in pyproject.toml and remove config from setup.cfg * simplify some of the CI targets and start updating the pytest targets * cleanup run_infrastructure * update ci_suite and other CI nox targets * add nox as a dev-requirement * get dev commands working, sans quickstart * tweak to the compose_down constant * get the new pylint target passing * remove analytics_id * updated the changelog * Apply suggestions from code review Co-authored-by: Paul Sanders <[email protected]> * fix the xenon command * add a few tweaks to the worker dev command * fix an import issue * update dockerignore * move mssql to the bottom of the datastore list Co-authored-by: Paul Sanders <[email protected]> Co-authored-by: Paul Sanders <[email protected]>
* Changed Debian base image * Update the python version to be the same as used in the fidesops app image Co-authored-by: Paul Sanders <[email protected]> * Updated changelog Co-authored-by: Dave Quinlan <[email protected]> Co-authored-by: Paul Sanders <[email protected]>
* Change docker password environment variable to MSSQL_SA_PASSWORD * Revert password environment variable name * Set MSSQL user to root in docker-compose * Revert setting root user in docker-compose * Change environment variable name from REQUIRE_MSSQL to SKIP_MSSQL_INSTALLATION * Add healthcheck to mssql compose file * Modify healthcheck command * Revert healthcheck * Try for more loging information * Another try for logging * Another try for logging * Try running only mssql tests to avoid timeout * Revert mssql only flag * Extend time out to try to get logs * Revert extra logging and extended timeout * Set network mode to host * Make mssql run on its own * Remove network from docker-compose and only run mssql in integration * Increase integration test logging * Revert mssql only * Use cache for docker * Fix workflow error * Fix workflow error * Fix target * Fix make traget * Fix make traget * Revert cache * Verify that mssql is running from pytest fixture * Add restart to mssql * Revert rester in docker-compose * Revert wait for mssql in pytest fixture * Wait for mssql to be ready before adding test data * Add count of retries * Update CHANGELOG Co-authored-by: Paul Sanders <[email protected]>
We've defined generic issue templates for all Ethyca repos here: https://github.com/ethyca/.github/tree/main/.github/ISSUE_TEMPLATE Removing the templates from this repo allows it to automatically pull in the organization templates for consistency.
Bumps [sqlalchemy-redshift](https://github.com/sqlalchemy-redshift/sqlalchemy-redshift) from 0.8.8 to 0.8.10. - [Release notes](https://github.com/sqlalchemy-redshift/sqlalchemy-redshift/releases) - [Changelog](https://github.com/sqlalchemy-redshift/sqlalchemy-redshift/blob/main/CHANGES.rst) - [Commits](sqlalchemy-redshift/sqlalchemy-redshift@0.8.8...0.8.10) --- updated-dependencies: - dependency-name: sqlalchemy-redshift dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Create __init__.py * move all of the test files down one dir into `ops` * update test paths * update paths where needed * update the changelog
* Add release dispatch event action * Update `CHANGELOG.md`
* Reduce number of open connections: - Limit task concurrency to two per worker. - Create one Engine per celery process which opens up a connection pool. Create one Session per celery process and use that session across privacy requests. - Close the session after the privacy request has finished executing. This just resets the session and returns connections back to the pool. It can be reused. - Remove unnecessary places where session is closed manually because the session is being used as a context manager and is already closed through that. - Pass the same Session that the privacy request is using through to TaskResources to be re-used to create ExecutionLogs instead of opening up a new Session. - Don't close the session when passing it into the Execution Log, wait until the entire privacy request is complete/exited. * Define "self" for run_privacy_task - it's the task itself. For mypy's benefits, define that the session is a context manager. * Make a session non-optional for graph_task.run_access_request, graph_task.run_erasure, and for instantiating taskResources * Use missing db fixture. * Add missing db resource. * Update test to reflect new behavior that disabling a datasource while a request is in progress can cause related collections to be skipped once the current session is expired and the connection config has the most recent state. Because the same Session that is being used to run the PrivacyRequest is now being used for ExecutionLogs, the process of saving an ExecutionLog runs a session.commit() which expires the Session and causes the ConnectionConfig to have the most recent state the next time it is accessed. * Update CHANGELOG.
Bumps [gitpython](https://github.com/gitpython-developers/GitPython) from 3.1 to 3.1.27. - [Release notes](https://github.com/gitpython-developers/GitPython/releases) - [Changelog](https://github.com/gitpython-developers/GitPython/blob/main/CHANGES) - [Commits](gitpython-developers/GitPython@3.1.0...3.1.27) --- updated-dependencies: - dependency-name: gitpython dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Update dev_nox.py * spin up the app before running a container shell
* Correctly `isort` `versioneer` in files * Make `optional-requirements.txt` a source of truth * Update `CHANGELOG.md` * Include `optional_requirements.txt` in the package * Specify an encoding when opening requirements files Quiets the pylint warning * Remove superfluous variables * Match dependency names more strictly, add docstring
…res (#1178) * Delete fidesctl user routes in favor of fidesops user routes. They are practically copies of each other except fidesops user logout route has improved logic such as allowing the user to logout with a malformed or expired token. The fidesctl route was taking priority. * Use new CONFIG variable in test_saas_queryconfig * Adjust fidesops health endpoint test. We're using the fidesctl health endpoint now and they surface a version number while fidesops did not. * Add missing CONFIG variable in existing execution tests. Fix incorrect timescale test name. * Add missing timescale secrets to integration_test_config.toml.
…st mode" is True. (#1162)
* 1191-Duplicate key in fides admin ui package.json file * Updated CHANGELOG.md file
* Stop creating new engines as part of the process of running health checks which cause us to have too many idle connections opened against the application database. Use the same engine that is being shared across the "ops" API endpoints. - Remove unused get_db_for_health_check * Remove the ctl get_db used in the endpoints in favor of the ops get_db. * Mock get_db_health needs two parameters.
* Re-add SaaS template update script to startup hook Also update dependency getter that returns the shared db session directly to be importable from outside classes. * Use a context manager for db session Co-authored-by: Adam Sachs <[email protected]>
* Bump pre-commit from 2.9.3 to 2.20.0 (#853) (#1373) Bumps [pre-commit](https://github.com/pre-commit/pre-commit) from 2.9.3 to 2.20.0. - [Release notes](https://github.com/pre-commit/pre-commit/releases) - [Changelog](https://github.com/pre-commit/pre-commit/blob/main/CHANGELOG.md) - [Commits](pre-commit/pre-commit@v2.9.3...v2.20.0) --- updated-dependencies: - dependency-name: pre-commit dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * 1016 frontend ability for users to manually enter pii to an in progress subject request (#1377) * On successful connector creation the empty yaml text input displays a validation error * Updated caniuse-lite NPM dependency * Added min/max character limitation validation for Manual Webhook DSR Customization * Prevented Chakra Divider component opacity from being overridden at runtime. User could not see visible horizontal line on UI before. * Updated the Manual Processing detail form to not be submitted until the form is dirty. * Removed form dirty check on ManualProcessingDetail component * Update URL when user is navigating to DSR Customization screen when creating a Manual Webhook * Enable retries on saas connectors for failures at the http request level (#1376) * Add decorator to send method that retries throttles * Fix black/mypy * Fix pylint * Add tests for authenticated client * Small fixes and typos * Update CHANGELOG.md * Small changes. Update retry logic to not retry general exceptions Co-authored-by: Eduardo Armendariz <[email protected]> * Add Consent Request API (#1387) Co-authored-by: Paul Sanders <[email protected]> * Add new template for consent requets emails (#1405) * Add new template for consent requets emails * Remove analytics id * Update subject Co-authored-by: Paul Sanders <[email protected]> * Add authenticated route to get consent preferences (#1402) Co-authored-by: Paul Sanders <[email protected]> * Backend: Updating or Deleting Access Webhooks [#1388][#1389] (#1394) * If a manual webhook is deleted or disabled, check if there are any remaining active manual webhooks configured. If not, queue any Privacy Requests stuck in "requires_input" for processing. * In the "view_uploaded_manual_webhook_data", load cached webhook data for a privacy request in strict mode. If it fails (no data saved, extra field saved, field missing), return checked=True, so the user knows they need to reupload data for this webhook before it can be submitted. Return the data in non-strict mode, so we just show the overlap between the data saved and the fields defined. * Update changelog. * Move queue_requires_input_requests to the connection_endpoints where this is the only module it's being called - both where you update and delete a connection config. * Clarify docstring. * Braze Connector: Access Endpoints (#1248) * [#1393] Update Fidesops config with sane defaults where necessary (#1395) * add sane defaults * make subsections of config with complete defaults optional * lowercase database.enabled, set defaults for optional configs * update return type * updates changelog * make PORT an env var * cast env var to int * remove unnecessary unpinned dependency * bump fideslib version * bump fideslib to 3.1.4 * add defaults for the non optional config subclasses * set empty dict to default for config subclasses that require some fields * use .get() in assemble URL for correct error message, correct comment * update jwt_key type annotation * Explain Privacy Request Execution [#1397] (#1396) Co-authored-by: Cole <[email protected]> * Update docker command on privacy center step 4 (#1410) removing the typo ` . at the end of the command so that the command works. * 1319 consent UI api integration (#1407) * Add consent UI back in * Finish initial integration with consent api * WIP consent page * Get initial consent updating working * Improve button look and feel * Add untracked files * Format VerificationForm.tsx * Remove comments * Rename Privacy modal variables * Rename variable * 1401 admin UI persist redux store to localstorage (#1409) * 1401 - Admin UI: Persist Redux store to localStorage * Resolved React memory leak when user attempts to logout via the Subject Requests landing page * Resolved UI unit test failure * Rollback previous change * Removed blacklist property from Redux store config * Refactored due to recommended code review feedback * update footer links (#1406) * update footer links * changelog * Update CHANGELOG.md Co-authored-by: Paul Sanders <[email protected]> Co-authored-by: Sean Preston <[email protected]> Co-authored-by: Paul Sanders <[email protected]> * firebase auth integration (#1399) * Initial implementation of firebase auth connector Introduces firebase_admin sdk as a fidesops dependency Leverages request overrides to define custom functionality * Update changelog * Minimize unused client config in firebase auth config * Add user.photo_url as a supported Firebase Auth field Improve readability in access implementation * Fix field name to be more consistent with python SDK properties * Add clarifying comment to unused function docstring * Add photo_url field to firebase auth dataset * Properly test provider data in firebase auth integration. Also include test coverage for delete function, even thought it's not being invoked by default saas config. * Only add fields to result if popluated * Fix empty attribute logic within provider data block * Ignore pylint error for too many variables * fix up provider data check Co-authored-by: Adam Sachs <[email protected]> * Add Braze connector registry entry (#1418) * Braze SVG added * Fixing change log message Co-authored-by: Adrian Galvan <[email protected]> Co-authored-by: Adam Sachs <[email protected]> * Removing PII fields from conversations collection (#1386) * Fidesops v1.8.1 Release Checklist (#1420) * updates changelog * add UI testing steps to release checklist * DX Improvements from `v1.8.1` release (#1421) * stop swallowing integrity errors on dataset update * dont attach logging middleware if analytics opt_out is true * handle diff correctly * disable consider-using-f-string as we need this for logging statements to work effectively in fideslog * Updated CHANGELOG.md file from prior merge * get the server up and running * fix test collection * fix isort and black * update dataset annotations * Merge latest fidesops UI into unified ui * Fixing post UI merge issues * Fix formatting and import issues * get all unit tests passing * Fix cypress failure 🤞 * Remove auth.slice.test.ts Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Eduardo Armendariz <[email protected]> Co-authored-by: Eduardo Armendariz <[email protected]> Co-authored-by: Paul Sanders <[email protected]> Co-authored-by: Paul Sanders <[email protected]> Co-authored-by: Dawn Pattison <[email protected]> Co-authored-by: Noonari <[email protected]> Co-authored-by: Sean Preston <[email protected]> Co-authored-by: Cole <[email protected]> Co-authored-by: shawnplusplus <[email protected]> Co-authored-by: Andrew Jackson <[email protected]> Co-authored-by: Cole Isaac <[email protected]> Co-authored-by: Adam Sachs <[email protected]> Co-authored-by: Adam Sachs <[email protected]> Co-authored-by: Adrian Galvan <[email protected]> Co-authored-by: Thomas <[email protected]>
* Remove SaaS type enum and dynamically reference registered types Update tests to dynamically compare results rather than looking for static values. * Remove references to specific custom SaaSType Since we no longer have an enum for SaaSType, we don't need to have a specific 'custom' type - instead, users can simply create their own type dynamically. There is also now no more invalid 'type' value for saas configs, so we remove the test for that functionality. * Remove unused import * update changelog Co-authored-by: Adam Sachs <[email protected]>
* Rate limiter implementation copied from fidesops repo * Change enum values to not be plural * Add test using rate limiter from multiple threads * Update function names and test parameters * Update error messages to include better details * Update changelog * Add an additional comment to test * fix typo in test name * Fix typo in test comment * Fix one more typo * Run black one more time Co-authored-by: Eduardo Armendariz <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Code Changes
Steps to Confirm
Pre-Merge Checklist
CHANGELOG.mdDescription Of Changes
This PR covers another merge of
mainintounified-fides-2, and then ofunified-fides-2intomain