Skip to content
This repository has been archived by the owner on Nov 30, 2022. It is now read-only.

Move root-level docker files into docker/ subdir #877

Merged
merged 19 commits into from
Jul 16, 2022
Merged
Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
24fe2a7
Move root-level docker files into docker/ subdir
ThomasLaPiana Jul 14, 2022
94fde51
move all of the compose files
ThomasLaPiana Jul 14, 2022
8f8fe2f
remove root aux compose files, refactor away the no-db and worker com…
ThomasLaPiana Jul 14, 2022
2e2ec50
unify the app and worker dockerfiles into a single file that leverage…
ThomasLaPiana Jul 14, 2022
f634ef9
move python scripts into a subdir, fix paths in compose integration f…
ThomasLaPiana Jul 14, 2022
9772aba
fix the script tests
ThomasLaPiana Jul 14, 2022
ad8426f
use python sleep instead of system sleep, fix script paths
ThomasLaPiana Jul 14, 2022
20f0913
remove the analytics_id that accicentally got committed
ThomasLaPiana Jul 14, 2022
014ae7e
updated changelog
ThomasLaPiana Jul 14, 2022
6128026
move the sample sql data to a subdir of docker/ so it can be mounted
ThomasLaPiana Jul 15, 2022
a79cbe9
update the teardown command and fix the integration files
ThomasLaPiana Jul 15, 2022
251d1ee
fix more path typos
ThomasLaPiana Jul 15, 2022
3502c7b
more desperate tweaks
ThomasLaPiana Jul 15, 2022
312002c
fix mysql/mariadb/mongo tests
ThomasLaPiana Jul 15, 2022
68132eb
added an additional build step if mssql not there, all tests passing
ThomasLaPiana Jul 15, 2022
083c080
fix an accidental lowercasing
ThomasLaPiana Jul 15, 2022
8d6e88f
Apply suggestions from code review
ThomasLaPiana Jul 15, 2022
004fdd5
Update CHANGELOG.md
ThomasLaPiana Jul 15, 2022
f1de632
Merge remote-tracking branch 'origin/main' into ThomasLaPiana-cleanup…
Jul 16, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 33 additions & 16 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,10 @@ The types of changes are:
* `Fixed` for any bug fixes.
* `Security` in case of vulnerabilities.


## [Unreleased](https://github.com/ethyca/fidesops/compare/1.6.1...main)

### Added

* [Admin UI] Update Subject Request status filter to be a multiselect dropdown [#513](https://github.com/ethyca/fidesops/pull/764)
* Add support for multiple statuses to be selected for filtering subject requests [#660](https://github.com/ethyca/fidesops/pull/802)
* [Datastore Management] Include Icon with datastore connection (FE)[#671](https://github.com/ethyca/fidesops/pull/845)
Expand All @@ -30,54 +31,63 @@ The types of changes are:
* Adds endpoint to get the secrets required for different connectors [#795](https://github.com/ethyca/fidesops/pull/795)

### Changed

* Update clipboard icon in Admin UI [#838](https://github.com/ethyca/fidesops/pull/838)
* Stop masking uvicorn logs by default [#831](https://github.com/ethyca/fidesops/pull/831)
* Bump fideslib to handle base64 encoded password [#820](https://github.com/ethyca/fidesops/pull/820)
* Return identity data from application DB, instead of cache [#860](https://github.com/ethyca/fidesops/pull/860)
* Update admin ui to be served from the root route `/` [#720](https://github.com/ethyca/fidesops/pull/720)
* Refactored the root directory and various Docker-related files [#877](https://github.com/ethyca/fidesops/pull/877)

### Developer Experience

* Replace user authentication routes with fideslib routes [#811](https://github.com/ethyca/fidesops/pull/811)
* Reduce docker image size [846](https://github.com/ethyca/fidesops/pull/846)

### Docs

* Backend UI deployment [#827](https://github.com/ethyca/fidesops/pull/827)
* Fix publish_docs CI action [#818](https://github.com/ethyca/fidesops/pull/818)
* Reorganize docs and standardize formatting [#858](https://github.com/ethyca/fidesops/pull/858)
* Changed wording on Admin UI login page [#774](https://github.com/ethyca/fidesops/pull/774)
* Fixed typos in Admin UI [#774](https://github.com/ethyca/fidesops/pull/774)

### Fixed

* Resolve issue with MyPy seeing files in fidesops as missing imports [#719](https://github.com/ethyca/fidesops/pull/719)
* Fixed `check-migrations` Make command [#806](https://github.com/ethyca/fidesops/pull/806)
* Fix issue requiring separate install of snowflake-connector-python [#807](https://github.com/ethyca/fidesops/pull/807)
* [User Management] Create new user gives HTTP 422 Unprocessable Entity exception [#832] (https://github.com/ethyca/fidesops/pull/833)
* [User Management] Create new user gives HTTP 422 Unprocessable Entity exception [#832] (<https://github.com/ethyca/fidesops/pull/833>)
ThomasLaPiana marked this conversation as resolved.
Show resolved Hide resolved
* Fix error when running `make test-data` [862](https://github.com/ethyca/fidesops/pull/862)
* [User Management] Refactored New and Edit user pages to reduce duplicate code [#839]https://github.com/ethyca/fidesops/pull/839
* [User Management] Refactored New and Edit user pages to reduce duplicate code [#839]<https://github.com/ethyca/fidesops/pull/839>
ThomasLaPiana marked this conversation as resolved.
Show resolved Hide resolved
* Fix error when there are no scopes in `ClientDetail` [#830](https://github.com/ethyca/fidesops/pull/830)


## [1.6.1](https://github.com/ethyca/fidesops/compare/1.6.0...1.6.1)

### Added

* `fidesops worker` command for running a Celery worker [#673](https://github.com/ethyca/fidesops/pull/673/)

### Developer Experience

* Add fixture to clear tables between test [#680](https://github.com/ethyca/fidesops/pull/680)
* Reduce the size of the docker image [#707](https://github.com/ethyca/fidesops/pull/707)
* Parallelize CI safe checks to reduce run time [#717](https://github.com/ethyca/fidesops/pull/717)
* Add dependabot to keep dependencies up to date [#718](https://github.com/ethyca/fidesops/pull/718)
* Make running a worker node optional [#770](https://github.com/ethyca/fidesops/pull/770)

### Changed

* Base64 encode passwords on frontend [#749](https://github.com/ethyca/fidesops/pull/749)

### Docs

* Updated the tutorial installation to use main in fidesdemo [#715](https://github.com/ethyca/fidesops/pull/715)
* Added a page on how to use the datastore UI [#742](https://github.com/ethyca/fidesops/pull/742)
* Added a page on implementing and opting out of fideslog analytics [#751](https://github.com/ethyca/fidesops/pull/751)

### Fixed

* Make reading of environment variables case insensitive [#712](https://github.com/ethyca/fidesops/pull/712)
* Fix console warning in disable connection modal [#750](https://github.com/ethyca/fidesops/pull/750)
* Fix no such container error with docker-compose [#758](https://github.com/ethyca/fidesops/pull/758)
Expand All @@ -86,6 +96,7 @@ The types of changes are:
## [1.6.0](https://github.com/ethyca/fidesops/compare/1.5.3...1.6.0)

### Added

* Subject Request Details page [#563](https://github.com/ethyca/fidesops/pull/563)
* Restart Graph from Failure [#578](https://github.com/ethyca/fidesops/pull/578)
* Redis SSL Support [#611](https://github.com/ethyca/fidesops/pull/611)
Expand All @@ -105,8 +116,8 @@ The types of changes are:
* Sample dataset and access configuration for Zendesk (ticket endpoints) [#677](https://github.com/ethyca/fidesops/pull/677)
* Include number of records to be masked in masking endpoint's log message [#692](https://github.com/ethyca/fidesops/pull/692)
* Datastore Connection Landing Page [#674](https://github.com/ethyca/fidesops/pull/674)
* Added the ability to delete a datastore from the frontend [#683] https://github.com/ethyca/fidesops/pull/683
* Added the ability to disable/enable a datastore from the frontend [#693] https://github.com/ethyca/fidesops/pull/693
* Added the ability to delete a datastore from the frontend [#683] <https://github.com/ethyca/fidesops/pull/683>
ThomasLaPiana marked this conversation as resolved.
Show resolved Hide resolved
* Added the ability to disable/enable a datastore from the frontend [#693] <https://github.com/ethyca/fidesops/pull/693>
ThomasLaPiana marked this conversation as resolved.
Show resolved Hide resolved
* Adds Postgres and Redis health checks to health endpoint [#690](https://github.com/ethyca/fidesops/pull/690)
* Adds the ability to revoke a pending privacy request [#592](https://github.com/ethyca/fidesops/pull/592/files)
* Added health checks and better error messages on app startup for both db and cache [#686](https://github.com/ethyca/fidesops/pull/686)
Expand All @@ -130,8 +141,11 @@ The types of changes are:
* Use `v2.1.0` of `fideslib` [#705](https://github.com/ethyca/fidesops/pull/705)

### Docs

* Subject Request detail documentation for the UI [#702](https://github.com/ethyca/fidesops/pull/702)

### Fixed

* Fixed error with running mypy on M1 Macs [#630](https://github.com/ethyca/fidesops/pull/630)
* Fixed error with mypy on Python versions greater than 3.9.6 [#630](https://github.com/ethyca/fidesops/pull/630)
* Bumped fideslib to 2.0.4. This fixes the issue where alembic couldn't find the `fidesops.toml` file from its new location [#643](https://github.com/ethyca/fidesops/pull/643)
Expand All @@ -140,18 +154,21 @@ The types of changes are:
## [1.5.3](https://github.com/ethyca/fidesops/compare/1.5.2...1.5.3)

### Changed

* Database migrations now exist as part of the core `fidesops` package [#620](https://github.com/ethyca/fidesops/pull/620)

### Removed
* The `[package]` config section no longer exists [#620](https://github.com/ethyca/fidesops/pull/620)

* The `[package]` config section no longer exists [#620](https://github.com/ethyca/fidesops/pull/620)

### Changed

* Process privacy requests as Celery tasks and not background processes [#621](https://github.com/ethyca/fidesops/pull/621)

## [1.5.2](https://github.com/ethyca/fidesops/compare/1.5.1...1.5.2)

### Added

* Added OAuth2 authentication strategy for SaaS connectors [#555](https://github.com/ethyca/fidesops/pull/555)
* Added `FIDESOPS__SECURITY__LOG_LEVEL` configuration variable to allow controlling the log level [#579](https://github.com/ethyca/fidesops/pull/579)
* Added `DEBUG` logs at startup to view all configuration values [#579](https://github.com/ethyca/fidesops/pull/579)
Expand All @@ -161,22 +178,25 @@ The types of changes are:
* Pause Erasure Request Execution / Resume on Manual Input in [#571](https://github.com/ethyca/fidesops/pull/571/)

### Changed

* Use the `RuleResponse` schema within the `PrivacyRequestReposnse` schema [#580](https://github.com/ethyca/fidesops/pull/580)
* Updated the webserver to use `PORT` config variable from the `fidesops.toml` file [#586](https://github.com/ethyca/fidesops/pull/586)
* Updated `black-ci` makefile command to also check `tests/` directory [#594](https://github.com/ethyca/fidesops/pull/594)

### Developer Experience
* Adds a script for MSSQL schema exploration [#557](https://github.com/ethyca/fidesops/pull/581)

* Adds a script for MSSQL schema exploration [#557](https://github.com/ethyca/fidesops/pull/581)

## [1.5.1](https://github.com/ethyca/fidesops/compare/1.5.0...1.5.1) - 2022-05-27

### Added

* Added `FIDESOPS__DATABASE__ENABLED` and `FIDESOPS__REDIS__ENABLED` configuration variables to allow `fidesops` to run cleanly in a "stateless" mode without any database or redis cache integration [#550](https://github.com/ethyca/fidesops/pull/550)
* Pause Access Request Execution / Resume on Manual Input in [#554](https://github.com/ethyca/fidesops/pull/554)
* A `[package]` section of the `fidesops.toml` configuration file may specify the path to the `fidesops` package itself [#566](https://github.com/ethyca/fidesops/pull/566)

### Changed

* `MaskingStrategyFactory` and associated `MaskingStrategy` implementations now use a decorator-based registration system, to improve extensibility [#560](https://github.com/ethyca/fidesops/pull/560)
* Added `from __future__ import annotations` to `src/fidesops/util/logger.py` to maintain backward compatibility with Python < 3.9 [#569](https://github.com/ethyca/fidesops/pull/569)

Expand Down Expand Up @@ -204,16 +224,14 @@ The types of changes are:
* User creation, update and permissions in the Admin UI [#511](https://github.com/ethyca/fidesops/pull/511)
* Yaml support for dataset upload [#284](https://github.com/ethyca/fidesops/pull/284)


### Breaking Changes
* Update masking API to take multiple input values [#443](https://github.com/ethyca/fidesops/pull/443

* Update masking API to take multiple input values [#443](<https://github.com/ethyca/fidesops/pull/443>
ThomasLaPiana marked this conversation as resolved.
Show resolved Hide resolved

### Docs

* DRP feature documentation [#520](https://github.com/ethyca/fidesops/pull/520)


## [1.4.2](https://github.com/ethyca/fidesops/compare/1.4.1...1.4.2) - 2022-05-12

### Added
Expand All @@ -232,8 +250,8 @@ The types of changes are:
* Publish Fidesops to Pypi [#491](https://github.com/ethyca/fidesops/pull/491)
* DRP data rights endpoint [#526](https://github.com/ethyca/fidesops/pull/526)


### Changed

* Converted HTTP Status Codes to Starlette constant values [#438](https://github.com/ethyca/fidesops/pull/438)
* SaasConnector.send behavior on ignore_errors now returns raw response [#462](https://github.com/ethyca/fidesops/pull/462)
* Seed user permissions in `create_superuser.py` script [#468](https://github.com/ethyca/fidesops/pull/468)
Expand All @@ -243,17 +261,16 @@ The types of changes are:
* Fixing inconsistent SaaS connector integration tests [#473](https://github.com/ethyca/fidesops/pull/473)
* Add user data to login response [#501](https://github.com/ethyca/fidesops/pull/501)


### Breaking Changes
* Update masking API to take multiple input values [#443](https://github.com/ethyca/fidesops/pull/443

* Update masking API to take multiple input values [#443](<https://github.com/ethyca/fidesops/pull/443>
ThomasLaPiana marked this conversation as resolved.
Show resolved Hide resolved

### Docs

* Added issue template for documentation updates [#442](https://github.com/ethyca/fidesops/pull/442)
* Clarify masking updates [#464](https://github.com/ethyca/fidesops/pull/464)
* Added dark mode [#476](https://github.com/ethyca/fidesops/pull/476)


### Fixed

* Removed miradb test warning [#436](https://github.com/ethyca/fidesops/pull/436)
Expand All @@ -262,7 +279,7 @@ The types of changes are:
* Audit imports and references [#479](https://github.com/ethyca/fidesops/pull/479)
* Switch to using update method on PUT permission endpoint [#500](https://github.com/ethyca/fidesops/pull/500)


### Developer Experience

* Add script to seed initial Privacy Request [#487](https://github.com/ethyca/fidesops/pull/487)
* Add first and last name to `createsuperuser` script [#486](https://github.com/ethyca/fidesops/pull/486)
27 changes: 23 additions & 4 deletions Dockerfile.app → Dockerfile
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
##############
## Frontend ##
##############
FROM node:16 as frontend

WORKDIR /fidesops/clients/admin-ui
Expand All @@ -7,9 +10,11 @@ RUN npm install
RUN npm run export


#############
## Backend ##
#############
FROM --platform=linux/amd64 python:3.9.13-slim-buster as backend

ARG SKIP_MSSQL_INSTALLATION

# Install auxiliary software
RUN apt-get update && \
Expand All @@ -27,6 +32,7 @@ RUN apt-get update && \
&& rm -rf /var/lib/apt/lists/*


ARG SKIP_MSSQL_INSTALLATION
RUN echo "ENVIRONMENT VAR: SKIP_MSSQL_INSTALLATION $SKIP_MSSQL_INSTALLATION"

# SQL Server (MS SQL)
Expand Down Expand Up @@ -54,12 +60,25 @@ COPY . /fidesops
WORKDIR /fidesops
RUN pip install -e .

# Enable detection of running within Docker
ENV RUNNING_IN_DOCKER=true

############
## Worker ##
############
FROM backend as worker
CMD [ "fidesops", "worker" ]

#################
## Application ##
#################
## Set the image up to be the application
FROM backend as app

# Make a static files directory
RUN mkdir -p /fidesops/src/fidesops/build/static/

# Copy frontend build over
COPY --from=frontend /fidesops/clients/admin-ui/out/ /fidesops/src/fidesops/build/static/

# Enable detection of running within Docker
ENV RUNNING_IN_DOCKER=true

CMD [ "fidesops", "webserver" ]
53 changes: 0 additions & 53 deletions Dockerfile.worker

This file was deleted.

Loading