Add documentation for new nox commands (#981)

* Add documentation for new nox commands

* changelog

* missed make to nox edits

* review edits

* update `nox -s dev` to not open a shell, add it as a posarg option

* Apply suggestions from code review

* remove the analytics id

* update typo

* remove extra numbering

Co-authored-by: Thomas <[email protected]>

CHANGELOG.md

@@ -55,6 +55,7 @@ The types of changes are:
 ### Docs
 
 * Added zendesk and salesforce connection pages [#908](https://github.com/ethyca/fidesops/pull/908)
+* Added nox command documentation [#981](https://github.com/ethyca/fidesops/pull/981)
 
 ### Fixed
 

README.md

@@ -16,18 +16,20 @@ _A part of the [greater Fides ecosystem](https://github.com/ethyca/fides)._
 ## :rocket: Quick Start
 If you're looking for a more detailed introduction to fidesops, we recommend following [our tutorial here](https://ethyca.github.io/fidesops/tutorial/). 
 
-Run the quickstart in your terminal to give fidesops a test drive:
+### System Requirements
 
-```
-Install Docker: https://docs.docker.com/desktop/#download-and-install
-Install Make: brew install make
-```
+- [Docker](https://www.docker.com/products/docker-desktop) (20.10.8+)
+- [Python](https://www.python.org/downloads/) (3.8+)
+- [Nox](https://nox.thea.codes/en/stable/) (`pip install nox`)
+
+Run the quickstart in your terminal to give fidesops a test drive:
 
 ```bash
 git clone https://github.com/ethyca/fidesops.git
 cd fidesops
-make quickstart
+nox -s quickstart
 ```
+
 This runs fidesops in docker along with the necessary data stores.  It also spins up a test postgres
 database and a test mongo database to mimic your application.  This quickstart will walk you through executing privacy
 requests against your system by making a series of API requests to fidesops.

clients/ops/admin-ui/README.md

@@ -4,13 +4,13 @@ Admin UI for managing FidesOps privacy requests. A web application built in Next
 
 ## Running Locally
 
-1. Run `make server` in top-level `fidesops` directory, then run `make user` and follow prompts to create a user. Note that password requires 8 or more characters, upper and lowercase chars, a number, and a symbol.
+1. Run `nox -s create_user` and follow prompts to create a user. Note that password requires 8 or more characters, upper and lowercase chars, a number, and a symbol.
 2. In a new shell, `cd` into `clients/ops/admin-ui`, then run `npm run dev`.
 3. Nav to `http://localhost:3000/` and logged in using created user. The `email` field is simply the `user` that was created, not a valid email address.
 
 ## Testing Entire Request Flow
 
-1. Run the `fidesops` server with `make server`.
+1. Run the `fidesops` server with `nox -s dev`.
 2. Create a policy key through the API (using the fidesops Postman collection).
 3. Configure the `clients/privacy-center` application to use that policy by adding it to the appropriate request config in `config/config.json`.
 4. Run the Privacy Request center using `npm run dev`.

docs/README.md

@@ -1,9 +1,9 @@
 # Fidesops Docs Site
 # Fidesops Docs
 
-This is the docs site for Fidesops. To run the docs locally, use `make docs-serve` and visit `localhost:8000`
+This is the docs site for Fidesops. To run the docs locally, use `nox -s docs_serve` and visit `localhost:8000`
 
 
 1. `pip install -r requirements.txt`
 1. `cd fidesops/`
-1. `make docs-serve`
+1. `nox -s docs_serve`

docs/fidesops/docs/development/contributing_details.md

@@ -78,19 +78,19 @@ db.refresh()
 ```
 
 ### Connecting to the database
-When you run `make server` or `make server-shell`, the database will be spun up in a Docker container with port `5432` exposed on localhost. You can connect to it using the credentials found in `.fidesops.toml`, e.g.
+When you run `nox -s dev`, the database will be spun up in a Docker container with port `5432` exposed on localhost. You can connect to it using the credentials found in `.fidesops.toml`, e.g.
 
 - Hostname: `localhost`
 - Port: `5432`
-- Username: see `database.USER` in `.fidesops.toml`
-- Password: see `database.PASSWORD` in `.fidesops.toml`
+- Username: see `database.user` in `.fidesops.toml`
+- Password: see `database.password` in `.fidesops.toml`
 
 
 ### Alembic migrations
 
 Some common Alembic commands are listed below. For a comprehensive guide see: https://alembic.sqlalchemy.org/en/latest/tutorial.html. 
 
-The commands will need to be run inside a shell on your Docker containers, which can be opened with `make server-shell`.
+The commands will need to be run inside a shell on your Docker containers, which can be opened with `nox -s dev -- shell`.
 
 In the `/src/fidesops` directory:
 
@@ -129,6 +129,6 @@ Occasionally when developing you'll run into issues where it's beneficial to rem
 - Stop all running containers: `docker-compose down`
 - Delete all local containers: `docker rm -f $(docker ps -a -q)`
 - Delete all local Docker volumes: `docker volume rm $(docker volume ls -q)`
-- Remove temp. files, installed dependencies, all local Docker containers and all local Docker volumes: `make clean`
+- Remove temp. files, installed dependencies, all local Docker containers and all local Docker volumes: `nox -s clean`
 - Delete all stopped containers, all networks not used by a container, all dangling images, and all build cache: `docker system prune`
-- Recreate the project: `make compose-build`
+- Recreate the project: `nox -s "build(dev)"`

docs/fidesops/docs/development/documentation.md

@@ -27,7 +27,7 @@ Documentation (including both concepts and API references) is built and deployed
 If you're using VS Code Dev Containers, the docs will automatically be available at `localhost:8000`, otherwise you'll need to run the following command:
 
 ```bash
-make docs-serve
+nox -s docs_serve
 ```
 
 You'll see a status update as the docs build, and then an announcement that they are available on `http://127.0.0.1:8000`.

docs/fidesops/docs/development/overview.md

@@ -14,42 +14,17 @@ commands to give you different functionality.
 ### System Requirements 
 
 1. Install Docker: https://docs.docker.com/desktop/#download-and-install
-2. Install `make` locally (see Make on Homebrew (Mac), Make for Windows, or your preferred installation) 
-   1. `brew install make`
-3. [Create a fork of fidesops](https://docs.github.com/en/get-started/quickstart/fork-a-repo) 
-4. Clone your fork `git clone https://github.com/<your-fork-location>/fidesops.git`
-5. `cd fidesops`
-
-### Available make commands
-- `make server` - this spins up the Fastapi server and supporting resources (a Postgres database and a Redis cache), which you can visit at `http://0.0.0.0:8080`. Check out the docs at `http://0.0.0.0:8000/fidesops/`
-- `make integration-env` - spins up everything in make server, plus additional postgres, mongo, and mysql databases for you to execute privacy requests against.
-    - Try this out locally with our [fidesops Postman collection](../postman/Fidesops.postman_collection.json)
-    - `make integration-env datastores="mysql mariadb"`- brings up only mysql and mariadb datastores 
-- `make integration-shell` - spins up everything in make server, plus all Docker Compose specified datastores (Postgres, MySQL, MSSQL, Mongodb) and opens a server shell. This is most useful for running `pytest -k ...` commands within the integration shell to test connected datastores.
-- `make black`, `make mypy`, and `make pylint` - auto-formats code
-- `make check-all` - runs the CI checks locally and verifies that your code meets project standards
-- `make server-shell`-  opens a shell on the Docker container, from here you can run useful commands like:
-    - `ipython` - open a Python shell
-    - `cd src` then `alembic revision --autogenerate -m "adds enum for mysql connection type"` - auto-generates DB migration
-- `make pytest` - runs all unit tests except those that talk to integration databases
-- `make pytest-integration` - runs all integration tests, except those on external datastores.
-- `make pytest-integration datastores="postgres snowflake mssql"` - runs access integration tests for the Postgres, Snowflake and MSSQL environments. NB. This can be used to run integration tests on external datastores if those datastores are explicitly specified.
-- `make pytest-integration-external` - runs only external integration tests.
-- `make pytest-integration-erasure` - runs erasure integration tests.
-- `make reset-db` - tears down the fidesops postgres db, then recreates and re-runs migrations.
-- `make quickstart` - runs a quick, five minute quickstart that talks to the fidesops API to execute privacy requests
-- `make check-migrations` - verifies there are no un-run migrations 
-- `make docs-serve` - spins up just the docs, which you can visit at `http://0.0.0.0:8000/fidesops/`
-
-See [Makefile](https://github.com/ethyca/fidesops/blob/main/Makefile) for more options. 
-
+2. [Create a fork of fidesops](https://docs.github.com/en/get-started/quickstart/fork-a-repo) 
+3. Clone your fork `git clone https://github.com/<your-fork-location>/fidesops.git`
+4. `cd fidesops`
 
+### Available `nox` commands
+An up-to-date list of build commands is available by running `nox` from within the project directory.
 #### Issues 
 
 - MSSQL: Known issues around connecting to MSSQL exist today for Apple M1 users. M1 users that wish to install `pyodbc` locally, please reference the workaround [here](https://github.com/mkleehammer/pyodbc/issues/846).
 
-- Package not found: When running `make server`, if you get a `importlib.metadata.PackageNotFoundError: fidesops`, do `make server-shell`,
-and then run `pip install -e .`. Verify fidesops is installed with `pip list`.
+- Package not found: When running `nox -s dev`, if you get a `importlib.metadata.PackageNotFoundError: fidesops`, do `nox -s dev -- shell`, and then run `pip install -e .`. Verify fidesops is installed with `pip list`.
 
 
 ## Write your code

docs/fidesops/docs/development/testing.md

@@ -51,7 +51,7 @@ Fidesops has a few [`pytest` fixtures](https://docs.pytest.org/en/stable/fixture
 
 Fidesops uses `pytest` for unit testing. As with other `make` commands, you have the option to run `pytest` in command-line or in application shell:
 
-1. In shell: Once in the fidesops container shell (`make server-shell`, or `make integration-shell` if running integration tests), invoke `pytest` from the root fidesops directory:
+1. In shell: Enter the fidesops container shell using `nox -s dev -- shell`, or pass in the datastores you want to start for integration testing (i.e., `nox -s dev -- shell mssql mariadb`). You can then invoke `pytest` from the root fidesops directory:
 
 ```bash
 cd fidesops
@@ -61,7 +61,7 @@ pytest
 2. From regular command-line:
 
 ```bash
-make pytest
+nox -s pytest_unit
 ```
 
 ### Running specific tests
@@ -75,13 +75,10 @@ pytest tests/ops/integration/ -k api
 
 Other commands you may need are listed below. The full documentation can be found at: <https://docs.pytest.org/en/6.2.x/>.
 
-- Run all tests in a directory: `make pytestpath=path/to/dir/ pytest`
-- Run all tests in a file: `make pytestpath=path/to/file.py pytest`
-- Run all tests within a class within a file: `make pytestpath=path/to/file.py::ClassName pytest`
-- Run a specific test within a class within a file: `make pytestpath=path/to/file.py::ClassName::method_name pytest`
-- Run a specific test within a file: `make pytestpath=path/to/file.py::method_name pytest`
-- Run integration tests (access): `make pytest-integration`
-- Run integration tests (erasure): `make pytest-integration-erasure`
+- Run all unit tests, except those that talk to integration databases - `nox -s pytest_unit`
+- Run all integration tests, except those on external datastores - `nox -s pytest_integration`
+- Run all tests that rely on third-party databases and services - `nox -s pytest_integration_external`
+- Run all SaaS tests that rely on third-party databases and services - `nox -s pytest_saas`
 
 #### Debugging
 

docs/fidesops/docs/postman/using_postman.md

@@ -24,8 +24,8 @@ A minimal Postman collection is included to assist in setting up your fidesops c
       ![Add root client id and secret](../img/postman_images/add_root_client_id_and_secret.png)
 
 ## Bring up local servers and mock databases
-1. Run `make integration-env` in your terminal.
-      - This brings up the `fidesops` server, `redis`, the `fidesops` postgres database, and some mock external databases like `mongodb_example` and `postgres_example`. These mock databases are pre-populated with test data to represent your datastores.
+1. Run `nox -s dev -- <datastore>` in your terminal. 
+      - This brings up the `fidesops` server and the list of datastores specified, i.e. `nox -s dev -- postgres mongodb`. These mock datastores are pre-populated with test data to represent your datastores.
 
 !!! Note ""
       The following list of requests is kept in the `Minimum API calls to create an Access Privacy Request` folder. Some of the returned data will need to be saved as additional variables for use in other steps.

docs/fidesops/docs/saas_connectors/example_configs/stripe.md

@@ -23,7 +23,7 @@ Fidesops uses the following Stripe endpoints to retrieve and delete Personally I
 ## Connection Settings
 Fidesops provides a [Postman collection](../../postman/using_postman.md) for easily establishing connections to your third party applications. Additional connection instructions may be found in the [configuration guide](../saas_config.md).
 
-**Deletion requests** are fulfilled by masking PII via `UPDATE` endpoints. To [give fidesops permission](../../guides/configuration_reference.md#configuration-variable-reference) to remove PII using `DELETE` endpoints, ensure the `MASKING_STRICT` variable in your `fidesops.toml` file is set to `FALSE`. 
+**Deletion requests** are fulfilled by masking PII via `UPDATE` endpoints. To [give fidesops permission](../../guides/configuration_reference.md#configuration-variable-reference) to remove PII using `DELETE` endpoints, ensure the `masking_strict` variable in your `fidesops.toml` file is set to `FALSE`. 
 
 ## Example Stripe Configuration
 ```yaml

docs/fidesops/docs/tutorial/execute_privacy_request.md

@@ -116,7 +116,7 @@ moments for the file to appear):
 ```
 
 ## Issues?
-- Is `make server` running?
+- Is `nox -s dev` running?
 - [Reference the full script here](https://github.com/ethyca/fidesdemo/blob/main/flaskr/fidesops.py) for pieces you may be missing.
     - This script has more detailed logging and error handling.
 - [Make sure your dataset is annotated properly](https://github.com/ethyca/fidesdemo/blob/main/fides_resources/flaskr_postgres_dataset.yml)

docs/fidesops/docs/ui/local_dev.md

@@ -5,7 +5,7 @@ To test the UI locally, clone the [FidesOps repository](https://github.com/ethyc
 
 ### Creating the root user
 
-In the top-level `fidesops` directory, run `make user`.
+In the top-level `fidesops` directory, run `nox -s create_user`.
 
 A series of prompts will walk you through creating a username and password. Passwords require 8 or more characters, upper and lowercase characters, a number, and a symbol.
 
@@ -37,7 +37,7 @@ echo NEXTAUTH_SECRET=`openssl rand -base64 32` >> .env.local
 
 ## Backend deployment
 
-Fidesops automatically serves a version of the UI when running `make server`.
+Fidesops automatically serves a version of the UI when running `nox -s dev`.
 
 To deploy a full version of the UI from a backend, run the following from the root fidesops directory:
 
@@ -49,4 +49,4 @@ To deploy a full version of the UI from a backend, run the following from the ro
 
 This will build and place the Admin UI files into a location accessible by backend fidesops deployments.
 
-To test the UI, run `make server` from the root directory, and visit `http://0.0.0.0:8080/static/index.html`.
+To test the UI, run `nox -s dev` from the root directory, and visit `http://0.0.0.0:8080/index.html`.

noxfiles/dev_nox.py

@@ -2,22 +2,29 @@
 import nox
 from constants_nox import ANALYTICS_OPT_OUT, COMPOSE_SERVICE_NAME, RUN, START_APP
 from docker_nox import build
-from run_infrastructure import run_infrastructure
+from run_infrastructure import ALL_DATASTORES, run_infrastructure
 
 
 @nox.session()
 def dev(session: nox.Session) -> None:
-    """Spin up the entire application and open a development shell."""
+    """Spin up the application. Uses positional arguments for additional features."""
     build(session, "dev")
     session.notify("teardown")
-    datastores = session.posargs or None
+    datastores = [
+        datastore for datastore in session.posargs if datastore in ALL_DATASTORES
+    ] or None
+    open_shell = "shell" in session.posargs
     if not datastores:
-        # Run the webserver without integrations
-        session.run(*START_APP, external=True)
-        session.run(*RUN, "/bin/bash", external=True)
+        if open_shell:
+            session.run(*START_APP, external=True)
+            session.run(*RUN, "/bin/bash", external=True)
+        else:
+            session.run("docker-compose", "up", COMPOSE_SERVICE_NAME, external=True)
     else:
         # Run the webserver with additional datastores
-        run_infrastructure(open_shell=True, datastores=datastores)
+        run_infrastructure(
+            open_shell=open_shell, run_application=True, datastores=datastores
+        )
 
 
 @nox.session()

noxfiles/run_infrastructure.py

@@ -23,6 +23,7 @@
     "bigquery": ["BIGQUERY_KEYFILE_CREDS", "BIGQUERY_DATASET"],
 }
 EXTERNAL_DATASTORES = list(EXTERNAL_DATASTORE_CONFIG.keys())
+ALL_DATASTORES = DOCKERFILE_DATASTORES + EXTERNAL_DATASTORES
 IMAGE_NAME = "webserver"