Update data-migration documentation (#975)

## Fixes issue
#930

## Description of Changes
Add documentation to make data-migrations a more transparent process.

## Tests and linting
 - [x] This branch is up-to-date with the `develop` branch.
 - [x] `pytest` passes on my local development environment.
 - [x] `pre-commit` passes on my local development environment.

CONTRIB.md

@@ -11,8 +11,8 @@ To submit your changes for review you have to fork the repository, push your new
 Use [pull_request_template.md](/.github/pull_request_template.md) to create the description for your PR! (The template should populate automatically when you go to open the pull request.)
 
 ### Recommended privacy settings
-Whenever you make a commit with `git` the name and email saved locally is stored with that commit and will become part of the public history of the project. This can be an unwanted, for example when using a work computer. We recommond changing the email-settings in the github account at https://github.com/settings/emails and selecting "Keep my email addresses private" as well as "Block command line pushes that expose my email". Also find your github-email address of the form `<id>+<username>@users.noreply.github.com` in that section. Then you can change the email and username stored with your commits by running the following commands
-```
+Whenever you make a commit with `git` the name and email saved locally is stored with that commit and will become part of the public history of the project. This can be an unwanted, for example when using a work computer. We recommend changing the email-settings in the github account at https://github.com/settings/emails and selecting "Keep my email addresses private" as well as "Block command line pushes that expose my email". Also find your github-email address of the form `<id>+<username>@users.noreply.github.com` in that section. Then you can change the email and username stored with your commits by running the following commands
+```shell
 git config user.email "<your-github-email>"
 git config user.name "<your-github-username>"
 ```
@@ -34,17 +34,17 @@ Tests are executed via `make test`. If you're switching between the Docker and V
 
 To hop into the postgres container, you can do the following:
 
-```
-$ docker exec -it openoversight_postgres_1 /bin/bash
+```shell
+$ docker exec -it openoversight-postgres-1 bash
 # psql -d openoversight-dev -U openoversight
 ```
 
 or run `make attach`.
 
 Similarly to hop into the web container:
 
-```
-$ docker exec -it openoversight_web_1 /bin/bash
+```shell
+$ docker exec -it openoversight-web-1 bash
 ```
 
 Once you're done, `make stop` and `make clean` to stop and remove the containers respectively.
@@ -62,15 +62,15 @@ You will need to do these two things for the service account to work as a Gmail
 4. For production, save the email address associated with your service account to a variable named `OO_SERVICE_EMAIL` in a `.env` file in the base directory of this repository. For development and testing, update the `OO_SERVICE_EMAIL` variable in the `docker-compose.yml` file.
 
 Example `.env` variable:
-```bash
+```shell
 OO_SERVICE_EMAIL="[email protected]"
 ```
 
 In addition to needing a service account email, you also need an admin email address so that users have someone to reach out to if an action is taken on their account that needs to be reversed or addressed.
 For production, save the email address associated with your admin account to a variable named `OO_HELP_EMAIL` in a `.env` file in the base directory of this repository. For development and testing, update the `OO_HELP_EMAIL` variable in the `docker-compose.yml` file.
 
 Example `.env` variable:
-```bash
+```shell
 OO_HELP_EMAIL="[email protected]"
 ```
 
@@ -81,7 +81,7 @@ on Amazon Web Services.
 
 Once you have done this, you can put your AWS credentials in the following environmental variables:
 
-```sh
+```shell
 $ export S3_BUCKET_NAME=openoversight-test
 $ export AWS_ACCESS_KEY_ID=testtest
 $ export AWS_SECRET_ACCESS_KEY=testtest
@@ -96,7 +96,7 @@ Running `make dev` will create the database and persist it into your local files
 
 You can access your PostgreSQL development database via psql using:
 
-```sh
+```shell
 psql  -h localhost -d openoversight-dev -U openoversight --password
 ```
 
@@ -108,71 +108,88 @@ or
 `$ python test_data.py --cleanup` to delete the data
 
 ### Migrating the Database
-If you e.g. add a new column or table, you'll need to migrate the database using the Flask CLI. First we need to 'stamp' the current version of the database:
+You'll first have to start the Docker instance for the OpenOversight app using the command `make start`. To do this, you'll need to be in the base folder of the repository (the one that houses the `Makefile`).
+
+```shell
+$ make start
+docker-compose build
+...
+docker-compose up -d
+[+] Running 2/0
+ ✔ Container openoversight-postgres-1  Running                                                                                                                                                             0.0s
+ ✔ Container openoversight-web-1       Running
+```
 
-```sh
-$ cd OpenOversight/  # change directory to source dir
+From here on out, we'll be using the Flask CLI. First we need to 'stamp' the current version of the database:
+
+```shell
+$ docker exec -it openoversight-web-1 bash # 'openoversight-web-1' is the name of the app container seen in the step above
 $ flask db stamp head
+$ flask db migrate -m "[THE NAME OF YOUR MIGRATION]" # NOTE: Slugs are limited to 40 characters and will be truncated after the limit
 ```
 
 (Hint: If you get errors when running `flask` commands, e.g. because of differing Python versions, you may need to run the commands in the docker container by prefacing them as so: `docker exec -it openoversight_web_1 flask db stamp head`)
 
-Next make your changes to the database models in `models.py`. You'll then generate the migrations:
+Next make your changes to the database models in `OpenOversight/app/models/database.py`. You'll then generate the migrations:
 
-```sh
+```shell
 $ flask db migrate
 ```
 
 And then you should inspect/edit the migrations. You can then apply the migrations:
 
-```sh
+```shell
 $ flask db upgrade
 ```
 
-You can also downgrade the database using `flask db downgrade`.
+You can also downgrade the database using:
+
+```shell
+flask db downgrade
+```
 
 ## Using a Virtual Environment
 One way to avoid hitting version incompatibility errors when running `flask` commands is to use a virtualenv.  See [Python Packaging user guidelines](https://packaging.python.org/guides/installing-using-pip-and-virtualenv/) for instructions on installing virtualenv.  After installing virtualenv, you can create a virtual environment by navigating to the OpenOversight directory and running the below
 
-```bash
+```shell
 python3 -m virtualenv env
 ```
 
 Confirm you're in the virtualenv by running
 
-```bash
+```shell
 which python
 ```
 
 The response should point to your `env` directory.
 If you want to exit the virtualenv, run
 
-```bash
+```shell
 deactivate
 ```
 
 To reactivate the virtualenv, run
 
-```bash
+```shell
 source env/bin/activate
 ```
 
 While in the virtualenv, you can install project dependencies by running
 
-```bash
+```shell
 pip install -r requirements.txt
 ```
 
 and
 
-```bash
+```shell
 pip install -r dev-requirements.txt
 ```
 
 ## OpenOversight Management Interface
 In addition to generating database migrations, the Flask CLI can be used to run additional commands:
 
-```sh
+```shell
 $ flask --help
 Usage: flask [OPTIONS] COMMAND [ARGS]...
 
@@ -204,7 +221,7 @@ Commands:
 
 In development, you can make an administrator account without having to confirm your email:
 
-```sh
+```shell
 $ flask make-admin-user
 Username: redshiftzero
 Email: [email protected]
@@ -230,15 +247,15 @@ Next, in your terminal run `docker ps` to find the container id of the `openover
 
 ## Debugging OpenOversight - Use pdb with a test
 If you want to run an individual test in debug mode, use the below command.
-```bash
+```shell
 docker-compose run --rm web pytest --pdb -v tests/ -k <test_name_here>
 ```
 
 where `<test_name_here>` is the name of a single test function, such as `test_ac_cannot_add_new_officer_not_in_their_dept`
 
 Similarly, you can run all the tests in a file by specifying the file path:
 
-```bash
+```shell
 docker-compose run --rm web pytest --pdb -v path/to/test/file
 ```
 

OpenOversight/migrations/README.md

@@ -0,0 +1,4 @@
+## General Alembic Information
+Alembic provides for the creation, management, and invocation of change management scripts for a relational database, using SQLAlchemy as the underlying engine.
+
+For information on how to execute DB migrations, please visit this section in the `CONTRIB.md` file: [Link](../../CONTRIB.md#migrating-the-database)

OpenOversight/migrations/alembic.ini

@@ -1,15 +1,8 @@
-# A generic, single database configuration.
-
 [alembic]
-# template used to generate migration files
-# file_template = %%(rev)s_%%(slug)s
-
-# set to 'true' to run the environment during
-# the 'revision' command, regardless of autogenerate
-# revision_environment = false
-
+script_location = ./OpenOversight/migrations/
+file_template = %%(year)d-%%(month).2d-%%(day).2d-%%(hour).2d%%(minute).2d_%%(rev)s_%%(slug)s
+truncate_slug_length = 40
 
-# Logging configuration
 [loggers]
 keys = root,sqlalchemy,alembic
 

OpenOversight/migrations/env.py

@@ -2,35 +2,23 @@
 from logging.config import fileConfig
 
 from alembic import context
+from flask import current_app
 from sqlalchemy import engine_from_config, pool
 
 
 # this is the Alembic Config object, which provides
 # access to the values within the .ini file in use.
 config = context.config
 
-# Interpret the config file for Python logging.
-# This line sets up loggers basically.
 fileConfig(config.config_file_name)
 logger = logging.getLogger("alembic.env")
 
-# add your model's MetaData object here
-# for 'autogenerate' support
-# from myapp import mymodel
-# target_metadata = mymodel.Base.metadata
-from flask import current_app  # noqa: E402
-
 
 config.set_main_option(
     "sqlalchemy.url", current_app.config.get("SQLALCHEMY_DATABASE_URI")
 )
 target_metadata = current_app.extensions["migrate"].db.metadata
 
-# other values from the config, defined by the needs of env.py,
-# can be acquired:
-# my_important_option = config.get_main_option("my_important_option")
-# ... etc.
-
 
 def run_migrations_offline():
     """Run migrations in 'offline' mode.

database/README.md

@@ -1,11 +1,9 @@
 # Database Setup
 
 ## Schema/Table Creation
-
 Running `make dev` in the docker environment will create and persist the database.
 
 ## Database Diagram
-
 ![](relationships.real.large.png)
 
 See more detailed database schema information [here](https://disman.tl/oo-docs/).

tasks.py

@@ -36,12 +36,12 @@
 
 
 # make sure we are logging in UTC and have millisecond granularity
-def _formatTime(record, _):
+def _format_time(record, _):
     return str(datetime.datetime.utcfromtimestamp(record.created)) + "Z"
 
 
 for h in logging.getLogger().handlers:
-    h.formatter.formatTime = _formatTime
+    h.formatter.formatTime = _format_time
 
 
 @dataclass