From 862b669452cae9fee9938476bbd2bf02dec63538 Mon Sep 17 00:00:00 2001 From: Michael Plunkett <5885605+michplunkett@users.noreply.github.com> Date: Tue, 18 Jul 2023 15:12:24 -0500 Subject: [PATCH] Update data-migration documentation (#975) https://github.com/lucyparsons/OpenOversight/issues/930 Add documentation to make data-migrations a more transparent process. - [x] This branch is up-to-date with the `develop` branch. - [x] `pytest` passes on my local development environment. - [x] `pre-commit` passes on my local development environment. --- CONTRIB.md | 71 +++++++++++++++++----------- OpenOversight/migrations/README | 1 - OpenOversight/migrations/README.md | 4 ++ OpenOversight/migrations/alembic.ini | 13 ++--- OpenOversight/migrations/env.py | 14 +----- database/README.md | 2 - 6 files changed, 52 insertions(+), 53 deletions(-) delete mode 100644 OpenOversight/migrations/README create mode 100644 OpenOversight/migrations/README.md diff --git a/CONTRIB.md b/CONTRIB.md index da21eebba..1706ea091 100644 --- a/CONTRIB.md +++ b/CONTRIB.md @@ -11,8 +11,8 @@ To submit your changes for review you have to fork the repository, push your new Use [PULL_REQUEST_TEMPLATE.md](/PULL_REQUEST_TEMPLATE.md) to create the description for your PR! (The template should populate automatically when you go to open the pull request.) ### Recommended privacy settings -Whenever you make a commit with `git` the name and email saved locally is stored with that commit and will become part of the public history of the project. This can be an unwanted, for example when using a work computer. We recommond changing the email-settings in the github account at https://github.com/settings/emails and selecting "Keep my email addresses private" as well as "Block command line pushes that expose my email". Also find your github-email address of the form `+@users.noreply.github.com` in that section. Then you can change the email and username stored with your commits by running the following commands -``` +Whenever you make a commit with `git` the name and email saved locally is stored with that commit and will become part of the public history of the project. This can be an unwanted, for example when using a work computer. We recommend changing the email-settings in the github account at https://github.com/settings/emails and selecting "Keep my email addresses private" as well as "Block command line pushes that expose my email". Also find your github-email address of the form `+@users.noreply.github.com` in that section. Then you can change the email and username stored with your commits by running the following commands +```shell git config user.email "" git config user.name "" ``` @@ -34,8 +34,8 @@ Tests are executed via `make test`. If you're switching between the Docker and V To hop into the postgres container, you can do the following: -``` -$ docker exec -it openoversight_postgres_1 /bin/bash +```shell +$ docker exec -it openoversight-postgres-1 bash # psql -d openoversight-dev -U openoversight ``` @@ -43,8 +43,8 @@ or run `make attach`. Similarly to hop into the web container: -``` -$ docker exec -it openoversight_web_1 /bin/bash +```shell +$ docker exec -it openoversight-web-1 bash ``` Once you're done, `make stop` and `make clean` to stop and remove the containers respectively. @@ -78,7 +78,7 @@ For more information about these settings, please see the [Flask-Mail](https://f Regardless of implementation, save the email address associated with your service account to a variable named `OO_SERVICE_EMAIL` in a `.env` file in the base directory of this repository. For development and testing, update the `OO_SERVICE_EMAIL` variable in the `docker-compose.yml` file. Example `.env` variable: -```bash +```shell OO_SERVICE_EMAIL="sample_email@domain.com" ``` @@ -86,7 +86,7 @@ In addition to needing a service account email, you also need an admin email add For production, save the email address associated with your admin account to a variable named `OO_HELP_EMAIL` in a `.env` file in the base directory of this repository. For development and testing, update the `OO_HELP_EMAIL` variable in the `docker-compose.yml` file. Example `.env` variable: -```bash +```shell OO_HELP_EMAIL="sample_admin_email@domain.com" ``` @@ -97,7 +97,7 @@ on Amazon Web Services. Once you have done this, you can put your AWS credentials in the following environmental variables: -```sh +```shell $ export S3_BUCKET_NAME=openoversight-test $ export AWS_ACCESS_KEY_ID=testtest $ export AWS_SECRET_ACCESS_KEY=testtest @@ -112,7 +112,7 @@ Running `make dev` will create the database and persist it into your local files You can access your PostgreSQL development database via psql using: -```sh +```shell psql -h localhost -d openoversight-dev -U openoversight --password ``` @@ -124,71 +124,88 @@ or `$ python test_data.py --cleanup` to delete the data ### Migrating the Database -If you e.g. add a new column or table, you'll need to migrate the database using the Flask CLI. First we need to 'stamp' the current version of the database: +You'll first have to start the Docker instance for the OpenOversight app using the command `make start`. To do this, you'll need to be in the base folder of the repository (the one that houses the `Makefile`). + +```shell +$ make start +docker-compose build +... +docker-compose up -d +[+] Running 2/0 + ✔ Container openoversight-postgres-1 Running 0.0s + ✔ Container openoversight-web-1 Running +``` -```sh -$ cd OpenOversight/ # change directory to source dir +From here on out, we'll be using the Flask CLI. First we need to 'stamp' the current version of the database: + +```shell +$ docker exec -it openoversight-web-1 bash # 'openoversight-web-1' is the name of the app container seen in the step above $ flask db stamp head +$ flask db migrate -m "[THE NAME OF YOUR MIGRATION]" # NOTE: Slugs are limited to 40 characters and will be truncated after the limit ``` (Hint: If you get errors when running `flask` commands, e.g. because of differing Python versions, you may need to run the commands in the docker container by prefacing them as so: `docker exec -it openoversight_web_1 flask db stamp head`) -Next make your changes to the database models in `models.py`. You'll then generate the migrations: +Next make your changes to the database models in `OpenOversight/app/models/database.py`. You'll then generate the migrations: -```sh +```shell $ flask db migrate ``` And then you should inspect/edit the migrations. You can then apply the migrations: -```sh +```shell $ flask db upgrade ``` -You can also downgrade the database using `flask db downgrade`. +You can also downgrade the database using: + +```shell +flask db downgrade +``` ## Using a Virtual Environment One way to avoid hitting version incompatibility errors when running `flask` commands is to use a virtualenv. See [Python Packaging user guidelines](https://packaging.python.org/guides/installing-using-pip-and-virtualenv/) for instructions on installing virtualenv. After installing virtualenv, you can create a virtual environment by navigating to the OpenOversight directory and running the below -```bash +```shell python3 -m virtualenv env ``` Confirm you're in the virtualenv by running -```bash +```shell which python ``` The response should point to your `env` directory. If you want to exit the virtualenv, run -```bash +```shell deactivate ``` To reactivate the virtualenv, run -```bash +```shell source env/bin/activate ``` While in the virtualenv, you can install project dependencies by running -```bash +```shell pip install -r requirements.txt ``` and -```bash +```shell pip install -r requirements-dev.txt ``` ## OpenOversight Management Interface In addition to generating database migrations, the Flask CLI can be used to run additional commands: -```sh +```shell $ flask --help Usage: flask [OPTIONS] COMMAND [ARGS]... @@ -220,7 +237,7 @@ Commands: In development, you can make an administrator account without having to confirm your email: -```sh +```shell $ flask make-admin-user Username: redshiftzero Email: jen@redshiftzero.com @@ -246,7 +263,7 @@ Next, in your terminal run `docker ps` to find the container id of the `openover ## Debugging OpenOversight - Use pdb with a test If you want to run an individual test in debug mode, use the below command. -```bash +```shell docker-compose run --rm web pytest --pdb -v tests/ -k ``` @@ -254,7 +271,7 @@ where `` is the name of a single test function, such as `test_ac Similarly, you can run all the tests in a file by specifying the file path: -```bash +```shell docker-compose run --rm web pytest --pdb -v path/to/test/file ``` diff --git a/OpenOversight/migrations/README b/OpenOversight/migrations/README deleted file mode 100644 index 2500aa1bc..000000000 --- a/OpenOversight/migrations/README +++ /dev/null @@ -1 +0,0 @@ -Generic single-database configuration. diff --git a/OpenOversight/migrations/README.md b/OpenOversight/migrations/README.md new file mode 100644 index 000000000..a905a93b1 --- /dev/null +++ b/OpenOversight/migrations/README.md @@ -0,0 +1,4 @@ +## General Alembic Information +Alembic provides for the creation, management, and invocation of change management scripts for a relational database, using SQLAlchemy as the underlying engine. + +For information on how to execute DB migrations, please visit this section in the `CONTRIB.md` file: [Link](../../CONTRIB.md#migrating-the-database) diff --git a/OpenOversight/migrations/alembic.ini b/OpenOversight/migrations/alembic.ini index f8ed4801f..adcf2dfcf 100644 --- a/OpenOversight/migrations/alembic.ini +++ b/OpenOversight/migrations/alembic.ini @@ -1,15 +1,8 @@ -# A generic, single database configuration. - [alembic] -# template used to generate migration files -# file_template = %%(rev)s_%%(slug)s - -# set to 'true' to run the environment during -# the 'revision' command, regardless of autogenerate -# revision_environment = false - +script_location = ./OpenOversight/migrations/ +file_template = %%(year)d-%%(month).2d-%%(day).2d-%%(hour).2d%%(minute).2d_%%(rev)s_%%(slug)s +truncate_slug_length = 40 -# Logging configuration [loggers] keys = root,sqlalchemy,alembic diff --git a/OpenOversight/migrations/env.py b/OpenOversight/migrations/env.py index 4b78a45a3..c40bf6a8f 100644 --- a/OpenOversight/migrations/env.py +++ b/OpenOversight/migrations/env.py @@ -4,6 +4,7 @@ from logging.config import fileConfig from alembic import context +from flask import current_app from sqlalchemy import engine_from_config, pool @@ -11,28 +12,15 @@ # access to the values within the .ini file in use. config = context.config -# Interpret the config file for Python logging. -# This line sets up loggers basically. fileConfig(config.config_file_name) logger = logging.getLogger("alembic.env") -# add your model's MetaData object here -# for 'autogenerate' support -# from myapp import mymodel -# target_metadata = mymodel.Base.metadata -from flask import current_app # noqa: F401 - config.set_main_option( "sqlalchemy.url", current_app.config.get("SQLALCHEMY_DATABASE_URI") ) target_metadata = current_app.extensions["migrate"].db.metadata -# other values from the config, defined by the needs of env.py, -# can be acquired: -# my_important_option = config.get_main_option("my_important_option") -# ... etc. - def run_migrations_offline(): """Run migrations in 'offline' mode. diff --git a/database/README.md b/database/README.md index e6c3fe567..6f0ac76ee 100644 --- a/database/README.md +++ b/database/README.md @@ -1,11 +1,9 @@ # Database Setup ## Schema/Table Creation - Running `make dev` in the docker environment will create and persist the database. ## Database Diagram - ![](relationships.real.large.png) See more detailed database schema information [here](https://disman.tl/oo-docs/).