Upstream batch 3 #360
Merged
Upstream batch 3 #360
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lucyparsons#930 Add documentation to make data-migrations a more transparent process. - [x] This branch is up-to-date with the `develop` branch. - [x] `pytest` passes on my local development environment. - [x] `pre-commit` passes on my local development environment.
## Fixes issues lucyparsons#941 lucyparsons#427 ## Description of Changes Changed the following column names: - `star_date` -> `start_date` - `descrip` -> `description` ## Tests and linting - [x] This branch is up-to-date with the `develop` branch. - [x] `pytest` passes on my local development environment. - [x] `pre-commit` passes on my local development environment. - [x] Data-migration output: ```shell $ docker exec -it openoversight-web-1 bash $ flask db stamp head /usr/local/lib/python3.11/site-packages/flask_limiter/extension.py:293: UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend. warnings.warn( [2023-07-18 17:16:06,001] INFO in __init__: OpenOversight startup INFO [alembic.runtime.migration] Context impl PostgresqlImpl. INFO [alembic.runtime.migration] Will assume transactional DDL. INFO [alembic.runtime.migration] Running stamp_revision -> 93fc3e074dcc $ flask db migrate -m "rename 'star_date'" /usr/local/lib/python3.11/site-packages/flask_limiter/extension.py:293: UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend. warnings.warn( [2023-07-18 17:17:01,906] INFO in __init__: OpenOversight startup ... Generating /usr/src/app/OpenOversight/migrations/versions/2023-07-18-1717_9ce70d7ebd56_rename_star_date.py ... done $ flask db upgrade [2023-07-18 17:18:49,546] INFO in __init__: OpenOversight startup INFO [alembic.runtime.migration] Context impl PostgresqlImpl. INFO [alembic.runtime.migration] Will assume transactional DDL. INFO [alembic.runtime.migration] Running upgrade 93fc3e074dcc -> 9ce70d7ebd56, rename 'star_date' ... (env) % make start docker-compose build ... [+] Running 2/2 ✔ Container openoversight-postgres-1 Started 0.2s ✔ Container openoversight-web-1 Started 0.3s (env) % docker exec -it openoversight-web-1 bash $ flask db stamp head /usr/local/lib/python3.11/site-packages/flask_limiter/extension.py:293: UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend. warnings.warn( [2023-07-18 19:18:26,742] INFO in __init__: OpenOversight startup INFO [alembic.runtime.migration] Context impl PostgresqlImpl. INFO [alembic.runtime.migration] Will assume transactional DDL. $ flask db migrate -m "rename 'descrip' to 'description'" ... INFO [alembic.autogenerate.compare] Detected added column 'unit_types.description' INFO [alembic.autogenerate.compare] Detected removed index 'ix_unit_types_descrip' on 'unit_types' INFO [alembic.autogenerate.compare] Detected added index 'ix_unit_types_description' on '['description']' INFO [alembic.autogenerate.compare] Detected removed column 'unit_types.descrip' Generating /usr/src/app/OpenOversight/migrations/versions/2023-07-18-1921_eb0266dc8588_rename_descrip_to_description.py ... done $ flask db upgrade /usr/local/lib/python3.11/site-packages/flask_limiter/extension.py:293: UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend. warnings.warn( [2023-07-18 19:33:12,354] INFO in __init__: OpenOversight startup INFO [alembic.runtime.migration] Context impl PostgresqlImpl. INFO [alembic.runtime.migration] Will assume transactional DDL. INFO [alembic.runtime.migration] Running upgrade 9ce70d7ebd56 -> eb0266dc8588, rename 'descrip' to 'description' $ ```
Renaming old migration files to follow new naming schema and adding messages where they are missing. - [x] This branch is up-to-date with the `develop` branch. - [x] `pytest` passes on my local development environment. - [x] `pre-commit` passes on my local development environment.
lucyparsons#323 Standardizing column names and changing `timestamp` datatypes to `timestamptz`. To make sure the times were presented accurately, I created a function that was called before any other ones to set the `TIMEZONE` in the session object. From there, I used a filter that applies the user's timezone to any dates that are presented. This session object goes away after `60` minutes of no requests, though that value is configurable should someone to change it. The solution is essentially based off these two solutions: - https://stackoverflow.com/a/75271114 - https://stackoverflow.com/a/49891626 Column name changes: - `descriptions`: `date_created` -> `created_at`, `date_updated` -> `updated_at` - `notes`: `date_created` -> `created_at`, `date_updated` -> `updated_at` - `raw_images`: `date_image_inserted`: `created_at`, `date_image_taken`: `taken_at` As a `timestamp`: <img width="1097" alt="Screenshot 2023-07-19 at 1 45 05 PM" src="https://github.com/lucyparsons/OpenOversight/assets/5885605/34ead7a9-e2de-43b7-a414-b50b7f0de0cb"> As a `timestamptz`: <img width="1095" alt="Screenshot 2023-07-19 at 1 52 46 PM" src="https://github.com/lucyparsons/OpenOversight/assets/5885605/d6a34acb-aa98-43d7-929a-9d80dc01416c"> As a `timestamp` (testing the `downgrade` command): <img width="1093" alt="Screenshot 2023-07-19 at 1 53 06 PM" src="https://github.com/lucyparsons/OpenOversight/assets/5885605/26c06464-1816-4ca2-8c6d-f5091f79dc9c"> Description and Note `created_at` confirmation: <img width="524" alt="Screenshot 2023-07-20 at 5 46 46 PM" src="https://github.com/lucyparsons/OpenOversight/assets/5885605/486e3690-2509-42db-89f2-990a89ea222c"> `descriptions.updated_at`: `2023-07-19 19:08:54.587185 +00:00` -> `July 19, 2023 2:08 PM` `notes.updated_at`: `2023-07-19 19:06:51.464656 +00:00` -> `July 19, 2023 2:06 PM` What it will actually look like: <img width="525" alt="Screenshot 2023-07-20 at 5 55 19 PM" src="https://github.com/lucyparsons/OpenOversight/assets/5885605/5eb6a2d7-975f-4047-827c-1e71d0ff028a"> - [x] This branch is up-to-date with the `develop` branch. - [x] `pytest` passes on my local development environment. - [x] `pre-commit` passes on my local development environment. - [x] Data-migration output: <details><summary>Column Name Ouput</summary> ```shell $ docker exec -it openoversight-web-1 bash $ flask db stamp head [2023-07-18 20:22:15,836] INFO in __init__: OpenOversight startup INFO [alembic.runtime.migration] Context impl PostgresqlImpl. INFO [alembic.runtime.migration] Will assume transactional DDL. $ flask db migrate -m "standarize datetime field names" [2023-07-18 20:27:19,780] INFO in __init__: OpenOversight startup INFO [alembic.runtime.migration] Context impl PostgresqlImpl. INFO [alembic.runtime.migration] Will assume transactional DDL. ... INFO [alembic.autogenerate.compare] Detected added column 'descriptions.created_at' INFO [alembic.autogenerate.compare] Detected added column 'descriptions.updated_at' INFO [alembic.autogenerate.compare] Detected removed column 'descriptions.date_updated' INFO [alembic.autogenerate.compare] Detected removed column 'descriptions.date_created' INFO [alembic.autogenerate.compare] Detected added column 'notes.created_at' INFO [alembic.autogenerate.compare] Detected added column 'notes.updated_at' INFO [alembic.autogenerate.compare] Detected removed column 'notes.date_updated' INFO [alembic.autogenerate.compare] Detected removed column 'notes.date_created' INFO [alembic.autogenerate.compare] Detected added column 'raw_images.created_at' INFO [alembic.autogenerate.compare] Detected added column 'raw_images.taken_at' INFO [alembic.autogenerate.compare] Detected removed index 'ix_raw_images_date_image_inserted' on 'raw_images' INFO [alembic.autogenerate.compare] Detected removed index 'ix_raw_images_date_image_taken' on 'raw_images' INFO [alembic.autogenerate.compare] Detected added index 'ix_raw_images_created_at' on '['created_at']' INFO [alembic.autogenerate.compare] Detected added index 'ix_raw_images_taken_at' on '['taken_at']' INFO [alembic.autogenerate.compare] Detected removed column 'raw_images.date_image_inserted' INFO [alembic.autogenerate.compare] Detected removed column 'raw_images.date_image_taken' Generating /usr/src/app/OpenOversight/migrations/versions/2023-07-18-2027_07ace5f956ca_standarize_datetime_field_names.py ... done $ flask db upgrade [2023-07-18 20:34:18,812] INFO in __init__: OpenOversight startup INFO [alembic.runtime.migration] Context impl PostgresqlImpl. INFO [alembic.runtime.migration] Will assume transactional DDL. INFO [alembic.runtime.migration] Running upgrade eb0266dc8588 -> 07ace5f956ca, standardize datetime field names $ ``` </details> <details><summary>Datatype Output</summary> ```shell % docker exec -it openoversight-web-1 bash $ flask db stamp head /usr/local/lib/python3.11/site-packages/flask_limiter/extension.py:293: UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend. warnings.warn( [2023-07-19 18:49:21,592] INFO in __init__: OpenOversight startup INFO [alembic.runtime.migration] Context impl PostgresqlImpl. INFO [alembic.runtime.migration] Will assume transactional DDL. INFO [alembic.runtime.migration] Running stamp_revision -> eb0266dc8588 $ flask db upgrade [2023-07-19 18:49:46,272] INFO in __init__: OpenOversight startup INFO [alembic.runtime.migration] Context impl PostgresqlImpl. INFO [alembic.runtime.migration] Will assume transactional DDL. INFO [alembic.runtime.migration] Running upgrade eb0266dc8588 -> 07ace5f956ca, standardize datetime field names INFO [alembic.runtime.migration] Running upgrade 07ace5f956ca -> 1931b987ce0d, convert timestamp to timestamptz $ flask db downgrade [2023-07-19 18:51:59,673] INFO in __init__: OpenOversight startup INFO [alembic.runtime.migration] Context impl PostgresqlImpl. INFO [alembic.runtime.migration] Will assume transactional DDL. INFO [alembic.runtime.migration] Running downgrade 1931b987ce0d -> 07ace5f956ca, convert timestamp to timestamptz $ flask db upgrade [2023-07-19 18:52:28,084] INFO in __init__: OpenOversight startup INFO [alembic.runtime.migration] Context impl PostgresqlImpl. INFO [alembic.runtime.migration] Will assume transactional DDL. INFO [alembic.runtime.migration] Running upgrade 07ace5f956ca -> 1931b987ce0d, convert timestamp to timestamptz $ ``` </details> --------- Co-authored-by: abandoned-prototype <[email protected]>
## Fixes issue lucyparsons#985 ## Description of Changes This addition was made so that the below text does not pop up when running the application. ```console 2023-07-21 15:45:07 openoversight-web-1 | warning package.json: No license field ``` ## Tests and linting - [x] This branch is up-to-date with the `develop` branch. - [x] `pytest` passes on my local development environment. - [x] `pre-commit` passes on my local development environment.
## Fixes issue lucyparsons#323 ## Description of Changes Added test to cover an empty string timezone being passed.
Partially addresses this: lucyparsons#797 Add `state` column to the `departments` table and update all relevant tests and features. This is the first step to addressing the Department state searching feature. While taking care of the `.format` in `models/database.py` for `Department`, I took care of the other ones as well. - [x] This branch is up-to-date with the `develop` branch. - [x] `pytest` passes on my local development environment. - [x] `pre-commit` passes on my local development environment. <details><summary>DB Migration Output</summary> ```console $ flask db upgrade [2023-07-26 15:52:20,432] INFO in __init__: OpenOversight startup INFO [alembic.runtime.migration] Context impl PostgresqlImpl. INFO [alembic.runtime.migration] Will assume transactional DDL. INFO [alembic.runtime.migration] Running upgrade 1931b987ce0d -> 18f43ac4622f, add state column to departments $ flask db downgrade [2023-07-26 15:52:25,802] INFO in __init__: OpenOversight startup INFO [alembic.runtime.migration] Context impl PostgresqlImpl. INFO [alembic.runtime.migration] Will assume transactional DDL. INFO [alembic.runtime.migration] Running downgrade 18f43ac4622f -> 1931b987ce0d, add state column to departments $ flask db upgrade [2023-07-26 15:52:30,441] INFO in __init__: OpenOversight startup INFO [alembic.runtime.migration] Context impl PostgresqlImpl. INFO [alembic.runtime.migration] Will assume transactional DDL. INFO [alembic.runtime.migration] Running upgrade 1931b987ce0d -> 18f43ac4622f, add state column to departments $ ``` </details>
## Description of Changes Removed unused and outdated image files. ## Tests and linting - [x] This branch is up-to-date with the `develop` branch. - [x] `pytest` passes on my local development environment. - [x] `pre-commit` passes on my local development environment.
Ran the `pre-commit autoupdate` command and added the [`flynt`](https://github.com/ikamensh/flynt) package to `pre-commit` to make sure we don't have any `.format()` strings in the repository. - [x] This branch is up-to-date with the `develop` branch. - [x] `pytest` passes on my local development environment. - [x] `pre-commit` passes on my local development environment.
lucyparsons#797 Added state abbreviation dropdown to the department creation and editing page. Create Department: <img width="877" alt="Screenshot 2023-07-24 at 12 00 46 PM" src="https://github.com/lucyparsons/OpenOversight/assets/5885605/ddef0ffe-b381-4fde-9b8f-8c925be8e0fe"> Edit Department: <img width="1470" alt="Screenshot 2023-07-26 at 4 44 21 PM" src="https://github.com/lucyparsons/OpenOversight/assets/5885605/adbcc13a-61c7-4b22-bf33-49ae21061fda"> Show state if it exists for department on list screen: <img width="1185" alt="Screenshot 2023-07-27 at 10 45 15 AM" src="https://github.com/lucyparsons/OpenOversight/assets/5885605/7530746f-715b-4d76-a7ed-3679ab3a5383"> Show state if it exists for department on download screen: <img width="1222" alt="Screenshot 2023-07-27 at 10 48 25 AM" src="https://github.com/lucyparsons/OpenOversight/assets/5885605/454a4c09-6031-4bf5-b1ab-480d47be046a"> - [x] This branch is up-to-date with the `develop` branch. - [x] `pytest` passes on my local development environment. - [x] `pre-commit` passes on my local development environment.
## Fixes issue lucyparsons#996 ## Description of Changes Added gzip compression to HTML, JS, and CSS file responses via the [`Flask-Compress` package](https://github.com/colour-science/flask-compress). Network tab of `http://localhost:3000/department/1` without compression: <img width="821" alt="Screenshot 2023-07-31 at 12 36 41 PM" src="https://github.com/lucyparsons/OpenOversight/assets/5885605/1ab41e56-a976-4e3c-83dc-fe3ef0bac16a"> Network tab of `http://localhost:3000/department/1` with compression: <img width="821" alt="Screenshot 2023-07-31 at 12 32 38 PM" src="https://github.com/lucyparsons/OpenOversight/assets/5885605/fc09cbec-17b9-4640-9a9d-2348a1b973d3"> ## Tests and linting - [x] This branch is up-to-date with the `develop` branch. - [x] `pytest` passes on my local development environment. - [x] `pre-commit` passes on my local development environment.
lucyparsons#797 Added 'Federal Agency' to list of accepted states, corrected formatting of numbers on browse screen, moved choice lists to `utils` folder, and modified formatter definitions. Browse: <img width="662" alt="Screenshot 2023-08-01 at 4 07 34 PM" src="https://github.com/lucyparsons/OpenOversight/assets/5885605/2318176e-3815-4040-b81a-a3e598ea861c"> Update: <img width="593" alt="Screenshot 2023-08-01 at 4 09 19 PM" src="https://github.com/lucyparsons/OpenOversight/assets/5885605/d2b569b1-e11e-4fdf-9160-672e3ed5a9fb"> View officer: <img width="1159" alt="Screenshot 2023-08-01 at 4 09 33 PM" src="https://github.com/lucyparsons/OpenOversight/assets/5885605/b0d821d7-9a88-4cd6-aa18-067e578d8111"> Header for listing officers: <img width="791" alt="Screenshot 2023-08-01 at 4 09 54 PM" src="https://github.com/lucyparsons/OpenOversight/assets/5885605/b430f38c-ab03-4d61-a543-eedc708213b4"> - [x] This branch is up-to-date with the `develop` branch. - [x] `pytest` passes on my local development environment. - [x] `pre-commit` passes on my local development environment.
I looked around the application and we don't actually use the file at all, so I wiped it. - [x] This branch is up-to-date with the `develop` branch. - [x] `pytest` passes on my local development environment. - [x] `pre-commit` passes on my local development environment.
AetherUnbound
force-pushed
the
upstream-batch-3
branch
from
3890e01
to
86a3ce7
Compare
AetherUnbound
approved these changes
Oct 9, 2023
|
One note for us: many of our data mungers use |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of Changes
Third batch of upstream changes
Notes for Deployment
There are a couple migrations that need to be run for this set of changes!
Screenshots (if appropriate)
Tests and linting
I have rebased my changes on
mainjust lintpassesjust testpasses