Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Upstream batch 3 #360

Merged
merged 13 commits into from
Oct 9, 2023
Merged

Upstream batch 3 #360

merged 13 commits into from
Oct 9, 2023

Commits on Oct 9, 2023

  1. Update data-migration documentation (lucyparsons#975) 

    lucyparsons#930

Add documentation to make data-migrations a more transparent process.

 - [x] This branch is up-to-date with the `develop` branch.
 - [x] `pytest` passes on my local development environment.
 - [x] `pre-commit` passes on my local development environment.
    @michplunkett @AetherUnbound
    michplunkett authored and AetherUnbound committed Oct 9, 2023
    Configuration menu
    Copy the full SHA
    4f27885 View commit details
    Browse the repository at this point in the history

  2. Rename star_date and descrip (lucyparsons#977) 

    ## Fixes issues
lucyparsons#941
lucyparsons#427

## Description of Changes
Changed the following column names:
- `star_date` -> `start_date`
- `descrip` -> `description`

## Tests and linting
 - [x] This branch is up-to-date with the `develop` branch.
 - [x] `pytest` passes on my local development environment.
 - [x] `pre-commit` passes on my local development environment.
 - [x]  Data-migration output:

```shell
$ docker exec -it openoversight-web-1 bash
$ flask db stamp head
/usr/local/lib/python3.11/site-packages/flask_limiter/extension.py:293: UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend.
  warnings.warn(
[2023-07-18 17:16:06,001] INFO in __init__: OpenOversight startup
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running stamp_revision  -> 93fc3e074dcc
$ flask db migrate -m "rename 'star_date'"
/usr/local/lib/python3.11/site-packages/flask_limiter/extension.py:293: UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend.
  warnings.warn(
[2023-07-18 17:17:01,906] INFO in __init__: OpenOversight startup
...
  Generating /usr/src/app/OpenOversight/migrations/versions/2023-07-18-1717_9ce70d7ebd56_rename_star_date.py ...  done
$ flask db upgrade
[2023-07-18 17:18:49,546] INFO in __init__: OpenOversight startup
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade 93fc3e074dcc -> 9ce70d7ebd56, rename 'star_date'

...

(env) % make start
docker-compose build
...
[+] Running 2/2
 ✔ Container openoversight-postgres-1  Started                                                                                                                                                             0.2s 
 ✔ Container openoversight-web-1       Started                                                                                                                                                             0.3s 
(env) % docker exec -it openoversight-web-1 bash
$ flask db stamp head
/usr/local/lib/python3.11/site-packages/flask_limiter/extension.py:293: UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend.
  warnings.warn(
[2023-07-18 19:18:26,742] INFO in __init__: OpenOversight startup
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
$ flask db migrate -m "rename 'descrip' to 'description'"
...
INFO  [alembic.autogenerate.compare] Detected added column 'unit_types.description'
INFO  [alembic.autogenerate.compare] Detected removed index 'ix_unit_types_descrip' on 'unit_types'
INFO  [alembic.autogenerate.compare] Detected added index 'ix_unit_types_description' on '['description']'
INFO  [alembic.autogenerate.compare] Detected removed column 'unit_types.descrip'
  Generating /usr/src/app/OpenOversight/migrations/versions/2023-07-18-1921_eb0266dc8588_rename_descrip_to_description.py ...  done
$ flask db upgrade
/usr/local/lib/python3.11/site-packages/flask_limiter/extension.py:293: UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend.
  warnings.warn(
[2023-07-18 19:33:12,354] INFO in __init__: OpenOversight startup
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade 9ce70d7ebd56 -> eb0266dc8588, rename 'descrip' to 'description'
$
```
    @michplunkett @AetherUnbound
    michplunkett authored and AetherUnbound committed Oct 9, 2023
    Configuration menu
    Copy the full SHA
    c71ffd4 View commit details
    Browse the repository at this point in the history

  3. Rename migration files (lucyparsons#979) 

    Renaming old migration files to follow new naming schema and adding
messages where they are missing.

 - [x] This branch is up-to-date with the `develop` branch.
 - [x] `pytest` passes on my local development environment.
 - [x] `pre-commit` passes on my local development environment.
    @abandoned-prototype @AetherUnbound
    abandoned-prototype authored and AetherUnbound committed Oct 9, 2023
    Configuration menu
    Copy the full SHA
    3b3279b View commit details
    Browse the repository at this point in the history

  4. Change timestamp datatypes to timestamptz (lucyparsons#978) 

    lucyparsons#323

Standardizing column names and changing `timestamp` datatypes to
`timestamptz`. To make sure the times were presented accurately, I
created a function that was called before any other ones to set the
`TIMEZONE` in the session object. From there, I used a filter that
applies the user's timezone to any dates that are presented. This
session object goes away after `60` minutes of no requests, though that
value is configurable should someone to change it.

The solution is essentially based off these two solutions:
- https://stackoverflow.com/a/75271114
- https://stackoverflow.com/a/49891626

Column name changes:
- `descriptions`: `date_created` -> `created_at`, `date_updated` ->
`updated_at`
- `notes`: `date_created` -> `created_at`, `date_updated` ->
`updated_at`
- `raw_images`: `date_image_inserted`: `created_at`, `date_image_taken`:
`taken_at`

As a `timestamp`:
<img width="1097" alt="Screenshot 2023-07-19 at 1 45 05 PM"
src="https://github.com/lucyparsons/OpenOversight/assets/5885605/34ead7a9-e2de-43b7-a414-b50b7f0de0cb">

As a `timestamptz`:
<img width="1095" alt="Screenshot 2023-07-19 at 1 52 46 PM"
src="https://github.com/lucyparsons/OpenOversight/assets/5885605/d6a34acb-aa98-43d7-929a-9d80dc01416c">

As a `timestamp` (testing the `downgrade` command):
<img width="1093" alt="Screenshot 2023-07-19 at 1 53 06 PM"
src="https://github.com/lucyparsons/OpenOversight/assets/5885605/26c06464-1816-4ca2-8c6d-f5091f79dc9c">

Description and Note `created_at` confirmation:
<img width="524" alt="Screenshot 2023-07-20 at 5 46 46 PM"
src="https://github.com/lucyparsons/OpenOversight/assets/5885605/486e3690-2509-42db-89f2-990a89ea222c">

`descriptions.updated_at`: `2023-07-19 19:08:54.587185 +00:00` -> `July
19, 2023 2:08 PM`
`notes.updated_at`: `2023-07-19 19:06:51.464656 +00:00` -> `July 19,
2023 2:06 PM`

What it will actually look like:
<img width="525" alt="Screenshot 2023-07-20 at 5 55 19 PM"
src="https://github.com/lucyparsons/OpenOversight/assets/5885605/5eb6a2d7-975f-4047-827c-1e71d0ff028a">

 - [x] This branch is up-to-date with the `develop` branch.
 - [x] `pytest` passes on my local development environment.
 - [x] `pre-commit` passes on my local development environment.
 - [x] Data-migration output:

<details><summary>Column Name Ouput</summary>

```shell
$ docker exec -it openoversight-web-1 bash
$ flask db stamp head
[2023-07-18 20:22:15,836] INFO in __init__: OpenOversight startup
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
$ flask db migrate -m "standarize datetime field names"
[2023-07-18 20:27:19,780] INFO in __init__: OpenOversight startup
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
...
INFO  [alembic.autogenerate.compare] Detected added column 'descriptions.created_at'
INFO  [alembic.autogenerate.compare] Detected added column 'descriptions.updated_at'
INFO  [alembic.autogenerate.compare] Detected removed column 'descriptions.date_updated'
INFO  [alembic.autogenerate.compare] Detected removed column 'descriptions.date_created'
INFO  [alembic.autogenerate.compare] Detected added column 'notes.created_at'
INFO  [alembic.autogenerate.compare] Detected added column 'notes.updated_at'
INFO  [alembic.autogenerate.compare] Detected removed column 'notes.date_updated'
INFO  [alembic.autogenerate.compare] Detected removed column 'notes.date_created'
INFO  [alembic.autogenerate.compare] Detected added column 'raw_images.created_at'
INFO  [alembic.autogenerate.compare] Detected added column 'raw_images.taken_at'
INFO  [alembic.autogenerate.compare] Detected removed index 'ix_raw_images_date_image_inserted' on 'raw_images'
INFO  [alembic.autogenerate.compare] Detected removed index 'ix_raw_images_date_image_taken' on 'raw_images'
INFO  [alembic.autogenerate.compare] Detected added index 'ix_raw_images_created_at' on '['created_at']'
INFO  [alembic.autogenerate.compare] Detected added index 'ix_raw_images_taken_at' on '['taken_at']'
INFO  [alembic.autogenerate.compare] Detected removed column 'raw_images.date_image_inserted'
INFO  [alembic.autogenerate.compare] Detected removed column 'raw_images.date_image_taken'
  Generating /usr/src/app/OpenOversight/migrations/versions/2023-07-18-2027_07ace5f956ca_standarize_datetime_field_names.py ...  done
$ flask db upgrade
[2023-07-18 20:34:18,812] INFO in __init__: OpenOversight startup
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade eb0266dc8588 -> 07ace5f956ca, standardize datetime field names
$
```
</details>

<details><summary>Datatype Output</summary>

```shell
% docker exec -it openoversight-web-1 bash
$ flask db stamp head
/usr/local/lib/python3.11/site-packages/flask_limiter/extension.py:293: UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend.
  warnings.warn(
[2023-07-19 18:49:21,592] INFO in __init__: OpenOversight startup
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running stamp_revision  -> eb0266dc8588
$ flask db upgrade
[2023-07-19 18:49:46,272] INFO in __init__: OpenOversight startup
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade eb0266dc8588 -> 07ace5f956ca, standardize datetime field names
INFO  [alembic.runtime.migration] Running upgrade 07ace5f956ca -> 1931b987ce0d, convert timestamp to timestamptz
$ flask db downgrade
[2023-07-19 18:51:59,673] INFO in __init__: OpenOversight startup
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running downgrade 1931b987ce0d -> 07ace5f956ca, convert timestamp to timestamptz
$ flask db upgrade
[2023-07-19 18:52:28,084] INFO in __init__: OpenOversight startup
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade 07ace5f956ca -> 1931b987ce0d, convert timestamp to timestamptz
$
```
</details>

---------

Co-authored-by: abandoned-prototype <[email protected]>
    @michplunkett @abandoned-prototype @AetherUnbound
    2 people authored and AetherUnbound committed Oct 9, 2023
    Configuration menu
    Copy the full SHA
    c9432df View commit details
    Browse the repository at this point in the history

  5. Add license to package.json (lucyparsons#986) 

    ## Fixes issue
lucyparsons#985

## Description of Changes
This addition was made so that the below text does not pop up when
running the application.
```console
2023-07-21 15:45:07 openoversight-web-1       | warning package.json: No license field
```

## Tests and linting
 - [x] This branch is up-to-date with the `develop` branch.
 - [x] `pytest` passes on my local development environment.
 - [x] `pre-commit` passes on my local development environment.
    @michplunkett @AetherUnbound
    michplunkett authored and AetherUnbound committed Oct 9, 2023
    Configuration menu
    Copy the full SHA
    12b9ac4 View commit details
    Browse the repository at this point in the history

  6. Add additional /timezone test (lucyparsons#989) 

    ## Fixes issue
lucyparsons#323

## Description of Changes
Added test to cover an empty string timezone being passed.
    @michplunkett @AetherUnbound
    michplunkett authored and AetherUnbound committed Oct 9, 2023
    Configuration menu
    Copy the full SHA
    a6f5497 View commit details
    Browse the repository at this point in the history

  7. Add state to departments table (lucyparsons#981) 

    Partially addresses this:
lucyparsons#797

Add `state` column to the `departments` table and update all relevant
tests and features. This is the first step to addressing the Department
state searching feature.

While taking care of the `.format` in `models/database.py` for
`Department`, I took care of the other ones as well.

 - [x] This branch is up-to-date with the `develop` branch.
 - [x] `pytest` passes on my local development environment.
 - [x] `pre-commit` passes on my local development environment.

<details><summary>DB Migration Output</summary>

```console
$ flask db upgrade
[2023-07-26 15:52:20,432] INFO in __init__: OpenOversight startup
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade 1931b987ce0d -> 18f43ac4622f, add state column to departments
$ flask db downgrade
[2023-07-26 15:52:25,802] INFO in __init__: OpenOversight startup
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running downgrade 18f43ac4622f -> 1931b987ce0d, add state column to departments
$ flask db upgrade
[2023-07-26 15:52:30,441] INFO in __init__: OpenOversight startup
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade 1931b987ce0d -> 18f43ac4622f, add state column to departments
$
```
</details>
    @michplunkett @AetherUnbound
    michplunkett authored and AetherUnbound committed Oct 9, 2023
    Configuration menu
    Copy the full SHA
    27269a8 View commit details
    Browse the repository at this point in the history

  8. Remove unused image files (lucyparsons#990) 

    ## Description of Changes
Removed unused and outdated image files.

## Tests and linting
 - [x] This branch is up-to-date with the `develop` branch.
 - [x] `pytest` passes on my local development environment.
 - [x] `pre-commit` passes on my local development environment.
    @michplunkett @AetherUnbound
    michplunkett authored and AetherUnbound committed Oct 9, 2023
    Configuration menu
    Copy the full SHA
    dbdf23e View commit details
    Browse the repository at this point in the history

  9. Update pre-commit packages and add flynt (lucyparsons#991) 

    Ran the `pre-commit autoupdate` command and added the
[`flynt`](https://github.com/ikamensh/flynt) package to `pre-commit` to
make sure we don't have any `.format()` strings in the repository.

 - [x] This branch is up-to-date with the `develop` branch.
 - [x] `pytest` passes on my local development environment.
 - [x] `pre-commit` passes on my local development environment.
    @michplunkett @AetherUnbound
    michplunkett authored and AetherUnbound committed Oct 9, 2023
    Configuration menu
    Copy the full SHA
    fc7e4c3 View commit details
    Browse the repository at this point in the history

  10. Add state field to DepartmentForm (lucyparsons#988) 

    lucyparsons#797

Added state abbreviation dropdown to the department creation and editing
page.

Create Department:
<img width="877" alt="Screenshot 2023-07-24 at 12 00 46 PM"
src="https://github.com/lucyparsons/OpenOversight/assets/5885605/ddef0ffe-b381-4fde-9b8f-8c925be8e0fe">

Edit Department:
<img width="1470" alt="Screenshot 2023-07-26 at 4 44 21 PM"
src="https://github.com/lucyparsons/OpenOversight/assets/5885605/adbcc13a-61c7-4b22-bf33-49ae21061fda">

Show state if it exists for department on list screen:
<img width="1185" alt="Screenshot 2023-07-27 at 10 45 15 AM"
src="https://github.com/lucyparsons/OpenOversight/assets/5885605/7530746f-715b-4d76-a7ed-3679ab3a5383">

Show state if it exists for department on download screen:
<img width="1222" alt="Screenshot 2023-07-27 at 10 48 25 AM"
src="https://github.com/lucyparsons/OpenOversight/assets/5885605/454a4c09-6031-4bf5-b1ab-480d47be046a">

 - [x] This branch is up-to-date with the `develop` branch.
 - [x] `pytest` passes on my local development environment.
 - [x] `pre-commit` passes on my local development environment.
    @michplunkett @AetherUnbound
    michplunkett authored and AetherUnbound committed Oct 9, 2023
    Configuration menu
    Copy the full SHA
    9d21d47 View commit details
    Browse the repository at this point in the history

  11. Add gzip compression for HTML, JS, and CSS files (lucyparsons#999) 

    ## Fixes issue
lucyparsons#996

## Description of Changes
Added gzip compression to HTML, JS, and CSS file responses via the
[`Flask-Compress`
package](https://github.com/colour-science/flask-compress).

Network tab of `http://localhost:3000/department/1` without compression:
<img width="821" alt="Screenshot 2023-07-31 at 12 36 41 PM"
src="https://github.com/lucyparsons/OpenOversight/assets/5885605/1ab41e56-a976-4e3c-83dc-fe3ef0bac16a">

Network tab of `http://localhost:3000/department/1` with compression:
<img width="821" alt="Screenshot 2023-07-31 at 12 32 38 PM"
src="https://github.com/lucyparsons/OpenOversight/assets/5885605/fc09cbec-17b9-4640-9a9d-2348a1b973d3">


## Tests and linting
 - [x] This branch is up-to-date with the `develop` branch.
 - [x] `pytest` passes on my local development environment.
 - [x] `pre-commit` passes on my local development environment.
    @michplunkett @AetherUnbound
    michplunkett authored and AetherUnbound committed Oct 9, 2023
    Configuration menu
    Copy the full SHA
    d2d27a1 View commit details
    Browse the repository at this point in the history

  12. Add 'Federal Agency' to list of accepted states (lucyparsons#998) 

    lucyparsons#797

Added 'Federal Agency' to list of accepted states, corrected formatting
of numbers on browse screen, moved choice lists to `utils` folder, and
modified formatter definitions.

Browse:
<img width="662" alt="Screenshot 2023-08-01 at 4 07 34 PM"
src="https://github.com/lucyparsons/OpenOversight/assets/5885605/2318176e-3815-4040-b81a-a3e598ea861c">

Update:
<img width="593" alt="Screenshot 2023-08-01 at 4 09 19 PM"
src="https://github.com/lucyparsons/OpenOversight/assets/5885605/d2b569b1-e11e-4fdf-9160-672e3ed5a9fb">

View officer:
<img width="1159" alt="Screenshot 2023-08-01 at 4 09 33 PM"
src="https://github.com/lucyparsons/OpenOversight/assets/5885605/b0d821d7-9a88-4cd6-aa18-067e578d8111">

Header for listing officers:
<img width="791" alt="Screenshot 2023-08-01 at 4 09 54 PM"
src="https://github.com/lucyparsons/OpenOversight/assets/5885605/b430f38c-ab03-4d61-a543-eedc708213b4">

 - [x] This branch is up-to-date with the `develop` branch.
 - [x] `pytest` passes on my local development environment.
 - [x] `pre-commit` passes on my local development environment.
    @michplunkett @AetherUnbound
    michplunkett authored and AetherUnbound committed Oct 9, 2023
    Configuration menu
    Copy the full SHA
    1f5f1bd View commit details
    Browse the repository at this point in the history

  13. Delete setup.py (lucyparsons#1001) 

    I looked around the application and we don't actually use the file at
all, so I wiped it.

 - [x] This branch is up-to-date with the `develop` branch.
 - [x] `pytest` passes on my local development environment.
 - [x] `pre-commit` passes on my local development environment.
    @michplunkett @AetherUnbound
    michplunkett authored and AetherUnbound committed Oct 9, 2023
    Configuration menu
    Copy the full SHA
    86a3ce7 View commit details
    Browse the repository at this point in the history