Show department last updated dates #214

sea-kelp · 2022-10-02T19:33:20Z

Description of Changes

Fix #82.

Adds officer, incident, and assignment last updated date for each department to the browse page.

I just threw the updated dates under the department header but let me know if you can think of a better way to display the dates!

Notes for Deployment

Make sure alembic is run!

Screenshots (if appropriate)

SPD has last updated dates for officers, incidents, and assignments while APD has none

Tests and linting

I have rebased my changes on main
just lint passes
just test passes

AetherUnbound · 2022-10-09T03:23:25Z

What steps do you think we'll need to take in order to migrate this in production?

AetherUnbound · 2022-10-09T03:30:23Z

When I try the following steps locally, I get an error 😰 I'm not sure what is going on, but I see a migration which drops the migrate_version table?? how are any migrations happening?

git checkout main
just fresh-start
gh pr checkout 214
just db upgrade

sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedTable) table "migrate_version" does not exist

[SQL: 
DROP TABLE migrate_version]
(Background on this error at: https://sqlalche.me/e/14/f405)

sea-kelp · 2022-10-09T05:49:49Z

When I try the following steps locally, I get an error cold_sweat I'm not sure what is going on, but I see a migration which drops the migrate_version table?? how are any migrations happening?
1. `git checkout main`

2. `just fresh-start`

3. `gh pr checkout 214`

4. `just db upgrade`
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedTable) table "migrate_version" does not exist

[SQL: 
DROP TABLE migrate_version]
(Background on this error at: https://sqlalche.me/e/14/f405)

In your local dev environment, you'll need to run just db stamp while on the main branch to set the current migration (this usually doesn't need to be done in prod since alembic records the migrations that have run but idk if we've run migrations before). Then just db upgrade should just work

AetherUnbound · 2022-10-11T03:34:57Z

Got it! I was able to run just db stamp both locally and in production, and just db upgrade ran fine locally too 😄

AetherUnbound

So I've been thinking about this a bit more - I wonder if we might be able to show 3 dates: officer updated, assignments updated, and incidents updates.

Officers may be updated by volunteers (demographics and other info), so displaying that date might be useful. We may want to have that be an auto-update/TIMESTAMP NOW column.

Assignments will be updated programmatically and in bulk, but those could also be updated by volunteers. Perhaps that should be auto-updated as well?

Incidents makes sense to have separately too since that's another dimension by which we could update info!

What do you think?

AetherUnbound · 2022-10-11T03:36:38Z

OpenOversight/app/models.py

@@ -59,6 +60,24 @@ def toCustomDict(self):
            "unique_internal_identifier_label": self.unique_internal_identifier_label,
        }

+    def latest_incident_date(self) -> datetime.date:


Should we cache these at all? Would that make sense to do/affect anything?

Yup, that makes a lot of sense for these values! Do you have any suggestions on what to use for caching? I know there's the built-in functools cache but we probably want to be able to set a TTL on these dates

Added cachetools caching!

AetherUnbound · 2022-10-11T03:38:52Z

OpenOversight/app/templates/browse.html

-          <p>
+
+       <div>
+         <i class="dept-{{department.id}}">Officers Updated: {{department.latest_officer_update()}}</i>


This displays as None if there's no date, could we use "No data" in those cases instead?

Yup, changed!

sea-kelp · 2022-10-11T05:48:17Z

Got it! I was able to run just db stamp both locally and in production, and just db upgrade ran fine locally too smile

Great! (And just to re-iterate, we only need to run it once in prod to tell it to skip all the previously committed migrations)

What do you think?

So what I'm hearing is that we want to add an auto-updating date_updated column to the officers, assignments, and incidents tables and list that per-department. Is that correct?

AetherUnbound · 2022-10-11T18:26:59Z

Yes! Nailed it exactly 🚀

sea-kelp · 2022-10-13T01:12:08Z

OpenOversight/app/models.py

@@ -39,6 +42,27 @@
 )


+date_updated_cache = TTLCache(maxsize=1024, ttl=60)


I forgot to update the ttl but does a 12 hour ttl make sense?

Yes I think so!

sea-kelp · 2022-10-13T01:13:10Z

OpenOversight/app/models.py

+    date_updated = db.Column(
+        db.DateTime, default=func.now(), onupdate=func.now(), index=True
+    )


This might make bulk updates really slow since we'd be updating lots of values in this column

I think even at scale that shouldn't be too impactful, the rows are getting updated anyway and in Postgres that means pulling out the whole row at once. I think we should be okay!

AetherUnbound

Looks great! Can't wait to get this out!

sea-kelp added 2 commits October 2, 2022 12:19

Add Department last updated dates

c9a4a59

Refactor repetitive test code

822aa9a

sea-kelp requested a review from a team as a code owner October 2, 2022 19:33

AetherUnbound reviewed Oct 11, 2022

View reviewed changes

Show date_updated for officers, incidents, and assignments

909ea8c

sea-kelp commented Oct 13, 2022

View reviewed changes

sea-kelp requested a review from AetherUnbound October 13, 2022 01:15

Update cache TTL to 12 hours

306b2b5

AetherUnbound approved these changes Oct 15, 2022

View reviewed changes

sea-kelp merged commit 07a10fe into main Oct 15, 2022

sea-kelp deleted the 82/show-dept-last-updated-dates branch October 15, 2022 20:06

abandoned-prototype mentioned this pull request Mar 27, 2023

Backport from Orca Collective - Step 3 lucyparsons/OpenOversight#908

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Show department last updated dates #214

Show department last updated dates #214

sea-kelp commented Oct 2, 2022 •

edited

Loading

AetherUnbound commented Oct 9, 2022

AetherUnbound commented Oct 9, 2022

sea-kelp commented Oct 9, 2022

AetherUnbound commented Oct 11, 2022

AetherUnbound left a comment

AetherUnbound Oct 11, 2022

sea-kelp Oct 11, 2022

sea-kelp Oct 13, 2022

AetherUnbound Oct 11, 2022

sea-kelp Oct 13, 2022

sea-kelp commented Oct 11, 2022

AetherUnbound commented Oct 11, 2022

sea-kelp Oct 13, 2022

AetherUnbound Oct 15, 2022

sea-kelp Oct 13, 2022

AetherUnbound Oct 15, 2022

AetherUnbound left a comment

		@@ -39,6 +42,27 @@
		)


		date_updated_cache = TTLCache(maxsize=1024, ttl=60)

Show department last updated dates #214

Show department last updated dates #214

Conversation

sea-kelp commented Oct 2, 2022 • edited Loading

Description of Changes

Notes for Deployment

Screenshots (if appropriate)

Tests and linting

AetherUnbound commented Oct 9, 2022

AetherUnbound commented Oct 9, 2022

sea-kelp commented Oct 9, 2022

AetherUnbound commented Oct 11, 2022

AetherUnbound left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sea-kelp commented Oct 11, 2022

AetherUnbound commented Oct 11, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AetherUnbound left a comment

Choose a reason for hiding this comment

sea-kelp commented Oct 2, 2022 •

edited

Loading