Implement deletion policy #464

matthiasschaub · 2024-06-10T13:47:05Z

Todo:

delete bbox on non-consent
~~should Bbox class have from string (from wkt function)?~~
uncomment esri layer
check if autovacuum is enabled
~~Fix tests by setup database on each run (not just on writing into the database)~~

This PR implements two cleanup functions as well as stores more metadata in the database:

cleanup_map_frames: Cleanup of old map frames for which there are uploaded sketch maps without consent. This cleanup is executed by celery beat every hour.
cleanup_blobs: Cleanup of uploaded sketch maps without consent after vector and raster results are generated. This cleanup is executed as callback by celery after uploaded sketch maps are processed.
Store map generation parameter in the database

Possible improvements:

client_celery.select_file call in tasks.py should return not only the file but also layer and bbox. This would enable to load those only at the place needed and not in routes.py and pass it to tasks.py. (locality of behaviour)
avoid code duplication in database clients
maybe rename select_file() to a more specific name since it does not only return file

Closes #447

Move metadata of a request (of a sketch map) to the map frame table instead of in the blob (uploaded files) table.

move test from test_celery_tasks.py to test_database_client_celery.py

sketch_map_tool/database/client_celery.py

joker234 · 2024-06-18T09:42:25Z

sketch_map_tool/database/client_flask.py

    create_query = """
    CREATE TABLE IF NOT EXISTS blob(
        id SERIAL PRIMARY KEY,
+        uuid UUID,


rename to something like map_frame_uuid or map_uuid

joker234 · 2024-06-18T09:44:00Z

sketch_map_tool/database/client_celery.py

+
+    Only set file and name to null. Keep metadata.
+    """
+    # TODO: Wait one day until deletion or check celery task status?


either implement ts interval of 24 hours or delete only for one uuid

Implemented delete by map frame uuid.

joker234 · 2024-06-18T09:51:29Z

sketch_map_tool/models.py

@@ -17,6 +17,16 @@ class Bbox:
    lon_max: float
    lat_max: float

+    @property


add test function

joker234 · 2024-06-18T10:02:25Z

sketch_map_tool/tasks.py

+def cleanup_map_frames():
+    """Cleanup map frames stored in the database."""
+    db_client_celery.cleanup_map_frames()
+    # TODO: run Vacuum or Vacuum full?


resolve before merging

Autovacuum should be enabled by default. Check first. If so, this should be enough.

Re-create VCR cassettes with out exchanging image in body against a dummy image. Use git lfs.

instead of string

joker234

some recommendations:

mock arcgis request in definitions.py
remove .gitattributes
link config.py (defaults) in documentation

has never been used

matthiasschaub · 2024-08-01T09:26:03Z

some recommendations:

* mock arcgis request in `definitions.py`

* remove `.gitattributes`

* link config.py (defaults) in documentation

All of those has been addressed.

matthiasschaub changed the title ~~Delete policy~~ Overhaul consent and implement delete policy Jun 10, 2024

matthiasschaub added 9 commits June 11, 2024 14:52

start working on periodic task for cleanup

559c8dc

feat(db): update files with uuid from qr-code

45ee063

refactor(qr-code): rename function to avoid alias

a582b9d

refactor(db): read qr-code in insert file function

96287e2

feat(bbox): add centroid and string functions

25cd8cc

feat(db): write attributes of uploaded file to db

8504a55

test(tasks): test cleanup old map frames

c4f42ad

refactor(db): remove unused update files func

f1cc306

refactor(cleanup): remove uuid argument

adf3e6a

matthiasschaub force-pushed the delete-policy branch from f307169 to adf3e6a Compare June 11, 2024 12:52

matthiasschaub added 8 commits June 11, 2024 15:43

feat(celery/config): schedule cleanup task

f1368a3

refactor: store request metadata in db

5fd6b93

Move metadata of a request (of a sketch map) to the map frame table instead of in the blob (uploaded files) table.

docs: improve docstring

09154d7

feat: cleanup blobs without consent

ae2827c

move test from test_celery_tasks.py to test_database_client_celery.py

refactor: use celery group primitive to run tasks

94d7b79

feat: use celery chain primitive to run cleanup

8319584

feat(db): cleanup upload files in database

48ea54a

refactor: rename cleanup to cleanup_map_frames

f5a44ad

matthiasschaub force-pushed the delete-policy branch from b861281 to f5a44ad Compare June 13, 2024 08:06

matthiasschaub marked this pull request as ready for review June 13, 2024 08:13

matthiasschaub requested review from Gigaszi and joker234 June 13, 2024 08:21

matthiasschaub changed the title ~~Overhaul consent and implement delete policy~~ Implement delete policy Jun 13, 2024

matthiasschaub changed the title ~~Implement delete policy~~ Implement deletion policy Jun 13, 2024

matthiasschaub commented Jun 18, 2024

View reviewed changes

sketch_map_tool/database/client_celery.py Outdated Show resolved Hide resolved

joker234 reviewed Jun 18, 2024

View reviewed changes

matthiasschaub and others added 3 commits June 18, 2024 18:45

feat: only delete blobs by uuid

a953754

test(models): add validation of centroid

a805133

fix(test): round centroid to right count of digits

56444b7

matthiasschaub added 19 commits July 17, 2024 18:29

test: remove unnecessary open/close of db conn

827d694

test: fix mock task attrib status & do not close db conn

9743008

fix: disable quality report generation

204d9c7

test: disable replace image content by vcr

3d45ecb

test: re-create VCR cassettes

f059719

Re-create VCR cassettes with out exchanging image in body against a dummy image. Use git lfs.

build: remove obsolete docker-compose version

df921a0

build: rename docker-compose.yaml to compose.yaml

f6d3793

fix(ui): disable quality map check

92c6b17

docs: add command to connect to db

f91216d

docs: add setup docs for apple mac m2

186e817

refactor(ui): update consent text on /digitize page

41514cb

update consent text and made it opt-in

ae89ab1

build: limit resources of docker compose services

dc00a82

test: re-approve approval tests

f6549fa

docs: update dev docs w/ info about db

ffad4a7

refactor: remove unused db client functions

77d985b

refactor(osm-quality-report): return empty BytesIO

669d237

instead of string

Merge remote-tracking branch 'origin/main' into delete-policy

0b02f98

docs: update path to proj in IDE setup section

6421e85

joker234 reviewed Jul 31, 2024

View reviewed changes

matthiasschaub added 6 commits July 31, 2024 18:09

remove .gitattributes

61a95d0

has never been used

docs: reference config.py in config docs

1a77c0e

fix: update ohsome quality api url

4018819

tests(vcr): ignore requests to arcgis.com

9cd8642

tests: re-create vcr cassettes

fbb8e7c

update gitignore to ignore celerybeat-schedule files

637bd16

joker234 approved these changes Aug 1, 2024

View reviewed changes

matthiasschaub merged commit 4d8e329 into main Aug 1, 2024
1 check passed

matthiasschaub deleted the delete-policy branch August 1, 2024 09:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement deletion policy #464

Implement deletion policy #464

matthiasschaub commented Jun 10, 2024 •

edited

Loading

joker234 Jun 18, 2024

Gigaszi Jun 23, 2024

joker234 Jun 18, 2024

matthiasschaub Jun 24, 2024

joker234 Jun 18, 2024

Gigaszi Jun 23, 2024

joker234 Jun 18, 2024

matthiasschaub Jun 24, 2024

joker234 left a comment

matthiasschaub commented Aug 1, 2024 •

edited

Loading

Implement deletion policy #464

Implement deletion policy #464

Conversation

matthiasschaub commented Jun 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joker234 left a comment

Choose a reason for hiding this comment

matthiasschaub commented Aug 1, 2024 • edited Loading

matthiasschaub commented Jun 10, 2024 •

edited

Loading

matthiasschaub commented Aug 1, 2024 •

edited

Loading