Support query by date_modified field #2009

DavisRayM · 2021-02-04T09:47:51Z

Changes / Features implemented

Add _date_modified into the generated Instance JSON
Update tests
Add created attribute to the data list XML response Instance representations

Steps taken to verify this change does what is intended

Updated tests

Side effects of implementing this change

Introduces new migrations that regenerate the JSON representation of all un-deleted Instances. Possible chance of downtime while the migration takes place

Closes #2001

ivermac

Looks good. Quick question, why do we need the migration file?

ukanga · 2021-02-04T12:28:02Z

onadata/apps/logger/models/instance.py

@@ -389,6 +386,7 @@ def get_full_dict(self, load_existing=True):
                self.date_created = submission_time()

            doc[SUBMISSION_TIME] = self.date_created.strftime(MONGO_STRFTIME)
+            doc[DATE_MODIFIED] = self.date_modified.strftime(MONGO_STRFTIME)


We do have this check https://github.com/onaio/onadata/pull/2009/files#diff-98ae21dd41854ad6194bdbca52b97c4d2b0231f6d06b3ee5ee73c4c88a135e27R385-R386 for date_created ensuring it is not null, should we have the same for date_modified?

DavisRayM · 2021-02-04T14:13:36Z

Looks good. Quick question, why do we need the migration file?

It's there so that the JSON is regenerated to enable users to start using the _date_modified filter. The JSON doesn't seem like it's updated till a Submission is edited

ukanga · 2021-02-04T15:15:17Z

onadata/apps/logger/migrations/0062_auto_20210202_0248.py

+# pylint: skip-file
+# Generated by Django 2.2.16 on 2021-02-02 07:48
+
+from django.db import migrations
+from onadata.apps.logger.models.instance import Instance
+
+
+def regenerate_instance_json(apps, schema_editor):
+    """
+    Regenerate Instance JSON
+    """
+    for inst in Instance.objects.filter(deleted_at__isnull=True):
+        inst.json = inst.get_full_dict(load_existing=False)
+        inst.save()
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('logger', '0061_auto_20200713_0814'),
+    ]
+
+    operations = [
+        migrations.RunPython(regenerate_instance_json)
+    ]


We will need a strategy to run this in our production environment, other deployments might be okay.

ukanga · 2021-02-04T15:18:21Z

onadata/apps/logger/models/instance.py

@@ -388,6 +385,12 @@ def get_full_dict(self, load_existing=True):
            if not self.date_created:
                self.date_created = submission_time()

+            if not self.date_modified:
+                doc[DATE_MODIFIED] = ""


Could we not include the field or use the date_created value, since that would ideally be the same value on the initial creation of the record? The latter approach is what we did for the DATE_CREATED field.

ukanga · 2021-02-12T13:28:39Z

onadata/apps/logger/models/instance.py

+            if not self.date_modified:
+                doc[DATE_MODIFIED] = self.date_created


Should this be getting, the date format treatment as below or is it handled elsewhere?

doc[DATE_MODIFIED] = self.date_created.strftime(MONGO_STRFTIME)

ukanga · 2021-02-15T10:55:48Z

onadata/apps/logger/models/instance.py

+            if not self.date_modified:
+                doc[DATE_MODIFIED] = self.date_created.strftime(
+                    MONGO_STRFTIME)
+            else:
+                doc[DATE_MODIFIED] = self.date_modified.strftime(
+                    MONGO_STRFTIME)
+


Simplify

if not self.date_modified: self.date_modified= self.date_created doc[DATE_MODIFIED] = self.date_modified.strftime( MONGO_STRFTIME)

- Add '_date_modified' header

Retrieve _date_modified value from the actual CSV

Add the _date_modified field to the sample data for the Data endpoint

DavisRayM requested review from ukanga, ivermac and gstuder-ona February 4, 2021 09:47

DavisRayM changed the title ~~Support query by date_modified field~~ [WIP] Support query by date_modified field Feb 4, 2021

ivermac previously approved these changes Feb 4, 2021

View reviewed changes

DavisRayM dismissed ivermac’s stale review via 5f0907d February 4, 2021 12:01

ukanga reviewed Feb 4, 2021

View reviewed changes

DavisRayM force-pushed the 2001-search-using-last-modified branch from 5f0907d to 9056773 Compare February 4, 2021 14:27

ukanga reviewed Feb 4, 2021

View reviewed changes

DavisRayM force-pushed the 2001-search-using-last-modified branch 4 times, most recently from a764dd7 to 31638e0 Compare February 11, 2021 14:05

DavisRayM changed the title ~~[WIP] Support query by date_modified field~~ Support query by date_modified field Feb 12, 2021

DavisRayM force-pushed the 2001-search-using-last-modified branch from 108d71a to c813e1f Compare February 12, 2021 12:42

ukanga reviewed Feb 12, 2021

View reviewed changes

ukanga reviewed Feb 15, 2021

View reviewed changes

DavisRayM added 11 commits February 15, 2021 14:13

Test that it's possible to filter by date_modified

e4e9947

Add "_date_modified" into Instance JSON

efcfa70

Add 'created' attribute to Data list XML representation

5b87476

Format files

bd12fd2

Add '_date_modified' to the ADDITIONAL_COLUMNS list

3a1a61d

Add DATE_MODIFIED into the METADATA_FIELDS list

eaf5905

Update tests

01427d5

Check if date_modified is null

303253a

Format files

1a5c0e0

Modify ordering of additional labels

b94ab52

Set '_date_modified' to the date_created value if not set on Instance

edc9fb5

DavisRayM added 9 commits February 15, 2021 14:13

Update tests

d2c9ff8

Format files

21807d6

Update test fixtures

c8b27b9

- Add '_date_modified' header

Update tests

cac47f4

Add '_date_modified' to the XForm headers list

e1d445d

Format date_created value before assigning to _date_modified field

6113cf5

Update tests

79ed7bd

Retrieve _date_modified value from the actual CSV

Update documentation

73f0095

Add the _date_modified field to the sample data for the Data endpoint

Set date_modified to date_created if null

e7f4ef2

DavisRayM force-pushed the 2001-search-using-last-modified branch from 8f2534b to e7f4ef2 Compare February 15, 2021 11:13

Trigger CI tests

c7202b7

DavisRayM force-pushed the 2001-search-using-last-modified branch from 7479bb7 to c7202b7 Compare February 15, 2021 11:50

ukanga approved these changes Feb 15, 2021

View reviewed changes

DavisRayM merged commit ca1838a into master Feb 16, 2021

DavisRayM deleted the 2001-search-using-last-modified branch February 16, 2021 13:59

DavisRayM mentioned this pull request Feb 23, 2021

Tag release v2.5.1 #2020

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support query by date_modified field #2009

Support query by date_modified field #2009

DavisRayM commented Feb 4, 2021

ivermac left a comment

ukanga Feb 4, 2021

DavisRayM commented Feb 4, 2021

ukanga Feb 4, 2021

ukanga Feb 4, 2021 •

edited

Loading

ukanga Feb 12, 2021

ukanga Feb 15, 2021

		if not self.date_modified:
		doc[DATE_MODIFIED] = self.date_created

Support query by date_modified field #2009

Support query by date_modified field #2009

Conversation

DavisRayM commented Feb 4, 2021

Changes / Features implemented

Steps taken to verify this change does what is intended

Side effects of implementing this change

ivermac left a comment

Choose a reason for hiding this comment

ukanga Feb 4, 2021

Choose a reason for hiding this comment

DavisRayM commented Feb 4, 2021

ukanga Feb 4, 2021

Choose a reason for hiding this comment

ukanga Feb 4, 2021 • edited Loading

Choose a reason for hiding this comment

ukanga Feb 12, 2021

Choose a reason for hiding this comment

ukanga Feb 15, 2021

Choose a reason for hiding this comment

ukanga Feb 4, 2021 •

edited

Loading