RestService not called on submission update

[evanxg852000]

Rest services get called when a new submission is sent, but the service is never called uppon updating the same submission. Is this intentional? how can I mitigate this?
I am keen on working on this if need be as we so desperately need this feature.

[jnm]

Thanks for offering to work on this! A few questions:

1. Are you using REST Services through KPI (see screenshot below), or are you using KoBoCAT directly?
   
2. It was intentional that POSTs to external services would only be made when submissions were first retrieved. I think the initial use-case was for people who manage their data outside of KoBo and therefore don't really use our editing features. How would you like this to work? Would another POST request be made after editing, containing the whole submission with any edited responses, or would there be a PUT / PATCH or something else to distinguish it as an update?

[evanxg852000]

1. I am using through KPI as in the screenshot.
2. I think it would be good to send the whole submission data containing updated fields and letting the consumer decide any change difference. However, it makes sense to send updates via PUT to make the distinction clear. This can save consumers from checking if the resource already exists.

[jnm]

To simply POST the whole thing again after editing should be a quick hack that you can apply to your instance if you need this urgently:

kobocat/onadata/apps/viewer/models/parsed_instance.py

Lines 339 to 340 in 3de1e63

	if success and created:
	call_service(self)

Try removing the and created condition.

[edit: removing and created is not enough because, as kryptt describes below, KPI will refuse to POST again and return a HTTP 409 response to KoBoCAT]

To do this properly would be more involved; I think we'd have to:

1. Add a new option for re-submitting edits in the KPI UI;
2. Somehow communicate that option to KC so that KC knows whether to resubmit edits to KPI;
   • Or, have KC always resubmit edits to KPI, but have them somehow tagged as edits instead of new submissions. Then, KPI can decide whether to re-send them to the external server;
3. Decide whether to use PUT or POST (or make it configurable?)
4. Make sure duplicate submission IDs don't cause a problem with KPI's REST service logging;
5. Be done and enjoy?

[kryptt]

After removing the and created bit;

I get the following error in the kobocat logs:

-------------- kobocat_main_worker@kobocat v4.2.1 (windowlicker)
---- **** -----
--- * *** * -- Linux-4.15.0-66-generic-x86_64-with-Ubuntu-16.04-xenial 2019-11-25 13:36:57
-- * - **** ---

• ** ---------- [config]
• ** ---------- .> app: onadata:0x7f1fba88e090
• ** ---------- .> transport: redis://redis-main.craft.swcoe.natlab.research.philips.private:6379/2
• ** ---------- .> results: redis://redis-main.craft.swcoe.natlab.research.philips.private:6379/2
• *** --- * --- .> concurrency: 2 (prefork)
  -- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
  --- ***** -----
  -------------- [queues]
  .> kobocat_queue exchange=kobocat_queue(direct) key=kobocat_queue

[tasks]
. onadata.apps.logger.tasks.fix_root_node_names
. onadata.apps.logger.tasks.generate_stats_zip
. onadata.apps.restservice.tasks.service_definition_task
. onadata.apps.stats.tasks.stat_log
. onadata.apps.viewer.models.parsed_instance.update_mongo_instance
. onadata.apps.viewer.tasks.create_analyser_export
. onadata.apps.viewer.tasks.create_csv_export
. onadata.apps.viewer.tasks.create_csv_zip_export
. onadata.apps.viewer.tasks.create_external_export
. onadata.apps.viewer.tasks.create_kml_export
. onadata.apps.viewer.tasks.create_sav_zip_export
. onadata.apps.viewer.tasks.create_xls_export
. onadata.apps.viewer.tasks.create_zip_export
. onadata.apps.viewer.tasks.delete_export
. onadata.apps.viewer.tasks.email_mongo_sync_status
. onadata.apps.viewer.tasks.log_stuck_exports_and_mark_failed
. onadata.celery.debug_task

ERROR 2019-11-25 14:12:11,409 tasks 488 139774313666304 service_definition_task - 409 Client Error: CONFLICT for url: http://manage.craft.swcoe.natlab.research.philips.internal/assets/aK9WQ2PncvSWf3sqcNzVgo/hook-s
ignal/
Traceback (most recent call last):
File "/srv/src/kobocat/onadata/apps/restservice/tasks.py", line 26, in service_definition_task
service.send(rest_service.service_url, data)
File "/srv/src/kobocat/onadata/apps/restservice/services/kpi_hook.py", line 28, in send
response.raise_for_status()
File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 940, in raise_for_status
raise HTTPError(http_error_msg, response=self)
HTTPError: 409 Client Error: CONFLICT for url: http://manage.craft.swcoe.natlab.research.philips.internal/assets/aK9WQ2PncvSWf3sqcNzVgo/hook-signal/

[taylordowns2000]

Hi all, @aleksa-krolls and I are running into this same issue with an @OpenFn integration. @evanxg852000 , @kryptt , @jnm , did you ever figure out your problem? Relatedly, I want to flag a potential bug/strange behaviour we've noticed:

1. As a workaround for updated submissions not being sent via Rest Services, our middleware periodically fetches submissions using this api: https://kf.kobotoolbox.org/api/v2/assets/blah/data/?format=json
2. We see that the _uuid and meta/instanceID of the updated submission actually change on each edit, making updating a record in an external system really hard.
3. We see that a meta/deprecatedID key gets added to the submission object, but that this key only keeps the last change, so if a form is edited more than one, the only way to link the 3rd version to the 1st is by also looking at the 2nd.
4. We notice that _id doesn't change across submissions, but it looks like an 8 digit integer and I've got no idea if it's unique across forms. (My guess is no, and that we cannot use this as a uuid for updating a record in an external system.)

Are we right in our understanding of how things work in 2-4 above? If so, how can someone take an updated form and use it to update a destination system? (In a way that survives beyond the first edit.)

Finally, is there any uuid for a submission, not for a particular submission revision? This, it would seem, would be the silver bullet—a global identifier for each submission. It's a very common requirement outside Kobo... wondering if we could push for Kobo to add a uuid for each submission internally as well! (Or, at least, a way to link "different submissions"—if that's how edits are handled technically—together.)

Thanks in advance for the consideration. Happy to take a crack at a PR if you point me in the right direction.

Taylor

[jnm]

Sorry for the slow response, @taylordowns2000, but you're right about points 2 and 3. This is dictated by the OpenRosa specification: https://docs.getodk.org/openrosa-metadata/.

Point 4 isn't quite right: _id is unique across forms (on the same server). This is a KoBoCAT-specific thing and not part of the OpenRosa spec; it's simply the Postgres primary key:

kobocat/onadata/apps/viewer/models/parsed_instance.py

Lines 233 to 237 in cbaa4e2

	def to_dict_for_mongo(self):
	d = self.to_dict()
	data = {
	UUID: self.instance.uuid,
	ID: self.instance.id,

kobocat/onadata/libs/utils/common_tags.py

Line 6 in cbaa4e2

ID = u"_id"

To answer your last question:

Finally, is there any uuid for a submission, not for a particular submission revision?

…not by default, but you can add one to your form by creating a calculate question with the calculation column containing once(uuid()), where:

• uuid() generates a UUID using the same mechanism as meta/instanceID: https://docs.getodk.org/form-operators-functions/#uuid
• once() ensures uuid() is not called during edits: https://docs.getodk.org/form-operators-functions/#once

Example XLSForm:

name	type	label	calculation
silver_bullet	calculate	unchanging uuid	once(uuid())
anything	text	type anything

XForm produced from that XLSForm, where you can see that /meta/instanceID is concat('uuid:', uuid()):

<?xml version="1.0" encoding="utf-8"?>
<h:html xmlns="http://www.w3.org/2002/xforms" xmlns:ev="http://www.w3.org/2001/xml-events" xmlns:h="http://www.w3.org/1999/xhtml" xmlns:jr="http://openrosa.org/javarosa" xmlns:odk="http://www.opendatakit.org/xforms" xmlns:orx="http://openrosa.org/xforms" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <h:head>
    <h:title>unchanging uuid</h:title>
    <model>
      <instance>
        <abJaKwDGoUrJz8hm38Uhut id="abJaKwDGoUrJz8hm38Uhut" version="vwLWuEdV8neuus26ARK7w2">
          <formhub>
            <uuid/>
          </formhub>
          <silver_bullet/>
          <anything/>
          <__version__/>
          <meta>
            <instanceID/>
          </meta>
        </abJaKwDGoUrJz8hm38Uhut>
      </instance>
      <bind calculate="once(uuid())" nodeset="/abJaKwDGoUrJz8hm38Uhut/silver_bullet" type="string"/>
      <bind nodeset="/abJaKwDGoUrJz8hm38Uhut/anything" type="string"/>
      <bind calculate="'vwLWuEdV8neuus26ARK7w2'" nodeset="/abJaKwDGoUrJz8hm38Uhut/__version__" type="string"/>
      <bind calculate="concat('uuid:', uuid())" nodeset="/abJaKwDGoUrJz8hm38Uhut/meta/instanceID" readonly="true()" type="string"/>
      <bind calculate="'1560320e8115401c89abbdea384b52c6'" nodeset="/abJaKwDGoUrJz8hm38Uhut/formhub/uuid" type="string"/>
    </model>
  </h:head>
  <h:body>
    <input ref="/abJaKwDGoUrJz8hm38Uhut/anything">
      <label>type anything</label>
    </input>
  </h:body>
</h:html>

[taylordowns2000]

Thank you, @jnm , and no worries at all on the delay. It looks like once(uuid()) is exactly what we need.

On my question 4 (can we rely on _id to be unique?) I get you—and it seems that we can but only if the client is using a single server/single database. I like your once(uuid()) solution more as it scales across deployments. (I'm partial to uuids generally—have just always found them super useful!)

Thanks again! Any updates on the original issue here? Can REST services also make POST requests with the entire form content when an update takes place?

[jnm]

On my question 4 (can we rely on _id to be unique?) I get you—and it seems that we can but only if the client is using a single server/single database.

Right, it's the single database that's the key, here. KoBo doesn't have sharding or anything, so as long as you're dealing with the same installation of KoBo, _id is unique. It doesn't matter if you have load balancing across many front-end servers, as we do: they all connect to the same Postgres database. If you want to do something fancy with replication, that's fine too; _id will remain unique.

There are situations where _id will fail you, though. A few come to mind:

• You operate multiple instances of KoBo with separate Postgres databases, and those multiple instances of KoBo all POST submissions to some other service that you run. Your other service would see duplicate _ids here.
• Someone develops a process that depends on KoBo _ids being unique and unchanging, but then they decide to move their data (e.g. using ODK Briefcase) from one KoBo instance to another. To their surprise, the _ids are now totally different!

I'd go ahead with once(uuid()) instead of _id. I'm trying to think of disadvantages:

• You have to remember to add it to your forms 😄
• It is generated by the client, so the server can't totally guarantee that it's unique. Client bugs, such as reusing a UUID but changing some other part of the XML submission content, or server bugs like Duplicate UUIDs/submissions #470 will lead to duplicate once(uuid()) (and, for that matter, _uuid) values in your data set.

Thanks again! Any updates on the original issue here? Can REST services also make POST requests with the entire form content when an update takes place?

No, and yes 😉 KoBo hasn't pursued this at all, but other people are interested. I'm going to mention you over at kobotoolbox/kpi#2781.

[mtyszler]

Any updates here?

I'm setting up some workflows, and triggering a post call after submit an edition is quite useful