Metadata parity between the JSON and XForm data endpoints

[gstuder-ona]

Suggested Feature / Enhancement

Currently OnaData-specific metadata like "_submitted_by" , "_review_status" ,"_review_comment", "_last_edited" are not available in the XForm API - this makes the data unavailable to XForm-based ETLs and other tools.

Benefits of implementing the feature/enhancement

We'll need the submitted_by and review fields for clients that need to audit who was editing their data. Arguably it would also make the API more consistent if the format didn't determine which data was available.

Suggested implementation plan(Steps to be taken to implement feature)

Audit the metadata fields added to the JSON data endpoint and ensure they're added as submission-item attributes in the XForm endpoint.

Random note - last_edited isn't quite the same as last_modified, and including both may be important so a user can know who last modified a form via the UI.

@DavisRayM - just because you've touched this code before

[alepietrobon]

heavy +1 :)

[ukanga]

@gstuder-ona Do you mean on the XForm XML? Can you be specific, please?

Note, this information will not show up in the XForm XML and is not intended to show up on the XForm XML.

[gstuder-ona]

@ukanga I think I was pretty clear in the above - ona-specific metadata (as opposed to XForm metadata like start/end) should not show up in the XForm XML data itself. However, the XML envelope that comes back <submission-batch><submission-item/> is the place this metadata should appear. We currently add metadata there, it's just adding new fields.

This is pretty much the same as injecting metadata as _underscored fields in the JSON result, though IMO it's actually kind of better as there's no potential for confusing the metadata from the data.

I am 100% on board with the XForm results coming back with no processing, this is particular OnaData metadata that is never stored in XForm.

[DavisRayM]

@gstuder-ona Adding most of the metadata fields currently provided by the JSON endpoint is pretty do-able and it'll be fine to add to the submission-item attributes

But there are a few fields namely: _notes, and any other metadata that contains a list of objects that would be a pain to add to the attributes.

If the _notes field is needed, we've come up with 3 approaches we can take to add them:

1. Add them as URL links to the objects i.e <submission-item notes="https://api.ona.io/api/v1/notes/1" />
2. Add them to the meta list within the data element. Least favorite option IMO; We'd be modifying the data sent by ODK Collect
3. Add an additional <server-meta/> element within the <submission-item/>:

<submission-batch>
  <submission-item submissionData="" dateCreated="" formVersion="" objectID="">
    <data/>
    <server-meta>
       <notes/>
       <tags/>
     <server-meta/>
   <submission-item/>
<submission-batch/>

PS: The notes field usually just contains the value of the _review_comment metadata plus a few other values; I've attached the sample values for the two below...

"_notes": [
  {
    "id": 97240,
    "note": "Testing",
    "owner": "davisraym",
    "created_by": 71070,
    "instance_field": "_review_status"
  }
],
"_review_status": "1",
 "_review_comment": "Testing"

CC: @denniswambua

[alepietrobon]

Gonna let @gstuder-ona confirm here, but option 3 seems the most reasonable at a glance

[gstuder-ona]

Option 3 seems fine with me - though I'm a little worried that it could be technically allowed that the form data element has a name of "server-meta". Is that accurate? Arguably it's not a huge deal as the ordering of XML elements can be used to parse, so the first element is always just the data.

Upshot is I'm okay with #3 but leave it up to @DavisRayM and the OnaData team to decide whether it makes sense to have an explicit data wrapper element like <submission-item><submission><data /></submission><server-meta /></submission-item>. It should keep us more backwards-compatible not to add the wrapper, so there's pros either way.