Skip to content

Commit

Permalink
Merge pull request #2114 from onaio/2107-xml-metadata
Browse files Browse the repository at this point in the history 
Add metadata fields present in the JSON response to the XML response
  • Loading branch information
@DavisRayM
DavisRayM authored Jul 13, 2021
2 parents dcfc7a0 + b743d8f commit 472b466
Show file tree
Hide file tree
Showing 4 changed files with 136 additions and 26 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/docker-image-build.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ jobs:
- name: Get the branch name
id: get-branch-name
if: github.event_name == 'push'
run: echo ::set-output name=BRANCH::${GITHUB_REF/refs/\heads\//}
run: echo "##[set-output name=BRANCH;]$(echo ${GITHUB_REF#refs/heads/})"

- name: Login to DockerHub
uses: docker/login-action@v1
Expand Down
74 changes: 68 additions & 6 deletions docs/data.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -132,9 +132,9 @@ Example

curl -X GET https://api.ona.io/api/v1/data?owner=ona

Get Submitted data for a specific form
GET JSON list of submitted data for a specific form
------------------------------------------
Provides a list of json submitted data for a specific form.
Provides a JSON list of submitted data for a specific form.

Note: Responses are automatically paginated when requesting a list of data that surpasses 10,000 records.

Expand Down Expand Up @@ -195,6 +195,68 @@ Response
....
]

GET XML list of submitted data for a specific form
--------------------------------------------------

Provides an XML list of submitted data for a specific form.

.. raw:: html
<pre class="prettyprint">
<b>GET</b> /api/v1/data/<code>{pk}</code>.xml
</pre>

Example
^^^^^^^^
::

curl -X GET https://api.ona.io/api/v1/data/574.xml

Response
^^^^^^^^^
::

<submission-batch serverTime="2021-07-02T08:16:24.304534+00:00">
<submission-item bambooDatasetId="" dateCreated="2021-07-02T08:16:24.091445+00:00" duration="" edited="False" formVersion="2014111" lastModified="2021-07-02T08:16:24.206278+00:00" mediaAllReceived="True" mediaCount="1" objectID="1957" reviewComment="" reviewStatus="" status="submitted_via_web" submissionTime="2021-07-02T08:16:24" submittedBy="bob" totalMedia="1">
<data id="transportation_2011_07_25" version="2014111">
<transport>
<available_transportation_types_to_referral_facility>none</available_transportation_types_to_referral_facility>
<loop_over_transport_types_frequency>
<ambulance></ambulance>
<bicycle></bicycle>
<boat_canoe></boat_canoe>
<bus></bus>
<donkey_mule_cart></donkey_mule_cart>
<keke_pepe></keke_pepe>
<lorry></lorry>
<motorbike></motorbike>
<taxi></taxi>
<other></other>
</loop_over_transport_types_frequency>
</transport>
<image1 type="file">1335783522563.jpg</image1>
<meta>
<instanceID>uuid:5b2cc313-fc09-437e-8149-fcd32f695d41</instanceID>
</meta>
</data>
<linked-resources>
<attachments>
<id>50</id>
<name>1335783522563.jpg</name>
<xform>574</xform>
<filename>bob/attachments/574_transportation_2011_07_25/1335783522563.jpg</filename>
<instance>1957</instance>
<mimetype>image/jpeg</mimetype>
<download_url>/api/v1/files/50?filename=bob/attachments/574_transportation_2011_07_25/1335783522563.jpg</download_url>
<small_download_url>/api/v1/files/50?filename=bob/attachments/574_transportation_2011_07_25/1335783522563.jpg&amp;suffix=small</small_download_url>
<medium_download_url>/api/v1/files/50?filename=bob/attachments/574_transportation_2011_07_25/1335783522563.jpg&amp;suffix=medium</medium_download_url>
</attachments>
</linked-resources>
</submission-item>
<submission-item>
...
</submission-item>
</submission-batch>

Get FLOIP flow results for a specific form
------------------------------------------
Provides a list of rows of submitted data for a specific form. Each row contains 6 values as specified |FLOIPSubmissionAPI|. The data is accessed from the data endpoint by specifiying the header ``Accept: "application/vnd.org.flowinterop.results+json"``.
Expand Down Expand Up @@ -264,7 +326,7 @@ Response

Paginate data of a specific form
---------------------------------
Returns a list of json submitted data for a specific form using page number and the number of items per page. Use the ``page`` parameter to specify page number and ``page_size`` parameter is used to set the custom page size.
Returns a list of JSON or XML submitted data for a specific form using page number and the number of items per page. Use the ``page`` parameter to specify page number and ``page_size`` parameter is used to set the custom page size.

- ``page`` - Integer representing the page.
- ``page_size`` - Integer representing the number of records that should be returned in a single page.
Expand All @@ -274,8 +336,8 @@ There are a few important facts to note about retrieving paginated data:
1. The maximum number of items that can be requested in a page via the ``page_size`` query param is 10,000
2. Information regrading transversal of the paginated responses can be found in `the Link header <https://tools.ietf.org/html/rfc5988>`_ returned in the response. *Note: Some relational links may not be present depending on the page accessed i.e the ``first`` relational page link won't be present on the first page response*

Example
^^^^^^^^
JSON Example
^^^^^^^^^^^^^
::

curl -X GET https://api.ona.io/api/v1/data/328.json?page=1&page_size=4
Expand All @@ -292,7 +354,7 @@ Sample response with link header
...
Link: <http://localhost:8000/api/v1/data/2?page=2&page_size=1>; rel="next", <http://localhost:8000/api/v1/data/2?page=3&page_size=1>; rel="last"

**Response:** ::
**JSON Response:** ::

[
{
Expand Down
51 changes: 32 additions & 19 deletions onadata/apps/api/tests/viewsets/test_data_viewset.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2598,20 +2598,16 @@ def test_data_query_ornull(self):

def test_data_list_xml_format(self):
"""Test DataViewSet list XML"""
# create form
xls_file_path = os.path.join(
os.path.dirname(os.path.abspath(__file__)),
"../fixtures/tutorial/tutorial.xls"
)
self._publish_xls_file_and_set_xform(xls_file_path)

# create submission
xml_submission_file_path = os.path.join(
os.path.dirname(os.path.abspath(__file__)),
"..", "fixtures", "tutorial", "instances",
"tutorial_2012-06-27_11-27-53_w_uuid.xml"
)
self._make_submission(xml_submission_file_path)
media_file = "1335783522563.jpg"
self._make_submission_w_attachment(os.path.join(
self.this_directory, 'fixtures',
'transportation', 'instances', 'transport_2011-07-25_19-05-49_2',
'transport_2011-07-25_19-05-49_2.xml'),
os.path.join(self.this_directory, 'fixtures',
'transportation', 'instances',
'transport_2011-07-25_19-05-49_2', media_file))

view = DataViewSet.as_view({'get': 'list'})
request = self.factory.get('/', **self.extra)
formid = self.xform.pk
Expand All @@ -2625,13 +2621,30 @@ def test_data_list_xml_format(self):
instance = self.xform.instances.first()
returned_xml = response.content.decode('utf-8')
server_time = ET.fromstring(returned_xml).attrib.get('serverTime')
edited = instance.last_edited is not None
submission_time = instance.date_created.strftime(MONGO_STRFTIME)
attachment = instance.attachments.first()
expected_xml = (
f'<?xml version="1.0" encoding="utf-8"?>\n<submission-batch serverTime="{server_time}">' # noqa
f'<submission-item dateCreated="{instance.date_created.isoformat()}" formVersion="{instance.version}" lastModified="{instance.date_modified.isoformat()}" objectID="{instance.id}">' # noqa
'<tutorial id="tutorial"><name>Larry\n Again</name><age>23</age><picture>1333604907194.jpg</picture>' # noqa
'<has_children>0</has_children><gps>-1.2836198 36.8795437 0.0 1044.0</gps><web_browsers>firefox chrome safari' # noqa
'</web_browsers><meta><instanceID>uuid:729f173c688e482486a48661700455ff</instanceID></meta></tutorial>' # noqa
'</submission-item></submission-batch>')
'<?xml version="1.0" encoding="utf-8"?>\n'
f'<submission-batch serverTime="{server_time}">' # noqa
f'<submission-item bambooDatasetId="" dateCreated="{instance.date_created.isoformat()}" duration="" edited="{edited}" formVersion="{instance.version}"' # noqa
f' lastModified="{instance.date_modified.isoformat()}" mediaAllReceived="{instance.media_all_received}" mediaCount="{ instance.media_count }" objectID="{instance.id}" reviewComment="" reviewStatus=""' # noqa
f' status="{instance.status}" submissionTime="{submission_time}" submittedBy="{instance.user.username}" totalMedia="{ instance.total_media }">' # noqa
f'<transportation id="{instance.xform.id_string}" version="{instance.version}">' # noqa
'<transport>'
'<available_transportation_types_to_referral_facility>none</available_transportation_types_to_referral_facility>' # noqa
'<loop_over_transport_types_frequency><ambulance></ambulance><bicycle></bicycle><boat_canoe></boat_canoe><bus></bus><donkey_mule_cart></donkey_mule_cart><keke_pepe></keke_pepe><lorry></lorry><motorbike></motorbike><taxi></taxi><other></other></loop_over_transport_types_frequency>' # noqa
'</transport>'
'<image1 type="file">1335783522563.jpg</image1>'
'<meta><instanceID>uuid:5b2cc313-fc09-437e-8149-fcd32f695d41</instanceID></meta>' # noqa
'</transportation>'
'<linked-resources>'
'<attachments>'
f'<id>{attachment.id}</id><name>1335783522563.jpg</name><xform>{instance.xform.id}</xform><filename>{ attachment.media_file.name }</filename><instance>{ instance.id }</instance><mimetype>image/jpeg</mimetype><download_url>/api/v1/files/{attachment.id}?filename={ attachment.media_file.name }</download_url><small_download_url>/api/v1/files/{attachment.id}?filename={ attachment.media_file.name }&amp;suffix=small</small_download_url><medium_download_url>/api/v1/files/{attachment.id}?filename={ attachment.media_file.name }&amp;suffix=medium</medium_download_url></attachments>' # noqa
'</linked-resources>'
'</submission-item>'
'</submission-batch>'
)
self.assertEqual(expected_xml, returned_xml)

def test_invalid_xml_elements_not_present(self):
Expand Down
35 changes: 35 additions & 0 deletions onadata/libs/serializers/data_serializer.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,9 @@
from onadata.apps.logger.models.instance import Instance, InstanceHistory
from onadata.apps.logger.models.xform import XForm
from onadata.libs.serializers.fields.json_field import JsonField
from onadata.libs.utils.common_tags import (
METADATA_FIELDS, NOTES, TAGS, DATE_MODIFIED, VERSION, GEOLOCATION,
XFORM_ID, ATTACHMENTS, XFORM_ID_STRING, UUID)
from onadata.libs.utils.logger_tools import remove_metadata_fields
from onadata.libs.utils.dict_tools import (dict_lists2strings, dict_paths2dict,
query_list_to_dict,
Expand Down Expand Up @@ -110,6 +113,15 @@ class Meta:
model = Instance
fields = ('xml', )

def _convert_metadata_field_to_attribute(self, field: str) -> str:
"""
Converts a metadata field such as `_review_status` into
a camel cased attribute `reviewStatus`
"""
split_field = field.split('_')[1:]
return split_field[0] + ''.join(
word.title() for word in split_field[1:])

def to_representation(self, instance):
ret = super(
DataInstanceXMLSerializer, self).to_representation(instance)
Expand All @@ -124,6 +136,29 @@ def to_representation(self, instance):
'@objectID': str(instance.id)
}
ret.update(instance_attributes)
excluded_metadata = [
NOTES, TAGS, GEOLOCATION, XFORM_ID, DATE_MODIFIED, VERSION,
ATTACHMENTS, XFORM_ID_STRING, UUID]
additional_attributes = [
(self._convert_metadata_field_to_attribute(metadata), metadata)
for metadata in METADATA_FIELDS
if metadata not in excluded_metadata]
for attrib, meta_field in additional_attributes:
meta_value = instance.json.get(meta_field, "")
if not isinstance(meta_value, str):
meta_value = str(meta_value)
ret.update({
f"@{attrib}": meta_value
})

# Include linked resources
linked_resources = {
'linked-resources': {
'attachments': instance.json.get(ATTACHMENTS),
'notes': instance.json.get(NOTES)
}
}
ret.update(linked_resources)
return ret


Expand Down

0 comments on commit 472b466

Please sign in to comment.