Enhance performance when exporting data on endpoint api/v1/data/<form_id>.<format>
#2460
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kelvin-muchiri
changed the title
kelvin-muchiri
changed the title
api/v1/data/<form_id>
kelvin-muchiri
force-pushed
the
data-viewset-export-performance
branch
from
f843300
to
93d9368
Compare
kelvin-muchiri
requested review from
FrankApiyo,
ukanga,
KipSigei and
ciremusyoka
and removed request for
FrankApiyo and
ukanga
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
kelvin-muchiri
changed the title
api/v1/data/<form_id>api/v1/data/<form_id>.<format>
kelvin-muchiri
force-pushed
the
data-viewset-export-performance
branch
from
24e4aac
to
0ec8593
Compare
the default ordering by id is making queries for run extremely slow when exporting large amounts of data.
to sort data by id in ascending order, the query parameter sort={"_id":1} will be used. For more info read https://github.com/onaio/onadata/blob/main/docs/data.rst#sort-submitted-data-of-a-specific-form-using-existing-fields
Notes are excluded when exporting the csv. The column _notes is usually added in the CSV but its always overriden to be blank as per the implmentation in the CSVDataFrameBuilder class. So this test case is futile
fix flaky tests by ensuring queryset is always ordered
fixing the rules would require alot of refactor so disabling the rules will suffice for now
kelvin-muchiri
force-pushed
the
data-viewset-export-performance
branch
from
1c48bd3
to
b25599a
Compare
KipSigei
reviewed
Aug 21, 2023
| @@ -815,6 +844,7 @@ def generate_external_export( # noqa C901 | |||
| filter_query = options.get("query") | |||
| meta = options.get("meta") | |||
| token = options.get("token") | |||
| sort = options.get("sort") | |||
There was a problem hiding this comment.
@kelvin-muchiri Could we add documentation for this change?
There was a problem hiding this comment.
@KipSigei Added the documentation
KipSigei
reviewed
Aug 21, 2023
| @@ -653,76 +658,114 @@ def list(self, request, *args, **kwargs): | |||
|
|
|||
| return Response(serializer.data) | |||
|
|
|||
| return custom_response_handler(request, xform, query, export_type) | |||
| return custom_response_handler(request, xform, query, export_type, sort=sort) | |||
There was a problem hiding this comment.
I dont think its necessary to have a new sort param since you can get it from request
KipSigei
approved these changes
Aug 23, 2023
Merged
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes / Features implemented
api/v1/data/<form_id>.<formatis making queries run extremely slow when exporting large amounts of data. To sort data by id in ascending order, the query parametersort={"_id":1}will be applied to the endpoint while to sort data by id in descending order the query parametersort={"_id"-1}. For further reference read Sort submitted data of a specific form using existing fieldsCSVDataFrameBuilder._query_datamethod which was querying the database twice despite data being passed toCSVDataFrameBuilder.to_flat_csv_export. Make use of the data passed to ensure lose coupling between the utility classCSVDataFrameBuilderand the code responsible for querying the data.onadata.apps.api.tests.viewsets.test_note_viewset.TestNoteViewSet.test_csv_export_form_w_notes. Notes are excluded when exporting the csv. The column_notesis usually added in the CSV but the value in each record is always overriden to be blank as per the implementation in theCSVDataFrameBuilderclass. So this test case is futileSteps taken to verify this change does what is intended
Side effects of implementing this change
Faster when exporting data on endpoint
api/v1/data/<form_id>.csvBefore submitting this PR for review, please make sure you have:
Closes #