Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data viewset retrieval optimisations #1966

Merged
merged 5 commits into from
Dec 10, 2020
Merged

Conversation

DavisRayM
Copy link
Contributor

@DavisRayM DavisRayM commented Dec 9, 2020

Changes / Features implemented

  • Introduced new SUBMISSION_RETRIEVAL_THRESHOLD setting
  • Disable Queryset ordering after an XForms' submission count surpasses the SUBMISSION_RETRIEVAL_THRESHOLD setting
  • Disable ETag functionality after an XForms' submission count surpasses the SUBMISSION_RETRIEVAL_THRESHOLD setting
  • Add CountOverridablePaginator and CountOverridablePageNumberPagination
  • Switch the default pagination class for the Data viewset to CountOverridablePageNumberPagination

TODO

  • Set a limit on the number of records that can be returned at one time when pagination is not used; Separate PR... Will most likely break 3rd Party applications that rely on this functionality

Steps taken to verify this change does what is intended

  • N/A

Side effects of implementing this change

  • After a certain threshold data retrieval (even via pagination) is not ordered and the ETag functionality is disabled

deleted_at__isnull=True).values_list('id', flat=True)
num_of_submissions = sum(
merged_form.xforms.values_list(
'num_of_submissions', flat=True))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should the above also filter by deleted_at is not null i.e.

merged_form.xforms.filter(deleted_at__isnull=True).values_list('num_of_submissions', flat=True)

If the above is true, can the query in line 417 and 420 be merged into 1 to retrieve both the id and num_of_submissions ?

Comment on lines 428 to 434
# Disable ordering for XForms with Submissions that are more
# than the SUBMISSION_RETRIEVAL_THRESHOLD
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or

# Enable ordering for XForms with Submissions that are less
# than the SUBMISSION_RETRIEVAL_THRESHOLD

😄

…tion" classes

The two classes are custom classes derived from the Django's Paginator and
RestFrameworks' PageNumberPagination that allow the count property on
the paginator to be overriden by a passed in value
Disable ETag functionality for forms that have submissions that surpass
the "SUBMISSION_RETRIEVAL_THRESHOLD" setting
@DavisRayM DavisRayM force-pushed the data-retrieval-improvements branch 3 times, most recently from 9116add to 1412968 Compare December 10, 2020 07:21
Disable submission ordering when a Form has submissions that surpass a
configurable "SUBMISSION_RETRIEVAL_THRESHOLD" limit
@DavisRayM DavisRayM merged commit 427844a into master Dec 10, 2020
@DavisRayM DavisRayM deleted the data-retrieval-improvements branch December 10, 2020 11:42
@DavisRayM DavisRayM mentioned this pull request Dec 10, 2020
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants