Ensure Pagination and sorting are implemented on the data endpoint #2113
Conversation
WinnyTroy
force-pushed
the
2108_sort_paginated_resonse
branch
5 times, most recently
from
b6ae263
to
4442ae1
Compare
WinnyTroy
changed the title
WinnyTroy
force-pushed
the
2108_sort_paginated_resonse
branch
2 times, most recently
from
a7fcdd9
to
20303c8
Compare
WinnyTroy
force-pushed
the
2108_sort_paginated_resonse
branch
from
20303c8
to
ad1b77e
Compare
WinnyTroy
force-pushed
the
2108_sort_paginated_resonse
branch
3 times, most recently
from
62964f3
to
e2ca6f7
Compare
| should_paginate: | ||
| # Unpack generator object to list | ||
| self.object_list = list(self.object_list) | ||
| else: |
There was a problem hiding this comment.
Seems like the pagination headers are skipped if the object_list is a generator object?
There was a problem hiding this comment.
Yes.
I assume, from the previously changes I observed a scenerio whereby data returned in JSON format was to be streamed, with no pagination.
I left the else condition to continue to handle such kinds of requests, i.e that the data returned is a generator objects and we do not want to paginate the returned response data.
A good example can be found on this test, we first return a generator object, and we dont want to paginate it, but use the limit flag instead.
There was a problem hiding this comment.
IMO we should always paginate requests if requested no matter if we're streaming the data... It's still going to be a pain if a user requests a lot of records while streaming data is enabled
There was a problem hiding this comment.
We also have a few tests that use pagination and stream data
onadata/onadata/apps/api/tests/viewsets/test_data_viewset.py
Lines 653 to 654 in e2ca6f7
WinnyTroy
force-pushed
the
2108_sort_paginated_resonse
branch
5 times, most recently
from
6e40760
to
9421052
Compare
| if isinstance(self.object_list, types.GeneratorType): | ||
| # Unpack generator object to list | ||
| self.object_list = list(self.object_list) | ||
| elif should_paginate: |
There was a problem hiding this comment.
What happens if it's a generator and the user asked for pagination ?
There was a problem hiding this comment.
It will mostly result to a generator object if sort query param is passed for the JSON result.
If page and page_size headers are passed, the result will be unpacked and paginated appropriately.
There was a problem hiding this comment.
Still seems like if it's a generator object the pagination headers won't be included.... Seems like it's unpacked but the pagination headers are not included
WinnyTroy
force-pushed
the
2108_sort_paginated_resonse
branch
2 times, most recently
from
01f4fd2
to
5f5ec80
Compare
| should_paginate: | ||
| if isinstance(self.object_list, types.GeneratorType): | ||
| # Unpack generator object to list | ||
| self.object_list = list(self.object_list) |
There was a problem hiding this comment.
This converts the stream to a list and breaks the stream data functionality
There was a problem hiding this comment.
Can we paginate when fetching from the db?
WinnyTroy
force-pushed
the
2108_sort_paginated_resonse
branch
3 times, most recently
from
29c157c
to
ef3c04b
Compare
WinnyTroy
force-pushed
the
2108_sort_paginated_resonse
branch
from
ef3c04b
to
9b85544
Compare
WinnyTroy
changed the title
Changes / Features implemented
Steps taken to verify this change does what is intended
Side effects of implementing this change
Before submitting this PR for review, please make sure you have:
Closes #2108