Stream JSON responses #164
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Note that:
|
dracos
force-pushed
the
stream-responses
branch
2 times, most recently
from
9ebcbdd
to
0466f06
Compare
| def output_html(request, title, areas, **kwargs): | ||
| kwargs['json_url'] = request.get_full_path().replace('.html', '') | ||
| kwargs['title'] = title | ||
| kwargs['areas'] = sorted_areas(areas) | ||
| kwargs['indent_areas'] = kwargs.get('indent_areas', False) |
There was a problem hiding this comment.
Wasn't this the line you decided to chop?
There was a problem hiding this comment.
Yeah, but then I need to pass it in the template_area iterator so thought I might as well leave it for that.
There was a problem hiding this comment.
But it doesn't need to be in kwargs so changed for the below.
|
Aside from those little inline comments, 👍 |
Create an iterator of the callback header, content, and footer. Both HttpResponse and StreamingHttpResponse handle this fine.
We subclass dict so that we can create an iterator that json will use.
This works around https://code.djangoproject.com/ticket/24240 which otherwise dies when trying to cope with a StreamingHttpResponse containing unicode strings.
This works around https://code.djangoproject.com/ticket/24242 so that gzipped streamed responses aren't larger than the original content.
It does this in quite a horrible way, by putting a flag in the template, and then inserting the iterable of areas in place of that flag. Also make sure we cache template loading. Thanks to Dave Arter for the defaultiter implementation.
dracos
force-pushed
the
stream-responses
branch
from
babcbaa
to
b19f2f9
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When we start, we have an iterable
QuerySet. So this changesadd_codesto be iterable, passes aniterdict[1] rather than adicttooutput_json, changesoutput_jsonto use aStreamingHttpResponseby default, and updates the callback middleware to work with streaming or non-streaming responses. This hopefully reduces memory usage for one particular request from c. 2Gb to 200Mb. Similarly foroutput_html, we keep everything as an iterable, doing something horrible with a template in order to put our iteration in the middle.[1] An
iterdictis adictsubclass that does nothing but provide an iterable foritems/iteritems, in order so that thejsonpackage will think it's a dictionary, iterate over it and output an iterator. Otherwise we'd need to load everything into a dictionary in memory to iterate it all back out...