Enable request body streaming #12
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Previously we allowed only File and StringIO objects as an input to FormData::File, but we can generalize that to any IO object that responds to #read and #size (which includes Tempfile, ActionDispatch::Http::UploadedFile etc).
That way different operating systems won't attempt to convert newline characters to their internal representation, instead the file content will always be retrieved byte-for-byte as is.
Previously Pathname was implicitly supported, though extracting filename wasn't working. With the recent refactoring this stopped working, so we make the Pathname support explicit.
By changing all components to use an IO object as a base, we can implement a common IO interface for all components, which delegates to the underlying IO object. This enables streaming multipart data into the request body, avoiding loading the whole multipart data into memory when File parts are backed by File objects. See httprb/http#409 for the new streaming API.
By delegating handling strings to CompositeIO we can remove a lot of the StringIO.new clutter when instantiating CompositeIO objects.
|
I'm not able to further reduce cyclomatic complexity of |
|
@janko-m sure, or just increase the global cyclomatic complexity rating. That looks fine for what it is, which means the global limit is probably a bit too low. |
|
I'll review both http pr and this one over this week (give me couple of days - away from home atm) |
We increase it for FormData::CompositeIO#read.
This way we're not creating a new string for each chunk read, instead each chunk will be read into an existing string object (a "buffer"), replacing any previous content.
ixti
reviewed
May 7, 2017
lib/http/form_data/composite_io.rb
Outdated
| class CompositeIO | ||
| # @param [Array<IO>] ios Array of IO objects | ||
| def initialize(*ios) | ||
| @ios = ios.flatten.map { |io| io.is_a?(String) ? StringIO.new(io) : io } |
There was a problem hiding this comment.
Probably we should fail if io is neither String nor IO
There was a problem hiding this comment.
Also, CompositeIO seems to be be internal class, and I don't see any point in allowing passing *any amount of args... Instead it should simply accept a single argument as flat array, thus here we will just map it.
lib/http/form_data/multipart.rb
Outdated
| @@ -34,30 +32,19 @@ def content_type | |||
| # | |||
| # @return [Integer] | |||
| def content_length | |||
There was a problem hiding this comment.
Probably worth get rid of this method and use alias instead?
|
@ixti Updated 😃 |
ixti
pushed a commit
that referenced
this pull request
May 8, 2017
This is a squashed commit of [PR#12][] with some tiny cleanups applied on top of that. [PR#12]: #12 Allow any IO object in FormData::File ------------------------------------- Previously we allowed only File and StringIO objects as an input to `FormData::File`, but we can generalize that to any IO object that responds to `#read` and `#size` (which includes `Tempfile`, `ActionDispatch::Http::UploadedFile` etc). Open File for given path in binary mode --------------------------------------- That way different operating systems won't attempt to convert newline characters to their internal representation, instead the file content will always be retrieved byte-for-byte as is. Officially support Pathname in FormData::File.new ------------------------------------------------- Previously Pathname was implicitly supported, though extracting filename wasn't working. With the recent refactoring this stopped working, so we make the Pathname support explicit. Make all components into IO objects ----------------------------------- By changing all components to use an IO object as a base, we can implement a common IO interface for all components, which delegates to the underlying IO object. This enables streaming multipart data into the request body, avoiding loading the whole multipart data into memory when File parts are backed by File objects. See httprb/http#409 for the new streaming API. Make CompositeIO convert strings to StringIOs --------------------------------------------- By delegating handling strings to CompositeIO we can remove a lot of the StringIO.new clutter when instantiating CompositeIO objects. Use a buffer when reading IO files in CompositeIO ------------------------------------------------- This way we're not creating a new string for each chunk read, instead each chunk will be read into an existing string object (a "buffer"), replacing any previous content.
|
Merged via 5688433 |
|
@ixti Thank you 🎉 Once new version |
netbsd-srcmastr
pushed a commit
to NetBSD/pkgsrc
that referenced
this pull request
Dec 9, 2017
This version is required by newer ruby-http gem 3.0.0. Upstream changes: (from CHANGES.md) ## 2.0.0 (2017-10-01) * [#17](httprb/form_data#17) Add CRLF character to end of multipart body. [@mhickman][] ## 2.0.0.pre2 (2017-05-11) * [#14](httprb/form_data#14) Enable streaming for urlencoded form data. [@janko-m][] ## 2.0.0.pre1 (2017-05-10) * [#12](httprb/form_data#12) Enable form data streaming. [@janko-m][]
netbsd-srcmastr
pushed a commit
to NetBSD/pkgsrc
that referenced
this pull request
Mar 17, 2018
## 2.1.0 (2018-03-05) * [#21](httprb/form_data#21) Rewind content at the end of `Readable#to_s`. [@janko-m][] * [#19](httprb/form_data#19) Fix buffer encoding. [@HoneyryderChuck][] ## 2.0.0 (2017-10-01) * [#17](httprb/form_data#17) Add CRLF character to end of multipart body. [@mhickman][] ## 2.0.0.pre2 (2017-05-11) * [#14](httprb/form_data#14) Enable streaming for urlencoded form data. [@janko-m][] ## 2.0.0.pre1 (2017-05-10) * [#12](httprb/form_data#12) Enable form data streaming. [@janko-m][]
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request allows multipart data to be streamed into the request body using the new streaming API in HTTP.rb (httprb/http#409). The advantage of streaming is that, when File parts are backed by files on the filesystem, multipart data doesn't have to be loaded whole into memory before its sent as the request body, but rather only small parts will be loaded into memory at a time.
First I needed to modify
FormData::Fileto support any IO object, not justFileorStringIO, so that the logic can be simpler. I think this feature is useful in general, as explained in httprb/http#409 (comment).I also modified the File object to be opened in binary mode, to prevent any conversions of carriage returns and line feeds by different operating systems. This is not related to this PR, but I noticed it and wanted to do it so that I don't forget.
FormData::Filehas implicitly supportedPathnameobjects (which is used in tests), but the previous refactoring has forced me to make this support explicit. I think that's good, since some folks might be relying on it, and it's nice to supportPathnamefor file paths in general.Finally, I modified all components to be backed by an IO object, so that they can all implement a common IO interface (
FormData::Readable), enabling them to be composed usingCompositeIO. Most of theCompositeIOimplementation was borrowed frommultipart-post. I needed to include#rewindin the interface so that#to_scan be called multiple times.