Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

implemented streaming parsing multipart/form-data as turbo takes 3x memory of uploading file(s) size during parsing multipart/form-data request #367

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

gary8520
Copy link

@gary8520 gary8520 commented Oct 4, 2023

isolated function which parse headers of multipart from parse_multipart_data() add kwargs.streaming_multipart_bytes in httpserver let user to parse multipart data in streaming and saved huge file (excess kwargs.large_body_bytes or 512 if no setting) to /tmp

the 3x memory of files as instance:
file about 100M so the multipart/form-data body is about 100M

  1. in function iostream.IOStream:_read_to_buffer() self._read_buffer:append_right(ptr, sz) would expand the buffer size to 100M
  2. in function iostream.IOStream:_consume(loc) chunk = ffi.string(ptr + self._read_buffer_offset, loc) converting/coping c string to lua string takes 100M
  3. in function httputil.parse_multipart_data(data, boundary) argument[1] = data:sub(v1, b2) slicing the huge file content as new string takes 100M

the solution contains in the PR as below:

  1. streaming parsing will parse every chunk to prevent to expand buffer size
  2. parsing chunk with raw buffer struct to prevent to convert lua string
  3. saving the hug file under /tmp in chunk as in handler user can os.rename() the file under /tmp to where they want

Testing result:

$ luajit examples/multipart.lua

response ok and the file uploaded/received are identical

image

image

isolated function which parse headers of multipart from parse_multipart_data()
add kwargs.streaming_multipart_bytes in httpserver let user to parse multipart data in streaming
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant