uploading large files in tornado using #stream_request_body - file-upload

Question

#stream_request_body
class StreamHandler(RequestHandler):
def post(self):
self.temp_file.close()
def prepare(self):
max_buffer_size = 4 * 1024**3 # 4GB
self.request.connection.set_max_body_size(max_buffer_size)
self.temp_file = open("test.txt","w")
def data_received(self, chunk):
self.temp_file.write(chunk)
With the above code I am able to upload file but in raw data form as shown below
-----------------------------6552719992117258671800152707
Content-Disposition: form-data; name="dest"
csv
-----------------------------6552719992117258671800152707
Content-Disposition: form-data; name="carrier"
bandwidth
-----------------------------6552719992117258671800152707
Content-Disposition: form-data; name="file1"; filename="test.csv"
Content-Type: text/csv
And the content of uploaded file follows here.
How do I get the request parameters parsed from the file and separate the data of the file uploaded?
Is there any other way to upload large files(around 2 GB) in tornado

Answer 1

This is the multipart protocol used by HTML forms. Tornado can currently only parse this if it sees all the data at once, not in a streaming upload. There's a third-party library that should be able to handle this: https://github.com/siddhantgoel/streaming-form-data. See this issue for more: https://github.com/tornadoweb/tornado/issues/1842
If you control the client, it may be simpler to use a plain HTTP PUT instead of the HTML multipart form protocol. This doesn't require any special handling on the server side