Accessing multiple files in a multipart request - mule

Most of the documentation in Mule regarding multipart relates to making a request and not processing an incoming request.
I'm looking to receive multiple files in an incoming request as an array and then loop through them to upload to another service (Salesforce).
Due to the use case I'm sending two properties:
data which is metadata to create a case record
files which is an array of files to be associated with the case record
My Postman request is:
This is the payload I receive in a Mule 4 application (redacted the file data itself):
----------------------------778704367722595657997650
Content-Disposition: form-data; name="data"
Content-Type: application/json
[
{
"subject": "test
}
]
----------------------------778704367722595657997650
Content-Disposition: form-data; name="files"; filename="logo.jpeg"
Content-Type: image/jpeg
jpegdata
----------------------------778704367722595657997650
Content-Disposition: form-data; name="files"; filename="icon.png"
Content-Type: image/png
pngdata
----------------------------778704367722595657997650--
I attempt to get the files with Dataweave:
payload.parts.files
But this returns only the first file 'logo.jpeg'
I've attempted to send the files as files[] but it doesn't seem to send as an array.
It seems that I should loop through the boundary instead? If so, should my request set the boundary so it's the same each time?
Or is there a different way to access the files?
Secondarily, is multipart the correct method or should I utilize application/json instead?
The use case is to send files from a contact form in another application to Salesforce.

When DataWeave parses a multi part input it outputs an object in which each key is the name of the part. Your input has two entries with the same key name. As usual in DataWeave you should use the multi valued selector to get all the values with the same key: payload.parts.*files

Related

Multipart/form-data broken after apiKit router Mule4

having issues with reading data from multipart/form-data after the request goes through ApiKit router in Mule4 app.
There is a RAML in place, used in ApiKit router to validate & route the requests.
#%RAML 1.0
title: ACC race data API
description: API for designing and updating race data
version: 1
protocols: [ HTTPS, HTTP ]
/acc/reverse/entrylist:
description: Used to manipulate entrylist for ACC reverse grid.
post:
description: Creates new entrylist based on supplied quali result, race one result and race one entrylist.
queryParameters:
positionsRotated:
description: Parameter defining how many positions to rotate.
required: false
type: number
example: 30
body:
multipart/form-data:
properties:
qualifyResult:
description: File containing qualifying result.
type: file
fileTypes: ['application/json']
required: true
raceOneResult:
description: File containing race 1 result.
type: file
fileTypes: ['application/json']
required: true
The validation works fine - if the data are sent in incorrect format, the exception is returned. For correctly formated data request is routed to the flow post:\acc\reverse\entrylist:multipart\form-data:acc-race-data-config.
Between the steps, payload format is changed (by Mule) from payload in readable form (see below) to java.io.ByteArrayInputStream#379ebdd5
Readable format of payload data (received to the app) before ApiKit router:
----------------------------180928595588258919887097
Content-Disposition: form-data; name="qualifyResult"; filename="json1.json"
Content-Type: application/json
{
"json1": "1"
}
----------------------------180928595588258919887097
Content-Disposition: form-data; name="raceOneResult"; filename="json2.json"
Content-Type: application/json
{
"json2": "2"
}
----------------------------180928595588258919887097--
The following dataweave script works fine if used before apiKit, but it doesn't work in the flow called by ApiKit:
%dw 2.0
output application/json
---
payload.parts[1].content
The example of output if above DW is used before ApiKit:
{
"json2": "2"
}
The example of output if the same DW is used after ApiKit:
org.mule.runtime.core.api.expression.ExpressionRuntimeException: "javax.mail.internet.ParseException - Missing start boundary, while reading `payload` as MultiPart.
Trace:
at main (Unknown)" evaluating expression: "%dw 2.0
output application/json
---
payload.parts[1].content".
Testing in Anypoint Studio 7.8.0, supposed to be used on Mule4-CE runtime once finished and ready to deploy.
Using Postman v8.5.1 for testing. sending form-data body with qualifyResult and raceOneResult parts containing JSON data, default headers, basic auth, query param positionsRotated=30.
Url called: https://localhost:443/api/acc/reverse/entrylist?positionsRotated=30
Tried to generate multipart/form-data body manually using a RAW type of payload, but the results were the same. Everything works fine if there is no ApiKit.. but I would like to use it to validate request validity.
Thanks to everyone replying for any useful hints!
I tested a similar scenario I had with the latest versions and I didn't had that problem. If you are using older versions of the HTTP connector and APIKit module try upgrading to the latest releases. Anypoint Studio has a feature to detect newer releases of connectors: https://docs.mulesoft.com/studio/7.9/update-modules

uploading large files in tornado using #stream_request_body

#stream_request_body
class StreamHandler(RequestHandler):
def post(self):
self.temp_file.close()
def prepare(self):
max_buffer_size = 4 * 1024**3 # 4GB
self.request.connection.set_max_body_size(max_buffer_size)
self.temp_file = open("test.txt","w")
def data_received(self, chunk):
self.temp_file.write(chunk)
With the above code I am able to upload file but in raw data form as shown below
-----------------------------6552719992117258671800152707
Content-Disposition: form-data; name="dest"
csv
-----------------------------6552719992117258671800152707
Content-Disposition: form-data; name="carrier"
bandwidth
-----------------------------6552719992117258671800152707
Content-Disposition: form-data; name="file1"; filename="test.csv"
Content-Type: text/csv
And the content of uploaded file follows here.
How do I get the request parameters parsed from the file and separate the data of the file uploaded?
Is there any other way to upload large files(around 2 GB) in tornado
This is the multipart protocol used by HTML forms. Tornado can currently only parse this if it sees all the data at once, not in a streaming upload. There's a third-party library that should be able to handle this: https://github.com/siddhantgoel/streaming-form-data. See this issue for more: https://github.com/tornadoweb/tornado/issues/1842
If you control the client, it may be simpler to use a plain HTTP PUT instead of the HTML multipart form protocol. This doesn't require any special handling on the server side

Onenote API (REST) - PATCH append - "must include a 'commands'" error when Commands is already supplied (?!)

Note: I'm pretty sure nothing's wrong with the PATCH query, I had it working before with 'Content-type':'application/json' and a constructed json file:
[
{
'target':'|TARGET_ID|',
'action':'append',
'content':'|HTML|'
}
]
For the purposes of this, the header supplied (authentication bearer is correct and will be omitted)
'Content-type':'multipart/form-data; Boundary=sectionboundary'
(note: Boundary=sectionboundary is in the same line)
Attempting to pass the following body as a PATCH to
https://www.onenote.com/api/v1.0/pages/|GUID|/content
returns a
"code":"20124","message":"A multi-part PATCH request must include a 'commands' part containing the PATCH action JSON structure." :
--sectionboundary
Content-Disposition: form-data; name="Commands"
Content-Type: application/json
[
{
'target':'|TARGET_ID|',
'action':'append',
'content':'|HTML|'
}
]
--sectionboundary
Content-Disposition: form-data; name="image-part-name"
Content-Type: image/png
|BINARY_IMAGE_DATA|
--sectionboundary--
As you can see, there's a Commands section already. Using smallcaps 'commands' doesn't help, and the correct syntax should be "Commands" as per the OneNote Dev Center documentation.
PS: |TARGET_ID| |HTML| |GUID| and |BINARY_DATA| are replaced with the correct content at runtime. Due to privacy constraints, the fact that you may use a different schema than I do, and how long |BINARY_IMAGE_DATA| actually is, I will not show the actual input unless required to solve the problem.
Would like to know if I missed anything - thanks in advance.
PPS: Yes, I realize i've omitted the img tag inside |HTML| somewhere. It shouldn't have anything to do with code 20124, and if I got it wrong should return another thing entirely.
Based on investigating the request information you shared, I can confirm that the PATCH request referenced as part of the correlation you provided does not match your posted header information.
The correlated PATCH request shows up as a multi-part request with only a single part that has Media Type "TEXT/HTML" and not "Application/JSON". Can you please check and confirm your request content ?
Let us continue to discuss this on email. If you still face issues calling the API, please write to me at machandw#microsoft.com
Regards,
Manoj

Encoding Issue apending NSData to Post

I have got problem uploading a file to a server through POST
I build the request body by loading a html file and inserting the data:
------WebKitFormBoundaryMAsmIFCr12izlsoH
Content-Disposition: form-data; name="token"
%#
------WebKitFormBoundaryMAsmIFCr12izlsoH
Content-Disposition: form-data; name="userfile"; filename="%#"
Content-Type: AES256/encrypted
%#
------WebKitFormBoundaryMAsmIFCr12izlsoH--
Content-Disposition: form-data; name="submit"
Datei absenden
------WebKitFormBoundaryMAsmIFCr12izlsoH--
But to insert the the data fia string with format, it must be a string. So I alloc initWithData:data encoding NSASCIIStringEncoding cause the data can not be displayed in UTF8. The rest of the request is in UTF8, so afterwards if need dataUsingEncoding:NSUTF8StringEncoding. This breaks the data. There some bytes on the wrong place and so on. So I tried an other way, I spitted the html into two parts, and made a sandwich of it. html, data, html Unfortunately the request now looks like the second html is part of the file, so the file is broken again.
Do you know a better method?
Thanks in advance,
Jannes
I think trying to use stringWithFormat is fundamentally flawed. NSString is conceptually an array of Unicode data, but the data you are trying to put together may not be valid Unicode.
You'd be better off converting the various header portions to NSData instances and then concatenating those with the data you want to send.

Fiddler add binary file data to POST

I'm try to add binary file data directly to the request body of a POST call so I can simulate a file upload. However, I tried setting a 'before request' breakpoint and using 'insert file' but I couldn't seem to get that to work. I also tried to modify CustomRules.js to inject the file but couldn't figure out how to load binary data via JScript. Is there an easy solution here?
I'm sure this is a new feature in the year since this question was answered, but thought I'd add it anyhow:
There's a blue "[Upload file]" link in Composer now on the right side under the URL textbox. This will create a full multipart/form-data request. If you use this, you'll notice in the body you now have something that looks like this:
<#INCLUDE C:\Some\Path\my-image.jpg#>
In my case, I just wanted to POST the binary file directly with no multipart junk, so I just put the <#INCLUDE ... #> magic in the request body, and that sends the binary file as the body.
In order to send multipart/form-data, this receipe will be helped.
In upper panel (Http header), set Content-Type as below. Other values are automatically resolved.
Content-Type: multipart/form-data; boundary=-------------------------acebdf13572468
And, input Response Body at the below panel as follows.
---------------------------acebdf13572468
Content-Disposition: form-data; name="description"
the_text_is_here
---------------------------acebdf13572468
Content-Disposition: form-data; name="file"; filename="123.jpg"
Content-Type: image/jpg
<#INCLUDE *C:\Users\Me\Pictures\95111c18-e969-440c-81bf-2579f29b3564.jpg*#>
---------------------------acebdf13572468--
The import rules are,
Content-Type should have two more - signs than boundary words in body.
The last of the body should be ended with two - signs.
In Fiddler script: (in Fiddler: Rules... Customize Rules), find the OnBeforeRequest function, and add a line similar to:
if (oSession.uriContains("yourdomain"))
{
oSession.LoadRequestBodyFromFile("c:\\temp\\binarycontent.dat");
}
since version 2.0, the Request Body has an "Upload File..." link that allows you to post/upload binary data.