Parse multipart/form-data response in rust / reqwest - pdf

I'm relatively new to rust and using reqwest to fetch a PDF document from a REST API endpoint. The content-type of the response is multipart/form-data; boundary=bc1f6465-6738-4b46-9d9d-b9ae36afa8cb with two parts:
--bc1f6465-6738-4b46-9d9d-b9ae36afa8cb
Content-Disposition: form-data; name="metadata"
Content-Type: application/json
{"documentId":"QkNfRENfSUwwMDEsRTA1OEU3ODQtMDAwMC1DNzY5LTg1MjktMTRFRkI5RTBFNjRF"}
--bc1f6465-6738-4b46-9d9d-b9ae36afa8cb
Content-Disposition: form-data; name="document"; filename=document.pdf
Content-Type: application/pdf
%PDF-1.4
<binary content>
%%EOF
--bc1f6465-6738-4b46-9d9d-b9ae36afa8cb--
I want now to save the PDF document in the 2nd part as a valid PDF file on disk. But the multipart functionality within reqwest seems to create new multipart requests whereas I need to parse a multipart response.
My code to download the file:
use reqwest::{self, header::AUTHORIZATION};
fn main() {
let url = "https://example.com/rest/document/123";
let authorization_header = String::from("Bearer ") + access_token.as_str();
let res = client.get(url)
.header(AUTHORIZATION, &authorization_header)
.send()
.expect("Error calling API");
}
Any hint on how to process the multipart/form-data response is appreciated!

Related

Karate multipart file array uses deprecated multipart/mixed content-type

When I upload an array of files as below
Given path 'files', 'multiple'
And multipart file files = { read: 'test.pdf', filename: 'upload-name1.pdf', contentType: 'application/pdf' }
And multipart file files = { read: 'test.pdf', filename: 'upload-name2.pdf', contentType: 'application/pdf' }
When method post
Then status 202
The files are added using the multipart/mixed content type eg:
Content-Type: multipart/form-data; boundary=176626466ce00050
--176626466ce00050
content-disposition: form-data; name="files"
content-type: multipart/mixed; boundary=892503f32da73ceb
--892503f32da73ceb
content-disposition: attachment; filename="upload-name1.pdf"
content-type: application/pdf
content-transfer-encoding: binary
...
--892503f32da73ceb
content-disposition: attachment; filename="upload-name2.pdf"
content-length: 553202
content-type: application/pdf
content-transfer-encoding: binary
...
--892503f32da73ceb--
--176626466ce00050
multipart/mixed was defined in RFC2388 but was later deprecated in RFC7578 so some servers such as Jetty will not support it.
Is it possible to override the default behaviour and use multiple form-data Content-Dispositions instead?
I am using Karate 1.3.0 and am pretty sure this wasn`t an issue with earlier versions
I'm unable to simulate this, so maybe you should follow this process: https://github.com/karatelabs/karate/wiki/How-to-Submit-an-Issue
This is what I tried:
* url 'https://httpbin.org/post'
* multipart file foo1 = { read: 'test.pdf', contentType: 'application/pdf' }
* multipart file foo2 = { read: 'test.pdf', contentType: 'application/pdf' }
* method post
You can see from the response that the server detects 2 multi-part files and it all looks ok.
I also see Karate print this in place of the request payload:
Mixed: content-disposition: form-data; name="foo1"; filename="test.pdf"
content-type: application/pdf; charset=UTF-8
content-length: 6514
Completed: true
IsInMemory: true
Mixed: content-disposition: form-data; name="foo2"; filename="test.pdf"
content-type: application/pdf; charset=UTF-8
content-length: 6514
Completed: true
IsInMemory: true
So it looks ok to me or unable to replicate. We simply use Netty to build multi-part payloads. You are welcome to dig into the code and recommend (or contribute a PR) in case we are using Netty wrong or if Netty has a bug.
For completeness, 1.3.1 has an alternate way of supporting an array of files, so you can try this approach:
* url 'https://httpbin.org/post'
# just use the same name, and behind the scenes an array of multi-parts will be sent in the request body
* def first = { name: 'foo', read: 'test.pdf', contentType: 'application/pdf' }
* def second = { name: 'foo', read: 'test.pdf', contentType: 'application/pdf' }
# note how we support an array of files, which can be programmatically built
# here we use enclosed javascript (refer docs) for convenience, note the round-brackets
* multipart files ([ first, second ])
* method post

Spark Java - strange behavior when uploading files

In my project I want to try to upload files, here is the part of the code responsible for this:
MultipartConfigElement multipartConfigElement =
new MultipartConfigElement(
"/tmp_files",
avatarSize,
avatarSize,
1024
);
request.raw().setAttribute(
"org.eclipse.jetty.multipartConfig",
multipartConfigElement
);
Part uploadedFile = request.raw().getPart("file");
And a request to upload a file using Idea's http client:
POST http://localhost:8080/users/me/avatar
Content-Type: multipart/form-data; boundary=abcd
Authorization: Bearer {{authToken}}
--abcd
Content-Disposition: form-data; name="file"; filename="test.png"
< /Users/user1/resources/test.png
--abcd--
where test.png is a regular picture.
But when I try to load in this code place:
Part uploadedFile = request.raw().getPart("file");
I get an error:
java.nio.file.NoSuchFileException: /tmp_files/MultiPart11851484240893602177
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:218)
at java.base/java.nio.file.Files.newByteChannel(Files.java:375)
at java.base/java.nio.file.Files.createFile(Files.java:652)
It can be assumed that this error is due to the fact that there are no write permissions to the root of the file system (I'm testing on mac os, under the user).
But if i try to upload another file - which is just a zip file then everything works.
POST http://{{host}}/users/me/avatar
Content-Type: multipart/form-data; boundary=abcd
Authorization: Bearer {{authToken}}
--abcd
Content-Disposition: form-data; name="file"; filename="file123.zip"
< /Users/18493151/develop/icandev/api-gateway/src/main/resources/file123.zip
--abcd--
and no exception in this line:
Part uploadedFile = request.raw().getPart("file");
Why is this happening? Why does the result depend on the file type?
sparkjava version 2.9.4

Getting error in file upload using karate api

Could anyone please assist me to file upload functionality using Karate API? I have tried many ways, but getting error message as
"[{"title":"QUERY.BIZ.004","status":500,"detail":"Error in uploading document","timestamp":"2021-12-01T09:04:01.033+01:00"}]"
PAYLOAD DETAILS
metadata: {"key":"FILE_NAME","value":"karate-logo"}
metadata: {"key":"FILE_EXTENSION","value":"jpg"}
metadata: {"key":"TAG","value":"REQUEST"}
metadata: {"key":"DOC_TYP","value":"00008"}
file: (binary)
REQUEST HEADERS
Accept: application/json
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Connection: keep-alive
Content-Length: 2368
Content-Type: multipart/form-data; boundary=----WebKitFormBoundary9A1eYQihw4rdVq9f
Below mentioned karate API code which I used in the framework
Given url posturl
And path 'document'
And header id = '1608672'
And header Content-Type = 'multipart/form-data'
And multipart file file = { read: 'classpath:dataDrivenPayload/karate-logo.jpg', filename: 'karate-logo.jpg', contentType: 'image/jpg' }
And multipart field metadata = {"key":"FILE_NAME","value":"karate-logo"}, {"key":"FILE_EXTENSION","value":"jpg"}, {"key":"TAG","value":"REQUEST"}, {"key":"DOC_TYP","value":"00008"}
When method POST
Then status 200
I think the metadata needs to be sent as multiple "parts".
Try something like this:
* url 'https://httpbin.org/anything'
* multipart file metadata = { value: '{"key":"FILE_NAME","value":"karate-logo"}' }
* multipart file metadata = { value: '{"key":"FILE_EXTENSION","value":"jpg"}' }
* method post
* status 200
Otherwise, please use these instructions to troubleshoot, and work with your server-side team if possible: github.com/karatelabs/karate/issues/1645#issuecomment-862502881

Creating multipart/mixed request

I have a very specific case I'm trying to test with Karate.
PUT https://test-api.com/endpoint
Content-Type: multipart/mixed; boundary=BOUNDARY
--BOUNDARY
Content-Type: application/vnd.api+json
{"type": "json-api-object"}
--BOUNDARY
Content-Disposition: attachment; name="fieldname"; filename="filename.jpg"
Content-Type: image/jpeg
Content-Encoding: base64
<binary data>
--BOUNDARY--
Examples show multipart/mixed requests, but they don't show how to set the content-type header on each part. I tried using And multipart header... but that didn't parse correctly.
https://github.com/intuit/karate/blob/master/karate-demo/src/test/java/demo/upload/upload.feature
If I can get this figured out with your help, I'll make a PR against the examples to hopefully help someone in the future.
Yeah this is the first time I'm seeing a need for a custom content-type for each part. This will need a change in the code, so yes an example will expedite a fix.
Meanwhile you can customize the content-type for normal requests.
I had success using mulipart files
* configure headers = {"Content-Type": 'multipart/mixed'}
* configure charset = null
* def mFiles = {}
* set mFiles.jsondata = { value: {'type': 'json-api-object'}, contentType: 'application/vnd.api+json' }
* set mFiles.fieldname = { read: 'classpath:path/to/somePhoto.jpg', filename: 'filename.jpg', contentType: 'image/jpeg' }
Given url "https://my-api.com/put"
And multipart files mFiles
When method PUT
According to this comment, multipart file is preferred for anything other than a string in a multipart field.

Multi-Part upload has no payload

I am trying to do a multi-part upload from my javascript application and it works fine in the chrome browser but fails without error when run in electron.
I can't find information, why it fails in electron, but maybe some of you have an idea? :)
var fileContent = new Uint8Array([64,65,66,67]);
var fileBlob = new Blob([fileContent], {type: 'application/octet-stream'});
var data = new FormData();
data.set('file', fileBlob, 'blob.bin');
fetch('http://my.end.point', {
method: 'POST',
body: data
});
When I run that code in Chrome, I can see the 4 bytes payload in the network tab and my end point receives the 4 bytes. If I do the same in electron, the payload in the network tab is empty and the end point receives the multi-part request with the payload missing. I also tried using XMLHttpRequest instead of fetch but that shows exactly the same behavior.
Request Payload from within electron:
------WebKitFormBoundaryNXzPiiAvBttdDATr
Content-Disposition: form-data; name="file"; filename="blob.bin"
Content-Type: application/octet-stream
------WebKitFormBoundaryNXzPiiAvBttdDATr--
Request payload from withon chrome browser:
------WebKitFormBoundarywTEtXn4z3NFt3sAb
Content-Disposition: form-data; name="file"; filename="blob.bin"
Content-Type: application/octet-stream
#ABC
------WebKitFormBoundarywTEtXn4z3NFt3sAb--
Does someone know, why it doesn't work from within electron?