Generate HTTP POST form with multipart-form-data without curl - http-headers

So I'm trying to generate HTTP POST form in my embedded application. However I get server 400 error that indicates that something is wrong with my post. I do not have any curl-like libraries, or such, so I need to form the post header from scratch.
const static char *post_header = "POST /v1/avs/speechrecognizer/recognize HTTP/1.1\r\n\
Host: access-alexa-na.amazon.com\r\n\
Authorization: Bearer %s\r\n\
Content-Type: multipart/form-data; boundary=BOUNDARY1234\r\n\
Transfer-Encoding: chunked\r\n\
Content-Length: %d\r\n\
\r\n\r\n\
--BOUNDARY1234\r\n\
Content-Disposition: form-data; name=\"metadata\"\r\n\
Content-Type: application/json; charset=UTF-8\r\n\
\r\n\
{\"messageHeader\": {},\"messageBody\": {\"profile\": \"alexa-close-talk\",\"locale\": \"en-us\",\"format\": \"audio/L16; rate=16000; channels=1\"}}\r\n\
\r\n\r\n\
--BOUNDARY1234\r\n\
Content-Disposition: form-data; name=\"audio\"\r\n\
Content-Type: audio/L16; rate=16000; channels=1\r\n\n";
After the last "\n" I have the wav header and payload itself. I don't have null termination between the wav header and the last request header content. Altough I've tried it and it doesn't seem to make any difference.
My authentication token should be OK (I've verified it with curl). I've used these scripts (https://miguelmota.com/blog/alexa-voice-service-with-curl/) and Amazon documentation as a base. The blogpost has a script that generates multipart payload and it's identical (compared binary dumps) to mine. My only obvious questionmarks are the first part of the query:
"POST /v1/avs/speechrecognizer/recognize HTTP/1.1\r\n\
Host: access-alexa-na.amazon.com\r\n\
Authorization: Bearer %s\r\n\
Content-Type: multipart/form-data; boundary=BOUNDARY1234\r\n\
Transfer-Encoding: chunked\r\n\
Content-Length: %d\r\n\
\r\n\r\n\"
and the curl call with especially the --data-binary part. Should it effect the request body shomehow?
curl -X POST \
-H "Authorization: Bearer ${TOKEN}" \
-H "Content-Type: multipart/form-data; boundary=${BOUNDARY}" \
--data-binary #multipart_body.txt \
https://access-alexa-na.amazon.com/v1/avs/speechrecognizer/recognize \
> response.txt
Any ideas anyone? I'm gettin a bit frustrated with this.
EDIT 1: Just to clarify. The total size of the data is about 200kbytes with the audio data included. The header size with the token is about 1200bytes. I'm sending the stuff in 1k blobs and I get the error after 4k or so. So I don't manage to send the whole thing before the server responds with the error. Also some of the similar cases in Amazon side indicates that 400 in this case points to problem with the header. However they aren't manually forming the posts so I cannot see the whole thing anywhere.
EDIT2:
Also as this is chunked data, I wonder how it affects this?
I mean if I fe chunk the header into parts defined by the --BOUNDARY1234 and max of 512 bytes, how would that work? I mean fe:
--BOUNDARY1234\r\n\ Content-Disposition: form-data; name=\"metadata\"\r\n\ Content-Type: application/json;
charset=UTF-8\r\n\ \r\n\ {\"messageHeader\": {},\"messageBody\":
{\"profile\": \"alexa-close-talk\",\"locale\": \"en-us\",\"format\":
\"audio/L16; rate=16000; channels=1\"}}\r\n\ \r\n\r\n\
Should the there be chunk size right in the start of the transfer before the --BOUNDARY1234 or does the "Content-Disposition" or "Content-Type" affect this somehow? Or should I add the chunk size only to binary payload? Problem here is that the max send block with my HW is 1k. And the total header size is ~1,5k.

Related

cURL vs MIME: POSTing a file

Question
I have written a very simple API using Flask, and I would like to upload a file to it using a POST command. I can easily make it work using cURL, but not so much using a logic app.
I have been using the Mozilla MIME Guide trying to construct the HTTP call, but I am not sure what to use in the header and body.
What I know is:
I would like to be able to send any file type, so I think I have to use the following:
Content-Type: application/octet-stream
Content-Disposition: attachment; filename="filename.xxx"
I have my file encoded with Base64, so I need to write that somehow, and place it in the body
I would like to use chunking. Does this make any difference?
My API
from flask import Flask, request, redirect
app = Flask(__name__)
#app.route('/', methods=['POST'])
def print_hello():
if request.files:
request.files['file'].save("/home/ebbemonster/cool_file.txt")
return "Hello World"
return "Goodbye World"
if __name__=="__main__":
app.run(host='0.0.0.0')
cURL
curl -X POST 13.81.62.87:5000 -F file=#GH019654.MP4
Logic App
So I figured out how to convert a cURL POST to a HTTP header/body.
Fetching header/body details
# Logging post call
curl -X POST XX.XX.XX.XX:5000 -F file=#GH019654.MP4 --trace-ascii post.log
# Fetching header
head -n 30 post.log
=> Send header, 216 bytes (0xd8)
0000: POST / HTTP/1.1
0011: Host: XX.XX.XX.XX:5000
0029: User-Agent: curl/7.58.0
0042: Accept: */*
004f: Content-Length: 300745456
006a: Content-Type: multipart/form-data; boundary=--------------------
00aa: ----ec1aab65fb2d68fd
00c0: Expect: 100-continue
# Fetching body
sed -n '18,25p' post.log
0000: --------------------------ec1aab65fb2d68fd
002c: Content-Disposition: form-data; name="file"; filename="GH019654.
006c: MP4"
0072: Content-Type: application/octet-stream
009a:
009c: ....ftypmp41 ...mp41....mdatGPRO#...HD7.01.01.70.00LAJ9022436601
00dc: 517...................................1...US.8'.f..C328132710684
011c: 1.HERO7 Black........................E.....1...US.8'.f..........
# Fetching end of body
tail -n 30 google.log
02c2: --------------------------ec1aab65fb2d68fd--
== Info: HTTP 1.0, assume close after body
Logic App Header/Body

Postman shows "Could not get any response" even though response is OK

I have a WCF service which I make API requests to.
This API call returns a JSON response object and also is able to return it in GZIP compression as well when "gzip" value is used in "Accept-Encoding" header.
The problem is when I try to get the response in GZIP, Postman shows "Could not get any response" although I see response and response's content are OK (200 status code) in Fiddler and can easily decompress the response content in my C# client.
I took a look in Postman Console but all I see is "Error: incorrect header check".
I hardly tried to find any documentation regarding this header check but couldn't find any.
These are the request headers:
POST /correction/v1/document?lang=US HTTP/1.1
Content-Type: text/plain
Accept-Encoding: gzip
User-Agent: PostmanRuntime/7.6.0
Accept: */*
content-length: 630
Connection: close
These are the response headers:
HTTP/1.1 200 OK
Content-Length: 512
Content-Type: application/json; charset=utf-8
Content-Encoding: gzip
Server: Microsoft-HTTPAPI/2.0
Date: Sun, 24 Feb 2019 14:05:50 GMT
Connection: close
The only thing I suspect is wrong is this message from Fiddler:
I integrated this code into mine in order to use GZIP in WCF.
https://github.com/carlosfigueira/WCFSamples/tree/master/MessageEncoder/GZipEncoderAndAutoFormatSelection
Basically, it captures the response before returning to client and use GZipStream for compression.
I got the same issue, I added the following header to fix this issue.
Accept-Encoding : *
I was able to solve a similar issue by using the header Accept-Encoding: */* or if you want to be specific do Accept-Encoding: */* that way the HTTP client will be able to process the response based on the type of encoding received, in the case of a gzip, it will decode the response and show it as normal text.
For me, I removed 'Accept-Encoding' in the request header.
I got this issue when the REST service was returning a zip content (aka. WinZip format). I solved the error by compressing the data using 7zip to produce true gzip format.

Sending post request to API with multipart content-type

I am trying to send a post request though an api. The call requires:
content-type: multipart/form-data; boundary=[boundary_number]
I have used Charles HTTP proxy to watch see what headers/content I need to send.
My Request: (basically copied from Charles' multipart section)
--324a08fa-6b58-424a-a1ad-691123d9d04b
Content-Disposition: form-data; name="message[body]"
Content-Transfer-Encoding: binary
Content-Type: text/plain; charset=utf-8
Content-Length: 5
Text!
--324a08fa-6b58-424a-a1ad-691123d9d04b--
** My Headers:**
My Result:
When I post this in postman, the response just displays 'Loading'
I can't seem to satisfy the content I was wondering if anyone can point me in the right direction.
Any help would be greatly appreciated!
Cheers!
The response is not as expected because the Content-Length of HTTP request body is incorrect. It should be 240, not 238.
For HTTP request in Charles screenshot, the message[body] data is qqq, which is also indicated by Content-Length: 3. However, in your HTTP request in postman, the message[body] data is Text!. While its own Content-length is correct (5), the Content-length of the whole request body is not changed accordingly -- it's still 238, which should be increased to 240.

Attach a file (picture) to a conversation

I would like to be able to attach a file to a conversation, using the REST API. Is it possible? There is a «attachments» to /conversations/{convId}/messages/{itemId} but is that usable? How? The description of that field is not available.
The file API of Circuit supports the upload of attachments. As soon as you received your access token you can POST a message with the byte data. The following example wold upload a file with name test.jpg
POST /rest/v2/fileapi HTTP/1.1
Host: local.circuit.com
Authorization: Bearer <access token>
Content-Length: 100
Content-Disposition: attachment; filename="test.jpg"
Cache-Control: no-cache
<your content in binary form here>
Usually I am using Postman for my tests since it is very easy to use and supports OAuth 2.0 token generation (https://www.getpostman.com/)
You will receive an result that looks like
{"fileId":"fb211fd6-df53-4b82-824d-986dac47b3e7","attachmentId":"ZmIyMT..."}
If you want to validate your upload you can check it via
GET /rest/v2/fileapi?fileid=fb211fd6-df53-4b82-824d-986dac47b3e7 HTTP/1.1
Host: local.circuit.com
Authorization: Bearer <access token>
Cache-Control: no-cache
Well that was the easy part, now that you have uploaded the file to the backend you must attach it to a conversation item. Today we do not support UPDATE, i.e. you need to create a new one.
POST /rest/v2/conversations/<conv ID>/messages HTTP/1.1
Host: local.circuit.com
Authorization: Bearer <access token>
Content-Type: application/x-www-form-urlencoded
Cache-Control: no-cache
content=New+Text+Message&attachments=ZmIyMT...
You have to pass the generated attachment ID. After the execution of this requests the file is attached to the conversation.
If you skip the second step the file will not be linked to any conversation, is only accessible by the user who initiates the upload and will be deleted within the next 24 - 48h automatically.
Hope this helps, let me know if you have additional questions.

Including Content-Transfer-Encoding header in RestSharp multi-part messages?

I am using RestSharp to upload multi-part messages (basically, form data with a file attached at the end). The receiving service seems to require that I specify the Content-Transfer-Encoding header on each part or I get a 400 error on the POST.
I can't find any way in RestSharp to set this, or to specify "headers" that are actually inside the body of the parts. What I need to produce is a payload that looks like this:
POST http://server/object/create HTTP/1.1
User-Agent: RestSharp/105.2.3.0
Content-Type: multipart/form-data; boundary=-----------------------------28947758029299
Host: server:8080
Content-Length: 540
-------------------------------28947758029299
Content-Type: application/xml;charset=US-ASCII
Content-Disposition: form-data; name="form"
Content-Transfer-Encoding: 8bit
<form>this is some form data in XML format</form>
-------------------------------28947758029299
Content-Disposition: form-data; name="1.eml"; filename="1.email"
Content-Type: application/octet-stream; charset=ISO-8859-1
Content-Transfer-Encoding: binary
This is arbitrary file content to attach to the form.
-------------------------------28947758029299--
So far, this is what I've got (I'm prototyping in PowerShell but the results are identical in C#):
$request = New-Object RestSharp.RestRequest("iss", [RestSharp.Method]::POST)
$request.AddParameter("application/xml;charset=UTF-8", $form, [RestSharp.ParameterType]::RequestBody) | Out-Null
$request.AddFile("att.txt", "C:\Development\att.txt", "application/octet-stream; charset=ISO-8859-1") | Out-Null
$response = $rest.Execute($request)
which does everything I need except the Content-Transfer-Encoding part.
Is this possible with RestSharp, or should I drop down to use the HTTP client directly?