I am using MSXML2.XMLHTTP60 to send text messages via VBA using a web server. I cannot understand why the € symbol is not displayed when receiving a text message. Other special characters, such as ò,à,è etc are displayed after a conversion function I wrote (for example à is encoded as "%E0"). I suppose that web server is expecting charset iso 8859-1 which doe not support € symbol. Therefore how can I solve this problem?
If your request is a POST request then you can specify header for Content-Type with encoding e.g. like this:
objHTTP.Open "POST", ...
objHTTP.setRequestHeader "Content-Type", "text/html; charset=utf-8"
But for GET request the URL with possible query string parameters will be encoded as ASCII. Read e.g. this post.
Using UTF-8 as your character encoding should solve such problems. It may also remove the need for your conversion function. I'm not sure how to set the encoding in your web server, but that's usually well documented.
Related
I have to test the below URL that contains special char:
url\search.html?q=*&Filters=template_fac_s:3f87222389034212a868c5c0cd12cacc|market_l1_fac_sm:e73956c2506442399dd73ed1eb0ec165
I am getting this error:
Response message: Non HTTP response message: Illegal character in query at index 121: url/search.html?q=*&Filters=template_fac_s:3f87222389034212a868c5c0cd12cacc|market_l1_fac_sm:e73956c2506442399dd73ed1eb0ec165
I think I need to encode the URL but I do not know how to do it . Does anyone know how to do this , can someone help.
Thanks.
JMeter provides __urlencode function which replaces characters which are not allowed in URL with url-encoded equivalents
References:
Understanding HTML Form Encoding: URL Encoded and Multipart Forms
Apache JMeter Functions - An Introduction
I'm trying to make calls to openstreetmap (specifically to https://nominatim.openstreetmap.org/search?amenity=charging_station&format=json&q=Elly-Beinhorn-Ring+2,12529+Schönefeld
) and from any webbrowser it works fine. When calling via Excel VBA is complains that the string is not UTF-8. I haven't used fiddler yet but to me it is clear that srequest should be in UTF-8 format. I found another solution to convert the url to ascii but I'd prefer to have it in utf-8 here's some sample code which creates the error:
Dim httpObject As Object
Set httpObject = CreateObject("MSXML2.XMLHTTP")
srequest = "https://nominatim.openstreetmap.org/search?amenity=charging_station&format=json&q=Elly-Beinhorn-Ring+2,12529+Schönefeld"
httpObject.Open "GET", srequest, False
httpObject.setRequestHeader "Content-Type", "application/x-www-form-urlencoded; charset=UTF-8"
httpObject.send
sgetResult = httpObject.responsetext
Debug.Print sgetResult
If anybody can help how to force MSXML2 to make an utf-8 call, i'd be happy. at the moment my solution is to replace German Umlauts (ä,ü,ö) with (ae,ue,oe) which works but is not what I want. Other solutions like URLEncode() have created more chaos by also converting + and , to values starting with %.
I could solve my problem by replacing this line:
Set httpObject = CreateObject("MSXML2.XMLHTTP")
with this one:
Set httpObject = CreateObject("MSXML2.ServerXMLHTTP")
Forcing MSXML2 to utf-8 is not the right way to go. All browsers convert to ASCII before sending the request. Easiest way to see this is in the Network tab in your browser Dev tools. Chrome translates your URL to
https://nominatim.openstreetmap.org/search?amenity=charging_station&format=json&q=Elly-Beinhorn-Ring+2,12529+Sch%C3%B6nefeld
So the easy way is to encode ö to %C3%B6 (as described here: https://www.fileformat.info/info/unicode/char/00f6/index.htm) and the other characters in a similar way. There are a lot of encoding methods available out there. Have a look
I've read conflicting and somewhat ambiguous replies to the question "How is a multipart HTTP request content length calculated?". Specifically I wonder:
What is the precise content range for which the "Content-length" header is calculated?
Are CRLF ("\r\n") octet sequences counted as one or two octets?
Can someone provide a clear example to answer these questions?
How you calculate Content-Length doesn't depend on the status code or media type of the payload; it's the number of bytes on the wire. So, compose your multipart response, count the bytes (and CRLF counts as two), and use that for Content-Length.
See: http://httpwg.org/specs/rfc7230.html#message.body.length
The following live example should hopefully answer the questions.
Perform multipart request with Google's OAuth 2.0 Playground
Google's OAuth 2.0 Playground web page is an excellent way to perform a multipart HTTP request against the Google Drive cloud. You don't have to understand anything about Google Drive to do this -- I'll do all the work for you. We're only interested in the HTTP request and response. Using the Playground, however, will allow you to experiment with multipart and answer other questions, should the need arise.
Create a test file for uploading
I created a local text file called "test-multipart.txt", saved somewhere on my file system. The file is 34 bytes large and looks like this:
We're testing multipart uploading!
Open Google's OAuth 2.0 Playground
We first open Google's OAuth 2.0 Playground in a browser, using the URL https://developers.google.com/oauthplayground/:
Fill in Step 1
Select the Drive API v2 and the "https://www.googleapis.com/auth/drive", and press "Authorize APIs":
Fill in Step 2
Click the "Exchange authorization code for tokens":
Fill in Step 3
Here we give all relevant multipart request information:
Set the HTTP Method to "POST"
There's no need to add any headers, Google's Playground will add everything needed (e.g., headers, boundary sequence, content length)
Request URI: "https://www.googleapis.com/upload/drive/v2/files?uploadType=multipart"
Enter the request body: this is some meta-data JSON required by Google Drive to perform the multipart upload. I used the following:
{"title": "test-multipart.txt", "parents": [{"id":"0B09i2ZH5SsTHTjNtSS9QYUZqdTA"}], "properties": [{"kind": "drive#property", "key": "cloudwrapper", "value": "true"}]}
At the bottom of the "Request Body" screen, choose the test-multipart.txt file for uploading.
Press the "Send the request" button
The request and response
Google's OAuth 2.0 Playground miraculously inserts all required headers, computes the content length, generates a boundary sequence, inserts the boundary string wherever required, and shows us the server's response:
Analysis
The multipart HTTP request succeeded with a 200 status code, so the request and response are good ones we can depend upon. Google's Playground inserted everything we needed to perform the multipart HTTP upload. You can see the "Content-length" is set to 352. Let's look at each line after the blank line following the headers:
--===============0688100289==\r\n
Content-type: application/json\r\n
\r\n
{"title": "test-multipart.txt", "parents": [{"id":"0B09i2ZH5SsTHTjNtSS9QYUZqdTA"}], "properties": [{"kind": "drive#property", "key": "cloudwrapper", "value": "true"}]}\r\n
--===============0688100289==\r\n
Content-type: text/plain\r\n
\r\n
We're testing multipart uploading!\r\n
--===============0688100289==--
There are nine (9) lines, and I have manually added "\r\n" at the end of each of the first eight (8) lines (for readability reasons). Here are the number of octets (characters) in each line:
29 + '\r\n'
30 + '\r\n'
'\r\n'
167 + '\r\n'
29 + '\r\n'
24 + '\r\n'
'\r\n'
34 + '\r\n' (although '\r\n' is not part of the text file, Google inserts it)
31
The sum of the octets is 344, and considering each '\r\n' as a single one-octet sequence gives us the coveted content length of 344 + 8 = 352.
Summary
To summarize the findings:
The multipart request's "Content-length" is computed from the first byte of the boundary sequence following the header section's blank line, and continues until, and includes, the last hyphen of the final boundary sequence.
The '\r\n' sequences should be counted as one (1) octet, not two, regardless of the operating system you're running on.
If an http message has Content-Length header, then this header indicates exact number of bytes that follow after the HTTP headers. If anything decided to freely count \r\n as one byte then everything would fall apart: keep-alive http connections would stop working, as HTTP stack wouldn't be able to see where the next HTTP message starts and would try to parse random data as if it was an HTTP message.
\n\r are two bytes.
Moshe Rubin's answer is wrong. That implementation is bugged there.
I sent a curl request to upload a file, and used WireShark to specifically harvest the exact actual data sent by my network. A methodology that everybody should agree is more valid than on online application somewhere gave me a number.
--------------------------de798c65c334bc76\r\n
Content-Disposition: form-data; name="file"; filename="requireoptions.txt"\r\n
Content-Type: text/plain\r\n
\r\n
Pillow
pyusb
wxPython
ezdxf
opencv-python-headless
\r\n--------------------------de798c65c334bc76--\r\n
Curl, which everybody will agree likely implemented this correctly:
Content-Length: 250
> len("2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d646537393863363563333334626337360d0a436f6e74656e742d446973706f736974696f6e3a20666f726d2d646174613b206e616d653d2266696c65223b2066696c656e616d653d22726571756972656f7074696f6e732e747874220d0a436f6e74656e742d547970653a20746578742f706c61696e0d0a0d0a50696c6c6f770d0a70797573620d0a7778507974686f6e0d0a657a6478660d0a6f70656e63762d707974686f6e2d686561646c6573730d0a2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d646537393863363563333334626337362d2d0d0a")
500
(2x250 = 500, copied the hex stream out of WireShark.)
I took the actual binary there. The '2d' is --- which starts the boundary.
Please note, giving the wrong count to the server treating 0d0a as 1 rather than 2 octets (which is insane they are octets and cannot be compound), actively rejected the request as bad.
Also, this answers the second part of the question. The actual Content Length is everything here. From the first boundary to the last with the epilogue --\r\n, it's all the octets left in the wire.
I am using an API that needs the | character in the URL. I tried changing that character to %7C, but the API rejects it. Now the issue is NSURLConnection gives me this if I include a | in the url:
NSURLErrorDomain Code=-1000 "bad URL" UserInfo=0x610000272bc0 {NSLocalizedDescription=bad URL, NSUnderlyingError=0x600000247b60 "bad URL"}
How can include the | in my URL?
You can't: the pipe character is a so called unsafe character and must always be percent escaped to form a valid URL. See RFC 1738 for the syntax of a valid URL.
You should contact the developers of the service you are using and ask them to change this peculiarity of their API's behavior.
I'm using the MediaWiki API to update some pages with an experimental robot.
This robot uses the Java Apache HTTP-client library to update the pages.
(...)
PostMethod postMethod = new PostMethod("http://mymediawikiinstallation/w/api.php");
postMethod.addParameter("action","edit");
postMethod.addParameter("title",page.replace(' ', '_'));
postMethod.addParameter("summary","trying to fix this accent problem");
postMethod.addParameter("text",content);
postMethod.addParameter("basetimestamp",basetimestamp);
postMethod.addParameter("starttimestamp",starttimestamp);
postMethod.addParameter("token",token);
postMethod.addParameter("notminor","");
postMethod.addParameter("format","xml");
int status = httpClient.executeMethod(postMethod);
(...)
However the 'content' string contains some accents. System.out.prinln(content) looks OK, but the accentuated characters in the wiki look bad. E.g. 'Val�rie' instead of 'Valérie'.
How can I fix this?
OK, changing the request header fixed the problem.
postMethod.setRequestHeader( "Content-Type", "application/x-www-form-urlencoded; charset=utf-8");
In my PHP code to talk to the Mediawiki API I used urlencode to encode the title parameter, and this seems to work fine.