So I want to retrieve some JSON data which changes frequently from Windows gadget, by using Microsoft.XMLHTTP ActiveXObject. The problem is that it returns cached version of the page instead of requesting new one.
I have no control over the server, and I can't use usual hack of sending extra parameter because server returns error if I send any parameters.
I've googled this to death and the best information is in this Stackoverflow question, but none of the answers work for me; I haven't been able to find a way to use ServerXMLHTTP from gadget Javascript. How do I either use ServerXMLHTTP, or prevent caching in a way other than adding random parameter in the URL?
try using POST instead of GET
request = new ActiveXObject("Microsoft.XMLHTTP");
request.onreadystatechange = callback;
request.open("POST", "server.php", true);
request.setRequestHeader("Content-Type", "application/x-www-form-urlencoded");
request.send();
references:
Prevent Chrome from caching AJAX requests
and my own experience.
Adding a random number worked for me in the context of VBA, eg
myURL= "http://acme.com/someapp/rest/someendpointurl?randombit=" & Rnd()
Dim WinHttpReq As Object
Set WinHttpReq = CreateObject("Microsoft.XMLHTTP")
WinHttpReq.Open "GET", myURL, False, "username", "password"
WinHttpReq.send
myURL = WinHttpReq.responseBody
If WinHttpReq.status = 200 Then
Set oStream = CreateObject("ADODB.Stream")
oStream.Open
oStream.Type = 1
oStream.Write WinHttpReq.responseBody
oStream.SaveToFile saveToPath, 2 ' 1 = no overwrite, 2 = overwrite
oStream.Close
End If
Thank you Sean Cull your solution worked for me in the context of VBA.
(shame I'm not allowed to upvote it so reposting here for reference)
Can you not add a random element to the end of your URL ?
your http://acme.com/someapp/rest/someendpointurl?randombit=20171701143500
Related
I'm trying to scrap data from a website using MSXML2.XMLHTTP object on VBA environment (Excel) and I cannot figure out how to solve this problem! The website is the following:
http://www.detran.ms.gov.br/consulta-de-debitos/
You guys can use the following test data to fill the form:
Placa: oon5868
Renavam: 1021783231
I want to retrieve data like "chassi", with the data above that would be " 9BD374121F5068077".
I do not have problems parsing the html document, the difficult is actually getting the information as response! Code below:
Sub SearchVehicle()
Dim strPlaca As String
Dim strRenavam As String
strPlaca = "oon5868"
strRenavam = "01021783231"
Dim oXmlPage As MSXML2.XMLHTTP60
Dim strUrl As String
Dim strPostData As String
Set oXmlPage = New MSXML2.XMLHTTP60
strUrl = "http://www2.detran.ms.gov.br/detranet/nsite/veiculo/veiculos/retornooooveiculos.asp"
strPostData = "placa=" & strPlaca & "&renavam=" & strRenavam
oXmlPage.Open "POST", strUrl, False
oXmlPage.setRequestHeader "Content-Type", "application/x-www-form-urlencoded"
oXmlPage.send strPostData
Debug.Print oXmlPage.responseText
End Sub
The strURL used in the POST method ".../retornooooveiculos.asp" is the one google developer tools and fiddler showed me that was the correct address the website was posting the payload.
When manually accessed, the website retrieve the correct information, but running my code I always get the following response on the .responseText:
<html>Acesse: <b><a href='http://www.detran.ms.gov.br target='_parent'>www.detran.ms.gov.br</a></b></html>
HELP PLEASE, I'm getting crazy trying to solve this puzzle! Why do I get redirected like this?
I need the "CHASSI" information and can't find the correct http Request to do this!
Try the below approach. It should fetch you the content you are after. The thing is you need to supply the Cookie copied from your Request Headers fields in order for your script to work which you can find using devtools.
Sub SearchVehicle()
Const URL As String = "http://www2.detran.ms.gov.br/detranet/nsite/veiculo/veiculos/retornooooveiculos.asp"
Dim HTTP As New ServerXMLHTTP60, HTML As New HTMLDocument
Dim elem As Object, splaca$, srenavam$, qsp$
splaca = "oon5868"
srenavam = "01021783231"
qsp = "placa=" & splaca & "&renavam=" & srenavam
With HTTP
.Open "POST", URL, False
.setRequestHeader "User-Agent", "Mozilla/5.0"
.setRequestHeader "Cookie", "ISAWPLB{07D08995-E67C-4F44-91A1-F6A16337ECD6}={286E0BB1-C5F9-4439-A2CE-A7BE8C3955E0}; ASPSESSIONIDSCSDSCTB=AGDPOBEAAPJLLMKKIGPLBGMJ; 69137927=967930978"
.setRequestHeader "Content-Type", "application/x-www-form-urlencoded"
.send qsp
HTML.body.innerHTML = .responseText
End With
For Each elem In HTML.getElementsByTagName("b")
If InStr(elem.innerText, "Chassi:") > 0 Then MsgBox elem.ParentNode.NextSibling.innerText: Exit For
Next elem
End Sub
Once again: fill in the Cookie field by collecting it using your devtools (from Request Headers section), if for some reason my provided Cookie doesn't work for you. Thanks.
Output I'm getting:
9BD374121F5068077
Hi guys I have this sub I want to download a file from URL but everytime when I run it WinHttpReq.Status contains 406.
Sub DownloadFile()
Dim myURL As String
myURL = "https://YourWebSite.com/?your_query_parameters"
Dim WinHttpReq As Object
Set WinHttpReq = CreateObject("Microsoft.XMLHTTP")
WinHttpReq.Open "GET", myURL, False, "username", "password"
WinHttpReq.send
myURL = WinHttpReq.responseBody
If WinHttpReq.Status = 200 Then
Set oStream = CreateObject("ADODB.Stream")
oStream.Open
oStream.Type = 1
oStream.Write WinHttpReq.responseBody
oStream.SaveToFile "C:\file.csv", 2 ' 1 = no overwrite, 2 = overwrite
oStream.Close
End If
End Sub
The 406 status code means that, although the server understood and processed the request, the response is of a form the client cannot understand. A client sends, as part of a request, headers indicating what types of data it can use, and a 406 error is returned when the response is of a type not in that list.
Eg. If you ask the server to send a GIF picture, but it can only send plain text and PNG pictures you will receive a 406 status code meaning your server understood your question but cannot fulfill it.
So you should include an Accept header specifying which type of media you want your server to send to your client (your VBA), and that type should be a type that your server can actually deliver.
Example how to send headers:
WinHttpReq.SetRequestHeader "Content-Type", "text/xml;charset=utf-8"
WinHttpReq.SetRequestHeader "Accept", "text/xml"
Of course nobody can tell you which media type is the correct because we don't know your server and which type of media it can provide.
I have been trying access a web site and OCR a captcha.
But this captcha part I have no idea how to proceed.
Can someone walk me through this?
1. Access website
2. Donwload captcha image (?)
3. OCR it
Any help is appreciated.
Downloading a file from the web can be done with the Microsoft.XMLHTTP object:
Dim myURL As String
myURL = "http://www.somesite.com/captcha.png"
Dim WinHttpReq As Object
Set WinHttpReq = CreateObject("Microsoft.XMLHTTP")
WinHttpReq.Open "GET", myURL, False
WinHttpReq.Send
myURL = WinHttpReq.ResponseBody
If WinHttpReq.Status = 200 Then
Set oStream = CreateObject("ADODB.Stream")
oStream.Open
oStream.Type = 1
oStream.Write WinHttpReq.ResponseBody
oStream.SaveToFile ("C:\captcha.png")
oStream.Close
End If
Snippet Copied from here
However, In order to run OCR on the image you just downloaded, you'll need to use an ActiveX component,
After googling, I came up with this component, and I haven't found anything free.
NOTE: Further research (and thick skin, and lots of coffee) has lead me here where it suggests that the store cannot receive the uid and pwd in the URL, and I gather from the PHP examples here that the transmission needs to be encrypted. Is this my problem? If so I have not been able to find anything on how to work around this.
I would be happy if the open() command prompted the UID & PWD window to open and I can manually input the values. Does anyone have a suggestion. Thanks!
I am learning how to use the BigCommerce APIs and I am programming in VB / XL to be able to directly post / retrieve from all fields in the store's DB.
I have never coded on this before (although I get around VB OK) and I am stuck. I have the following code:
Const URL As String = "https://www.myurl.com/api/v2/brands.json"
Public Sub Test()
Dim xmlHttp As Object
Set xmlHttp = CreateObject("MSXML2.ServerXMLHTTP.6.0")
xmlHttp.Open "GET", URL, False, "myid", "mytoken"
xmlHttp.setRequestHeader "Content-Type", "application/json"
xmlHttp.send
Dim html As MSHTML.HTMLDocument
Set html = New MSHTML.HTMLDocument
html.body.innerHTML = xmlHttp.ResponseText
Range("A1").Value = html.body.innerHTML
End Sub
What I get back is "401" or more precisely:
[{"status":401,"message":"No credentials were supplied in the request."}]
The credentials are valid and working, as if I place the URL in my browser and submit it, the pop up for UID & PWD comes up, and once I place the values I am using in the code, the resulting BRANDS list appears in the browser.
Additionally, if I add a "rand num" to the URL to ensure the returned data is not cached as suggested here, then the request looks like this:
xmlHttp.Open "GET", URL & "&t=" & WorksheetFunction.RandBetween(1, 99), False, `"myid", "mytoken"`
and the response back from BC changes to:
406</STATUS><MESSAGE>The requested content type is not available.</MESSAGE></ERROR></ERRORS>
I have also tried this (where UID and PWD are the correct values):
xmlHttp.Open "GET", URL & "&user=UID" & "&password=PWD", False
and i get:
406</STATUS><MESSAGE>The requested content type is not available.</MESSAGE></ERROR></ERRORS>
Can anyone help me understand pls. I have followed examples at MSDN, but I am still stuck. The BC API site is poor in examples (none actually!) for VB.
Thank you
Sorted! BC required this:
Set xmlHttp = CreateObject("MSXML2.ServerXMLHTTP.6.0")
xmlHttp.Open "GET", URL_Cat, UID, PWD
xmlHttp.setRequestHeader "Content-Type", "application/json"
xmlHttp.setRequestHeader "Authorization", "Basic " & Base64Encode(UID & ":" & PWD)
xmlHttp.send
I got Base64Encode code from here. Hope this helps someone.
I need a excel macro (vba) to select a file from box.com by iterating though existing folders and at the same time I need to upload the file from my machine to box.com folder using excel macro. I am searching for a long time on net. But no use. Please help or try to give some ideas how to achieve this.
Thanks in advance.
-Edit
I am using the below code for getting authentication token. But I am getting an error message at the place of .send(url). Error message is "The server name or address could not be resolved".
Function getAuthToken()
Dim WinHttpReq As WinHttp.WinHttpRequest
Dim api_key As String
api_key = "{api_key}"
Set WinHttpReq = New WinHttp.WinHttpRequest
strUrl = "https://www.box.net/api/1.0/rest?action=get_ticket&api_key=" & api_key
WinHttpReq.Open Method:="GET", url:=strUrl, async:=False
WinHttpReq.Send
getTicket = WinHttpReq.responseText
Debug.Print getTicket
End Function
Not being a vba expert, I suspect that you'll get more answers if you tag your question with a vba tag. However, some quick scanning around shows that vba can call REST apis by doing something like this:
Dim MyURL as String
MyURL = "http://api.box.com/2.0/folders/0.xml"
Set objHTTP = CreateObject("MSXML2.ServerXMLHTTP")
With objHTTP
.Open "GET", MyURL, False
.setRequestHeader "Content-Type", "application/x-www-form-urlencoded"
.setRequestHeader "Authorization", "BoxAuth api_key=<your api key>&auth_token=<your auth token>
.send (MyURL)
End With
I'll defer to a real VBA expert, but something roughly along these lines should work.
Yeah, this is frustrating. I tried code like Peter's using both WinHttp.WinHttpRequest.5.1 and MSXML2.ServerXMLHTTP and with both I just get a zero-length string back. No error message or anything.
I installed cURL and tested the URL. It works fine there. The script below also works fine with a generic JSON web service, like jsonplaceholder.typicode.com.
All this makes me think that Box.com is receiving the message, detecting it is coming from a non-approved source, and returning nothing . . . probably for security reasons.
Option Explicit
Const URL As String = "https://api.box.com/2.0/folders/0 -H ""Authorization: Bearer MyToken"""
'Const URL As String = "https://jsonplaceholder.typicode.com/posts/1"
Sub Test()
Dim winHTTP As Object
' Set winHTTP = CreateObject("WinHttp.WinHttpRequest.5.1")
Set winHTTP = CreateObject("MSXML2.ServerXMLHTTP")
winHTTP.Open "GET", URL
winHTTP.setRequestHeader "Content-Type", "application/json"
winHTTP.send
Debug.Print winHTTP.ResponseText
If Len(winHTTP.ResponseText) = 0 Then
MsgBox "blank string returned"
Else
Dim objResponse As Object
Set objResponse = JsonConverter.ParseJson(winHTTP.ResponseText) 'Converter from Tim Hall - https://github.com/VBA-tools/VBA-JSON
End If
End Sub