I have searched for similar "403" threads but could not find an answer.
I have the following URL which is valid and working -
www.investing.com/equities/apple-computer-inc-earnings
And my code:
Sub GetSourceCode()
Dim XMLHTTP As Object
Dim URL As String
Dim data As String
URL = "http://www.investing.com/equities/apple-computer-inc-earnings"
Set XMLHTTP = CreateObject("WinHttp.WinHttpRequest.5.1")
XMLHTTP.Open "GET", URL, False
XMLHTTP.send
data = XMLHTTP.responseText
End Sub
However, The string "data" is returned with a "403 Forbidden" error instead of the source code, which is what I am trying to achieve.
Any solution or a workaround (or a duplicate reference) would be highly appreciated.
Thanks
set request header just before .send
XMLHTTP.SetRequestHeader "User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
Related
Making my first steps on HTTP within VBA.
Already managed to get cookie value in order to make a request. The problem I'm having is that the file I want to download is not in the first response. When I analyse using HTTP Header Live, the browser receives several responses and only the last one is the file, a PDF that is generated after a query sent by the user. The only thing I'm getting is the first response that I'm displaying with a MsgBox. Can someone help me solving this problem. Made some searches through the web but haven't found yet, a solution.
The code I am using is:
Sub Test()
Dim WinHttpReq As Object
Set WinHttpReq = CreateObject("WinHTTP.WinHTTPrequest.5.1")
With WinHttpReq
.Open "POST", myURL, False ', "username", "password"
.send
x = .getResponseHeader("Set-Cookie")
i = InStr(x, ";")
x = Left(x, i - 1)
MsgBox x
End With
With WinHttpReq
.Open "GET", myURL, False
.Option(WinHttpRequestOption_EnableRedirects) = True
.SetRequestHeader "Content-Type", "application/pdf"
.SetRequestHeader "Accept-Encoding", "gzip, deflate, br"
.SetRequestHeader "Cookie", x
.SetRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:62.0) Gecko/20100101 Firefox/62.0"
.send
End With
MsgBox (WinHttpReq.getAllResponseHeaders())
If WinHttpReq.Status = 200 Then
Set oStream = CreateObject("ADODB.Stream")
With oStream
.Type = 1
.Open
.Write WinHttpReq.responseBody
.SaveToFile "C:\Users\xxx\Desktop\file.pdf", 2
.Close
End With
End If
Set WinHttpReq = Nothing
Set oStream = Nothing
End Sub
Analyzing with Firefox, when I enter the URL, I receive two responses with 200 OK. I wonder how can I get the second response? The site uses a javascript that interprets the query I send and returns a PDF file. The file name changes according to the user and the query.
Now, i reached the following point. I make a first request to get the cookie, then a second to get an ETag from a diferent address. The problem is that when i make the third request to download the file, the filename is apparently generated by the server (APACHE?), based on the ETag and something I'm not being able to find. The last 5 numbers from the ETag change in the filename.
For example:
ETag - 1543932096000
File - 1543932095115.xxxxx.address.com.5245.idp.pdf
Since I don't have the filename, i cannot download the file with an httprequest.
Help?
I've written a very tiny script in vba using POST request. However, when I run it, I get nothing as result except for a blank message. I've tried to fill in the request parameter accordingly. Perhaps, I can't notice which should be included in the parameter. The page I'm dealing with contains several images in it's right panel. When an image is clicked the request about which i'm talking here is sent to the server and brings back the result and displays new information concerning its' flavor under it. My goal is to parse all the flavors connected to each images. Anyways, I'm trying to attach all the things necessary to find out what i'm missing. Thanks in advance.
This is what I got from chrome developer tools to prepare the POST request:
"https://www.dropbox.com/s/zjn0ahixhu58miq/RequestStatus.txt?dl=0"
Here is what I'm trying with:
Sub PostReq()
Dim http As New XMLHTTP60, html As New HTMLDocument
Dim ArgumentStr As String
ArgumentStr = "opt=flavor&opt1=207&opt2=47&ip=105"
With http
.Open "POST", "https://www.optigura.com/product/ajax/details.php", False
.setRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"
.setRequestHeader "Content-type", "application/x-www-form-urlencoded"
.setRequestHeader "Accept", "application/json, text/javascript, */*; q=0.01"
.send ArgumentStr
html.body.innerHTML = .responseText
End With
MsgBox http.responseText
End Sub
This is the original link to the webpage:
"https://www.optigura.com/uk/product/gold-standard-100-whey/"
Your code sets a request header like so:
.setRequestHeader "Content-type", "application/x-www-form-urlencoded"
so the script is going to expect the argument string to be URL encoded (which it isn't).
Try either encoding the string, or send the request using "GET" instead.
Finally, I've made it. To receive the required response it is necessary to send a GET request first then again send a POST request using the response from that get request. Here is the working one:
Sub httpPost()
Dim http As New XMLHTTP60, html As New HTMLDocument
Dim ArgumentStr As String
ArgumentStr = "opt=flavor&opt1=207&opt2=47&ip=105"
With http
.Open "GET", "https://www.optigura.com/uk/product/gold-standard-100-whey/", False
.send
End With
With http
.Open "POST", "https://www.optigura.com/product/ajax/details.php", False
.setRequestHeader "X-Requested-With", "XMLHttpRequest"
.setRequestHeader "Content-type", "application/x-www-form-urlencoded"
.setRequestHeader "Accept", "application/json, text/javascript, */*; q=0.01"
.send ArgumentStr
html.body.innerHTML = .responseText
End With
MsgBox http.responseText
End Sub
I wish I could use some macros to send Dashboard summary text over the work Telegram. I know that for me to send by telegram through webhook (GET and POST html) and I researched and found this topic here:
How can I send an HTTP POST request to a server from Excel using VBA?
Set objHTTP = CreateObject("MSXML2.ServerXMLHTTP")
URL = "https://api.telegram.org/bot<token>/METHOD_NAME"
objHTTP.Open "POST", URL, False
objHTTP.setRequestHeader "User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
objHTTP.send("")
Replacing < Token > and < Method_name >
https://core.telegram.org/bots/api
However, when I debug the code it locks up excel and does not come back anymore. Any idea what that might be?
It worked when I deployed MSXML2.ServerXMLHTTP to MSXML2.ServerHTTP60.
Sub fff()
Set objHTTP = CreateObject("MSXML2.ServerHTTP60")
URL = "https://api.telegram.org/bot<token>/sendMessage?chat_id="id"&text=test"
objHTTP.Open "POST", URL, False
objHTTP.setRequestHeader "User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
objHTTP.send ("")
Cells(1, 1).Value = objHTTP.ResponseText
End Sub
I have written a short macro [below] to test a GET call. This macro works fine for several people that have tested it, however one person (using same version of excel - 2013) is getting a timeout error. I have allowed Excel to connect to the internet in settings, and have enabled macros. Any suggestions are appreciated.
Thanks!
Set objHTTP = CreateObject("Msxml2.ServerXMLHTTP.6.0")
Dim result As String
URL = "http://www.google.com"
With objHTTP
.Open "GET", URL, False
.SetRequestHeader "User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
.send ("")
End With
Change object type: Set objHTTP = CreateObject("Microsoft.XMLHTTP")
Hope this help!
I have written some code to retrieve url, but i am getting #Value error. Is anything wrong in this code,
Public Function Rurl(ByVal URL As String)
Dim http As Object
Set http = CreateObject("WinHttp.WinHttpRequest.5.1")
http.Option(WinHttpRequestOption_UserAgentString) = "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)"
http.Option(WinHttpRequestOption_EnableRedirects) = True
If Not InStr(URL, "://") > 0 Then URL = "http://" & URL
http.Open "GET", URL
http.Send
Rurl = http.GetResponseHeader("Location")
Set http = Nothing
End function
You don't say where you're getting the error, so I'm going to assume it's at this line:
Rurl = http.GetResponseHeader("Location")
Something to ask yourself: what will your code do if the site at the supplied URL doesn't redirect?
The answer is that your code will give you an error at the above line which you don't handle anywhere in your code, very likely resulting in the #VALUE! error that you're seeing.
I'd suggest adding some error checking to ensure your function works in all situations. So, give this a go:
Public Function Rurl(ByVal URL As String)
On Error GoTo ErrorHandler
Dim http As Object
Set http = CreateObject("WinHttp.WinHttpRequest.5.1")
http.Option(WinHttpRequestOption_UserAgentString) = "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)"
http.Option(WinHttpRequestOption_EnableRedirects) = True
If Not InStr(URL, "://") > 0 Then URL = "http://" & URL
http.Open "GET", URL
http.Send
Rurl = http.GetResponseHeader("Location")
Set http = Nothing
Exit Function
ErrorHandler:
Rurl = "" ' or you can say something like: "No redirection".
Resume Next
End Function
If an error occurs anywhere in your function, the error handler will set the return value of your function to something sensible, clean up and exit the function. If no error occurs, everything should work like before. We're just adding a bit of code to handle potential errors.
trap for the 302 status code then get the Location variable from the http header.
strUrl = "https://xx123.abc.com/"
Dim http As Object
Set http = CreateObject("WinHttp.WinHttpRequest.5.1")
With http
.Open "GET", strUrl, False
.setRequestHeader "Content-Type", "text/css" '"application/x-www-form-urlencoded"
.setRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0"
.Option(WinHttpRequestOption_EnableRedirects) = False
.Send
d = .waitForResponse()
If (.Status = 302) Then
temp = .responseText
headers = .getAllResponseHeaders()
cookie = .getResponseHeader("Set-Cookie")
redirectedURL = .getResponseHeader("Location")
End If
End With