Web Scraping: xmlHTTP GET request not working - vba

I am trying to scrape flashscore.com with VBA using XMLHTTP request. It is making multiple javascript based requests to the server. One request i am interested in (as visible in the chrome developer tools) is this:
but when I send the request, it fails. The code is
With CreateObject("MSXML2.XMLHTTP")
.Open "GET", "https://d.flashscore.com/x/feed/f_1_0_5_en_1", False
.send
End With
Can anybody guide me why i can not retrieve the content like this? Why the request is failing? Thanks

Related

Azure logic app API http response with excel file download using Postman

Created azure logic app HTTP request it gives response for normal JSON schema However, I want to attach SharePoint excel sheet when I trigger the request from Postman.
1.How to used content type or schema to download the attached file. when postman request sent.
2.is that possible to download when you hit API through logic app
3.Generated HTTP POST URL is working
For your requirement, I test it in my side. It seems we do not need to set any value for "Content-Type" in headers of response. Please refer to my logic app below:
Then when you request the logic app url in postman, please choose "Send and Download" instead of "Send".
After that, you can download the file when request the url in postman.

Microsoft Graph v1.0 download file content not working

I am trying to download a word document file named Test.docx located in Sharepoint Document Library from within a Visio File macro referring to the Microsoft documentation at
https://learn.microsoft.com/en-us/graph/api/driveitem-get-content?view=graph-rest-1.0&tabs=http
I am able to get my site ID and Item ID ending up to the below URL,
https://graph.microsoft.com/v1.0/sites/{site-id}/drive/items/{item-id}
Since I want to download the file content in binary and save it locally, I add "content" to the url as below,
https://graph.microsoft.com/v1.0/sites/{site-id}/drive/items/{item-id}/content
My code looks as below in VBA environment for the client as below with an active Access Token,
myURL=https://graph.microsoft.com/v1.0/sites/{site-id}/drive/items/{item-id}/content
Set winHttpReq = New XMLHTTP60
winHttpReq.Open "GET", myURL
winHttpReq.setRequestHeader "Authorization", "Bearer " & AccessToken
winHttpReq.send
I am getting an error as "Access is denied." at the line winHttpReq.send and control exits out of the function.
Files.Read.All, Files.ReadWrite.All, Sites.Read.All, Sites.ReadWrite are the permissions already given for the API access.
Also, I did another call fetching the "#microsoft.graph.downloadUrl" download URL but I did not see any binary content in the response.
My questions are,
Is my approach correct?
I need the file binary data to be downloaded using graph API but adding /content to the URL gives this error. How can I get the file using API?
Where can I fetch the binary content from the response to the url of "#microsoft.graph.downloadUrl" of the file to download and save the file locally?
Is there any other approach to download a file using the graph API
that will work in VBA environment?

VBA Proxy auto-configuration for http requests

I need to build a client for a web api in VBA and it needs to work behind a proxy (with authentication). I've been looking at the WinHttp.WinHttpRequest and MSXML2.XMLHTTP/ServerXMLHTTP classes. It turns out that:
XMLHTTP automatically detects proxy settings provided through a proxy.pac file (good)
WinHttpRequest doesn't (bad)
However, on the other hand:
XMLHTTP automatically follows redirects, and there's no way of disabling this behaviour (bad)
WinHttpRequest doesn't (good)
Since I'd like to be able to have my cake and eat it, is there a way to get automatic proxy configuration for a component such as WinHttpRequest that doesn't follow redirects blindly?
The VBA-Web project might help you with your pastry eating problem.
https://github.com/VBA-tools/VBA-Web
I guess what you wish to do would go something like:
Dim client As New WebClient
With client
.BaseUrl = "https://www.google.com"
.ProxyUsername = <user>
.ProxyPassword = <password>
.EnableAutoProxy = True
End With
Dim request As New WebRequest
With request
.Method = WebMethod.HttpGet
.Format = WebFormat.PlainText
End With
Dim response As WebResponse
Set response = client.Execute(request)

Parse.com Push Object/Message from Excel VBA

I am trying to write a VBA script to push messages from Excel to my decives using the Parse.com framework. So far I succeeded in sending push data messages from the Parse.com website to multiple devices so that part should be good.
Problem is getting VBA to send a Push or even data to Parse.com in general.
At the moment I'm just trying to get the connection to work and push some data in to my TestObject Class. I am using the following code and authentication seems to go how it's supposed to be now! The PUT object gives an error: invalid json
Sub Parse()
TargetURL = "https://api.parse.com/1/classes/TestObject"
Set HTTPReq = CreateObject("WinHttp.WinHttpRequest.5.1") '
HTTPReq.Open "POST", TargetURL, False
HTTPReq.setRequestHeader "X-Parse-Application-Id", "xxxxxx"
HTTPReq.setRequestHeader "X-Parse-REST-API-Key", "xxxxxx"
HTTPReq.setRequestHeader "Content-Type", "application/json"
HTTPReq.send ("{foo:bar}")
MsgBox (HTTPReq.ResponseText)
End Sub
New Error:
Invalid Json
Have you set up authentication? From the Parse.com REST API reference:
Request Format
For POST and PUT requests, the request body must be JSON, with the Content-Type header set to application/json.
Authentication is done via HTTP headers. The X-Parse-Application-Id header identifies which application you are accessing, and the X-Parse-REST-API-Key header authenticates the endpoint. In the examples that follow, the keys for your app are included in the command. You can use the drop-down to construct example code for other apps.
You may also authenticate your REST API requests using basic HTTP authentication. For example, to retrieve an object you could set the URL using your Parse credentials in the following format:
https://myAppID:javascript-key=myJavaScriptKey#api.parse.com/1/classes/GameScore/Ed1nuqPvcm
For Javascript usage, the Parse Cloud supports cross-origin resource sharing, so that you can use these headers in conjunction with XMLHttpRequest.
HTTPReq.send ("{""foo"":""bar""}")
Posting will be successful.

XmlHttp Request Basic Authentication Issue

I have the following code that creates a serverside object of the xmlhttp class. I am trying to connect to a site that requires basic authentication. I am able to get this to work with the code below.
What's the problem? Well I'm passing the credentials using the open call. That alone is not enough. I must also set the authorization header with the manually calculated base64 encoded username:password combination. If I try to set the header without passing the credentials to the open call, it fails. Call me crazy, but when I pass the credentials to the open, that's all I should have to do. If I set the header, that's all I should have to do. Right? Doing both seems like something isn't right. Right?
Is this a bug or a glitch?
Additional background is:
IIS 5 & ASP Classic
The error received when one of the two items is ommitted is an HTTP Status 401:
"You are not authorized to view this page
You do not have permission to view this directory or page using the credentials that you supplied because your Web browser is sending a WWW-Authenticate header field that the Web server is not configured to accept."
Since IIS is making the request I'm not able to inspect it with Fiddler :-(
Set xmlhttp = Server.CreateObject("MSXML2.ServerXMLHTTP.6.0")
xmlhttp.setTimeouts 5000, 5000, 10000, 10000 'ms - resolve, connect, send, receive
xmlhttp.open "GET", "http://example.com/", False, "username", "password"
xmlhttp.setRequestHeader "Authorization", "Basic dXNlcm5hbWU6cGFzc3dvcmQ="
xmlhttp.send
I believe you're hitting this known limitation (or bug) that can be boiled down to msxml2 lacking (or having incorrect) support for "negotiated" authentication mechanisms, which means you have to force the issue (bypass the incorrectly-conducted negotiation) exactly by adding the authorization header yourself as you're doing.