*UPDATE AT THE END
I need help with using an API to authenticate into https://connect.garmin.com/signin/.
I am using VBA and Power Query to automate the collecting of workout data from my Garmin account. As far as I can tell, the website uses cookie based authentication and a CSRF token.
I am testing my API calls in Postman, building the authentication request in VBA, and performing data collection in Power Query with the authenticated cookies. (I would use Power Query for the whole project, but it is not able to return the response headers/authenticated cookies)
I am trying to replicate the browser actions by collecting the cookies from response headers and CSRF token from the HTML body before making the authentication POST request, but I am not having much success. When I try to authenticate using this method the response status is 200 and response body is the login page with a "Something went wrong" message.
I have tried to follow OmegaStripes answer from this question to get a handle on the cookies,
How to set and get JSESSIONID cookie in VBA
Is there a special way to handle CSRF tokens in Postman or VBA MSXML2.ServerXMLHTTP? Is there something I am fundamentally not understanding about cookie authentication?
If you need any further info from me, please let me know. Any help is greatly appreciated!
My modified version of OmegaStripes code is below,
Option Explicit
Sub GetCookie()
Dim sUrl, sRespHeaders, sRespText, aSetHeaders, aList, aSetBody, sCSRFToken, sBody
sBody = "username=USERNAME&password=PASSWORD&embed=false&_csrf="
' get cookie 1
sUrl = "https://connect.garmin.com/signin/"
XmlHttpRequest "GET", sUrl, Array(), "", sRespHeaders, sRespText
ParseResponse "^Set-(Cookie): (\S*?=\S*?);[\s\S]*?$", sRespHeaders, aSetHeaders
' get cookie 2 and CSRF Token
sUrl = "https://sso.garmin.com/sso/signin?service=https%3A%2F%2Fconnect.garmin.com%2Fmodern%2F&webhost=https%3A%2F%2Fconnect.garmin.com%2Fmodern%2F&source=https%3A%2F%2Fconnect.garmin.com%2Fsignin%2F&redirectAfterAccountLoginUrl=https%3A%2F%2Fconnect.garmin.com%2Fmodern%2F&redirectAfterAccountCreationUrl=https%3A%2F%2Fconnect.garmin.com%2Fmodern%2F&gauthHost=https%3A%2F%2Fsso.garmin.com%2Fsso&locale=en_US&id=gauth-widget&cssUrl=https%3A%2F%2Fconnect.garmin.com%2Fgauth-custom-v1.2-min.css&privacyStatementUrl=https%3A%2F%2Fwww.garmin.com%2Fen-US%2Fprivacy%2Fconnect%2F&clientId=GarminConnect&displayNameShown=false&consumeServiceTicket=false&generateExtraServiceTicket=true&generateTwoExtraServiceTickets=false&generateNoServiceTicket=false&globalOptInShown=true&globalOptInChecked=false&connectLegalTerms=true&locationPromptShown=true&showPassword=true"
XmlHttpRequest "GET", sUrl, aSetHeaders, "", sRespHeaders, sRespText
' parse project names
ParseResponse "^Set-(Cookie): (\S*?=\S*?);[\s\S]*?$", sRespHeaders, aSetHeaders
sCSRFToken = GetCSRFToken("name=" & Chr(34) & "_csrf" & Chr(34) & " value=" & Chr(34), sRespText)
' get authenticated cookies
sUrl = "https://sso.garmin.com/sso/signin?service=https%3A%2F%2Fconnect.garmin.com%2Fmodern%2F&webhost=https%3A%2F%2Fconnect.garmin.com%2Fmodern%2F&source=https%3A%2F%2Fconnect.garmin.com%2Fsignin%2F&redirectAfterAccountLoginUrl=https%3A%2F%2Fconnect.garmin.com%2Fmodern%2F&redirectAfterAccountCreationUrl=https%3A%2F%2Fconnect.garmin.com%2Fmodern%2F&gauthHost=https%3A%2F%2Fsso.garmin.com%2Fsso&locale=en_US&id=gauth-widget&cssUrl=https%3A%2F%2Fconnect.garmin.com%2Fgauth-custom-v1.2-min.css&privacyStatementUrl=https%3A%2F%2Fwww.garmin.com%2Fen-US%2Fprivacy%2Fconnect%2F&clientId=GarminConnect&displayNameShown=false&consumeServiceTicket=false&generateExtraServiceTicket=true&generateTwoExtraServiceTickets=false&generateNoServiceTicket=false&globalOptInShown=true&globalOptInChecked=false&connectLegalTerms=true&locationPromptShown=true&showPassword=true"
XmlHttpRequest "POST", sUrl, aSetHeaders, sBody & sCSRFToken, sRespHeaders, sRespText
' parse project names
ParseResponse "^Set-(Cookie): (\S*?=\S*?);[\s\S]*?$", sRespHeaders, aSetHeaders
End Sub
Sub XmlHttpRequest(sMethod, sUrl, aSetHeaders, sPayload, sRespHeaders, sRespText)
Dim aHeader
With CreateObject("MSXML2.ServerXMLHTTP")
'.SetOption 2, 13056 ' SXH_SERVER_CERT_IGNORE_ALL_SERVER_ERRORS
.Open sMethod, sUrl, False
For Each aHeader In aSetHeaders
.SetRequestHeader aHeader(0), aHeader(1)
Next
.Send (sPayload)
sRespHeaders = .GetAllResponseHeaders
sRespText = .ResponseText
End With
End Sub
Sub ParseResponse(sPattern, sResponse, aData)
Dim oMatch, aTmp, sSubMatch
If IsEmpty(aData) Then
aData = Array()
End If
With CreateObject("VBScript.RegExp")
.Global = True
.MultiLine = True
.Pattern = sPattern
For Each oMatch In .Execute(sResponse)
If oMatch.SubMatches.Count = 1 Then
PushItem aData, oMatch.SubMatches(0)
Else
aTmp = Array()
For Each sSubMatch In oMatch.SubMatches
PushItem aTmp, sSubMatch
Next
PushItem aData, aTmp
End If
Next
End With
End Sub
Function GetCSRFToken(sPattern, sResponse)
Dim lStart, lLength, sSubMatch
lStart = InStr(1, sResponse, sPattern) + Len(sPattern)
lLength = InStr(lStart, sResponse, Chr(34)) - lStart
GetCSRFToken = Mid(sResponse, lStart, lLength)
End Function
Sub PushItem(aList, vItem)
ReDim Preserve aList(UBound(aList) + 1)
aList(UBound(aList)) = vItem
End Sub
EDIT
I have changed approach to creating an instance of Internet Explorer, and automating the log in process. Not ideal, but I am able to sign in successfully. My issue now is I am not able to retrieve the HttpOnly Cookies required to maintain the logged in session using,
getCookie = objIE.document.Cookie
I have tried another of OmegaStripes answers (coincidentally) Retrieve ALL cookies from Internet Explorer but was unsuccessful.
Any suggestions on how to get this working with MSXML2.ServerXMLHTTP, an IE instance, or any other method would be greatly appreciated!
Related
I'm trying to catch a Response to a POST Request using XMLHTTP using the code below
Dim XMLPage As New MSXML2.XMLHTTP60
Dim HTMLDoc As New MSHTML.HTMLDocument
Dim htmlEle1 As MSHTML.IHTMLElement
Dim htmlEle2 As MSHTML.IHTMLElement
Dim URL As String
Dim elemValue As String
URL = "https://www.informadb.pt/pt/pesquisa/?search=500004064"
XMLPage.Open "GET", URL, False
XMLPage.send
HTMLDoc.body.innerHTML = XMLPage.responseText
For Each htmlEle1 In HTMLDoc.getElementsByTagName("div")
Debug.Print htmlEle1.className
If htmlEle1.className = "styles__SCFileModuleFooter-e6rbca-1 kUUNkj" Then
elemValue = Trim(htmlEle1.innerText)
If InStr(UCase$(elemValue), "CONSTITU") > 0 Then
'Found Value
Exit For
End If
End If
Next htmlEle1
The problem is that I can't find the ClassName "styles__SCFileModuleFooter-e6rbca-1 kUUNkj", because I notice that when I manually insert the value (500004064) in the search box of the URL : https://www.informadb.pt/pt/pesquisa/, the Web Page generates addicinal traffic and turns up at an end point URL : https://www.informadb.pt/pt/pesquisa/empresa/?Duns=453060832, where that className can be found in the Request ResponseText.
My goal is to use the First URL to retrieve the Duns number '453060832', to be able to access the information in the ResponseText of the EndPoint URL. And to catch Duns Number, I need to find a way to get the Endpoint URL, or try to get The POST request response below, and get that value using JSON parser:
{'TotalResults': 1,
'NumberOfPages': 1,
'Results': [{'Duns': '453060832',
'Vat': '500004064',
'Name': 'A PANIFICADORA CENTRAL EBORENSE, S.A.',
'Address': 'BAIRRO DE NOSSA SENHORA DO CARMO,',
'Locality': 'ÉVORA',
'OfficeType': 'HeadOffice',
'FoundIn': None,
'Score': 231.72766,
'PageUrl': '/pt/pesquisa/empresa/?Duns=453060832'}]}
I'm not being able to capture what is really happening using the XMLHTTP Browser request, that seems to be the below steps:
navigate to https://www.informadb.pt/pt/pesquisa/?search=500004064
Webpage generates additional traffic
Amongst that additional traffic is an API POST XHR request which
returns search results as JSON. That request goes to
https://www.informadb.pt/Umbraco/Api/Search/Companies and includes
the 500004064 identifier amongst the arguments within the post body
Based on the API results the browser ends up at the following URI
https://www.informadb.pt/pt/pesquisa/empresa/?Duns=453060832
Can someone help me please, I have to do it using VBA.
Thanks in advance.
A small example how to POST data to your website using VBA, and how to use bare-bones string processing to extract data from the result, as outlined in my comments above.
Function GetVatId(dunsNumber As String) As String
With New MSXML2.XMLHTTP60
.open "POST", "https://www.informadb.pt/Umbraco/Api/Search/Companies", False
.setRequestHeader "Content-Type", "application/json"
.send "{""Page"":0,""PageSize"":5,""SearchTerm"":""" & dunsNumber & """,""Filters"":[{""Key"":""districtFilter"",""Name"":""Distrito"",""Values"":[]},{""Key"":""legalFormFilter"",""Name"":""Forma Jurídica"",""Values"":[]}],""Culture"":""pt""}"
If .status = 200 Then
MsgBox "Response: " & .responseText, vbInformation
GetVatId = Mid(.responseText, InStr(.responseText, """Vat"":""") + 7, 9)
Else
MsgBox "Repsonse status " & .status, vbExclamation
End If
End With
End Function
Usage:
Dim vatId As String
vatId = GetVatId("453060832") ' => "500004064"
For a more robust solution, you should use a JSON parser and -serializer, something like https://github.com/VBA-tools/VBA-JSON.
Question
Is it possible to call Microsoft Graph API using VBA code?
If yes, how to handle O365 authorization? I have seen plenty of topics saying to create an application in Microsoft Azure to get a token but I am surprised I must do that for a simple local use.
What I tried
After discovering Microsoft Graph, I tried this API in Graph Explorer
https://graph.microsoft.com/v1.0/planner/tasks
I was able to create a task in planner!
Consequently, in my mind, it was possible to call this API from VBA code executed directly in Outlook.
I created this macro in Outlook:
Sub TaskPlannerCreation()
Dim PlannerService As New MSXML2.XMLHTTP60
Dim sData As Variant
sData = " { "" ""planId"": ""K9Zv2QHm1U-GSAhd-PTGZfdFeOn"",""bucketId"": ""b6NVNiEIQkGZeBBzn7kWqJvAGvvs"",""title"": ""Outlook task"" } "
With PlannerService
.Open "POST", "https://graph.microsoft.com/v1.0/planner/tasks", False
.setRequestHeader "Content-Type", "application/json"
.setRequestHeader "Accept", "application/json"
.setRequestHeader "User-Agent", "xx"
.Send (sData)
I have an Authorization error with
error code 401
UPDATE on 12-03-2020 :
Solution found to get a Graph Api token analysing URL when calling Graph Explorer (works perfectly for me) :
Function GetToken()
Dim xml As New MSXML2.XMLHTTP60
Dim doc As MSHTML.HTMLDocument
Dim urltoken As String
'copy paste the URL you see when calling Microsoft Graph Explorer and add prompt + domain_hint parameters
urltoken = "https://login.microsoftonline.com/common/oauth2/v2.0/authorize?response_mode=form_post&nonce=graph_explorer&mkt=fr-FR&client_id={clientid}&response_type=token&scope=openid profile User.ReadWrite User.ReadBasic.All Sites.ReadWrite.All Contacts.ReadWrite People.Read Notes.ReadWrite.All Tasks.ReadWrite Mail.ReadWrite Files.ReadWrite.All Calendars.ReadWrite&prompt=none&domain_hint=organizations"
xml.Open "GET", urltoken, False
xml.Send
If xml.readyState = 4 And xml.Status = 200 Then
Set doc = New MSHTML.HTMLDocument
doc.Body.innerHTML = xml.responseText
GetToken = doc.getElementsByName("access_token")(0).Value
sSuccess = True
Else
MsgBox "Error" & vbNewLine & "Ready state: " & xml.readyState & _
vbNewLine & "HTTP request status: " & xml.Status
sSuccess = False
End If
Set xml = Nothing
End Function
So using VBA for calling Graph API is possible :)
So the code you show is only partially correct. Here is what I found to actually work. (This is with what you provided as I actually found a Json parser to work with the data better than the innerHTML methods, I also had to use a different version of MSXML since the one you reference wasnt working for me.)
Function GetToken()
Dim xml As New MSXML2.XMLHTTP60
Dim doc As MSHTML.HTMLDocument
Dim urltoken As String
'copy paste the URL you see when calling Microsoft Graph Explorer and add prompt + domain_hint parameters
urltoken = "https://login.microsoftonline.com/{tenent id}/oauth2/v2.0/token"
xml.Open "POST", urltoken, False
xml.Send("client_id={clientid}&scope=https://graph.microsoft.com/.default&grant_type=client_credentials&client_secret=(cleint secret}")
If xml.readyState = 4 And xml.Status = 200 Then
Set doc = New MSHTML.HTMLDocument
doc.Body.innerHTML = xml.responseText
GetToken = doc.getElementsByName("access_token")(0).Value
sSuccess = True
Else
MsgBox "Error" & vbNewLine & "Ready state: " & xml.readyState & _
vbNewLine & "HTTP request status: " & xml.Status
sSuccess = False
End If
Set xml = Nothing
End Function
I try to download the profile picture from some users in my organization. I'm getting the access_token, but in the next step get the error message:
The token contains no permissions, or permissions can not be
understood.
My app has in my point of view all required permissions:
2
Here is my used code:
Sub Test_GetToken()
Dim xml As New MSXML2.XMLHTTP60
Dim url As String
Dim Json As Object
url = "https://login.microsoftonline.com/tenant_id/oauth2/v2.0/token"
xml.Open "POST", url, False
xml.setRequestHeader "application", "x-www-form-urlencoded"
xml.Send ("client_id=1234678&scope=https%3A%2F%2Fgraph.microsoft.com%2F.default&client_secret=12345678&grant_type=client_credentials")
Set Json = JsonConverter.ParseJson(xml.responseText)
access_token = Json("access_token")
token_type = Json("token_type")
expires_in = Json("expires_in")
ext_expires_in = Json("ext_expires_in")
' trying to get the photo
url = "https://graph.microsoft.com/v1.0/users/user1#OUTLOOK.DE/photo/$value"
xml.Open "GET", url, False
xml.setRequestHeader "application", "x-www-form-urlencoded"
xml.setRequestHeader "Content-Type", "text/json"
xml.setRequestHeader "Authorization", token_type & " " & access_token
xml.Send ("")
'Debug.Print token_type & " " & access_token
'Debug.Print xml.getAllResponseHeaders
Debug.Print xml.responseText
Set xml = Nothing
End Sub
Does anybody has an ideas where the mistake is?
There are several things wrong here:
There isn't an application header in HTTP
x-www-form-urlencoded is an incomplete Content-Type (it should be `application/x-www-form-urlencoded).
Your Content-Type should be application/json
You cannot use the Client Credentials OAuth grant to access a Microsoft Account (i.e. a personal #Outlook address). Client Credentials can only access data from users within your tenant. In order to access personal account data, you need to have the user authenticate using either the Authorization Code or Implicit OAuth grants.
I got Admin Consent and now it works. thanks to Marc for improving my code and getting the solution.
I'm trying to scrap data from a website using MSXML2.XMLHTTP object on VBA environment (Excel) and I cannot figure out how to solve this problem! The website is the following:
http://www.detran.ms.gov.br/consulta-de-debitos/
You guys can use the following test data to fill the form:
Placa: oon5868
Renavam: 1021783231
I want to retrieve data like "chassi", with the data above that would be " 9BD374121F5068077".
I do not have problems parsing the html document, the difficult is actually getting the information as response! Code below:
Sub SearchVehicle()
Dim strPlaca As String
Dim strRenavam As String
strPlaca = "oon5868"
strRenavam = "01021783231"
Dim oXmlPage As MSXML2.XMLHTTP60
Dim strUrl As String
Dim strPostData As String
Set oXmlPage = New MSXML2.XMLHTTP60
strUrl = "http://www2.detran.ms.gov.br/detranet/nsite/veiculo/veiculos/retornooooveiculos.asp"
strPostData = "placa=" & strPlaca & "&renavam=" & strRenavam
oXmlPage.Open "POST", strUrl, False
oXmlPage.setRequestHeader "Content-Type", "application/x-www-form-urlencoded"
oXmlPage.send strPostData
Debug.Print oXmlPage.responseText
End Sub
The strURL used in the POST method ".../retornooooveiculos.asp" is the one google developer tools and fiddler showed me that was the correct address the website was posting the payload.
When manually accessed, the website retrieve the correct information, but running my code I always get the following response on the .responseText:
<html>Acesse: <b><a href='http://www.detran.ms.gov.br target='_parent'>www.detran.ms.gov.br</a></b></html>
HELP PLEASE, I'm getting crazy trying to solve this puzzle! Why do I get redirected like this?
I need the "CHASSI" information and can't find the correct http Request to do this!
Try the below approach. It should fetch you the content you are after. The thing is you need to supply the Cookie copied from your Request Headers fields in order for your script to work which you can find using devtools.
Sub SearchVehicle()
Const URL As String = "http://www2.detran.ms.gov.br/detranet/nsite/veiculo/veiculos/retornooooveiculos.asp"
Dim HTTP As New ServerXMLHTTP60, HTML As New HTMLDocument
Dim elem As Object, splaca$, srenavam$, qsp$
splaca = "oon5868"
srenavam = "01021783231"
qsp = "placa=" & splaca & "&renavam=" & srenavam
With HTTP
.Open "POST", URL, False
.setRequestHeader "User-Agent", "Mozilla/5.0"
.setRequestHeader "Cookie", "ISAWPLB{07D08995-E67C-4F44-91A1-F6A16337ECD6}={286E0BB1-C5F9-4439-A2CE-A7BE8C3955E0}; ASPSESSIONIDSCSDSCTB=AGDPOBEAAPJLLMKKIGPLBGMJ; 69137927=967930978"
.setRequestHeader "Content-Type", "application/x-www-form-urlencoded"
.send qsp
HTML.body.innerHTML = .responseText
End With
For Each elem In HTML.getElementsByTagName("b")
If InStr(elem.innerText, "Chassi:") > 0 Then MsgBox elem.ParentNode.NextSibling.innerText: Exit For
Next elem
End Sub
Once again: fill in the Cookie field by collecting it using your devtools (from Request Headers section), if for some reason my provided Cookie doesn't work for you. Thanks.
Output I'm getting:
9BD374121F5068077
I want to import data from Anedot, a credit card processing firm, using a HTTP GET request from an MS Access program. Anedot uses a RESTful API and has provided help on there website: https://anedot.com/api/v2
I want to do this with VBA, and associate the import with a button on an MS Access form. I've read that this only possible with XML. Do I create the XML file with VBA?
I'd greatly appreciate some background information on how to get this done, as most of it is flying over my head. I don't really know where to begin and I'm having trouble finding anything useful on google.
So far I've realized I'll need to reference their API via a URL link (which they provide), and that I'll have to authorize my account using my username and a token ID. But how can I do this in VBA?
Thanks.
First of all try to make a request to API using basic authorization. Take a look at the below code as the example:
Sub Test()
' API URL from https://anedot.com/api/v2
sUrl = "https://api.anedot.com/v2/accounts"
' The username is the registered email address of your Anedot account
sUsername = "mymail#example.com"
' The password is your API token
sPassword = "1e56752e8531647d09ec8ab20c311ba928e54788"
sAuth = TextBase64Encode(sUsername & ":" & sPassword, "us-ascii") ' bXltYWlsQGV4YW1wbGUuY29tOjFlNTY3NTJlODUzMTY0N2QwOWVjOGFiMjBjMzExYmE5MjhlNTQ3ODg=
' Make the request
With CreateObject("MSXML2.XMLHTTP")
.Open "GET", sUrl, False
.SetRequestHeader "Authorization", "Basic " & sAuth
.Send
Debug.Print .ResponseText
Debug.Print .GetAllResponseHeaders
End With
End Sub
Function TextBase64Encode(sText, sCharset) ' 05 10 2016
Dim aBinary
With CreateObject("ADODB.Stream")
.Type = 2 ' adTypeText
.Open
.Charset = sCharset ' "us-ascii" for bytes to unicode
.WriteText sText
.Position = 0
.Type = 1 ' adTypeBinary
aBinary = .Read
.Close
End With
With CreateObject("Microsoft.XMLDOM").CreateElement("objNode")
.DataType = "bin.base64"
.NodeTypedValue = aBinary
TextBase64Encode = Replace(Replace(.Text, vbCr, ""), vbLf, "")
End With
End Function
Put your credentials to sUsername and sPassword variables, choose the appropriate URL from API help page and put it to sURL. Then you can parse JSON response from the server (currently you will see the response for /v2/accounts request in Immediate window).
It's a fairly lengthy question to be honest, but lets start with some code to get you going.
This Class Module ("clsXMLHttpMonitor") should help:
Option Explicit
Dim XMLHttpReq As MSXML2.ServerXMLHTTP
Dim RequestedVar As String
Dim P As Object
Public Sub Initialize(ByVal uXMLHttpRequest As Object, Optional RequestedValue As String = "")
RequestedVar = RequestedValue
Set XMLHttpReq = uXMLHttpRequest
End Sub
Sub ReadyStateChangeHandler()
If XMLHttpReq.ReadyState = 4 Then
If XMLHttpReq.Status = 200 Then
'Process the response here
Debug.Print "200 recieved"
Set P = JSON.parse(XMLHttpReq.responseText)
Else
If XMLHttpReq.Status = 404 Then
'Handle it
End If
End If
End If
End Sub
Function returnResponseHeaders() As String
returnResponseHeaders = XMLHttpReq.getAllResponseHeaders
XMLHttpReq.Send
End Function
Function returnFullText() As String
If XMLHttpReq.ReadyState = 4 Then
If XMLHttpReq.Status = 200 Then
returnFullText = XMLHttpReq.responseText
Else
returnFullText = "-1"
End If
Else
returnFullText = ""
End If
End Function
End Function
Use it like this:
Set XMLHttpReq = New MSXML2.ServerXMLHTTP
Set XMLHttpMon = New clsXMLHttpMonitor
XMLHttpMon.Initialize XMLHttpReq
XMLHttpReq.OnReadyStateChange = XMLHttpMon
XMLHttpReq.Open "POST", URL, True
XMLHttpReq.Send strPayload
As you seem to request a Json response from a URL, you can study the Json modules here for a full implementation that collects the Json response in a collection, which you then can use in your code or save to a table. See the demo module for examples:
VBA.CVRAPI