Trouble reusing the same Http after passing it from a sub to a function - vba

I've written a script using xhr to parse the link of the first post from this website and then passed the link and the http to a function to fetch the title from it's inner page. It's important that I pass both the link and the http to the function in order to reuse the same http that I have used in the first place.
My script appears to be working in the right way but I'm not sure I did the whole thing in the right way. The reason for my confusion is that I get the result when I use like getHTTP(ByVal Http, ByVal link) As Variant. I even get the result when I go for getHTTP(ByVal Http, ByVal link) As String or getHTTP(ByVal Http, ByVal link). Moreover, I didn't explicitly define link as String or Http as XMLHTTP60 within function parameters.
I've tried with (working flawlessly):
Function getHTTP(ByVal Http, ByVal link) As Variant
Dim Html As New HTMLDocument, oTitle$
With Http
.Open "GET", link, False
.send
Html.body.innerHTML = .responseText
oTitle = Html.querySelector("h1[itemprop='name'] > a").innerText
getHTTP = oTitle
End With
End Function
Sub GetInfo()
Const base$ = "https://stackoverflow.com"
Const Url$ = "https://stackoverflow.com/questions/tagged/web-scraping"
Dim Http As New XMLHTTP60, Html As New HTMLDocument
Dim firstLink$, postTitle$
With Http
.Open "GET", Url, False
.send
Html.body.innerHTML = .responseText
firstLink = base & Replace(Html.querySelector(".summary .question-hyperlink").getAttribute("href"), "about:", "")
postTitle = getHTTP(Http, firstLink)
MsgBox postTitle
End With
End Sub
What is the right way to pass http between sub and function in order to reuse the same http?

Something like this might be appropriate:
Sub GetInfo()
Const base As String = "https://stackoverflow.com"
Const url As String = "https://stackoverflow.com/questions/tagged/web-scraping"
Dim Html As HTMLDocument
Dim firstLink As String, postTitle As String
firstLink = base & Replace(GetPage(url).querySelector(".summary .question-hyperlink") _
.getAttribute("href"), "about:", "")
Debug.Print firstLink
postTitle = GetPage(firstLink).querySelector("h1[itemprop='name'] > a").innerText
Debug.Print postTitle
End Sub
Function GetPage(url As String) As HTMLDocument
Dim Html As HTMLDocument
Static Http As XMLHTTP60
If Http Is Nothing Then
Set Http = New XMLHTTP60
'log in here
End If
With Http
.Open "GET", url, False
.send
If .Status = 200 Then
Set Html = New HTMLDocument
Html.body.innerHTML = .responseText
Else
Debug.Print .Status
'warn user
End If
End With
Set GetPage = Html
End Function
Above all assumes the "happy path" so probably would benefit from some exception handling...

Something like the following should do what you wanna achieve:
Function getHTTP(ByVal Http, link) As String
With Http
.Open "GET", link, False
.send
getHTTP = .responseText
End With
End Function
Sub GetInfo()
Const base$ = "https://stackoverflow.com"
Const Url$ = "https://stackoverflow.com/questions/tagged/web-scraping"
Dim Http As Object, Html As New HTMLDocument
Dim firstLink$
Set Http = CreateObject("MSXML2.XMLHTTP")
Html.body.innerHTML = getHTTP(Http, Url)
firstLink = base & Replace(Html.querySelector(".summary .question-hyperlink").getAttribute("href"), "about:", "")
Html.body.innerHTML = getHTTP(Http, firstLink)
MsgBox Html.querySelector("h1[itemprop='name'] > a").innerText
End Sub

If I've understood your issue correctly and you need changes made to the http instance in the function to be passed to your calling sub then passing http ByRef will be enough. Passing the variable ByRef passes changes made to the http instance in the function to the sub for use outside of the function. So changes made to http in the function will be passed to the sub and will not be lost. You will be using the same instance of the http object in both places. Passing http ByVal won't pass the changes made to http in the function to the sub. Change your function parameters declaration as follows.
Function getHTTP(ByRef Http, ByVal link) As Variant

Related

Catch the POST Request Response and the Redirected URL from XMLHTTP request with VBA

I'm trying to catch a Response to a POST Request using XMLHTTP using the code below
Dim XMLPage As New MSXML2.XMLHTTP60
Dim HTMLDoc As New MSHTML.HTMLDocument
Dim htmlEle1 As MSHTML.IHTMLElement
Dim htmlEle2 As MSHTML.IHTMLElement
Dim URL As String
Dim elemValue As String
URL = "https://www.informadb.pt/pt/pesquisa/?search=500004064"
XMLPage.Open "GET", URL, False
XMLPage.send
HTMLDoc.body.innerHTML = XMLPage.responseText
For Each htmlEle1 In HTMLDoc.getElementsByTagName("div")
Debug.Print htmlEle1.className
If htmlEle1.className = "styles__SCFileModuleFooter-e6rbca-1 kUUNkj" Then
elemValue = Trim(htmlEle1.innerText)
If InStr(UCase$(elemValue), "CONSTITU") > 0 Then
'Found Value
Exit For
End If
End If
Next htmlEle1
The problem is that I can't find the ClassName "styles__SCFileModuleFooter-e6rbca-1 kUUNkj", because I notice that when I manually insert the value (500004064) in the search box of the URL : https://www.informadb.pt/pt/pesquisa/, the Web Page generates addicinal traffic and turns up at an end point URL : https://www.informadb.pt/pt/pesquisa/empresa/?Duns=453060832, where that className can be found in the Request ResponseText.
My goal is to use the First URL to retrieve the Duns number '453060832', to be able to access the information in the ResponseText of the EndPoint URL. And to catch Duns Number, I need to find a way to get the Endpoint URL, or try to get The POST request response below, and get that value using JSON parser:
{'TotalResults': 1,
'NumberOfPages': 1,
'Results': [{'Duns': '453060832',
'Vat': '500004064',
'Name': 'A PANIFICADORA CENTRAL EBORENSE, S.A.',
'Address': 'BAIRRO DE NOSSA SENHORA DO CARMO,',
'Locality': 'ÉVORA',
'OfficeType': 'HeadOffice',
'FoundIn': None,
'Score': 231.72766,
'PageUrl': '/pt/pesquisa/empresa/?Duns=453060832'}]}
I'm not being able to capture what is really happening using the XMLHTTP Browser request, that seems to be the below steps:
navigate to https://www.informadb.pt/pt/pesquisa/?search=500004064
Webpage generates additional traffic
Amongst that additional traffic is an API POST XHR request which
returns search results as JSON. That request goes to
https://www.informadb.pt/Umbraco/Api/Search/Companies and includes
the 500004064 identifier amongst the arguments within the post body
Based on the API results the browser ends up at the following URI
https://www.informadb.pt/pt/pesquisa/empresa/?Duns=453060832
Can someone help me please, I have to do it using VBA.
Thanks in advance.
A small example how to POST data to your website using VBA, and how to use bare-bones string processing to extract data from the result, as outlined in my comments above.
Function GetVatId(dunsNumber As String) As String
With New MSXML2.XMLHTTP60
.open "POST", "https://www.informadb.pt/Umbraco/Api/Search/Companies", False
.setRequestHeader "Content-Type", "application/json"
.send "{""Page"":0,""PageSize"":5,""SearchTerm"":""" & dunsNumber & """,""Filters"":[{""Key"":""districtFilter"",""Name"":""Distrito"",""Values"":[]},{""Key"":""legalFormFilter"",""Name"":""Forma Jurídica"",""Values"":[]}],""Culture"":""pt""}"
If .status = 200 Then
MsgBox "Response: " & .responseText, vbInformation
GetVatId = Mid(.responseText, InStr(.responseText, """Vat"":""") + 7, 9)
Else
MsgBox "Repsonse status " & .status, vbExclamation
End If
End With
End Function
Usage:
Dim vatId As String
vatId = GetVatId("453060832") ' => "500004064"
For a more robust solution, you should use a JSON parser and -serializer, something like https://github.com/VBA-tools/VBA-JSON.

WinHttp.WinHttpRequest.5.1 URL ENCODE

I'm trying to use the Qrickt API:
https://qrickit.com/qrickit_apps/qrickit_api.php
to create a QRCode for Google Map address in VBA.
To do this I have to send a Http request like this:
"http://qrickit.com/api/qr.php?d=http://google.com/maps?q=Via+Roma,+1+Milano&qrsize=150&t=p&e=m"
The API documentation says:
*For non-English and special characters, url encode your data first.
The problem is that I cannot manage to pass an encoded address to the API.
If I pass a string such as "Via+Roma", or "Via%20Roma", the generated QRCode URL is always
http://maps.google.com/maps?q=Via Roma, 1 Milano
so the QRCode image is created, but phone do not open directly google maps.
Can somehome help me?
Here's the code:
Public Function f_QRCode(ByVal Address As String, ByVal Destination As String) As Boolean
On Error GoTo Err_Handler
Const ApiPath As String = "https://qrickit.com/api/qr.php?d=http://maps.google.com/maps?q="
Dim WinHttpReq As Object '\\ Oggetto che serve al download del verbale
Dim fic As Integer
Dim buffer() As Byte
Dim URL As String
'\\ Costruisco l'URL
URL = ApiPath + "Via%20Roma%2C%%201%20Milano" + "&qrsize=150&t=p&e=m"
'\\ Creo l'oggetto per la connessione
Set WinHttpReq = CreateObject("WinHttp.WinHttpRequest.5.1")
WinHttpReq.Open "POST", URL, False
WinHttpReq.setRequestHeader "Content-Type", "application/x-www-form-urlencoded"
WinHttpReq.send
If WinHttpReq.Status = 200 Then
fic = FreeFile
Open Destination For Binary Lock Read Write As #fic
buffer = WinHttpReq.responseBody
Put #fic, , buffer
Close #fic
f_QRCode = True
Else
MsgBox "Error"
End If
ExitHere:
Erase buffer
Set WinHttpReq = Nothing
Exit Function
Err_Handler:
Resume ExitHere
End Function
Their API accepts GET requests, and you're sending a POST.
Try:
URL = ApiPath + "Via%20Roma%2C%%201%20Milano" + "&qrsize=150&t=p&e=m"
Set WinHttpReq = CreateObject("WinHttp.WinHttpRequest.5.1")
WinHttpReq.Open "GET", URL, False
WinHttpReq.send
I would add that you might consider using the function EncodeURL for encoding.
Application.EncodeURL("url")

MSXML2.XMLHTTP works in a standalone function, access denied when called from a running procedure

I have a very simple function for returning the HTML code behind a web page from a VBA app:
Function GetSource(sURL As String) As Variant
' Purpose: To obtain the HTML text of a web page
' Receives: The URL of the web page
' Returns: The HTML text of the web page in a variant
Dim oXHTTP As Object, n As Long
Set oXHTTP = CreateObject("MSXML2.XMLHTTP")
oXHTTP.Open "GET", sURL, False
oXHTTP.send
GetSource = oXHTTP.responsetext
Set oXHTTP = Nothing
End Function
It works beautifully when I call it directly -- I get everything there is. However, whenever I try to call it from a running procedure, I get an "access denied" error (-2147024891).
I've tried playing around with the Internet Explorer object, but it only returns a fraction of what MSXML2.XMLHTTP returns -- and not what I want. Can anybody tell me how to overcome the error or get the Internet Explorer object to return what MSXML2.XMLHTTP returns?
This is what I use:
Public Function getHTTP(ByVal sReq As String) As String
On Error GoTo onErr
With CreateObject("MSXML2.XMLHTTP")
.Open "GET", sReq, False: .Send
getHTTP = StrConv(.responseBody, 64)
End With
Exit Function
onErr: MsgBox "Error "&Err &": "&Err.Description,49,"Error opening site"
End Function
It's the same idea as you're using with error handling added... I've never had an issue with it.
More Information:
MSDN: XMLHttpRequest object (Methods & Properties)
Wikipedia: XMLHttpRequest (XHR)

Web Scraping using VBA and MSXML2.XMLHTTP library

I'm trying to scrap data from a website using MSXML2.XMLHTTP object on VBA environment (Excel) and I cannot figure out how to solve this problem! The website is the following:
http://www.detran.ms.gov.br/consulta-de-debitos/
You guys can use the following test data to fill the form:
Placa: oon5868
Renavam: 1021783231
I want to retrieve data like "chassi", with the data above that would be " 9BD374121F5068077".
I do not have problems parsing the html document, the difficult is actually getting the information as response! Code below:
Sub SearchVehicle()
Dim strPlaca As String
Dim strRenavam As String
strPlaca = "oon5868"
strRenavam = "01021783231"
Dim oXmlPage As MSXML2.XMLHTTP60
Dim strUrl As String
Dim strPostData As String
Set oXmlPage = New MSXML2.XMLHTTP60
strUrl = "http://www2.detran.ms.gov.br/detranet/nsite/veiculo/veiculos/retornooooveiculos.asp"
strPostData = "placa=" & strPlaca & "&renavam=" & strRenavam
oXmlPage.Open "POST", strUrl, False
oXmlPage.setRequestHeader "Content-Type", "application/x-www-form-urlencoded"
oXmlPage.send strPostData
Debug.Print oXmlPage.responseText
End Sub
The strURL used in the POST method ".../retornooooveiculos.asp" is the one google developer tools and fiddler showed me that was the correct address the website was posting the payload.
When manually accessed, the website retrieve the correct information, but running my code I always get the following response on the .responseText:
<html>Acesse: <b><a href='http://www.detran.ms.gov.br target='_parent'>www.detran.ms.gov.br</a></b></html>
HELP PLEASE, I'm getting crazy trying to solve this puzzle! Why do I get redirected like this?
I need the "CHASSI" information and can't find the correct http Request to do this!
Try the below approach. It should fetch you the content you are after. The thing is you need to supply the Cookie copied from your Request Headers fields in order for your script to work which you can find using devtools.
Sub SearchVehicle()
Const URL As String = "http://www2.detran.ms.gov.br/detranet/nsite/veiculo/veiculos/retornooooveiculos.asp"
Dim HTTP As New ServerXMLHTTP60, HTML As New HTMLDocument
Dim elem As Object, splaca$, srenavam$, qsp$
splaca = "oon5868"
srenavam = "01021783231"
qsp = "placa=" & splaca & "&renavam=" & srenavam
With HTTP
.Open "POST", URL, False
.setRequestHeader "User-Agent", "Mozilla/5.0"
.setRequestHeader "Cookie", "ISAWPLB{07D08995-E67C-4F44-91A1-F6A16337ECD6}={286E0BB1-C5F9-4439-A2CE-A7BE8C3955E0}; ASPSESSIONIDSCSDSCTB=AGDPOBEAAPJLLMKKIGPLBGMJ; 69137927=967930978"
.setRequestHeader "Content-Type", "application/x-www-form-urlencoded"
.send qsp
HTML.body.innerHTML = .responseText
End With
For Each elem In HTML.getElementsByTagName("b")
If InStr(elem.innerText, "Chassi:") > 0 Then MsgBox elem.ParentNode.NextSibling.innerText: Exit For
Next elem
End Sub
Once again: fill in the Cookie field by collecting it using your devtools (from Request Headers section), if for some reason my provided Cookie doesn't work for you. Thanks.
Output I'm getting:
9BD374121F5068077

Trying to integrate an HTTP GET request in my MS-Access database program

I want to import data from Anedot, a credit card processing firm, using a HTTP GET request from an MS Access program. Anedot uses a RESTful API and has provided help on there website: https://anedot.com/api/v2
I want to do this with VBA, and associate the import with a button on an MS Access form. I've read that this only possible with XML. Do I create the XML file with VBA?
I'd greatly appreciate some background information on how to get this done, as most of it is flying over my head. I don't really know where to begin and I'm having trouble finding anything useful on google.
So far I've realized I'll need to reference their API via a URL link (which they provide), and that I'll have to authorize my account using my username and a token ID. But how can I do this in VBA?
Thanks.
First of all try to make a request to API using basic authorization. Take a look at the below code as the example:
Sub Test()
' API URL from https://anedot.com/api/v2
sUrl = "https://api.anedot.com/v2/accounts"
' The username is the registered email address of your Anedot account
sUsername = "mymail#example.com"
' The password is your API token
sPassword = "1e56752e8531647d09ec8ab20c311ba928e54788"
sAuth = TextBase64Encode(sUsername & ":" & sPassword, "us-ascii") ' bXltYWlsQGV4YW1wbGUuY29tOjFlNTY3NTJlODUzMTY0N2QwOWVjOGFiMjBjMzExYmE5MjhlNTQ3ODg=
' Make the request
With CreateObject("MSXML2.XMLHTTP")
.Open "GET", sUrl, False
.SetRequestHeader "Authorization", "Basic " & sAuth
.Send
Debug.Print .ResponseText
Debug.Print .GetAllResponseHeaders
End With
End Sub
Function TextBase64Encode(sText, sCharset) ' 05 10 2016
Dim aBinary
With CreateObject("ADODB.Stream")
.Type = 2 ' adTypeText
.Open
.Charset = sCharset ' "us-ascii" for bytes to unicode
.WriteText sText
.Position = 0
.Type = 1 ' adTypeBinary
aBinary = .Read
.Close
End With
With CreateObject("Microsoft.XMLDOM").CreateElement("objNode")
.DataType = "bin.base64"
.NodeTypedValue = aBinary
TextBase64Encode = Replace(Replace(.Text, vbCr, ""), vbLf, "")
End With
End Function
Put your credentials to sUsername and sPassword variables, choose the appropriate URL from API help page and put it to sURL. Then you can parse JSON response from the server (currently you will see the response for /v2/accounts request in Immediate window).
It's a fairly lengthy question to be honest, but lets start with some code to get you going.
This Class Module ("clsXMLHttpMonitor") should help:
Option Explicit
Dim XMLHttpReq As MSXML2.ServerXMLHTTP
Dim RequestedVar As String
Dim P As Object
Public Sub Initialize(ByVal uXMLHttpRequest As Object, Optional RequestedValue As String = "")
RequestedVar = RequestedValue
Set XMLHttpReq = uXMLHttpRequest
End Sub
Sub ReadyStateChangeHandler()
If XMLHttpReq.ReadyState = 4 Then
If XMLHttpReq.Status = 200 Then
'Process the response here
Debug.Print "200 recieved"
Set P = JSON.parse(XMLHttpReq.responseText)
Else
If XMLHttpReq.Status = 404 Then
'Handle it
End If
End If
End If
End Sub
Function returnResponseHeaders() As String
returnResponseHeaders = XMLHttpReq.getAllResponseHeaders
XMLHttpReq.Send
End Function
Function returnFullText() As String
If XMLHttpReq.ReadyState = 4 Then
If XMLHttpReq.Status = 200 Then
returnFullText = XMLHttpReq.responseText
Else
returnFullText = "-1"
End If
Else
returnFullText = ""
End If
End Function
End Function
Use it like this:
Set XMLHttpReq = New MSXML2.ServerXMLHTTP
Set XMLHttpMon = New clsXMLHttpMonitor
XMLHttpMon.Initialize XMLHttpReq
XMLHttpReq.OnReadyStateChange = XMLHttpMon
XMLHttpReq.Open "POST", URL, True
XMLHttpReq.Send strPayload
As you seem to request a Json response from a URL, you can study the Json modules here for a full implementation that collects the Json response in a collection, which you then can use in your code or save to a table. See the demo module for examples:
VBA.CVRAPI