Accessing webservice with credentials using vba - vba

I'm working on a longer script for Access and at one point it is necessary to check a webservice for the latest version of a file (the filename). This webservice is only accessible via a browser with an URL like https://webservice.example.com:1234/Server/test.jsp?parameter=value then it is necessary to authenticate with the standard browser username password pop up.
Of course I could skip this pop up if I'd use something like https://user:password#webservice.example.com:1234/Server/test.jsp?parameter=value instead. (Note that it is not about security at this point the password only exists for the sake of having a password and it's totally acceptable to store it as clear text)
At the moment I already use the following working code to get information from another
website:
Dim appIE As Object
Dim sURL as String, infoStr as String
Set appIE = GetObject("new:{D5E8041D-920F-45e9-B8FB-B1DEB82C6E5E}") 'class id of InternetExplorerMedium
sURL = "https://webservice.example.com:1234/Server/test.jsp?parameter=value"
With appIE
.Navigate sURL
.Visible = False
End With
Do While appIE.Busy Or appIE.ReadyState <> 4
DoEvents
Loop
infoStr = appIE.Document.getElementsByTagName("body").item.innerText
However, if I add the credentials to the URL as I would do in the browser
sURL = "https://user:password#webservice.example.com:1234/Server/test.jsp?parameter=value"
I will get the following error:
Runtime error '-2146697202 (800c000e)': method 'navigate' of object
'IWebBrowser2' failed
Does anybody know why it is failing if I add the credentials or has anybody an idea how to do this differently?

If your website requires Basic authentication, it's relatively easy to authenticate using a basic authentication header.
We need to be able to Base64 encode content, so first we need to define a helper function for that:
Public Function ToBase64(Bytes() As Byte) As String
Dim XMLElement As Object
Set XMLElement = CreateObject("Msxml2.DOMDocument.6.0").createElement("tmp")
XMLElement.DataType = "bin.base64"
XMLElement.nodeTypedValue = Bytes
ToBase64 = Replace(XMLElement.Text, vbLf, "")
End Function
Then, a second helper to create a basic authentication header:
Public Function CreateBasicAuthHeader(Username As String, Password As String) As String
'Assuming ASCII encoding, UTF-8 is harder
CreateBasicAuthHeader = "Authorization: Basic " & ToBase64(StrConv(Username & ":" & Password, vbFromUnicode))
End Function
A quick validation shows that ?CreateBasicAuthHeader("Aladdin", "OpenSesame") returns Authorization: Basic QWxhZGRpbjpPcGVuU2VzYW1l, which is the expected header according to Wikipedia
Then, you can use this in the Navigate method:
Dim appIE As Object
Dim sURL as String, infoStr as String
Set appIE = GetObject("new:{D5E8041D-920F-45e9-B8FB-B1DEB82C6E5E}") 'class id of InternetExplorerMedium
sURL = "https://webservice.example.com:1234/Server/test.jsp?parameter=value"
With appIE
.Navigate sURL, Headers:=CreateBasicAuthHeader("MyUsername", "MyPassword")
.Visible = False
End With
Do While appIE.Busy Or appIE.ReadyState <> 4
DoEvents
Loop
infoStr = appIE.Document.getElementsByTagName("body").item.innerText
This assumes that the server either expects ASCII encoding, or your username and password are both only ASCII characters and the server expects UTF-8 encoding.

Related

MSXML2.XMLHTTP works in a standalone function, access denied when called from a running procedure

I have a very simple function for returning the HTML code behind a web page from a VBA app:
Function GetSource(sURL As String) As Variant
' Purpose: To obtain the HTML text of a web page
' Receives: The URL of the web page
' Returns: The HTML text of the web page in a variant
Dim oXHTTP As Object, n As Long
Set oXHTTP = CreateObject("MSXML2.XMLHTTP")
oXHTTP.Open "GET", sURL, False
oXHTTP.send
GetSource = oXHTTP.responsetext
Set oXHTTP = Nothing
End Function
It works beautifully when I call it directly -- I get everything there is. However, whenever I try to call it from a running procedure, I get an "access denied" error (-2147024891).
I've tried playing around with the Internet Explorer object, but it only returns a fraction of what MSXML2.XMLHTTP returns -- and not what I want. Can anybody tell me how to overcome the error or get the Internet Explorer object to return what MSXML2.XMLHTTP returns?
This is what I use:
Public Function getHTTP(ByVal sReq As String) As String
On Error GoTo onErr
With CreateObject("MSXML2.XMLHTTP")
.Open "GET", sReq, False: .Send
getHTTP = StrConv(.responseBody, 64)
End With
Exit Function
onErr: MsgBox "Error "&Err &": "&Err.Description,49,"Error opening site"
End Function
It's the same idea as you're using with error handling added... I've never had an issue with it.
More Information:
MSDN: XMLHttpRequest object (Methods & Properties)
Wikipedia: XMLHttpRequest (XHR)

How to login to a website using vba?

I am trying to login the mentioned website using vba but unable to get through.kindly assist. The website i am trying to logging contains a form where we are suppose to fill our credentials.though, the website is opening using the below enlisted code but nothing happens post that.
Dim HTMLDoc as HTMLDocument
Dim My Browser as Internet Explorer
Sub MYRED()
Dim MyHTML_Element As IHTMLElement
Dim MyURL as String
On Error GoTo Err_Clear
MyURL = "https://www.Markit.com"
Set MyBrowser = New InternetExplorerMedium
MyBrowser.Silent = True
MyBrowser.navigate MyURL`enter code here`
MyBrowser. Visible = True`enter code here`
Do
Loop Until MyBrowser.readyState = READYSTATE_COMPLETE
Set HTMLDoc = MyBrowser. Document
'Useridele = "input_5ac52c16-cbba-4fb4-b284-857fac1f55fd"
MyBrowser.document.getElementById ("input_5ac52c16-cbba-4fb4-b284- 857fac1f55fd").Value = "user#example.com"
'HTMLDoc.all.input_5ac52c16-cbba-4fb4-b284-857fac1f55fd.Value = "atul.sanwal#markit.com" 'Enter your email id here
passele = "input_b14b8d03-75b6-49b5-a61a-602672036046"
MyBrowser.document.getElementById("input_b14b8d03-75b6-49b5-a61a- 602672036046").Value = "***************"
'HTMLDoc.all.passele.Value = "12345678" 'Enter your password here
For Each My Html_Element in HTMLDoc.getElementsByTagName("input")
If MyHTML_Element.Type = "submit" Then MyHTML_Element.Click: Exit For
Next
Err_Clear:
If Err <> 0 Then
Err.Clear
Resume Next
End If
End Sub
As I can see this:
"input_5ac52c16-cbba-4fb4-b284-857fac1f55fd")
I'm guessing that this ID is generated every time you are making request
(yep, everytime I'm refreshing this site this element got new ID ), so
You cannot pass static ID as key to get elementById You should try other options to get this specific element You want to fill with data.
Maybe Your library supports gettibng by attribute so, You can get element with atribute
name='username'
One additional proposition :
You are using html to make request via web Interface, but You could try to ommit loging in just to make pure requests.
As I see when filling data and pushing Submit my browser is sending
POST request :
POST /home/UFEExternalSignOnServlet HTTP/1.1
Host: products.markit.com
I dont know how large is your knowledge about WebAPI and etc. but You can write to me if need more help

Fetching webpage source when the response is different in logged in and logged out state

With my code (given below), I capture source of my website's webpage and then I am trying to find a string with InStr and process it with Split. Everything is working but I have an issue where I need help.
When I am logged in to my website supplying the username and password, I can see the string in the webpage source but when I am in a logged out state, my string is not visible in the webpage source. What changes do I need in the code which will tell the computer that I am logged in.
Is this a cache/cookies issue? if yes, how to code it so that it honors the browser cache/cookies remembering that the user is logged in and gets the source as it shows up in logged in state?
Appreciate a reply. Thanks
Sub source()
Dim oHttp As New WinHttp.WinHttpRequest
Dim sURL As String
Dim webpageSource As String
sURL = "http://www.google.com"
oHttp.Open "GET", sURL, False
oHttp.send
webpageSource = oHttp.ResponseText
End Sub

MSXML2.XMLHTTP page request: How do you make sure you get ALL of the final HTML code?

I've used this simple subroutine for loading HTML documents from the web for some time now with no problems:
Function GetSource(sURL As String) As Variant
' Purpose: To obtain the HTML text of a web page
' Receives: The URL of the web page
' Returns: The HTML text of the web page in a variant
Dim oXHTTP As Object, n As Long
Set oXHTTP = CreateObject("MSXML2.XMLHTTP")
oXHTTP.Open "GET", sURL, False
oXHTTP.send
GetSource = oXHTTP.responsetext
Set oXHTTP = Nothing
End Function
but I've run into a situation where it only loads part of a page most of the time (not always -- sometimes it loads all of the expected HTML code). If you SAVE the HTML of the page to another file on the web from a browser, the subroutine will always read it with no problem.
I'm guessing that the issue is timing -- that the dynamic page registers "done" while a script is still filling in details. Sometimes it completes in time, other times it doesn't.
Has anyone ever encountered this behavior before and surmounted it? It seems that there should be a way of capturing via the MSXML2.XMLHTTP object exactly what you'd get if went to the page and chose the save to HTML option.
If you'd like to see the behavior for yourself, here's a sample of a page that doesn't load consistently:
http://www.tiff.net/festivals/thefestival/programmes/specialpresentations/mr-turner
and here's a saved HTML file of that same page:
http://tofilmfest.ca/2014/film/fest/Mr_Turner.htm
Is there any known workaround for this?
I found a workaround that gives me what I want. I control Internet Explorer programmatically and invoke a three-second delay after I tell it to navigate to a page to enable the content to finish loading. Then I extract the HTML code by using an IHTMLElement from Microsoft's HTML library. It's not pretty, but it retrieves all of the HTML code for every page I've tried it with. If anybody has a better way accomplishing the same end, feel free to show off.
Function testbrowser() As Variant
Dim oIE As InternetExplorer
Dim hElm As IHTMLElement
Set oIE = New InternetExplorer
oIE.Height = 600
oIE.Width = 800
oIE.Visible = True
oIE.Navigate "http://www.tiff.net/festivals/thefestival/programmes/galapresentations/the-riot-club"
Call delay(3)
Set hElm = oIE.Document.all.tags("html").Item(0)
testbrowser = hElm.outerHTML
End Function
Sub delay(ByVal secs As Integer)
Dim datLimit As Date
datLimit = DateAdd("s", secs, Now())
While Now() < datLimit
Wend
End Sub
Following Alex's suggestion, here's how to do it without a brute force fixed delay:
Function GetHTML(ByVal strURL as String) As Variant
Dim oIE As InternetExplorer
Dim hElm As IHTMLElement
Set oIE = New InternetExplorer
oIE.Navigate strURL
Do While (oIE.Busy Or oIE.ReadyState <> READYSTATE_COMPLETE)
DoEvents
Loop
Set hElm = oIE.Document.all.tags("html").Item(0)
GetHTML = hElm.outerHTML
Set oIE = Nothing
Set hElm = Nothing
End Function

Crawl to final URL from within Excel VBA

I have a list of domain names, and many of them redirect me to the same domain. For instance... foo1.com, foo2.csm and foo3.com all deposit me at foo.com.
I'm trying to deduplicate the list of domains by writing a VBA script to load the final page and extract it's URL.
I've started from this article which retrieve's the page's title (http://www.excelforum.com/excel-programming-vba-macros/355192-can-i-import-raw-html-source-code-into-excel.html), but can't figure out how to modify it to get the final URL (from which I can extract the URL.
Can anyone please point me in the right direction?
Add a reference to "Microsoft XML, v3.0" (or whatever version you have)
Sub tester()
Debug.Print CheckRedirect("adhpn2.com")
End Sub
Function CheckRedirect(URL As String)
If Not UCase(URL) Like "HTTP://*" Then URL = "http://" & URL
With New msxml2.ServerXMLHTTP40
.Open "HEAD", URL, False
.send
CheckRedirect = .getOption(-1)
End With
End Function
Try this, need to look at .LocationURL:
Public Function gsGetFinalURL(rsURL As String) As String
Dim ie As Object
Set ie = CreateObject("InternetExplorer.Application")
With ie
.navigate rsURL
Do While .Busy And Not .ReadyState = 4
DoEvents
Loop
gsGetFinalURL = .LocationURL
.Quit
End With
Set ie = Nothing
End Function
I haven't tried it on a huge variety of URLs, just the one you gave and a couple of others. If it is an invalid URL it will return what is passed. You can use the code from the original function to check and handle accordingly.