Excel VBA: output the result of a WinHTTPRequest in Excel? - vba

How do I output the result of a WinHTTPRequest in Excel?
For example, the following code queries the stock quote of Apple from a webpage but it doesn't output anything:
Sub GetQuotes()
Dim XMLHTTP As Object, html As Object, pontod As Object
On Error Resume Next
Set oHtml = New HTMLDocument
With CreateObject("WINHTTP.WinHTTPRequest.5.1")
.Open "GET", "http://www.reuters.com/finance/stocks/overview?symbol=AAPL.O", False
.send
oHtml.body.innerHTML = .responseText
End With
'Price
Set pontod = oHtml.getElementsByClassName("sectionQuote nasdaqChange")(0).getElementsByTagName("span")(1)
MsgBox pontod.innerText
End Sub
While this runs perfectly for the name:
Sub GetQuotes2()
Dim XMLHTTP As Object, html As Object, pontod As Object
On Error Resume Next
Set oHtml = New HTMLDocument
With CreateObject("WINHTTP.WinHTTPRequest.5.1")
.Open "GET", "http://www.reuters.com/finance/stocks/overview?symbol=AAPL.O", False
.send
oHtml.body.innerHTML = .responseText
End With
'Name
Set pontod = oHtml.getElementById("sectionTitle").getElementsByTagName("h1")(0)
MsgBox pontod.innerText
End Sub
I'd like to be able to fetch the whole page and look for specific HTML elements in it, but how do I manage to see the whole response from the query?

As Jeeped said above, the method getElementsByClassName doesn't work on an XML request.
However, by looking at the webpage you're trying to scrape, you can work-around the issue by using this line:
Set pontod = oHtml.getElementById("headerQuoteContainer").getElementsByTagName("span")(1)
instead of this one:
Set pontod = oHtml.getElementsByClassName("sectionQuote nasdaqChange")(0).getElementsByTagName("span")(1)
As you can observe from the HTML structure of the webpage:
... your price is not only the second span element of the first div with class names sectionQuote and nasdaqChange, but also the second span element of the unique object with id headerQuoteContainer.
Hence, scrape it from there will avoid you to use the invalid method getElementsByClassName (which is a valid HTML method, but not when the HTML is an XML response) for the classic getElementById().

Related

Extract Data from URL VBA

I am trying to get Addresses Data from URL but facing some error. I am just beginner in VBA, i did not Understand where is problem in my code. wish somebody can help me to get right solution.
here I attached Image and also my VBA code
here is my Code
Public Sub IE_GetLink()
Dim sResponse As String, HTML As HTMLDocument
Dim url As String
Dim Re As Object
Set HTML = New HTMLDocument
Set Re = CreateObject("MSXML2.XMLHTTP")
'On Error Resume Next
url = "http://markexpress.co.in/network1.aspx?Center=360370&Tmp=1656224682265"
With Re
.Open "GET", url, False
.setRequestHeader "If-Modified-Since", "Sat, 1 Jan 2000 00:00:00 GMT"
.send
sResponse = StrConv(.responseBody, vbUnicode)
End With
Dim Title As Object
With HTML
.body.innerHTML = sResponse
Title = .querySelectorAll("#colspan")(0).innerText
End With
MsgBox Title
End Sub
Please help me ...
Several things.
What is wrong with your code:
Title should be a string as you are attempting to assign the return of .innerText to it. You have declared it as an object which would require SET keyword (and the removal of the .innerText accessor).
Colspan is an attribute not an id so your css selector list is incorrect.
Furthermore, looking at what the page actually does, there is a request for an additional document which actually has the info you need. You need to take the centre ID you already have and change the URI you make a request to.
Then, you want only the first td in the target table. Change your CSS selector list to target that.
Public Sub GetInfo()
Dim HTML As MSHTML.HTMLDocument
Dim re As Object
Set HTML = New MSHTML.HTMLDocument
Set re = CreateObject("MSXML2.XMLHTTP")
Dim url As String
Dim response As String
url = "http://crm.markerp.in/NetworkDetail.aspx?Center=360370&Tmp="
With re
.Open "GET", url, False
.setRequestHeader "If-Modified-Since", "Sat, 1 Jan 2000 00:00:00 GMT"
.send
response = .responseText
End With
Dim info As String
With HTML
.body.innerHTML = response
info = .querySelector("#tblDisp td").innerText
End With
MsgBox info
End Sub

VBA web page scroll

I am getting a problem to scroll document in a proper position, also getting a problem to capture a proper detail in excel here is my Code please Sir suggest me where I am getting wrong
here i try with following code still getting some error
Public Sub GData()
'On Error Resume Next
Dim html As HTMLDocument
Dim Re, Cr, cipherDict As Object
Dim sResponse, cipherKey, Str, SG As String
Dim myArr, RsltArr(14) As Variant
Set Re = CreateObject("vbscript.regexp")
Set Cr = CreateObject("MSXML2.XMLHTTP")
Set cipherDict = CreateObject("Scripting.Dictionary")
Set html = New HTMLDocument
URL = "https://www.google.com/maps/place/Silky+Beauty+Salon/#22.2932632,70.7723656,17z/data=!3m1!4b1!4m5!3m4!1s0x3959ca1278f4820b:0x44e998d30e14a58c!8m2!3d22.2932632!4d70.7745543"
With Cr
.Open "GET", URL, False
.setRequestHeader "If-Modified-Since", "Sat, 1 Jan 2000 00:00:00 GMT"
.send
sResponse = StrConv(.responseBody, vbUnicode)
s = .responseText
End With
With html
.body.innerHTML = sResponse
title = .querySelector("section-hero-header-title-title").innerText
phone = .querySelector("[data-item-id^=phone] [jsan*=text]").innerText
webSite = .querySelector("[aria-label^=Website] [jsan*=text]").innerText
End With
datarw = ActiveSheet.Cells(ActiveSheet.Rows.Count, "A").End(xlUp).Row + 1
ActiveSheet.Cells(datarw, 1).Value = title
ActiveSheet.Cells(datarw, 5).Value = phone
ActiveSheet.Cells(datarw, 7).Value = webSite
ActiveSheet.Cells(datarw, 1).Select
ActiveSheet.Rows(datarw).WrapText = False
End Sub
Looks like you can use combinations of different combinators (^ starts with and * contains) to search for substrings in attributes on the page to get your target nodes. Using descendant combinators to specify the relationship between attributes being used for anchoring.
Test if matched node Is Not Nothing before attempting to access either an attribute value or .innerText
Dim phone as Object, webSite As Object, title As Object
Set title = ie.document.querySelector(".section-hero-header-title-title")
Set phone = ie.document.querySelector("[data-item-id^=phone] [jsan*=text]")
Set website = ie.document.querySelector("[aria-label^=Website] [jsan*=text]")
If Not phone Is Nothing Then
'clean phone.innerText as appropriate
End If
If Not website Is Nothing Then
'clean website.innerText as appropriate
End If
To get the appropriate protocol for the website address, if missing, you can use the cleaned website address you have in a regex to pull the protocol from earlier in the html where it sits in a script tag.
Read about
css selectors: https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Selectors
querySelector: querySelector and querySelectorAll vs getElementsByClassName and getElementById in JavaScript

VBA GetElementsById Method "Object Variable Not Set"

I'm trying to select the main menu ID of this page http://greyhoundstats.co.uk/index.php labeled ("menu_wholesome") in order to get their hyperlinks later on. In the HTML document, there are two tags with this ID, a <div> and its child element <ul>, but when i search for them with the code below, i get the object variable not set" error.
Option Explicit
Public Const MenuPage As String = "http://greyhoundstats.co.uk/index.php"
Sub BrowseMenus()
Dim XMLHTTPReq As New MSXML2.XMLHTTP60
Dim HTMLDoc As New MSHTML.HTMLDocument
Dim MainMenuList As MSHTML.IHTMLElement
Dim aElement As MSHTML.IHTMLElementCollection
Dim ulElement As MSHTML.IHTMLUListElement
Dim liElement As MSHTML.IHTMLLIElement
XMLHTTPReq.Open "GET", MenuPage, False
XMLHTTPReq.send
HTMLDoc.body.innerText = XMLHTTPReq.responseText
Set MainMenuList = HTMLDoc.getElementById("menu_wholesome")(0) '<-- error happens here
End Sub
Anyone knows why getElementsById can't find the refered ID, although it is part of the HTML document set? I know that this method is supposed to return a unique ID, but when we have the same one refered by other tags i also know that i will return the first ID found which should be the <div id="menu_wholesome"> part of the HTML page being requested.
Firstly: You want to work and set the innerHTML as you intend to traverse a DOM document.
Secondly: This line
Set MainMenuList = HTMLDoc.getElementById("menu_wholesome")(0)
It is incorrect. getElementById returns a single element which you cannot index into. You index into a collection.
Please note: Both div and and ul lead to the same content.
If you want to select them separately use querySelector
HTMLDoc.querySelector("div#menu_wholesome")
HTMLDoc.querySelector("ul#menu_wholesome")
The above target by tag name first then the id attribute.
If you want a collection of ids then use querySelectorAll to return a nodeList of matching items. Ids should be unique to the page but sometimes they are not!
HTMLDoc.querySelectorAll("#menu_wholesome")
You can then index into the nodeList e.g.
HTMLDoc.querySelectorAll("#menu_wholesome").item(0)
VBA:
Option Explicit
Public Const MenuPage As String = "http://greyhoundstats.co.uk/index.php"
Sub BrowseMenus()
Dim sResponse As String, HTMLDoc As New MSHTML.HTMLDocument
Dim MainMenuList As Object, div As Object, ul As Object
With CreateObject("MSXML2.XMLHTTP")
.Open "GET", MenuPage, False
.setRequestHeader "If-Modified-Since", "Sat, 1 Jan 2000 00:00:00 GMT"
.send
sResponse = StrConv(.responseBody, vbUnicode)
End With
sResponse = Mid$(sResponse, InStr(1, sResponse, "<!DOCTYPE "))
HTMLDoc.body.innerHTML = sResponse
Set MainMenuList = HTMLDoc.querySelectorAll("#menu_wholesome")
Debug.Print MainMenuList.Length
Set div = HTMLDoc.querySelector("div#menu_wholesome")
Set ul = HTMLDoc.querySelector("ul#menu_wholesome")
Debug.Print div.outerHTML
Debug.Print ul.outerHTML
End Sub
It is unclear what are you trying to achieve. I just fixed the current problem you are having at this moment. .getElementById() deals with an individual element so when you treats it as a collection of element then it will throws that error. If you notice this portion getElementBy and getElementsBy, you can see the variation as to which one is a collection of elements (don't overlook the s). You can only use (0) or something similar when you make use of getElementsBy.
You should indent your code in the right way so that others can read it without any trouble:
Sub BrowseMenus()
Const MenuPage$ = "http://greyhoundstats.co.uk/index.php"
Dim HTTPReq As New XMLHTTP60, HTMLDoc As New HTMLDocument
Dim MainMenuList As Object
With HTTPReq
.Open "GET", MenuPage, False
.send
HTMLDoc.body.innerHTML = .responseText
End With
Set MainMenuList = HTMLDoc.getElementById("menu_wholesome")
End Sub

VBA: Run-time error 424: Object required when trying to web scrape

I'm trying to update various fund sizes using morgninstar.co.uk. The code worked fine until it suddenly stopped and gave an error:
"Run-time error 424: Object required".
The exact line where the error occurs is:
Set allData = IE.document.getElementById("overviewQuickstatsDiv").getElementsByTagName("tbody")(0)
The idea is to ultimately scan the whole "tbody"-tag and look for the line "Fund Size" inside "tr" and "td"-tags. When "Fund Size" is found, the code would return the 3rd "td"-tag (actual fund size).
After this I'd add a loop to loop through a list of funds that I've got.
As the code stopped working completely, I haven't got this far yet. Here I'm just trying to check if the code returns the actual fund size.
Since there are not always 3 "td"-tags inside the "tr"-tags, I'll still have to construct some sort of IF-statement to fix that issue.
But for now I'd just want to know how I could get the code running again? I've spent great deal of time searching for an answer but as it seems that this is a variable type problem the solution depends on the situation.
I'm using Excel 2010 and Internet Explorer 11.
URL in easy form to copy-paste:
http://www.morningstar.co.uk/uk/funds/snapshot/snapshot.aspx?id=F0GBR04BKW
Sub testToScrapeWholeTbodyTag()
'Microsoft Internet Controls
'Microsoft HTML Object Library
'Microsoft Shell Controls and Automation
'======Opens URL======
Dim IE As Object
Set IE = CreateObject("internetexplorer.application")
With IE
.navigate "http://www.morningstar.co.uk/uk/funds/snapshot/snapshot.aspx?id=F0GBR04BKW"
.Visible = False
End With
While IE.Busy
DoEvents
Wend
'======Got from internet, fixed a previous error. However, I'm not 100% sure what this does======
Dim sh
Dim eachIE As Object
Do
Set sh = New Shell32.Shell
For Each eachIE In sh.Windows
If InStr(1, eachIE.LocationURL, "http://www.morningstar.co.uk/uk/funds/snapshot/snapshot.aspx?id=F0GBR04BKW") Then
Set IE = eachIE
IE.Visible = False '"This is here because in some environments, the new process defaults to Visible."
Exit Do
End If
Next eachIE
Loop
Set eachIE = Nothing
Set sh = Nothing
'======Looks for the "Fund Size"======
'Trying to look for "Fund Size" inside "tr"-tag and if found, return the value in the 3rd "tr"-tag
Set allData = IE.document.getElementById("overviewQuickstatsDiv").getElementsByTagName("tbody")(0) 'Run-time error 424: Object required
row1 = allData.getElementsByTagName("tr")(5).Cells(0).innerHTML
row2 = allData.getElementsByTagName("tr")(5).Cells(1).innerHTML
row3 = allData.getElementsByTagName("tr")(5).Cells(2).innerHTML
If Left(row1, 9) = "Fund Size" Then
Worksheets("Sheet3").Range("B3") = Split(row3, ";")(1)
End If
Debug.Print allData.getElementsByTagName("tr")(5).Cells(0).innerHTML '"Fund Size"
Debug.Print allData.getElementsByTagName("tr")(5).Cells(2).innerHTML 'Actual fund size
IE.Quit
Set IE = Nothing
End Sub
EDIT:
Switched method. Now the problem is to get the fund size extracted. So the below code works as it is but I'd need to add a couple of lines to get the fund size out of it. This is my first time using this method so it may well be that I've just not understood some really basic thing. Still, I wasn't able to find a solution to this on my own.
Sub XMLhttpRequestTest()
'Microsoft XML, v 6.0
'Microsoft HTML object library
Dim HTMLDoc As New HTMLDocument
Dim ohttp As New MSXML2.XMLHTTP60
Dim myurl As String
Dim TRelements As Object
Dim TRelement As Object
myurl = "http://www.morningstar.co.uk/uk/funds/snapshot/snapshot.aspx?id=F0GBR04BKW"
ohttp.Open "GET", myurl, False
ohttp.send
HTMLDoc.body.innerHTML = ohttp.responseText
With HTMLDoc.body
Set TRelements = .getElementsByTagName("tr")
For Each TRelement In TRelements
Debug.Print TRelement.innerText
Next
End With
End Sub
You can use a css selector of
#overviewQuickstatsDiv td.line.text
And then select the element at index 4
# means id. . = className.
Option Explicit
Public Sub XMLhttpRequestTest()
'Microsoft XML, v 6.0
'Microsoft HTML object library
Dim HTMLDoc As New HTMLDocument, ohttp As New MSXML2.XMLHTTP60
Const URL As String = "http://www.morningstar.co.uk/uk/funds/snapshot/snapshot.aspx?id=F0GBR04BKW"
Dim TRelements As Object, TRelement As Object
With ohttp
.Open "GET", URL, False
.send
HTMLDoc.body.innerHTML = .responseText
Debug.Print HTMLDoc.querySelectorAll("#overviewQuickstatsDiv td.line.text")(4).innerText
'Your other stuff
End With
End Sub

Error when changing IE automation code to XML

I recently started working with XML automation and after changing some basic IE automation code over, I seem to be getting an error. Here's the HTML:
<tbody>
<tr class="group-2 first">
<td class="date-col">
<a href="/stats/matches/mapstatsid/48606/teamone-vs-merciless">
<div class="time" data-time-format="d/M/yy" data-unix="1498593600000">27/6/17</div>
</a>
</td>
......SOME MORE HTML HERE......
</tr>
......SOME MORE HTML HERE......
</tbody>
And here's the code i'm using in Excel VBA:
Sub readData()
Dim XMLPage As New MSXML2.XMLHTTP60
Dim html As New MSHTML.HTMLDocument
XMLPage.Open "GET", "https://www.hltv.org/stats/matches", False
XMLPage.send
If XMLPage.Status <> 200 Then MsgBox XMLPage.statusText
html.body.innerHTML = XMLPage.responseText
For Each profile In html.getElementsByTagName("tbody")(0).Children
Debug.Print profile.getElementsByClassName("date-col")(0).getElementsByTagName("a")(0).getAttribute("href") 'Run time error '438' here
Next
End Sub
I'm getting the Run time error '438' at the debug print code. seems to be happening when getting the class but i'm unsure why. It works fine if I use this for example:
Debug.Print profile.innertext
Worked for me:
Sub readData()
Dim XMLPage As New MSXML2.XMLHTTP60
Dim html As New MSHTML.HTMLDocument, links, a, i
XMLPage.Open "GET", "https://www.hltv.org/stats/matches", False
XMLPage.send
If XMLPage.Status <> 200 Then MsgBox XMLPage.statusText
html.body.innerHTML = XMLPage.responseText
Set links = html.querySelectorAll("td.date-col > a")
Debug.Print links.Length
For i = 0 To links.Length - 1
Debug.Print links(i).href
Next
Set links = Nothing
Set html = Nothing
End Sub
FYI when I used For Each to loop over the links collection Excel would reliably crash, so I'd stay with the loop shown
profile refers to a row, and profile.cells(0) will refer to the first column in that row. So try...
profile.cells(0).getElementsByTagName("a")(0).getAttribute("href")
Also, profile should be declared as HTMLTableRow.
The URL you are using isn't serving valid XML, but it's recoverable with some simple regex replacements. Once we have some valid XML, we can load that into a DOM document and use XPath to select the nodes as required:
Option Explicit
'Add references to:
' - MSXML v3
' - Microsoft VBScript Regular Expressions 5.5
Sub test()
Const START_MARKER As String = "<table class=""stats-table matches-table"">"
Const END_MARKER As String = "</table>"
With New MSXML2.XMLHTTP
.Open "GET", "https://www.hltv.org/stats/matches", False
.send
If .Status = 200 Then
'The HTML isn't valid XHTML, so we can't just use the http.XMLResponse DOMDocument
'Let's extract the HTML table
Dim tableStart As Long
tableStart = InStr(.responseText, START_MARKER)
Dim tableEnd As Long
tableEnd = InStr(tableStart, .responseText, END_MARKER)
Dim tableHTML As String
tableHTML = Mid$(.responseText, tableStart, tableEnd - tableStart + Len(END_MARKER))
'The HTML table has invalid img tags (let's add a closing tag with some regex)
With New RegExp
.Global = True
.Pattern = "(\<img [\W\w]*?)"">"
Dim tableXML As String
tableXML = .Replace(tableHTML, "$1"" />")
End With
'And load an XML document from the cleaned up HTML fragment
Dim doc As MSXML2.DOMDocument
Set doc = New MSXML2.DOMDocument
doc.LoadXML tableXML
End If
End With
If Not doc Is Nothing Then
'Use XPath to select the nodes we need
Dim nodes As MSXML2.IXMLDOMSelection
Set nodes = doc.SelectNodes("//td[#class='date-col']/a/#href")
'Enumerate the URLs
Dim node As IXMLDOMAttribute
For Each node In nodes
Debug.Print node.nodeTypedValue
Next node
End If
End Sub
Output:
/stats/matches/mapstatsid/48606/teamone-vs-merciless
/stats/matches/mapstatsid/48607/merciless-vs-teamone
/stats/matches/mapstatsid/48608/merciless-vs-teamone
/stats/matches/mapstatsid/48600/wysix-vs-fnatic-academy
/stats/matches/mapstatsid/48602/skitlite-vs-nexus
/stats/matches/mapstatsid/48604/extatus-vs-forcebuy
/stats/matches/mapstatsid/48605/extatus-vs-forcebuy
/stats/matches/mapstatsid/48599/planetkey-vs-gatekeepers
/stats/matches/mapstatsid/48603/gatekeepers-vs-planetkey
/stats/matches/mapstatsid/48595/wysix-vs-gambit
/stats/matches/mapstatsid/48596/kinguin-vs-playing-ducks
/stats/matches/mapstatsid/48597/spirit-academy-vs-tgfirestorm
/stats/matches/mapstatsid/48601/spirit-academy-vs-tgfirestorm
/stats/matches/mapstatsid/48593/fnatic-academy-vs-gambit
/stats/matches/mapstatsid/48594/alternate-attax-vs-nexus
/stats/matches/mapstatsid/48590/pro100-vs-playing-ducks
/stats/matches/mapstatsid/48583/extatus-vs-ex-fury
/stats/matches/mapstatsid/48589/extatus-vs-ex-fury
/stats/matches/mapstatsid/48584/onlinerol-vs-forcebuy
/stats/matches/mapstatsid/48591/forcebuy-vs-onlinerol
/stats/matches/mapstatsid/48581/epg-vs-veni-vidi-vici
/stats/matches/mapstatsid/48588/epg-vs-veni-vidi-vici
/stats/matches/mapstatsid/48592/veni-vidi-vici-vs-epg
/stats/matches/mapstatsid/48582/log-vs-gatekeepers
/stats/matches/mapstatsid/48586/gatekeepers-vs-log
/stats/matches/mapstatsid/48580/spraynpray-vs-epg
/stats/matches/mapstatsid/48579/quantum-bellator-fire-vs-spraynpray
/stats/matches/mapstatsid/48571/noxide-vs-masterminds
/stats/matches/mapstatsid/48572/athletico-vs-legacy
/stats/matches/mapstatsid/48578/node-vs-avant
/stats/matches/mapstatsid/48573/funky-monkeys-vs-grayhound
/stats/matches/mapstatsid/48574/grayhound-vs-funky-monkeys
/stats/matches/mapstatsid/48575/hegemonyperson-vs-eclipseo
/stats/matches/mapstatsid/48577/eclipseo-vs-hegemonyperson
/stats/matches/mapstatsid/48566/masterminds-vs-tainted-black
/stats/matches/mapstatsid/48562/grayhound-vs-legacy
/stats/matches/mapstatsid/48563/noxide-vs-riotous-raccoons
/stats/matches/mapstatsid/48564/avant-vs-dark-sided
/stats/matches/mapstatsid/48565/avant-vs-dark-sided
/stats/matches/mapstatsid/48567/eclipseo-vs-uya
/stats/matches/mapstatsid/48568/uya-vs-eclipseo
/stats/matches/mapstatsid/48560/uya-vs-new4
/stats/matches/mapstatsid/48561/new4-vs-uya
/stats/matches/mapstatsid/48559/jaguar-sa-vs-miami-flamingos
/stats/matches/mapstatsid/48558/spartak-vs-binary-dragons
/stats/matches/mapstatsid/48557/kungar-vs-spartak
/stats/matches/mapstatsid/48556/igamecom-vs-fragsters
/stats/matches/mapstatsid/48554/nordic-warthogs-vs-aligon
/stats/matches/mapstatsid/48555/binary-dragons-vs-kungar
/stats/matches/mapstatsid/48550/havu-vs-rogue-academy
Looking at the MSHTML.HTMLDocument reference there is no method getElementsByClassName.
You will need to loop through each row in the tbody you are selecting and then get the first td in that row and then get the first link in that td and read the href attribute from it. You could alternately compare the class attribute of the td but since it is the first element in the row there is no need to do that.