Web scraping - create object for IE - vba

Sub Get_Data()
Set ie = CreateObject("InternetExplorer.Application")
ie.Visible = True
ie.Navigate "http://www.scramble.nl/military-database/usaf"
Do While ie.Busy
Application.Wait DateAdd("s", 1, Now)
Loop
SendKeys "03-3114"
SendKeys "{ENTER}"
End Sub
The code below searches for keyboard typed value 03-3114 and gets a data in the table. If I 'd like to search for value which is already in cell A1 and scrape values from table for "Code, Type, CN, Unit" in cell range ("B1:E1") what should I do?

You are using SendKeys which are highly unreliable :) Why not find the name of the textbox and the search button and directly interact with it as shown below?
Sub Get_Data()
Dim ie As Object, objInputs As Object
Set ie = CreateObject("InternetExplorer.Application")
ie.Visible = True
ie.Navigate "http://www.scramble.nl/military-database/usaf"
Do While ie.readystate <> 4: DoEvents: Loop
'~~> Get the ID of the textbox where you want to output
ie.Document.getElementById("serial").Value = "03-3114"
'~~> Here we try to identify the search button and click it
Set objInputs = ie.Document.getElementsByTagName("input")
For Each ele In objInputs
If ele.Name Like "sbm" Then
ele.Click
Exit For
End If
Next
End Sub
Note: To understand how I got the names serial and sbm, refer to the explanation given just above the image below.
The code below searches for keyboard typed value 03-3114 and gets a data in the table. If I 'd like to search for value which is already in cell A1 and scrape values from table for "Code, Type, CN, Unit" in cell range ("B1:E1") what should I do?
Directly put the value from A1 in lieu of the hardcoded value
ie.Document.getElementById("serial").Value = Sheets("Sheet1").Range("A1").Value
To get the values from the table, identify the elements of the table by right clicking on it in the browser and clicking on "Inspect/Inspect Element(In Chrome it is just Inspect)" as shown below.
I can give you the code but I want you to do it yourself. If you are still stuck then update the question with the code that you tried and then we will take it from there.
Interesting read: html parsing of cricinfo scorecards

Related

VBA- wanted to click element which has multiple frames

With the previous question's answer I stepped ahead however still stuck on one point where I wanted to click the element but it seems it has 2 frames and thats the reason when I am running code its giving error that element not found, please check attachment and below code and advise.
Sub activeBexIE_Final()
Application.ScreenUpdating = False
Application.DisplayAlerts = False
'On Error Resume Next
Dim Perm_bot As New Selenium.IEDriver
Perm_bot.Get "official link"
Perm_bot.Wait 2000
Perm_bot.FindElementById("logonuidfield").SendKeys "XYZ"
Perm_bot.SendKeys Perm_bot.Keys.Tab
Perm_bot.SendKeys "PQR"
Perm_bot.SendKeys Perm_bot.Keys.Enter
Perm_bot.Wait 40000
Perm_bot.switchToFrame
Perm_bot.FindElementById("iframe_Roundtrip_9223372036563636042")
Perm_bot.FindElementById("BUTTON_OPEN_SAVE_btn1_acButton").SendKeys
Perm_bot.Keys.Enter
Perm_bot.Wait 30000' *till here I am sucessful- Opening Url-login-click on
"Open" option*
*'from here I am unable to move further, pls check attachment"select
layout" like how I wanted to select and also check the attachment "frame"
and "element" which is having the HTML details, not sure if this particular
element has 2 element, if yes then how I should write the script and after
this I wanted to do tab 7 times and then enter, kindly help for further
script, thanks in advance
'Perm_bot.switchToFrame Perm_bot.FindElementById("urPopupOuter0")
Perm_bot.switchToFrame Perm_bot.FindElementById("urPopupInner0")
Perm_bot.FindElementById("LOAD_state_tigen4_tlv1_list_unid6_tv").Click
SendKeys "{TAB}"
SendKeys "{TAB}"
SendKeys "{TAB}"
SendKeys "{TAB}"
SendKeys "{TAB}"
SendKeys "{TAB}"
SendKeys "{TAB}"
SendKeys "{ENTER}"
Perm_bot.Wait 10000
Application.DisplayAlerts = True
Application.ScreenUpdating = True
'perm_dot.Quit
'Set perm_dot = Nothing
End Sub
Select Layout
FrameElement
I believe your concern is your iframes are having similar attributes and you have to switch to each one of it before tabing 7 times.
Approach would be
Just like getting all windows handles and then loop you can use common xpath for iframe for e.g
findelements(By.Xpath("//iframes))
this will return iframe elements and then you can use loop and traverse and use
SwitchToFrame().
If your webpage is containing multiple frames and you need to perform an action on each frame too then I suggest trying to create an object for those frames and assign it to the 'document' object and then try to perform the action on each frame.
Sub demo()
Dim URL As String
Dim IE, doc1, doc2 As Object
Dim frame1 As HTMLFrameElement
Dim frame2 As HTMLFrameElement
Set IE = CreateObject("InternetExplorer.Application")
IE.Visible = True
URL = "D:\Tests\parent.html"
IE.Navigate URL
Do While IE.ReadyState = 4: DoEvents: Loop 'Do While
Do Until IE.ReadyState = 4: DoEvents: Loop 'Do Until
Set frame1 = IE.document.getElementByID("urpopupouter0")
Set frame2 = IE.document.getElementByID("urpopupinner0")
Set doc1 = frame1.contentWindow.document
Set doc2 = frame2.contentWindow.document
IE.document.getElementByID("btn1").Click
doc1.getElementByID("btn1").Click
doc2.getElementByID("btn1").Click
Set IE = Nothing
End Sub
I tested the above code on my side and it works fine with the multiple frames on a webpage.
Test result:
VBA code is clicking the button on each frame and on the parent web page which changes the forecolor of the button to red.
Further, you can try to modify the code as per your own requirements.

Return Address from Google term search to excel using VBA

I am familiar with StackOverflow but have just recently signed up. I am trying to search a Hotel on google and return the address in Excel using VBA. Below is a photo of what Information I am trying to return from Google. From my research, I was able to find a VBA that allowed me to return the Results stats.
Would it be possible to modify my code and return the box at the top of my google search?
I would really appreciate your help! Below is the VBA I am using to return search results.
Sample Image - Red Roof Inn & Address
Sub SearchGoogle()
Dim ie As Object
Dim form As Variant
Dim button As Variant
Dim LR As Integer
Dim var As String
Dim var1 As Object
LR = Cells(Rows.Count, 1).End(xlUp).Row
For x = 2 To LR
var = Cells(x, 1).Value
Set ie = CreateObject("internetexplorer.application")
ie.Visible = True
With ie
.Visible = True
.navigate "http://www.google.co.in"
While Not .readyState = READYSTATE_COMPLETE
Wend
End With
'Wait some to time for loading the page
While ie.Busy
DoEvents
Wend
Application.Wait (Now + TimeValue("0:00:02"))
ie.document.getElementById("lst-ib").Value = var
'Here we are clicking on search Button
Set form = ie.document.getElementsByTagName("form")
Application.Wait (Now + TimeValue("0:00:02"))
Set button = form(0).onsubmit
form(0).submit
'wait for page to load
While ie.Busy
DoEvents
Wend
Application.Wait (Now + TimeValue("0:00:02"))
Set var1 = ie.document.getElementById("resultStats")
Cells(x, 2).Value = var1.innerText
ie.Quit
Set ie = Nothing
Next x
End Sub
Right now your code loads the page and then loads the value of the resultStats element.
So the section of your code that you will need to alter is:
Set var1 = ie.document.getElementById("resultStats")
Cells(x, 2).Value = var1.innerText
The first step to your problem is to understand the DOM of the HTML page you are attempting to use, in this case Google. I would suggest using a browser to navigate the DOM as it would give you a good idea of what the whole page is doing.
If you are aiming to do this on a macro basis you will need a path through the DOM that will always take you where you want to go. I would suggest having two pages with different searches open so that you can check you hypothesis as you go.
For example the boxes that you refer to seem to be located in a class called kp-header from knowing this you can build out your path through the DOM to return the text value displayed on screen. Again you will need to do your own investigations to find the best stating point for your search as kp-header was just the first potently helpful result I could find.
Although please note that depending on the speed you are loading these webpages you may hit a limit from google as they discourage scraping. What would be a better option to avoid these limits and to avoid yourself having to investigate all of google's DOM would be to try and incorporate one of google's API's

VBA checkbox website interaction

I'm currently experiencing a problem with VBA. I'm trying to tick checkbox on an external website. Here is a html snippet that I'm working with:Html code from external website
I have redacted any confidential information from the snippet.
Here is my VBA code:
Set elements = objIE.document.getElementsByTagName("input")
For Each ele In elements
ele.toString
If ele.Value = "xxx" Then ele.Click
Next
So in this code the elementsvariable is an object and the loop variable ele is not being populated at all. I need to compare that the value in the check box contains value xxx. I'm not too experienced in VBA.
Any help would be appreciated
Thanks
I really don't know if that works. But in VBA the
.Value
property of a Checkbox is eighter true or false. The property of the text that is shown is:
.Caption
This code is looping properly:
Sub test()
Set objIE = CreateObject("InternetExplorer.Application")
Dim elements, ele
objIE.Visible = True
For Each ip In Sheets("Sheet1").Range("A2:A13").Value
objIE.Navigate ip
Do Until Not objIE.Busy And objIE.ReadyState = 4
DoEvents
Loop
Set elements = objIE.document.getElementsByTagName("input")
For Each ele In elements
ele.toString
If ele.Value = "xxx" Then ele.Click
Next
Next
End Sub

How can I grab URLs contained in webpages?

I'm trying to get URLs from within an external webpage using a macro. Here's my current code:
Sub GoToWebSite()
Dim IE As Object
Application.ScreenUpdating = False
Set IE = CreateObject("InternetExplorer.Application")
With IE
.Navigate "www.website.com/careers/"
.Visible = True
End With
Application.ScreenUpdating = True
Set IE = Nothing
End Sub
From here, I want to supply the macro with a particular URL, tell it to search for particular text within www.website.com/careers/, then tell it to grab the hyperlink corresponding to the text, and paste the hyperlink in a cell in a spreadsheet. So for example, search for "Sales" then paste the URL corresponding to "Sales" in a particular cell.
There's no way to select an element based on its innerText, so you'll need to iterate the anchor/links node list and check each to see if it's the one you're looking for.
For example:
Dim objLink
For Each objLink in IE.document.getElementsByTagName("a")
If StrComp(objLink.innerText, "sales", vbTextCompare) = 0 Then
' Found the link matching our text. Display its URL...
Debug.Print objLink.href
Exit For
End If
Next

VBA and Internet Explorer: fill in an input box

I'm new to making interactions between VBA and Internet Explorer, but I've read many things online and couldn't figure out the problem in my code. I just want to retrieve the 'Username' box on a website and add a value inside. So I retrieved all input boxes into a collection of HTML elements, but then that collection is empty:
Dim Collect As IHTMLElementCollection
With IE
.navigate "http:xxxxxxxxxx"
.Visible = True
End With
Do While IE.Busy
Loop
Set Collect = IE.document.getElementsByTagName("input")
MsgBox Collect.Length
End Sub
This will give a message box with "0". If I toggle a breakpoint before the end of the code and I "watch" the variable Collect, I can see there are 17 items inside, one of them being the username 'inputbox', with name 'tfUserName'. Can you help me please?
EDIT: I found that the problem comes from this code:
Do While IE.Busy
Loop
Which I replaced with this:
Do Until IE.readyState = READYSTATE_COMPLETE
DoEvents
Loop
And now everything works fine. Thank you for your responses.
Validate the collection against null instead to determine if it contains elements
If Not Collect Is Nothing Then
For Each htmlElement In Collect
If Not htmlElement.getAttribute("username") Is Nothing Then
htmlElement.setAttribute("value", "licou6")
Exit For
End If
Next
End If