I;m in the process of learning the web scraping using VBA and I've come across the website listed within my code which is Java build.
My goal is to click Next button from the VBA code, but I can not identify it when looking at "behind the scene" code. When using "Inspect" element i see the button reference but when I list all the links in my excel it is not there.
Then I looked at the page using CTR-U (in Chrome) and the page looks completely different and has a lot of Java Code. I am not familiar with Java therefore could you please describe how is the mechanism of decoding such a Java page?
Here is my code:
Sub first_template()
Dim ie As New InternetExplorer
Dim doc As HTMLDocument
ie.Top = 0
ie.Left = 0
ie.Visible = True
ie.navigate "http://www.amaaonline.com/member-directory/"
Do
DoEvents
Loop Until ie.readyState = READYSTATE_COMPLETE
Set input_elements = ie.document.getElementsByTagName("a")
i = 4
For Each element In input_elements
Cells(i, 1).Value = element.innerHTML
i = i + 1
Next element
ie.Quit
Set ie = Nothing
End Sub
Related
I have previously used selenium webdriver for sending messages to whatsapp via VBA. But now, I need some pure vba codes to send text from excel sheet. What I discovered InternetExplorer does not support whatsapweb. I searched the internet I discovered some tutorial where it uses APPACTIVATE method for using CHROME.
AppActivate "Chrome"
But I was not able to use chrome with this code. Can anyone help with code below/ suggesting links will also be benefital for sending messages through vba to whatsapp without third party applications.
Sub Automate_IE_Load_Page()
'This will load a webpage in IE
Dim i As Long
Dim URL As String
Dim IE As Object
Dim objElement As Object
Dim objCollection As Object
'Create InternetExplorer Object
Set IE = CreateObject("InternetExplorer.Application")
'Set IE.Visible = True to make IE visible, or False for IE to run in the background
IE.Visible = True
'Define URL
URL = "https://web.whatsapp.com/"
'Navigate to URL
IE.navigate URL
' Statusbar let's user know website is loading
Application.StatusBar = URL & " is loading. Please wait..."
' Wait while IE loading...
'IE ReadyState = 4 signifies the webpage has loaded (the first loop is set to avoid inadvertently skipping over the second loop)
Do While IE.readyState = 4: DoEvents: Loop 'Do While
Do Until IE.readyState = 4: DoEvents: Loop 'Do Until
'Webpage Loaded
Application.StatusBar = URL & " Loaded"
'Unload IE
Set IE = Nothing
Set objElement = Nothing
Set objCollection = Nothing
End Sub
On internet I saw excel formula where it hyperlinks the text message to whatsapp from formula bar, but HYPERLINK formula opens the edge browser not IE. Could not I use also Edge browser in vba library instead IE.Application?
=IF(G3="","",HYPERLINK("https://web.whatsapp.com/send?phone=%2B"&VLOOKUP(B3,'Employee Master'!A:D,4,0)&"&text=" & H3&"&app_absent=1&send=1",")"))
I've written a script in vba using IE to get the titles of different hotel names from a webpage. The hotel names traverse multiple pages through pagination.
My scraper can keep clicking on the next button successfully while parsing the titles from each page until ther is no more click left to perform. The parser is doing is job just perfect. All I wish to know is a simple logic I've asked below.
My question: How the content of each page is rightly coming through even when I didn't use this Set Htmldoc = IE.document line just after the .click? When a click is initiated, the scraper goes to a new page with new content. How come it gets updated with new content from each page as my defined do loop comes after with IE block?
This is the script:
Sub GetTitles()
Const Url As String = "https://www.tripadvisor.com/Hotels-g147237-Caribbean-Hotels.html"
Dim IE As New InternetExplorer, Htmldoc As HTMLDocument, post As Object, R&
With IE
.Visible = True
.navigate Url
While .Busy = True Or .readyState < 4: DoEvents: Wend
Set Htmldoc = .document
End With
Do
For Each post In Htmldoc.getElementsByClassName("listing") ''how this "Htmldoc" gets updated
With post.getElementsByClassName("property_title")
If .Length Then R = R + 1: Cells(R, 1) = .Item(0).innerText
End With
Next post
If Not Htmldoc.querySelector(".standard_pagination span[onclick*='pagination_next']") Is Nothing Then
Htmldoc.querySelector(".standard_pagination span[onclick*='pagination_next']").Click
Application.Wait Now + TimeValue("00:00:05")
''I didn't use anything like "Set Htmldoc = IE.document" but it still works flawlessly
Else:
Exit Do
End If
Loop
IE.Quit
End Sub
The script is not faulty. Though, you are using it without fully understanding is certainly troublesome.
When you do this Set Htmldoc = .document you are setting the IE's document for later use.
When you do this Htmldoc.querySelector(".standard_pagination span[onclick*='pagination_next']").Click javascript comes in play and updates the content of the page (i.e document).
You may believe that the document has changed but its only being updated. In reality,there is no navigation happening at all.
Add the following and see how the page/document remains the same, just the content changes.
'/ Url before Next button click
Debug.Print "Before Click " & Htmldoc.Url
Htmldoc.querySelector(".standard_pagination span[onclick*='pagination_next']").Click
'/ Url after Next button click
Debug.Print "After Click " & Htmldoc.Url
Since the document, once set remains the same and the updated content has same layout/DOM (that is how mostly programmers code, most likely all the pages are being rendered using a template) hence your code works perfectly fine. Net to net for your do loop, nothing changed.
Set Htmldoc = .document
gets a pointer to the DOM. When it changes the Htmldoc is pointing at the new content. No need to do a new Set Htmldoc
I've written a script in vba in combination with IE to click on some dots available on a map in a web page. When a dot is clicked, a small box containing relevant information pops up.
Link to that website
I would like to parse the content of each box. The content of that box can be found using class name contentPane. However, the main concern here is to generate each box by clicking on those dots. When a box shows up, it looks how you can see in the below image.
This is the script I've tried so far:
Sub HitDotOnAMap()
Const Url As String = "https://www.arcgis.com/apps/Embed/index.html?webmap=4712740e6d6747d18cffc6a5fa5988f8&extent=-141.1354,10.7295,-49.7292,57.6712&zoom=true&scale=true&search=true&searchextent=true&details=true&legend=true&active_panel=details&basemap_gallery=true&disable_scroll=true&theme=light"
Dim IE As New InternetExplorer, HTML As HTMLDocument
Dim post As Object, I&
With IE
.Visible = True
.navigate Url
While .Busy = True Or .readyState < 4: DoEvents: Wend
Set HTML = .document
End With
Application.Wait Now + TimeValue("00:0:07") ''the following line zooms in the slider
HTML.querySelector("#mapDiv_zoom_slider .esriSimpleSliderIncrementButton").Click
Application.Wait Now + TimeValue("00:0:04")
With HTML.querySelectorAll("[id^='NWQMC_VM_directory_'] circle")
For I = 0 To .Length - 1
.item(I).Focus
.item(I).Click
Application.Wait Now + TimeValue("00:0:03")
Set post = HTML.querySelector(".contentPane")
Debug.Print post.innerText
HTML.querySelector("[class$='close']").Click
Next I
End With
End Sub
when I execute the above script, it looks like it is running smoothly but nothing happens (I meant, no clicking) and it doesn't throw any error either. Finally it quits the browser gracefully.
This is how a box with information looks like when a dot gets clicked.
Although I've used hardcoded delay within my script, they can be fixed later as soon as the macro starts working.
Question: How can I click each of the dots on that map and collect the relevant information from the popped-up box? I only expect to have any solution using Internet Explorer
The data are not the main concern here. I would like to know how IE work in such cases so that I can deal with them in future cases. Any solution other than IE is not I'm looking for.
No need to click on each dots. Json file has all the details and you can extract as per your requirement.
Installation of JsonConverter
Download the latest release
Import JsonConverter.bas into your project (Open VBA Editor, Alt + F11; File > Import File)
Add Dictionary reference/class
For Windows-only, include a reference to "Microsoft Scripting Runtime"
For Windows and Mac, include VBA-Dictionary
References to be added
Download the sample file here.
Code:
Sub HitDotOnAMap()
Const Url As String = "https://www.arcgis.com/sharing/rest/content/items/4712740e6d6747d18cffc6a5fa5988f8/data?f=json"
Dim IE As New InternetExplorer, HTML As HTMLDocument
Dim post As Object, I&
Dim data As String, colObj As Object
With IE
.Visible = True
.navigate Url
While .Busy = True Or .readyState < 4: DoEvents: Wend
data = .document.body.innerHTML
data = Replace(Replace(data, "<pre>", ""), "</pre>", "")
End With
Dim JSON As Object
Set JSON = JsonConverter.ParseJson(data)
Set colObj = JSON("operationalLayers")(1)("featureCollection")("layers")(1)("featureSet")
For Each Item In colObj("features")
For j = 1 To Item("attributes").Count - 1
Debug.Print Item("attributes").Keys()(j), Item("attributes").Items()(j)
Next
Next
End Sub
Output
I'm currently trying to automate my IE at work with VBA and Excel, because we have some repetitive tasks to complete in an SAP System which is called within the Internet Explorer.
So far I am able to start IE and navigate to the starting page of our System.
I looked on the source code from the Sap System to search for an id from a div (lets say its 'SampleID'). But if I try to click it via VBA I only get returned 'nothing'. So I tried to print out the source code via VBA Debug.Print to look at the problem. And here's the thing:
The source code that gets printed is completely different from the Website's Source Code. In the printed code for example is a iframe that I can't find in the website source code. So that's the reason I can't find my div and any other objects, because they are not in the code that VBA is searching through.
Can anyone tell me what's going wrong? Why are the source codes completely different?
Sub stgGo(){
Dim i As Long
Dim IE As InternetExplorerMedium
Dim objElement As Object
Dim objCollection As Object
Set IE = New InternetExplorer
IE.Visible = True
IE.Navigate ("URL")
'waiting 10 seconds to be really sure javascript is ready
Application.Wait (Now + TimeValue("0:00:10"))
'printing out the false source code
Debug.Print strHTML = myIE.Document.body.innerHTML
Set objCollection = IE.document.getElementsByTagName("div")
i = 0
While i < objCollection.Length
If objCollection(i).ID = "SampleID" Then
objCollection(i).Click
End If
i = i + 1
Wend
End Sub
I have a VERY basic script I am using with an excel spreadsheet to populate a form. It inserts data into the fields then waits for me to click the submit button on each page. Then it waits for the page to load and fills in the next set of fields. I click submit, next page loads, etc. Finally, on the last two pages I have to do some manual input that I can't do with the spreadsheet. Code thus far is shown below.
My question is, at the end of what I have there now, how can I make the system wait for me to fill out that last page, then once I submit it realize that it has been submitted and loop back to the beginning, incrementing the row on the spreadsheet so that we can start over and do the whole thing again for the next student?
As you will probably be able to tell I am not a programmer, just a music teacher who does not savor the idea of filling out these forms manually for all 200 of my students and got the majority of the code you see from a tutorial.
Function FillInternetForm()
Dim IE As Object
Set IE = CreateObject("InternetExplorer.Application")
'create new instance of IE. use reference to return current open IE if
'you want to use open IE window. Easiest way I know of is via title bar.
IE.Navigate "https://account.makemusic.com/Account/Create/?ReturnUrl=/OpenId/VerifyGradebookRequest"
'go to web page listed inside quotes
IE.Visible = True
While IE.busy
DoEvents 'wait until IE is done loading page.
Wend
IE.Document.All("BirthMonth").Value = "1"
IE.Document.All("BirthYear").Value = "2000"
IE.Document.All("Email").Value = ThisWorkbook.Sheets("queryRNstudents").Range("f2")
IE.Document.All("Password").Value = ThisWorkbook.Sheets("queryRNstudents").Range("e2")
IE.Document.All("PasswordConfirm").Value = ThisWorkbook.Sheets("queryRNstudents").Range("e2")
IE.Document.All("Country").Value = "USA"
IE.Document.All("responseButtonsDiv").Click
newHour = Hour(Now())
newMinute = Minute(Now())
newSecond = Second(Now()) + 3
waitTime = TimeSerial(newHour, newMinute, newSecond)
Application.Wait waitTime
IE.Document.All("FirstName").Value = ThisWorkbook.Sheets("queryRNstudents").Range("a2")
IE.Document.All("LastName").Value = ThisWorkbook.Sheets("queryRNstudents").Range("b2")
IE.Document.All("Address1").Value = "123 Nowhere St"
IE.Document.All("City").Value = "Des Moines"
IE.Document.All("StateProvince").Value = "IA"
IE.Document.All("ZipPostalCode").Value = "50318"
End Function
I would use the events of the IE, more specifically the form, something like this, using MSHTML Controls library.
Private WithEvents IEForm As MSHTML.HTMLFormElement
Public Sub InternetExplorerTest()
Dim ie As SHDocVw.InternetExplorer
Dim doc As MSHTML.HTMLDocument
Set ie = New SHDocVw.InternetExplorer
ie.Visible = 1
ie.navigate "http://stackoverflow.com/questions/tagged/vba"
While ie.readyState <> READYSTATE_COMPLETE Or ie.Busy
DoEvents
Wend
Set doc = ie.document
Set IEForm = doc.forms(0)
End Sub
Private Function IEForm_onsubmit() As Boolean
MsgBox "Form Submitted"
End Function