Excel VBA Web Scraping - Object Required Error - vba

I have a small piece of code which tried to search in a website and tells whether there are any results or not.
It works very good for me (#India) but not for my colleague (#US). He checked in 2 of his PCs but gets same error.
After I click on search button, ie.Document becomes totally blank, also ie.Document.ChildNodes.Length returns 0.
Sub iTest()
Call ProcessRecord("First_Name", "Middle_Name", "Last_Name", "1/1/2015")
End Sub
Private Function ProcessRecord(fName As String, mName As String, lName As String, dob As String) As Boolean
Dim results As String, idx%
Dim ie As New InternetExplorer
ie.Visible = True
Navigate ie, "http://ws.ocsd.org/ArrestWarrants/default.aspx"
Do While ie.Busy Or ie.ReadyState <> READYSTATE_COMPLETE
Application.Wait Now + TimeSerial(0, 0, 1)
Loop
Do While ie.Busy Or ie.Document.ReadyState <> "complete"
Application.Wait Now + TimeSerial(0, 0, 1)
Loop
' fill values in webpage
ie.Document.getElementById("FirstName").Value = fName
ie.Document.getElementById("MiddleName").Value = mName
ie.Document.getElementById("LastName").Value = lName
ie.Document.getElementById("DOB").Value = dob
' click on search button
ie.Document.getElementById("btnSearch").Click
' wait for results
Do While ie.Busy Or ie.ReadyState <> READYSTATE_COMPLETE
Application.Wait Now + TimeSerial(0, 0, 1)
Loop
Do While ie.Busy Or ie.Document.ReadyState <> "complete"
Application.Wait Now + TimeSerial(0, 0, 1)
Loop
' check results
results = ie.Document.getElementById("lblResults").innerText '<< It gives Object Required error here. Because ie.Document has no element after I click on search button.
If results = "No Results Found." Then
MsgBox "Not found", vbExclamation
Else
MsgBox "Found", vbExclamation
End If
ie.Quit
End Function

Have you tried the below?
Do While ie.Busy Or ie.ReadyState <> READYSTATE_COMPLETE
DoEvents
Loop
Appears to be working for me, might just be a delay in internet connection for your US friend.

Related

Query Web Table on Current Website

I am running into a bit of a problem. Normally when I pull a table I use the "data from web" tool in excel, however I now have quite a few places I need to pull data that first require me to enter a username and password. I figured out some code for that (though probably not the most elegant) but realized that once I get to my desired page I have no idea how to extract the table. Here is what I have so far.
Sub Login()
Sheets("IOL").Select
Set ie = CreateObject("InternetExplorer.application")
ie.Visible = True
ie.Navigate ("https://internalsite.company.com/secure/login" & ActiveCell)
Do
If ie.ReadyState = 4 Then
ie.Visible = True
Exit Do
Else
DoEvents
End If
Loop
ie.Document.forms(0).all("badgeBarcodeId").Value = "00000"
ie.Document.forms(0).submit
'used because it redirects to a new page after submitting and I couldn't figure out how to make it wait for the new page to load before proceeding.
Application.Wait (Now + TimeValue("0:00:02"))
ie.Document.forms(0).all("password").Value = "00000"
ie.Document.forms(0).submit
End Sub
After the login is accomplished I would like to go to http://internalsite.company.com/csv and import the csv directly into a sheet. Anytime I make a new connection it makes me log in again so I figure there has to be a way to extract the file without adding a new connection. I'm pretty new with more complex VBA so bear with me.
I was able to get this code to do the job, but it is more preferable to get the CSV directly instead of the table. Sometimes the table doesn't like to load.
Sub Login()
Dim clip As DataObject
Dim ieTable As Object
Set ie = CreateObject("InternetExplorer.application")
ie.Visible = True
ie.Navigate ("https://internalsite1.company.com/secure/login" & ActiveCell)
Do
If ie.ReadyState = 4 Then
ie.Visible = True
Exit Do
Else
DoEvents
End If
Loop
ie.Document.forms(0).all("badgeBarcodeId").Value = "00000"
ie.Document.forms(0).submit
Do While ie.Busy: DoEvents: Loop
Do Until ie.ReadyState = 4: DoEvents: Loop
ie.Document.forms(0).all("password").Value = "000000"
ie.Document.forms(0).submit
Do While ie.Busy: DoEvents: Loop
Do Until ie.ReadyState = 4: DoEvents: Loop
ie.Navigate "http://internalsite2.company.com/site/Inbound?filter=1To3Days"
Do While ie.Busy: DoEvents: Loop
Do Until ie.ReadyState = 4: DoEvents: Loop
Set ieTable = ie.Document.all.Item("DataTables_Table_0")
If Not ieTable Is Nothing Then
Set clip = New DataObject
clip.SetText "" & ieTable.outerHTML & ""
clip.PutInClipboard
Workbooks("Production Meeting Dashboard.xlsm").Activate
Sheets("IOL").Select
Range("A1").Select
ActiveSheet.PasteSpecial Format:="Unicode Text", link:=False, _
DisplayAsIcon:=False, NoHTMLFormatting:=True
End If
End Sub

Log-in to Website

I'm trying to create an Excel macro that will log me into this website. I've written macros to log me into to other websites, but this one is somehow different.
When I use the following code to programmatically insert the username and password and then programmatically click the "log-in" button, the webpage comes back with the message
Error: please provide a username and Error: Please provide a
password
Sub LogIn()
' Open IE and navigate to the log-in page
Set ie = CreateObject("InternetExplorer.Application")
With ie
.Visible = True
.navigate "https://www.prudential.com/login"
' Loop until the page is fully loaded
Do Until Not ie.Busy And ie.ReadyState = 4
DoEvents
Loop
Application.Wait (Now + TimeValue("0:00:10"))
' Enter the necessary information on the Login web page and click the submit button
ie.document.getElementById("username").Value = "abcdef"
ie.document.getElementById("password").Value = "12345678"
ie.document.getElementsByClassName("btn btn-primary btn-login btn-sm-block analytics-login")(0).Click
' Loop until the page is fully loaded
Do Until Not ie.Busy And ie.ReadyState = 4
DoEvents
Loop
End With
' Do stuff
' Quit IE
ie.Quit
End Sub
I can see that my code has inserted the username and password into the webpage boxes, but for some reason after successfully clicking the submit button, the username and password are not being detected.
What code will successfully insert the username and password so that it can be processed by the web page? You'll know that your code works when you get the following error message:
We are unable to verify your username and password. Please try again.
Thanks for your help!
It actually wants you to send a key, so you can bypass this request like this:
ie.document.getElementById("username").Focus
ie.document.getElementById("username").Value = "abcde"
Application.SendKeys "f", True
ie.document.getElementById("password").Focus
ie.document.getElementById("password").Value = "1234567"
Application.SendKeys "8", True
ie.document.getElementsByClassName("btn btn-primary btn-login btn-sm-block analytics-login")(0).Click
Edit:
Sub GetData()
Dim ie As InternetExplorer
Dim desc As IHTMLElement
Set ie = New InternetExplorer
With ie
.navigate "https://www.prudential.com/login"
.Visible = True
End With
Do While (ie.Busy Or ie.ReadyState <> READYSTATE_COMPLETE)
DoEvents
Loop
ie.document.getElementById("username").Focus
Application.SendKeys "abcdef"
Application.Wait (Now + TimeValue("0:00:01"))
ie.document.getElementById("password").Focus
Application.SendKeys "12345678"
Application.Wait (Now + TimeValue("0:00:01"))
ie.document.getElementsByClassName("btn btn-primary btn-login btn-sm-block analytics-login")(0).Click
Set ie = Nothing
End Sub
Try the code below:
Option Explicit
Const READYSTATE_COMPLETE As Long = 4
Const URL_String As String = "https://www.prudential.com/login"
Sub Login()
Dim IE As Object
Dim ObjElement As Object
' using Late Binding
Set IE = CreateObject("InternetExplorer.Application")
With IE
.navigate URL_String
.Visible = True
End With
Do While (IE.Busy Or IE.ReadyState <> READYSTATE_COMPLETE)
DoEvents
Loop
IE.document.getElementByid("username").Value = "abcdef"
IE.document.getElementByid("password").Value = "12345678"
' Optional: I didn't need the wait time
Application.Wait (Now + TimeValue("0:00:01"))
Set ObjElement = IE.document.getElementsByClassName("btn btn-primary btn-login btn-sm-block analytics-login")
ObjElement(0).Click
Set IE = Nothing
End Sub
Another approach might be something like the following.
Sub Get_Logged_In()
Dim IE As New InternetExplorer, HTML As HTMLDocument
Dim post As Object, elem As Object
With IE
.Visible = True
.navigate "https://www.prudential.com/login"
While .Busy = True Or .readyState < 4: DoEvents: Wend
Set HTML = .document
End With
With HTML
Set post = .querySelector("#username")
post.Focus
post.innerText = "abcdef"
Set elem = .querySelector("#password")
elem.Focus
elem.innerText = "12345678"
.querySelector("button[data-qa='Login Button']").Click
End With
IE.Quit
End Sub
Reference to add to the library:
1. Microsoft Internet Controls
2. Microsoft HTML Object Library
In case anyone else has a similar problem I was finally able to get Sendkeys to work. Still, if anyone can come up with a non-Sendkeys solution it would be much preferred.
Two important points in getting Sendkeys to work are 1) use of
Set WSS = CreateObject("WScript.Shell")
and 2) to be sure to run the macro from Excel (e.g. tools-macro-macros) and not from the VB Editor (running from the editor will simply insert the text somewhere within the code and not on the target webpage).
' Open IE and navigate to the log-in page
Application.StatusBar = "Opening IE and logging in to the website"
Set WSS = CreateObject("WScript.Shell")
my_username = "abcdef"
my_pw = "12345678"
Set ie = CreateObject("InternetExplorer.Application")
With ie
.Visible = True
.navigate "https://www.prudential.com/login"
.Top = 0
.Left = 0
.Height = 1025
.Width = 1925
' Loop until the page is fully loaded
Do Until Not ie.Busy And ie.ReadyState = 4
DoEvents
Loop
Application.Wait (Now + TimeValue("0:00:10"))
WSS.SendKeys "{TAB}{TAB}{TAB}{TAB}{TAB}{TAB}{TAB}{TAB}{TAB}{TAB}{TAB}{TAB}{TAB}{TAB}{TAB}{TAB}{TAB}{TAB}{TAB}{TAB}{TAB}{TAB}{TAB}{TAB}{TAB}{TAB}{TAB}{TAB}{TAB}{TAB}{TAB}"
Application.Wait (Now + TimeValue("0:00:01"))
WSS.SendKeys my_username
Application.Wait (Now + TimeValue("0:00:01"))
WSS.SendKeys "{TAB}"
Application.Wait (Now + TimeValue("0:00:01"))
WSS.SendKeys my_pw
Application.Wait (Now + TimeValue("0:00:01")
ie.document.getElementsByClassName("btn btn-primary btn-login btn-sm-block analytics-login")(0).Click
' Loop until the page is fully loaded
Do Until Not ie.Busy And ie.ReadyState = 4
DoEvents
Loop
End With
' You're now logged in, do your stuff..

Excel VBA multiple For loop not working

I seem to be stuck in an weird issue. I have the Excel VBA code below to visit a website and enter a userID (from the userID column A in sheet 1) and retrieve the name of the user which shows up after hitting the submit button then continues with the rest of the userIDs.
Public Sub TEST()
TestPage = "http://test.com/"
Dim IE As New InternetExplorer
Dim Doc As HTMLDocument
Dim GetElem As Object
Dim GetElem2 As Object
IE.Visible = True
IE.navigate TestPage
Do
DoEvents
Loop Until IE.readyState = READYSTATE_COMPLETE
Application.Wait (Now + TimeValue("0:00:04"))
Set Doc = IE.document
CurRow = Sheet1.UsedRange.Rows.Count
Do While CurRow > 0
Application.Wait (Now + TimeValue("0:00:4"))
'Find/Get the userID textbox and enter the current userID
For Each GetElem In Doc.getElementsByTagName("input")
If GetElem.ID = "query" Then 'I could just do getElementByID later
GetElem.Value = Sheet1.Range("A" & CurRow).Value
End If
Next
'Find and click the submit button
For Each GetElem2 In Doc.getElementsByTagName("button")
If GetElem2.Type = "submit" Then
GetElem2.Click
Application.Wait (Now + TimeValue("0:00:03"))
End If
Next
CurRow = CurRow - 1
Loop
End Sub
The problem is the code works only once. It enters the first userID into the text box and hits Submit. When it loops and tries to enter the next userID though, the code get stuck in a loop.
If I remove the entire 2nd For-Next loop it works (although it doesn't submit, it enters each of the userIDs in the text box).
On top of that, if I use the F8 debugging the code step by step, everything works fine. Only getting problems when fully running the code.:(
Public Sub TEST()
TestPage = "http://test.com/"
Dim IE As New InternetExplorer
Dim Doc As HTMLDocument
Dim GetElem As Object
Dim GetElem2 As Object
IE.Visible = True
IE.navigate TestPage
Do
DoEvents
Loop Until IE.readyState = READYSTATE_COMPLETE
Application.Wait (Now + TimeValue("0:00:04"))
Set Doc = IE.document
CurRow = Sheet1.UsedRange.Rows.Count
For Each GetElem In Doc.getElementsByTagName("input")
If GetElem.ID = "query" Then 'I could just do getElementByID later
GetElem.Value = Sheet1.Range("A" & CurRow).Value
For Each GetElem2 In Doc.getElementsByTagName("button")
If GetElem2.Type = "submit" Then
GetElem2.Click
Application.Wait (Now + TimeValue("0:00:03"))
End If
Next GetElem2
End If
CurRow = CurRow + 1
Next GetElem
End Sub

VBA Internet Explorer Application gives different results for each function call

I'm trying to automate a task in excel that requires opening a webpage, navigating to a link on that page, and then clicking on a button on the second page to download an .xlsx file.
I've written a script that should do this. However, the response I get from the webpage is not always the same. In particular, sometimes this will return a download from the first page and sometimes it will navigate to the second page and not download anything, once or twice it has done both.
My sense is that this has to do with how long it takes for InternetExplorer.application to complete a request. I can't figure out how to troubleshoot this though, given that I tell the script to wait for IE.application to complete its request.
Sub DoBrowse2()
'For Each lnk In Sheets("Sheet4").Hyperlinks
'Range(lnk).Hy.Follow
'Next
Dim i As Long
Dim URL As String
Dim BaseURL As String
Dim ToURL As String
Dim IE As Object
Dim objElement As Object
Dim objCollection As Object
Dim HWNDSrc As Long
Dim html As IHTMLDocument
Set IE = CreateObject("InternetExplorer.Application")
URL = Range("B2").Hyperlinks(1).Address
IE.Navigate URL
IE.Visible = True
Application.StatusBar = URL & " is loading. Please wait..."
Do While IE.ReadyState = 4: DoEvents: Loop
Do Until IE.ReadyState = 4: DoEvents: Loop
Application.StatusBar = URL & " Loaded"
'Set html = IE.Document
'Dim elements As IHTMLElementCollection
'Set elements = html.all
For Each itm In IE.Document.all
If itm.className = "datagrid" Then
For Each el In itm.Document.all
Debug.Print "hello"
If el.className = "ujump" And Right(el.innerText, 12) = "Constituents" Then
'Debug.Print el.innerText
ToURL = el.getAttribute("data-subset")
BaseURL = "http://datastream.thomsonreuters.com/navigator/search.aspx?dsid=ZUCH002&AppGroup=DSAddin&host=Metadata&prev=scmTELCMBR&s=D&subset="
ToURL = BaseURL & ToURL
'Debug.Print ToURL
IE.Navigate ToURL
IE.Visible = True
Do While IE.Busy
Debug.Print "in busy loop"
Application.Wait DateAdd("s", 1, Now)
Loop
GoTo end_of_for
End If
Next
End If
Next
end_of_for:
Debug.Print ("STOP STOP STOP STOP STOP")
Dim Script As String
For Each itm In IE.Document.all
If itm.className = "lgc excel" Then
Debug.Print "hello world"
Debug.Print itm.getAttribute("onclick")
itm.Click
Do While IE.Busy
Debug.Print "app busy"
Application.Wait DateAdd("s", 1, Now)
Loop
Exit For
End If
Next
End Sub
Thanks in advance for your help.
Use this to determine whether IE page has been fully loaded, it always must be both of these conditions:
Do Until ie.ReadyState = 4 And ie.Busy = False
DoEvents
Loop
Even with code above if there are scripts on the page, some content may be loaded after ie.ReadyState = 4 And ie.Busy = False condition is met and either easy way, but inefficient and unreliable Application.Wait can be used or you can try finding elements on the website which inform about loading state and determine the state by their visible attributes etc.
Part of your code is wrong and causes an endless loop:
Do While IE.ReadyState = 4: DoEvents: Loop
Do Until IE.ReadyState = 4: DoEvents: Loop
It makes DoEvents fire while readystate is complete and also until it reaches complete status.
Narrow down a collection of all elements:
For Each itm In IE.Document.all
to a specific collection for better performance when possible, for example:
For Each itm In IE.Document.GetElementsByTagName("div")

Reference Cell in VBA Script

I was curious if there is a way to reference a specific cell within the following VBA script. The cell contains a date in yyyymmdd format, which would go in the commented section below.
Sub OpenData()
Set IE = CreateObject("InternetExplorer.Application")
IE.Visible = False
IE.navigate "http://website.com/'desired cell value'/subdirectory/file.txt"
'Check for good connection to web page loop!
Do
If IE.readyState = 4 Then
IE.Visible = True
Exit Do
Else
DoEvents
End If
Loop
'Wait for window to open!
Application.wait (Now + TimeValue("0:00:02"))
'MsgBox "Done"
IE.Visible = True
Eric
Have you tried format?
'...
IE.navigate "http://website.com/" & Format (Cells(1,1).Value, “dd/MM/yyyy”) & "/subdirectory/file.txt" '...