Pause script till website fully loaded - Excel VBA - vba

I'm currently trying to create a sheet which will extract tracking information for parcels sent out. I've worked out the following code for the time being but encounter the following issues:
The code continues before the page fully loads, I suspect this may be because after the initial loading is complete, it runs a script and refreshes.
If mouse is not rolling over Internet Explorer, high probability of a human verification with images. I understand this may not be possible to avoid but is there any way I can pause the script while someone completes the verification?
Sub RoyalTrack()
Dim i As Long
Dim ie As Object
Dim t As String
Set ie = CreateObject("InternetExplorer.Application")
With ie
.Visible = True
.Navigate "https://www.royalmail.com/track-your-item#/tracking-results/SF511991733GB"
.Resizable = True
End With
While ie.ReadyState <> 4 Or ie.Busy: DoEvents: Wend
Dim full As Variant
Dim latest As Variant
full = ie.Document.getElementsByClassName("c-tracking-history")(0).innerText
latest = ie.Document.getElementsByClassName("tracking-history-item ng-scope")(0).innerText
MsgBox full
MsgBox latest
End Sub

Managed to figure it out. Added a 2 second wait after page loads to allow loading and an error handler to identify if the required property is available.
Sub RoyalTrack()
Dim i As Long
Dim ie As Object
Dim t As String
Dim trackingN As String
Dim count As Integer
count = 2
Do While Worksheets("Sheet1").Range("D" & count).Value <> ""
Set ie = CreateObject("InternetExplorer.Application")
trackingN = Worksheets("Sheet1").Range("D" & count).Value
With ie
.Visible = True
' Variable tracking SF-GB
.Navigate "https://www.royalmail.com/track-your-item#/tracking-results/" & trackingN
.resizable = True
End With
While ie.readyState <> 4 Or ie.Busy: DoEvents: Wend
Application.Wait (Now + TimeValue("0:00:02"))
Dim full As Variant
Dim latest As Variant
On Error Resume Next
latest = ie.document.getElementsByClassName("tracking-history-item ng-scope")(0).innerText
If Err Then
MsgBox "Prove your humanity if you can"
Err.Clear
End If
latest = ie.document.getElementsByClassName("tracking-history-item ng-scope")(0).innerText
Windows("Book1.xls").Activate
Sheets("Sheet1").Select
Range("E" & count).Value = latest
ie.Quit
Set ie = Nothing
count = count + 1
Loop
End Sub

Related

IE Automation - VBA Error - Runtime Error 70: Permission Denied

Friends, I'm very new to VBA and here I'm trying to scrape data from an internal website. The process goes like this: There are few serial numbers entered in the column A3 to End. The macro is supposed to navigate to the URL --> pick the serial number from excel --> Enter in the Search Field and Click Search. Once the result is populated on the page, it scrapes specific values and populates in the excel sheet.
The page opens up well, data is picked from excel and when the macro is reading the table cells it gives Runtime 70 Error. Below is my code for reference. Any help to fix is much appreciated.
Sub Type1_Data()
Dim ie As InternetExplorer
Dim html As MSHTML.HTMLDocument
Dim RowNumber, ColumnNumber As Long
RowNumber = 3
ColumnNumber = 0
Dim i As Long
Dim HTMLDoc As MSHTML.HTMLDocument
Dim Filt As MSHTML.IHTMLElement
Dim mtbl As MSHTML.IHTMLElement
Dim strempid As MSHTML.HTMLElementCollection
Dim strempid1 As MSHTML.HTMLElementCollection
Dim strempid2 As MSHTML.HTMLElementCollection
Dim strempid3 As MSHTML.HTMLElementCollection
Dim strempid4 As MSHTML.HTMLElementCollection
Dim strempid5 As MSHTML.HTMLElementCollection
Dim strempid6 As MSHTML.HTMLElementCollection
Set ie = New InternetExplorer
ie.Visible = False
ie.navigate ("URL")
Do While ie.READYSTATE = 4: DoEvents: Loop
Do Until ie.READYSTATE = 4: DoEvents: Loop
Set HTMLDoc = ie.document
xy:
If HTMLDoc.Title <> "Marketplace | Find a professional" Then
ie.Visible = True
GoTo xy
End If
ie.Visible = True
ThisWorkbook.Activate
Dim Ed As Integer
Ed = 3
While ThisWorkbook.Sheets("ProM Search").Cells(Ed, 1).Value <> 0
Ed = Ed + 1
Wend
Ed = Ed - 1
For i = 3 To Ed
Application.ScreenUpdating = True
Set UID = HTMLDoc.getElementById("navSelect")
Set Filt = HTMLDoc.getElementById("searchText")
Set mtbl = HTMLDoc.getElementsByTagName("Table")(23)
Application.Wait DateAdd("s", 1, Now)
HTMLDoc.getElementById("NLQTextArea").Value = ThisWorkbook.Sheets("ProM Search").Cells(i, 1).Value
HTMLDoc.getElementById("submitAction").Click
Set strempid = mtbl.getElementsByClassName("dojoxGridCell")(1)
Set strempid1 = mtbl.getElementsByClassName("dojoxGridCell")(2)
Set strempid2 = mtbl.getElementsByClassName("dojoxGridCell")(3)
Set strempid3 = mtbl.getElementsByClassName("dojoxGridCell")(7)
Set strempid4 = mtbl.getElementsByClassName("dojoxGridCell")(9)
Set strempid5 = mtbl.getElementsByClassName("dojoxGridCell")(11)
Set strempid6 = mtbl.getElementsByClassName("dojoxGridCell")(12)
ActiveCell.Offset(0, 1).Activate
ActiveCell.Value = strempid.innerText
ActiveCell.Offset(0, 1).Activate
ActiveCell.Value = strempid1.innerText
ActiveCell.Offset(0, 1).Activate
ActiveCell.Value = strempid2.innerText
ActiveCell.Offset(0, 1).Activate
ActiveCell.Value = strempid3.innerText
ActiveCell.Offset(0, 1).Activate
ActiveCell.Value = strempid4.innerText
ActiveCell.Offset(0, 1).Activate
ActiveCell.Value = strempid5.innerText
ActiveCell.Offset(0, 1).Activate
ActiveCell.Value = strempid6.innerText
ActiveCell.Offset(1, -7).Activate
DoEvents
If ActiveCell.Value = "" Then
MsgBox "Fetching Completed Successfully", vbExclamation, "ProM - Open Seat Search T2"
GoTo qt
End If
Next
qt:
ie.Quit
Set ie = Nothing
Set HTMLDoc = Nothing
End Sub
HTMLDoc.getElementById("submitAction").Click may cause a page refresh making any referenced elements become stale. Try working always of ie.document rather than setting into a variable when performing actions likely/known to cause page refresh/update. This is a common cause of a permission denied error.
Your If End If should probably be a loop with time out. You only need the one ie.visible = True. You rarely gain from setting Visible to false in my opinion. If you intend to hide it from a user you should do so from the start unless it interferes with functionality.
No point for Application.ScreenUpdating = True as it is never switched off within this sub. If this sub is called then you only need it once outside the loop. Your repeated ActiveCell.Offset(0, 1).Activate can just use a loop with Select Case and set the value direct without activate. Repeating the line Do While ie.READYSTATE = 4: DoEvents: Loop serves no purpose.
You could use Find method of range to determine the row in column 1 where 0 occurs rather than walking down the column. And tests to determine it is found and >=3.
This error means an attempt was made to write to a write-protected disk or to access a locked file. You could check if special permission is needed to access the worksheet. For detailed causes and solutions, you can refer to this doc.
Besides, you can also refer to my working sample about reading the cell value in VBA:
Sub LOADIE()
Set ieA = CreateObject("InternetExplorer.Application")
ieA.Visible = True
ieA.navigate "https://www.bing.com"
Do Until ieA.readyState = 4
DoEvents
Loop
Set doc = ieA.Document
Dim tempStr As String
tempStr = "sb_form_q"
doc.getElementById(tempStr).Value = ThisWorkbook.Sheets("SheetName").Range("E2").Value
End Sub

Excel VBA Error - Runtime Error 70: Permission Denied

I need help with a code that was written for me. The code is below. I works find perhaps 70% of the time, but every approx 20 rows it throws up
Runtime Error 70: Permission Denied
What would cause this, and is there any fix for it? Also, How can I change the code to skip any errors like this (or similar) and just move onto the next row.
Debugger throws up the line: If btn.ID = "gs_hdr_tsb" Then
The full script is below.
Dim ie As Object
Private Sub CommandButton1_Click()
Set ie = CreateObject("InternetExplorer.Application")
ie.Visible = True
ie.Navigate TextBox1.Text
Do
DoEvents
Loop Until ie.readystate = 4
Application.Wait (Now + TimeValue("00:00:02"))
End Sub
Private Sub CommandButton2_Click()
Dim i As Long
Dim prfl As String
Dim ws As Worksheet
Set ws = ActiveSheet
Dim INPTTG, btn As Object
Dim Profile_link As String
Dim H3tg As Object
Dim iedata As Object
If iedata Is Nothing Then
Set iedata = CreateObject("InternetExplorer.Application")
End If
iedata.Visible = True
For i = TextBox2.Text To TextBox3.Text
prfl = ws.Range("E" & i).Value
For Each INPTTG In ie.document.getelementsbytagname("input")
If INPTTG.classname = "gs_in_txt" Then
INPTTG.Value = ""
INPTTG.Value = prfl
Exit For
End If
Next INPTTG
'button
Application.Wait (Now + TimeValue("00:00:02"))
For Each btn In ie.document.getelementsbytagname("button")
If btn.ID = "gs_hdr_tsb" Then
btn.Click
Do
DoEvents
Loop Until ie.readystate = 4
Application.Wait (Now + TimeValue("00:00:02"))
End If
Next btn
'prfl
For Each H3tg In ie.document.getelementsbytagname("h3")
If H3tg.classname = "gsc_oai_name" And VBA.Trim(H3tg.innertext) = prfl Then
Call GetData(H3tg.getelementsbytagname("a").Item(0).href, iedata, i)
Exit For
End If
Next H3tg
TextBox2.Text = i
Next i
MsgBox "Completed"
iedata.Quit
Set iedata = Nothing
Application.DisplayAlerts = False
ThisWorkbook.Save
End Sub
Any help would be very appreciated.
Try this one.
Dim ie As Object
Private Sub CommandButton1_Click()
On error go to ErrMsg
Set ie = CreateObject("InternetExplorer.Application")
ie.Visible = True
ie.Navigate TextBox1.Text
Do
DoEvents
Loop Until ie.readystate = 4
Application.Wait (Now + TimeValue("00:00:02"))
ErrMsg:
End Sub
Private Sub CommandButton2_Click()
Dim i As Long
Dim prfl As String
Dim ws As Worksheet
Set ws = ActiveSheet
Dim INPTTG, btn As Object
Dim Profile_link As String
Dim H3tg As Object
Dim iedata As Object
On error go to msgErr
If iedata Is Nothing Then
Set iedata = CreateObject("InternetExplorer.Application")
End If
iedata.Visible = True
For i = TextBox2.Text To TextBox3.Text
prfl = ws.Range("E" & i).Value
For Each INPTTG In ie.document.getelementsbytagname("input")
If INPTTG.classname = "gs_in_txt" Then
INPTTG.Value = ""
INPTTG.Value = prfl
Exit For
End If
Next INPTTG
'button
Application.Wait (Now + TimeValue("00:00:02"))
For Each btn In ie.document.getelementsbytagname("button")
If btn.ID = "gs_hdr_tsb" Then
btn.Click
Do
DoEvents
Loop Until ie.readystate = 4
Application.Wait (Now + TimeValue("00:00:02"))
End If
Next btn
'prfl
For Each H3tg In ie.document.getelementsbytagname("h3")
If H3tg.classname = "gsc_oai_name" And VBA.Trim(H3tg.innertext) = prfl Then
Call GetData(H3tg.getelementsbytagname("a").Item(0).href, iedata, i)
Exit For
End If
Next H3tg
TextBox2.Text = i
Next i
MsgBox "Completed"
iedata.Quit
Set iedata = Nothing
Application.DisplayAlerts = False
ThisWorkbook.Save
MsgErr:
End Sub
There are several answers here on StackOverflow about this VB Error. Each answer or situation is unique in reality - although each existing answer states a different potential root cause (file permissions, folder permissions, name reuse, ranges, etc).
I would recommend narrowing down the root-cause by double clicking on the side of the stating function/code in order to mark a breakpoinnt (looks like a red dot) (Alternatively, you can right click on the line of the code - Select the Toggle and then Breakpoint).
Next, run your code, and it will stop in your breakpoint. You can then Step-Into/Over/Out your code and essentially find the line of code that is responsible for throwing your error code. (Step Into is F8, Step over is Shift+F8 ((Go To the Debug top menu to see more options)))
Once you identified the responsible line of code - you can start looking further.
In my case scenario, I was using a protected variable name "Date" (look into variable names). Once I renamed it to something else, the problem was fixed.
One possible cause of this error (as was my case) is an antivirus/antimalware program blocking certain libraries. Try temporarily disabling all antivirus/antimalware programs and test your script again. In my case, Malwarebytes
In MalWare bytes, you can disable this setting by going to:
Security > Advanced Settings > Application Hardening [tab]
On the row called "Disable Loading of VBScript Libraries", un-check the checkbox under "MS Office".

Pilot freight tracking macro working on one computer but not another

Recently our office upgraded to new laptops. Luckily in changing over from Windows 7 to 10 and from desktop to Laptops we did not experience any issues with running this macro other than needing to enable popups on Pilotdelivers.com. For a few weeks now we've run this macro on two different laptops and everything has gone smoothly.
Today, on one of the laptops the macro is no longer working correctly, but on the other it's fine. I checked to make sure popups were enabled and that both computers are running from the same excel worksheet on our network. I restarted the computer twice and ran the macro with no other applications open. Laptops are the same model, installed at the same time. Identical software installations and updates. The laptop that is having trouble is used by my less computer savvy co worker and the laptop that's working is mine. So it's possible that there's some setting he changed that he shouldn't have, but I'm not sure what to check.
It seems like it's skipping the portion of code that clicks a link to open a new tab.
What the macro is supposed to do:
copy the tracking number on the worksheet
open IE
insert the tracking number into the text box
click track
wait for the new page to load
click the tracking number link
wait for the new tab to load
close the first tab
check to see if the most recent update is DELIVERED
if yes it switches back to excel and inputs DELIVERED and the
delivery date, if not it looks at the most recent update and adds
that line to the worksheet.
What seems like the macro is doing:
copy the tracking number on the worksheet
open IE
insert the tracking number into the text box
click track
wait for the new page to load
seems to skip clicking on the tracking number link
wait for the current page to load (which is already loaded)
when looking for the latest update it grabs the tracking number
instead (since it's not the expected page)
check to see if it's marked DELIVERED
10.if yes it switches back to excel and inputs DELIVERED and the
delivery date, if not it looks at the most recent update and adds
that line to the worksheet
It seems like it skips some or all of this section of code:
Dim ieDOC As HTMLDocument
Set ieDOC = ie.document
Set htmlColl = ieDOC.getElementsByTagName("a")
For Each htmlInput In htmlColl
If htmlInput.ID = "clickElement" Then
htmlInput.Click
Exit For
End If
Next htmlInput
ie.Quit
Set shellWins = New ShellWindows
If shellWins.Count > 0 Then
Set ie2 = shellWins.Item(1)
End If
Full code below:
Sub PilotTracking()
Dim ProURL As String
Dim ie As Object
Dim ie2 As Object
Dim RowCount As Integer
Dim i As Integer
Dim html_Document As HTMLDocument
Dim htmlColl As MSHTML.IHTMLElementCollection
Dim htmlInput As MSHTML.HTMLInputElement
Dim shellWins As ShellWindows
Dim htmlColl2 As MSHTML.IHTMLElementCollection
Dim htmlInput2 As MSHTML.HTMLInputElement
Dim marker As Integer
RowCount = 0
ProURL = "http://www.pilotdelivers.com/"
Do While Not ActiveCell.Offset(RowCount, -5).Value = ""
Set ie = CreateObject("InternetExplorer.application")
With ie
.Visible = True
.navigate ProURL
Do Until Not ie.Busy And ie.readyState = 4: DoEvents: Loop
End With
Set Doc = ie.document 'works don't delete
Doc.getElementById("tbShipNum").innerHTML = ActiveCell.Offset(RowCount, -5).Value 'works don't delete
Doc.getElementById("btnTrack").Click 'works don't delete
Do Until Not ie.Busy And ie.readyState = 4: DoEvents: Loop
i = 0
Do While i < 4
WaitHalfSec
i = i + 1
Loop
Do Until Not ie.Busy And ie.readyState = 4: DoEvents: Loop
Dim ieDOC As HTMLDocument
Set ieDOC = ie.document
Set htmlColl = ieDOC.getElementsByTagName("a")
For Each htmlInput In htmlColl
If htmlInput.ID = "clickElement" Then
htmlInput.Click
Exit For
End If
Next htmlInput
ie.Quit
Set shellWins = New ShellWindows
If shellWins.Count > 0 Then
Set ie2 = shellWins.Item(1)
End If
i = 0
Do While i < 8
WaitHalfSec
i = i + 1
Loop
Do Until Not ie2.Busy And ie2.readyState = 4: DoEvents: Loop
Set htmlColl2 = ie2.document.getElementsByTagName("td")
For Each htmlInput2 In htmlColl2
If htmlInput2.className = "dxgv" Then
If ActiveCell.Offset(RowCount).Value = "" Then
ActiveCell.Offset(RowCount).Value = htmlInput2.innerText
Else
If ActiveCell.Offset(RowCount).Value <> "DELIVERED" Then
ActiveCell.Offset(RowCount, -2).Value = ""
Else
ActiveCell.Offset(RowCount, -2).Value = htmlInput2.innerText
End If
Exit For
End If
End If
Next htmlInput2
ie2.Quit
Set shellWins = Nothing
Set ie = Nothing
Set ie2 = Nothing
RowCount = RowCount + 1
Loop
Set shellWins = Nothing
Set ie = Nothing
Set ie2 = Nothing
End Sub
Sub WaitHalfSec()
Dim t As Single
t = Timer + 1 / 2
Do Until t < Timer: DoEvents: Loop
End Sub
Instead of this:
Set htmlColl = ieDOC.getElementsByTagName("a")
For Each htmlInput In htmlColl
If htmlInput.ID = "clickElement" Then
htmlInput.Click
Exit For
End If
Next htmlInput
You should be able to just do this:
ieDOC.getElementById("clickElement").Click
Id is supposed to be unique within a given page. I see you've used getElementById elsewhere, so was there some reason it's not used here?
I'd guess maybe the problem is this:
ie.Quit
so try commenting that out. Something to do perhaps with where new pages load (new window vs new tab?)
If you're having problems grabbing the correct IE document, then try something like:
Function GetIE(sLocation As String) As Object
Dim objShell As Object, objShellWindows As Object, o As Object
Dim sURL As String
Dim retVal As Object
Set retVal = Nothing
Set objShell = CreateObject("Shell.Application")
Set objShellWindows = objShell.Windows
For Each o In objShellWindows
sURL = ""
On Error Resume Next
'check the URL and if it's the one you want then
' assign it to the return value and exit the loop
sURL = o.document.Location
On Error GoTo 0
If sURL Like sLocation & "*" Then
Set retVal = o
Exit For
End If
Next o
Set GetIE = retVal
End Function
This function will return an IE window which matches the provided URL (i.e. the first on whose URL begins with the passed sLocation string)

Excel VBA Macro: Scraping data from site table that spans multiple pages

Thanks in advance for the help. I'm running Windows 8.1, I have the latest IE / Chrome browsers, and the latest Excel. I'm trying to write an Excel Macro that pulls data from StackOverflow (https://stackoverflow.com/tags). Specifically, I'm trying to pull the date (that the macro is run), the tag names, the # of tags, and the brief description of what the tag is. I have it working for the first page of the table, but not for the rest (there are 1132 pages at the moment). Right now, it overwrites the data everytime I run the macro, and I'm not sure how to make it look for the next empty cell before running.. Lastly, I'm trying to make it run automatically once per week.
I'd much appreciate any help here. Problems are:
Pulling data from the web table beyond the first page
Making it scrape data to the next empty row rather than overwriting
Making the Macro run automatically once per week
Code (so far) is below. Thanks!
Enum READYSTATE
READYSTATE_UNINITIALIZED = 0
READYSTATE_LOADING = 1
READYSTATE_LOADED = 2
READYSTATE_INTERACTIVE = 3
READYSTATE_COMPLETE = 4
End Enum
Sub ImportStackOverflowData()
'to refer to the running copy of Internet Explorer
Dim ie As InternetExplorer
'to refer to the HTML document returned
Dim html As HTMLDocument
'open Internet Explorer in memory, and go to website
Set ie = New InternetExplorer
ie.Visible = False
ie.navigate "http://stackoverflow.com/tags"
'Wait until IE is done loading page
Do While ie.READYSTATE <> READYSTATE_COMPLETE
Application.StatusBar = "Trying to go to StackOverflow ..."
DoEvents
Loop
'show text of HTML document returned
Set html = ie.document
'close down IE and reset status bar
Set ie = Nothing
Application.StatusBar = ""
'clear old data out and put titles in
'Cells.Clear
'put heading across the top of row 3
Range("A3").Value = "Date Pulled"
Range("B3").Value = "Keyword"
Range("C3").Value = "# Of Tags"
'Range("C3").Value = "Asked This Week"
Range("D3").Value = "Description"
Dim TagList As IHTMLElement
Dim Tags As IHTMLElementCollection
Dim Tag As IHTMLElement
Dim RowNumber As Long
Dim TagFields As IHTMLElementCollection
Dim TagField As IHTMLElement
Dim Keyword As String
Dim NumberOfTags As String
'Dim AskedThisWeek As String
Dim TagDescription As String
'Dim QuestionFieldLinks As IHTMLElementCollection
Dim TodaysDate As Date
Set TagList = html.getElementById("tags-browser")
Set Tags = html.getElementsByClassName("tag-cell")
RowNumber = 4
For Each Tag In Tags
'if this is the tag containing the details, process it
If Tag.className = "tag-cell" Then
'get a list of all of the parts of this question,
'and loop over them
Set TagFields = Tag.all
For Each TagField In TagFields
'if this is the keyword, store it
If TagField.className = "post-tag" Then
'store the text value
Keyword = TagField.innerText
Cells(RowNumber, 2).Value = TagField.innerText
End If
If TagField.className = "item-multiplier-count" Then
'store the integer for number of tags
NumberOfTags = TagField.innerText
'NumberOfTags = Replace(NumberOfTags, "x", "")
Cells(RowNumber, 3).Value = Trim(NumberOfTags)
End If
If TagField.className = "excerpt" Then
Description = TagField.innerText
Cells(RowNumber, 4).Value = TagField.innerText
End If
TodaysDate = Format(Now, "MM/dd/yy")
Cells(RowNumber, 1).Value = TodaysDate
Next TagField
'go on to next row of worksheet
RowNumber = RowNumber + 1
End If
Next
Set html = Nothing
'do some final formatting
Range("A3").CurrentRegion.WrapText = False
Range("A3").CurrentRegion.EntireColumn.AutoFit
Range("A1:C1").EntireColumn.HorizontalAlignment = xlCenter
Range("A1:D1").Merge
Range("A1").Value = "StackOverflow Tag Trends"
Range("A1").Font.Bold = True
Application.StatusBar = ""
MsgBox "Done!"
End Sub
There's no need to scrape Stack Overflow when they make the underlying data available to you through things like the Data Explorer. Using this query in the Data Explorer should get you the results you need:
select t.TagName, t.Count, p.Body
from Tags t inner join Posts p
on t.ExcerptPostId = p.Id
order by t.count desc;
The permalink to that query is here and the "Download CSV" option which appears after the query runs is probably the easiest way to get the data into Excel. If you wanted to automate that part of things, the direct link to the CSV download of results is here
You can improve this to parse out exact elements but it loops all the pages and grabs all the tag info (everything next to a tag)
Option Explicit
Public Sub ImportStackOverflowData()
Dim ie As New InternetExplorer, html As HTMLDocument
Application.ScreenUpdating = False
With ie
.Visible = True
.navigate "https://stackoverflow.com/tags"
While .Busy Or .READYSTATE < 4: DoEvents: Wend
Set html = .document
Dim numPages As Long, i As Long, info As Object, item As Object, counter As Long
numPages = html.querySelector(".page-numbers.dots ~ a").innerText
For i = 1 To 2 ' numPages ''<==1 to 2 for testing; use to numPages
DoEvents
Set info = html.getElementById("tags_list")
For Each item In info.getElementsByClassName("grid-layout--cell tag-cell")
counter = counter + 1
Cells(counter, 1) = item.innerText
Next item
html.querySelector(".page-numbers.next").Click
While .Busy Or .READYSTATE < 4: DoEvents: Wend
Set html = .document
Next i
Application.ScreenUpdating = True
.Quit '<== Remember to quit application
End With
End Sub
I'm not making use of the DOM, but I find it very easy to get around just searching between known tags. If ever the expressions you are looking for are too common just tweak the code a bit so that it looks for a string after a string).
An example:
Public Sub ZipLookUp()
Dim URL As String, xmlHTTP As Object, html As Object, htmlResponse As String
Dim SStr As String, EStr As String, EndS As Integer, StartS As Integer
Dim Zip4Digit As String
URL = "https://tools.usps.com/go/ZipLookupResultsAction!input.action?resultMode=1&companyName=&address1=1642+Harmon+Street&address2=&city=Berkeley&state=CA&urbanCode=&postalCode=&zip=94703"
Set xmlHTTP = CreateObject("MSXML2.XMLHTTP")
xmlHTTP.Open "GET", URL, False
On Error GoTo NoConnect
xmlHTTP.send
On Error GoTo 0
Set html = CreateObject("htmlfile")
htmlResponse = xmlHTTP.ResponseText
If htmlResponse = Null Then
MsgBox ("Aborted Run - HTML response was null")
Application.ScreenUpdating = True
GoTo End_Prog
End If
'Searching for a string within 2 strings
SStr = "<span class=""address1 range"">" ' first string
EStr = "</span><br />" ' second string
StartS = InStr(1, htmlResponse, SStr, vbTextCompare) + Len(SStr)
EndS = InStr(StartS, htmlResponse, EStr, vbTextCompare)
Zip4Digit = Left(Mid(htmlResponse, StartS, EndS - StartS), 4)
MsgBox Zip4Digit
GoTo End_Prog
NoConnect:
If Err = -2147467259 Or Err = -2146697211 Then MsgBox "Error - No Connection": GoTo End_Prog 'MsgBox Err & ": " & Error(Err)
End_Prog:
End Sub

VBA hanging on ie.busy and readystate check

I am trying to grab some football player data from a website to fill a privately used database. I've included the entire code below. This first section is a looper that calls the second function to fill a database. I've run this code in MSAccess to fill a database last summer and it worked great.
Now I am only getting a few teams to fill before the program gets hung up at
While IE.Busy Or IE.ReadyState <> READYSTATE_COMPLETE: DoEvents: Wend
I've searched countless websites regarding this error and tried changing this code by putting in sub function to wait a period of seconds or other work-arounds. None of those solve the issue. I've also tried running this on multiple computers.
The first computer made it through 3 teams (or three calls of the 2nd function). The second slower computer makes it through 5 teams. Both eventually hang. The 1st computer has Internet Explorer 10 and the second has IE8.
Sub Parse_NFL_RawSalaries()
Status ("Importing NFL Salary Information.")
Dim mydb As Database
Dim teamdata As DAO.Recordset
Dim i As Integer
Dim j As Double
Set mydb = CurrentDb()
Set teamdata = mydb.OpenRecordset("TEAM")
i = 1
With teamdata
Do Until .EOF
Call Parse_Team_RawSalaries(teamdata![RotoworldTeam])
.MoveNext
i = i + 1
j = i / 32
Status("Importing NFL Salary Information. " & Str(Round(j * 100, 0)) & "% done")
Loop
End With
teamdata.Close ' reset variables
Set teamdata = Nothing
Set mydb = Nothing
Status ("") 'resets the status bar
End Sub
Second function:
Function Parse_Team_RawSalaries(Team As String)
Dim mydb As Database
Dim rst As DAO.Recordset
Dim IE As InternetExplorer
Dim HTMLdoc As HTMLDocument
Dim TABLEelements As IHTMLElementCollection
Dim TRelements As IHTMLElementCollection
Dim TDelements As IHTMLElementCollection
Dim TABLEelement As Object
Dim TRelement As Object
Dim TDelement As HTMLTableCell
Dim c As Long
' open the table
Set mydb = CurrentDb()
Set rst = mydb.OpenRecordset("TempSalary")
Set IE = CreateObject("InternetExplorer.Application")
IE.Visible = False
IE.navigate "http://www.rotoworld.com/teams/contracts/nfl/" & Team
While IE.Busy Or IE.ReadyState <> READYSTATE_COMPLETE: DoEvents: Wend
Set HTMLdoc = IE.Document
Set TABLEelements = HTMLdoc.getElementsByTagName("Table")
For Each TABLEelement In TABLEelements
If TABLEelement.id = "cp1_tblContracts" Then
Set TRelements = TABLEelement.getElementsByTagName("TR")
For Each TRelement In TRelements
If TRelement.className <> "columnnames" Then
rst.AddNew
rst![Team] = Team
c = 0
Set TDelements = TRelement.getElementsByTagName("TD")
For Each TDelement In TDelements
Select Case c
Case 0
rst![Player] = Trim(TDelement.innerText)
Case 1
rst![position] = Trim(TDelement.innerText)
Case 2
rst![ContractTerms] = Trim(TDelement.innerText)
End Select
c = c + 1
Next TDelement
rst.Update
End If
Next TRelement
End If
Next TABLEelement
' reset variables
rst.Close
Set rst = Nothing
Set mydb = Nothing
IE.Quit
End Function
In Parse_Team_RawSalaries, instead of using the InternetExplorer.Application object, how about using MSXML2.XMLHTTP60?
So, instead of this:
Set IE = CreateObject("InternetExplorer.Application")
IE.Visible = False
IE.navigate "http://www.rotoworld.com/teams/contracts/nfl/" & Team
While IE.Busy Or IE.ReadyState <> READYSTATE_COMPLETE: DoEvents: Wend
Set HTMLdoc = IE.Document
Maybe try using this (add a reference to "Microsoft XML 6.0" in VBA Editor first):
Dim IE As MSXML2.XMLHTTP60
Set IE = New MSXML2.XMLHTTP60
IE.Open "GET", "http://www.rotoworld.com/teams/contracts/nfl/" & Team, False
IE.send
While IE.ReadyState <> 4
DoEvents
Wend
Dim HTMLDoc As MSHTML.HTMLDocument
Dim HTMLBody As MSHTML.htmlBody
Set HTMLDoc = New MSHTML.HTMLDocument
Set HTMLBody = HTMLDoc.body
HTMLBody.innerHTML = IE.responseText
I've generally found that MSXML2.XMLHTTP60 (and WinHttp.WinHttpRequest, for that matter) generally perform better (faster and more reliable) than InternetExplorer.Application.
I've found this post very helpful when I encountered similiar problem. Here is my solution:
I used
Dim browser As SHDocVw.InternetExplorer
Set browser = New SHDocVw.InternetExplorer
and
cTime = Now + TimeValue("00:01:00")
Do Until (browser.readyState = 4 And Not browser.Busy)
If Now < cTime Then
DoEvents
Else
browser.Quit
Set browser = Nothing
MsgBox "Error"
Exit Sub
End If
Loop
Sometimes page is loaded but code stops on DoEvents and goes on and on and on. Using this code it goes on only for 1 minute and if browser is not ready it quits the browser and exits sub.
I know this is a old post but. I have had the same problem with my code for downloading web site pictures using Excel VBA automation. Some sites wont let you download a image file using a link without first opening the link in a browser. However my code was getting hung up sometimes with when the objBrowser.visible was set to false with the folowing code
Do Until (objBrowser.busy = False And objBrowser.readyState = 4)
Application.Wait (Now + TimeValue("0:00:01"))
DoEvents 'browser.readyState = 4
Loop
the simple fix was to make the objBrowser.visible
I fixed it with
Dim Passes As Integer: Passes = 0
Do Until (objBrowser.busy = False And objBrowser.readyState = 4)
Passes = Passes + 1 'count loops
Application.Wait (Now + TimeValue("0:00:01"))
DoEvents
If Passes > 5 Then
'set size browser cannot set it smaller than 400
objBrowser.Width = 400 'set size
objBrowser.Height = 400
Label8.Caption = Passes 'display loop count
' position browser "you cannot move it off the screen" ready state wont change
objBrowser.Left = UserForm2.Left + UserForm2.Width
objBrowser.Top = UserForm2.Top + UserForm2.Height
objBrowser.Visible = True
DoEvents
objBrowser.Visible = False
End If
Loop
objBrowser only flashes for less than a second but it gets the job done!