I'm pretty new with VBA, but I've been muddling through to make a program for my team.
This piece of code works most of the time, but tends to hang on occasion. I can't figure out why it would hang sometimes, and work perfectly most of the time, so I'm now trying to figure out a better way to accomplish this loop. I know that this method of looping isn't the best way to do things, but not sure how to accomplish the task.
My webpage operates in a PEGA Web application, and the native IE ready state indicators are always 'ready' so I have to use the web application's ready state markers.
Can anyone help me out?
Public Sub WaitingForRS()
' FIND THE C360 WINDOW
Marker = 0
Set objShell = CreateObject("Shell.Application")
IE_count = objShell.Windows.Count
For x = 0 To (IE_count - 1)
On Error Resume Next
my_url = objShell.Windows(x).Document.Location
my_title = objShell.Windows(x).Document.title
If my_title Like "Coverage User" & "*" Then
Set C360Window = objShell.Windows(x)
Marker = 1
Exit For
Else
End If
Next
If Marker = 0 Then
MsgBox ("C360 window is not found. Please ensure C360 is open in Internet Explorer and try again")
Else
End If
'FIND THE READY STATE INDICATOR
RSIndicatorDocMarker = 0
RSIndicatorDataMarker = 0
Set RSIndicatorPage = C360Window.Document
Set RSIndicatorClass = RSIndicatorPage.getelementsbyclassname("document-statetracker")(0)
RSIndicatorCheck:
'CHECK THE READY STATE DOC STATUS
If RSIndicatorClass.getattribute("data-state-doc-status") = "ready" Then
RSIndicatorDocMarker = 1
Else: RSIndicatorDocMarker = 0
End If
'CHECK THE READY STATE
If RSIndicatorClass.getattribute("data-state-busy-status") = "none" Then
RSIndicatorDataMarker = 1
Else: RSIndicatorDataMarker = 0
End If
'Compare the RSIndicators
If RSIndicatorDataMarker = 1 And RSIndicatorDocMarker = 1 Then
Else: GoTo RSIndicatorCheck
End If
End Sub
Maybe try using OnTime instead of the tight loop you currently have:
Public Sub WaitingForRS()
Dim win As Object
Dim w As Object, el, ready As Boolean, idle As Boolean
For Each w In CreateObject("Shell.Application").Windows
If w.Name Like "*Internet*" Then
If w.Title Like "Coverage user*" Then
Set win = w
Exit For
End If
End If
Next
If Not win Is Nothing Then
Set el = win.document.getelementsbyclassname("document-statetracker")(0)
ready = (el.getattribute("data-state-doc-status") = "ready")
idle = (el.getattribute("data-state-busy-status") = "none")
If ready And idle Then
ProceedWithNextSteps win 'do whatever comes next: pass in the window
Else
'wait for a while then try again
Application.OnTime Now + TimeSerial(0, 0, 1), "WaitingForRS"
End If
Else
MsgBox "Window not found!"
End If
End Sub
Might want to add a time limit so it doesn't keep looping forever if the page isn't "ready".
Check out my answer on this post Excel VBA Submitting data via IE on an online MS Forms not working
# Idea 5 subheading. The WaitForLoadSETx() function has a manual time over-ride and double loop to help catch the state.
This was the best I could do in VBA with IE object. Ideally you want to learn Selenium, PhantomJS, or Puppeteer for actual browser manipulation. There are of course cool apps, folks have built on top of these libraries to help (puppeteer recorder chrome add in)
Or to keep things in Excel, use a simple XMLHTTPRequest object to cut the browser out of the equation, and deal solely with the request / response from the web page. This is a good alternative because it lets you focus on scraping the html content, in text form without the Javascript or waiting for page to load.
Related
I have a VB.NET 4.6.1 desktop app that has been using FindWindow and FindWindowEx for over 2 years with no issue to locate a MDI child window and capture the window caption text, it has worked flawlessly until recent.
The behavior now is my app can only successfully obtain the MDI client window handle if I go back to either the parent window or MDI client and click anywhere on either window, then return to my app and the process succeeds.
I have tried adding threading sleep events, running the action continuously in a loop multiple times, calling AppActivate method using process ID (thinking I just needed to execute again), my next workaround thought is to try and send a click event to the parent window prior to my action being executed or maybe to use Enumerate all child windows of the parent, hope someone can suggest something because I am at a roadblock, been doing this for years but this one doesn't make sense to me, I have the suspicion that it is related to recent ownership of the software company and them revising this section, but I have no idea why it would interfere with these root level API methods.
Sample Code:
MDIhWnd = FindWindowEx(ParenthWnd, IntPtr.Zero, "WindowsForms10.MDICLIENT.app.0.34f5582_r7_ad1", Nothing)
'Threading.Thread.Sleep(100)
'AppActivate(proc(0).Id)
If MDIhWnd = 0 Then
Threading.Thread.Sleep(100)
'Dim hw = GetTopWindow(ParenthWnd)
For i = 0 To 500
AppActivate(proc(0).Id)
MDIhWnd = FindWindowEx(ParenthWnd, IntPtr.Zero, "WindowsForms10.MDICLIENT.app.0.34f5582_r7_ad1", Nothing)
If MDIhWnd <> 0 Then
Exit For
End If
Next
End If
The solution for me was, based on the above suggestion, to use UI Automation, I
had never worked with it before, however after looking it over I gave a go and
found that it did indeed simplify my needs to capture window text from a 3rd party application window with MDI Client Interface.
Below is a lessor version in VB.NET of the process for anyone needing to do the
same thing:
Imports System.Windows.Automation
' You will also need references to UIAutomationClient, and UIAutomationTypes
Private Sub test_ui_automation()
Dim ParenthWnd As Integer = 0
Dim _AutomationElementA As System.Windows.Automation.AutomationElement = Nothing
Dim _AutomationElementB As System.Windows.Automation.AutomationElement = Nothing
Dim _AutomationElementC As System.Windows.Automation.AutomationElement = Nothing
Dim propCondition As Condition
Try
'Parent Windows Process Stuff
ParenthWnd = FindWindow(Nothing, "Application to Find")
_AutomationElementA = AutomationElement.FromHandle(ParenthWnd)
If _AutomationElementA Is Nothing Then
NotifyIcon1.BalloonTipIcon = ToolTipIcon.Error
NotifyIcon1.BalloonTipText = "Couldn't Locate Parent Window."
NotifyIcon1.Visible = True
NotifyIcon1.ShowBalloonTip(3000)
Exit Sub
End If
' MDI Client Stuff
' I used ClassNameProperty but other conditions are available
propCondition = New PropertyCondition(AutomationElement.ClassNameProperty, "WindowsForms10.MDICLIENT.app.0.34f5582_r7_ad1", PropertyConditionFlags.IgnoreCase)
_AutomationElementB = _AutomationElementA.FindFirst(TreeScope.Element Or TreeScope.Children, propCondition)
If _AutomationElementB Is Nothing Then
NotifyIcon1.BalloonTipIcon = ToolTipIcon.Warning
NotifyIcon1.BalloonTipText = "Application warning MDIClient not Available!"
NotifyIcon1.Visible = True
NotifyIcon1.ShowBalloonTip(3000)
Exit Sub
End If
' Final Stage Stuff Locate Window Containing Class with Caption
propCondition = New PropertyCondition(AutomationElement.ClassNameProperty, "WindowsForms10.Window.8.app.0.34f5582_r7_ad1", PropertyConditionFlags.IgnoreCase)
_AutomationElementC = _AutomationElementB.FindFirst(TreeScope.Element Or TreeScope.Children, propCondition)
If _AutomationElementC Is Nothing Then
NotifyIcon1.BalloonTipIcon = ToolTipIcon.Warning
NotifyIcon1.BalloonTipText = "Automation warning, MDI Details are open."
NotifyIcon1.Visible = True
NotifyIcon1.ShowBalloonTip(3000)
Exit Sub
End If
Caption = _AutomationElementC.Current.Name
' If needed you can now parse/strip any data needed from the Caption text.
' I had other processes here but could not include in the post.
Catch ex As Exception
MsgBox(ex.Message)
End Try
End Sub
Task:
So my first foray into Selenium and I am attempting to:
Find the number of pages in a pagination set listed at the bottom of https://codingislove.com/ This is purely to support task 2 by determining the loop end.
Loop over them
I believe these are linked but for those that want a single issue. I simply want to find the correct collection and loop over it to load each page.
The number of pages is, at time of writing, 6 as seen at the bottom of the webpage and shown below:
As an MCVE I simply want to find the number of pages and click my way through them. Using Selenium Basic.
What I have tried:
I have read through a number of online resources, I have listed but a few in references.
Task 1)
It seems that I should be able to find the count of pages using the Size property. But I can't seem to find the right object to use this with. I have made a number of attempts; a few shown below:
bot.FindElementsByXPath("//*[#id=""main""]/nav/div/a[3]").Size '<==this I think is too specific
bot.FindElementsByClass("page-numbers").Size
But these yield the run-time error 438:
"Object does not support this property or method"
And the following doesn't seem to expose the required methods:
bot.FindElementByCss(".navigation.pagination")
I have fudged with
bot.FindElementsByClass("page-numbers").Count + 1
But would like something more robust
Task 2)
I know that I can navigate to the next page, from page 1, with:
bot.FindElementByXPath("//*[#id=""main""]/nav/div/a[3]").Click
But I can't use this in a loop presumably because the XPath needs to be updated.
If not updated it leads to a runtime error 13.
As the re-directs follow a general pattern of
href="https://codingislove.com/page/pageNumber/"
I can again fudge my way through by constructing each URL in the loop with
bot.Get "https://codingislove.com/page/" & i & "/"
But I would like something more robust.
Question:
How do I loop over the pagination set in a robust fashion using selenium? Sure I am having a dense day and that there should be an easy to target appropriate collection to loop over.
Code - My current attempt
Option Explicit
Public Sub scrapeCIL()
Dim bot As New WebDriver, i As Long, pageCount As Long
bot.Start "chrome", "https://codingislove.com"
bot.Get "/"
pageCount = bot.FindElementsByClass("page-numbers").Count + 1 '
For i = 1 To pageCount 'technically can loop from 2 I know!
' bot.FindElementByXPath("//*[#id=""main""]/nav/div/a[3]").Click 'runtime error 13
' bot.FindElementByXPath("//*[#id=""main""]/nav/div/a[2]/span").Click ''runtime error 13
bot.Get "https://codingislove.com/page/" & i & "/"
Next i
Stop
bot.Quit
End Sub
Note:
Any supported browser will do. It doesn't have to be Chrome.
References:
Finding the number of pagination buttons in Selenium WebDriver
http://seleniumhome.blogspot.co.uk/2013/07/how-can-we-automate-pagination-using.html
Requirements:
Selenium Basic
ChromeDriver 2.37 'Or use IE but zoom must be at 100%
VBE Tools > references > Selenium type library
To click the element, it must be visible in the screen, so you need to scroll to the bottom of the page first (selenium might do this implicitly some times, but I don't find it reliable).
Try this:
Option Explicit
Public Sub scrapeCIL()
Dim bot As New WebDriver, btn As Object, i As Long, pageCount As Long
bot.Start "chrome", "https://codingislove.com"
bot.Get "/"
pageCount = bot.FindElementsByClass("page-numbers").Count
For i = 1 To pageCount
bot.ExecuteScript ("window.scrollTo(0,document.body.scrollHeight);")
Application.wait Now + TimeValue("00:00:02")
On Error Resume Next
Set btn = bot.FindElementByCss("a[class='next page-numbers']")
If btn.IsPresent = True Then
btn.Click
End If
On Error GoTo 0
Next i
bot.Quit
End Sub
Similar principle:
Option Explicit
Public Sub GetItems()
Dim i As Long
With New ChromeDriver
.Get "https://codingislove.com/"
For i = 1 To 6
.FindElementByXPath("//*[#id=""main""]/nav/div/a[3]").SendKeys ("Keys.PageDown")
Application.Wait Now + TimeValue("00:00:02")
On Error Resume Next
.FindElementByCss("a.next").Click
On Error GoTo 0
Next i
End With
End Sub
Reference:
'http://seleniumhome.blogspot.co.uk/2013/07/how-to-press-keyboard-in-selenium.html
If you're only interested in clicking through each of the pages (and getting the number of pages is just an aid to doing this) then you should be able to click this element until it's no longer there:
<span class="screen-reader-text">Next Page</span>
Using
bot.FindElementByXpath("//span[contains(text(), 'Next Page')]")
Have a loop click that link on each page load. Eventually it wont be there. Then use VBA's error/exception handling to handle whatever the equivalent of NoSuchElementException is in this implementation of WebDriver. You will need to re-find the element each time in the loop.
How about trying like this? Few days back I could figure out that there is an option .SendKeys("keys.END") which will lead you to the bottom of a page so that the driver can reach out the expected element to click. I used If Err.Number <> 0 Then Exit Do within the do loop so that if the scraper encounters any error, it will break out of loop as in, element not found error in this case when the clicking on the last page button is done.
Give this a shot:
Sub GetItems()
Dim pagenum As Object
With New ChromeDriver
.get "https://codingislove.com/"
Do
On Error Resume Next
Set pagenum = .FindElementByCss("a.next")
pagenum.SendKeys ("Keys.END")
Application.Wait Now + TimeValue("00:00:03")
pagenum.Click
If Err.Number <> 0 Then Exit Do
On Error GoTo 0
Loop
.Quit
End With
End Sub
Reference to add to the library:
Selenium Type Library
I'm hoping someone can help. I'm trying to speed up the process of filling a webform that must be completed dozens or hundreds of times with information stored in excel.
To do this, I need one button to open an IE window and navigate to a certain website's login page (I've figured this bit out). The user can then log in and navigate to the form that needs to be filled. Then, I'd like the user to be able to return to the excel page, click another button, which will automatically fill several drop downs and text boxes.
Within Excel, I already have some code to allow the user to search for the particular set of information that needs to go to the form, so all they should have to do is click the button to transfer it over. The first bit of the code is just this:
Public IE As Object
Public Sub OpenIE()
Set IE = CreateObject("InternetExplorer.Application")
IE.Visible = True
IE.Navigate "loginpage url"
End Sub
Where I'm having trouble, however, is having a different function access the same IE window once the user has logged in and navigated to the form. Right now I've got this:
Sub FillMacro()
Dim sh As Object, oWin As Object
Set sh = CreateObject("Shell.Application")
For Each oWin In sh.Windows
If TypeName(oWin.document) = "HTMLDocument" Then
Set IE = oWin
Exit For
End If
Next
IE.Visible = True
Application.Wait (Now + TimeValue("00:00:01"))
IE.document.getElementById("idec").Value = "John"
Application.Wait (Now + TimeValue("00:00:01"))
IE.document.getElementById("idee").Value = "Smith"
End Sub
Most of that I've gotten from other posts on this forum, and while I'm a bit of a novice at this, the problem seems to be that for some reason VBA can't find the text boxes with the id of LastName or FirstName. What's more, the IE.Visible = True doesn't bring the IE window back to the foreground, so I'm trying to find the proper line to do that. When I try to run the code, I get an "Object Required" error at:
IE.document.getElementById("idec").Value = "John"
I've tried searching this site and will continue to look, but any help in the meantime would be greatly appreciated!
On the Internet Explorer page, here is the line for the first text box I'm trying to fill:
<input name="componentListPanel:componentListView:1:component:patientLastNameContainer:patientLastName" class="input300" id="idec" type="text" maxlength="60" value="">
Why not automate logging process as well? Login could be stored in Excel and its value read by macro from cell.
As Tim Williams suggests, if there is Iframe on website, use (for me it works only with contentwindow included):
IE.document.getElementById("iFrameIdHere").contentwindow.document.getElemeântById("idec").Value = "John"
Instead of Application.Wait use:
Do Until IE.ReadyState = 4 And IE.Busy = False
DoEvents
Loop
It will save you a lot of time when page loads fast and prevent errors when loading exceeds wait time. Use it ONLY after page reloads (meaning after navigating or anything what causes page reloads, especially .click on HTML elements.
Use early binding, it's a bit faster than creating objects. It can increase performance by a few percent based on page loading speed, the faster pages load, the bigger increase.
Set IE = New InternetExplorer
Finally, you can toggle loading pictures, depending on whether you need to download images from website.
Public Sub ShowPictures(ByVal EnabledStatus As Boolean)
Public ScrapingCancelled as Boolean
Dim obj_Shell
Dim v_Result As Variant
Set obj_Shell = CreateObject("WScript.Shell")
'Reads the registry key that determines whether 'Show pictures' Internet Explorer advanced setting is enabled
v_Result = obj_Shell.RegRead("HKEY_CURRENT_USER\Software\Microsoft\Internet Explorer\Main\Display Inline Images")
Select Case v_Result
Case "yes" 'Pictures are displayed
If EnabledStatus = False Then _
obj_Shell.RegWrite "HKEY_CURRENT_USER\Software\Microsoft\Internet Explorer\Main\Display Inline Images", "no", "REG_SZ"
Case "no" 'Pictures are not displayed
If EnabledStatus = True Then _
obj_Shell.RegWrite "HKEY_CURRENT_USER\Software\Microsoft\Internet Explorer\Main\Display Inline Images", "yes", "REG_SZ"
Case Else
ScrapingCancelled = True
End Select
End Sub
No images loaded:
ShowPictures (0)
Images loaded:
ShowPictures (1)
A good practice is to set value to 1 in the end of macro.
I am really struggling to get my VBA code to wait until a website I am accessing, finishes an internal search it is performing.
So far I have tried:
Do While ie.Busy: DoEvents: Loop
Set doc = ie.document
Do While doc.readyState <> "complete": DoEvents: Loop
but it does not work. I could post the html 'refresh' function if that would help?
Any ideas would be greatly appreciated.
EDIT
It is using the following function which I have gathered from the source code:
var limit="5";
function beginrefresh()
{
if (!document.images) return;
if (limit==1)
document.getElementsByTagName("Form")[0].submit();
else
{
limit-=1;
window.status="Page will refresh in " + limit + " seconds.";
setTimeout("beginrefresh()",1000);
}
}
window.onload = function () { beginrefresh(); }
EDIT
I have cracked it and it was really simple in the end. I used the following to search for a known id on the page that was being loaded. So when it found this id it knew that the code could continue. A very useful technique IF you know an the name of an id on the page loading.
Do
Do While ie.Busy: DoEvents: Loop
Set doc = ie.document
Do While doc.readyState <> "complete": DoEvents: Loop
Set testobject = ie.document.getElementById("changeFilterCriteria")
If Not testobject Is Nothing Then Exit Do
Loop
EDIT
Or maybe not - it does work for about 5 or 6 times in a row but then crashes on
Set testobject = ie.document.getElementById("changeFilterCriteria")
IE comes up with 'internet explorer cannot display the webpage',
If I then click back and then refresh the page is still performing its internal search.
Any help would be gratefully received!
Jim
This is a tricky situation that I have come across numerous times as well. The traditional "Do...Until" loop with IE.ReadyState doesn't work after the first time the page is accessed, mostly because IE and Excel are out of synch at that point. The best approach I've found is to use a Boolean flag in a "Do...If" loop, where the flag is triggered by a reference to an element in the page you're refreshing. Until the page refreshes, the flag will be in one state. As long as the flag is in that state, the Do loop continues. So, somewhere after you've instructed the page to refresh, insert the following code (be sure to adapt for your specific circumstance):
Dim flag as Boolean: flag = False
Dim lng_cnt as long: long_cnt = 0
Dim elem_temp as IHTMLElement: Set elem_temp = IE.document.getElementById("*element_name*")
Do Unitl flag = True or lng_cnt = 30
If elem_temp Is Nothing Then
flag = False
Else
flag = True
Exit Do
End If
lng_cnt = lng_cnt +1
Application.Wait (Now() + TimeValue("00:00:01"))
Loop
Note: I added the counter (lng_cnt) to prevent infinite looping. It instructs the program to wait 1 second before continuing if the page hasn't loaded yet, and will discontinue after 30 seconds; because you never know if the page's server is down or whatever. Hope this helps.
When an action is performed, it will last for a while to get a message showing that this action is successful. Here I use a function to test if the message comes out:
For i = 0 To NumberOfiframe - 1
Set objPage = iframe(i).Object
Set objTag = objPage.GetElementsByTagName("span")
intTotalLink = objTag.Length - 1
For intCtr = 0 to intTotalLink
If objTag(intCtr) = null Then
Exit Function
End If
strContent = objTag(intCtr).InnerText
endPosition = InStr(1, strContent, "Not all transport requests yet")
If endPosition > 0 then
TRNotReleased = FALSE
Exit for
End If
Next
Next
while when the message comes out, QTP will pop up a window showing "Object Required: objTag(..)", my point is that QTP is not able to find the set object when Web UI changes.
As the message contains different information for each operation, is there any solutions for me to get over from this issue?
Thanks in advance.
When you use objPage.GetElementsByTagName you're getting a reference to a DOM object on the page, this is not a QTP test object but rather an object belonging to the browser. When the DOM in the browser changes the object you're holding onto ceases to be valid.
If you want to access the object after the HTML changes you'll have to ask QTP to retrieve it again. Perhaps like this:
For i = 0 To NumberOfiframe - 1
Set objPage = iframe(i).Object
Set objTag = objPage.GetElementsByTagName("span")
intTotalLink = objTag.Length - 1
For intCtr = 0 to intTotalLink
' Get the collection of objects again
Set objTagCurr = objPage.GetElementsByTagName("span")
Set currObj = objTagCurr(intCtr)