Wait Until Website Internal Search Has Completed Using VBA - vba

I am really struggling to get my VBA code to wait until a website I am accessing, finishes an internal search it is performing.
So far I have tried:
Do While ie.Busy: DoEvents: Loop
Set doc = ie.document
Do While doc.readyState <> "complete": DoEvents: Loop
but it does not work. I could post the html 'refresh' function if that would help?
Any ideas would be greatly appreciated.
EDIT
It is using the following function which I have gathered from the source code:
var limit="5";
function beginrefresh()
{
if (!document.images) return;
if (limit==1)
document.getElementsByTagName("Form")[0].submit();
else
{
limit-=1;
window.status="Page will refresh in " + limit + " seconds.";
setTimeout("beginrefresh()",1000);
}
}
window.onload = function () { beginrefresh(); }
EDIT
I have cracked it and it was really simple in the end. I used the following to search for a known id on the page that was being loaded. So when it found this id it knew that the code could continue. A very useful technique IF you know an the name of an id on the page loading.
Do
Do While ie.Busy: DoEvents: Loop
Set doc = ie.document
Do While doc.readyState <> "complete": DoEvents: Loop
Set testobject = ie.document.getElementById("changeFilterCriteria")
If Not testobject Is Nothing Then Exit Do
Loop
EDIT
Or maybe not - it does work for about 5 or 6 times in a row but then crashes on
Set testobject = ie.document.getElementById("changeFilterCriteria")
IE comes up with 'internet explorer cannot display the webpage',
If I then click back and then refresh the page is still performing its internal search.
Any help would be gratefully received!
Jim

This is a tricky situation that I have come across numerous times as well. The traditional "Do...Until" loop with IE.ReadyState doesn't work after the first time the page is accessed, mostly because IE and Excel are out of synch at that point. The best approach I've found is to use a Boolean flag in a "Do...If" loop, where the flag is triggered by a reference to an element in the page you're refreshing. Until the page refreshes, the flag will be in one state. As long as the flag is in that state, the Do loop continues. So, somewhere after you've instructed the page to refresh, insert the following code (be sure to adapt for your specific circumstance):
Dim flag as Boolean: flag = False
Dim lng_cnt as long: long_cnt = 0
Dim elem_temp as IHTMLElement: Set elem_temp = IE.document.getElementById("*element_name*")
Do Unitl flag = True or lng_cnt = 30
If elem_temp Is Nothing Then
flag = False
Else
flag = True
Exit Do
End If
lng_cnt = lng_cnt +1
Application.Wait (Now() + TimeValue("00:00:01"))
Loop
Note: I added the counter (lng_cnt) to prevent infinite looping. It instructs the program to wait 1 second before continuing if the page hasn't loaded yet, and will discontinue after 30 seconds; because you never know if the page's server is down or whatever. Hope this helps.

Related

Is there a better way to accomplish this loop?

I'm pretty new with VBA, but I've been muddling through to make a program for my team.
This piece of code works most of the time, but tends to hang on occasion. I can't figure out why it would hang sometimes, and work perfectly most of the time, so I'm now trying to figure out a better way to accomplish this loop. I know that this method of looping isn't the best way to do things, but not sure how to accomplish the task.
My webpage operates in a PEGA Web application, and the native IE ready state indicators are always 'ready' so I have to use the web application's ready state markers.
Can anyone help me out?
Public Sub WaitingForRS()
' FIND THE C360 WINDOW
Marker = 0
Set objShell = CreateObject("Shell.Application")
IE_count = objShell.Windows.Count
For x = 0 To (IE_count - 1)
On Error Resume Next
my_url = objShell.Windows(x).Document.Location
my_title = objShell.Windows(x).Document.title
If my_title Like "Coverage User" & "*" Then
Set C360Window = objShell.Windows(x)
Marker = 1
Exit For
Else
End If
Next
If Marker = 0 Then
MsgBox ("C360 window is not found. Please ensure C360 is open in Internet Explorer and try again")
Else
End If
'FIND THE READY STATE INDICATOR
RSIndicatorDocMarker = 0
RSIndicatorDataMarker = 0
Set RSIndicatorPage = C360Window.Document
Set RSIndicatorClass = RSIndicatorPage.getelementsbyclassname("document-statetracker")(0)
RSIndicatorCheck:
'CHECK THE READY STATE DOC STATUS
If RSIndicatorClass.getattribute("data-state-doc-status") = "ready" Then
RSIndicatorDocMarker = 1
Else: RSIndicatorDocMarker = 0
End If
'CHECK THE READY STATE
If RSIndicatorClass.getattribute("data-state-busy-status") = "none" Then
RSIndicatorDataMarker = 1
Else: RSIndicatorDataMarker = 0
End If
'Compare the RSIndicators
If RSIndicatorDataMarker = 1 And RSIndicatorDocMarker = 1 Then
Else: GoTo RSIndicatorCheck
End If
End Sub
Maybe try using OnTime instead of the tight loop you currently have:
Public Sub WaitingForRS()
Dim win As Object
Dim w As Object, el, ready As Boolean, idle As Boolean
For Each w In CreateObject("Shell.Application").Windows
If w.Name Like "*Internet*" Then
If w.Title Like "Coverage user*" Then
Set win = w
Exit For
End If
End If
Next
If Not win Is Nothing Then
Set el = win.document.getelementsbyclassname("document-statetracker")(0)
ready = (el.getattribute("data-state-doc-status") = "ready")
idle = (el.getattribute("data-state-busy-status") = "none")
If ready And idle Then
ProceedWithNextSteps win 'do whatever comes next: pass in the window
Else
'wait for a while then try again
Application.OnTime Now + TimeSerial(0, 0, 1), "WaitingForRS"
End If
Else
MsgBox "Window not found!"
End If
End Sub
Might want to add a time limit so it doesn't keep looping forever if the page isn't "ready".
Check out my answer on this post Excel VBA Submitting data via IE on an online MS Forms not working
# Idea 5 subheading. The WaitForLoadSETx() function has a manual time over-ride and double loop to help catch the state.
This was the best I could do in VBA with IE object. Ideally you want to learn Selenium, PhantomJS, or Puppeteer for actual browser manipulation. There are of course cool apps, folks have built on top of these libraries to help (puppeteer recorder chrome add in)
Or to keep things in Excel, use a simple XMLHTTPRequest object to cut the browser out of the equation, and deal solely with the request / response from the web page. This is a good alternative because it lets you focus on scraping the html content, in text form without the Javascript or waiting for page to load.

could you inform me what it is wrong? "VBA web login", it's not work

I'm studying VBA for excel
now it is for crawling web by excel
i don't know what it is wrong.
could you tell me the solution?
private Sub CommandButton1_Click()
Set ie = CreateObject("internetexplorer.application")
ie.Visible = True
ie.navigate "http://www.marso.co.kr/member/login.html"
Do
If ie.readyState = 4 Then
Exit Do
Else
DoEvents
End If
Loop
ie.Document.getElementById("member_id").Value = "11111"
ie.Document.getElementById("member_passwd").Value = "11111"
ie.Document.getElementsByClassName("btn_280_black").Click
End Sub
Solution
Thanks to #plutian for the heads up in the comments
Convert
ie.Document.getElementsByClassName("btn_280_black").Click
To
ie.Document.getElementsByClassName("btn_280_black")(0).Click
This needs to be done because getElementsbyClassName returns an Array (even if only one Element was found).
Old (not quite right) answer
I believe the issue you may be facing is as follows:
At this part
ie.Document.getElementById("member_id").Value = "11111"
ie.Document.getElementById("member_passwd").Value = "11111"
ie.Document.getElementsByClassName("btn_280_black").Click
You use Upper-case Value and Click while you should be using lower-case value & click.
Also your accessing of the button is a bit wrong, getElementsbyClassName returns an Array (even if only one Element was found).
This means you need to access the first Element of said array by adding [0] to your button call => ie.Document.getElementsByClassName("btn_280_black")[0].click.
And finally, just accessing the button's .click function won't call the function so you'll need to call it by adding () to the end of your call, turning it into ie.Document.getElementsByClassName("btn_280_black")[0].click()
So at the end it should look more like this:
ie.Document.getElementById("member_id").value = "11111"
ie.Document.getElementById("member_passwd").value = "11111"
ie.Document.getElementsByClassName("btn_280_black")[0].click()
Hope this helps, if any errors pop up while trying this out, please add them to your reply.

Loop over a set of pages with Selenium Basic (VBA)

Task:
So my first foray into Selenium and I am attempting to:
Find the number of pages in a pagination set listed at the bottom of https://codingislove.com/ This is purely to support task 2 by determining the loop end.
Loop over them
I believe these are linked but for those that want a single issue. I simply want to find the correct collection and loop over it to load each page.
The number of pages is, at time of writing, 6 as seen at the bottom of the webpage and shown below:
As an MCVE I simply want to find the number of pages and click my way through them. Using Selenium Basic.
What I have tried:
I have read through a number of online resources, I have listed but a few in references.
Task 1)
It seems that I should be able to find the count of pages using the Size property. But I can't seem to find the right object to use this with. I have made a number of attempts; a few shown below:
bot.FindElementsByXPath("//*[#id=""main""]/nav/div/a[3]").Size '<==this I think is too specific
bot.FindElementsByClass("page-numbers").Size
But these yield the run-time error 438:
"Object does not support this property or method"
And the following doesn't seem to expose the required methods:
bot.FindElementByCss(".navigation.pagination")
I have fudged with
bot.FindElementsByClass("page-numbers").Count + 1
But would like something more robust
Task 2)
I know that I can navigate to the next page, from page 1, with:
bot.FindElementByXPath("//*[#id=""main""]/nav/div/a[3]").Click
But I can't use this in a loop presumably because the XPath needs to be updated.
If not updated it leads to a runtime error 13.
As the re-directs follow a general pattern of
href="https://codingislove.com/page/pageNumber/"
I can again fudge my way through by constructing each URL in the loop with
bot.Get "https://codingislove.com/page/" & i & "/"
But I would like something more robust.
Question:
How do I loop over the pagination set in a robust fashion using selenium? Sure I am having a dense day and that there should be an easy to target appropriate collection to loop over.
Code - My current attempt
Option Explicit
Public Sub scrapeCIL()
Dim bot As New WebDriver, i As Long, pageCount As Long
bot.Start "chrome", "https://codingislove.com"
bot.Get "/"
pageCount = bot.FindElementsByClass("page-numbers").Count + 1 '
For i = 1 To pageCount 'technically can loop from 2 I know!
' bot.FindElementByXPath("//*[#id=""main""]/nav/div/a[3]").Click 'runtime error 13
' bot.FindElementByXPath("//*[#id=""main""]/nav/div/a[2]/span").Click ''runtime error 13
bot.Get "https://codingislove.com/page/" & i & "/"
Next i
Stop
bot.Quit
End Sub
Note:
Any supported browser will do. It doesn't have to be Chrome.
References:
Finding the number of pagination buttons in Selenium WebDriver
http://seleniumhome.blogspot.co.uk/2013/07/how-can-we-automate-pagination-using.html
Requirements:
Selenium Basic
ChromeDriver 2.37 'Or use IE but zoom must be at 100%
VBE Tools > references > Selenium type library
To click the element, it must be visible in the screen, so you need to scroll to the bottom of the page first (selenium might do this implicitly some times, but I don't find it reliable).
Try this:
Option Explicit
Public Sub scrapeCIL()
Dim bot As New WebDriver, btn As Object, i As Long, pageCount As Long
bot.Start "chrome", "https://codingislove.com"
bot.Get "/"
pageCount = bot.FindElementsByClass("page-numbers").Count
For i = 1 To pageCount
bot.ExecuteScript ("window.scrollTo(0,document.body.scrollHeight);")
Application.wait Now + TimeValue("00:00:02")
On Error Resume Next
Set btn = bot.FindElementByCss("a[class='next page-numbers']")
If btn.IsPresent = True Then
btn.Click
End If
On Error GoTo 0
Next i
bot.Quit
End Sub
Similar principle:
Option Explicit
Public Sub GetItems()
Dim i As Long
With New ChromeDriver
.Get "https://codingislove.com/"
For i = 1 To 6
.FindElementByXPath("//*[#id=""main""]/nav/div/a[3]").SendKeys ("Keys.PageDown")
Application.Wait Now + TimeValue("00:00:02")
On Error Resume Next
.FindElementByCss("a.next").Click
On Error GoTo 0
Next i
End With
End Sub
Reference:
'http://seleniumhome.blogspot.co.uk/2013/07/how-to-press-keyboard-in-selenium.html
If you're only interested in clicking through each of the pages (and getting the number of pages is just an aid to doing this) then you should be able to click this element until it's no longer there:
<span class="screen-reader-text">Next Page</span>
Using
bot.FindElementByXpath("//span[contains(text(), 'Next Page')]")
Have a loop click that link on each page load. Eventually it wont be there. Then use VBA's error/exception handling to handle whatever the equivalent of NoSuchElementException is in this implementation of WebDriver. You will need to re-find the element each time in the loop.
How about trying like this? Few days back I could figure out that there is an option .SendKeys("keys.END") which will lead you to the bottom of a page so that the driver can reach out the expected element to click. I used If Err.Number <> 0 Then Exit Do within the do loop so that if the scraper encounters any error, it will break out of loop as in, element not found error in this case when the clicking on the last page button is done.
Give this a shot:
Sub GetItems()
Dim pagenum As Object
With New ChromeDriver
.get "https://codingislove.com/"
Do
On Error Resume Next
Set pagenum = .FindElementByCss("a.next")
pagenum.SendKeys ("Keys.END")
Application.Wait Now + TimeValue("00:00:03")
pagenum.Click
If Err.Number <> 0 Then Exit Do
On Error GoTo 0
Loop
.Quit
End With
End Sub
Reference to add to the library:
Selenium Type Library

VBA-Excel: Filling forms in an IE Window

I'm hoping someone can help. I'm trying to speed up the process of filling a webform that must be completed dozens or hundreds of times with information stored in excel.
To do this, I need one button to open an IE window and navigate to a certain website's login page (I've figured this bit out). The user can then log in and navigate to the form that needs to be filled. Then, I'd like the user to be able to return to the excel page, click another button, which will automatically fill several drop downs and text boxes.
Within Excel, I already have some code to allow the user to search for the particular set of information that needs to go to the form, so all they should have to do is click the button to transfer it over. The first bit of the code is just this:
Public IE As Object
Public Sub OpenIE()
Set IE = CreateObject("InternetExplorer.Application")
IE.Visible = True
IE.Navigate "loginpage url"
End Sub
Where I'm having trouble, however, is having a different function access the same IE window once the user has logged in and navigated to the form. Right now I've got this:
Sub FillMacro()
Dim sh As Object, oWin As Object
Set sh = CreateObject("Shell.Application")
For Each oWin In sh.Windows
If TypeName(oWin.document) = "HTMLDocument" Then
Set IE = oWin
Exit For
End If
Next
IE.Visible = True
Application.Wait (Now + TimeValue("00:00:01"))
IE.document.getElementById("idec").Value = "John"
Application.Wait (Now + TimeValue("00:00:01"))
IE.document.getElementById("idee").Value = "Smith"
End Sub
Most of that I've gotten from other posts on this forum, and while I'm a bit of a novice at this, the problem seems to be that for some reason VBA can't find the text boxes with the id of LastName or FirstName. What's more, the IE.Visible = True doesn't bring the IE window back to the foreground, so I'm trying to find the proper line to do that. When I try to run the code, I get an "Object Required" error at:
IE.document.getElementById("idec").Value = "John"
I've tried searching this site and will continue to look, but any help in the meantime would be greatly appreciated!
On the Internet Explorer page, here is the line for the first text box I'm trying to fill:
<input name="componentListPanel:componentListView:1:component:patientLastNameContainer:patientLastName" class="input300" id="idec" type="text" maxlength="60" value="">
Why not automate logging process as well? Login could be stored in Excel and its value read by macro from cell.
As Tim Williams suggests, if there is Iframe on website, use (for me it works only with contentwindow included):
IE.document.getElementById("iFrameIdHere").contentwindow.document.getEleme‌ntById("idec").Value = "John"
Instead of Application.Wait use:
Do Until IE.ReadyState = 4 And IE.Busy = False
DoEvents
Loop
It will save you a lot of time when page loads fast and prevent errors when loading exceeds wait time. Use it ONLY after page reloads (meaning after navigating or anything what causes page reloads, especially .click on HTML elements.
Use early binding, it's a bit faster than creating objects. It can increase performance by a few percent based on page loading speed, the faster pages load, the bigger increase.
Set IE = New InternetExplorer
Finally, you can toggle loading pictures, depending on whether you need to download images from website.
Public Sub ShowPictures(ByVal EnabledStatus As Boolean)
Public ScrapingCancelled as Boolean
Dim obj_Shell
Dim v_Result As Variant
Set obj_Shell = CreateObject("WScript.Shell")
'Reads the registry key that determines whether 'Show pictures' Internet Explorer advanced setting is enabled
v_Result = obj_Shell.RegRead("HKEY_CURRENT_USER\Software\Microsoft\Internet Explorer\Main\Display Inline Images")
Select Case v_Result
Case "yes" 'Pictures are displayed
If EnabledStatus = False Then _
obj_Shell.RegWrite "HKEY_CURRENT_USER\Software\Microsoft\Internet Explorer\Main\Display Inline Images", "no", "REG_SZ"
Case "no" 'Pictures are not displayed
If EnabledStatus = True Then _
obj_Shell.RegWrite "HKEY_CURRENT_USER\Software\Microsoft\Internet Explorer\Main\Display Inline Images", "yes", "REG_SZ"
Case Else
ScrapingCancelled = True
End Select
End Sub
No images loaded:
ShowPictures (0)
Images loaded:
ShowPictures (1)
A good practice is to set value to 1 in the end of macro.

IBM Connections Activities

I need to Export IBM Connections Activity to csv file by VBA. Do you have any experience with this? It can be done manually by clicking on Edit Activity -> Export Activity -> Export
Edited:
Yes, I am using IBM Connections, interacting on web page in a browser. When I used inspect element, I saw that code for export button is this
<input class="lotusBtn" dojoattachpoint="exportAct_AP" value="Export" dojoattachevent="onclick:exporter" type="button">
so I tried something like this, but I am not still able to fully identify click action:
Sub IEACObject2()
Dim i As Long
Dim objCollection As Object
Set IE = CreateObject("InternetExplorer.Application")
IE.Visible = True
IE.Navigate "https://w3-connections.ibm.com/activities/service/html/mainpage#activitypage,25e2e3b1-c4e9-4448-95ae-b72432871095"
Do While IE.Busy
DoEvents
Loop
Set objCollection = IE.Document.getElementsByTagName("input")
i = 0
While i < objCollection.Length
If (objCollection(i).Type = "button" And objCollection(i).Value = "Export") Then
objCollection(i).Click
End If
i = i + 1
Wend
Set IE = Nothing
End Sub
there are API's available.
On Premises : https://www-10.lotus.com/ldd/lcwiki.nsf/xpAPIViewer.xsp?lookupName=IBM+Connections+5.5+API+Documentation
Cloud: https://www-10.lotus.com/ldd/appdevwiki.nsf
You should be able to find all the infos there.....
Maybe use the atom feed (XML) : https://YOUR-CONNECTIONS-SERVER/activities/service/atom2/descendants?nodeUuid=YOUR-ACTIVITY-UID
There is a button for this feed in every activity
I am sure VBA is able to deal with (atom) XML