I've got a bit of code know that works with pulling information from a cell and paste it into a search engine to search. Which works fine as a tested on both google and bing pulls back what I want to search.
Now I've changed the code which I think is correct to an internal website but now get and run-time error and automation error. It will load the site fine, Just not search through it.
I'm no expert at this so still trying to learn VBA. Also if you can advise any pointers on the best way to troubleshoot things likes.
As at the moment, the only way I figure it out is to break each line see what it does and find the fault. But this is a bit beyond me.
The only thing I think it could be but might be wrong is missing a reference
I've got the HTML Object library and Internet controls.
Run time error
This is the code from the site for the search box.
<input class="genInput" type="text" size="32" name="xx_quicksearch" value="" id="quicksearchbox">
This is my code,
Sub Test()
Dim ie As Object
Dim form As Variant
Dim button As Variant
Dim LR As Integer
Dim var As String
LR = Cells(Rows.Count, 1).End(xlUp).Row
For x = 2 To LR
var = Cells(x, 1).Value
Set ie = CreateObject("internetexplorer.application")
ie.Visible = True
With ie
.Visible = True
.navigate "*******"
While Not .readyState = READYSTATE_COMPLETE
Wend
End With
'Wait some to time for loading the page
While ie.Busy
DoEvents
Wend
Application.Wait (Now + TimeValue("0:00:02"))
ie.document.getElementById("quicksearch").Value = var
'code to click the button
Set form = ie.document.getElementsByTagName("form")
Application.Wait (Now + TimeValue("0:00:02"))
Set button = form(0).onsubmit
form(0).submit
'wait for page to load
While ie.Busy
DoEvents
Wend
Next x
End Sub
The id is different from what you posted. In: ie.document.getElementById("quicksearch").Value = var try typing "quicksearchbox" rather than just "quicksearch"
Related
Good afternoon all,
Please forgive me, I am new to VBA and have been practicing within my workbooks but for my work would be highly useful to branch out to Webscraping.
The following is my code.
Sub pullsalesdatafromonline()
Dim IE As Object
Dim doc As HTMLDocument
Set IE = CreateObject("InternetExplorer.Application")
IE.Visible = True
IE.navigate "https://www.peoplestuffuk.com/WFMMCDPRD/Login.jsp"
Do While IE.Busy Or IE.readyState <> 4
Application.Wait DateAdd("s", 1, Now)
Loop
Set doc = IE.document
doc.getElementById("txtUserID").Click
doc.getElementById("txtUserID").Value = Usernameinserted
doc.getElementById("txtPassword").Click
doc.getElementById("txtPassword").Value = PasswordInserted
doc.getElementById("Login").Click
IE.navigate "https://www.peoplestuffuk.com/WFMMCDPRD/LoginSubmit.jsp"
IE.navigate "https://www.peoplestuffuk.com/WFMMCDPRD/ModuleSelection.jsp"
IE.navigate "https://www.peoplestuffuk.com/WFMMCDPRD/dashboard/main/dashboardmain.jsp"
IE.navigate "https://www.peoplestuffuk.com/WFMMCDPRD/rrd/matrixReport.jsp?fromPM=N"
End Sub
Here is the element that I am trying to scrape from;
<INPUT onclick=javaScript:submitForm() class=button-t type=button value=Login>
<SPAN id=Login_span class=buttonbox></SPAN>
I've looked at several other links that have looked into firing the .onclick event but, can't quite seem to figure it out.
The error in VBA highlights the:
IE.navigate "https://www.peoplestuffuk.com/WFMMCDPRD/ModuleSelection.jsp"
line. When my username and password enter, the page refreshes/ tries to exectute and then empties the Login and password info and leaves me on the Login page.
Would greatly appreciate someone's input.
Thank you
If you had a demo username and password to share then there was an efficient way of logging in using xhr. However, as for your current attempt concern, they are not the ids you used them within .getElementById(). Given that the following approach should log you in. Make sure to uncomment the delay, if the script fails to do the doing.
Sub pullsalesdatafromonline()
Dim IE As Object, doc As HTMLDocument
Set IE = CreateObject("InternetExplorer.Application")
IE.Visible = True
IE.navigate "https://www.peoplestuffuk.com/WFMMCDPRD/Login.jsp"
While IE.Busy Or IE.readyState < 4: DoEvents: Wend
Application.Wait Now + TimeValue("00:00:03")
Set doc = IE.document
doc.querySelector("[name='txtUserID']").innerText = "username"
doc.querySelector("[name='txtPassword']").innerText = "password"
' Application.Wait Now + TimeValue("00:00:03")
doc.querySelector("input[value='Login']").Click
End Sub
I'm trying to simulate the interaction with Google through the IE app and going through the DOM to get the classes I need and all is fine, stepping though the code, except the .Click action which doesn't cause a crash but it doesn't do anything (page doesn't navigate) - Code and screenshot of HTML below:
Option Explicit
Private Sub Test_Automation()
Dim ie, doc, eInput, eButton, eButtons As Object
Dim sURL, sTest As String
Set ie = CreateObject("internetexplorer.application")
sURL = "https://www.google.co.uk/?gfe_rd=cr&ei=IpDvWK72LsjCaJCbjKAL&gws_rd=ssl"
sTest = "Test"
With ie
.Visible = True
.Navigate sURL
End With
Do While ie.Busy Or ie.readyState <> 4
DoEvents
Loop
Set doc = ie.document
Set eInput = doc.getElementByid("lst-ib")
Set eButtons = doc.getElementsByTagName("input")
eInput.Value = sTest
For Each eButton In eButtons
If (eButton.getattribute("name") = "btnK") Then
eButton.Click
Exit For
End If
Next
End Sub
Any advise on what I'm doing wrong would be great!
You can get rid of your For...Next loop at the bottom and replace it with this to click the button:
doc.forms(0).submit
The 0 can be changed to another number (such as 1 or 2) to click on a different button. If there are multiple buttons on a page that can be clicked on it will just take some trial and error to find out which number matches the button you want to click.
I have a VERY basic script I am using with an excel spreadsheet to populate a form. It inserts data into the fields then waits for me to click the submit button on each page. Then it waits for the page to load and fills in the next set of fields. I click submit, next page loads, etc. Finally, on the last two pages I have to do some manual input that I can't do with the spreadsheet. Code thus far is shown below.
My question is, at the end of what I have there now, how can I make the system wait for me to fill out that last page, then once I submit it realize that it has been submitted and loop back to the beginning, incrementing the row on the spreadsheet so that we can start over and do the whole thing again for the next student?
As you will probably be able to tell I am not a programmer, just a music teacher who does not savor the idea of filling out these forms manually for all 200 of my students and got the majority of the code you see from a tutorial.
Function FillInternetForm()
Dim IE As Object
Set IE = CreateObject("InternetExplorer.Application")
'create new instance of IE. use reference to return current open IE if
'you want to use open IE window. Easiest way I know of is via title bar.
IE.Navigate "https://account.makemusic.com/Account/Create/?ReturnUrl=/OpenId/VerifyGradebookRequest"
'go to web page listed inside quotes
IE.Visible = True
While IE.busy
DoEvents 'wait until IE is done loading page.
Wend
IE.Document.All("BirthMonth").Value = "1"
IE.Document.All("BirthYear").Value = "2000"
IE.Document.All("Email").Value = ThisWorkbook.Sheets("queryRNstudents").Range("f2")
IE.Document.All("Password").Value = ThisWorkbook.Sheets("queryRNstudents").Range("e2")
IE.Document.All("PasswordConfirm").Value = ThisWorkbook.Sheets("queryRNstudents").Range("e2")
IE.Document.All("Country").Value = "USA"
IE.Document.All("responseButtonsDiv").Click
newHour = Hour(Now())
newMinute = Minute(Now())
newSecond = Second(Now()) + 3
waitTime = TimeSerial(newHour, newMinute, newSecond)
Application.Wait waitTime
IE.Document.All("FirstName").Value = ThisWorkbook.Sheets("queryRNstudents").Range("a2")
IE.Document.All("LastName").Value = ThisWorkbook.Sheets("queryRNstudents").Range("b2")
IE.Document.All("Address1").Value = "123 Nowhere St"
IE.Document.All("City").Value = "Des Moines"
IE.Document.All("StateProvince").Value = "IA"
IE.Document.All("ZipPostalCode").Value = "50318"
End Function
I would use the events of the IE, more specifically the form, something like this, using MSHTML Controls library.
Private WithEvents IEForm As MSHTML.HTMLFormElement
Public Sub InternetExplorerTest()
Dim ie As SHDocVw.InternetExplorer
Dim doc As MSHTML.HTMLDocument
Set ie = New SHDocVw.InternetExplorer
ie.Visible = 1
ie.navigate "http://stackoverflow.com/questions/tagged/vba"
While ie.readyState <> READYSTATE_COMPLETE Or ie.Busy
DoEvents
Wend
Set doc = ie.document
Set IEForm = doc.forms(0)
End Sub
Private Function IEForm_onsubmit() As Boolean
MsgBox "Form Submitted"
End Function
Currently I have 2 pieces of code that work separately, but when used together they don't work properly.
The first code asks the user to input information which is stored. It then navigates to the correct webpage where it uses the stored user input information to navigate via filling and submitting a form. It arrives at the correct place.
The second code uses a specific URL via ie.navigate "insert url here" to navigate to the same place as the first code. It then scrapes URL data and stores it in a newly created sheet. It does this correctly.
When merging them I replace the navigation segment from the second code with the first code, but then it only stores the first 5 of 60 URLs as if it hadn't fully loaded the page before scraping data. It seems to skip the code directly after ie.document.forms(0).submit which is supposed to wait for the page to load before moving on to the scraping..
extra info: the button wasn't defined so I cannot just click it so I had to use ie.document.forms(0).submit
Summary of what I want the code to do:
request user input
store user input
open ie
navigate to page
enter user input into search field
select correct search category from listbox
submit form
'problem happens here
scrape url data
store url data in specific excel worksheet
The merged code:
Sub extractTablesData()
Dim ie As Object, obj As Object
Dim Var_input As String
Dim elemCollection As Object
Dim html As HTMLDocument
Dim Link As Object
Dim erow As Long
' create new sheet to store info
Application.DisplayAlerts = False
ThisWorkbook.Sheets("HL").Delete
ThisWorkbook.Sheets.Add.Name = "HL"
Application.DisplayAlerts = True
Set ie = CreateObject("InternetExplorer.Application")
Var_input = InputBox("Enter info")
With ie
.Visible = True
.navigate ("URL to the webpage")
While ie.readyState <> 4
DoEvents
Wend
'Input Term 1 into input box
ie.document.getElementById("trm1").Value = Var_input
'accessing the Field 1 ListBox
For Each obj In ie.document.all.Item("FIELD1").Options
If obj.Value = "value in listbox" Then
obj.Selected = True
End If
Next obj
' button undefined - using this to submit form
ie.document.forms(0).submit
'----------------------------------------------------------------
'seems to skip this part all together when merged
'Wait until IE is done loading page
Do While ie.readyState <> READYSTATE_COMPLETE
Application.StatusBar = "Trying to go to website…"
DoEvents
Loop
'----------------------------------------------------------------
Set html = ie.document
Set ElementCol = html.getElementsByTagName("a")
For Each Link In ElementCol
erow = Worksheets("HL").Cells(Rows.Count, 1).End(xlUp).Offset(1, 0).Row
Cells(erow, 1).Value = Link
Cells(erow, 1).Columns.AutoFit
Next
Application.StatusBar = “”
Application.ScreenUpdating = True
End With
End Sub
I've been stuck for quite some time on this and haven't found any solutions on my own so I'm reaching out. Any help will be greatly appreciated!
You mentioned you think the website might not be fully loaded. This is a common problem because of the more dynamic elements on a webpage. The easiest way to handle this is to insert the line:
Application.Wait Now + Timevalue("00:00:02")
This will force the code to pause for an additional 2 seconds. Insert this line below the code which waits for the page to load and this will give Internet Explorer a chance to catch back up. Depending on the website and the reliability of your connection to it I recommend adjusting this value anywhere up to about 5 seconds.
Most websites seem to require additional waiting like this, so handy code to remember when things don't work as expected. Hope this helps.
I solved this by using a completely different method. I used a query table with strings to go where I wanted.
Sub ExtractTableData()
Dim This_input As String
Const prefix As String = "Beginning of url"
Const postfix As String = "end of url"
Dim qt As QueryTable
Dim ws As Worksheet
Application.DisplayAlerts = False
ThisWorkbook.Sheets("HL").Delete
ThisWorkbook.Sheets.Add.Name = "HL"
Application.DisplayAlerts = True
This_input = InputBox("enter key info to go to specific url")
Set ws = ActiveSheet
Set qt = ws.QueryTables.Add( _
Connection:="URL;" & prefix & This_input & postfix, _
Destination:=Worksheets("HL").Range("A1"))
qt.RefreshOnFileOpen = True
qt.WebSelectionType = xlSpecifiedTables
'qt.webtables is key to getting the specific table on the page
qt.WebTables = 2
qt.Refresh BackgroundQuery:=False
End Sub
I have written a macro that will open a webpage, find the spots I need to put the data into, and then I want the macro to hit a prefill button, then hit Ok.
The page source code for the button is:
<input type="text" size="30" maxlength="40" name="name" id="name">
<input type="button" value="Prefill" onclick="prefill()">
I've been searching for answers all week and I think I have a basic understanding of how this is supposed to work by running a loop to search for it, but I'm having no luck in my endeavor of actually getting this to work.
Can someone show me the loop that will search my page for this button?
Thank you in advance.
Requested code so far
Private Sub Populate_Click()
Dim i As Long
Dim IE As Object
Dim objElement As Object
Dim objCollection As Object
Set IE = CreateObject("InternetExplorer.Application")
IE.Visible = True
IE.Navigate "website" 'make sure you've logged into the page
Do
DoEvents
Loop Until IE.READYSTATE = 3
Do
DoEvents
Loop Until IE.READYSTATE = 4
ActiveSheet.EnableCalculation = False
ActiveSheet.EnableCalculation = True
Call IE.document.getelementbyid("name").SetAttribute("value", ActiveSheet.Range("b2").Value)
Call IE.document.getelementbyid("aw_login").SetAttribute("value", ActiveSheet.Range("a2").Value)
Set objCollection = IE.document.getElementsByTagName("input")
i = 0
While i < objCollection.Length
If objCollection(i).Type = "button" And _
objCollection(i).Name = "Prefill" Then
Set objElement = objCollection(i)
End If
i = i + 1
Wend
objElement.Click
End Sub
Looks like you are pretty close, this is what I have used to do something similiar
With ie.document
Set elems = .getelementsbytagname("input")
For Each e In elems
If (e.getattribute("className") = "saveComment") Then
e.Click
Exit For
End If
Next e
End With
You will probably just have to change the if statement.
I also notice that your code refers to .Name = "Prefill" but your html snippet refers to .value = "Prefill"