Getting table data on website via VBA is very slow - vba

I need to get the table from website via VBA, but the data is very big.
I used array to save every data, and it cost me over 300 seconds to run it...
Is there any method to accelerate?
(I need to key in and click elements to get the table, can I use "msxml2.xmlhttp"?)
(I am not allowed to use SeleniumBasic.)
this is my code:
Sub ie_()
Application.ScreenUpdating = False
If Weekday(Date) > 1 Then
w1 = DateAdd("d", -Weekday(Date) + 1, Date)
w2 = DateAdd("d", -7, w1)
End If
date1 = Format(w2, "yyyy-mm-dd")
date2 = Format(w1, "yyyy-mm-dd")
Set IE = CreateObject("InternetExplorer.application")
With IE
.Visible = True
.navigate myURL
Set report_btn = .Document.getElementsByClassName("sys-btn")(18)
report_btn.Click
Set next_btn = .Document.getElementByid("btnQry")
next_btn.Click
'key in element
.Document.All.txtTxnTime_From.Value = date1
.Document.All.txtTxnTime_End.Value = date2
.Document.All.btnQuery.Click
'search and get the table. Because the data is huge, I used the loop to wait for it....
Set tbl = .Document.getElementByid("tb_Main")
Do While tbl Is Nothing
Application.Wait (Now + TimeValue("0:00:10"))
Set tbl = .Document.getElementByid("tb_Main")
Loop
End With
'save the data in table to the array
'this step cost almost 300 seconds....
Set trs = tbl.getElementsByTagName("tr")
Dim arr() As Variant
ReDim arr(trs.Length - 1, 25)
i = 0
For Each rw In tbl.Rows
j = 0
For Each cel In rw.Cells
arr(i, j) = cel.innertext
j = j + 1
Next
i = i + 1
Next
'get the data in array and write to the cell
For i = 0 To trs.Length - 1
For j = 0 To 25
Cells(i + 1, j + 1) = arr(i, j)
Next
Next
'close ie
IE.Quit
Set IE = Nothing
End Sub

Related

VBA refining range

I am attempting to draw data from a separate sheet and put it into a corresponding cell if the conditions are met. My code works, but it is not efficient. I do not know how to change the For Next loop so that it attempts to draw data only until the final entry. Right now I have it set to go a hundred or so cells further than I need so that I wouldn't have to update the code as often when I input new data to the data sheet (or at least that was the thought). Here is my code:
Sub LRearTest()
Dim R As Integer
Dim j As Integer
For j = 89 To 250
For R = 1 To 300
If Worksheets("Input").Cells(j, 22).Value >= Worksheets("1036L").Cells(R, 5).Value And Worksheets("Input").Cells(j, 22).Value <= Worksheets("1036L").Cells(R, 6).Value Then
Worksheets("Input").Cells(j, 20).Value = Worksheets("1036L").Cells(R, 3).Value
End If
Next R
Next j
End Sub
The problem is when I run this code it takes almost two minutes before it is over. I am not sure if it is because I have used j and r as integers or what. Also I have a dozen of these on one module so I am not sure if that contributes. The code works like I said, it is just far too slow. Help is greatly appreciated.
The point that I am checking is in Column V of Sheet "Input". Each of my columns that I want to populate, F - U, use the same data in column V. The sheets that I am comparing the data in column V against are labeled as 1030L, 1030R, 1031L, 1031R, 1032L, 1032R, 1033L, 1033R, 1034L, 1034R, 1034LA, 1034RA, 1035L, 1035R, 1036L, and 1036R. The data being compared is in the same columns in every sheet. Thank you
Something like this should work for you:
Sub LRearTest()
Dim wb As Workbook
Dim wsInput As Worksheet
Dim wsData As Worksheet
Dim aDataParams() As String
Dim aInput As Variant
Dim aData As Variant
Dim InputIndex As Long
Dim DataIndex As Long
Dim ParamIndex As Long
Dim MinCol As Long
Set wb = ActiveWorkbook
Set wsInput = wb.Sheets("Input")
'Adjust the column associations for each sheet as necessary
ReDim aDataParams(1 To 16, 1 To 3)
aDataParams(1, 1) = "1030L": aDataParams(1, 2) = "F"
aDataParams(2, 1) = "1030R": aDataParams(2, 2) = "G"
aDataParams(3, 1) = "1031L": aDataParams(3, 2) = "H"
aDataParams(4, 1) = "1031R": aDataParams(4, 2) = "I"
aDataParams(5, 1) = "1032L": aDataParams(5, 2) = "J"
aDataParams(6, 1) = "1032R": aDataParams(6, 2) = "K"
aDataParams(7, 1) = "1033L": aDataParams(7, 2) = "L"
aDataParams(8, 1) = "1033R": aDataParams(8, 2) = "M"
aDataParams(9, 1) = "1034L": aDataParams(9, 2) = "N"
aDataParams(10, 1) = "1034R": aDataParams(10, 2) = "O"
aDataParams(11, 1) = "1034LA": aDataParams(11, 2) = "P"
aDataParams(12, 1) = "1034RA": aDataParams(12, 2) = "Q"
aDataParams(13, 1) = "1035L": aDataParams(13, 2) = "R"
aDataParams(14, 1) = "1035R": aDataParams(14, 2) = "S"
aDataParams(15, 1) = "1036L": aDataParams(15, 2) = "T"
aDataParams(16, 1) = "1036R": aDataParams(16, 2) = "U"
'Find minimum column
MinCol = wsInput.Columns.Count
For ParamIndex = LBound(aDataParams, 1) To UBound(aDataParams, 1)
If wsInput.Columns(aDataParams(ParamIndex, 2)).Column < MinCol Then MinCol = wsInput.Columns(aDataParams(ParamIndex, 2)).Column
Next ParamIndex
'Based on minimum column, determine column indexes for each sheet/column pair
For ParamIndex = LBound(aDataParams, 1) To UBound(aDataParams, 1)
aDataParams(ParamIndex, 3) = wsInput.Columns(aDataParams(ParamIndex, 2)).Column - MinCol + 1
Next ParamIndex
With wsInput.Range("F89", wsInput.Cells(wsInput.Rows.Count, "V").End(xlUp))
If .Row < 89 Then
MsgBox "No data in sheet [" & wsInput.Name & "]"
Exit Sub
End If
aInput = .Value
End With
For ParamIndex = LBound(aDataParams, 1) To UBound(aDataParams, 1)
'Define data sheet based on current column
Set wsData = wb.Sheets(aDataParams(ParamIndex, 1))
aData = wsData.Range("C1", wsData.Cells(wsData.Rows.Count, "F").End(xlUp)).Value
For InputIndex = LBound(aInput, 1) To UBound(aInput, 1)
For DataIndex = LBound(aData, 1) To UBound(aData, 1)
If aInput(InputIndex, UBound(aInput, 2)) >= aData(DataIndex, 3) _
And aInput(InputIndex, UBound(aInput, 2)) <= aData(DataIndex, 4) Then
aInput(InputIndex, aDataParams(ParamIndex, 3)) = aData(DataIndex, 1)
Exit For
End If
Next DataIndex
Next InputIndex
Set wsData = Nothing
Erase aData
Next ParamIndex
wsInput.Range("F89").Resize(UBound(aInput, 1), UBound(aInput, 2)) = aInput
Set wb = Nothing
Set wsInput = Nothing
Set wsData = Nothing
Erase aInput
Erase aData
Erase aDataParams
End Sub

How to copy specific part of row VBA Excel to another sheet?

I solved it on my own. I added a for loop. Here is my working code. Thanks to everyone else for trying to help.
Sub runMatch()
Dim critRemID, listRemID, critRemIDstart, listRemIDstart As Range
Set critRemID = Worksheets("Enterprise - score").Cells(2, 1)
Set listRemID = Worksheets("Sheet1").Cells(2, 1)
Set critRemIDstart = Worksheets("Enterprise - score").Cells(2, 30)
Set listRemIDstart = Worksheets("Sheet1").Cells(2, 2)
Dim i, j, index As Integer
i = 0
j = 0
Do While critRemID.Offset(i, 0) <> ""
If critRemID.Offset(i, 0) = listRemID.Offset(j, 0) Then
For index = 0 To 84
critRemIDstart.Offset(i, index) = listRemIDstart.Offset(j, index).Value
Next index
i = i + 1
j = 0
index = 0
Else
If listRemID.Offset(j, 0) = "" Then
j = 0
i = i + 1
Else
j = j + 1
End If
End If
Loop
End Sub
I have two sheets, they each have a the same IDs on each sheet but
different sets of data.
I want to scan through the rows of data and if there is a match, copy
the entire row from a certain column to another certain column to the
end of one of the sheets.
Sheet 1 is the sheet I want to copy info into, on the end I've created
the same headers for the data I want to bring over from sheet 2.
the code below is what I have, I set a range up for the IDs and one
for where I want the copied cells to start
Dim critRemID, listRemID, critRemIDstart, listRemIDstart As Range
Set critRemID = Worksheets("Enterprise - score").Cells(2, 1)
Set listRemID = Worksheets("Sheet1").Cells(2, 1)
Set critRemIDstart = Worksheets("Enterprise - score").Cells(2, 30)
Set listRemIDstart = Worksheets("Sheet1").Cells(2, 90)
Dim i, j As Integer
i = 0
j = 0
Do While critRemID.Offset(i, 0) <> ""
If critRemID.Offset(i, 0) = listRemID.Offset(j, 0) Then
critRemIDstart.Row(i) = listRemIDstart.Row(j).Value
i = i + 1
j = 0
Else
j = j + 1
End If
Loop
I keep getting this error
Wrong number of arguments or invalid property assignment
I tried going a different route but kept getting confused as shown
below. I was trying to have it copy each cell one by one and once it
reached an empty cell, it would move onto the next ID on the main
sheet and start over but this does nothing, I think it keeps
increasing both IDs on the sheet and never finds a match.
If critRemID.Offset(i, 0) = listRemID.Offset(j, 0) Then
critRemIDstart.Offset(i, k) = listRemIDstart.Offset(j, l).Value
k = k + 1
l = l + 1
Else
If listRemIDstart.Offset(j, l) = "" Then
j = j + 1
l = 0
i = i + 1
k = 0
Else
j = j + 1
i = i + 1
l = 0
k = 0
End If
End if
any help is appreciated. Thanks.
Range.Find method could find the key easily.
Dim critRem, listRem As Worksheet
Set critRem = Worksheets("Enterprise - score")
Set listRem = Worksheets("Sheet1")
Dim critRemID, listRemID, cell, matchedCell As Range
With critRem
Set critRemID = .Range(.Cells(2, 1), .Cells(.Rows.Count, 1).End(xlUp))
End With
With listRem
Set listRemID = .Range(.Cells(2, 1), .Cells(.Rows.Count, 1).End(xlUp))
End With
For Each cell In critRemID
Set matchedCell = listRemID.Find(cell.Value)
If matchedCell Is Nothing Then 'ID is not found
'Do nothing
Else 'ID is found, matchedCell is pointed to column A now
cell.Offset(0, 29).Resize(1, 10) = matchedCell.Offset(0, 89).Resize(1, 10)
'offset(0,29) means offsetting right 29 columns
'resize(0,10) means resizing the range with 1 row and 10 columns width
'feel free to change the number for your data
End If
Next cell
Note: If you are confused about offset().resize(), there is another approach. cell.Row gives you the row that the data should be written into, and matchedCell.Row gives you the row that the ID matched. So you can access certain cell by something like listRem.Range("D" & matchedCell.Row)
Tried to do it using the loop.
Sub Anser()
Dim critRemID As Range
Dim listRemID As Range
Dim critRemIDstart As Range
Dim listRemIDstart As Range
'::::Change Sheet names and column numbers:::::
Set critRemID = Worksheets("Sheet1").Cells(2, 1)
Set listRemID = Worksheets("Sheet2").Cells(2, 1)
Set critRemIDstart = Worksheets("Sheet1").Cells(2, 2)
Set listRemIDstart = Worksheets("Sheet2").Cells(2, 2)
Dim i, j As Integer
i = 0
j = 0
Do
If critRemID.Offset(i, 0) = listRemID.Offset(j, 0) Then
critRemIDstart.Offset(i) = listRemIDstart.Offset(j)
i = i + 1
j = 0
Else
j = j + 1
End If
Loop While critRemID.Offset(i, 0) <> ""
End Sub
If as you say both sheets have the same IDs, then why not use a Vlookup function to bring the data into Sheet1, then simply copy the results and paste as values so you get rid of the formula on them cells?
Something like a loop running:
For i = 1 to LastRow
Sheet1.cells(i, YourColumnNumber).value = "=VLOOKUP(RC[-1], Sheet2!R1:R1048576, 3, False)"
Next i

Macro running slowly

I had a simple macro that used to run in a minute. But it's now running very slowly. It takes about an hour to run. Is this because of the loop I'm using? Can someone help me see what went wrong?
Sub Runtable()
Sheets("RateTable").Cells(1, "A") = "ID"
Sheets("RateTable").Cells(1, "B") = "Section"
Sheets("RateTable").Cells(1, "C") = "Gender"
Sheets("RateTable").Cells(1, "D") = "Age"
'
LastID = Sheets("Input").Cells(2, 22)
For ID = 0 To LastID
LastSet = Sheets("Input").Cells(2, 19)
For myRow = 2 To LastSet
Sheets("RateTable").Cells(ID * (LastSet - 1) + myRow, 1) = Sheets("Input").Cells(ID + 2, 1)
Next myRow
Next ID
'
Dim myMyRow As Long
Dim OutputMyRow As Long
OutputMyRow = 2
LastID = Sheets("Input").Cells(2, 22)
LastSection = Sheets("Input").Cells(2, 21)
LastAge = Sheets("Input").Cells(2, 20)
For ID = 0 To LastID
For Section = 0 To LastSection
For myMyRow = 2 To LastAge
Sheets("RateTable").Cells(OutputMyRow, 2).Value = Sheets("Input").Cells(Section - FirstID + 2, "N").Value
OutputMyRow = OutputMyRow + 1
Next myMyRow
Next Section
Next ID
'
EndGenderLoop = Sheets("Input").Cells(2, 23)
For myRow = 2 To EndGenderLoop
Sheets("RateTable").Cells(myRow, 3) = Sheets("Input").Cells(2, 17)
Next myRow
'
EndAgeLoop = Sheets("Input").Cells(2, 24)
For AgeCurve = 0 To EndAgeLoop
'
For myRow = 2 To 52
Sheets("RateTable").Cells(AgeCurve * 51 + myRow, 4) = Sheets("Input").Cells(myRow, 10)
Next myRow
Next AgeCurve
'
End Sub
Use a status bar to determine where the code is slowing down. Here's one site with simple code (included below in case that link fails) but there are many others. For code this simple to be running 60 times slower now compared to earlier could indicate something wrong with the computer. Have you restarted? Can you revert to a previous backup state?
Option Explicit
Sub StatusBar()
Dim x As Integer
Dim MyTimer As Double
'Change this loop as needed.
For x = 1 To 250
'Dummy Loop here just to waste time.
'Replace this loop with your actual code.
MyTimer = Timer
Do
Loop While Timer - MyTimer < 0.03
Application.StatusBar = "Progress: " & x & " of 250: " & Format(x / 250, "Percent")
DoEvents
Next x
Application.StatusBar = False
End Sub

How to correct a userform when error 13 is displayed in VBA?

I'm currently on a project that search in a product database all non-referenced product (blank fields). When I click on the button that opens a userform, error 13 is displayed, here is the code:
Dim i As Integer
Dim j As Integer
Dim t As Integer
Dim r As Integer
t = 1
While Feuil3.Cells(t, 1) <> ""
t = t + 1
Wend
t = t - 1
For r = 2 To t
If Feuil3.Cells(r, 3) = "" Then
i = 1
While Feuil2.Cells(i, 1) <> ""
i = i + 1
Wend
Feuil2.Cells(i, 1) = Feuil3.Cells(r, 2)
End If
Next
i = 1
While Feuil2.Cells(i, 1) <> ""
i = i + 1
Wend
For j = 2 To i
If Feuil2.Cells(j, 2) = "" Then
list51.AddItem Feuil2.Cells(j, 1)
End If
Next
End Sub
It appears that the error comes from this line:If Feuil3.Cells(r, 3) = "" Then
My skills in VBA are limited, do you have any idea on how to fix this problem?
Thanks,
Have a look at this. Should do the same just a lot less iteratively
Dim Feuil2Rng As Range, Feuil3Rng As Range
Dim c
With Feuil3
Set Feuil3Rng = .Range(.Cells(2, 1), .Cells(.Cells(.Rows.Count, 1).End(xlUp).Row, 1))
End With
For Each c In Feuil3Rng
If c.Offset(0, 2) = vbNullString Then
With Feuil2
.Cells(.Cells(.Rows.Count, 1).End(xlUp).Row + 1, 1) = c.Offset(0,1)
End With
End If
Next
With Feuil2
Set Feuil2Rng = .Range(.Cells(2, 1), .Cells(.Cells(.Rows.Count, 1).End(xlUp).Row, 1))
End With
For Each c In Feuil2Rng
If c.Offset(0, 1) = vbNullString Then
list51.AddItem c.Value2
End If
Next

VBA If used range contains a word/text/value, go back to previous step

I wrote a macro to download data from a website, after the website is fully loaded, it will scrap the data by the html tag, however, sometimes the data is incorrectly scraped due to unknown error, I want to add a checking after each variant 'x' completed, e.g. If the activesheet contains the word "中报",then go back to the step "'Select the Report Type" to re-do the scraping. Also, I know some of the variables/data types are not set at the very beginning. Could anyone help to solve this? Thanks in advance!
Sub GetFinanceData()
Dim x As Variant
Dim IE As Object
For x = 1 To 1584
Dim URL As String, elemCollection As Object
Dim t As Integer, r As Integer, c As Integer
Worksheets("Stocks").Select
Worksheets("Stocks").Activate
'Open IE and Go to the Website
'URL = "http://stock.finance.sina.com.cn/hkstock/finance/00001.html"
URL = Cells(x, 1)
Set IE = CreateObject("InternetExplorer.Application")
With IE
.navigate URL
.Visible = False
Do While .Busy = True Or .readyState <> 4
Loop
DoEvents
Worksheets.Add(After:=Worksheets(Worksheets.Count)).Name = _
ThisWorkbook.Worksheets("Stocks").Range("B" & x).Value 'You could even simplify it and just state the name as Cells(x,2)
'Select the Report Type
Set selectItems = IE.Document.getElementsByTagName("select")
For Each i In selectItems
i.Value = "zero"
i.FireEvent ("onchange")
Application.Wait (Now + TimeValue("0:00:05"))
Next i
Do While .Busy: DoEvents: Loop
ActiveSheet.Range("A1:K2000").ClearContents
ActiveSheet.Range("A1").Value = .Document.getElementsByTagName("h1")(0).innerText
ActiveSheet.Range("B1").Value = .Document.getElementsByTagName("em")(0).innerText
ActiveSheet.Range("A4").Value = Worksheets("Stocks").Cells(1, 4)
'Find and Get Table Data
tblNameArr = Array(Worksheets("Stocks").Cells(2, 4), Worksheets("Stocks").Cells(3, 4), Worksheets("Stocks").Cells(4, 4), Worksheets("Stocks").Cells(5, 4))
tblStartRow = 6
Set elemCollection = .Document.getElementsByTagName("TABLE")
For t = 0 To elemCollection.Length - 1
For r = 0 To (elemCollection(t).Rows.Length - 1)
For c = 0 To (elemCollection(t).Rows(r).Cells.Length - 1)
ActiveSheet.Cells(r + tblStartRow, c + 1) = elemCollection(t).Rows(r).Cells(c).innerText
Next c
Next r
ActiveSheet.Cells(r + tblStartRow + 2, 1) = tblNameArr(t)
tblStartRow = tblStartRow + r + 4
Next t
End With
' cleaning up memory
IE.Quit
Next x
End Sub
This is cleaned up quite a bit.
I added a SelectReportType: line label. Whenever you want to go back to that condition, use insert the line
Goto SelectReportType
And it will take you to that spot. The better way to do it would be to place that code in a separate function so you can call it anytime your test for "中报" is true. But I'm not following your code well enough to understand what you are doing to assist you with that.
Sub GetFinanceData()
Dim x As Variant
Dim IE As Object
Dim URL As String, elemCollection As Object
Dim t As Integer, r As Integer, c As Integer
Dim selectItems As Variant, i As Variant
Dim tblNameArr() As String
Dim tblStartRow As Long
Worksheets("Stocks").Select
Worksheets("Stocks").Activate
For x = 1 To 1584
'Open IE and Go to the Website
'URL = "http://stock.finance.sina.com.cn/hkstock/finance/00001.html"
URL = Cells(x, 1)
Set IE = CreateObject("InternetExplorer.Application")
With IE
.Navigate URL
.Visible = False
Do While .Busy = True Or .ReadyState <> 4
Loop
DoEvents
Worksheets.Add(After:=Worksheets(Worksheets.count)).name = _
ThisWorkbook.Worksheets("Stocks").Range("B" & x).Value 'You could even simplify it and just state the name as Cells(x,2)
SelectReportType:
'Select the Report Type
Set selectItems = IE.Document.getElementsByTagName("select")
For Each i In selectItems
i.Value = "zero"
i.FireEvent ("onchange")
Application.Wait (Now + TimeValue("0:00:05"))
Next i
Do While .Busy: DoEvents: Loop
ActiveSheet.Range("A1:K2000").ClearContents
ActiveSheet.Range("A1").Value = .Document.getElementsByTagName("h1")(0).innerText
ActiveSheet.Range("B1").Value = .Document.getElementsByTagName("em")(0).innerText
ActiveSheet.Range("A4").Value = Worksheets("Stocks").Cells(1, 4)
'Find and Get Table Data
tblNameArr = Array(Worksheets("Stocks").Cells(2, 4), Worksheets("Stocks").Cells(3, 4), Worksheets("Stocks").Cells(4, 4), Worksheets("Stocks").Cells(5, 4))
tblStartRow = 6
Set elemCollection = .Document.getElementsByTagName("TABLE")
For t = 0 To elemCollection.Length - 1
For r = 0 To (elemCollection(t).Rows.Length - 1)
For c = 0 To (elemCollection(t).Rows(r).Cells.Length - 1)
ActiveSheet.Cells(r + tblStartRow, c + 1) = elemCollection(t).Rows(r).Cells(c).innerText
Next c
Next r
ActiveSheet.Cells(r + tblStartRow + 2, 1) = tblNameArr(t)
tblStartRow = tblStartRow + r + 4
Next t
End With
' cleaning up memory
IE.Quit
Next x
End Sub