How to automate VBA macro to run through entire file? - vba

I have a VBA Macro that needs to run through a very large Excel file (about 550,000 rows).
I have found that the Macro can only run through about 20,000 lines at a time - any bigger than that and it gives an 'Overflow' error.
My only solution right now is to set j (row number) = 1 to 20,000, run it, then set j from 20,001 to 40,000, run it, then set j from 40,001 to 60,000 etc....all the way to 550,000.
This is a tedious process and I'm wondering is there any way to automate it so that I don't have to keep going back in to change the numbers?
Code looks something like:
Sub ExposedDays()
Dim j As Integer
Dim k As Integer
Dim l As Integer
Dim result As String
For j = 2 To 10000
k = 1
Do While Len(Worksheets("my8").Cells(j, k)) > 0
k = k + 1
Loop
Worksheets("my8").Cells(j, 131) = Worksheets("my8").Cells(j, k - 6)
Worksheets("my8").Cells(j, 132) = Worksheets("my8").Cells(j, k - 5)
.......

The integer data type allocates only enough memory to hold integers -32,768 to 32,767. When you:
dim j as integer
an 'Overflow' error will result when j exceeds 32,767. To solve this problem you need to assign a larger data type.
The Long data type is large enough to hold integers -2,147,483,648 to 2,147,483,647.
If you:
dim j as long
dim k as long
you should be able to avoid this error.
Hope this helps.

Related

Dim n variables where n is a user defined number

I have to dim n variables in my software, where n is a number that user types in the interface. I'd start with a for loop, in order to declare these variables, with something like:
for i = 0 to n
dim r.i as IRow = worksheet.CreateRow(i)
next
This is what I think but obviously I know that's wrong. Does anyone knows if this is possible? I can't create neither List(Of) nor arrays because these are rows of an excel file I'm trying to export, otherwise NPOI returns me error.
What does Excel have anything to do with not using a List? You can absolutely use worksheet.CreateRow() with a list:
Dim rows As New List(Of IRow)
For i As Integer = 0 to n - 1
rows.Add(worksheet.CreateRow(i))
Next
Now the name of each row is row(0), row(1), row(2) ... row(n - 1)
If you've tried this and NPOI is giving you an error, you should show that code around where the error is thrown and tell us what the error is.
Based on comments:
'Assuming everything has the same number of entries as ListBox1, per the comments
Dim rows As New List(Of IRow)
Dim row As IRow = worksheet.CreateRow(0)
row.CreateCell(0).SetCellValue("Time")
row.CreateCell(1).SetCellValue("hrr")
For i As Integer = 0 To ListBox1.Items.Count - 1
row = worksheet.CreateRow(i+1)
row.CreateCell(0).SetCellValue(ListBox1.Items(i))
row.CreateCell(1).SetCellValue(ListBox2.Items(i))
rows.Add(row)
Next i

VBA Excel - Generate real random numbers that check if result exist in given range

I am using following function to create random number and check if the resulting integer is in a given range:
Function RandBetweenInt(Lowest As Long, Highest As Long, Exclude As Range) As Long
Dim R As Long
Dim C As Range
Do
R = Lowest + Int(Rnd() * (Highest + 1 - Lowest))
For Each C In Exclude
If R = C Then Exit For
Next C
Loop Until C Is Nothing
RandBetweenInt = R
Application.Volatile
End Function
After calling the function RandBetweenInt(0, 999, tgt.Range("K1:K1000")) the result is correct, but I see that the resulting number is always the same (f.i 705) if I just close the file and open it again, so there is no a real random generation happening. Even if I create consecutive random numbers, the sequence seems to be always the same (705, 332, 107...).
For me is really important that each time I create the random number the result is really random. Do you have a hint?

Capabilities of a Long in VBA [duplicate]

I often see questions relating to Overflow errors with vba.
My question is why use the integer variable declaration instead of just defining all numerical variables (excluding double etc.) as long?
Unless you're performing an operation like in a for loop where you can guarantee that the value won't exceed the 32,767 limit, is there an impact on performance or something else that would dictate not using long?
Integer variables are stored as 16-bit (2-byte) numbers
Office VBA Reference
Long (long integer) variables are stored as signed 32-bit (4-byte) numbers
Office VBA Reference
So, the benefit is in reduced memory space. An Integer takes up half the memory that a Long does. Now, we are talking about 2 bytes, so it's not going to make a real difference for individual integers, it's only a concern when you are dealing with TONS of integers (e.g large arrays) and memory usage is critical.
BUT on a 32 bit system, the halved memory usage comes at a performance cost. When the processor actually performs some computation with a 16 bit integer (e.g. incrementing a loop counter), the value silently gets converted to a temporary Long without the benefit of the larger range of numbers to work with. Overflows still happen, and the register that the processor uses to store the values for the calculation will take the same amount of memory (32 bits) either way. Performance may even be hurt because the datatype has to be converted (at a very low level).
Not the reference I was looking for but....
My understanding is that the underlying VB engine converts integers to long even if its declared as an integer. Therefore a slight speed decrease can be noted. I have believed this for some time and perhaps thats also why the above statement was made, I didnt ask for reasoning.
ozgrid forums
This is the reference I was looking for.
Short answer, in 32-bit systems 2 byte integers are converted to 4 byte
Longs. There really is no other way so that respective bits correctly line
up for any form of processing. Consider the following
MsgBox Hex(-1) = Hex(65535) ' = True
Obviously -1 does not equal 65535 yet the computer is returning the correct
answer, namely
"FFFF" = "FFFF"
However had we coerced the -1 to a long first we would have got the right
answer (the 65535 being greater than 32k is automatically a long)
MsgBox Hex(-1&) = Hex(65535) ' = False
"FFFFFFFF" = "FFFF"
Generally there is no point in VBA to declare "As Integer" in modern
systems, except perhaps for some legacy API's that expect to receive an
Integer.
pcreview forum
And at long last I found the msdn documentation I was really truly looking for.
Traditionally, VBA programmers have used integers to hold small
numbers, because they required less memory. In recent versions,
however, VBA converts all integer values to type Long, even if they're
declared as type Integer. So there's no longer a performance advantage
to using Integer variables; in fact, Long variables may be slightly
faster because VBA does not have to convert them.
To clarify based on the comments: Integers still require less memory to store - a large array of integers will need significantly less RAM than an Long array with the same dimensions. But because the processor needs to work with 32 bit chunks of memory, VBA converts Integers to Longs temporarily when it performs calculations
So, in summary, there's almost no good reason to use an Integer type these days. Unless you need to Interop with an old API call that expects a 16 bit int, or you are working with large arrays of small integers and memory is at a premium.
One thing worth pointing out is that some old API functions may be expecting parameters that are 16-bit (2-byte) Integers and if you are on a 32 bit and trying to pass an Integer (that is already a 4-byte long) by reference it will not work due to difference in length of bytes.
Thanks to Vba4All for pointing that out.
Even though this post is four years old, I was curious about this and ran some tests. The most important thing to note is that a coder should ALWAYS declare a variable as SOMETHING. Undeclared variables clearly performed the worst (undeclared are technically Variant)
Long did perform the fastest, so I have to think that Microsoft's recommendation to always use Long instead of Integer makes sense. I'm guessing the same as true with Byte, but most coders don't use this.
RESULTS ON 64 BIT WINDOWS 10 LAPTOP
Code Used:
Sub VariableOlymics()
'Run this macro as many times as you'd like, with an activesheet ready for data
'in cells B2 to D6
Dim beginTIME As Double, trials As Long, i As Long, p As Long
trials = 1000000000
p = 0
beginTIME = Now
For i = 1 To trials
Call boomBYTE
Next i
Call Finished(p, Now - beginTIME, CDbl(trials))
p = p + 1
beginTIME = Now
For i = 1 To trials
Call boomINTEGER
Next i
Call Finished(p, Now - beginTIME, CDbl(trials))
p = p + 1
beginTIME = Now
For i = 1 To trials
Call boomLONG
Next i
Call Finished(p, Now - beginTIME, CDbl(trials))
p = p + 1
beginTIME = Now
For i = 1 To trials
Call boomDOUBLE
Next i
Call Finished(p, Now - beginTIME, CDbl(trials))
p = p + 1
beginTIME = Now
For i = 1 To trials
Call boomUNDECLARED
Next i
Call Finished(p, Now - beginTIME, CDbl(trials))
p = p + 1
End Sub
Private Sub boomBYTE()
Dim a As Byte, b As Byte, c As Byte
a = 1
b = 1 + a
c = 1 + b
c = c + 1
End Sub
Private Sub boomINTEGER()
Dim a As Integer, b As Integer, c As Integer
a = 1
b = 1 + a
c = 1 + b
c = c + 1
End Sub
Private Sub boomLONG()
Dim a As Long, b As Long, c As Long
a = 1
b = 1 + a
c = 1 + b
c = c + 1
End Sub
Private Sub boomDOUBLE()
Dim a As Double, b As Double, c As Double
a = 1
b = 1 + a
c = 1 + b
c = c + 1
End Sub
Private Sub boomUNDECLARED()
a = 1
b = 1 + a
c = 1 + b
c = c + 1
End Sub
Private Sub Finished(i As Long, timeUSED As Double, trials As Double)
With Range("B2").Offset(i, 0)
.Value = .Value + trials
.Offset(0, 1).Value = .Offset(0, 1).Value + timeUSED
.Offset(0, 2).FormulaR1C1 = "=ROUND(RC[-1]*3600*24,0)"
End With
End Sub
As noted in other answers, the real difference between int and long is the size of its memory space and therefore the size of the number it can hold.
here is the full documentation on these datatypes
http://msdn.microsoft.com/en-us/library/office/ms474284(v=office.14).aspx
an Integer is 16 bits and can represent a value between -32,768 and 32,767
a Long is 32 bits and can represent -2,147,483,648 to 2,147,483,647
and there is a LongLong which is 64 bits and can handle like 9 pentilion
One of the most important things to remember on this is that datatypes differ by both language and operating system / platform. In your world of VBA a long is 32 bits, but in c# on a 64 bit processor a long is 64 bits. This can introduce significant confusion.
Although VBA does not have support for it, when you move to any other language in .net or java or other, I much prefer to use the system datatypes of int16, int32 and int64 which allows me to b much more transparent about the values that can be held in these datatypes.
VBA has a lot of historical baggage.
An Integer is 16 bits wide and was a good default numeric type back when 16 bit architecture/word sizes were prevalent.
A Long is 32 bits wide and (IMO) should be used wherever possible.
I have taken #PGSystemTester's method and updated it to remove some potential variability. By placing the loop in the routines, this removes the time taken to call the routine (which is a lot of time). I have also turned off screen updating to remove any delays this may cause.
Long still performed the best, and as these results are more closely limited to the impacts of the variable types alone, the magnitude of variation is worth noting.
My results (desktop, Windows 7, Excel 2010):
Code used:
Option Explicit
Sub VariableOlympics()
'Run this macro as many times as you'd like, with an activesheet ready for data
'in cells B2 to D6
Dim beginTIME As Double, trials As Long, i As Long, p As Long
Dim chosenWorksheet As Worksheet
Set chosenWorksheet = ThisWorkbook.Sheets("TimeTrialInfo")
Application.EnableEvents = False
Application.Calculation = xlCalculationManual
Application.ScreenUpdating = False
trials = 1000000000 ' 1,000,000,000 - not 10,000,000,000 as used by #PGSystemTester
p = 0
beginTIME = Now
boomBYTE trials
Finished p, Now - beginTIME, CDbl(trials), chosenWorksheet.Range("B2")
p = p + 1
beginTIME = Now
boomINTEGER trials
Finished p, Now - beginTIME, CDbl(trials), chosenWorksheet.Range("B2")
p = p + 1
beginTIME = Now
boomLONG trials
Finished p, Now - beginTIME, CDbl(trials), chosenWorksheet.Range("B2")
p = p + 1
beginTIME = Now
boomDOUBLE trials
Finished p, Now - beginTIME, CDbl(trials), chosenWorksheet.Range("B2")
p = p + 1
beginTIME = Now
boomUNDECLARED trials
Finished p, Now - beginTIME, CDbl(trials), chosenWorksheet.Range("B2")
p = p + 1
Application.EnableEvents = True
Application.Calculation = xlCalculationAutomatic
Application.ScreenUpdating = True
chosenWorksheet.Calculate
End Sub
Private Sub boomBYTE(numTrials As Long)
Dim a As Byte, b As Byte, c As Byte
Dim i As Long
For i = 1 To numTrials
a = 1
b = 1 + a
c = 1 + b
c = c + 1
Next i
End Sub
Private Sub boomINTEGER(numTrials As Long)
Dim a As Integer, b As Integer, c As Integer
Dim i As Long
For i = 1 To numTrials
a = 1
b = 1 + a
c = 1 + b
c = c + 1
Next i
End Sub
Private Sub boomLONG(numTrials As Long)
Dim a As Long, b As Long, c As Long
Dim i As Long
For i = 1 To numTrials
a = 1
b = 1 + a
c = 1 + b
c = c + 1
Next i
End Sub
Private Sub boomDOUBLE(numTrials As Long)
Dim a As Double, b As Double, c As Double
Dim i As Long
For i = 1 To numTrials
a = 1
b = 1 + a
c = 1 + b
c = c + 1
Next i
End Sub
Private Sub boomUNDECLARED(numTrials As Long)
Dim a As Variant, b As Variant, c As Variant
Dim i As Long
For i = 1 To numTrials
a = 1
b = 1 + a
c = 1 + b
c = c + 1
Next i
End Sub
Private Sub Finished(i As Long, timeUSED As Double, trials As Double, initialCell As Range)
With initialCell.Offset(i, 0)
.Value = trials
.Offset(0, 1).Value = timeUSED
.Offset(0, 2).FormulaR1C1 = "=ROUND(RC[-1]*3600*24,2)"
End With
End Sub
This is a space vs necessity problem.
In some situations it's a necessity to use a long. If you're looping through rows in a large excel file, the variable that holds the row number should be a long.
However, sometimes you will know that an integer can handle your problem and using a long would be a waste of space (memory). Individual variables really don't make much of a difference, but when you start dealing with arrays it can make a big difference.
In VBA7, Integers are 2 bytes and longs are 4 bytes
If you have an array of 1 million numbers between 1 and 10, using an Integer array would take up about 2MB of RAM, compared to roughly 4MB of RAM for a long array.
As others already mentioned, a Long may take twice as much space as an Integer. As others also already mentioned, the high capacity of current computers means you will see no difference in performance whatsoever, unless you are dealing with extra extra extra large amounts of data:
Memory
Considering 1 million values, the difference between using Integers versus Longs would be of 2 bytes for each value, so that is 2 * 1 000 000 / 1,024 / 1024 = less than 2 MB of difference in your RAM, which is likely much less than 1% or even 0.1% of your RAM capacity.
Processing
Considering the benchmark done by PGSystemTester's, you can see a difference of 811 - 745 = 66 seconds between Longs and Integers, when processing 10 billion batches of 4 operations each. Reduce the number to 1 million of operations and we can expect 66 / 10 000 / 4 = less than 2ms of difference in execution time.
I personally use Integers and Longs to help readability of my code, particularly in loops, where an Integer indicates the loop is expected to be small (less than 1000 iterations), whereas a Long tells me the loop is expected to be rather large (more than 1000).
Note this subjective threshold is way below the Integer upper limit, I use Longs just to make the distinction between my own definitions of small and large.
I know it is an old post, but I found it while browsing and I also wanted to share my findings:
after #PGSystemTester's method I got those
and after AJD's
Intel i5-8500T CPU
8gigs of RAM
64bit system and Win10Enterprise 21H1 OS build 19043.2006
while excel is on Version 2108 and Build 14326.20852
I also don't know if it was influencing but I also got Rubberduck Vers. 2.5.2.5906
but it seems like my Integer is faster in both cases
Ty AJD and pgSystemTester. I confirm AJD results for Integer 33 and Long 20. Yet Long is much much faster only because your test program is small enough to fit entirely in the processor cache memory. The fastest ram is L1 cache and that is only 32kB for data and 32kB for program. Testing Byte, Integer and Long for one or several arrays that barely fit into 32kB of L1 data cache will give you totally different or opposite results regarding the speeds.
In my case for the same arrays that totals 120kB for Integers and 240kB for Long i had the same result for Long as for Integer.
That is because changing to Long the same arrays totals double the size compared with Integer arrays, and so, more and more of data had fall outside L1 cache due to the change to Long. To reach data outside L1 cache took much more clocks or time.
Therefore your test is good only as a test yet in real life is misleading as msdn.microsoft recommendation to use Long regardless. Also those who emphasize that ram size is double for Long had not emphasize the consequence for the processor waiting time to reach data outside of L1 cache or even worst outside L2 cache or outside L3 cache. For each outside L1, L2 and L3 the time to reach the data will increase dramatically and this is most important for speed.
To summarize:
if your data fit inside the L1 cache then Long is fastest but
that is only 4k of data times 4Bytes for Long = 16kB (because other programs and OS will populate the rest of 32k of L1 cache),
Byte and Integer will drastically increase the speed for arrays in size
of at least 16kB, because changing to Long will increase size and this
will force more data to reside outside the fastest L1 cache ram.
Try the same test but instead of Dim a As Byte use Dim a() As Byte , example:
Dim a() As Byte, b() As Byte, c() As Byte
ReDim a(7, 24, 60), b(24, 7, 60), c(24, 60, 7)
Dim h As Long, loops As Long: Dim i As Long, j As Long, k As Long ' these i, j, k always As Long
loops=1
For h = 1 To loops
For i = 1 To 6: For j = 0 To 23: For k = 1 To 58
a(i, j, k) = a(i + 1, j, k): b(j, i, k) = b(j, i - 1, k)
c(j, k, i) = a(i - 1, j, k + 1) + b(j, i - 1, k - 1)
Next k: Next j: Next i
For i = 6 To 1 Step -1: For j = 23 To 0 Step -1: For k = 58 To 1 Step -1
a(i, j, k) = a(i + 1, j, k): b(j, i, k) = b(j, i - 1, k)
c(j, k, i) = a(i - 1, j, k + 1) + b(j, i - 1, k - 1)
Next k: Next j: Next i
Next h
First set "loops" to 1 to see how long it takes. Then increase it gradually aiming for several seconds for As Bytes. It will take longer for As Integer, and even longer for As Long...
The size of each of the 3 array is 8x25x61 = 12200 and this is
12200 kB times 3 = 36600 kB for As Byte ,
24400 kB times 3 = 73200 kB for As Integer ,
48800 kB times 3 = 146400 kB for As Long .
Run the same code with Dim a() As Integer, b() As Integer, c() As Integer, then the same with Dim a() As Long, b() As Long, c() As Long and so on.
Now if you increase one dimension 20 times you will expect a 20 increase in duration but it will be much more because now data will fall outside the L2 cache (1MB shared for all 4 cores).
If you increase one dimension 200 times you will expect a 200 increase in duration but it will be again much more because now data will fall outside the L3 cache (6-8MB shared for all 4 cores and the same 8MB for 8 cores or 16MB if ryzen 5800...).
I can't understand why after 20 years or more the L1 cache is only 64kB when it can be at least 16x16=256kB . With 16bits for row address and 16bits for column address you have to read only 32bits and that is one read for 32 bits processor. I suspect that is because perhaps the core still works on 16bits (8 for row + 8 for column address, 8x8=64kB) or worst on only 8bits.
After testing please post your results.

Overflow when processing historical price data (VBA)

I'm writing a macro that reads price data from an excel file, write them on another excel file and does basic calculations (e.g. returns). The data is made of 2600 x 100 data points.
When I run it i have an overflow error! looks like it's handling too much data. Is it an issue with the computer's memory? (i have a modern laptop with 4GB Ram) or does it have to do with the way the data is stored during the calulations? What I don't get is that if I try to do that kind of calculations in the spreadsheet directly, i won't have an overflow message.
Btw, I'm trying to avoid using Access.
Thanks for your help.
Sam
Dim i As Long
Dim j As Long
Dim k As Long
Dim m As Long
Dim n As Long
Dim o As Long
' FYI yearsconsidered = 10
For j = 1 To Group(k).TickerCount
For m = 1 To Quotes(j, k).ContinuationCount
For i = 1 To YearsConsidered * 261
If Sheets(j).Cells(i - GivenPeriod, 4 * m - 3) = 0 Then
Sheets(j).Cells(i, 4 * m - 2) = 0
Else
Sheets(j).Cells(i, 4 * m - 2) = (Sheets(j).Cells(i, 4 * m - 3) - Sheets(j).Cells(i - GivenPeriod, 4 * m - 3)) / Sheets(j).Cells(i - GivenPeriod, 4 * m - 3)
End If
Next i
Next m
Next j
I have experienced a similar situation when I'm looping through a large number of data points. For some reason, this does not happen if I code the formula entry into a range of cells (at the same time); like when you manually select a range in the spreadsheet, enter the formula once, then hit Ctrl+Enter. This will require for the proper use of the $ signs (absolute vs relational positions).
It's hard to tell exactly what the issue is without seeing the code, so not sure if this is your problem...

Trouble with Dynamic Arrays

As a noobie to VBA, I am having a Hell of a time understanding how arrays, dynamic arrays in specific, work. I am an Industrial Engineering student, so most of my coding has been done with MatLab.
My question is: Why do I keep getting Run-time error '9' "Subscript out of range" for the following code:
Sub FunWithArrays()
Dim DynArray()
Dim i As Integer
Dim j As Integer
j = 1
For i = 1 To 10
DynArray(j) = i
j = j + 1
ReDim DynArray(UBound(DynArray) + 1)
MsgBox DynArray(i)
Next
End Sub
Any and all help is appreciated!
As people in the comments have mentioned, dynamic arrays are just that: dynamic. That is to say that if you declare an array without dimension, as you have here with Dim DynArray() then it doesn't at this point have any "slots" to store anything.
When an array is declared this way, you then need to Redim it to tell it how many slots you want it to have (presumably after determining this from some other data or user input).
The full code for this would be:
Sub FunWithArrays()
Dim DynArray()
Dim i As Integer
Dim j As Integer
j = 1
Redim DynArray(10)
' redimensioned the array to hold 10 items (since
' you've set the loop as i=1 to 10 you must already
' know that the array wants to hold 10 things)
For i = 1 To 10
'ReDim Preserve DynArray(i)
' this is how you could do it if you didn't know in
' advance that you were going to loop 10 times,
' but as somebody else said, this is a slow way to do it.
DynArray(i) = j
j = j + 1
MsgBox DynArray(i)
' Generally instead of a messagebox here I would use:
' Debug.Print DynArray(i)
' this then prints out the value (on a new line each time)
' into the immediate window which means you can see the
' output without having to click "OK" each time.
Next
End Sub
You can also redimension the array within the loop, or later on, to hold a different number of items, but if you want to keep the items already stored in the array you must use ReDim:
ReDim Preserve DynArray(i)
It's also good practice in general to declare the array type. When you use Dim DynArray() this creates an array of type Variant which can hold anything (String, Double, Object, etc.). If you explicitly declare the type e.g. Dim DynArray() as Integer then it'll only allocate enough memory to hold integers, which is more efficient. For many applications the difference in speed will not make a difference, but when you're dealing with looping through something thousands of times it can matter.