VBA: Big Data and the use of Arrays - vba

Question

So I'm working with large 28,000 line plus data+. Plus possibly 5 other spreadsheets to cross reference against.
I keep being told Arrays are faster but can it be explained to me it seems they are faster where you can read and write large chunks of data into the array at a time. Which is something where I can understand where there might be a speed overhead reduction.
Or is it right to say Arrays are just plain faster than say....
Worksheet.range("A1").Value=AOtherWorksheet.range("A1").Value
It just appears somewhat magical if that's the case as could get why reading in blocks of variants would be faster but don't necessarily get why reading off a sheet into a array and then off array into second sheet would be faster. Have I misunderstood I'm just trying to tease that specific part out.
Any other tricks comments for automating large spreadsheets welcome but was mainly focused on understanding this titbit.

Answer 1

I think the magic is caused by complexity - each cell carries with it a lot of "baggage"
Hundreds of settings for its environment, and most of them are about cell formatting
Height, Width, RowHeight, ColumnWidth, Column, Row, Font, IndentLevel, etc
To see all properties, observe the watch window for Sheet1.Range("A1")
(properties with a + next to them are complex objects with their own set of properties)
The main reason for optimizing with arrays is to avoid all formatting
Each cell "knows" about all settings regardless if they are changed or not, and carries all this "weight" around. Most users, most of the times only care about the value in the cell, and never touch the formatting. In rare occasions you may be stuck working directly with the range object if you need to modify each individual cell's .Borders, .Interior.Color, .Font, etc, and even then, there are ways of grouping similarly formatted cells and modifying the attributes of the entire group at once
.
To continue with the baggage analogy (and this is stretching it a bit): at an airport, if I need to refill a pen for passenger "John Doe" from his luggage already on the plane, in a utility room at the back of the airport, I will be able to do it (I have all the info I need), but it'll take me time to go back and forth, carrying that luggage. And for one passenger it can be done in a reasonable amount of time, but how much longer would it take to refill 20K pens, or 100K, or a million ? (ONE - BY - ONE)
I view the Range <-> VBA interaction the same way: working with individual cells one at the time, is like carrying each individual luggage for a million passengers, to the utility room at the back of the airport. This is what this statement does:
Sheet1.Range("A1:A1048576").Value = Sheet2.Range("A1:A1048576").Value
as opposed to extracting all pens, from all suitcases at once, refilling them, and placing them all back
.
Copying the range object to an array is isolating one of the properties for each cell - its Value ("the pen"), from all the others settings (Excel is extremely efficient about this). We now have an array of only the values, and no other formatting settings. Modify each value individually in memory, then place them all back into the range object:
Dim arr as Variant
arr = Sheet2.Range("A1:A1048576") 'Get all values from Sheet2 into Sheet1
Sheet1.Range("A1:A1048576") = arr
.
This is where the Copy / Paste parameters are different as well:
Sheet2.Range("A1:A1048576").Copy
Sheet1.Range("A1:A1048576").PasteSpecial xlPasteAll
.
Timers for Rows: 1,048,573
xlPasteAll - Time: 0.629 sec; (all values + all formatting)
xlPasteAllExceptBorders - Time: 0.791 sec
xlPasteAllMergingConditionalFormats - Time: 0.782 sec; (no merged cells)
xlPasteAllUsingSourceTheme - Time: 0.791 sec
xlPasteColumnWidths - Time: 0.004 sec
xlPasteComments - Time: 0.000 sec; (comments test is too slow)
xlPasteFormats - Time: 0.497 sec; (format only, no values, no brdrs)
xlPasteFormulas - Time: 0.718 sec
xlPasteFormulasAndNumberFormats - Time: 0.775 sec
xlPasteValidation - Time: 0.000 sec
xlPasteValues - Time: 0.770 sec; (conversion from formula to val)
xlPasteValuesAndNumberFormats - Time: 0.634 sec
.
Another aspect, beyond arrays, are the types of indexes for data structures
For most situations arrays are acceptable, but when there is a need for better performance, there are the Dictionary and Collection objects
One array inefficiency is that for finding elements we need to iterate over each one
A more convenient option could be to access specific items, a lot faster
Dim d As Object 'Not indexed (similar to linked lists)
Set d = CreateObject("Scripting.Dictionary") 'Native to VB Script, not VBA
d.Add Key:="John Doe", Item:="31" 'John Doe - 31 (age); index is based on Key
d.Add Key:="Jane Doe", Item:="33"
Debug.Print d("Jane Doe") 'Prints 33
Dictionaries also have the very useful and fast method of checking items d.Exists("John Doe"), which returns True or False without errors (but Collections don't). With an array you'd have to loop over potentially all items to find out
I think one of the fastest ways to extract unique values for large columns is to combine arrays and dictionaries
Public Sub ShowUniques()
With Sheet1.UsedRange
GetUniques .Columns("A"), .Columns("B")
End With
End Sub
Public Sub GetUniques(ByRef dupesCol As Range, uniquesCol As Range)
Dim arr As Variant, d As Dictionary, i As Long, itm As Variant
arr = dupesCol
Set d = CreateObject("Scripting.Dictionary")
For i = 1 To UBound(arr)
d(arr(i, 1)) = 0 'Shortcut to add new items to dictionary, ignoring dupes
Next
uniquesCol.Resize(d.Count) = Application.Transpose(d.Keys)
'Or - Place d itms in new array (resized accordingly), and place array back on Range
' ReDim arr(1 To d.Count, 1 To 1)
' i = 1
' For Each itm In d
' arr(i, 1) = itm
' i = i + 1
' Next
' uniquesCol.Resize(d.Count) = arr
End Sub
From: Col A To: Col B
1 1
2 2
1 3
3
Dictionaries don't accept duplicate keys, just ignores them