Cosolidating large excel files, cant go around buffer overflow - vba

Question

I am trying to consolidate multiple large excel files into one single file using the following code
Sub Macro1()
Application.DisplayAlerts = False
Dim Country As String
Dim i As Integer
Dim j As Integer
Dim k As Integer
k = 2
For i = 1 To 50
Windows("Try2").Activate
Country = Worksheets("Names").Cells(i, 1).Value
Workbooks.Open Filename:= "C:path\" & Country & " "
ActiveWorkbook.Sheets("Main").Activate
finalrow = Cells(Rows.Count, 1).End(xlUp).Row
Workbooks(Country).Sheets("Main").Range(Cells(1, 1), Cells(10000, 64)).Copy
Workbooks("Try2").Sheets("Output").Activate
Workbooks("Try2").Sheets("Output").Cells(k, 2).PasteSpecial xlPasteValues
Range(Cells(k, 1), Cells(k + 10000, 1)) = Country
finalrow2 = 10002 + k
k = finalrow2 + 1
Workbooks(Country).Sheets("Main").Activate
Workbooks(Country).Close SaveChanges:=False
Next i
End Sub
However after consolidating 2-3 files excel it is throwing buffer overflow error. We have around 50 files. I am trying to figure out if this is just a problem with excel unable to handle large files or there is something wrong with my code. Is there any way to guide excel to handle larger files sizes?

Answer 1

There's a couple of things here that could be throwing your macro off.
First, you're declaring i, j, and k as the 'integer' data type; problem with that is that the largest possible value an integer can hold is 32,767. If you go above that, you're going to get a buffer overflow.
Second, you're declaring variables that you don't use, and not declaring variables that you do use. As far as I can see, j isn't used anywhere in the code, but finalrow and finalrow2 are used but aren't declared anywhere.
I would suggest you replace your integer data types that could go very high with long data types (long integer). You might also want to use more descriptive names for your integers to make your code easier to read:
Dim iCycler as integer
Dim kStart as long
Dim FinalRow as Long
Dim FinalRow2 as Long
Run through that and see what you get.
EDITED TO ADD:
Here is a complete process that should suit you:
Sub Duplicator()
'Define the source file, sheet, and range
Dim wbkSource As Workbook
Dim shtSource As Worksheet
Dim rngSource As Range
'Define the target file, sheet and range
Dim wbkTarget As Workbook
Dim shtTarget As Worksheet
Dim rngTarget As Range
'Define the sheet with the list of countries
Dim shtControl As Worksheet
'Prepare control integers
Dim iLoop As Integer
Dim lLastRow As Long
'Define the target file as the active workbook
Set wbkTarget = ActiveWorkbook
Set shtTarget = wbkSource.Sheets("Output")
Set rngTarget = shtTarget.Range("A2")
Set shtControl = wbkTarget.Sheets("Names")
'Loop through the list
For iLoop = 1 To 50
'Open the source file and assign it to a variable.
Set wbkSource = Workbooks.Open("C:\path\" & shtControl.Cells(iLoop, 1).Value)
'Assign the source sheet
Set shtSource = wbkSource.Sheets("Main")
'Find the last row of data
lLastRow = shtSource.Range("A" & Rows.Count).End(xlUp).Row
'Use the last row to build a source range variable
Set rngSource = shtSource.Range("A1", "BL" & lLastRow)
'Check that there is space for the copy. If there is not, create a new sheet for the new data
If rngTarget.Row + lLastRow > shtTarget.Rows.Count Then
Set shtTarget = wbkTarget.Sheets.Add
shtTarget.Name = "Output 2"
Set rngTarget = shtTarget.Range("A2")
End If
'Use the size of rngSource to define the size of the target range
Set rngTarget = rngTarget.Resize(rngSource.Rows.Count, rngSource.Columns.Count)
'Duplicate the values over
rngTarget.Value = rngSource.Value
'Prepare the target range for the next loop
Set rngTarget = shtTarget.Range("A" & Rows.Count).End(xlUp).Offset(1, 0)
'Close the source file
wbkSource.Close False
Next iLoop
End Sub
By declaring and using variables throughout, the code should be easier both to write and to read. It should also run quicker, since it's using the range1.value = range2.value structure, which bypasses the rather slow clipboard. It also contains a check to make sure that you're not surpassing 1,048,576 rows of data, which would cause a crash.

Answer 2

quite along the lines of good coding practice Werff already explained to you, you could try this other (commented) code:
Sub Macro1()
Dim outputSht As Worksheet '<--| declare a variable to set your "output" sheet to
Dim countryData As Variant, countryNames As Variant '<--| declare arrays to store "country names" and "country data" in
Dim country As Variant '<-- "countries" looping variable
Application.Calculation = xlCalculationManual '<-- disable calculations
Application.ScreenUpdating = False '<-- disable screen updating
With Workbooks("Try").Worksheets("Names") '<--| reference "country names" worksheet
countryNames = Application.Transpose(.Range("A1", .Cells(.Rows.count, 1).End(xlUp)).Value) '<--| store country names in column "A" from row 1 down to last not empty row
End With
Set outputSht = Workbooks("Try").Worksheets("Output") '<--| set "output" worksheet
For Each country In countryNames '<-- loop through countries stored in 'countryNames'
With Workbooks.Open(FileName:="C:path\" & Country).Sheets("Main") '<--| open current country workbook and reference its "Main" sheet
countryData = .Range("BL1", .Cells(.Rows.count, 1).End(xlUp)).Value '<--| store current country data in 'countryData' array
.Parent.Close SaveChanges:=False '<--| close current country workbook
End With
With outputSht '<--| reference output sheet
With .Cells(.Rows.count, 1).End(xlUp).Offset(1).Resize(UBound(countryData, 1)) '<--|reference its column A range from first empty cell after last not empty cell down to as many rows as current country array has
.Value = country '<--| write current country name in referenced range
.Offset(, 1).Resize(, 64).Value = countryData '<--| write country data array content from column B rightwards
End With
End With
Next country
Application.Calculation = xlCalculationAutomatic
Application.ScreenUpdating = True
End Sub
since it uses arrays to store data, it suffers from the arrays maximum size limitation to 65536 rows. If your "countries" workbooks "Main" sheets have more rows than such a limit then different pasting values techniques must be adopted (like between range values)