Insert 10000+ rows in an Excel sheet - vba

Question

I am creating an Excel macro while pulls data from an Excel sheet (containing more than 10 thousand rows) and populates another Excel sheet.
I have to insert data row-wise since for every row I need to fetch data from 2-3 different sheets based on a column value (say, EMP_ID).
e.g. Excel file - 1 has
Emp_ID | Emp_Name | Age
Now based on each employee ID I need to fetch employee related data from 3 other Excel sheets. So I have to loop through 10k records.
When I execute the code the Excel application just hangs. I think this is because I try to insert data row-wise.
Can someone suggest a faster way to insert/update large number of rows.
I have already tried using Variant/Array to store data and then populate the sheets. But it still doesn't seem to work.
NOTE: I am reading records from the Source file into a RecordSet.
I have already added:
Application.DisplayAlerts = False,
Application.ScreenUpdating = False,
Application.Calculation = xlCalculationManual
And then set it back to default.

Answer 1

I know this is not a direct answer, but sometimes it's better to teach how the work has to be done, intead of simply reply.
Your work needs to be done with Access (or any other dbms). You have to define three tables, each indexed by Emp_ID, and all the staff become simple.

Answer 2

I tend to agree with Sergio.
If using a database is totally not an option, using an array is the way to go.
I have already tried using Variant/Array to store data and then populate the sheets. But it still doesn't seem to work.
Can you show the code you tried?
This works for me:
Dim arData() As Variant
' ... calculate number of rows and columns ...
ReDim arData(1 To numRows, 1 To numCols)
' ... populate arData ...
' Define range with identical dimensions as arData, e.g. insert in second row
Set rng = sh.Range(sh.Cells(2, 1), sh.Cells(numRows + 1, numCols))
' Transfer array to range (this is fast!)
rng.Value = arData

Answer 3

Also agree, Excel isn't really the tool for this.
If you're stuck with it then try the following:
Read all lookup sheets just once into collections using class objects as your data structures. For example, create a class called Employee and add the appropriate properties.
Public ID As Long
Public Age As Integer
Public Name As String
To read them, you'd code it like this...
Private mEmployeeCol As Collection
Dim ws As Worksheet
Dim empData As Employee
Dim v As Variant
Dim r As Long
Set ws = ThisWorkbook.Worksheets("employee stuff")
v = ws.UsedRange.Value2
Set mEmployeeCol = New Collection
For r = LBound(v, 1) To UBound(v, 1)
Set empData = New Employee
empData.ID = v(r, 1)
empData.Name = v(r, 2)
empData.Age = v(r, 3)
mEmployeeCol.Add empData, Key:=CStr(empData.ID)
Next
To look up the values, do it so ...
Set empData = mEmployeeCol(CStr(ID))
v(r, [your col]) = empData.ID
Then, definitely, DEFINITELY populate the final sheet with an array of variants. It's pretty straight forward ...
Dim v(1 To 10000, 1 To 50) As Variant
ws.Range("A1").Resize(UBound(v, 1), UBound(v, 2)).Value = v