compare huge text files using vba - vba

Question

I gotta serious problem here.. any kind of help is much appreciated!!
I have two huge text files (130 MB)each with thousands of records in each. I need to compare the two files using vba or by any means and generate a spreadsheet which includes the header and with two additional columns. The two additional columns will be the file name and in the next column it should display in which particular column is error. Each record will be having multiple discrepancies. One file can have the records which cannot be found in the other file. So this condition should also be recorded in the spreadsheet.
Example:
Media Events: Taking one record from each.
00000018063|112295|000|**0009**|
PROL:
00000018063|112295|000|**0013**|
In the above example, the records are from two files. The highlighted ones are the differences between the records. So the output should be like this..
HH_NUMBER | CLASS_DATE | MV_MIN DURATION File Mismatc Mismatch Reason
00000018063 | 112295 | 000 **0009** Media Events Mismatches in DURATION
00000018063 | 112295 | 000 **0013** PROL Mismatches in DURATION
00000011861 | 112295 | 002 0126 Media Events missing in PROL file

Answer 1

It seems there are three problems here:
1) Find matching records (first column) between two files.
2) Compare records that match on the first column - if there is a difference, record what the difference is
3) If a record exists in one file but not the other, record that.
I am going to assume that the two "huge files" are in fact separate sheets in the same excel workbook, and that the records are sorted on the first key. This will speed up processing significantly. But speed is a secondary concern, I assume. I also assume there is a third sheet where you put the output.
Here is an outline of VBA code - you will have to do a bit of work to get it "just right" for your application, but I hope this gets you going.
Sub compare()
Dim s1 as Worksheet
Dim s2 as Worksheet
Dim col1 as Range
Dim col2 as Range
Dim c as Range
Dim record1 As Range, record2 As Range, output As Range
Dim m
Dim numCols as Integer
numCols = 5 ' however many columns you want to compare over
Set s1 = Sheets("Media")
Set s2 = Sheets("Pro")
Set output = Sheets("output").Range("A2")
Application.ScreenUpdating = False
s1.Select
Set col1 = Range("A2", [A2].End(xlDown));
s2.Select
Set col2 = Range("A2", [A2].End(xlDown));
On Error Resume Next
For Each c in col1.Cells
m = Application.Match(c.Value, col2, 0);
If isError(m) Then
' you found a record in 1 but not 2
' record this in your output sheet
output.Value = "Record " & c.Value & " does not exist in Pro"
Set output = output.Offset(1,0) ' next time you write output it will be in the next line
' you will have to do the same thing in the other direction - test all values
' in 2 against 1 to see if any records exist in 2 that don't exist in 1
Else
' you found matching records
Set record1 = Range(c, c.offset(0, numCols))
Set record2 = Range(col2.Cells(m,1), col2.Cells(m,numCols))
' now you call another function to compare these records and record the result
' using the same trick as above to "go to the next line" - using output.Offset(1,0)
End If
Next c
End Sub

Answer 2

You could do this with formulas:
See
MS KB: Use Excel to compare two lists of data
Me Excel.com - Creating a list of non-matching values
ExcelExperts.com - Extracting non-matching entries from two columns in a third column
To give you an idea, basically, if you have two lists in columns A & B, you could use formulas like below in columns C and D to show the matching or non-matching:
In C1,
=If(isna(match(A1,B:B,0)),A1,"")
and, in D1
=IF(Isna(Match(B1,A:A,0)),B1,"")
both copied down.
FURTHER READING:
Excel Index Function and Match Function - Contextures MVP
Excel VLOOKUP and Index & Match - Excel User MVP
Excel User MVP - Excel’s Best Lookup Method: INDEX-MATCH