Merge many CSV files - vba

Question

I have set of 500 csv files. Each file has four columns and variable number of rows.
I want to merge all of these csv's into one common sheet. If someone can help me in doing this in PowerShell, it would be great.
Sample Data in Excel 1:
Name Age Marks Class
A 15 100 5
B 20 88 6
Sample Data in Excel 2:
Name Age Marks Class
C 11 99 2
Output :
Name Age Marks Class
A 15 100 5
B 20 88 6
C 11 99 2

Answer 1

If all the CSV files are in one folder then:
$res = #()
ls *.csv | %{
$temp = Import-CSV $_
$res += $temp
}
$res | Export-CSV .\ALLINFO.csv -NoTypeInformation
The break down:
$res = #() - Make an array called $res that will hold all the data. This isn't strictly required. You could do it in a way that appends to a result file directly.
ls *.csv | - Find all the CSV files in the folder and pass them to the next command.
%{$temp = Import-CSV $_; $res += $temp} - Take each of those files, import the CSV data into a holder variable called $temp. Add the contents of $temp to the collector variable $res. Again it is not necessary to use the intermediate $tamp variable, I just find it more clear to do so.
$res | Export-CSV .\ALLINFO.csv -NoTypeInformation - Now that the data from all the files is in $res, export $res to a new file.

Answer 2

If the files are large then you could merge them as text documents. This is a lot faster than importing csv-objects, but it requires that the properties and the order in which they're placed are equal in all files. Example:
$files = Get-ChildItem "*.csv"
#Get header
$text = #(Get-Content -Path $files[0].FullName -TotalCount 1)
$files | ForEach-Object {
#Get text but skip header
$text += Get-Content -Path $_.FullName | Select-Object -Skip 1
}
#Save merged csv
$text | Set-Content Output.csv
Output.csv
Name;Age;Marks;Class
A;15;100;5
B;20;88;6
C;11;99;2
You could optimize it even more by replacing Get-Content for [System.IO.File]::ReadAllLines() etc. but I skipped that now as it's more complicated/hard to read.
UPDATE: Added alternative solution that saves the output-file part for part as Ansgar suggested.
$outputfile = "Output.csv"
$files = Get-ChildItem "*.csv"
#Get header
Get-Content -Path $files[0].FullName -TotalCount 1 | Set-Content -Path $outputfile
$files | ForEach-Object {
#Get text but skip header
Get-Content -Path $_.FullName | Select-Object -Skip 1
} | Add-Content -Path $outputfile

Answer 3

In your case, the sort name is optional depending on whether the merge should also reorder the contents (obviously, you can sort on a different parameter as well). Same stipulation as above - all .csv files in one directory.
dir c:\directory_containing_your\*.csv | Import-Csv | sort name | Export-Csv -Path c:\output.csv -NoTypeInformation
From the ScriptingGuy.

Answer 4

Here's a heavily-commented solution that uses VBA in Excel to combine the CSVs. The strategy here is this:
Set your references up-front, most importantly the strDir variable (which is a string representing the directory that holds all your CSVs)
Loop through the directory
Open each CSV
Copy the appropriate contents from each CSV
Paste the contents to the output workbook
Repeat the loop until all files have been iterated over
Hope this helps!
Option Explicit
Public Sub CombineCSVsInFolder()
Dim strFile As String, strDir As String
Dim wbkSource As Workbook, wbkOutput As Workbook
Dim wksSource As Worksheet, wksOutput As Worksheet
Dim lngLastRowSource As Long, lngLastRowOutput As Long
Dim rngSource As Range, rngOutput As Range
Dim blnFirst As Boolean
'Set references up-front
strDir = "c:\stack\my_csvs\" '<~ edit this line with the CSV directory
strFile = Dir(strDir)
blnFirst = True
Set wbkOutput = Workbooks.Add
Set wksOutput = wbkOutput.ActiveSheet
Application.ScreenUpdating = False
'Loop through the CSV directory
While (strFile <> "")
'Assign source CSV files
Set wbkSource = Workbooks.Open(strDir & strFile)
Set wksSource = wbkSource.ActiveSheet
'Assign boundaries of area to copy and output
lngLastRowSource = LastRowNum(wksSource)
lngLastRowOutput = LastRowNum(wksOutput)
With wksOutput
Set rngOutput = .Cells(lngLastRowOutput + 1, 1)
End With
'If this is the first time through, include headers, otherwise do not
If blnFirst = False Then
With wksSource
Set rngSource = .Range(.Cells(2, 1), .Cells(lngLastRowSource, 4))
End With
'Special case for first iteration to correct source and output ranges
Else
With wksSource
Set rngSource = .Range(.Cells(1, 1), .Cells(lngLastRowSource, 4))
End With
With wksOutput
Set rngOutput = .Cells(1, 1)
End With
blnFirst = False
End If
'Execute copy, close source and repeat
rngSource.Copy rngOutput
wbkSource.Close
strFile = Dir
Wend
'Turn screen updates back on
Application.ScreenUpdating = True
End Sub
'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
'INPUT : Sheet, the worksheet we'll search to find the last row
'OUTPUT : Long, the last occupied row
'SPECIAL CASE: if Sheet is empty, return 1
Public Function LastRowNum(Sheet As Worksheet) As Long
If Application.WorksheetFunction.CountA(Sheet.Cells) <> 0 Then
LastRowNum = Sheet.Cells.Find(What:="*", _
LookIn:=xlFormulas, _
SearchOrder:=xlByRows, _
SearchDirection:=xlPrevious).Row
Else
LastRowNum = 1
End If
End Function