Removing stray CRLF embedded in a CSV file column - vb.net

Question

I need some direction on how to solve a problem I am working on. The root issue is that I need to work with CSV files in another program. The source system that creates the CSV files does not strip out CRLF in any of the data fields that get exported (meaning some fields have an embedded CRLF). As a result I receive a CSV file that has malformed rows in it. My end goal is an utility that will
check the first column of each row (which if correct is a GUID with a length of 36, or
count the columns in each row (which is the example below).
In the example below I am looking at the column count. If the correct count is 18 then I want it to write that row to a new file. If the column count is not correct I want to remove the CRLF from that row until the column count is correct.
Again, two ways to solve the issue that I know of:
Check the length of the first column for a length of 36 (before the first comma and excluding the first row which is the title row), or
count the columns and remove any trailing CRLF until the column count is equal to 18 (the total column count).
My issue with the code so far is being able to write out a valid row to a new file. Currently it writes out System.String[] instead of the actual row.
Public Class Form1
Private Sub btnFixit_Click(sender As Object, e As EventArgs) Handles btnFixit.Click
Dim iBadRowNumber As Integer = vbNull
Dim strFixedFile As System.IO.StreamWriter = My.Computer.FileSystem.OpenTextFileWriter(Me.txtFixedFile.Text, True)
Using MyReader As New Microsoft.VisualBasic.FileIO.TextFieldParser(Me.txtBaselineFileToProcess.Text)
MyReader.TextFieldType = FileIO.FieldType.Delimited
MyReader.SetDelimiters(",")
Dim currentRow As String()
While Not MyReader.EndOfData
Try
currentRow = MyReader.ReadFields()
If currentRow.Count = 18 Then
strFixedFile.WriteLine(currentRow)
Else
' Future code here to fix the line
End If
Catch ex As Microsoft.VisualBasic.FileIO.MalformedLineException
MsgBox("Line " & ex.Message &
"is not valid and will be skipped.")
End Try
End While
End Using
strFixedFile.Close()
End Sub
End Class
Here is an example of 2 correct rows with one incorrect row in the middle. In this example the row beginning with Sometown is really part of the prior row. I have also seen that one true row may be broken into three or more partial rows like similar to what you see in the Sometown row.
CustomerId,CustomerName,Status,Type,CustomerNumber,DBA,Address1,Address2,City,State,ZipCode,WebAddr,EMail,SalesCode,ServiceCode,DivisionCode,BranchCode,DepartmentCode
6d0125cd-70cf-4048-9ee1-8d9682e426a5,"Smith,James",Active,Customer,8,,103 Long Dr,,AnotherTown,NJ,000000,,,!!S,!%9,!!#,!!#,"!""."
35ed375c-c226-4879-a789-469cae63383c,"Doe, John",Active,Customer,55281,,28 Short Drive,,
Sometown,CA,12345,,
email#domain.com,"!$,",!$^,!!#,!!#,!!K
a5972bce-408f-4def-b77c-4ae0148dd045,"Duck,Donald",Active,Customer,25,,236 North Main St,,Mytown,PA,11111,,,!!2,!%9,!!#,!!#,"!""."
There may be much more elegant ways to perform the specific task. I am open either to corrections to my logic above or a totally different way to solve the problem either in VB.net or PowerShell.

Answer 1

Normally, csv can have multiline fields without a problem. But those need to be surrounded with quotes.
In your example this doesn't seem the case, but on the other hand there is no multiline field either, the field with value Sometown starts at a new line. So I wonder if this is the original data.
In case your multiline fields are surrounded with quotes you need to inform your parser about it.
But even with the single lines you will have problems caused by the fields with a seperator inside. Luckily those are quoted (as they should be), so you need to set the TextFieldParser.HasFieldsEnclosedInQuotes property as wel.
Now, if your multiline fields happen to be quoted (as they should be), the above setting should solve everything.
Update
You could do something like this:
currentRow = MyReader.ReadFields()
If currentRow.Count = 18 Then
strFixedFile.WriteLine(currentRow)
Else
'Write current row without newline
'Read next line/row
'WriteLine this row
End If
But you'll have to take care of fields like "Smith,James" with a seperator inside. Make sure your parser handles quoted fields properly (see above).

Answer 2

The most straightforward approach would probably be a variation of your first validation check:
Read the file line-by-line and keep both the current and the previous line in a buffer.
Check if the beginning of the line is a proper GUID (e.g. with a regular expression).
If the current line does not start with a GUID, append it to the previous line.
Otherwise write the previous line to the output file unless it's empty, then replace it with the current line.
I don't know VB.net, but in PowerShell it would look somewhat like this:
$reader = New-Object IO.StreamReader ('C:\path\to\input.csv')
$writer = New-Object IO.StreamWriter ('C:\path\to\output.csv', $false)
$writer.WriteLine($reader.ReadLine()) # copy CSV header
$output = '' # output buffer
$current = '' # pre-buffered current line from input file
while ($reader.Peek() -ge 0) {
# read line into pre-buffer
$current = $reader.ReadLine()
$hasGUID = $current -match '^[a-f0-9]{8}(-[a-f0-9]{4}){3}-[a-f0-9]{12},'
# append line to output buffer if it doesn't have a GUID, otherwise
# write the output buffer to file if it contains data and move the
# current line to the output buffer
if (-not $hasGUID) {
$output += $current
} else {
if ($output) { $writer.WriteLine($output) }
$output = $current
}
}
# write remaining pre-buffered line (if there is one)
if ($current -and $hasGUID) { $writer.WriteLine($current) }
$reader.Close(); $reader.Dispose()
$writer.Close(); $writer.Dispose()