Well, I have successfully coded my unformatted text data. I used comma as a delimiter and if I am posting it here so that it can help students like me.
Try
Using Reader As New TextFieldParser(Application.StartupPath & "\data.txt")
Reader.TextFieldType = FieldType.FixedWidth
Reader.SetFieldWidths(20, 20, 1, 2)
Dim currentRow As String()
While Not Reader.EndOfData
Try
currentRow = Reader.ReadFields()
Dim oWrite As New System.IO.StreamWriter(Application.StartupPath & "\FormattedData.txt")
For Each newString In currentRow
oWrite.Write(newString & ",")
Next
oWrite.WriteLine()
oWrite.Close()
Catch ex As Exception
End Try
End While
End Using
Catch ex As Exception
End Try
Now I have a question too: How can I omit the last comma from the output while importing to access database?
The easiest way to do this (that I have found) is to write the comma for the last record on the next pass, but don't do it on the first loop:
Dim fFirstEntry As Boolean = True
For Each newString In currentRow
If fFirstEntry Then
fFirstEntry = False
Else
oWrite.Write(",")
End If
oWrite.Write(newString)
Next
Dim sb As New StringBuilder()
For Each newString In currentRow
sb.Append(newString).Append(",")
Next
sb.Length -= 1
oWrite.Write(sb.ToString())
Note that the StringBuilder is more efficient than direct string manipulation. The latter generates a lot of garbage on the heap, which has to be collected by the garbage collector later.
Related
Like many on here, I am new to programming and mainly focus on web development. I have written a program cobbled together from help on here that works perfectly. I take a CSV file and inject it into an SQL database. I am getting a "MalformedLineException" line exception on the last line of the CSV file and believe it is because the header line is not being skipped.
Would love some help on working out how to skip the first line from my code below:
Private Sub subProcessFile(ByVal strFileName As String)
'This is the file location for the CSV File
Using TextFileReader As New Microsoft.VisualBasic.FileIO.TextFieldParser(strFileName)
'removing the delimiter
TextFileReader.TextFieldType = Microsoft.VisualBasic.FileIO.FieldType.Delimited
TextFileReader.SetDelimiters(",")
ProgressBar1.Value = 0
Application.DoEvents()
'variables
Dim TextFileTable As DataTable = Nothing
Dim Column As DataColumn
Dim Row As DataRow
Dim UpperBound As Int32
Dim ColumnCount As Int32
Dim CurrentRow As String()
'Loop To read in data from CSV
While Not TextFileReader.EndOfData
Try
CurrentRow = TextFileReader.ReadFields()
If Not CurrentRow Is Nothing Then
''# Check if DataTable has been created
If TextFileTable Is Nothing Then
TextFileTable = New DataTable("TextFileTable")
''# Get number of columns
UpperBound = CurrentRow.GetUpperBound(0)
''# Create new DataTable
For ColumnCount = 0 To UpperBound
Column = New DataColumn()
Column.DataType = System.Type.GetType("System.String")
Column.ColumnName = "Column" & ColumnCount
Column.Caption = "Column" & ColumnCount
Column.ReadOnly = True
Column.Unique = False
TextFileTable.Columns.Add(Column)
ProgressBar1.Value = 25
Application.DoEvents()
Next
clsDeletePipeLineData.main()
End If
Row = TextFileTable.NewRow
'Dim Rownum As Double = Row
'If Rownum >= 1715 Then
' MsgBox(Row)
'End If
For ColumnCount = 0 To UpperBound
Row("Column" & ColumnCount) = CurrentRow(ColumnCount).ToString
Next
TextFileTable.Rows.Add(Row)
clsInsertPipeLineData.main(CurrentRow(0).ToString, CurrentRow(1).ToString, CurrentRow(2).ToString, CurrentRow(3).ToString, CurrentRow(4).ToString, CurrentRow(5).ToString, CurrentRow(6).ToString, CurrentRow(7).ToString, CurrentRow(9).ToString)
ProgressBar1.Value = 50
Application.DoEvents()
End If
Catch ex As _
Microsoft.VisualBasic.FileIO.MalformedLineException
MsgBox("Line " & ex.Message &
"is not valid and will be skipped.")
End Try
End While
ProgressBar1.Value = 100
Application.DoEvents()
clsMailConfirmation.main()
TextFileReader.Dispose()
MessageBox.Show("The process has been completed successfully")
End Using
"MalformedLineException" says that Line cannot be parsed using the current Delimiters, to fix it, adjust Delimiters so the line can be parsed correctly, or insert exception-handling code in order to handle the line.
Someone encountered similar question, maybe its reply can help you.
I want to output the current row if there is an error but I'm getting a message that the current record is nothing.
Here is my code:
Dim currentRow As String()
Using MyReader As New FileIO.TextFieldParser(filenametoimport)
MyReader.TextFieldType = FileIO.FieldType.Delimited
MyReader.SetDelimiters(",")
While Not MyReader.EndOfData
Try
currentRow = MyReader.ReadFields()
ImportLine(currentRow)
Catch ex As FileIO.MalformedLineException
report.AppendLine()
report.AppendLine($"[{currentrow}]")
report.AppendLine("- record is malformed and will be skipped. ")
Continue While
End Try
End While
end Using
I need to output the currentrow so that I can report to the user that there was a bad record.
report.AppendLine($"[{currentrow}]")
I understand that the value would be null if the parse failed but is there a way to get the current record?
How do I output this record if it failed to parse the record?
Thanks for the assistance!
You can't get the raw data directly in the exception, but you can at least get the line number where the error occurred. You may be able to use that line number to go back and find the offending record:
Dim currentRow As String()
Using MyReader As New FileIO.TextFieldParser(filenametoimport)
MyReader.TextFieldType = FileIO.FieldType.Delimited
MyReader.SetDelimiters(",")
While Not MyReader.EndOfData
Try
currentRow = MyReader.ReadFields()
ImportLine(currentRow)
Catch ex As FileIO.MalformedLineException
report.AppendLine($"{vbCrLf}- record at line {ex.LineNumber} is malformed and will be skipped. ")
End Try
End While
End Using
TextFieldParser also provides access to the underlying stream, and provides a ReadLine() method, so if you're really desparate to write the code you could walk back the stream to the previous line ending and then call MyReader.ReadLine() to get the record (which would in turn advance the stream again to where you expect).
I did not get a compile error on MyReader.SetDelimiters(",") but I changed it to an array anyway. The report.AppendLine($"[{currentrow}]") line probably doesn't expect an array. That line I altered to provide a string.
Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
Dim currentRow As String() = Nothing
Using MyReader As New FileIO.TextFieldParser("filenametoimport")
MyReader.TextFieldType = FileIO.FieldType.Delimited
MyReader.SetDelimiters({","})
While Not MyReader.EndOfData
Try
currentRow = MyReader.ReadFields()
ImportLine(currentRow)
Catch ex As FileIO.MalformedLineException
report.AppendLine()
report.AppendLine($"[{String.Join(",", currentRow)}]")
report.AppendLine("- record is malformed and will be skipped. ")
Continue While
End Try
End While
End Using
End Sub
EDIT
As per comments by # Joel Coehoorn and # ErocM if the row is null you could provide the content of the previous row so they errant row could be located.
Dim LastGoodRow As String()
While Not MyReader.EndOfData
Try
currentRow = MyReader.ReadFields()
ImportLine(currentRow)
LastGoodRow = currentRow
Catch ex As FileIO.MalformedLineException
report.AppendLine()
report.AppendLine($"[{String.Join(",", LastGoodRow)}]")
report.AppendLine("- record following this row is malformed and will be skipped. ")
Continue While
End Try
End While
I have a vb project that imports a csv file and some of the data contains commas. The fields with the commas are in double quotes.
I am creating a datagridview from the header row of the csv then importing the remainder of the file into the dgv but the fields with commas are causing a problem. The fields are not fixed width.
I think I need a way to qualify the commas as a delimiter based on double quote or some other method of importing the data into the dgv.
Thanks
Using objReader As New StreamReader(FName)
Dim line As String = objReader.ReadLine()
Do While objReader.Peek() <> -1
line = objReader.ReadLine()
Dim splitLine() As String = line.Split(",")
DataGridView1.Rows.Add(splitLine)
Application.DoEvents()
Loop
End Using
Example Data:
1,"VALIDFLAG, NOGPS",0,1.34,3.40,0.17,1
Thinks very much for the suggestions.
I am going to use textfieldparser for my import.
Using MyReader As New Microsoft.VisualBasic.FileIO.TextFieldParser(FName)
MyReader.TextFieldType = Microsoft.VisualBasic.FileIO.FieldType.Delimited
MyReader.Delimiters = New String() {","}
Dim currentRow As String()
Dim firstline As Boolean = True
While Not MyReader.EndOfData
Try
currentRow = MyReader.ReadFields()
If firstline = True Then
firstline = False
Else
Me.DataGridView1.Rows.Add(currentRow)
End If
Catch ex As Microsoft.VisualBasic.FileIO.MalformedLineException
MsgBox("Line " & ex.Message & " is invalid. Skipping")
End Try
Application.DoEvents()
End While
End Using
FASTER,WW0011,"CTR ,REURN,ALT TUBING HELIUM LEAK",DEFAULT test,1,3.81,test
I need to get the result of the following line as
Arr(0) =faster
Arr(1) =WW0011
Arr(2) =CTR ,REURN,ALT TUBING HELIUM LEAK
Arr(3) =DEFAULT test
Arr(4) =faster
Arr(5) = 1
Arr(6)=3.81
Arr(7) = test
I tried using split, but the problem is on Arr(2)
could anyone please give me a solution
You could use the TextFieldParser class which will take care of situations like this. Set the HasFieldEnclosedInQuotes property to true. Here is an example from MSDN (slightly altered):
Using MyReader As New Microsoft.VisualBasic.FileIO.TextFieldParser("c:\logs\bigfile")
MyReader.TextFieldType = Microsoft.VisualBasic.FileIO.FieldType.Delimited
MyReader.Delimiters = New String() {","}
'Set this to ignore commas in quoted fields.
MyReader.HasFieldsEnclosedInQuotes = True
Dim currentRow As String()
'Loop through all of the fields in the file.
'If any lines are corrupt, report an error and continue parsing.
While Not MyReader.EndOfData
Try
currentRow = MyReader.ReadFields()
' Include code here to handle the row.
Catch ex As Microsoft.VisualBasic.FileIO.MalformedLineException
MsgBox("Line " & ex.Message & " is invalid. Skipping")
End Try
End While
End Using
I use this function alot myself
Private Function splitQuoted(ByVal line As String, ByVal delimeter As Char) As String()
Dim list As New List(Of String)
Do While line.IndexOf(delimeter) <> -1
If line.StartsWith("""") Then
line = line.Substring(1)
Dim idx As Integer = line.IndexOf("""")
While line.IndexOf("""", idx) = line.IndexOf("""""", idx)
idx = line.IndexOf("""""", idx) + 2
End While
idx = line.IndexOf("""", idx)
list.Add(line.Substring(0, idx))
line = line.Substring(idx + 2)
Else
list.Add(line.Substring(0, Math.Max(line.IndexOf(delimeter), 0)))
line = line.Substring(line.IndexOf(delimeter) + 1)
End If
Loop
list.Add(line)
Return list.ToArray
End Function
Use a for loop to iterate the string char by char!
I have this code to read a CVS file. It reads each line, devides each line by delimiter ',' and stored the field values in array 'strline()' .
How do I extract only required fields from the CSV file?
For example if I have a CSV File like
Type,Group,No,Sequence No,Row No,Date (newline)
0,Admin,3,345678,1,26052010 (newline)
1,Staff,5,78654,3,26052010
I Need only the value of columns Group,Sequence No and date.
Thanks in advance for any ideas.
Dim myStream As StreamReader = Nothing
' Hold the Parsed Data
Dim strlines() As String
Dim strline() As String
Try
myStream = File.OpenText(OpenFile.FileName)
If (myStream IsNot Nothing) Then
' Hold the amount of lines already read in a 'counter-variable'
Dim placeholder As Integer = 0
strlines = myStream.ReadToEnd().Split(Environment.NewLine)
Do While strlines.Length <> -1 ' Is -1 when no data exists on the next line of the CSV file
strline = strlines(placeholder).Split(",")
placeholder += 1
Loop
End If
Catch ex As Exception
LogErrorException(ex)
Finally
If (myStream IsNot Nothing) Then
myStream.Close()
End If
End Try
1) DO NOT USE String.Split!!
CSV data can contain comma's, e.g.
id,name
1,foo
2,"hans, bar"
Also as above you would need to handle quoted fields etc... See CSV Info for more details.
2) Check out TextFieldParser - it hadles all this sort of thing.
It will handle the myriad of different escapes you can't do with string.split...
Sample from: http://msdn.microsoft.com/en-us/library/cakac7e6.aspx
Using MyReader As New Microsoft.VisualBasic.FileIO.TextFieldParser("C:\TestFolder\test.txt")
MyReader.TextFieldType = FileIO.FieldType.Delimited
MyReader.SetDelimiters(",")
Dim currentRow As String()
While Not MyReader.EndOfData
Try
currentRow = MyReader.ReadFields()
Dim currentField As String
For Each currentField In currentRow
MsgBox(currentField)
Next
Catch ex As Microsoft.VisualBasic.FileIO.MalformedLineException
MsgBox("Line " & ex.Message & "is not valid and will be skipped.")
End Try
End While
End Using
The MyReader.ReadFields() part will get you an array of strings, from there you'll need to use the index etc...
PK :-)
Maybe instead of only importing selected fields, you should import everything, then only use the ones you need.