How do I improve perfomance creating DataTable in VB.net? - vb.net

Question

I have following code and it's slow to load first time. CSV file is about 4mb 16000 rows.
If Session("tb") Is Nothing Then
Dim str As String()
If (IsNothing(Cache("csvdata"))) Then
str = File.ReadAllLines(Server.MapPath("~/test/feed.csv"))
Cache.Insert("csvdata", str, Nothing, DateTime.Now.AddHours(12), TimeSpan.Zero)
Else
str = CType(Cache("csvdata"), Array)
End If
Dim dt As New DataTable
dt.Columns.Add("Shape", GetType(System.String))
dt.Columns.Add("Weight", GetType(System.Double))
dt.Columns.Add("Color", GetType(System.String))
dt.Columns.Add("Clarity", GetType(System.String))
dt.Columns.Add("Price", GetType(System.Int32))
dt.Columns.Add("CutGrade", GetType(System.String))
For i As Integer = 1 To str.Length - 1
Dim pattern As String = ",(?=([^""]*""[^""]*"")*[^""]*$)"
Dim rgx As New Regex(pattern)
Dim t As String = rgx.Replace(str(i), "\")
Dim s As String() = t.Split("\"c)
Dim pr As Int32 = CType(s(5), Int32)
Dim fpr As Int32
Dim rate As Double
Select Case pr
Case Is < 300
rate = 2
Case 301 To 600
rate = 1.7
Case Is > 600
rate = 1.16
End Select
fpr = Math.Round(pr * rate)
Dim a As String() = {s(1), s(2), s(3), s(4), fpr, s(40)}
dt.Rows.Add(a)
Next
Session("tb") = dt
ListView1.DataSource = dt
ListView1.DataBind()
Else
Dim x As DataTable = CType(Session("tb"), DataTable)
ListView1.DataSource = x
ListView1.DataBind()
End If
csv file is cached and I assume this can share with everyone.
(one person loads once in 12 hours)
Once I create Session, the page loads fast as well.
So, creating Datatable seems to be the slow process.
This is first time to deal with datatable and I'm sure someone can point out what I'm doing wrong.
Thank you
UPDATE:
I have changed Cache to the original Datatable instead of CSV file.
It loads fast now, but I would like to know if this is a bad idea or not.
Cache.Insert("csvdata", dt, Nothing, DateTime.Now.AddHours(12), TimeSpan.Zero)
Once it's stored in Cache, I can run Query against it using Linq.
SAMPLE CSV first 3 rows
Supplier ID,Shape,Weight,Color,Clarity,Price / Carat,Lot Number,Stock Number,Lab,Cert #,Certificate Image,2nd Image,Dimension,Depth %,Table %,Crown Angle,Crown %,Pavilion Angle,Pavilion %,Girdle Thinnest,Girdle Thickest,Girdle %,Culet Size,Culet Condition,Polish,Symmetry,Fluor Color,Fluor Intensity,Enhancements,Remarks,Availability,Is Active,FC-Main Body,FC- Intensity,FC- Overtone,Matched Pair,Separable,Matching Stock #,Pavilion,Syndication,Cut Grade,External Url
9349,Round,1.74,F,VVS1,13650.00,,IM-95-188-243,ABC,11228,,,7.81|7.85|4.62,59.00,62.00,34.00,13.00,,,Medium,,0,None,,Excellent,Very Good,Blue,Medium,,"",Not Specified,Y,,,,False,True,,,,Very Good,http://www.test/teste.
9949,Round,1.00,I,VVS1,6059.00,,IM-95-189-C021,ABC,212197,,,6.37|6.42|3.96,61.90,54.00,34.50,16.00,,,Thin,Slightly Thick,0,None,,Excellent,Good,,None,,"Additional pinpoints are not shown.",Guaranteed Available,Y,,,,False,True,,,,Very Good,http://www.test/test.

Answer 1

Look into using a TextFieldParser to read the CSV instead of splitting the strings yourself.
Also, if you use a List(Of CustomClass) where CustomClass has the Shape, Weight, Color, etc. properties you can avoid the unnecessary overhead of the DataTable and you can still do your LINQ queries against the List.
Pardon my C#, I do not have VB.NET installed on this box.
public class Gemstone
{
public string Shape { get; set; }
public double Weight { get; set; }
public string Color { get; set; }
}
static void Main(string[] args)
{
TextFieldParser textFieldParser = new TextFieldParser("data.txt");
textFieldParser.Delimiters = new string[] {","};
textFieldParser.ReadLine(); // skip header line
List<Gemstone> list = new List<Gemstone>(16000); // allocate the list with your best calculated guess of its final size
while(!textFieldParser.EndOfData)
{
string[] fields = textFieldParser.ReadFields();
Gemstone gemstone = new Gemstone();
gemstone.Shape = fields[1];
gemstone.Weight = Double.Parse(fields[2]);
gemstone.Color = fields[3];
list.Add(gemstone);
}

Answer 2

FYI I just found this whole TextFieldParser thing, and I do a LOT of text file parsing so I tested it....
On an 11mb file, with about 5200 rows and 300 columns.
It was 25% of the speed i was using when putting in a datatable. It was about 15% of the speed when I removed the datatable code:
Dim DataTable As New DataTable()
Dim StartTime As Long = Now.Ticks
Dim Reader As New FileIO.TextFieldParser("file.txt")
Reader.TextFieldType = FileIO.FieldType.Delimited
Reader.SetDelimiters(vbTab)
Reader.HasFieldsEnclosedInQuotes = False
Dim Header As Boolean = True
While Not Reader.EndOfData
Dim Fields() As String = Reader.ReadFields
If Header Then
For I As Integer = 1 To 320
DataTable.Columns.Add("Col" & I)
Next
Header = False
Else
If Mid(Fields(0), 1, 1) <> "#" Then DataTable.Rows.Add(Fields)
End If
End While
Debug.Print((Now.Ticks - StartTime) / 10000 & "ms")
Dim DataTable2 As New DataTable()
StartTime = Now.Ticks
For I As Integer = 1 To 320
DataTable2.Columns.Add("Col" & I)
Next
For Each line As String In System.IO.File.ReadAllLines("file.txt")
Dim NVP() As String = Split(line, vbTab)
If Mid(line, 1, 1) <> "#" Then DataTable2.Rows.Add(NVP)
Next
Debug.Print((Now.Ticks - StartTime) / 10000 & "ms")
With the datable code removed:
Dim StartTime As Long = Now.Ticks
Dim Reader As New FileIO.TextFieldParser("file.txt")
Reader.TextFieldType = FileIO.FieldType.Delimited
Reader.SetDelimiters(vbTab)
Reader.HasFieldsEnclosedInQuotes = False
Dim Header As Boolean = True
While Not Reader.EndOfData
Dim Fields() As String = Reader.ReadFields
End While
Debug.Print((Now.Ticks - StartTime) / 10000 & "ms")
StartTime = Now.Ticks
For Each line As String In System.IO.File.ReadAllLines("file.txt")
Dim NVP() As String = Split(line, vbTab)
Next
Debug.Print((Now.Ticks - StartTime) / 10000 & "ms")
Kinda surprising to me, but I guess the datatable has more functionality. Another new thing I discover that I'll never use :(