Need to Batch a Large DataTable and write each batch to a Text file - VB.Net - vb.net

Question

I have a requirement that I need to query a DB and fetch the records in a Data Table. The Data Table has 20,000 records.
I need to batch these records in Batches of 100 records each and write these batches into a individual Text files.
Till now I have been able to batch the records in batches of 100 each using IEnumerable(of DataRow).
I am now facing issue in writing the IEnumeable(Of DatRow) to a Text File.
My code is a below:
Dim strsql = "Select * from myTable;"
Dim dt as DataTable
Using cnn as new SqlConnection(connectionString)
cnn.Open()
Using dad as new SqlAdapter(strsql ,cnn)
dad.fill(dt)
End Using
cnn.Close()
End Using
Dim Chunk = getChunks(dt,100)
For each chunk as IEnumerable(Of DataRow) In Chunks
Dim path as String = "myFilePath"
If Not File.Exists(myFilePath) Then
//** Here I will write my Batch into the File.
End If
Next
Public Iterator Function getChunks(byVal Tab as DataTable, byVal size as Integer) as IEnumerable (Of IEnumerable(of DataRow))
Dim chunk as List(Of DataRow) = New List(of DataRow)(size)
For Each row As DataRow in tab.Rows
chunk.Add(row)
if chunk.Count = size Then
Yield chunk
chunk = New List(of DataRow0(size)
Next
if chunk.Any() Then Yield chunk
End Function
Need your help to write the IEneumerable of DataRows into a Text file for each Batch of Records.
Thanks
:)

Answer 1

Your existing code is needlessly complex. If this is all you're doing, then using a datatable is unnecessary/unwise; this is one of the few occasions I would advocate using a lower level datareader to keep the memory impact low
Writing a db table to a file, quick, easy and low memory consumption:
Dim dr = sqlCommand.ExecuteReader()
Dim sb as New StringBuilder
Dim lineNum = -1
Dim batchSize = 100
While dr.Read()
'turn the row into a string for our file
For x = 0 to dr.FieldCount -1
sb.Append(dr.GetString(x)).Append(",")
Next x
sb.Length -= 1 'remove trailing comma
sb.AppendLine()
'keep track of lines written so we can batch accordingly
lineNum += 1
Dim fileNum = lineNum \ batchSize
File.AppendAllText($"c:\temp\file{fileNum}.csv", sb.ToString())
'clear the stringbuilder
sb.Length = 0
End While
If you really want to use a datatable, there isn't anything stopping you swapping this while dr For a For Each r as DataRow in myDatatable.Rows
Please note, this isn't an exercise in creating a fully escaped csv, nor formatting the data; it is demonstrating the concept of having a firehose of data and simply writing it to N different files by utilising the fact that doing an integer divide on every number from 0 to 99 will result in 0 (and hence go in file 0) and then very number from 1 to 199 will result in 1 (and hence lines go in file 1) etc, and doing this process on a single stream of data, or single iteration of N items
You could build the file lines in the string builder and write them every batchSize if lineNum Mod batchSize = batchSize - 1, if you feel that it would be more efficient than calling file appendalltext (which opens and closes the file)

Answer 2

Tested this with a table of a little over 1,500 records and 10 fields. The file creation took a little over 5 seconds (excluding data access). All things being equal (which I know they are not) that would be over 13 seconds writing the files.
Since your problem was with the iterator I assume the there were no memory issues with the DataTable.
You can include more than one database object in a Using block by using a comma to designate a list of objects in the Using.
Private Sub OPCode()
Dim myFilePath = "C:\Users\xxx\Documents\TestLoop\DataFile"
Dim strsql = "Select * from myTable;"
Dim dt As New DataTable
Using cnn As New SqlConnection(connectionString),
cmd As New SqlCommand(strsql, cnn)
cnn.Open()
dt.Load(cmd.ExecuteReader)
End Using
sw.Start()
Dim StartRow = 0
Dim EndRow = 99
Dim FileNum = 1
Dim TopIndex = dt.Rows.Count - 1
Do
For i = StartRow To EndRow
Dim s = String.Join("|", dt.Rows(i).ItemArray)
File.AppendAllText(myFilePath & FileNum & ".txt", s & Environment.NewLine)
Next
FileNum += 1
StartRow += 100
EndRow += 100
If EndRow >= TopIndex Then
EndRow = TopIndex
End If
Loop Until StartRow >= TopIndex
sw.Stop()
MessageBox.Show(sw.ElapsedMilliseconds.ToString)
End Sub

Answer 3

I thought your code was a great use of the iteration function.
Here is the code for your iterator.
Public Iterator Function getChunks(ByVal Tab As DataTable, ByVal size As Integer) As IEnumerable(Of IEnumerable(Of DataRow))
Dim chunk As List(Of DataRow) = New List(Of DataRow)(size)
For Each row As DataRow In Tab.Rows
chunk.Add(row)
If chunk.Count = size Then
Yield chunk
chunk = New List(Of DataRow)(size)
End If
Next
If chunk.Any() Then Yield chunk
End Function
Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
Dim dt = LoadDataTable()
Dim myFilePath As String = "C:\Users\xxx\Documents\TestLoop\DataFile"
Dim FileNum = 1
For Each chunk As IEnumerable(Of DataRow) In getChunks(dt, 100)
For Each row As DataRow In chunk
Dim s = String.Join("|", row.ItemArray)
File.AppendAllText(myFilePath & FileNum & ".txt", s & Environment.NewLine)
Next
FileNum += 1
Next
MessageBox.Show("Done")
End Sub
You just needed to nest the For Each to get at the data rows.