Performance improvement on vb.net code - vb.net

I need to write 50 million records with 72 columns into text file, the file size is growing as 9.7gb .
I need to check each and every column length need to format as according to the length as defined in XML file.
Reading records from oracle one by one and checking the format and writing into text file.
To write 5 crores records it is taking more than 24 hours. how to increase the performance in the below code.
Dim valString As String = Nothing
Dim valName As String = Nothing
Dim valLength As String = Nothing
Dim valDataType As String = Nothing
Dim validationsArray As ArrayList = GetValidations(Directory.GetCurrentDirectory() + "\ReportFormat.xml")
Console.WriteLine("passed xml")
Dim k As Integer = 1
Try
Console.WriteLine(System.DateTime.Now())
Dim selectSql As String = "select * from table where
" record_date >= To_Date('01-01-2014','DD-MM-YYYY') and record_date <= To_Date('31-12-2014','DD-MM-YYYY')"
Dim dataTable As New DataTable
Dim oracleAccess As New OracleConnection(System.Configuration.ConfigurationManager.AppSettings("OracleConnection"))
Dim cmd As New OracleCommand()
cmd.Connection = oracleAccess
cmd.CommandType = CommandType.Text
cmd.CommandText = selectSql
oracleAccess.Open()
Dim Tablecolumns As New DataTable()
Using oracleAccess
Using writer = New StreamWriter(Directory.GetCurrentDirectory() + "\FileName.txt")
Using odr As OracleDataReader = cmd.ExecuteReader()
Dim sbHeaderData As New StringBuilder
For i As Integer = 0 To odr.FieldCount - 1
sbHeaderData.Append(odr.GetName(i))
sbHeaderData.Append("|")
Next
writer.WriteLine(sbHeaderData)
While odr.Read()
Dim sbColumnData As New StringBuilder
Dim values(odr.FieldCount - 1) As Object
Dim fieldCount As Integer = odr.GetValues(values)
For i As Integer = 0 To fieldCount - 1
Dim vals As Array = validationsArray(i).ToString.ToUpper.Split("|")
valName = vals(0).trim
valDataType = vals(1).trim
valLength = vals(2).trim
Select Case valDataType
Case "VARCHAR2"
If values(i).ToString().Length = valLength Then
sbColumnData.Append(values(i).ToString())
'sbColumnData.Append("|")
ElseIf values(i).ToString().Length > valLength Then
sbColumnData.Append(values(i).ToString().Substring(0, valLength))
'sbColumnData.Append("|")
Else
sbColumnData.Append(values(i).ToString().PadRight(valLength))
'sbColumnData.Append("|")
End If
Case "NUMERIC"
valLength = valLength.Substring(0, valLength.IndexOf(","))
If values(i).ToString().Length = valLength Then
sbColumnData.Append(values(i).ToString())
'sbColumnData.Append("|")
Else
sbColumnData.Append(values(i).ToString().PadLeft(valLength, "0"c))
'sbColumnData.Append("|")
End If
'sbColumnData.Append((values(i).ToString()))
End Select
Next
writer.WriteLine(sbColumnData)
k = k + 1
Console.WriteLine(k)
End While
End Using
writer.WriteLine(System.DateTime.Now())
End Using
End Using
Console.WriteLine(System.DateTime.Now())
'Dim Adpt As New OracleDataAdapter(selectSql, oracleAccess)
'Adpt.Fill(dataTable)
Return Tablecolumns
Catch ex As Exception
Console.WriteLine(System.DateTime.Now())
Console.WriteLine("Error: " & ex.Message)
Console.ReadLine()
Return Nothing
End Try

Related

Split output result by n and loop

I'm solving an issue where I need to create one PDF form.
That PDF form has 8 sections where I need to put info about and looks like shown on picture (only 4 shown).
The point is that my query will return 0 - n different results. So I need to split by 8 and post on different pages.
I tried like shown below but that seems not to work since I always load a new document. Does anyone have some advice how to make it?
Try
Dim sCommand As OleDb.OleDbCommand
sCommand = New OleDb.OleDbCommand("SELECT a,b,c Query to fetch n results ", _dbCon)
sCommand.CommandTimeout = 0
Dim _dbREADER As OleDb.OleDbDataReader
_dbREADER = sCommand.ExecuteReader
Dim dt As DataTable = New DataTable()
dt.Load(_dbREADER)
Dim totalPages As Integer = dt.Rows.Count / 8
Dim currentPage As Integer = 1
Dim rowCounter As Long = 0
If dt.Rows.Count > 0 Then
For Each row In dt.Rows
rowCounter += 1
If rowCounter = 8 Then
currentPage += 1
rowCounter = 0
End If
_pdfDocumentOutput = System.IO.Path.GetTempPath() & "MailingForm_" & currentPage & ".pdf"
SaveFromResources(_pdfDocument, My.Resources.template)
Using reader As New PdfReader(_pdfDocument)
Using stamper As New PdfStamper(reader, New IO.FileStream(_pdfDocumentOutput, IO.FileMode.Create))
Dim fontName As String = System.IO.Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.Fonts), "SCRIPTIN.ttf")
Dim bf As BaseFont = BaseFont.CreateFont(fontName, BaseFont.IDENTITY_H, BaseFont.NOT_EMBEDDED)
Dim pdfForm As AcroFields = stamper.AcroFields
pdfForm.AddSubstitutionFont(bf)
pdfForm.SetField(rowCounter - 1 & "0", row("Customer")) 'Checks the top radiobutton of the VrPr4 field
pdfForm.SetField(rowCounter - 1 & "1", row("Address"))
pdfForm.SetField(rowCounter - 1 & "2", row("Location"))
stamper.FormFlattening = True
End Using
End Using
Next
End If
Status.Text = "Store info loaded ! "
Catch ex As Exception
Status.Text = ex.Message
End Try
I found the solution by splitting tables
Private Shared Function SplitTable(ByVal originalTable As DataTable, ByVal batchSize As Integer) As List(Of DataTable)
Dim tables As List(Of DataTable) = New List(Of DataTable)()
Dim i As Integer = 0
Dim j As Integer = 1
Dim newDt As DataTable = originalTable.Clone()
newDt.TableName = "Table_" & j
newDt.Clear()
For Each row As DataRow In originalTable.Rows
Dim newRow As DataRow = newDt.NewRow()
newRow.ItemArray = row.ItemArray
newDt.Rows.Add(newRow)
i += 1
If i = batchSize Then
tables.Add(newDt)
j += 1
newDt = originalTable.Clone()
newDt.TableName = "Table_" & j
newDt.Clear()
i = 0
End If
If row.Equals(originalTable.Rows(originalTable.Rows.Count - 1)) Then
tables.Add(newDt)
j += 1
newDt = originalTable.Clone()
newDt.TableName = "Table_" & j
newDt.Clear()
i = 0
End If
Next
Return tables
End Function
And after that loop through each one of table . And put all results to one file
Dim tables = SplitTable(dt, 8)

How to change value of /GrpPwd/ Cell in all Rows?

I need to perform this SQL Query:
UPDATE Resurses SET GrpPwd = #GrpPwd1 WHERE Resurs = #Resurs1
For all rows in my Access Database using VB.NET
How can i do it?
I use this code but id doesn't work:
Password Generator:
Dim charset As String = nalf
Dim r As New Random()
Dim lenPass As Integer = r.Next(minLength, maxLength)
Dim str As String = String.Empty
For i As Integer = 0 To lenPass - 1
str += charset(r.Next(0, charset.Length))
Next
Return str
End Function
Update rows
If DataGridView1.CurrentRow.Cells(3).Value = "Yes" Then
nres = DataGridView1.CurrentRow.Cells(0).Value
nalf = DataGridView1.CurrentRow.Cells(6).Value
nsym = DataGridView1.CurrentRow.Cells(1).Value
Dim parol1 As String
Dim pwd As String = pass99(nsym, nsym) '
parol1 = pwd
Dim cmd As New OleDbCommand()
Dim Conn As New OleDb.OleDbConnection("Provider=Microsoft.ACE.oledb.12.0; Data source=" + bpath)
cmd.Connection = Conn
Conn.Open()
For i As Integer = 0 To k - 1
cmd.CommandText = String.Format("UPDATE Resurses SET GrpPwd = #GrpPwd1{0} WHERE Resurs = #Resurs1{0};", i)
cmd.Parameters.Add(String.Format("#GrpPwd1{0}", i), OleDbType.WChar).Value = parol1
cmd.Parameters.Add(String.Format("#Resurs1{0}", i), OleDbType.Integer).Value = nres
Next
cmd.ExecuteNonQuery()
Conn.Close()
MsgBox("All rows are updated!")
End If
f

VB: Count number of columns in csv

So, quite simple.
I am importing CSVs into a datagrid, though the csv always has to have a variable amount of columns.
For 3 Columns, I use this code:
Dim sr As New IO.StreamReader("E:\test.txt")
Dim dt As New DataTable
Dim newline() As String = sr.ReadLine.Split(";"c)
dt.Columns.AddRange({New DataColumn(newline(0)), _
New DataColumn(newline(1)), _
New DataColumn(newline(2))})
While (Not sr.EndOfStream)
newline = sr.ReadLine.Split(";"c)
Dim newrow As DataRow = dt.NewRow
newrow.ItemArray = {newline(0), newline(1), newline(2)}
dt.Rows.Add(newrow)
End While
DG1.DataSource = dt
This works perfectly. But how do I count the number of "newline"s ?
Can I issue a count on the number of newlines somehow? Any other example code doesn't issue column heads.
If my csv file has 5 columns, I would need an Addrange of 5 instead of 3 and so on..
Thanks in advance
Dim sr As New IO.StreamReader(path)
Dim dt As New DataTable
Dim newline() As String = sr.ReadLine.Split(","c)
' MsgBox(newline.Count)
' dt.Columns.AddRange({New DataColumn(newline(0)),
' New DataColumn(newline(1)),
' New DataColumn(newline(2))})
Dim i As Integer
For i = 0 To newline.Count - 1
dt.Columns.AddRange({New DataColumn(newline(i))})
Next
While (Not sr.EndOfStream)
newline = sr.ReadLine.Split(","c)
Dim newrow As DataRow = dt.NewRow
newrow.ItemArray = {newline(0), newline(1)}
dt.Rows.Add(newrow)
End While
dgv.DataSource = dt
End Sub
Columns and item values can be added to a DataTable individually, using dt.Columns.Add and newrow.Item, so that these can be done in a loop instead of hard-coding for a specific number of columns. e.g. (this code assumes Option Infer On, so adjust as needed):
Public Function CsvToDataTable(csvName As String, Optional delimiter As Char = ","c) As DataTable
Dim dt = New DataTable()
For Each line In File.ReadLines(csvName)
If dt.Columns.Count = 0 Then
For Each part In line.Split({delimiter})
dt.Columns.Add(New DataColumn(part))
Next
Else
Dim row = dt.NewRow()
Dim parts = line.Split({delimiter})
For i = 0 To parts.Length - 1
row(i) = parts(i)
Next
dt.Rows.Add(row)
End If
Next
Return dt
End Function
You could then use it like:
Dim dt = CsvToDataTable("E:\test.txt", ";"c)
DG1.DataSource = dt

How can I be alerted if my recursion decides to start skipping loops?

I have a nested For / Next recursion at the heart of a web app I am trying to develop. When I programmed it, I verified its output on an example that nested 3-5 levels and thought all was well.
Lately, I decided to verify an entry in the result set that was nested 12 to 15 levels deep. The resulting percentage was incorrect because some progeny were not recursed.
Because my result set can take so long to run (8-13 hours for the more indepth queries) and I could run the same test on a subset (~150,000 records) of the records in the full database table (~1,200,000 records), I trialled it against the smaller table.
Low and behold it appears to work perfectly. A search that was returning ~56,000 records, returned ~126,000 records (meaning it was previously skipping out a lot of recursions). I verified random sample results to be correct.
Comparing result sets from the same query done on the two database tables, it seems that the missed recursions start to show up with some (not all) of the records that are nested beyond 12 levels deep.
The troubling thing for me is that I need to know when my result set is suspect without having to search the result set for dropped nesting.
Here is the code for the recursive sub:
' lookup table for up to 63 generations '
Dim percentage() As Double = {100, 50, 25, 12.5, 6.25, 3.125, ect. . .}
' DataTable to display results of looping through the db '
Dim t As DataTable
Dim c As DataColumn
Dim r As DataRow
' columns are: id, name, dad, mom, gender, year born, trait, percentage '
Private Sub GetPct(ByRef progeny As List(Of Int32), ByRef gender As List(Of String), ByVal generations As Int16, ByVal count As Int16)
Dim nxtGeneration As Int16 = generations + 1
Dim nxtPercentage As Double = percentage(nxtGeneration)
For i As Int16 = 0 To count
Dim dbConn As New SqlConnection(connString)
Dim j As Int16 = -1
Dim prog As New List(Of Int32)
Dim gndr As New List(Of String)
If gender(i) = "M" Then
Dim dreader As SqlDataReader
Dim dgetComm As New SqlCommand("d_get", dbConn)
dgetComm.CommandType = CommandType.StoredProcedure
dgetComm.Parameters.Add("#id", SqlDbType.Int)
dgetComm.Parameters("#id").Value = prog(i)
Using dbConn
Try
dbConn.Open()
dreader = dgetComm.ExecuteReader()
If dreader.HasRows = True Then
While dreader.Read()
j += 1
Dim updated As DataRow = t.Rows.Find(dreader(0))
If updated Is Nothing Then
t.BeginLoadData()
r = t.NewRow()
r(0) = dreader(0)
r(1) = dreader(1)
r(2) = dreader(2)
r(3) = dreader(3)
r(4) = dreader(4)
r(5) = dreader(5)
r(6) = dreader(6)
r(7) = nxtPercentage
t.Rows.Add(r)
t.EndLoadData()
prg.Add(dreader(0))
gnd.Add(dreader(4))
Else
prg.Add(dreader(0))
gnd.Add(dreader(4))
updated(7) += nxtPercentage
End If
End While
End If
dgetComm.Dispose()
dgetComm = Nothing
dreader.Close()
dreader = Nothing
Catch ex As Exception
' modify when going live '
lblDetails.Text &= "Error loading to table with get of dam" & Err.Description
End Try
End Using
GetPct(prg, gnd, nxtG, j)
Else
Dim sreader As SqlDataReader
Dim sgetComm As New SqlCommand("s_get", dbConn)
sgetComm.CommandType = CommandType.StoredProcedure
sgetComm.Parameters.Add("#id", SqlDbType.Int)
sgetComm.Parameters("#id").Value = prog(i)
Using dbConn
Try
dbConn.Open()
sreader = sgetComm.ExecuteReader()
If sreader.HasRows = True Then
While sreader.Read()
j += 1
Dim updated As DataRow = t.Rows.Find(srdr(0))
If updated Is Nothing Then
t.BeginLoadData()
r = t.NewRow()
r(0) = sreader(0)
r(1) = sreader(1)
r(2) = sreader(2)
r(3) = sreader(3)
r(4) = sreader(4)
r(5) = sreader(5)
r(6) = sreader(6)
r(7) = nxtPercentage
t.Rows.Add(r)
t.EndLoadData()
prg.Add(sreader(0))
gnd.Add(sreader(4))
Else
prg.Add(sreader(0))
gnd.Add(sreader(4))
updated(7) += nxtPercentage
End If
End While
End If
sgetComm.Dispose()
sgetComm = Nothing
sreader.Close()
sreader = Nothing
Catch ex As Exception
' modify when going live'
lblDetails.Text &= "Error loading to table with get of sire" & Err.Description
End Try
End Using
GetPct(prg, gnd, nxtG, j)
End If
Next i
End Sub

How to move to next row in dataset and display in hyperlink

I'm writing an email application that will be used to send HTML news articles to clients.
I'm using a dataset to return the headlines to display to the client. When I loop through the dataset the the latest record is returned but latest headline link is not displayed. So the outputted HTML is the same headline everytime, which is the first record in the dataset. How do I move to the next record in the data set and get the outputted HTML to display the next/correct headline?
Here is a sample of my code:
'Code to populate dataset
Public Function GetHeadline(ByVal ArticleID As Integer) As DataSet
Try
Dim objConn As SqlConnection = New SqlConnection()
objConn.ConnectionString = myConnectionString
objConn.Open()
ds = New DataSet
ds.Clear()
Dim sqlCommand As String = "SomeSql"
Dim objCmd As SqlCommand = New SqlCommand(sqlCommand, objConn)
Dim dataAdapter As SqlDataAdapter = New SqlDataAdapter(objCmd)
dataAdapter.Fill(ds)
Catch ex As Exception
MsgBox(ex.ToString)
GetHeadline = Nothing
End Try
Return ds
End Function
'Code to populate link
If GroupID = 4 Then
iLocation1 = HTMLbody.IndexOf("{!HeadlineID")
While iLocation1 > 0
iLocation2 = HTMLbody.IndexOf("}", iLocation1)
sHeadLineTag = HTMLbody.Substring(iLocation1 + 1, iLocation2 - iLocation1 - 1)
dtReport = clsGlobal.globalReportCatalog.GetHeadline2()
clsGlobal.globalReportCatalog.SetHeadlinePropertiesFromRow(dtReport.Rows(0))
With clsGlobal.globalReportCatalog
For i As Integer = 0 To dtReport.Rows.Count
If i < dtReport.Rows.Count - 1 Then
i = i + 1
End If
Dim ID As Integer = dtReport.Rows(i)("ArticleID")
sHyperTag = "" & .HeadlineReportName & " - " & .HeadlineTitle & ""
sHeadlineDescription = .HeadlineDescription
HTMLbody = HTMLbody.Replace("{!Report.Description}", sHeadlineDescription)
Next
End With
I don't see why you need
For i As Integer = 0 To dtReport.Rows.Count
If i < dtReport.Rows.Count - 1 Then
i = i + 1
End If
Can't you use
Dim ID As Integer = dtReport.Rows(dtReport.Rows.Count - 1)("ArticleID")
or was there supposed to be a row movenext in the loop you forgot?