sqlite update in a loop take to long - vb.net

I am updating only 20 rows in a loop but the best performance I get is around 2.5 seconds for the 20 rows.
What can I do to make this simple update faster?
This is what Iv'e tried:
Dim connectionString As New SQLite.SQLiteConnection("data source=" & Application.StartupPath & "\db.db3; Version=3;")
Dim connection = New SQLite.SQLiteConnection(connectionString)
connection.Open()
Dim com3 As New System.Data.SQLite.SQLiteCommand(connection)
For Each row As DataRow In table.Rows
com3.CommandText = "UPDATE Results SET LastScore=" & row.Item("position") & " WHERE TeamID='" & row.Item("teamid") & "'"
com3.ExecuteNonQuery()
Next
connection.Close()
AND
Dim connectionString As New SQLite.SQLiteConnection("data source=" & Application.StartupPath & "\db.db3; Version=3;")
Using con As New SQLiteConnection(connectionString)
con.Open()
Using c As New SQLiteCommand(con)
c.CommandText = "UPDATE Results SET LastScore= #Pos WHERE TeamID = #ID"
For Each row As DataRow In table.Rows
c.Parameters.AddWithValue("#Pos", row.Item("position"))
c.Parameters.AddWithValue("#ID", row.Item("teamid"))
c.ExecuteNonQuery()
Next
End Using
con.Close()
End Using

Thanks to muffi's answer this is now resolved. Add BeginTransaction()
Dim connectionString As New SQLite.SQLiteConnection("data source=" & Application.StartupPath & "\db.db3; Version=3;")
Using con As New SQLiteConnection(connectionString)
con.Open()
Using t As SQLiteTransaction = con.BeginTransaction()
Using c As New SQLiteCommand(con)
c.CommandText = "UPDATE Results SET LastScore= #Pos WHERE TeamID = #ID"
For Each row As DataRow In table.Rows
c.Parameters.AddWithValue("#Pos", row.Item("position"))
c.Parameters.AddWithValue("#ID", row.Item("teamid"))
c.ExecuteNonQuery()
Next
End Using
t.Commit()
End Using
con.Close()
End Using

Related

select multiple column from excel in VBA

How can I select multiple column in Excel file using Vba. because I have excel file contains 580 columns.
I try using Query below, but its only can select 255 columns.
Dim ExcelConnection As New System.Data.OleDb.OleDbConnection("Provider=Microsoft.ACE.OLEDB.12.0; Data Source=" & TxtPath.Text & "; Extended Properties=""Excel 12.0 Xml; HDR=Yes""")
ExcelConnection.Open()
Dim expr As String = "select * FROM [Sheet1$]"
Dim objCmdSelect As OleDbCommand = New OleDbCommand(expr, ExcelConnection)
Dim objDR As OleDbDataReader
Dim SQLconn As New SqlConnection()
Dim ConnString As String = "data source=" & DBServer1 & "; initial catalog=" & DBDataBase1 & "; persist security info=true; user id=" & DBUSer1 & "; password=" & DBPass1
SQLconn.ConnectionString = ConnString
SQLconn.Open()
Using bulkCopy As SqlBulkCopy = New SqlBulkCopy(SQLconn)
bulkCopy.DestinationTableName = "TMP_UPLOAD_ISP"
Try
objDR = objCmdSelect.ExecuteReader
bulkCopy.WriteToServer(objDR)
objDR.Close()
SQLconn.Close()
Catch ex As Exception
MsgBox(ex.ToString)
End Try
I try by using select column header (220 Columns to select) ,its getting error until I reduce down to 100 columns.
Dim ExcelConnection As New System.Data.OleDb.OleDbConnection("Provider=Microsoft.ACE.OLEDB.12.0; Data Source=" & TxtPath.Text & "; Extended Properties=""Excel 12.0 Xml; HDR=Yes""")
ExcelConnection.Open()
Dim expr As String = "select Plan_Year,Colom4,Colom6,Colom7,Colom21,Colom22,Colom24,Colom25,Colom27,Colom28,Colom33,Colom34,Colom38,Colom39,Colom41,Colom42,Colom47,Colom48,Colom49,Colom50,Colom51,Colom52,Colom66,Colom67,Colom69,Colom70,Colom72,Colom73,Colom78,Colom79,Colom83,Colom84,Colom86,Colom87,Colom92,Colom93,Colom94,Colom95,Colom96,Colom97,Colom111,Colom112,Colom114,Colom115,Colom117,Colom118,Colom123,Colom124,Colom128,Colom129,Colom131,Colom132,Colom137,Colom138,Colom139,Colom140,Colom141,Colom142,Colom156,Colom157,Colom159,Colom160,Colom162,Colom163,Colom168,Colom169,Colom173,Colom174,Colom176,Colom177,Colom182,Colom183,Colom184,Colom185,Colom186,Colom187,Colom201,Colom202,Colom204,Colom205,Colom207,Colom208,Colom213,Colom214,Colom218,Colom219,Colom221,Colom222,Colom227,Colom228,Colom229,Colom230,Colom231,Colom232,Colom246,Colom247,Colom249,Colom250,Colom252,Colom253,Colom258,Colom259,Colom263,Colom264,Colom266,Colom267,Colom272,Colom273,Colom274,Colom275,Colom276,Colom277,Colom303,Colom304,Colom306,Colom307,Colom309,Colom310,Colom315,Colom316,Colom320,Colom321,Colom323,Colom324,Colom329,Colom330,Colom331,Colom332,Colom333,Colom334,Colom348,Colom349,Colom351,Colom352,Colom354,Colom355,Colom360,Colom361,Colom365,Colom366,Colom368,Colom369,Colom374,Colom375,Colom376,Colom377,Colom378,Colom379,Colom393,Colom394,Colom396,Colom397,Colom399,Colom400,Colom406,Colom407,Colom410,Colom411,Colom413,Colom414,Colom419,Colom420,Colom421,Colom422,Colom423,Colom424,Colom438,Colom439,Colom441,Colom442,Colom444,Colom445,Colom450,Colom451,Colom455,Colom456,Colom458,Colom459,Colom464,Colom465,Colom466,Colom467,Colom468,Colom469,Colom483,Colom484,Colom486,Colom487,Colom489,Colom490,Colom495,Colom496,Colom500,Colom501,Colom503,Colom504,Colom509,Colom510,Colom511,Colom512,Colom513,Colom514,Colom528,Colom529,Colom531,Colom532,Colom534,Colom535,Colom540,Colom541,Colom545,Colom546,Colom548,Colom549,Colom554,Colom555,Colom556,Colom557,Colom558,Colom559 FROM [Sheet1$]"
Dim objCmdSelect As OleDbCommand = New OleDbCommand(expr, ExcelConnection)
Dim objDR As OleDbDataReader
Dim SQLconn As New SqlConnection()
Dim ConnString As String = "data source=" & DBServer1 & "; initial catalog=" & DBDataBase1 & "; persist security info=true; user id=" & DBUSer1 & "; password=" & DBPass1
SQLconn.ConnectionString = ConnString
SQLconn.Open()
Using bulkCopy As SqlBulkCopy = New SqlBulkCopy(SQLconn)
bulkCopy.DestinationTableName = "TMP_UPLOAD_ISP"
Try
objDR = objCmdSelect.ExecuteReader
bulkCopy.WriteToServer(objDR)
objDR.Close()
SQLconn.Close()
Catch ex As Exception
MsgBox(ex.ToString)
End Try
How can I Select column that I want to select (220 columns intermittently).
You are using the ACE provider and that means you are bound by the limitations of MS Access. Access has a maximum field limit of 255 (see: Access 2016 specifications).
Depending on the source worksheet, you could use a range query to retrieve multiple result sets and then merge them.
Select * From [SheetName$A1:IU]
then
Select * From [SheetName$UI1:SQ]
Column "UI" is column number 255 and column "SQ" is column number 511.
If you are not familiar with this range syntax, omitting the row on the end column tells it to seek from the starting row until no data is found in the column range.

vb.net Using SqlConnection doesn't free the file from use

I am working with a local .mdf file and I execute some queries to the database and I am using USING blocks to make sure the SqlConnection and SqlReader are disposed of correctly.
I then try to read the data of the file to generate a MD5 Hash of the file but it says the file is still in use.
The code isn't the cleanest it is my first time working with the Sql in a VB.NET app.
SQL Insert:
Dim finalW As String = ""
Dim finalO() As String
Dim currentcounter As Integer = 0
For Each Dir As String In System.IO.Directory.GetDirectories(Pathfinder)
Dim dirInfo As New System.IO.DirectoryInfo(Dir)
Dim temp As New List(Of String)
For Each currentFile In Directory.GetFiles(Pathfinder & "\" & dirInfo.Name & "\", "*.png", SearchOption.TopDirectoryOnly)
temp.Add(Path.GetFileName(currentFile))
Next
If temp.Count <> 0 Then
finalW = temp.Find(AddressOf GetNewIcon)
finalO = temp.FindAll(AddressOf GetOldIcon).ToArray
If finalW <> "" Then
Using con As New SqlClient.SqlConnection("Data Source=(LocalDB)\MSSQLLocalDB;AttachDbFilename=""" & PathFinal & "ImaginiDB.mdf"";Integrated Security=True")
con.Open()
Using cmd = con.CreateCommand()
cmd.CommandType = CommandType.Text
cmd.CommandText = "INSERT NewIcon (Name) VALUES ('" & finalW.Trim() & "')"
cmd.Connection = con
cmd.ExecuteNonQuery()
End Using
currentcounter = currentcounter + 1
Dim Id As String = ""
Using command = con.CreateCommand()
command.CommandType = CommandType.Text
command.CommandText = "SELECT * FROM NewIcon WHERE Name='" & finalW & "'"
Using reader As SqlDataReader = command.ExecuteReader()
While reader.Read()
Id = reader(0)
End While
End Using
End Using
For Each item As String In finalO
Using cmd2 = con.CreateCommand()
cmd2.CommandType = CommandType.Text
cmd2.CommandText = "INSERT OldIcon (NID,Name) VALUES ('" & Id & "','" & item.ToString.Trim() & "')"
cmd2.Connection = con
cmd2.ExecuteNonQuery()
End Using
currentcounter = currentcounter + 1
Next
Dim cur As Long = currentcounter * 100 / counter
SetProgress(cur)
End Using
End If
End If
Next
GC.Collect()
GC.WaitForPendingFinalizers()
SetLabel4Text("FINISHED IMPORT", Color.Red)
MD5 Generation ran after this process is finished:
Public Function GenMD5(ByVal Filename As String) As String
Dim MD5 = System.Security.Cryptography.MD5.Create
Dim Hash As Byte()
Dim sb As New System.Text.StringBuilder
Using st As New IO.FileStream(Filename, IO.FileMode.Open, IO.FileAccess.Read, IO.FileShare.Read)
Hash = MD5.ComputeHash(st)
End Using
For Each b In Hash
sb.Append(b.ToString("X2"))
Next
Return sb.ToString
End Function
So as mentioned in the comment by #jmcilhinney
Different connections use different pool as the MSDN says:
When a connection is first opened, a connection pool is created based
on an exact matching algorithm that associates the pool with the
connection string in the connection.
Thus I decided to implement the method:
SqlConnection.ClearPool(connection As SqlConnection)
I placed this just before my END USING:
Dim cur As Long = currentcounter * 100 / counter
SetProgress(cur)
SqlConnection.ClearPool(con)
End Using

SQL Update not updating records in access

Trying to update the values of Comment and ProgressValue inside a table in access. The message box at the end pops up but none of the values are changed.
Sub UpdateWeeklyReport()
Dim con As OleDbConnection
Dim com As OleDbCommand
con = New OleDbConnection("provider=microsoft.jet.oledb.4.0;data source=C:\ProjectDatabase.mdb")
com = New OleDbCommand("Update WeeklyReport Set Comment = #Comment, ProgressValue = #ProgressValue Where [EntryDate]='" & CBDate.SelectedValue.ToString & "' AND [AdminNo]=" & CBAdmin.SelectedValue & " AND [ClassCode]='" & CBClass.SelectedItem.ToString & "'", con)
con.Open()
com.Parameters.Add("#Comment", OleDbType.LongVarChar).Value = txtComment.Text
com.Parameters.Add("#ProgressValue", OleDbType.Integer).Value = CBProgress.Text
com.ExecuteNonQuery()
con.Close()
MessageBox.Show("Report Changed")
intialconnection()
End Sub
End Class

Excel to Sql adding certain rows to sql server

cmd.CommandText = (Convert.ToString("SELECT * From [") & excelSheet) & "] Order By" & columnCompany & "OFFSET 0 ROWS FETCH NEXT 100000 ROWS ONLY "
I am trying to map data form excel to an sql table but the excel table has a lot of records in it, so I run out of memory if I try to add all of the records at once. So what I am trying to do is use this line of code
cmd.CommandText = (Convert.ToString("SELECT * From [") & excelSheet) & "] Order By B OFFSET 0 ROWS FETCH NEXT 100000 ROWS ONLY "
So that 100000 row will be added. Then I intend to clear the data table and do the same only from 100001 to 200000 and so on.
But I cant get this to work correctly, it works fine when it is just SELECT TOP 100000 FROM... but then I dont know how to get the next 100000 record. I think the Order By is where the issue is as I'm not sure how to get the correct column.
Here is my code Below
Private Sub btnMap_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles btnMap.Click
'Allows columns to be moved mapped in the correct order
Dim nbColumnsToTransfer As Integer = dgvMapped.Columns.GetColumnCount(1)
Dim indexes As List(Of Integer) = (From column As DataGridViewColumn In dgvLoadedata.Columns.Cast(Of DataGridViewColumn)() _
Take nbColumnsToTransfer _
Order By column.DisplayIndex _
Select column.Index).ToList()
Dim columnCompany As Integer = indexes(0)
Dim columnEmail As Integer = indexes(1)
Dim columnFirstName As Integer = indexes(2)
Dim columnSurname As Integer = indexes(3)
Dim columnPostCode As Integer = indexes(4)
Dim columnTelephone As Integer = indexes(5)
Dim columnEmployeeBand As Integer = indexes(6)
Dim DeleteConnection As New Data.SqlClient.SqlConnection
DeleteConnection.ConnectionString = "Data Source=DC01\SQLEXPRESS;Initial Catalog=YCM;User ID=sa;Password=S0rtmypc!"
DeleteConnection.Open()
Dim sqlDTM As String = "Delete From TempMapped"
Dim command As New SqlCommand(sqlDTM, DeleteConnection)
command.ExecuteNonQuery()
Dim sqlDTSR As String = "Delete From TempSplitReq"
command = New SqlCommand(sqlDTSR, DeleteConnection)
command.ExecuteNonQuery()
DeleteConnection.Close()
If chkContactSplitReq.Checked = True Then
Dim dt As New DataTable()
Using con As New OleDbConnection(conString)
Using cmd As New OleDbCommand()
Using oda As New OleDbDataAdapter()
cmd.CommandText = (Convert.ToString("SELECT * From [") & excelSheet) & "] Order By" & columnCompany & "OFFSET 0 ROWS FETCH NEXT 100000 ROWS ONLY ""
cmd.Connection = con
con.Open()
oda.SelectCommand = cmd
oda.Fill(dt)
con.Close()
End Using
End Using
End Using
Dim connection As String = "Data Source=DC01\SQLEXPRESS;Initial Catalog=YCM;User ID=sa;Password=S0rtmypc!"
Using cn As New SqlConnection(connection)
cn.Open()
Using copy As New SqlBulkCopy(cn)
copy.ColumnMappings.Add(columnCompany, 0)
copy.ColumnMappings.Add(columnEmail, 1)
copy.ColumnMappings.Add(columnFirstName, 2)
copy.ColumnMappings.Add(columnPostCode, 3)
copy.ColumnMappings.Add(columnTelephone, 4)
copy.DestinationTableName = "TempSplitReq"
copy.WriteToServer(dt)
End Using
End Using
Dim SplitConnection As New Data.SqlClient.SqlConnection
SplitConnection.ConnectionString = "Data Source=DC01\SQLEXPRESS;Initial Catalog=YCM;User ID=sa;Password=S0rtmypc!"
SplitConnection.Open()
Dim sqlSplit As String = "TempSplitReq_SaveToMapped"
Dim commandSplit As New SqlCommand(sqlSplit, SplitConnection)
commandSplit.ExecuteNonQuery()
SplitConnection.Close()
Else
Dim dt As New DataTable()
Using con As New OleDbConnection(conString)
Using cmd As New OleDbCommand()
Using oda As New OleDbDataAdapter()
cmd.CommandText = (Convert.ToString("SELECT * From [") & excelSheet) + "] "
cmd.Connection = con
con.Open()
oda.SelectCommand = cmd
oda.Fill(dt)
con.Close()
End Using
End Using
End Using
Dim connection As String = "Data Source=DC01\SQLEXPRESS;Initial Catalog=YCM;User ID=sa;Password=S0rtmypc!"
Using cn As New SqlConnection(connection)
cn.Open()
Using copy As New SqlBulkCopy(cn)
copy.ColumnMappings.Add(columnCompany, 0)
copy.ColumnMappings.Add(columnEmail, 1)
copy.ColumnMappings.Add(columnFirstName, 2)
copy.ColumnMappings.Add(columnSurname, 3)
copy.ColumnMappings.Add(columnPostCode, 4)
copy.ColumnMappings.Add(columnTelephone, 5)
copy.DestinationTableName = "TempMapped"
copy.WriteToServer(dt)
End Using
End Using
End If
MsgBox("Data Mapping Complete")
End Sub
A SqlBulkCopy object has a BatchSize property. I have a similar situation and set
copy.BatchSize = 1000
And mine works with a large spreadsheet import. Adjust this property in your environment for better performance.

Populating Datagrid

I am creating a pageant scoring system,
I have this functions for populating my datagridview's first column using query:
Public Sub searchContestant()
If comboCategory.SelectedIndex <> 0 Then
Dim id As Integer = Login.JudgeBindingSource1.Current("Cont_id")
Dim critID As Integer = Convert.ToInt32(lblID.Text)
'search the contest record by the contest_id
Dim con As New SqlConnection
Dim reader As SqlDataReader
Dim reader2 As SqlDataReader
Dim dt = New DataTable()
Dim nc As New DataGridViewTextBoxColumn
DataGridView1.Columns.Clear()
DataGridView1.Rows.Clear()
Try
con.ConnectionString = "Data Source=BROWNIE\SQLEXPRESS;Initial Catalog=PSS;Integrated Security=True"
Dim cmd As New SqlCommand("SELECT * from Criteria where Cat_id = '" & critID & "' ", con)
Dim cmd2 As New SqlCommand("SELECT * from Contestant where Cont_id = '" & id.ToString() & "' ", con)
con.Open()
' Execute Query
reader = cmd.ExecuteReader()
nc.Name = "Contestants"
nc.Width = 250
nc.ReadOnly = True
DataGridView1.Columns.Add(nc)
While reader.Read()
nc = New DataGridViewTextBoxColumn
nc.Name = "(" & reader.GetString(2) & "%)" & reader.GetString(1)
nc.HeaderText = "(" & reader.GetString(2) & "%)" & reader.GetString(1)
nc.Width = 150
DataGridView1.Columns.Add(nc)
End While
Catch ex As Exception
MessageBox.Show("Error while connecting to SQL Server." & ex.Message)
Finally
con.Close() 'Whether there is error or not. Close the connection. '
End Try
Try
'populate contestant rows
con.ConnectionString = "Data Source=BROWNIE\SQLEXPRESS;Initial Catalog=PSS;Integrated Security=True"
Dim cmd2 As New SqlCommand("SELECT * from Contestant where Cont_id = '" & id.ToString() & "' ", con)
con.Open()
reader2 = cmd2.ExecuteReader()
While reader2.Read()
nc = New DataGridViewTextBoxColumn
nc.Width = 100
DataGridView1.Rows.Add(reader2.GetString(2).ToString())
End While
Catch ex As Exception
MessageBox.Show("Error while connecting to SQL Server." & ex.Message)
Finally
con.Close() 'Whether there is error or not. Close the connection. '
End Try
End If
End Sub
And for succeeding columns :
Public Sub searchCriteria()
Dim con As New SqlConnection
Dim reader4 As SqlDataReader
Dim dt = New DataTable()
Dim nc As New DataGridViewTextBoxColumn
DataGridView1.Columns.Clear()
Try
'populate Criteria
con.ConnectionString = "Data Source=BROWNIE\SQLEXPRESS;Initial Catalog=PSS;Integrated Security=True"
Dim cmd As New SqlCommand("SELECT * from Criteria where Cat_id = '" & lblID.Text & "' ", con)
con.Open()
reader4 = cmd.ExecuteReader
While reader4.Read
nc = New DataGridViewTextBoxColumn
nc.Name = reader4.GetString(1).ToString & Chr(13) & reader4.GetString(2).ToString & "%"
nc.Width = 100
DataGridView1.Columns.Add(nc)
End While
Catch ex As Exception
MessageBox.Show("Error while connecting to SQL Server." & ex.Message)
Finally
con.Close() 'Whether there is error or not. Close the connection. '
End Try
End Sub
This is the output:
And everytime a judge login, I insert data into the system with this function:
Public Sub makeTransaction()
Dim con2 As New SqlConnection
'get the judge id upon logging in
Dim judgeID = Login.JudgeBindingSource1.Current("Judge_id")
'get the contest id of the judge
Me.JudgeTableAdapter.FillByJudgeID(Me.PSSDataSet.Judge, judgeID)
Dim JudgeContestid = Login.JudgeBindingSource1.Current("Cont_id")
Me.ContestantTableAdapter.FillByContestID(Me.PSSDataSet.Contestant, JudgeContestid)
'get the contestant id
Me.ContestTableAdapter.FillByContestID(Me.PSSDataSet.Contest, JudgeContestid)
Dim contestID As Integer = ContestBindingSource.Current("Cont_id")
'get the category of the contest
Me.CategoryTableAdapter.FillByContestID(Me.PSSDataSet.Category, Convert.ToInt32(contestID))
For Each Category As DataRow In Me.PSSDataSet.Category.Rows
Me.CriteriaTableAdapter.FillByCategoryID(Me.PSSDataSet.Criteria, Category("Cat_id"))
For Each Contestant As DataRow In Me.PSSDataSet.Contestant.Rows
For Each Criteria As DataRow In Me.PSSDataSet.Criteria.Rows
TransactionsTableAdapter.Insert(JudgeContestid, Contestant("Con_id"), Criteria("Cri_id"), 0)
Next
Next
Next
End Sub
Now my problem is, how can I populate the datagrid using the data recently made by MakeTransaction() function?